Noise or Signal: The Role of Image Backgrounds in Object Recognition
Authors: Kai Yuanqing Xiao
Logan Engstrom
Andrew Ilyas
Aleksander Madry
Venue: ICLR 2021 Poster (2021)
Keywords: Backgrounds
Model Biases
Robustness
Computer Vision

Abstract: We assess the tendency of state-of-the-art object recognition models to depend on signals from image backgrounds. We create a toolkit for disentangling foreground and background signal on ImageNet images, and find that (a) models can achieve non-trivial accuracy by relying on the background alone, (b) models often misclassify images even in the presence of correctly classified foregrounds--up to 88% of the time with adversarially chosen backgrounds, and (c) more accurate models tend to depend on backgrounds less. Our analysis of backgrounds brings us closer to understanding which correlations machine learning models use, and how they determine models' out of distribution performance.


Published as a conference paper at ICLR 2021
NOISE OR SIGNAL: THE ROLE OF IMAGE BACK- GROUNDS IN OBJECT RECOGNITION
Kai Xiao, Logan Engstrom, Andrew Ilyas, Aleksander Madry
MIT
(kaix, rngstrom, a yas, madry) @mi edu
ABSTRACT
We assess the tendency of state-of-the-ar object recognition models to depend on signals from image backgrounds. We create toolkit for disentangling foreground and background signal on ImageNet images, and find that (a) models can achieve non-trivial accuracy by relying on the background alone, (b) models often misclas- sify images even in the presence of correctly classified foregrounds-up to 88% of the time with udversarially chosen backgrounds, and (c) more accurate models tend to depend on backgrounds less. Our analysis of backgrounds brings us closer to understanding which correlations machine learning models use, and how they determine models' out of distribution performance.
INTRODUCTION
Object recognition models are typically trained to minimize loss on given dataset, and evaluated by the accuracy they attain on the corresponding test set. In this paradigm, model performance can be improved by incorporating any generalizing correlation between images and their labels into decision-making. However, the actual model reliability and robustness depend on the specific set of correlations that is used, and on how those correlations are combined. Indeed, outside of the training distribution, model predictions can deviate wildly from human expectations either due to relying on correlations that humans do not perceive (Jetley et al. 20188] Ilyas al. 2019] Jacobsen al. 2019), or due to overusing correlations, such as texture (Geirhos et al. 2019: Baker et al. 2018) and color (Yip & Sinha 2002), that humans do use (but a lesser degree). Characterizing the correlations that models depend on thus has important implications for understanding model behavior, in general.
Image backgrounds are a natural source of correlation between images and their labels in object recognition. Indeed, prior work has shown that models may use backgrounds in classification (Zhang
al.
2007 Ribeiro al., 2016;] Zhu et al., 2017 Rosenfeld al., 2018 Zech al.) 2018 Barbu
et al. 2019: Shetty et al. 2019 Sagawa et al. 2020 Geirhos et al. 2020). and suggests that even human vision makes use of image context for scene and object recognition (Torralba 2003). this work, aim to obtain a deeper and more holistic understanding of how current state-of-the-arr image classifiers utilize image backgrounds. To this end, in contrast to most of the prior work (which tends study relatively small and often newle-curated image datasets"). our focus is on ImageNet (Russakovsky et al. (013)- one of the largest and most widely used datasets, with state-of-the-ar training methods, architectures, and tre-trained models tuned work well for it.
Zhu et al. (017)) analyze ImageNet classification (focusing on the older, AlexNet model) to find that AlexNet achieves small but non-trivial test accuracy on dataset consisting of only backgrounds (where foreground objects are replaced by black rectangles). While sufficient for establishing that backgrounds can be used for classification, we aim to go beyond those initial explorations to get a more fine-grained understanding of the relative importance of backgrounds and foregrounds, for newer, of-e-oo-thhe-a models, and to provide a versatile toolkit for others to use. Specifically, we investigate the extent to which models rely on backgrounds, the implications of this reliance, and how models' use of backgrounds has evolved over time. Concretely:
We create a suite of datasets that help disentangle (and control for different aspects of) the impact of foreground and background signals on classification. The code and datasets
We discuss these works in greater detail in Section Related Works.

---

Published as a conference paper at ICLR 2021
are publicly available for others to use in this repository:
https://github..
Using the aforementioned toolkit, we characterize models' reliance image backgrounds. We find that image backgrounds alone suffice for fairly successful classification and that changing background signals decreases average-case performance. In fact, we further show that by choosing backgrounds in an adversarial manner, we can make standard models misclassify 88% images as the background class.
We demonstrate that standard models not only use but require backgrounds for correctly classifying large portions of test sets (35% on our benchmark).
We study the impact of backgrounds on classification for variety of classifiers, and find that models with higher ImageNet test accuracy tend to simultaneously have higher accuracy on image backgrounds alone and have greater robustness changes image background.
METHODOLOGY
To properly gauge image backgrounds' role in image classification, construct synthetic dataset for disentangling background from foreground signal: ImageNet-9.
Original
Only-BG-B BG-B
Only-BG-T
No-FG
insect
bird
insect
bird
Only-FG
Mixed-Same
Mixed-Ramd
Mixed-Nes
instrument
insect
insect
instrument
Figure 1: Variations the synthetic dataset ImageNet-9, as described Table 1 We label each image with its pre-trained ResNet-50 dlssifiiaaionnggrennt if corresponding with the original label; red, if not. The model correctly classifies the image as "insect" when given: the original image, only the background, and two cases where the original foreground present but the background changes. Note that, in particular, the model fails in two cases when the original foreground is present but the background changes in MIXED-NEXT ONLY-FG).
Base dataset: ImageNet-9. We organize a subset of ImageNet into new dataset with nine coarse-grained classes and call it ImageNet-9 (IN-9) To create it, we group together ImageNet classes sharing an ancestor in the WordNet (Miller 1995) hierarchy. We use coarse-grained classes because there are not enough images with annotated bounding boxes (which we need to disentangle backgrounds and foregrounds) to use the standard labels. The resulting IN-9 dataset is dlas-bblaacedd and has 45405 training images and 4050 testing images. While we can (and do) apply our methods on the full ImageNet dataset as well, we choose to focus this searse-graines version of ImageNet because of its higher-fil fidelity images. We describe the dataset creation process detail and discuss the advantages of focusing IN-9 in Appendix
"heee classes are dog, bi rd, vehicle, repti 1e, carnivore, insect, i instrument, primate,
and fi sh.
2

---

Published as a conference paper at ICLR 2021
Variations of ImageNet-9 From this base set of images, which call the ORIGINAL version of IN-9, we create seven other synthetic variations designed to understand the impact backgrounds. We use both rectangular bounding boxes and the foreground segmentation algorithm GrabCut (Rother et , 2004), as implemented in OpenCV, to disentangle backgrounds and foregrounds. We visualize these variations in Figure [ and provide detailed reference in Table These subdatasets of IN-9 differ only in how they process the foregrounds and backgrounds of each constituent image.
Larger dataset: IN-9L We finally create dataset called IN-9L that consists of all the images in ImageNet corresponding to the classes in ORIGINAL (rather than just the images that have associated bounding boxes). This dataset has about 180k training images total. We leverage this larger dataset to train better generalizing models, and prefer to analyze models trained on IN-9I. whenever possible.
Table 1: The 8 modified subdatasets created from ImageNet-9, which are visualized in Figure The foreground detection method refers how the pixels corresponding to the foreground are found. GrabCut refers to the foreground segmentation algorithm implemented in OpenCV. Random backgrounds the last three datasets are taken from ONLY-BG-T, For more details see Appendix
Name
Foreground
Background
Foreground Detection Method
ORIGINAL
Unmodified
Unmodified
ONLY-BG-I
Black
Unmodified
Bounding Box
ONLY-BG-T
Tiled background
Unmodified
Bounding Box
No-FG
Black
Unmodified
GrabCut
ONLY-FG
Unmodified
Black
GrabCut
MIXED-SAME SAME
Unmodified
Random BG of the same class Random BG of random class
GrabCut
MIXED-RAND
Unmodified
GrabCut
MIXED-NEXT
Unmodified
Random BG of the next class
GrabCut
QUANTIFYING RELIANCE ON BACKGROUND SIGNALS
With ImageNet-9 in hand, we now assess the role of image backgrounds in classification.
100
Testing dataset
80
Same as train
Test Accuracy
I ORIGINAL
60
40
20
0
ONLY-BG-B
ONLY-BG-T Training dataset
No-FG
Figure 2: We train models on each the "background-only" datasets, then evaluate each on its corresponding test well the ORIGINAL test set. Even though the model only learns from background signal, it achieves (much) better than random performance on both the corresponding test and ORIGINAL. Here, random guessing would give 11.11% (the dotted line).
Backgrounds suffice for classification. Prior work has found that models are able make accurate predictions based on backgrounds alone; we begin by directly quantifying this ability. Looking at the ONLY-BG-T, ONLY-BG-B, and No-FG datasets, background-only training sets generalize reasonably well to both their corresponding test sets and unmodified images from the ORIGINAL test set (around 40-50% for every model, far above the
find (cf. Figure 2 that models trained these

---

Published as a conference paper at ICLR 2021
Table 2: Performance of state-of-the-art computer vision models on selected test sets of ImageNet- 9. We include both pre-trained ImageNet models and models of different architectures that we train on IN-9L. The BG-GAP is defined as the difference test accuracy between MIXED-SAME and MIXED-RAND and helps assess the tendency of such models to rely on background signal. Architectures are sorted by their test accuracies on ImageNet and ORIGINAL for pre-trained and IN-9L-trained models, respectively. Shaded in grey are the two architectures that can be directly compared across datasets (ResNet-50 and
Pre-trained ImageNet
Trained IN-9L
MobileNet-13
EfficientNet-b
ResNeet50
WRN- 50x2
ShuffleNer
ResNet- 550
WRN-50x2
VGG16-BN
DPN-02
AlexNet
Test dataset
ImageNet 67.9%
77.2%
77.6%
78.5%
80.0%
ORIGINAL
95.5%
96. 1%
96.9%
96.6%
97.2%
86.7%
95.7%
96.3%
97.2%
97.6%
ONLY-BG-T 16.3%
16.5%
17.4%
18.8%
17.6%
41.5%
43.6%
43.6%
45.1%
45.7%
MIXED-S SAME
84.0%
86.2%
91.0%
88.3%
90.5%
76.2%
86.7%
89.9%
90.6%
91.0%
MIXED-RAND
73.2%
76.3%
84.3%
81.4%
86.1%
54.2%
69.4%
75.6%
78.0%
78.0%
BG-gap 10.8%
9.9%
6.7%
6.9%
4.4%
22.0%
17.3%
14.3%
12.6%
13.0%
baseline of 11 1 representing random classification). Our results confirm that image backgrounds contain signal that models can accurately classify standard images with.
Models exploit background signal for classification. We discover that models can misclassify due to background signal, especially when the background class does not match that of the foreground. As a demonstration, we study model accuracies on the MIXED-RAN RAND dataset, where image backgrounds randomized and thus provide no information about the correct label. By comparing test accuracies on MIXED MIXED-RAND and MIXED-SAME SAME where images have coasssconns backgrounds, we can measure classifiers' dependence on the correct background. We denote the resulting accuracy gap between MIXED-SAME and MIXED-RAND as the BG-GAP; this difference represents the drop in model accuracy due to changing the class signal from the background. In Table 2 we observe BG-GAP of 13-22% and 4-11% for models trained on IN-9L and ImageNet, respectively, suggesting that backgrounds often mislead state-of-the-an models even when the correct foreground is present.
More Training Data can reduce the BA-GAP Our results indicate that ImageNet-trainee models are less dependent on backgrounds than their IN-9L-trained counterparts- they have smaller (but still sigmificaat) BG-GAP, and perform worse when predicting solely based on backgrounds (i.e., on the ONLY-BG-T dataset). We explore two ways that ImageNet differs from IN-91. to understand this Aienomena-Imann ImageNet has (a) more datapoints than IN-91., and (b) more fine-grat fine-grained class structure. Figure FigureBJshow shows that more training data reduces the BG-GAP, particularly when the training dataset size approaches the size of ImageNet. This indicates that training much more data (and thus, more backgrounds) can reduce (but not eliminate) the effect of backgrounds on model predictions. An ablation study of ImageNet's more fine- fine-grained class structure does not find strong evidence supporting its helpfulness (cf. Appendix B)
Models are vulnerable to adversarial backgrounds. To understand how worst-case backgrounds impact models' performance, we evaluate model robustness to adversarially chosen backgrounds. We find that 88% foregrounds are susceptible to such backgrounds; that is, for these foregrounds, there is a background that causes the classifier to classify the resulting irreground-backgroung as the background class. For a finer grained look, we also evaluate image backgrounds based on their attack success rate (ASR), i.c., how frequently they cause models to predict the (background) class in the presence of a conflicting foreground class. As an example, Figure 4 shows the five backgrounds with the highest ASR for the insect class-these backgrounds (extracted from insect images in ORIGINAL) fool a --- ResNet-50 model into predicting insect on up to 52% of
combination
"MIXD--SAM controls for artifacts from image processing presented in MIXED-RANE RAND. For further discussion, see Appendix D
4

---

Published as a conference paper at ICLR 2021
Effects of Training on More Data
80
Test Dala
Tes:: Accuracy
70
Mixed-Same
60
ImageNet-99
Imageliet
50
Full Size
40
Full IN-91 Size
30
Dataset size relative to imageNet-9
Figure 3: We compare test accuracies on MIXED-SAME and MIXED- RAND and observe that training with more data reduces the BG-GAP (BG-GAP measures the effect of backgrounds on model predictions). While this trend is true for models trained on both IN-9 and ImageNet, the trend is most noticeable for models trained on the largest training set, the full ImageNet dataset-this is shown on the far right side of the graph.
non-insect foregrounds. We plot histogram of ASR over all insect backgrounds in Figure 24 of the Appendix-it has long tail. Similar results are observed for other classes well (ef. Appendix G
42%
44%
45%
45%
52%
insect, 66 55%
Figure 4: The adversarial backgrounds that most frequently fool IN-9L-trained models into classify- ing a given foreground as insect, ordered by the percentage of foregrounds fooled. The total portion of images that can be fooled (by any background from this class) is 66.55%.
Training on MIXED-RAND reduces background dependence. Next, we explore how to reduce models' dependence on background. To this end, we train models on MIXED-RAME -ANDD. a synthetic dataset where background signals are decorrelated from class labels. As we would expect, MIXED- RAND-trained models extract less signal from backgrounds: evaluation results show that MIXED- RAND models perform poorly (15% accuracy-baree higher than random) on datasets with only backgrounds and no foregrounds, (ONLY-BG-1 ONLY-BG-B).
Indeed, such models are also more accurate datasets where backgrounds do not match foregrounds. In Figure we observe that a MIXED-R MIXED-RAN RAND, and 22.3% higher accuracy on MIXED-NEXT dataset where background signals classsconsy mismatch foregrounds. (Recall that MIXED- NEXT images have foregrounds from class y mixed with backgrounds from class + labeled as class y.) The MIXED-RE BEIED--RAD-URRR RAND-trained model also has little variation (at most 3.8%) in accuracy across all five test sets that contain the correct foreground.
ORIGINAT.-traing counterpart MIXED-RAND-traee AND-trained model has 17.3% higher accuracy than its
Qualitatively, the MIXED-RAND- RAND-trained model also appears to place more relative importance on foreground pixels than the ORIGINAL- model; the saliency maps of the models in Figure show that the MIXED-R AnD-rraiedd model's saliency maps highlight more foreground pixels than those of ORIGINAI-trained models.
A fine grained look at dependence backgrounds. We analyze models' reliance back- grounds at an baage-b--imae level and ask: for which images does introducing backgrounds help hurt classifiers' performance? To this end, for each image in ORIGINAL,, decompose
5

---

Published as a conference paper at ICLR 2021
100
80
Test Accuracy
I MIXED-IN -EXT MIXED-RAND RAND MIXED-SANE ONLY-FG
60
40
ORIGINAL
20
0
MIXED-RAN -AND Training dataset
ORIGINAL
Figure 5: We compare the test performance of model trained on the synthetic MIXED-RANT dataset with a model trained on ORIGINAL. We evaluate these models on variants of IN-9 that contain identical foregrounds. For the ORIGINAL-trained model, test performance decreases significantly when the background signal modified during testing. However, the MIXED-RE MIXED-RAND-traiie AND-traieed model is robust background changes, albeit the cost of lower accuracy images from ORIGINAL..
Image
Original Saliency
Mixed-Rand Saliency
Image
Original Saliency Mixed-Rand Saliency
Figure 6: Saliency maps for the the ORIGINAL and MIXED-RAME models on two images. As expected, the LIXED-RAND model appears to place more importance on foreground pixels.
how models use foreground and background signals by examining classifiers' predictions on the corresponding image in -AXED-RAND and ONLY-BG-T. Here, ONLY-BG-T predictions as a proxy for which class the foreground and background signals (alone) point towards, respectively. We categorize each image based on how its background and foreground signals impact classification; we list the categories in Table and show the counts for each category as a histogram per classifier in Figure 7 Our results show that while few backgrounds induce misclassification (see Appendix H for examples), large fraction of images require backgrounds for correct dlasification-apppoiimaaaaa
the MIXED-R R AND and
35% on the ORIGINAL trained classifiers, as calculated by combining the "BG Required" and "BG+FG Required" categories.
Further insights derived from IN-9 are discussed in the Appendix We focus on key findings in this section, but also include more comprehensive results and examples of other questions that can be explored by using the toolkit of IN-9 in the Appendix.
4 BENCHMARK PROGRESS AND BACKGROUND DEPENDENCE
In the previous sections, we demonstrated that standard image classification models exploit signals from backgrounds. Considering that these models result from progress on standard computer vision benchmarks, a natural question is: what extent have improvements image classification benchmarks resulted from exploiting background correlations? And relatedly, how has model robustness misleading background signals evolved over time?
As first step towards answering these questions, we study the progress made by ImageNet models synthetic IN-9 dataset variations. In Figure g we plot accuracy our synthetic datasets against ImageNet accuracy for each of the architectures considered. As evidenced by the lines of best fit in Figure accuracy increases the original ImageNet benchmark generally correspond to

---

Published as a conference paper at ICLR 2021
Table 3: Prediction categories we study for a given image-model pair. For a given image, model can make differing predictions based on the presence or absence of its gpeegroudd/bcckgrommg each possible case based on how the background classification relates to the full image classification and the foreground classification. To proxy classifying full images, foregrounds, and backgrounds separately, we classify ORIGINAL. MIXED-RAN and ONLY-BG-T (respectively). "BG Irrelevant" demarcates images where the foreground classification result is the same as that of the full image (in terms of correctness). We show illustrative examples BG Required and BG+FG Required below.
We label
Label
Correct Prediction
Correct Prediction Correct Prediction
on Full Image
Foreground
Background
BG Required
Fools
BG+FG Required
X
BG+FG Fools
BG Irrelevant
VIX
Vix
Original
Mixed-Rand
Only-BG-T
Original
Mixed-Rand
Only-BG-T
BG+FG Required
BG uirrdd
bird
insect
bird
fish
vehicle
bird
accuracy increases on all of the synthetic datasets. This includes the ONLY-BG datasets indicating that models do improve at extracting correlations from image backgrounds.
Indeed, the ONLY-B BG trend observed in Figure suggests that either (a) image classification models can only attain their reported accuracies in the presence of background signals; or (b) these models result of optimization technique,
carry an implicit bias towards features in the background, model class, etc.-in this case, we may need explicit regularization (e.g., through distributionally robust optimization (Sagawa et al. 2020) related techniques) to obtain models invariant to these background features. The ONLY- BG trend does not indicate that models are failing per se; could also indicate that models learn to depend backgrounds because they are necessary for correctly classifying certain images due to quirks in the ImageNet dataset.
Still, models' relative improvement in accuracy on classifying ONLY-BG-D at a slower (absolute) rate than MIXED-RAND. MIXED-S AAME and MIXED-NEXT. Furthermore, the performance gap between the MIXED datasets and the others (most notably, between MIXED-RAM AND and MIXED-S AME; between MIXED- and MIXED-RAND: and consequently between MIXED-NEXT and MIXED-SAME) trends towards closing, indicating that models not only are becoming better at using foreground features, but also are becoming more robust misleading background features (MIXED- RAND and MIXED-NEXT) Finally, models also improve in accuracy faster on No-FG (which has foreground shape but no texture) than on ONLY-BG-T, which implies that better models are using foreground shape features more effectively.
dataset variants is promising- moddls improve
Overall, the accuracy trends observed from testing ImageNet models on our synthetic datasets reveal that better models (a) are capable of exploiting background correlations, but (b) are increasingly robust to changes in background, suggesting that invariance to background features may not necessarily come at the cost benchmark accuracy.
5 RELATED WORK
Prior works on contextual bias from image background show that background correlations can be predictive (Torralba 2003) and can influence model decisions. ZZhang al. (2007) find that (a) a bag-
'Here, we highlight works analyzing contextual hias from image backgrounds specifically, and discuss contextual hias generally and foreground segmentation in Appendix
7

---

Published as a conference paper at ICLR 2021
4,000
Number of examples
3,000
BG Irrelevant
BG Required
2,000
BG Fools
BG+FG Required
BG+FG Fools
1,000
BINND-RAND
ORIGINAL Training dataset
ONLY-BG-T
We categorize each test set image based on how a model classifies the full image, the background alone, and the foreground alone (cf. Table The model trained ORIGINAL needs the background for correct classification on 35% of images (measured by adding "BG Required" and "BG+FG Required), while a model trained on MIXED-RAN is much less reliant background. The model trained on ONLY-BG- requires the background most, as expected; however, the model often misclassifies both the full image and the background, so the "BG Irrelevant" subset is still sizable.
Figure 7:
Accuracy on synthetic datasets
0.8
ONLY-BG: BG-B
0.6
MIXED-NEX
MIXED-RAN
0.4
MIXED- -SME
No-FG
ResNet-50
0.2
MobileNet-v33s
0.55
0.6
0.65
0.75 ImageNet Accuracy
0.7
0.8
0.85
0.9
Figure 8: Measuring progress on each of the synthetic ImageNet-9 datasets with respect to progress on the standard ImageNet test set. Higher accuracy on ImageNet generally corresponds to higher accuracy each of the constructed datasets, but the rate which accuracy grows varies based on the types of features present in each dataset. Each pre-trained model corresponds to a vertical line on the plot-we we mark ResNet-50 and MobileNet-v3? models for reference.
of-features object detection algorithm depends on image backgrounds in the PASCAL dataset and (b) using this algorithm
training set with varying backgrounds leads to better generalization, [Beery et al. (2018) and Barbu et al. (2019) collect new test datasets of animals and objects, respectively. Barbu et (209)) focus on object classes that also exist in ImageNet, and their
contains front of unconventional backgrounds and in unfamiliar orientations. Both works show that computer vision models experience significant accuracy drops when trained on data with one set of backgrounds and tested on data with another, [Sagawa et al.) (2020) create synthetic dataset Waterbirds, where waterbirds and landbirds from one dataset are combined with water and land backgrounds from another. They show that a model's 's reliance on spurious correlations with the background can be harmful for small subgroups of data where those spurious correlations no longer hold (e.g, landbirds water backgrounds). [Rosenfeld et al. (2018) analyze background dependence for object detection (as opposed to classification) models on the MS-COCO dataset. They transplant object from one image another image, and find that object detection models may detect the transplanted object differently depending on its location, and that the transplanted object may also cause mispredictions other objects in the image. Zech al. (2018) study medical imaging and
objects photographed
8

---

Published as a conference paper at ICLR 2021
show that a model learned to detect hospital-specifii metal token medical scans. The model then used each hospital's pneumonia prevalence rate to predict pneumonia fairly well (without learning much about actually detecting pneumonia).
The only work that, similarly to us, studies a large-scale dataset in this context is Zhu et al. (201 77) who analyze ImageNet and show that AlexNet can achieve nontrivial accuracy on a dataset similar to our ONLY-BG- B dataset. While sufficient for establishing that backgrounds can be used for classification, this dataset also introduces biases by adding large black rectangular patches all of the images (which our ONLY-BG-T dataset fixes). In comparison to Zhu et al. (0117) and the other prior works, we: (a) properly segment foregrounds and backgrounds using the GrabCut algorithm instead of relying on rectangular bounding boxes; (b) create dataset variations that allow us to measure not just model performance without foregrounds, but also the relative influence of foregrounds and backgrounds on model predictions: (c) control for the effect of image artifacts by focusing on comparisons between the MIXED SAME and MIXED-RR ANDD datasets; (d) study model robustness to adversarial backgrounds; (e) study larger and more recent set of classifiers (He et al., 2
Zagoruyko ko & Komodakis) 2016; Tan Le 2019); (f) show how improvements they give on ImageNet relate to background dependence; and (g) make our benchmarking toolkit publicly accessible for others to use and build on.
6 DISCUSSION AND CONCLUSION
In this work, we study the extent to which classifiers rely on image backgrounds. To this end, we create a toolkit for measuring the precise role of background and foreground signal that involves constructing new test datasets that contain different amounts of each. Through these datasets we establish both the usefulness of background signal and the tendency of our models to depend on backgrounds, even when relevant foreground features are present. Our results show that our models are not robust to changes in the background, either in the adversarial case, or in the average case.
As most ImageNet images have human-recognizann recognizable foreground objects, models appear to rely on background more than humans on that dataset. The fact that models can be fooled by adversarial background changes 88% of all images highlights how poorly computer vision models may perform in an outto--distribution setting. However, contextual information like the background can still be useful in certain settings. After all, humans do use backgrounds as context in visual processing, and the background may be necessary if the foreground is blurry or distorted (Torralba 2003). Therefore, reliance on background is a nuanced question that merits further study.
On one hand, our findings provide evidence that models succeed by using background correlations, which may be undesirable in some applications. On the other hand, we find that advances in classifiers have given rise to models that use foregrounds more effectively and are more robust to changes in the background. To obtain even more robust models, we may want to draw inspiration from successes in training on the MIXED-RAN RAND dataset (a dataset designed to neutralize background signal-oo cf. Table D.. related data- augmmemtmmtnnio techniques (Shetty et al., 2019), and training algorithms like distributionally robust optimization (Sagawa et al., 2020) and model-based robust learning (Robey) cet al., 2020). Overall, the toolkit and findings in this work help us to better understand models and to monitor our progress toward the goal of reliable machine learning.
REFERENCES
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Deep convolutional networks do not classify based on global object shape. In PLOS Computational Biology, 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, C Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Neural Information Processing Systems, 2019.
Sara Beery, Grant van Hom, and Pietro Perona. Recognition in terra incognita. In European Conference on Computer Vision (ECCV), 2018.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. In IEEE Transactions on Image Processing, 2004.

---

Published as a conference paper at ICLR 2021
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference Learning Representations. 2019.
Robert Geirhos, fom-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. In arXiv preprint arXiv:2004.07780, 2020.
Michael Harville, Gaile Gordon, and John Woodfill. Foreground segmentation using adaptive mixture models in color and depth. In IEEE Workshop Detection and Recognition of Events in Video, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Andrew Hyas, Shibani Santurkar, Logan Engstrom, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Neural Information Processing Systems (NeurIPS), 2019.
Jom-Heerikk Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability. In International Contemporary on Learning Representations (ICLR), 2019.
Saumya Jetley, Nicholas Lord, and Philip Torr. With friends like these, who needs adversaries? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A. Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Computer Vision (ECCV), 2012.
George Miller. Wordnet: a lexical database for english, In Communications of the ACM, 1995.
Choi Myung Jin, Antonio Torralba, and Alan S. Willsky. Context models and out-of-context objects. In Pattern Recognition Letters, 2012.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. In International Conference Knowledge Discovery and Data Mining, 2016.
Alexander Robey, Hamed Hassani, and George J. Pappas. Model-based robust deep learning. In arXiv preprini arXiv:2005.1022 2020.
Amir Rosenfeld, Richard S. Zemel, and John K. Tsotsos. The elephant in the room. In arXiv preprint 180iv 03305, 2018
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. In ACM Transactions on Graphics, 2004.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. In International Journal of Computer Vision (IJCV. 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
Rakshith Shetty, Mario Fritz, and Bernt Schiele. Adversarial scene editing: Automatic object removal from weak supervision. Neural Information Processing Systems (NeurIPS), 2018.
Rakshith Shetty, Bernt Schicle, and Mario Fritz. Not using the car to see the sidewalk quantifying and sidewalk--uanttffing controlling the effects of context in classification and segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning (ICML), 2019.
10

---

Published as a conference paper at ICLR 2021
Antonio Torralba. Contextual priming for object detection. In International Journal of Computer Vision (IJCV), 2003.
Andrew W. Yip and Pawan Sinha. Contribution of color to face recognition. In Perception, 2002.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Generative image inpainting with contextual attention. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Sergey Zagonuyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), 2016.
John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. In PLoS Medicine, 2018.
Jianguo Zhang, Marcin Marszalek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. International Journal of Computer Vision (IJCV), 2007.
Zhuotun Zhu, Lingxi Xie, and Alan Yuille. Object recognition without and without objects. In International Joint Conference on Artificial Intelligence, 2017.

---

Published as a conference paper at ICLR 2021
A
DATASETS DETAILS
We choose the following 9 high-level classes.
Class
WordNet ID
Number of sub-classes
Dog
n02084071
116
Bird
n01503061
Vehicle
n04576211
42
Reptile
n01661091
Carnivore
n02075296
35
Insect
59955
27
Instrument
n03800933
26
Primate
n02469914
20
Fish
n02512053
16
Table 4: The 9 classes of ImageNet-9.
All datasets used in the paper are balanced by randomly removing images from classes that over-represented. We only keep as many images as the smallest post-modification synthetic dataset, so all synthetic datasets (except IN-9L) have the same number of images. We also use custom GUI to manually process the test set to improve data quality. For IN-9L, the only difference from using the corresponding classes
original ImageNet dataset that we balance the dataset.
For all images: we apply the following filters before adding each image to our datasets.
The image must have bounding box annotations.
For simplicity, each image must have exactly one bounding box. A large majority of images that have bounding box annotations satisfy this.
For images needing a properly segmented foreground: This includes the 3 MIXED datasets, ONLY-FG, and No-FG. We filter out images based on the following criteria.
Because images are cropped before they are fed into models, we require that less than 50% of the bounding box is removed by the crop, to ensure that the foreground still exists. Almost all images pass this filter.
The OpenCV foreground segmentation function cv2 grabCut (used extract the fore- ground shape) must work on the image. We remove images where it fails.
For the test set only, we manually remove images with foreground segmentations that retain significant portion of the background signal.
For the test set only, manually remove foreground segmentations that very bad (e.g. the segmentation selects part of the image, and that part doesn't contain the foreground object).
For images needing only background signal: This includes ONLY-BG-B and ONLY-BG-T. In this case, apply the following criteria:
The bounding box must not be too big (more than 90% of the image). The intent here is to
avoid ONLY-BG-B images being just a large black rectangle.
For the test only, we manually remove ONLY-BC images that still have an instance of the class even after removing the bounding box. This occurs when the bounding boxes are imperfect incomplete (e.g. only one of dogs in an image labeled with bounding box).
Creating the ONLY-BG-T dataset: We first make a "tiled" version of the background by finding the largest rectangular strip (horizontal vertical) outside the bounding box, and tiling the entire
12

---

Published as a conference paper at ICLR 2021
Image with Bounding Box
Tiled Background
Only-BG-T
Figure 9: Visualization of how ONLY-BG-T is created.
image with that strip. We then replace the removed foreground with the tiled background. visual example is provided in Figure We purposefully choose not use deep-learning-basee inpainting techniques such as (Shetty et al. 2018) to replace the removed foreground, as such methods could lead to biases that the inpainting model has learned from the data. For example, an inpainting model may learn that the best way to inpaint missing chunk of a flower is to place an insect there, which is something we want avoid.
Motivation for each IN-9 variation: We create ONLY-BG-B and ONLY-BG-T to remove the foreground completely, including the shape of the foreground object. We intend for ONLY-BG-B to be directly comparable to the prior work of Zhu et al. (2017) that uses similar methodology to evaluate older AlexNet models, while ONLY-BG-T a more natural-looking background that avoids black rectangles introduced in ONLY-BG-B.
The No-FG dataset created to retain the foreground shape, but not the texture. We can use it to assess the relative importance of foreground shape compared to foreground texture.
Finally, we create four datasets that have identical foregrounds but each have distinct background signals. ONLY-FG has a pure black background to go with the foreground. MIXED-SAME has MIXED-RAND has background signal
background signal from the same class as the foreground. from random class, it can be thought of as having neutral background signal. MIXED-NEXT has background signal from the next class, which will always be in conflict with the foreground. image processing pipeline equally present in
Any artifacts in the foreground that result from all four datasets. Thus, these datasets help to isolate how much backgrounds alone influence model predictions when the correct foreground exists in the image.
mull-ImageNet version of each synthetic variation: We also apply the same methodology for disentangling foreground and background signal to the entire ImageNet validation set, creating Full- Iall-ImageNet (Full-IN) versions of each of our 7 dataset variations.
We evaluate a pre- trained ResNet-50 on Full-IN for comparison in Table 5 and observe similar trends to ImageNet-9 that lead to similar conclusions on model background reliance. We choose to focus on ImageNet-9 results in the main paper because the following shortcomings of Full-IN.
Individual classes are quite small, as some classes have very few (or even zero) images that make it through our filters due to lack of proper annotated bounding boxes.
2. When bounding boxes do exist, their quality is often lower than those in the IN-9 classes. For example, many images of fruit contain multiple fruit, but only one will be properly annotated with bounding box.
3. When creating the MIXED-NEXT equivalent for Full-IN, the next class is often similar to the previous one. For example, many dog breeds occur consecutively in ImageNet. Thus, Full-IN's MIXED-NEL frequently has backgrounds that are similar to backgrounds from the foreground class.
13

---

Published as a conference paper at ICLR 2021
B EXPLAINING THE DECREASED BG-GAP OF PRE-TRAINED IMAGENET MODELS
We investigate two possible explanations for why pre-trained ImageNet models have a smaller BG- GAP than models trained on ImageNet-9. Understanding this phenomenon can help inform how models should be trained to be more background-robust We find slight improvements to background- robustness from training on more fine-grained classes. We find that training on larger datasets helps only slightly when the training dataset set size is smaller than IN-9I., but larger improvements occur when the training dataset size is bigger. Thus, we encourage training on larger datasets if reduced background robustness is the goal.
B. THE EFFECT OF FINE-GRAINEDNEN ON THE BG-GAP
One possible explanation is that training models to distinguish between finer- amer-grained classes forces them to focus more on the foreground, which contains relevant features for making those fine-grained distinctions, than the background, which may be fairly similar across sub-classes a high-level class. This suggests that asking models to solve more fine-grained tasks could improve model robustness to background changes.
To test the effect of fine- fie-grrineednes on ImageNet-9, make related dataset called IN-9LB that uses the same 9 high-level classes and can be cleanly modified into more fine-grained versions. Specifically, for IN-9LB 144 ImageNet classes. To create successively more fine-grained versions of the IN-9LB dataset, we group every n sub-classes together into higher-level class, for n {1, 8, 16). Here, corresponds to keeping all 144 ImageNet classes as they are, while n = 16 corresponds to only having 9 high-level classes, like ImageNet-9. Because we keep all images from those original ImageNet classes, this dataset is the same size as IN-9L.
choose exactly 16 sub-classes for each high-level class, for total of
1
We train models on IN-9LB at different levels of fine-grainedness and evaluate the BG-GAP of those models in Figure TO We find that fine-grained models have smaller BG-GAP as well as better performance on MIXED-NEXT, but the improvement is very slight and also comes at the cost of decreased accuracy on ORIGINAL. The BG-GAP of the most fine-grat tne-grained classifier is 2.3% smaller than the BG-GAP of the most coarse-gran goarseegr classifier, showing that fine-grainedness does improve mackground-robustness However, the improvement is still small compared to the size of the BG-GAP (which is 13.3% for the fine-grained classifier).
B.2 THE EFFECT OF LARGER DATASET SIZE ON THE BG-GAP
second possible explanation for why pre-trained ImageNet models have smaller BG-GAP is that training on larger datasets is important for background-robustres To evaluate this possibility. we train models on different-sized subsets of IN-9LBB The largest dataset we train on the full IN-9LB dataset, which is 4 times large as IN-9, and the smallest is 1/4 as large as IN-9, Figure II shows that increasing the dataset size does increase overall performance but only slightly decreases the BG-GAP.
Next, we train models on different-sizzi subsets of ImageNet; we use the pre-trained ResNet-50 ImageNet model for full-sized ImageNet, and we train ResNet-50 models subsets that are 1/2, 1/4, 1/8, 1/16, and 1/32 as large as ImageNet. In these cases, we observe in Figure [2 that training on more data does not help significantly when the training dataset sizes still small, but it does help more noticeably for models trained on 1/2 of ImageNet and all of ImageNet.
It is possible that having both fine-grained class structure and more training data simultaneously is important for background-robustnes Furthermore, more training data (from other classes that are not in IN-9L) may also be the cause of the increased mackrroundrrobustnes of pre-trained ImageNet models.
B.3 SUMMARY OF METHODS INVESTIGATED TO REDUCE THE BG-GAP
In Figure I3 we compare the BG-GAP of ResNet-50 models trained on different datasets and with different methods to ResNet-50 pre-trained ImageNet. We explore f--robust training, increasing
14

---

Published as a conference paper at ICLR 2021
Effects of Training on Fine-grained Data
100
90
Test Accuracy
Test Data
80
Mixed-Next
Mixed-Rand
70
Mixed-Same
Original
60
50
18
36
72
144
Number of Training Classes (log scale)
Figure 10: We train models on IN-9LB at different levels of fine-grainedness (more training classes is more fine-grained). The BG-GAP, or the difference between the test accuracies on MIXED-SAME and MIXED-RAND decreases as we make the classification task more fine-grained, but the decrease small compared to the size of the BG-GAP.
Effects of Training on More Data
80
Test Accuracy
Test Data
70
Mixed-Next Mixed-Rand
60
Mixed-Same
50
Original
40
30
0.25
0.50
1.00
2.00
4.00
Dataset size relative to ImageNet-9
Figure 11: We train models on different-sized subsets of IN-9LB. The largest training set we use the full IN-9LB dataset, which is 4 times larger than ImageNet-9, While performance all test datasets improves as the amount of training data increases, the BG-GAP has almost the same size regardless of the amount of training data used.
dataset size, and making the classification task more fine- graineri and find that none of these methods reduces the BG-GAP much as pre-training on ImageNet. The only method that reduces the BG-GAP significantly more is training MIXED-RAND AND. Furthermore, the same trends hold true we measure the difference between MIXED-S AME and MIXED-NEXT as opposed the BG-GAP (the difference between MIXED-SAME and MIXED-BR -AND).
15

---

Published as a conference paper at ICLR 2021
Effects of Training on More Data (Trained ImageNet)
100
Test Accuracy
Test Data
Mixed-Same
Mixed-Rand
Mixed-Next
Original
Full ImageNet Size
40
1
2
16
32
Dataset size relative to ImageNet-9
Figure 12: We train models on different-size subsets of ImageNet. We use pre-trained ResNet-50 for the rightmost datapoints corresponding to training the full ImageNet dataset, which is about 30 times larger than ImageNet-9. The BG-GAP begins to decrease when the training dataset set size sufficiently large.
Mixed-Same vs. Mixed-Rand Accuracy for Different Models
100
80
Test Accuracy
60
Mixed-Same
40
Mixed-Rand
IN
IN-9LB
20
IN-9L
IN-9
Pre-trained IN
IN-9LB Coarse
IN IN-9LB Fine
IN-9L
IN-9 (Mixed Rand)
IN-9
IN-9 (L2-robust)
IN-9 (Linf-robust)
Training Dataset (and Method)
Figure 13:
We compare various different methods of training models and measure their BG-GAP, the difference between MIXED-SAM and MIXED-RAND test accuracy. We find that (1) Pre- trained IN models have surprisingly small BG-GAP. (2) Increasing fine- mne-grainedness (IN-9LB Coarse vs. IN-9LB Fine) and dataset size (IN-9 vs. IN-9L) decreases the BG-GAP only slightly. (3) f,-robust training does not help. (4) Training on MIXED-RAME (cf. Section Iisioon]apee appeers to be the most effective strategy for reducing the BG-GAP, For such model. the MIXED-SAME and MIXED-RAN accuracies are nearly identical.
16

---

Published as a conference paper at ICLR 2021
C TRAINING AND EVALUATION DETAILS
For all models, use fairly standard training settings for ImageNet-style models. We train for 200 epochs using SGD with batch size of 256, a learning rate of 0.1 (with learning rate drops every 50 epochs), momentum parameter of 0.9, weight decay of 1e-4, and data augmentation (random resized crop, random horizontal flip, and color jitter). Unless specified, we always use a standard ResNet-50 architecture (He al. 2016). For the experiment depicted in Figure [T] we found that using a smaller learning rate of 0.01 was necessary for training to converge on the smallest training sets. Thus, we used that same learning rate for all models in Figure
When evaluating ImageNet classifiers on IN-9, we map all ImageNet predictions to their correspond- ing coarse- -rrsiid class in IN-9. For example, we map both gi ant schnauzer and Irish terrier to dog, and both goldf llldfish and tiger shark to FISH. If an ImageNet classifier outputs class that has no corresponding coarse-grained class IN-9, we consider the prediction incorrect.
D ADDITIONAL EVALUATION RESULTS
We include full results of training models every synthetic IN-9 variation and then testing them on every synthetie IN-9 variation T In addition to being more comprehensive, this table and these IN-9 variations can help answer variety of questions, of which we provide three examples here. Finally, we also evaluate a pre-trained model on Full-ImageNet (Full-IN) versions of each synthetic IN-9 variation for comparison.
How does more training data affect model performance with and without object shape?
We already show closely related results the effect of more training data the BG-GAP in Figure IT] Here, we compare model test performance on the No-FG and ONLY-BG- test sets. Both replace the foreground with black, but only No-FG retains the foreground shape.
By comparing the models trained on ORIGINAL and IN-91. -1 (4x more training data), we find that
1. The ORIGINAL-trained model performs about 9% better on No-FG than ONLY-BG-B, indicating that it can slightly improve accuracy by using object shape.
2. The IN-9L-trained model performs about 22% better on No-FG than ONLY-BG- showing that it can improve accuracy far more by using object shape.
Furthermore, both models perform very similarly on ONLY-BG-B.. Thus, this suggests that more training data may allow models to learn to use object shape more effectively. Understanding this phenomena further could help inform model training and dataset collection if the goal is to train models that are able to leverage shape effectively.
How much information is leaked from the size of the foreground bounding box?
The scale of an object already gives signal correlated with the object class (Torralba 2003). Even though they are designed avoid having foreground signal, the background-onjj datasets ONLY- BG-B and ONLY-BG-T may inadvertently leak information about object scale due to the bounding box sizes being recognizable.
To gauge the extent of this leakage, we can measure how models trained on datasets where only the foreground signal has useful correlation (MIXED-RAL -ANN or ONLY-FG) perform on the background- only test sets. We find that there is small signal leakage from bounding box size alone-a a model trained on ONLY-FG achieves about 23% background-only test accuracy, suggesting that it is able to exploit the signal leakage to some degree. A model trained on MIXED-RAND achieves about 15% background-onl test accuracy, just slightly better than random, perhaps because it is harder for models to measure (and thus, make use of) object scale when training MIXED-RAND
The existence of a small amount of information leakage in this case shows the importance of comparing MIXED-SAMI (as opposed to just ORIGINAL) with MIXED-RAND and MIXED-NEXT when assessing model dependence on backgrounds. Indeed, the MIXED datasets may contain (1) image processing artifacts, such rough edges from the foreground processing, and (2) small traces
17

---

Published as a conference paper at ICLR 2021
Object Size vs. Model Accuracy on Only-BG-T
1.0
Overall Accuracy
0.8
Accuracy
0.4
0.2
0.0
0
10
20
30
40
50
60
70
80
90
Bounding Box Size (as percent of Image Size)
Figure 14: Comparing model accuracy ONLY-BG-T across different foreground object bounding box sizes. We observe that the model more likely to succeed when shown only image backgrounds if the removed foreground objects have smaller bounding boxes. The dotted line represents the overall accuracy of the model on ONLY-BG-T (averaged over all bounding box sizes).
of the original background. This makes it important to control for both factors when measuring how models react to varying background signal.
How does foreground bounding box size affect accuracy ONLY-BG-T?
We further find that models are more able to predict accurately using the background signal alone when the foreground object is smaller-this is visualized in Figure [4] Intuitively this result makes sense, as most state-of-the-ar models are trained with cropping- remove small foreground objects from training images. Thus, models are actually trained to succeed when small foreground objects are cropped out, and our toolkit confirms that this indeed the case.
data augmentation, which can
Trained
Test Dataset ONLY-RG--
MIXID-NIS NEXT
MINED-RAND
MIXED-SAMI:
No-PG
ONLY-RG-T
ONLY-EG
ORIGINAL
IN-91
MIXED-NEXT MIXED-RAND
78.07
$3.28
48.49
16.20
11.19
8.22
59.60
12.32
46.44
71.09
71.53
71.33
26.72
15.33
74,89
73.23
67.53
MIXED-SAME
No-FG
13.70
18.74
12.79
70.1
36.79
42.52
31.48
48.94
47.62
ONLY-BG-B
10.35
15.11
38,3
37.85
$4.30
42,51
21.38
12.10
41.01
ONLY-BG- BG-T
11.48
17.09
45.80
40.44
38.49
50.25
19.19
49.06
47.94
ONEY-PG
33.04
47.63
ORIGINAL
IN-9L
71.21
75.00
89.90
55.78
34.02
43.60
84.12
96.32
94.61
NmngeNet
82.99
84.32
90.99
$2.69
12.69
17.36
90.17
96.89
95.33
ImageNet IN)l-IN)
51.47
48.69
64.34
21.70
7.98
9.51
59.19
76.07
Table 5: The test accuracies, in percentages, of ResNet-50 models trained on all variants of ImageNet- and pre-trained ImageNet ResNet-50. The bottom row and the second-to-last-ron test the same pre-trained ImageNet model; however, the bottom row tests the model on the Full-IN version of each dataset variation. Testing on Full-IN shows similar trends as testing on ImageNet-9, Note that the MIXED-NER NEXT test accuracy is actually higher than the MIXED-RA RAND test accuracy in the bottom row because the next class often very similar to the previous class in Full-IN.
What about other ways of modifying the background signal?
One can modify the background in various other ways-for for example, instead of replacing the background with black as in ONLY-FG, the background can be blurred in the BG-BLURRED image of Figure 15 As expected, blurred backgrounds are still slightly correlated with the correct class. Thus. test accuracies for standard models this dataset are higher than ONLY-FG, but lower than on MIXED- SAME (which has signal from random class-aligned backgrounds that are not blurred). While do not investigate all possible methods of modifying background signal,
18

---

Published as a conference paper at ICLR 2021
Original
Only-FG
BG-Blurred
Figure 15: Backgrounds can also be modified in other ways; for example, it can be blurred. Our evaluations this dataset show similar results.
we believe that the variations we do examine in ImageNet-9 already improve our understanding of how background signals matter. Investigating other variations could provide an even more nuanced understanding of what parts of the background are most important.
E ADDITIONAL RELATED WORKS AND EXPLICIT COMPARISONS
There has been prior work on mitigating contextual bias in image classification, the influence of background signals on various datasets, and techniques like foreground segmentation that we leverage.
Mitigating Contextual Bias: (Khosla et al. 2012) focuses on mitigating dataset-specific contextual bias and proposes learning SVMs with both general weights and dataset-specific weights, while (Myung Jin et al. 2012) creates an out-of-context detection task with 209 out-of-contexi images and suggests using graphical models to solve it. (Shetty et al. 2019) focuses on the role of co- occurring objects as context in the MS-COCO dataset, and uses object removal to show that (a) models can still predict removed object when only co-occurring objects are shown, and (b) special data-augmentation can mitigate this.
Explicit Comparison to Prior Works Studying the Influence of Backgrounds: In comparison to prior works on the influence of image backgrounds (described in Section 5). our work contributes the following.
We develop toolkit for analyzing the background dependence of ImageNet classifiers, the most common benchmark for computer vision progress. Only (Zhu et al. 2017). which we compare to in Section 5 also focuses ImageNet.
The test datasets we create separate and mix foreground and background signals in various ways (cf. Table [. allowing us to study the sensitivity of models to these signals in a more fine-grained manner.
Our toolkit for separating foreground and background can be applied without human- annotated foreground segmentation, which prior works on MS-COCO and Waterbirds rely on. This important because foreground segmentation annotations are hard to collect and do not exist for ImageNet.
We study the extent of background dependence in the extreme case of adversarial back- grounds.
We focus on better vision models, including ResNet (He et al. 2016), Wide ResNet(Zagoruyko & Komodakis 2016), and EfficientNet (Tan & Le 2019).
We evaluate how improvements on the ImageNet benchmark have affected background dependence (cf. Section[4).
We will publicly release our toolkit (code and datasets) for benchmarking background dependence that others can also use better understand their own models. Our toolkit is compatible with any ImageNet-trained model.
Foreground Segmentation and Image Inpainting: In order to create IN-9 and its variants. we rely on OpenCV's implementation of the foreground segmentation algorithm GrabCut (Rother al.
19

---

Published as a conference paper at ICLR 2021
2004). Foreground segmentation is branch of computer vision that seeks to automatically extract the foreground from an image (Harville et al. 2001). After finding the foreground, we remove it and simply replace the foreground with copies of parts of the background. Other works solve this problem, called image inpainting, either using exemplar-based methods (Criminisi et al. 20004) or using deep learning Yu al. (2018); [Shetty al. (2018). (Shetty al., 2018) both detects the foreground for removal and inpaints the removed region. However, more advanced inpaintings techniques can be slow and inaccurate when the region that must be inpainted is relatively large (Shetty al. 2018). which is the case for many ImageNet images. Exploring better ways of segementing the foreground and inpainting the removed foreground could improve our analysis toolkit further.
20

---

Published as a conference paper at ICLR 2021
F ADDITIONAL EXAMPLES OF SYNTHETIC DATASETS
We randomly sample an image from each class, and display all synthetic variations that image, as well as the predictions of pre-trained ResNet-50 (trained IN-9L) each variant.
Original
Only-BG-B
Only-BG-T
No-FG
dog
vehicle
vehicle
dog
Only-FG
Mixed-Same
Mixed-Rand
Mixed-Next
dog
dog
dog
dog
Figure 16: ImageNet-9 variations Dog.
Original
Only-BG-B
Only-BG-T T
No-FG
bird
fish
bird
bird
Only-FG
Mixed-Same
Mixed-Rand
Mixed-Ned Next
bird
bird
bird
bird
Figure 17: ImageNet-9 variations
Bird.
21

---

Published as a conference paper at ICLR 2021
Original
Only-BG-B
Only-BG-T
No-FG
vehicle
vehicle
vehicle
vehicle
Only-FG
Mixed-Same
Mixed-Rand
Mixed -ext
instrument
vehicle
vehicle
instrument
Figure 18: ImageNet-9 variations-Vehiclle..
Original
Only- Only-BG-B
Only-BG-T
No-FG
reptile
bird
dog
bird
Only-FG
Mixed-Same
Mixed-Rand Rand
Mixed-Nest
reptile
reptile
reptile
reptile
Figure ImageNet-9 variations- Reptile..
22

---

Published as a conference paper at ICLR 2021
Original
Only-BG-B
Only-BG-T
No-FG
carnivore
primate
bird
bird
Only-FG
Mixed-Same
Mixed-Rame
Mixed-Nee
carnivore
carnivore
carnivore
carnivore
Figure 20: ImageNet-9 variations-Carivvon
Original
Only-BG-B
Only-BG-T
No-FG
instrument
bird
bird
insect
Only-FG
Mixed-Same
Mixed-Rand
Mixed- Mixed-Next
instrument
instrument
bird
bird
Figure 21: ImageNet-9 variations Instrument.
23

---

Published as a conference paper at ICLR 2021
Original
Only-BG-B
Only-BG-T
No-FG
primate
instrument
carnivore
dog
Only-FG
Mixed-Same
Mixed-Rand
Mixed-Ni Mixed-Next
dog
primate
primate
dog
Figure 22: ImageNet-9 variations Primate.
Original
Only-BG-B
Only-BG-T
No-FG
fish
insect
fish
dog
Only-FG
Mixed-Same
Mixed-Rand Rand
Mixed-Nest
fish
fish
fish
fish
Figure 23: ImageNet-9 variations Fish.
24

---

Published as a conference paper at ICLR 2021
Target class: insect
60
40
Count
20
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Adversarial success rate
Figure 24: Histogram of insect backgrounds grouped by how often they cause (non-insect) fore- grounds to be classified as insect by a IN-9L-trained model. We visualize the five backgrounds that fool the classifier on the largest percentage of images in Figure
G ADVERSARIAL BACKGROUNDS
We compute the adversarial background attack success rate for models Table 6 MIXED-RAND model more adversarially background robust than the ORIGINAL model, it is less adversarially background robust than the IN-9L model. The model trained on all of ImageNet is the most adversarially background robust of all models. This suggests that increasing training dataset size (IN-9L) has bigger effect on adversarial background robustness than randomizing backgrounds during training MIXED-RANDI On the other hand, the MIXED-RR RAND model has a much lower BG-GA AP than the IN-9I. model, indicating that models with a smaller BG-GAP are not necessarily robust adversarial backgrounds, and vice versa.
While the
Training Dataset
ORIGINAL
MIXED-RAND
IN-9L
ImageNet
Attack Success Rate
99.0%
93.5%
88.0%
77.7%
Table 6: Adversarial backgrounds attack success rates for models analyzed this paper. The ORIGINAL and the MIXED-RAND are trained equally small datasets, IN-9L is trained on 4x more data, and the ImageNet model is trained on the most data.
Next, we visualize the attack success rate distribution of the different backgrounds from the insect class in Figure 24 The long tail of the distribution indicates that many backgrounds are especially capable fooling models.
Finally, we include the 5 most fooling backgrounds for all classes, the fool rate for each of those backgrounds, and the total fool rate across all backgrounds from that class (on the left of each row) below.
25

---

Published as a conference paper at ICLR 2021
17%
18%
19%
20%
21%
dog. 61, 1.7%%
Figure 25: Most adversarial backgrounds Dog.
32%
34%
35%
36%
38%
bird. 64. .2%
Figure 26: Most adversarial backgrounds Bird.
41%
41%
43%
44%
47%
vehicle, 67 -7.7%%
Figure 27: Most adversarial backgrounds Vehicle.
26

---

Published as a conference paper at ICLR 2021
22%
22%
23%
25%
28%
roptile, 54, 9..9%%
Figure 28: Most adversarial backgrounds-
17%
18%
20%
21%
28%
carnivore. 11%
Figure 29: Most adversarial Carmivor
45%
46%
46%
46%
47%
instrument, 73.19%
Figure 30: Most adversarial backgrounds Instrument.
25%
27%
28%
33%
39%
primate, 63. 8.3%
Figure 31: Most adversarial backgrounds Primate.
43%
44%
44%
44%
46%
fish, 69. 9.69%
Figure 32: Most adversarial backgrounds- Fish.
27

---

Published as a conference paper at ICLR 2021
H EXAMPLES OF FOOLING BACKGROUNDS IN UNMODIFIED IMAGES
We visualize examples of images where the background the full original image actually fools models in Figure 33 For these images, models classify the foreground alone correctly, but they predict the same wrong class on the full image and the background. We denote these images as "BG Fools" in Table and Figure 7 While this category is relatively rare (accounting for just 3% of the ORIGINAL EELGINAL-trained model's 's predictions), they reveal subset of original images where background signal hurts classifier performance. Qualitatively, we observe that these images all have confusing or misleading backgrounds.
Original Classification:
insect
instrument vehicle
fish reptile
fish insect
vehicle instrument
Random Background:
bird
Figure 33: Images that are incorrectly classified (as the class on the top row, which is the same class that their background alone from ONLY-BG-T is classified as), but are correctly classified (as the class on the bottom row) when the background is randomized. Note that these images have confusing backgrounds that could be associated with another class.
28


Reviews:
[{"review_id": "jNzCrbDiNln", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors presented a comprehensive study on the role of background in image classification. They designed a new set of data and a lot of experiments to find answers to the following questions: (1) How much decrease in classification accuracy if the background signal is removed? (2) Can a model successfully classify an image solely based on its background? (3) Will an image be misclassified if the image's background is replaced by a different background? (4) With the advance of the model architecture, are the more advanced models like ResNet more robust to background effect? \n\nThe paper is well written, and the figures and tables are clearly presented. The newly created dataset ImageNet-9, that contains background- and foreground-free images, are publicly available. \n\nComments:\n- I am not sure if I understand the purpose for presenting the ONLY-BG-T results in Figure 7. In my opinion, by comparing the BG-Required numbers in MIXED-RAND and ORIGINAL models, it is already clear enough to demonstrate that the background is a necessary component for many images to obtain correct classification. \n- As shown in the section of related work, similar topics have been studied before. One of the main contributions of this paper is the newly created dataset. It can be generally useful for ML research in robustness and out-of-distribution detection.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_QiYUV2Ek83", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think this a very good contribution to ICLR given the topic and the quality of the submission (originality, contribution to the stare of the art, experimental evidence, etc) although the study might need to be supported in a more theoretical framework to make it worth of an oral presentation (I would recommend a poster or short presentation)\n\n Some of the strong points of the submission are summarized as follows:\n\n1.\tStudies in the interpretability of the results of deep learning models is a very important aspect, as well as the robustness of the obtained models in a variety of circumstances and under adversarial attacks. \n2.\tA sufficient introduction and motivations sections, but I would suggest introducing the state of the art at the beginning of the paper as it would help to get a better grasp of how the works builds upon previous work.\n3.\tThe state of the art (despite the previous comment) contextualizes the subject matter in a succinct but comprehensive manner. Although there are certain aspects that could be improved, such as including a table outlining in a clearer manner the contributions of the authors in this context.\n4.\tThe experimental design is good, showing a careful analysis to validate the proposal and several ablation studies to assess the validity of the authors' hypotheses\n5.\tThe foundations for the method are presented in great detail in a formalized manner and provides sufficient elements (i.e. examples) to assess the validity of the proposed approach.\n\n\nHowever, there are certain things that in my opinion could be improved:\n\n1.\tThe authors make a very interesting contribution that leverages knowledge from several research areas and thus, sometimes the contributions with regards to the state of the art are difficult to follow. I would suggest making a table summarizing the main features of some previous works so the readers can better grasp the limitations of those previous works and understand better the improvements in each of the areas outlined in this research\n2.\tThe organization of the paper is confusing, some effort should be given at creating a clearer layout that makes the paper easier to read and follow the flow of ideas.\n3.\tFuture work could be further elaborated and discussion in specific domains (medical imaging, for instance) could be further discussed.\n4.\tThe abstract mentions that the proposed work can be used as a blueprint for assessing the out of distribution performance of deep learning models, but this aspect is not sufficiently explored in my opinion.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WsTxwdJ1OV0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe submission performs similar foreground-background analysis for object recognition as in [1], but with more modern networks in mind. As such, the main takeaways indicate that this phenomenon still exists - networks today continue to suffer from background bias as they did four years ago with AlexNet, although maybe to a lesser extent. This submission curates more careful evaluation setups by using segmentation of foreground objects, tiled backgrounds to create multiple datasets that serve to illustrate the trends in a more disambiguated way. \n\nIs there some way to quantify the overall diversity of adversarial backgrounds? For example, is it possible that owing to strong correlations of a few objects with easily-learned backgrounds, these few backgrounds always cause misclassification for other objects? Could there be a way to detect such backgrounds?\n\nThe Appendix says \"The ORIGINAL-trained model performs similarly on NO-FG and ONLY-BG-B, indicating that it does not use object shape effectively”  but there seems to be a 10% gap in Table 5, indicating that the shape mask is fairly useful. The IN-9L numbers seem 21% up instead of 13%. Am I misreading this table?\n\nRe. \"Indeed, the ONLY-BG trend observed in Figure 8 suggests that…”, could an additional possibility be that around 20% of  classes are fully correlated with their backgrounds? I.e. how can we know how much of the findings are to do with model \"failure\", and not dataset quirks?\n\nIn summary,\n\n(+) While it is not particularly surprising to learn that backgrounds can be misleading even with the correct foreground or that there exists vulnerability to adversarially picked backgrounds, given the evidence we have of background biases already, it can be useful to have a quantification of the “BG-gap” for a range of modern models.\n\n(-) The takeaways are mostly already recognised from the many works that have pointed out reliance of object recognition models on backgrounds. The experiments provide a quantified view of how much modern networks trained on the particular datasets rely on backgrounds, but it is unclear how widely applicable this information is, given that this only analyses a specific dataset. The curated datasets might be useful for benchmarking progress; however, if one were to set up the goal of providing such a testset, then perhaps it might be more appropriate to curate an entire testset of adversarial backgrounds alone (rather than mixed-rand) across a range of modern networks and for all of ImageNet, which, along with the usual test set would provide a background-robustness sanity check (with the caveat from the authors that backgrounds may actually be informative when the foreground is confusing).\n\n\n[1] Object recognition with and without objects, Zhu et al. 2017\n\nPost Rebuttal:\nI appreciate the authors' responses. The \"novelty\" over Zhu et al. was never under question in my review, I was mostly confused about how to weigh the significance of the findings, how useful it is to know some numbers for the version of the dataset created by the authors (which is not really the original Imagenet classification task), and if the submission actually does \"pinpoint\" what the problems are, how and when they manifest, to what extent the dataset is responsible vs. the training choices. Having read the other reviews, responses, looked at the updates, I'm still unsure --  if there were something in this paper that was new or surprising and not more or less already known from existing works (perhaps not precise numbers, but then the paper is essentially using a synthetic, modified Imagenet anyway), I'd be more enthusiastic about pushing up the rating. But as of now, I'm retaining my initial rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oc2VoXeE3JY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper studies the effect of background noise on image classification tasks for neural networks. The paper suggests the following based on empirical results from ImageNet classifiers.\n\n1. It is possible to achieve reasonable accuracy by just using the background information.\n2. Image classification models suffer from a decrease in accuracy if inference images have a different background.\n3. Image classification models have higher accuracy tends to depend on the background image less.\n\nThe paper also introduces a toolkit for evaluating the ImageNet classifiers' dependence on background images. \n\n##########################################################################\n\nPros:\n1. The paper provides a detailed quantitative analysis of the effects of using different backgrounds (both training and testing). It constructs various synthetic test dataset that analyzes different scenarios.\n2. The result sections (sections 3 and 4) are well structured and carefully study the impact of using different backgrounds. \n3. The proposed toolkit can be used to evaluate the model's reliance on the background.\n\n##########################################################################\n\nCons:\n1. The key concern about the paper is the lack of novelty. While the synthetic dataset was constructed to study the effects of background in detail, the findings from the paper are not new. \n2. The experiments do not study/relate how data augmentation techniques affect background reliance. The paper also mentions OOD techniques such as distributionally robust optimization (Sagawa et al, 2020), but does not study how these techniques affect background reliance. Moreover, I cannot find a discussion on which factors in training might force the model to less rely on image backgrounds (or robust to foreground images).\n3. The dataset mainly used, IN-9, is also a small dataset that contains less than 50,000 train images. Moreover, the paper only considers the ImageNet type dataset. Some results may not hold for the other datasets. The authors do not address this in the manuscript.\n4. There are some minor concerns about the experimental set-up used in the paper that I describe in the section below.\n5. Writings can be significantly improved. \n\n##########################################################################\n\nQuestions:\n1. On page 5, it says, \"MIXED-RAND models perform poorly on datasets with only backgrounds and no foregrounds.\" What is the insight from this experiment? Does this imply that the model might be learning shape features as it is doing better than random?\n2. For images processed with GrabCut, wouldn't the model use shape-related features along with the background? If it is learning the shape features, can the positive correlation in section 4 mean that a stronger model might be learning more shape features?\n3. What is each point in figure 8 represent? Is it using different architectures?\n4. Appendix B.1, doesn't the change in the number of classes also result in a change in total dataset size?\n5. What is the main insight of this paper when training neural networks?\n6. What are the main contributions of the toolkit? How accurate is the segmentation?\n\nMinor notes:\n* On page 1, \"standard models misclassify 88% of images ...\" --> does this refer to 88% of test images?\n* On page 1, I don't understand this sentence: \"tend to simultaneously exploit background correlations more.\" Is there a stronger correlation between backgrounds for more accurate models? \n* For figure 2, what is the test accuracy for a model trained on the original dataset (original/original)?\n* In table 2, \"on select test sets ...\" --> selected.\n* In figure 3, I don't understand what it means by \"note that the gap decreases much more on the right side of the graph.\"\n* It isn't easy to interpret table 3. It will be easier to understand if it contains some illustrative examples.\n\n##########################################################################\n\nI raised my score based on the author's response.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the tendency of image recognition models to depend on image backgrounds, and propose a suite of datasets to study this phenomenon.

All the reviewers agree that the paper investigates an important problem, is well-written and contains several interesting insights that should be of interest to the community. I recommend acceptance.

Decision: Accept (Poster)