Contemplating Real-World Object Classification
Authors: ali borji
Venue: ICLR 2021 Poster (2021)
Keywords: object recognition
deep learning
ObjectNet
Robustness

Abstract: Deep object recognition models have been very successful over benchmark
datasets such as ImageNet. How accurate and robust are they to distribution
shifts arising from natural and synthetic variations in datasets? Prior research on
this problem has primarily focused on ImageNet variations (e.g., ImageNetV2,
ImageNet-A). To avoid potential inherited biases in these studies, we take a
different approach. Specifically, we reanalyze the ObjectNet dataset recently
proposed by Barbu et al. containing objects in daily life situations. They showed
a dramatic performance drop of the state of the art object recognition models on
this dataset. Due to the importance and implications of their results regarding
the generalization ability of deep models, we take a second look at their analysis.
We find that applying deep models to the isolated objects, rather than the entire
scene as is done in the original paper, results in around 20-30% performance
improvement. Relative to the numbers reported in Barbu et al., around 10-15%
of the performance loss is recovered, without any test time data augmentation.
Despite this gain, however, we conclude that deep models still suffer drastically
on the ObjectNet dataset. We also investigate the robustness of models against
synthetic image perturbations such as geometric transformations (e.g., scale,
rotation, translation), natural image distortions (e.g., impulse noise, blur) as well
as adversarial attacks (e.g., FGSM and PGD-5). Our results indicate that limiting
the object area as much as possible (i.e., from the entire image to the bounding
box to the segmentation mask) leads to consistent improvement in accuracy and
robustness. Finally, through a qualitative analysis of ObjectNet data, we find that
i) a large number of images in this dataset are hard to recognize even for humans,
and ii) easy (hard) samples for models match with easy (hard) samples for humans.
Overall, our analysis shows that ObjecNet is still a challenging test platform that
can be used to measure the generalization ability of models. The code and data
are available in [masked due to blind review].

Published as a conference paper at ICLR 2021
CONTEMPLATING REAL-WORLD OBJECT CLASSIFICATION
Ali Borji
HCL. America
al borj i gma i com
ABSTRACT
Deep object recognition models have been very successful over benchmark datasets such as ImageNet. How accurate and robust are they to distribution shifts arising from natural and synthetic variations datasets? Prior research on this problem has primarily focused on ImageNet variations (e.g., ImageNetV2, ImageNet-A). To avoid potential inherited biases in these studies, we take a different approach. Specifically, we reanalyze the ObjectNet dataset' recently proposed by Barbu et al. containing objects in daily life situations. They showed dramatic performance drop of the state of the art object recognition models this dataset. Due to the importance and implications of their results regarding the generalization ability of deep models, we take a second look at their analysis. We find that applying deep models to the isolated objects, rather than the entire scene as is done in the original paper, results in around 20-30% performance improvement. Relative the numbers reported in Barbu al., around 10-15% of the performance loss is recovered, without any test time data augmentation. Despite this gain, however, we conclude that deep models still suffer drastically on the ObjectNet dataset. We also investigate the robustness of models against synthetic image perturbations such geometric transformations (e.g., scale, rotation, translation). natural image distortions (e.g., impulse noise, blur) as well as adversarial attacks (e.g., FGSM and PGD-5). Our results indicate that limiting the object area as much as possible (i.e., from the entire image to the bounding box to the segmentation mask) leads to consistent improvement in accuracy and robustness. Finally, through qualitative analysis of ObjectNet data, we find that i) large number of images this dataset are hard to recognize even for humans, and ii) easy (hard) samples for models match with easy (hard) samples for humans. Overall, our analyses show that ObjecNet is still challenging test platform for evaluating the generalization ability of models. Code and data are available at https:/ g thub. com/ al liborjii jeeeebbeeettttteeeeeeeee www.RRaeaaallllili isti..
INTRODUCTION
Object recognition can be said to be the most basic problem in vision sciences. It is required in the early stages of visual processing before a system, be it a human or a machine, can accomplish other tasks such as searching, navigating, or grasping. Application of a convolutional neural network architecture (CNN) known as LeNet (LeCun et al., 1998), albeit with new bells and whistles (Krizhevsky et al., 2012), revolutionized not only computer vision but also several other areas. With the initial excitement gradually damping, researchers have started to study the shortcomings of deep models and question their generalization ability. From prior research, we already know that CNNs: a) lack generalization to out of distribution samples (e.g.. Recht et al. (2019); Barbu et al. (2019); Shankar et al. (2020); Taori et al. (2020); Koh et al. (2020)). Even after being exposed to many different instances of the same object category, they fail to fully capture the concept. In stark contrast, humans can generalize from only few examples (a.k.a few-shot learning), b) perform poorly when applied to transformed versions of the same object. In other words, they
/ http::::///bjeccettttdeee dev/
2See https: /oppnreeeeeeeeeeeeeeeeee net/ forum21 d-Q4RUywJ 1kgr for reviews and discussions. prelimnary version of this work has been published Arxiv (Borji, 2020).
"Classification of an object appearing lonely in image. For images containing multiple objects, object localization or detection is required first.

---

Published as a conference paper at ICLR 2021
not invariant to spatial transformations (e.g.. translation, in-plane and in-depth rotation, scale) as shown in (Azulay & Weiss, 2019; Engstrom et al., 2019; Fawzi & Frossard. 2015), as well as noise corruptions (Hendrycks & Dietterich, 2019; Geirhos et al., 2018b), and c) are vulnerable to imperceptible adversarial image perturbations (Szegedy et al., 2013; Goodfellow et al., 2014; Nguyen al., 2015). Majority of these works, however, have used either the ImageNet dataset its variations, and thus might be biased towards ImageNet characteristics. Utilizing a very challenging dataset that has been proposed recently, known as ObjectNet (Barbu al., 2019), here we seek to answer how well the state of the art CNNs generalize to real world object recognition scenarios. We also explore the role of spatial context in object recognition and answer whether it is better to use cropped objects (using bounding boxes) or segmented objects to achieve higher accuracy and robustness. Furthermore, we study the relationship between object recognition, scene understanding. and object detection. These are important problems that have been less explored.
Several datasets have been proposed for training and testing object recognition models, and to study their generalization ability (e.g., ImageNet by Deng et al. (2009), Places by Zhou et al. (2017), CIFAR by Krizhevsky et al. (2009), NORB by LeCun et al. (2004), and iLab20M by Borji et al. (2016)) As the most notable one, ImageNet dataset has been very instrumental for gauging the progress in object recognition over the past decade. A large number of studies have tested new ideas by training deep models on ImageNet (from scratch), or by finetuning pre-trained (on ImageNet) classification models on other datasets. With the ImageNet being retired, the state of the object recognition problem remains unclear. Several questions such as out of distribution generalization, "superhuman performance" (He et al., 2016) and invariance to transformations persist. To rekindle the discourse, recently Barbu al. (2019) introduced the ObjectNet dataset which according their claim has less bias than other recognition datasets*. This dataset is supposed to be used solely as a test set and comes with a licence that disallows the researchers to finetune models on it. Images are pictured by Mechanical Turk workers using a mobile app in variety of backgrounds, rotations, and imaging viewpoints. ObjectNet contains 50,000 images across 313 categories, out of which 113 are in common with ImageNet categories. Astonishingly, Barbu et al. found that the state of the art object recognition models perform drastically lower on ObjectNet compared to their performance on ImageNet (about 40-45% drop). Our principal goal here it to revisit the Barbu et al.'s analysis and measure the actual performance drop on ObjectNet compared to ImageNet. To this end, we limit our analysis to the 113 overlapped categories between the two datasets. We first annotate the objects in the ObjectNet scenes by drawing boxes around them. We then apply a number of deep models on these object boxes and find that models perform significantly better now, compared to their performance on the entire scene (as is done in Barbu et. al). Interestingly, and perhaps against the common belief, we also find that training and testing models on segmented objects, rather than the object bounding box or the full image, leads to consistent improvement in accuracy and robustness over a range of classification tasks and image transformations (geometric, natural distortions, and adversarial attacks). Lastly, we provide qualitative (and somewhat anecdotal) analysis of extreme cases in object recognition for humans and machines.
2 RELATED WORK
Robustness against synthetic distribution shifts. Most research on assessing model robustness has been focused on synthetic image perturbations (e.g., spatial transformations, noise corruptions, simulated weather artifacts, temporal changes (Gu et al., 2019), and adversarial examples) perhaps because it is easy to precisely define, implement, and apply them to arbitrary images. While models have improved significantly in robustness these distribution shifts (e.g.. Zhang (2019); Zhang et al. (2019); Cohen & Welling (2016)), they are still not as robust as humans. Geirhos et al. (2018b) showed that humans are more tolerant against image manipulations like contrast reduction, additive than models. Further, humans and models behave differently
noise, or novel eidolon-distortio (witnessed by different error patterns) as the signal gets weaker. Zhu et al. (2016) contrast the influence of the foreground object and image background on the performance of humans and models.
Robustness against natural distribution shifts. Robustness on real data a clear challenge for deep neural networks. Unlike synthetic distribution shifts, it is difficult to define distribution shifts that occur naturally in the real-world (such as subtle changes in scene composition, object types, and lighting conditions). Recht et al. (2019) closely followed the original ImageNet creation process
*OhjectNet dataset, however, has it own biases. It consists of indoor objects that are available to many people, are mobile, are top large, too small, fragile dangerous.
2

---

Published as a conference paper at ICLR 2021
Figure 1: Sample images from the ObjectNet dataset along with our manually annotated object bounding boxes from Chairs, Teapots and i-shirts categories. The leftmost column shows three chair examples from the ImageNct dataset. ImageNct scenes often have single isolated object in them whereas images in the ObjectNet dataset contain multiple objects. Further, ObjectNet objects cover wider range of variation contrast, rotation, scale, and occlusion compared to ImageNet objects (See arguments in Barbu et al. (2019)). In total, we annotated 18,574 images across 113 categories in common between the datasets. This figure modified from Figure in Barbu et al. (2019).
InageNett
ObjectNat
Chais
Chais by
backgrond
Chains by
stewpoine
Teapuis
Talito
to build a new test set called ImageNetV2. They reported performance gap of about 11% (top-1 acc.) between the performance of the best deep models on this dataset and the original test set. Similar observations have been made by Shankar al. (2020). By evaluating 204 ImageNet models in 213 different test conditions, Taori et al. (2020) found that a) current synthetic robustness does not imply natural robustness. In other words, robustness measures for synthetic distribution shifts are weakly predictive of robustness on the natural distribution shifts, b) robustness measurements should control for accuracy since higher robustness can sometimes explained by the higher accuracy on standard unperturbed test set, and c) training models on larger and more diverse data improves robustness but does not lead to full closure of the performance gap. A comprehensive benchmark of distribution shifts in the wild, known WILDS, has recently been published by Koh et al. (2020), encompassing different data modalities including vision. In D' Amour et al. (2020), authors regard "underspecification" a major challenge the credibility and generalization of modern machine learning pipelines. An ML pipeline is underspecified when it returns models that perform very well on held-out test sets during training but perform poorly deployment time.
Contextual interference. Context plays a significant role in pattern recognition and visual reasoning (e.g.. Bar (2004); Torralba & Sinha (2001); Rabinovich et al. (2007); Heitz & Koller (2008); Galleguillos & Belongie (2010)). The extent to which visual context is being used by deep models still unclear. Unlike models, humans are very good at exploiting context when it helpful and discard it when it causes ambiguity. In other words, deep models do not understand what is the foreground object and what constitutes the background Nagarajan et al. (2020) mention that ML models utilize features (e.g., image background) which are spuriously correlated with the label during training. This makes them fragile at the test time when statistics slightly differ. As we argue here, this is one of the main reasons why deep models are vulnerable to geometric and adversarial perturbations. Geirhos et al. (2020) have studied this phenomenon under the "shortcut learning" terminology from broader perspective.
Insights from human vision. CNNs turn out to be good models of human vision and can explain the first feed-forward sweep of information (See Kriegeskorte (2015) for review). They, however, differ from human visual processing in several important ways. Current object recognition methods do not rely on segmentation, whereas figure-ground segmentation plays significant role in human vision, in particular for the encoding of spatial relations between 3D object parts (Biederman, 1987: Serre, 2019). Some computer vision works, predating deep learning, have also shown that premeenmn the image before applying the recognition algorithms, improves the accuracy (Malisiewicz Efros, 2007; Rabinovich et al., 2007; Rosenfeld Weinshall, 201 1). Unlike the human vision system, CNNs are hindered drastically in crowded scenes (e.g.. Volokitin et al. (2017)). CNNs rely more on texture whereas humans pay more attention to shape (Geirhos et al., 2018a). Utilizing minimal recognizable images, Ullman et al. (2016) argued that the human visual system uses features and processes that are not used by current deep models.
$ As an example, consider a model that is trained to classify camels vs. cows, with camels always shown in sandy backgrounds and cows shown against grassy backgrounds. Although such model does well during training, it gets confused when presented with cows in sandy backgrounds at test time (Beery et al., 2018). See also Rosenfeld al. (2018) for another example in the context of object detection
3

---

Published as a conference paper at ICLR 2021
Figure 2: Performance of deep object recognition models on ObjectNet dataset. Results of our analysis by applying AlexNet, VGG-19, and ResNet-152 models to object bounding boxes (i.e., isolated objects) are shown in blue (overlaid in Fig. I from the ObjectNet paper). Feeding object boxes models instead of the entire scene improves the accuracy about 10-15%.
100
ImageNai Top-5
ImageNet Top
Using our code
90
Beonding Box (kp-3)
Objeatie
80
Objeshas Vop-1
hull kullimage (ftp-2) hull Image (Tpp Ski
Objee japer
Oup sasi
70
Accunay %
20-30% performance gain
Accuracy
60
-0
25-35% performance drop
40-45%
performance
40
drop
The right panel shows the results using our code (thus leading to fair comparison). Performance improvement is more significant now which is ~ 20-30%. We did not use test time data augmentation here.
30
5720
20
ObjectNet Top-5 (box)
ObjectNet Top-1 (box)
10
Reater
Goopleter
Hosteer
accetoona)
Recognizers year
NleSNa 2012
VGG-L 19 2014
ResNet-132
2010 Inception-el
NASNet.1 A 2017
2018 PNASNa:55
2018
3 EXPERIMENTS AND RESULTS
3.1 ACCURACY AND ROBUSTNESS AGAINST NATURAL DISTRIBUTION SHIFTS
critic of Barbu et al. (2019). Barbu et al.'s work is great contribution to the field to answer how well object recognition models generalize to the real-world circumstances and to control for biases in data collection. It, however, suffers from a major shortcoming that is making no distinction between "object detection" and "object recognition". This confusion brings along several concerns:
1. They use the term "object detector" to refer to "object recognition" models. Object detection and object recognition are two distinct, yet related, tasks. Each one has its own models, datasets, evaluation measures, and inductive biases. For example, as shown in Fig. I, images in object recognition datasets (e.g.. ImageNet) often contain single object, usually from a closeup view, whereas scenes in object detection datasets (e.g., MS COCO (Lin et al., 2014), OpenImages (Kuznetsova et al., 2018)) usually have multiple objects. Objects in the detection datasets vary more in some parameters such as occlusion and size. For instance, there is a larger variation in object scale in detection datasets (Singh & Davis, 2018). This discussion also relates to the distinction between "scene understanding" and "object recognition". To understand a complex scene, as humans we look around, fixate individual objects to recognize them, and accumulate information over fixations to perform more complex tasks such as answering question or describing an event. To avoid biases in recognition datasets (e.g., typical scales or object views), we propose to (additionally) use detection datasets to study object recognition. We will discuss this further in Section 4.
2. Instead of applying models to isolated objects. Barbu et al. apply them to cluttered scenes containing multiple objects. Unlike ImageNet where the majority of images include only a single object, ObjectNet images have multiple objects in them and are often more cluttered. Therefore, the drop in performance of models on ObjectNet can be merely due to the fact that pretrained models ImageNet have been trained on individual objects.
3. In addition to top-1 accuracy, Barbu et al. also report top-5 accuracy. One might argue that this may suffice in dealing with scenes containing multiple objects. Top-5 accuracy was first introduced in Russakovsky et al. (2015) to remedy the issues with the top-1 accuracy. The latter can be overly stringent by penalizing predictions that appear in the image but do not correspond to the target label. Top-5 accuracy itself, however, has two shortcomings. First, a model can still be penalized if all of the five guesses exist in the image, but none is the image label. Both scores fall short in addressing the images with counter-intuitive labels (c.g., when non-salient objects are labeled; Appx. E). Second, on fine- grained classification tasks (ImageNet has several fine-grained classes e.g., dogs), allowing five
4

---

Published as a conference paper at ICLR 2021
predictions can make certain class distinctions trivial (Shankar et al., 2020). For example, there are five turtles in the ImageNet class hierarchy (mud turtle, box turtle, loggerhead turtle, leatherback turtle, and terrapin) that are difficult to distinguish. A classifier may trick the score by generating all of these labels for a turtle image to ensure it predicts the correct label. Shankar al. proposed to use multi-label accuracy as alternative top-5 score. Each image has a set of target labels (i.e., multi-label annotations). A prediction is marked correct if it corresponds however, may favor model that generates correct labels but may confuse the locations over a model that is spatially more precise but misses some objects (See also Beyer al. (2020)). Regardless, since multi-label annotations for ObjectNet are not available, we report both top-1 and top-5 scores when feeding isolated objects to models.
any of the target labels for that image. This score,
Bounding box annotation. The 113 object categories in the ObjectNet dataset, overlapped with the ImageNet, contain 18,574 images in total. On this subset, the average number of images per category is 164.4 (min=55, max=284). Fig. 8 in Appx. A shows the distribution of the number of images per category on this dataset (envelope and dish drying rack are the most and least frequent objects, respectively). We drew a bounding box around the object corresponding to the category label of each image. If there were multiple nearby objects from the same category (e.g., chairs around a table), we tried to include all of them in the bounding box. Some example scenes and their corresponding bounding boxes are given in Fig. 1. Appx. H shows more stats on ObjectNet.
Object recognition results. We employ six widely-used state of the art deep neural networks including AlexNet (Krizhevsky et al.. 2012), VGG-19 (Simonyan & Zisserman, 2014), GoogLeNet (Szegedy et al., 2015), ResNet- 152 (He et al., 2016). Inception-v3 (Szegedy et al., 2016)", and MNASNet (Tan et al., 2019). AlexNet, VGG-19, and ResNet-152 have also been used in the ObjectNet paper (Barbu et al., 2019). We use the PyTTrrch implementation of these models?. Since the code from the ObjectNet paper is unavailable (at the time of preparing this work), in addition to applying models to bounding boxes and plotting the results on top of the results from the ObjectNet paper, we also run our code both the bounding boxes and the full images. This allows fair comparison and helps mitigate possible inconsistency in data processing methods (e.g., different data normalization schemes or test time data augmentation such as rotation, scale, color jittering, cropping, etc.).
Fig. 2 shows an overlay of our results in Fig. 1 from the ObjectNet paper. As can be seen, applying models to the object bounding box instead of the entire scene improves the accuracy about 10- 10-15%. Although the gap is narrower now, models still significantly underperform on ObjectNet than the ImageNet dataset. Using our code, the improvement going from full image to bounding boxes is around 20-30% across all tested models (the right panel in Fig. 2). Our results using the full image are lower than Barbu et al.'s results using the full image (possibly because we do not utilize data augmentation). This relative difference entails that applying their code bounding boxes will likely improve the performance beyond 10% that we obtained here. Assuming 25% gain in performance on top of their best results when using boxes, will still not close the performance gap which indicates that ObjectNet remains a challenging dataset for testing object recognition models.
Breakdown of accuracy over the 113 categories is shown in Appx. B (Figs, 9 & 10 over isolated objects and Figs. 11 & 12 over the full image). Interestingly. in both cases, almost all models, except GoogLeNet on isolated objects and AlexNet on the full image, perform the best over the safety pin category. Inspecting the images from this class, we found that they have single safety pin often held by a person (perhaps about the same distance from the camera thus similar scales). The same story is true about the banana class which is the second easiest category using the bounding boxes. This object becomes much harder to recognize when using the full image (26.88% vs. 70.3% using boxes) which highlights the benefit of applying models isolated objects rather than scenes.
3.2 ACCURACY AND ROBUSTNESS AGAINST SYNTHETIC DISTRIBUTION SHIFTS
3.2.1 ROBUSTNESS AGAINST COMMON IMAGE CORRUPTIONS
Previous work has shown that ImageNet-trainedd CNNs generalize poorly over a wide range of image distortions (e.g.. Hendrycks & Dietterich (2019); Azulay & Weiss (2019); Dodge & Karam (2017)). These works, however, have applied CNNs to the whole scene. Here, we
BBarbu et al. have used Inception-v4. "ttpps: //tp::////tt/h///////// http://ttttrhhooo///eeeeoee wwwp/oos///ooooooooooooooo.oo.oo stabl eeteeeettrchhii i oniio//mmn /moee1 1
5

---

Published as a conference paper at ICLR 2021
top-1
top-5
60-
tp-1
up3
Bounding Box
VCG-23 Incept sn
155
NcsNe-352
1xp-5
Severity
Sevvritty
Putl Image
Figure 3: Left panels) Average top-1 and top-5 accuracy of models over 1130 images from the ObjectNet dataset corrupted by 14 natural image distortions. Right two panels) Average classification accuracy per each of the 113 categories (left) and each of the 14 distortion types (right). In all cases, applying models to the isolated object (using bounding boxes) leads to higher robustness than applying them to the full image.
ask whether applying the models to the bounding boxes can improve robustness against image distortions. Following Hendrycks & Dietterich (2019), we systematically test how model accuracy degrades if images are corrupted by 14 different types of distortions including Gaussian noise, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost. fog, brightness, contrast, elastic transform and JPEG compression at 3 levels of corruption severity. Fig. 36 (Appx. F) shows sample images along with their distortions. Ten images from each of the 113 categories of ObjectNet (1 130 images in total) were fed to three models including VGG-19, Inception-v3, and ResNet-152.
Aggregate results over the full image and the object bounding box (both resized to 224 x 224 pixels) are shown in Fig. 3. All three models are more robust when applied to the object bounding box than the full image all corruption levels, using both top-1 and top-5 scores (left two panels). Among models, ResNet-152 performs better and is the most robust model. It is followed by the Inception-v3 model. For nearly all of the 113 object categories, using bounding boxes leads to higher robustness than using the full image (the third panel). Similarly, using bounding boxes results in higher robustness against all distortion types (the right-most pancl). Across distortion types, shown in Figs. 37 & 38 (Appx. F), ResNet- 152 consistently outperforms the other two models at all severity levels, followed by Inception-v3. It seems that models are hindered more by impulse noise, frost, zoom blur, and snow distortions. The top-1 accuracy at severity level 2 on these distortions is below 20%. Overall, we conclude that limiting the object area only to the bounding box leads not only to higher prediction accuracy but also to higher robustness against image distortions. Extrapolating this approach, can we improve robustness by shrinking the object region even further by using the segmentation masks? We will thoroughly investigate this question in the next subsections.
3.2.2 ROBUSTNESS AGAINST ADVERSARIAL PERTURBATIONS
Despite being very accurate, CNNs are highly vulnerable to adversarial inputs (Szegedy et al., 2013: Goodfellow et al., 2014). These inputs are crafted carefully and maliciously by adding small imperceptible perturbations to them (c.g., altering the value of a pixel up to units under the -nomm; pixels in the range [0, 255]). Here we apply the ImageNet pretrained models to 1130 images that were selected above. The models are tested against the Fast Gradient Sign Method (FGSM) (Goodfellow al., 2014) at two perturbation budgets in the untargeted white-box setting.
Table. 1 shows the results. We find that models more resilient against the FGSM attack when applied to the bounding box than the full image. While the input size is the same in both cases (224x224), the adversary has more opportunity to mislead the classifier in the full image case since a larger fraction of pixels play an insignificant role in the decisions made by the network. This aligns with observations from the visualization tools (e.g.. Selvaraju et al. (2017)) revealing that CNNs indeed rely only on a small subset of image pixels to elicit a decision. One might argue that the lower robustness on the full images could be due to training and test discrepancy (i.e., training models on single objects and applying them to the entire scene). To address this, in the next subsection we train and test models in the same condition.
Full Image
Bounding Box
Model
2/255
8/255
c= 2/255 6 8/255
VGG- 19
0.53/2.48
0.09/0.71
3.27/10.44
1.06/5.66
Inception v3
3.18/10.62
1.42/4.42
9.03/25.13
4.87/15.6
ResNet- 152
2.39/10.34
1.15/4.96
10.62/26.64 6.64/19.73
Table 1: Robust accuracy (top- (/to-// against FGSM attack.

---

Published as a conference paper at ICLR 2021
3.3 THE INFLUENCE OF THE SURROUNDING CONTEXT ON ROBUSTNESS
Despite large body of literature on whether and how much visual context
MNIST
with by masking (clean)
0.-
w/b masking (clean)
with by masking (notse)
benefits CNNs in terms of accuracy and robustness the matter has not been settled yet (e.g., Bar (2004);
08
masking
0.7
FashionMNIST
with masking (clean) masking (nose)
Torralba & Sinha (2001);
04
bg masking (noise)
Rabinovich et al. (2007);
Rosenfeld et al. (2018);
02
Heitz & Koller (2008);
Divvala al. (2009); Zhu et al. (2016); Xiao et al. (2020); Malisiewicz & Efros (2007)). To study how context surrounding an object impacts model accuracy and robustness in more detail, we conducted two experiments. In the first one, we trained two CNNs (2 conv layers, each followed by a pooling layer and 2 final fe layers) on MNIST and Fashion MNIST datasets, for which it is easy to derive the foreground masks (Figs. 39 & 40; Appx. G). CNNs were trained on either the original clean images or the foreground objects placed against the FGSM attack w/o background subtraction. With background subtraction, we essentially assume that the adversary has access only to the foreground object (i.e., effectively removing the perturbations that fall on the background). As results in Fig. show, background subtraction improves the robustness substantially using both models and over both datasets.
00
0.00
0.05
0.10
0.35
0.20
0.25
0.30
0.00
Epailam
Figure 4: The effect of background subtraction (a.k.a foreground detection) on adversarial robustness (here against the FGSM attack). Two models are trained and tested on clean and noisy data from MNIST (left) and FashionMNIST (right) datascts. In the noise case, the object is overlaid in a white noise field (no noise on the object itself).
white noise background. We then tested the models
To examine whether the above conclusion generalizes to more complex natural scenes, we ran a second experiment. First, selected images from ten classes of the MS COCO dataset including chair, car, book, bot t tle, di dinning table, umbrella, boat, mot motorcycle, sheep, and cow. Objects from these classes come with segmentation mask (one object per image; 100 images per category; 1000 images in total). Around 32.7% of the image pixels fall inside the object bounding box and around 58.1% of the bounding box pixels fall inside the object mask. Fig. 5 shows a sample chair alongside its bounding box and its segmentation mask.
We then trained three ResNet-18 models (finetuned on ImageNet). one per each input type: 1) full image, 2) bounding box, and 3) segmented object (placed in Models were trained on 70 images per category (700 in total) for 10 epochs and were then tested on the remaining 30 images per category. An attempt was made to tune the parameters to attain the best test accuracy in each case (e.g., by avoiding overfitting). The test accuracy of models in order are 66.9%, 78%, and 80.3%. One reason behind lower prediction accuracy using boxes might be because multiple objects may fit inside
Full image (FGSMI
full image (RGD
Rounding Box (POSH]
50
Bounding (PCD-S)
7D
Mask
a dark background).
(PGD-5)
60
22
Full mage
Bounding Box
Seg. Mask
Seq
2055
#255
36/255
Seading
severity
Figure 5: Model accuracy against adversarial perturbations (left) and noise corruptions (middle). The right panel shows a sample chair image along with its bounding box and segmentation mask.
Majority of such works are focused on model accuracy Taori et al. (2020) argue that robustness scores should control for accuracy as more predictive models in general more robust. To avoid this issue we used models that have about the same standard accuracy.
7

---

Published as a conference paper at ICLR 2021
full image Boundiy Bex Seg.
50
12000
orgge
(bright)
Figure 6: Performance (top-1) 1) of the ResNet- 18 model against geometric transformations.
the bounding box (e.g., for elongated objects such as broom). Model performance against FGSM and Lx PGD-5 (Projected Gradient Descent by Madry et al. (2017)) adversarial attacks are shown in Fig. 5 (left panel). We observe that training models on segmented objects leads to higher adversarial robustness against both types of attacks. The improvement is more pronounced at higher perturbations. We also considered a condition in which we masked the perturbations that fall on the background, denoted "Seg. Mask FG" in the figure. We noticed even higher robustness against the attacks by removing the background perturbations. These results encourage using foreground detection as an effective adversarial defense.
The middle panel in Fig. 5 shows model robustness against noise corruptions (averaged over the 14 distortions used in Section 3.2.1). Here again, we find that using segmentation masks leads to higher robustness compared to the full image and object boxes. "Seg. Mask + FG" leads to the best robustness among the input types. While it might be hard draw a general conclusion regarding the superiority of the segmentation masks over bounding boxes in object recognition accuracy, our investigation suggests that using masks leads to a significant boost in adversarial robustness with little or no drop in standard accuracy. Our results offer an upper bound in the utility of segmentation masks robustness. More work is needed to incorporate this feat in CNNs (i.e., using attention).
3.3.1 ROBUSTNESS AGAINST GEOMETRIC TRANSFORMATIONS
We also tested the ResNet- 18 model (i.e., trained over the full image, the bounding box, and the segmented object on ObjectNet; as above) against three geometric transformations including scaling, in-plane rotation, and horizontal translation. Fig. shows the results over the 300 test images that were used in the previous subsection. We find that the model trained on segmentation masks is more robust than the other two models over all three geometric transformations, followed by the models trained on the object bounding boxes and the full image, in order.
3.4 QUALITATIVE INSPECTION OF OBJECTNET IMAGES AND ANNOTATIONS
During the annotation of ObjectNet images, we came across the following observations: a) Some objects look very different when they are in motion (c.g., the fan in row 4 of Fig. 34 in Appx. D), or when they are shadowed or occluded by other objects (e.g., the hammer in Fig. 34 row 4), b) Some object instances differ a lot from the typical instances the same class (e.g., the helmet in Fig. 34 row 5; the orange in Fig. 33 row 5), c) Some objects can be recognized only through reading their captions (e.g., the pet food container in Fig. 33 row 2), d) Some images have wrong labels (e.g., the pillow in Fig. 33 row 2; the skirt in Fig. 33 row 1; the tray in Fig. 34 row 2; See also Appx. E). e) Some objects are extremely difficult for humans (c.g., the tennis racket in Fig. 34 row 4; the shovel in Fig. 33 row 4; the tray in Fig. 33 row 1). f) In many images, objects are occluded by hands holding them (c.g., the sock and the shovel in Fig. 33 row 4). g) Some objects are hard to recognize in dim light (e.g., the printer in Fig. 33 row 2), and h) Some categories are often confused with other categories in the same set. Example sets include (bath towel, bed sheet, full sized towel, dishrag or hand towel), (sandal, dress shoe (men), running shoe). {1-shirt, dress, sweater, suit jacket, skirt). (ruler, spatula, pen, match), (padlock, combination lock). J. and (envelope, letter).
The left panel in Fig. 7 shows four easy (highly confident correct predictions) and four hard (highly confident misclassifications) for ResNet-152 over six ObjectNet categories. In terms of the difficulty level, casy(difficult) objects for models appear casy(diffiell)) to humans too. Also, our qualitative inspection shows that ObjectNet includes large number of objects that can be recognized only after a careful examination (the right panel in Fig. 7). More examples are given in Appx. C.
8

---

Published as a conference paper at ICLR 2021
Figure 7: Left) Four easiest (highest classification confidence; 1st rows) and four hardest (lowest confidence; 2nd rows) samples per category from the ObjectNet dataset for the ResNet-152 model (predictions shown in blue). The casy samples seem to be casier for humans (qualitatively), while the harder ones also seem be harder. Right) A collection of challenging objects from the ObjectNet dataset for humans. Can you guess the category of the annotated objects in these images? Keys are: row 1: skirt, fan, desk lamp. safety pin, still camera, and spatula, row 2: vase, shovel, vacuum cleaner, printer, remote control, and pet food container, and row 3: sandal, vase, match, spatula, stuffed animal, and shovel. Appxs. C & D contain more examples.
4 TAKEAWAYS AND DISCUSSION
Our investigation reveals that deep models perform significantly better when applied to isolated objects rather than the entire scene. The reason behind this two-fold. First, there is less variability in single objects compared to scenes containing multiple objects. Second, deep models (used here and also in ObjectNet paper) have been trained on ImageNet images which are less cluttered compared to the ObjectNet images. We anticipate that training models from scratch on large scale datasets that contain isolated objects will likely result in even higher accuracy. Assuming around 30% increase in performance (at best) top of Barbu et al.'s results using bounding boxes still leaves a large gap of at least 15% between ImageNet and ObjectNet which means that ObjectNet is indeed much harder. It covers a wider range of variations than ImageNet including object instances, viewpoints, rotations, occlusions, etc which pushes the limits of object recognition models. Hence, despite its limitations and biases, ObjectNet dataset remains great platform to test deep models in realistic situations.
We envision four research directions for the future work in this area. First, background subtraction is a promising mechanism and should be investigated further over large scale datasets (given the availability of high-resolution masks; c-g., MS COCO). We found that it improves robustness substantially over various types of image perturbations and attacks. Humans can discern the foreground object from the image background with high precision. This feat might be the key to robustness and hints towards an interplay and feedback loop between recognition and segmentation that is currently missing in CNNs. Second, measuring human performance on ObjectNet will provide useful baseline for gauging model performance. Barbu et. al report accuracy of around 95% when they asked subjects to mention the objects that are present in the scene. This task, however, is different from recognizing isolated objects similar to the regime that was considered here (i.e., akin to rapid scene categorization tasks; See Serre et al. (2007)). Besides, error patterns of models and humans (e.g., Borji & Itti (2014)), in addition crude accuracy measures, will inform us about the differences in object recognition mechanisms between humans and machines. could be that models work in a completely different fashion than the human visual system. Third, as discussed in Section 3.1, multi-label prediction accuracy is more appropriate for evaluating recognition models. Annotating all objects in ObjectNet images will thus provide an additional dimension to assess models. In this regard, we propose a new task where the goal is to recognize objects in their natural contexts. This task resembles (cropped) object recognition and object detection, but it is slightly different (i.e., the goal here is to recognize an object limited by bounding box given all available information in the scene). This is essentially an argument against the recognition-dettccion dichotomy. Finally, it would be interesting to how well the state of the art object detectors perform on the ObjectNet dataset (c.g., over overlapped classes between ObjectNet and MS COCO (Lin al., 2014)). We expect a significant drop in detection performance since it is hard to recognize objects in this dataset.
From a broader perspective, our study reinforces the idea that there is more to scene understanding then merely learning statistical correlations. In particular, background subtraction and visual context crucial in robust recognition and demand further investigation in future studies.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184):1-22 2019.
Moshe Bar. Visual objects in context. Nature Reviews Neuroscience, 5(8):617, 2004.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh
Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems, pp- 9448-9458, 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 456-473, 2018.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.0755, 2020.
Irving Biederman.
theory of human image understanding.
Psychological review, 94(2):115, 1987.
Ali Borji. Objectnet dataset: Reanalysis and correction. arXiv preprint
Ali Borji and Laurent Itti. Human computer in scene and object recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 113-120, 2014.
Ali Borji, Saeed Izadi, and Laurent Itti. ilab-20ml A large-scale controlled object dataset to investigate deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2221-2230 2016.
Taco Cohen and Max Welling. Group equivariant convolutional networks. 2016.
International
conference machine learning, pp.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03395. 2020.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large- scale hierarchical image database. In 2009 IEEE conference computer vision and patlern recognition, pp. 248-255. Icee, 2009.
Santosh K Divvala, Derek Hoiem, James H Hays, Alexei A Efros, and Martial Hebert. An empirical study of context in object detection. In 2009 IEEE Conference on computer vision and Patlern Recognition, pp. 1271-1278. IEEE, 2009.
Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recognition performance under visual distortions. In 20/7 26th international conference on computer communication and networks (ICCCN), pp. 1-7. IEEE, 2017.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. International Conference on Machine Learning, pp. 1802-1811, 2019.
Alhussein Fawzi and Pascal Frossard. Manitest: Are classifiers really invariant? arXiv preprint arXiv:1507.06535. 2015.
Carolina Galleguillos and Serge Belongie. Context based object categorization: A critical survey. Computer vision and image understanding, 114(6):11-7222 2010.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12233 2018a.
10

---

Published as a conference paper at ICLR 2021
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in neural information processing systems, pp. 7538-7550, 2018b.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(1 1:665-673, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1442.6572, 2014.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoe Le, and Jonathon Shlens. Using videos evaluate image model robustness. arXiv preprint arviv: 1904. 10076, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Geremy Heitz and Daphne Koller. Learning spatial context: Using stuff to find things. In European conference on computer vision, pp. 30-43. Springer, 2008
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. 122611, 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Sara Beery, et al. Wilds: A benchmark of in- tn-the-wild distribution shifts. arXiv preprint arXiv:2012.07421, 2020.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097- 1105, 2012.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00900 2018
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324, 1998.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. 11-104. IEEE, 2004.
Han S Lee, Alex A Agarwal, and Junmo Kim. Why do deep neural networks still not recognize these images?: A qualitative analysis on failure cases of imagenet classification. arXiv preprint arXiv:1709.03439 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1XXv:1706.0688. 2017.
11

---

Published as a conference paper at ICLR 2021
Tomasz Malisiewicz and Alexei A Efros. Improving spatial support for objects via multiple segmentations. 2007.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of -fttt---iitribution generalization. arXiv preprint arXiv:2010.15775, 2020.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 427-436, 2015.
Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge Belongie. Objects in context. In ICCV, volume I, pp. 5. Citeseer, 2007.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019.
Amir Rosenfeld and Daphna Weinshall, Extracting foreground masks towards object recognition. In 2011 International Conference on Computer Vision, pp. 1371-1378. IEEE, 2011.
Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv: /808. 03305, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):2112522 2015.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee baased localization, In Proceedings of the IEEE international conference on computer vision, pp. 618- 626, 2017.
Thomas Serre. Deep learning: the good, the bad, and the ugly. Annual Review of Vision Science, 5: 399-426, 2019.
Thomas Serre, Aude Oliva, and Tomaso Poggio. A feedforward architecture accounts for rapid categorization. Proceedings of the national academy of sciences, 104(15):6424 6429, 2007.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang, Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy on imagenet. In International Conference on Machine Learning (ICML), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X9.v11009.555 2014.
Bharat Singh and Larry Davis. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2577-3558) 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXiv:1312.6799 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
12

---

Published as a conference paper at ICLR 2021
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification, arXiv preprint arXiv:2007.00644, 2020.
Antonio Torralba and Pawan Sinha. Statistical context priming for object detection. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 1, pp. 763-770. IEEE, 2001.
Shimon Ullman, Liav Assif, Ethan Fetaya, and Daniel Harari. Atoms of recognition in human and computer vision. Proceedings of the National Academy of Sciences, 113(10):2744-2749 2016.
Anna Volokitin, Gemma Roig, and Tomaso A Poggio. Do deep neural networks suffer from crowding? In Advances in Neural Information Processing Systems, pp. 3628-5638 2017.
Kai Xiao, Logan Engstrom, Andrew llyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006. 09994, 2020.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint 1eXiv:1901.080 2019.
Richard Zhang. Making convolutional networks shift-invariant again.
arXiv preprint
arXiv:1904. 111.904..111..... 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. JEEE transactions on pattern analysis and machine intelligence, 4056))1452-1464, 2017.
Zhuotun Zhu, Lingxi Xie, and Alan Yuille, Object recognition with and without objects, 2016.
13

---

Published as a conference paper at ICLR 2021
A
FREQUENCY OF THE IMAGES PER CATEGORY
frequency
frequency
88
000
05
000
5
Mixing Salad Bowl-
Envelope-
Drinking
Sandal
Speaker
Bucket
Bottle cap-
Lipstick-
Band Aid
Wetch
Paintbrush
Trash bin
Vese
Water bottle-
Wine bottle
Backpack
TV
Banana
Weight scale
Candle
Measuring cup
Broom
Ladle
Cellphone
Salt shaker
Desk lamp
Pitcher
Bread
Laptop (open)
Chair
Ruler
Beer bottle
Drill
Mug
Bath towel
Running shoe
Lemon
bilet paper roll
Vacuum cleaner
Can opener
Swimming trunks
Bicycle
Dishrag hand towel
Computer mouse.
Pillow
Alarm clock.
Butcher's knife
Hammer
Stuffed animal
Winter glove
Pen
Orange
Wallet
Helmet
Coffee/Trench press.
Weight (exercise)
Plate
Pop can
Fan
Plunger
Sunglesses
Bench
Binder (closed)
Tle.
Remote control.
Printer
Sock
Wek
Basket
Lampshade
Doormat
Wheel
Safety pin
Microwave
dryer
Monitor
Iron clothes)
Frying pan.
Strainer
Tespot
Jeans
Match
Necklace
Skirt
T-shirt
Whistle
Pill bottle
Still Camera
Lighter
Combination lock
Spatula
Portable
Suit jacket
Dress shoe (men)
Nail (fastener)
Plastic bag
Tennis racket
Paper towel
Shovel
Padlock
Letter opener
bassee
Sleeping bag...
Clothes hamper
Drying rack plates
Soap dispenser
Keyboard
Sweater
Soup Bowl
Umbrella
Try.
Figure& 8: Frequency of the images (one object label per image) over the 113 categorics of overapee with the ImageNet. The right bar chart is the continuation of of the left one.
14

---

Published as a conference paper at ICLR 2021
B MODEL ACCURACY PER CATEGORY USING BOXES VS. FULL IMAGE
Sno
Hone Sale
Figure 9: Performance of models on object bounding boxes (from left to right: AlexNet, VGG, and GoogLeNet).
15

---

Published as a conference paper at ICLR 2021
Prac
201
Wedk
Now
Smplene
Vocum Dtowe
ande
Hensrngo co
Long N1000
Chcur
Seac
tork
SE
Figure 10: Performance of models object bounding boxes (from left right: ResNet, Inception3, and MNASNet).
16

---

Published as a conference paper at ICLR 2021
H D
Oring
Sult
Wilt Prace
foltr
Lowethede
Money
Mong
Dene
Contraation
1op
Eniclooo gland
SE
Figure 11: Performance of models the entire image (from left to right: AlexNet, VGG, and GoogLeNet).
17

---

Published as a conference paper at ICLR 2021
15
3rB
Driing not
Cer
Movig Tran
Centreetin
Ribro Seldd
Rirve Eorc-
veo
Fac
SE
Figure 12: Performance of models on the entire image (from left to right: ResNet, Inception3, and MNASNet).
18

---

Published as a conference paper at ICLR 2021
C EASIEST AND HARDEST OBJECTS FOR THE RESNET- 152 MODEL
"ha did data dal cuile 10551.5
Toe clss, ditaal deut
12
2
10
2
9
3
12
8
4
7
7
10
8
Correctly classified; highest confidences
Siga cirlareco J530.00
contares tharts
cordedd 26191
9:59
(b) Correctly classified; lowest confidences
dighal Pod wull 1806.
mul ad Par
HINE EINAAIT AMD 1611
8
7
12
5
(c) Misclassified; highest confidences
Tacurily dul deta dak
The snuli pocu dgual de1t
THE endlgg dioc dhia
THE cooc diaal deik Pee
(d) Misclassified; lowest confidences
Figure 13: Correctly classified and misclassified examples from the Alarm clock class by the ResNet model.
19

---

Published as a conference paper at ICLR 2021
Fro seder
3791700
rafdeoo Lenn
Pull Leau
And loae
(a) Correctly classified; highest confidences
ozáaaa 2815
2k0837
(b) Correctly classified; lowest confidences
Ond OREAnCE 27617.0
Cad 2013412
1Nd
ThE Dxppetac Sogpeted inderro 17EE
(c) Misclassified; highest confidences
WST rock gytho yuthon cakco 2007
and Cot
27
(d) Misclassified; lowest confidences
Figure 14: Correctly classified and misclassified examples from the Banana class by the ResNet model.
20

---

Published as a conference paper at ICLR 2021
Tel 8010 Ad GLIRIG
Txe Band-A LaBAni
uerd kd LABATI
Correctly classified; highest confidences
Band
Dasd
cebbery 119599
rabiber 1682
cybberr 7687
BAND-AI
DAVND HILD
(b) Correctly classified; lowest confidences
Exp melebwe A811
1kg
wilting eV-iven
mobbee 11011
rediew 1011-0
(c) Misclassified; highest confidences
Créldees 182221
spanily carldrncc LTRARI
conkdencs 1800.00
crldirss 1986
(d) Misclassified; lowest confidences
Figure 15: Correctly classified and misclassified examples from the Band-Aid class by the ResNet model.
21

---

Published as a conference paper at ICLR 2021
InA pot
235
(a) Correctly classified; highest confidences. Only three benches correctly classified.
Frd banviner tanitec bel arde salites hande!
Pad thei
and malles porteg
Frd timade
(b) Misclassified; highest confidences
parkbbach
OLE00 to 1502
On Verue 2013.0
Grbbim 2. 180544
GRTANE 1535.0
Misclassified; lowest confidences
Figure 16: Correctly classified and misclassified examples from the Bench class by the ResNet model.
22

---

Published as a conference paper at ICLR 2021
(a) Correctly classified; highest confidences. Only two plates were correctly classified.
Wid doneerrrr CAREETTAAAEE PCCE gaila vaflac
RTd ircicaad NES PaleEry Sanion
Tie and sprllcdiyy badisds AccEes bakks sod led 2o0s
The diox
(b) Misclassified; highest confidences
300 CRESeC 158175
Find deraaader aarooiiing nochine cal fale 104311
Tole Co6ker 2011
oallee 1E9T
(c) Misclassified; lowest confidences
Figure 17: Correctly classified and misclassified examples from the Plate class by the ResNet model.
23

---

Published as a conference paper at ICLR 2021
Thet
Teet Booe
The keer
Correctly classified; highest confidences
Pic lanc
(b) Correctly classified; lowest confidences
Not UVAN PM20
Misclassified; highest confidences
sefferee 220012
confdencs 1124
conlitiee 118316
coridencs IND
(d) Misclassified; lowest confidences
Figure Correctly classified misclassified examples from Broom class by ResNet model.
24

---

Published as a conference paper at ICLR 2021
Toe Lrbitree 6096
(a) Correctly classified; highest confidences
Bai vaile
(b) Correctly classified; lowest confidences
Ned w2t2 po
For ogencs
060.00
(c) Misclassified; highest confidences
Pocigral 203 9K
Poccoande lepe lph
Sae our dei Leen light Jelisrr
(d) Misclassified; lowest confidences
Figure 19: Correctly classified and misclassified examples from the Candle class by the ResNet model.
25

---

Published as a conference paper at ICLR 2021
net Pare DA SlA
The party SA
no parte On, S0a
Fnd demtrra Dowe
Correctly classified; highest confidences
redi 201
vulll Fur 2013
(b) Correctly classified; lowest confidences
SAIE PENCDAES NADeevi Busta cc 206
cediove 201
(c) Misclassified; highest confidences
welkee 2007.8
confibrns- 2006
crbArc 232-21
(d) Misclassified; lowest confidencess
Figure 20: Correctly classified and misclassified examples from Fan class by the ResNet model.
26

---

Published as a conference paper at ICLR 2021
not 14C 100
NATARAJ E21 SCALE
(a) Correctly classified; highest confidences
conldincs
colldins 2000.00
ON 201124441
(b) Correctly classified; lowest confidences
THA dill TAM
(c) Misclassified; highest confidences
Indd aiiio Valedon
and dedial
ride
Wediialla del WitEIIETTT 0/21 kens
(d) Misclassified; lowest confidences
Figure 21: Correctly classified and misclassified examples from the Ruler class by the ResNet model.
27

---

Published as a conference paper at ICLR 2021
The widram Gcttbane 3700223
wadyya 39022-22
Correctly classified; highest confidences
for waraaa
Pho uildy ort
su55 crlldena 24
(b) Correctly classified; lowest confidences
Pic led0
and Nedib
and couedhor
Tred COENC TOTS PuEDe, Sump
(c) Misclassified; highest confidences
Pecivatters 095560 189018
crelorsos 196-41
Misclassified; lowest confidences
Figure 22: Correctly classified and misclassified examples from the Safety-pin class by the ResNet model.
28

---

Published as a conference paper at ICLR 2021
ILIRI 1003
CabAnr 3595
Crliurse 25 056408
LYBARCE 0547
(a) Correctly classified; highest confidences
Text Rosp
Tae kaod
(b) Correctly classified; lowest confidences
ood 1::31:1
callro FEEL
cunldress 14B
ipat
Crlarics sanc 20
bark
(c) Misclassified; highest confidences
Fse Super
Tal Seade
Teo Seppll
Tal SE30
Misclassified; lowest confidences
Figure 23: Correctly classified and misclassified examples from the Teapot class by the ResNet model.
29

---

Published as a conference paper at ICLR 2021
Sefteaee $22004
collstnce 102213
Mrveervaa zunisanc 100
eviidece 0830
(a) Correctly classified; highest confidences
Tae Ideisios waw www.dass.o. syder
Tue servici aebeeio setwer
(b) Correctly classified; lowest confidences
Ned nen by
fdivaras Inbenerrppien Red neator
not beacthodd banctects
Fed mands
upiee
(c) Misclassified; highest confidences
PRC IREECE
WLVVOR
Pret: wwwreshar weherr dw0.. 1ogg coNftIooe
Trd lome
botteeaee Spaderr (pedter confibero
(d) Misclassified; lowest confidences
Figure 24: Correctly classified and misclassified examples from the TV class by the ResNet model.
30

---

Published as a conference paper at ICLR 2021
Tae add
Tae aid
Tae and
Tac ALd
Correctly classified; highest confidences
oefiens 220064
caréirnc DiNeor
confdbece LYBS
CHrddrcs 2389
(b) Correctly classified; lowest confidences
Tas Rock
Pacc uodi
(c) Misclassified; highest confidences
soci coaftsnnon 19s1.17
Tal
(d) Misclassified; lowest confidences
Figure 25: Correctly classified and misclassified examples from the Sock class the ResNet model.
31

---

Published as a conference paper at ICLR 2021
Poe salioo de dRANe FUMM
nuggare ERAM Wes pavee den diioo
sundiaie GRAM CaMN Pull surg
TLe sang SHE MUM your Pakk
(a) Correctly classified; highest confidences
THe merfmm del slown vedeo
Yoer end poves del slowws shakks
Thi singlinnuy de plason stades
The seg.eegnn del Jmmn. 1100
(b) Correctly classified; lowest confidences
Yot sung
Nec Meefleiin eo dediylooss
Due laa 28
Pust
Telt dove
Frx gell
5300
(c) Misclassified: highest confidences
RIm gaure shades 100
ang shndes
TIee Aag mue deawes hark
nee ardaad hovter Mett Dooes
glens
(d) Misclassified; lowest confidences
Figure 26: Correctly classified and misclassified examples from the Sunglasses class by the ResNet model.
32

---

Published as a conference paper at ICLR 2021
howit, bodde Aldder
bondee, boatte Wlnder
The
how heat
Correctly classified; highest confidences
2103
(b) Correctly classified; lowest confidences
Tuac Widtue
Windue Beeon Ne1t
Livic 511911
The taw MivOOT1S
(c) Misclassified; highest confidences
Pee te coe
Pes Geed 1d
"he sin censaler' parse wodiier ag
Misclassified; lowest confidences
Figure 27: Correctly classified and misclassified examples from the Tie class by the ResNet model.
33

---

Published as a conference paper at ICLR 2021
coliseee 0080
conlsence 0180
(a) Correctly classified; highest confidences
Facccoie 2
(b) Correctly classified; lowest confidences
adorre crlden
(c) Misclassified; highest confidences
Tea inlaaiaa
The cotee rug
Tue cotee
187122
TALK NERDY
MMM
(d) Misclassified; lowest confidenees
Figure 28: Correctly classified and misclassified examples from the Mug class by the ResNet model.
34

---

Published as a conference paper at ICLR 2021
mLIA 39995
ELme 39523 55
ELORE NJ 33283 20
L9 350
(a) Correctly classified; highest confidences
The Sannw
Tha SAnPAr
The Faneeer
(b) Correctly classified; lowest confidences
End Tahiue
Peel ud crédenci
and cardones
daaniikk corldinc
(c) Misclassified; highest confidences
Thai Nammer
The Numeorr
The Nummee
The Komener
Misclassified; lowest confidences
Figure 29: Correctly classified and misclassified examples from the Hammer class by the ResNet model.
35

---

Published as a conference paper at ICLR 2021
Lie orddrca 005125
blocahas cor cwase 5014
maurtii bil,, allersi bal oToa teysA contionn (31240
offe
(a) Correctly classified; highest confidences
bila, bid rowara in Biiny al inreci affinc
Slerau Wedt Eiaal hell Por tonti Beyily
Wal kriliieel fo tanta hayiyy
(b) Correctly classified; lowest confidences
The came bhac
stamannlee offnne Stid
martin Non altaman V10
and
(c) Misclassified; highest confidences
Awd Lio
IsE nouttal biles
THE www.riiiiiii bis allssshblls affra world
FIE dfrna And eerem
And wintee
(d) Misclassified; lowest confidences
Figure 30: Correctly classified and misclassified examples from the Bicycle class by the ResNet model.
36

---

Published as a conference paper at ICLR 2021
prore nelibate Ican 10
phone
show
carf bari was
cer cereeee and
natires 1ets ia
Correctly classified; highest confidences
chore and wit E yo ce
Wat vriluie pelipiino call
show
and cill dut recid aer pharn
And colloarto redd cille dlarr
(b) Correctly classified; lowest confidences
blephen
can 0000007771) 731E33
car beoc 2011
cor peno d0
combencc 135333
(c) Misclassified; highest confidences
The cill Ielchera odlderphor ofll
The calltie hele celle www.hh.w....o..
The cobldee belept ore vellrer www.ehhwee....
The colite holee oflire dew.crl
(d) Misclassified; lowest confidences
Figure 31: Correctly classified and misclassified examples from the Cellphone class by the ResNet model.
37

---

Published as a conference paper at ICLR 2021
You bebendai: dui foling chul sooli
Tee barte dal Diling chul socing dei
Tha butter wa Sollng diac rocong
The butbe cuk Bollng
(a) Correctly classified; highest confidences
THE the Gleo 1ee
barber tha intbae
Toe
2017
(b) Correctly classified; lowest confidences
cpn VATe!
(c) Misclassified; highest confidences
nat VILLE Canrar ownn
supiig cand swoe 221..5
ceetires 270
(d) Misclassified; lowest confidences
Figure 32: Correctly classified and misclassified examples from the Chair class by the ResNet model.
38

---

Published as a conference paper at ICLR 2021
D SOME CHALLENGING EXAMPLES FOR HUMANS
Figure 33: A selection of challenging objects that hard be recognized by humans. Can you guess the category the annotated objects these images? Keys are follows:
row 1: (skirt, skirt, desk lamp, safety pin, still camera, spatula, tray),
row 2: (vase, pillow, sleeping bag, printer, remote control, pet food container, detergent), row 3: (vacuum cleaner, vase, vase, shovel, stuffed animal, sandal, sandal),
4: (sock, shovel, shovel, skirt, skirt, match, spatula),
row (padlock, padlock, microwave, orange, printer, trash bin, tray)
39

---

Published as a conference paper at ICLR 2021
Figure 34: A selection of challenging objects that are hard to be recognized by humans (continued from above).
Can you guess the category of the annotated objects in these images? Keys are follows:
row 1: (remote control, ruler, full sized towel, ruler, remote control, remote control, remote control), row (remote control, calendar, butter, bookend, rider, tray, desk lamp),
row 3: (envelope, envelope, drying rack for dishes, full sized towel, drying rack for dishes, drinking cup, desk lamp).
row 4: (desk lamp, desk lamp, dress, tennis racket, fan, fan, hammer),
5: (printer, toaster, printer, helmet, printer, printer, printer)
40

---

Published as a conference paper at ICLR 2021
E ANNOTATION ISSUES IN OBJECT RECOGNITION DATASETS
(a)
picket
(e)ffrican elephant
desktop compatee
Somputer
eqpecsoo maker
screwdavver
curpersar's kit
digital watch
deskSop
pió
mouse
deli:
witchook
TENTSG
striiner
collutar telepiame
mamiie
eqpren
lotler openee
flute
ohoe
pizzz
Band Asd
spoclight
morabboar
mek crab
matchstick
liptick
tudle
projectex
puckec
eest beht
black gold tick
dandaa spiiks
barn spicks
qpider
black, widdns
Some examples from the ImageNet dataset highlighting why multi-label annotations are necessary: (a) Multiple objects. Here, the image lahel is desk but screen, monitor, coffee mug and many more objects in the scene could count correct labels, (b) Non-salient object. A scene where the target label picket fence is counter intuitive hecause it appears in the image hackground, while classes groom, bow-tie, suit, gown, and possibly hoopskirt are more prominently displayed the foreground. с) Synonym or subset relationships: This image has ImageNet label African elephant, but can be labeled tusker as well, because every African elephant with tusks is tusker. d) Unclear images: This image is labeled lake shore. but could also be labeled scashore there not enough information in the scene distinguish the water body between lake sea. Image compiled from (Shankar al., 2020). Bottom: Some annotation problems in the ImageNet dataset. Example images along with their ground truth (GT) labels (red) and predicted classes (PC) by model (blue) are shown. Top-left) Similar labels, Top-right) non-salient GT, Bottom-left) challenging images, and Bottom-right) incorrect GT. Please see Lee et al. (2017) 7) for details.
Figure 35: Top:
41

---

Published as a conference paper at ICLR 2021
F ANALYSING MODEL ROBUSTNESS OVER NATURALLY DISTORTED IMAGES
Severity
1
2
3
brightness
contrast
defocus
blur
elastic
transform
fog
frost
gaussian
noise
glass blur
impulse
noise
jpeg
compression
motion blur
shot noise
snow
zoom blur
Figure 36: Sample images alongside their corruptions severity levels.
42

---

Published as a conference paper at ICLR 2021
0.
25
Dlo22 buT
blr
STOW
05
30
frest
Fgg
Inghhness
controct
Searity
dladic
25
3000rr
300eri
Figure Top-1 (dashed lines) and Top-5 (solid lines) accuracy models over natural image distortions 3 severity levels (using object bounding box).
43

---

Published as a conference paper at ICLR 2021
V00-3 Tellesa52
1.0
05
2.5
3.8
glo2G bur
bli
STOW
AECANCES
0.0
05
frest
Pgg
Inihhness
controct
25
Qeartty
Seanity
dladic Larraton
pop_compeooon
0.0
20
25
300010
Figure Top-1 (dashed lines) and Top-5 (solid lines) accuracy models over natural image distortions 3 severity levels (using full image).
44

---

Published as a conference paper at ICLR 2021
ADVERSARIAL DEFENSE USING FOREGROUND DETECTION ON MNIST G
AND FASHION MNIST
>
6-0
Sppo
8
S
8
S
Epp.
5
O
2
/
6-0-8
6-9-3
1-2-2
1-8
3-9-3
9-2-8
5-9.0
1-22
2-24
3-2
1
6
/
3
Eph 0.55
9
2
27
3
2'
2
5
4
3
7
Ep 0.1
4
4
Epo
ESX C15
4
Y
8
7
3
Esc. 0.15
3
6
3
3
Spx 0.2
3
Ept. 0.2
3
P
2
6-2-
2-+1
Epo 025
Epo. 0.25
3
2/66
4iv
1>2
6
4
9
3.+22
3
5
Sp 0.3
4
Epon
Epp. 0
5
8
5
6
3'2
Bon 000
>>
4
Spp EL3
8
g
B-2
9-21
EDR. 0:33
$
U
9
9
1
1=1
7:5
3-5-2
0-9.2
8:9-9
8->3
5:5E
3
8
$
Epo
Bac
7-33
9:0.3
1
9-6-7
2
5-5-3
Sp 83
3
3
Figure 39: Tef-left) Model, trained on clean MNIST data, is attacked by FGSM, Top-right) Foreground detection model that has been trained clean data. Bottom-left) Model, trained noisy MNIST data, attacked by FGSM, Bottom-right) Foreground detection of model that has been trained noisy data. Noisy data is created by overlaying an object over white noise i.e., noise x (1- mask) + object. We find that background subtraction together with edge detection improves robustness.
45

---

Published as a conference paper at ICLR 2021
Epaco
Epo.
Epr. 0.05
Epo:
S
9.2
EsP 0.3
5-9-9
1-5-3
1-9
9-11
3:50
9-1
8-00
1-95
2-255
9-21
Epk 0:55
Epx
4.56
6:44
1-55
0'6
1
8
0
7555
Ep 02
Epr 02
Expe 0.25
Egn 025
Epac D.3
Tpr 0.3
2-92
9-2
19:1
9-2
Epp
S
6-
6-0
Epp 05
Epp
Epc. 63
Eper
Epes 0.15
Epo
Sppe B.
Ep
296
755
Epe
Epe
Figure 40: Top-left) Model, trained clean MaSHionMNIST data, is attacked by FGSM, Top-right) Foreground detection over a model that has heen trained on clean data. Bottom-left) Model, trained on noisy FaSS data, is attacked by FGSM, Bottom Bottom-right) Foreground detection of model that has been trained on noisy data. Noisy data is created by overlaying object over white noise i.e., noise x (I- mask) + object. We find that background subtraction together with edge detection improves robustness.
46

---

Published as a conference paper at ICLR 2021
H
STATISTICS OF OBJECTΝET DATASET
object ratio
0.5
020
0.15
0.10
0.05
0.4
object size / immae size
0.8
0.6
0.4
0.2
0.2
0.4
0.6
0.8
Object scale
Figure 41: Top: Distribution of object scale in ObjectNet dataset (1 13 annotated classes), Bottom: Distribution single-object localization (dark green) and PASCAL VOC 2012 (light bluc)
object scale in ILSVRC2012-200 validation sets. Object scale is fraction of image area occupied by an average object instance. Please see Fig. 16 in Russakovsky et al. (2015).
25
100
125
150
175
200
0
50
100
150
200
Figure 42: Distribution of object location in ObjectNet (113 annotated classes).
47


Reviews:
[{"review_id": "h3ReLoWq6wa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Real-world applications frequently provide challenges that are not seen in common computer vision datasets like Imagenet, where images are blurry, dark, corrupted, objects are highly occluded, test objects may be out of distribution due to natural distribution shifts, etc.  This phenomenon was investigated in 2018 by Beery et al, who similarly found categorization is easier with localization in challenging real-world scenes (ie classifying cropped boxes). I would recommend taking a look at that paper (citation below) and including it in the related work.\n\nBeery, S., Van Horn, G., & Perona, P. (2018). Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 456-473).\n\n\nPros:\n\nThe authors provide many experiments digging into various aspects of what makes real-world object detection challenging. This is a useful reference point for future work.\n\nCons:\nThis paper re-analyzes generalization in the context of an existing dataset. There is not anything particularly novel about the analysis, and similar results have been shown on other real-world datasets.\nThe authors describe the image distortions they consider to be “natural”, but are applying them synthetically. It is not clear to me that applying a synthetic distortion of a type that can be seen in the real world is necessarily reflective of those realistic distortions in the wild.  It would be better to explicitly collect examples of these types of distortions in real data and compare against that.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XlynfQFTzr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper revisits ObjectNet dataset closely and found applying classifiers on object bounding box significantly reduces the gap between ImageNet and ObjectNet. The authors further investigates the robustness of CNNs against image perturbations and adversarial attacks, and found limiting the object area to their segmentation mask significantly improves model accuracy (and robustness). Qualitative evaluation is also performed over confident and less-confident / incorrect model predictions and find it correlates with human perception.\n\nPros:\n- More analyses like this paper does would help bridge the gap between ML/CV model performance in staged datasets and real-world scenarios\n- The experimentation conducted in this paper is comprehensive, accompanied with many in-depth inspection over a large dataset. The insights drawn from this paper would be invaluable for researchers working in this field.\n\nCons:\n- My major concern with this paper (and the main factor of rating it clear rejection) is the experimental setup used in section 3.3. From authors they \"selected images from ten classes of the ObjectNet dataset ... manually annotated the object of interest in each image\". Then \"Models were trained on 70 images per category\".  (also from Figure 39 \"In total we annotated 1K images across ten categories of the ObjectNet dataset.\"). If interpreted correctly, the models are trained on part of ObjectNet images which clearly violates dataset license \"ObjectNet may never be used to tune the parameters of any model.\" (https://objectnet.dev/download.html). \n\n- While appreciating the authors conducting study on model robustness, the conclusion drawn from several experiments seems to confuse \"performance\" v.s. \"robustness\", where the former indicates model have better accuracy, and the latter measures how model accuracy varies with increasing noise / perturbations. See more details below.\n\nSome (minor) comments:\n\n- Sec 3.1. 1) I share the concern from authors that \"object detector\" should be not confused with \"object recognition\" (or commonly used \"image classifier\"). Hopefully vision community could use more consistent terms across literatures.\n\n- Sec 3.1 1) While detection dataset would surely have more scale variation (and truncation / occlusion due to many objects are not in the center), it is not entirely clear that object in detection datasets \"vary more in parameters such as lighting, ..., and blur\".\n\n- Sec 3.1 1) It would be great to see more analysis on detection datasets (authors mentioned they will discuss in section 4, but only with very little analysis).\n\n- Sec 3.1, 2) While ImageNet and ObjectNet have distinct characteristics, having some stats on object size / spatial placement might better illustrate the gaps between these datasets.\n\n- Sec 3.1 3) Agree top-5 might make classifiers' life easy, but it is more of an eval metric rather than training loss (the model still need to predict top-1 class correct during training). Meanwhile, it is not very clear why multi-label annotation would bias against model that \"is spatially more precise but misses some objects\". That model should be evaluated against detection benchmarks, rather than object recognition (image classification) datasets.\n\n- Sec 3.1 \"bounding box annotation\". For multiple objects nearby with the same category, the annotation would include all of them in one bounding box. This might leads to bad aspect ratio? (in general, bounding box would also vary more in aspect ratio than images, and feeding bounding box into a square CNN seems to be less ideal)\n\n- Sec 3.1 \"object recognition results\". \"AlexNet, VGG-19, and GoogLeNet have also been used\", GoogLeNet should be \"ResNet-152\"? ObjectNet uses inception (GoogLeNet) v4 while authors use v3.\n\n- Sec 3.2.1 \"The higher the prediction accuracy, the higher the robustness against natural distortions\". This is not necessarily true. Looking at Figure 3, it seems all models and both image / bounding box schemes would have decayed performance w.r.t distortion severity, and their slopes seem similar.\n\n- Sec 3.3 \"Despite a large body of literature on whether and how much visual context benefits CNNs, the majority being focused on model accuracy, the matter has not been settled yet\". \"the matter\" means context for robustness? please clarify.\n\n- Sec 3.3 \" Around 35.5% and 12.6% of the image pixels fall inside the object bounding box and the foreground object, respectively. Around 58.5% of the bounding box pixels fall inside the object mask.\". here foreground object and object mask should be the same thing? if so, the numbers seem not matching (would expect box-to-image ratio * mask-to-box-ratio = mask-to-image ratio, or these numbers are normalized differently)?\n\n- Figure 5: it is clear that seg-mask actually isn't robust to adversarial attacks (accuracy dropped significantly), which contradicts with the claim from authors.\n\n- Sec 3.3 \"An attempt was made to tune the parameters to attain the best test accuracy in each case\", could authors elaborate? \n\n- Sec 3.3.1. here the results seem to indicate segmask is more robust (less variations). Would the segmentation mask itself be indicative for object categories? It might be interesting to predict object classes directly from those masks as a baseline.\n\n\nFinal review:\n\nThe authors updated the manuscript and removed tuning experiment on ObjectNet. I am still a bit concerned about the definition of \"robustness\", but the paper overall does look good for ICLR publication. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WTDDyFHGi15", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like the main ideas articulated in the paper, but find the writing lacks some clarity: \n\nSummary of paper: The paper  takes as a starting point the study from Barbu et al where the robustness of object recognition pipelines to be able to handle distribution shifts are studied by testing ImageNet trained architectures against ObjectNet.  The main point in the current paper is that the performance degradation seen in Barbu et al is due to the fact that the CNNs were processing the image with entire image as context  and when one only provides a sub-window around the objects of interest the resulting performance improves significantly.  The paper also describes experiments with various synthetic distorted data and finally examines details of ObjectNet dataset to illustrate that there are images that are hard to categorize even for humans. Thus, the paper concludes that object recognition on ObjectNet is still hard to solve.\n\nIt is clear that by using bounding boxes or even removing background from those bounding boxes,  the performance will be better (since the training was on ImageNet with single objects). So, in a way, they are kind-of recreating the training distribution in order to improve the performance.\n\nMain significance of the paper (Pros): Detailed study of performance of object recognition and the empirical finding that figure-ground segmentation may improve recognition.  Analysis of properties of ObjectNet and its challenges.\n\nOriginality/Novelty:  The paper is largely empirical and has a good discussion of the relevant background literature analyzing object recognition systems.  \n\nCons:  It has incremental insights. \n\nClarity of paper: The description of the experiments are at times unclear.    The structure of the paper could be simplified with a table or diagram that illustrates the logic behind the experimentation and conclusion.  There are multiple datasets used, the training scheme is sometimes on ImageNet and tested on ObjectNet and sometimes on selected categories of ObjectNet and test on the rest of ObjectNet.  \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4MFn1zi8XZy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "OVERVIEW:\nThe authors present a follow-up to the prior work of Barbu et al on the task of Object Recognition* (name confusion addressed in cons below). Barbu et al demonstrated that on a more realistic dataset like ObjectNet, models trained on a clean dataset like ImageNet suffer significant degradation. This work reduces the performance gap by cropping out the object using bounding box or mask information and running the recognition model on top of it. They do this for a variety of models (AlexNet, VGG-19, ResNet-152, Inception-v4, NASNet-A, PNASNet-5L) and transformations (image distortions, adversarial perturbations, context, geometric transformations). \n\nPROS:\n- The paper is well-written and tackles an important topic of object recognition* in the wild. It tries to move away from the ImageNet driven approach that is currently present in the community to a more realistic scenario. \n- They build on the prior work of Barbu et al and are able to reduce performance gap demonstrated by Barbu et al by using bounding box or mask cropped images of the object of interest.\n- They present a lot of experimental evaluation using a variety of models and transformations and demonstrate that their findings hold across all settings.\n\nCONS:\n- I agree with the authors that ImageNet with a single (or few) object present in the center of the image with clear foreground-background separation is unrealistic. ObjectNet is a better snapshot of the real world and models trained on ImageNet suffer in ObjectNet. However, the proposed approach to crop out the object using bounding box information or mask information is moving the data distribution from the real world setting of ObjectNet closer to the ideal setting of ImageNet. This then leads to an expected improvement in performance. I appreciate the thoroughness of the results and evaluation presented but it does not feel like a novel contribution in my opinion. \n- The authors present 4 future research directions in Section 4. I would be more willing to accept the paper if one of these research directions is incorporated as a contribution. For example, the last research direction of applying an object detection model trained with MS COCO on ObjectNet images instead of an image classification model trained with ImageNet is something that is doable. I would encourage the authors to even consider an object detection model trained on LVIS which has a larger number of object categories. This moves further away from object recognition* to object detection + classification but the latter is what we would typically encounter in a real-world scenario. Even here, I would need some contribution or novel analysis besides re-running current experiments with detection models.\n- I strongly recommend a change of name to \"Contemplating Real-World Object Classification\" (no caps and classification instead of recognition). In my understanding, object recognition is a super-set of classification, detection, segmentation, etc. ImageNet leads to Image Classification models even if they are technically object classification. But sticking with the terminology of Object Recognition because it was used by Barbu et al is misleading. I would prefer Object Classification be used because that is the task of interest in this work.\n\nREASON FOR RATING:\nI think there is an interesting problem of real-world object classification that is of significant importance and this work moves a little closer to analyzing possible ways to reduce the performance gap from ImageNet to ObjectNet. However, their key contribution is somewhat expected (not novel) and needs some more work before being conference-paper ready.\n\nUPDATE:\nI have read the author feedback and the other reviews/discussions. I keep my original rating of 5. I think multiple authors raised the question of novelty relative to Barbu et al and the authors argue that they demonstrate the importance of context (whole image vs bounding box vs instance mask) for object recognition. Section 3.3 and Figure 5 is helpful in demonstrating it. However, the experimental setup is very limited (700 train + 300 test). COCO has 110K train and 5K val images and many more objects. If you argue that only 10 categories are common between COCO and ObjectNet, how many are common between LVIS and ObjectNet? I would strongly encourage the authors to leverage these pre-trained models and sharpen their message & contributions. I think they provide empirical justifications (important to the community) for expected results in moving from image to bounding box (same comment from multiple reviewers) but they need to de-emphasize that aspect and emphasize their results on context and robustness. A revision and resubmission to a different conference is encouraged.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that overall the two-pronged message of the submission has utility.

1. That ObjectNet is continues to be difficult for models to understand and is a challenging test platform even when objects are isolated from their backgrounds. This is significant and not obvious. Cropping objects makes the distribution shift between ObjectNet and ImageNet far smaller, but the large remaining performance gap points to the fact that detectors are limited by their ability to recognize the foregrounds of objects not by their ability to isolate objects from their backgrounds.

2. That segmentation could be a promising direction for robustness to adversarial perturbations which has so far been overlooked.
Decision: Accept (Poster)