Contemplating Real-World Object Classification

1 INTRODUCTION . Object recognition3 can be said to be the most basic problem in vision sciences . It is required in the early stages of visual processing before a system , be it a human or a machine , can accomplish other tasks such as searching , navigating , or grasping . Application of a convolutional neural network architecture ( CNN ) known as LeNet ( LeCun et al. , 1998 ) , albeit with new bells and whistles ( Krizhevsky et al. , 2012 ) , revolutionized not only computer vision but also several other areas . With the initial excitement gradually damping , researchers have started to study the shortcomings of deep models and question their generalization ability . From prior research , we already know that CNNs : a ) lack generalization to out of distribution samples ( e.g. , Recht et al . ( 2019 ) ; Barbu et al . ( 2019 ) ; Shankar et al . ( 2020 ) ; Taori et al . ( 2020 ) ; Koh et al . ( 2020 ) ) . Even after being exposed to many different instances of the same object category , they fail to fully capture the concept . In stark contrast , humans can generalize from only few examples ( a.k.a few-shot learning ) , b ) perform poorly when applied to transformed versions of the same object . In other words , they 1https : //objectnet.dev/ 2See https : //openreview.net/forum ? id=Q4EUywJIkqr for reviews and discussions . A prelimnary version of this work has been published in Arxiv ( Borji , 2020 ) . 3Classification of an object appearing lonely in an image . For images containing multiple objects , object localization or detection is required first . are not invariant to spatial transformations ( e.g. , translation , in-plane and in-depth rotation , scale ) as shown in ( Azulay & Weiss , 2019 ; Engstrom et al. , 2019 ; Fawzi & Frossard , 2015 ) , as well as noise corruptions ( Hendrycks & Dietterich , 2019 ; Geirhos et al. , 2018b ) , and c ) are vulnerable to imperceptible adversarial image perturbations ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Nguyen et al. , 2015 ) . Majority of these works , however , have used either the ImageNet dataset or its variations , and thus might be biased towards ImageNet characteristics . Utilizing a very challenging dataset that has been proposed recently , known as ObjectNet ( Barbu et al. , 2019 ) , here we seek to answer how well the state of the art CNNs generalize to real world object recognition scenarios . We also explore the role of spatial context in object recognition and answer whether it is better to use cropped objects ( using bounding boxes ) or segmented objects to achieve higher accuracy and robustness . Furthermore , we study the relationship between object recognition , scene understanding , and object detection . These are important problems that have been less explored . Several datasets have been proposed for training and testing object recognition models , and to study their generalization ability ( e.g. , ImageNet by Deng et al . ( 2009 ) , Places by Zhou et al . ( 2017 ) , CIFAR by Krizhevsky et al . ( 2009 ) , NORB by LeCun et al . ( 2004 ) , and iLab20M by Borji et al . ( 2016 ) ) . As the most notable one , ImageNet dataset has been very instrumental for gauging the progress in object recognition over the past decade . A large number of studies have tested new ideas by training deep models on ImageNet ( from scratch ) , or by finetuning pre-trained ( on ImageNet ) classification models on other datasets . With the ImageNet being retired , the state of the object recognition problem remains unclear . Several questions such as out of distribution generalization , “ superhuman performance ” ( He et al. , 2016 ) and invariance to transformations persist . To rekindle the discourse , recently Barbu et al . ( 2019 ) introduced the ObjectNet dataset which according to their claim has less bias than other recognition datasets4 . This dataset is supposed to be used solely as a test set and comes with a licence that disallows the researchers to finetune models on it . Images are pictured by Mechanical Turk workers using a mobile app in a variety of backgrounds , rotations , and imaging viewpoints . ObjectNet contains 50,000 images across 313 categories , out of which 113 are in common with ImageNet categories . Astonishingly , Barbu et al . found that the state of the art object recognition models perform drastically lower on ObjectNet compared to their performance on ImageNet ( about 40-45 % drop ) . Our principal goal here it to revisit the Barbu et al. ’ s analysis and measure the actual performance drop on ObjectNet compared to ImageNet . To this end , we limit our analysis to the 113 overlapped categories between the two datasets . We first annotate the objects in the ObjectNet scenes by drawing boxes around them . We then apply a number of deep models on these object boxes and find that models perform significantly better now , compared to their performance on the entire scene ( as is done in Barbu et . al ) . Interestingly , and perhaps against the common belief , we also find that training and testing models on segmented objects , rather than the object bounding box or the full image , leads to consistent improvement in accuracy and robustness over a range of classification tasks and image transformations ( geometric , natural distortions , and adversarial attacks ) . Lastly , we provide a qualitative ( and somewhat anecdotal ) analysis of extreme cases in object recognition for humans and machines . 2 RELATED WORK . Robustness against synthetic distribution shifts . Most research on assessing model robustness has been focused on synthetic image perturbations ( e.g. , spatial transformations , noise corruptions , simulated weather artifacts , temporal changes ( Gu et al. , 2019 ) , and adversarial examples ) perhaps because it is easy to precisely define , implement , and apply them to arbitrary images . While models have improved significantly in robustness to these distribution shifts ( e.g. , Zhang ( 2019 ) ; Zhang et al . ( 2019 ) ; Cohen & Welling ( 2016 ) ) , they are still not as robust as humans . Geirhos et al . ( 2018b ) showed that humans are more tolerant against image manipulations like contrast reduction , additive noise , or novel eidolon-distortions than models . Further , humans and models behave differently ( witnessed by different error patterns ) as the signal gets weaker . Zhu et al . ( 2016 ) contrast the influence of the foreground object and image background on the performance of humans and models . Robustness against natural distribution shifts . Robustness on real data is a clear challenge for deep neural networks . Unlike synthetic distribution shifts , it is difficult to define distribution shifts that occur naturally in the real-world ( such as subtle changes in scene composition , object types , and lighting conditions ) . Recht et al . ( 2019 ) closely followed the original ImageNet creation process 4ObjectNet dataset , however , has it own biases . It consists of indoor objects that are available to many people , are mobile , are not too large , too small , fragile or dangerous . to build a new test set called ImageNetV2 . They reported a performance gap of about 11 % ( top-1 acc . ) between the performance of the best deep models on this dataset and the original test set . Similar observations have been made by Shankar et al . ( 2020 ) . By evaluating 204 ImageNet models in 213 different test conditions , Taori et al . ( 2020 ) found that a ) current synthetic robustness does not imply natural robustness . In other words , robustness measures for synthetic distribution shifts are weakly predictive of robustness on the natural distribution shifts , b ) robustness measurements should control for accuracy since higher robustness can sometimes be explained by the higher accuracy on a standard unperturbed test set , and c ) training models on larger and more diverse data improves robustness but does not lead to full closure of the performance gap . A comprehensive benchmark of distribution shifts in the wild , known as WILDS , has recently been published by Koh et al . ( 2020 ) , encompassing different data modalities including vision . In D ’ Amour et al . ( 2020 ) , authors regard “ underspecification ” a major challenge to the credibility and generalization of modern machine learning pipelines . An ML pipeline is underspecified when it returns models that perform very well on held-out test sets during training but perform poorly at deployment time . Contextual interference . Context plays a significant role in pattern recognition and visual reasoning ( e.g. , Bar ( 2004 ) ; Torralba & Sinha ( 2001 ) ; Rabinovich et al . ( 2007 ) ; Heitz & Koller ( 2008 ) ; Galleguillos & Belongie ( 2010 ) ) . The extent to which visual context is being used by deep models is still unclear . Unlike models , humans are very good at exploiting context when it is helpful and discard it when it causes ambiguity . In other words , deep models do not understand what is the foreground object and what constitutes the background5 . Nagarajan et al . ( 2020 ) mention that ML models utilize features ( e.g. , image background ) which are spuriously correlated with the label during training . This makes them fragile at the test time when statistics slightly differ . As we argue here , this is one of the main reasons why deep models are so vulnerable to geometric and adversarial perturbations . Geirhos et al . ( 2020 ) have studied this phenomenon under the “ shortcut learning ” terminology from a broader perspective . Insights from human vision . CNNs turn out to be good models of human vision and can explain the first feed-forward sweep of information ( See Kriegeskorte ( 2015 ) for a review ) . They , however , differ from human visual processing in several important ways . Current object recognition methods do not rely on segmentation , whereas figure-ground segmentation plays a significant role in human vision , in particular for the encoding of spatial relations between 3D object parts ( Biederman , 1987 ; Serre , 2019 ) . Some computer vision works , predating deep learning , have also shown that pre-segmenting the image before applying the recognition algorithms , improves the accuracy ( Malisiewicz & Efros , 2007 ; Rabinovich et al. , 2007 ; Rosenfeld & Weinshall , 2011 ) . Unlike the human vision system , CNNs are hindered drastically in crowded scenes ( e.g. , Volokitin et al . ( 2017 ) ) . CNNs rely more on texture whereas humans pay more attention to shape ( Geirhos et al. , 2018a ) . Utilizing minimal recognizable images , Ullman et al . ( 2016 ) argued that the human visual system uses features and processes that are not used by current deep models . 5As an example , consider a model that is trained to classify camels vs. cows , with camels always shown in sandy backgrounds and cows shown against grassy backgrounds . Although such a model does well during training , it gets confused when presented with cows in sandy backgrounds at test time ( Beery et al. , 2018 ) . See also Rosenfeld et al . ( 2018 ) for another example in the context of object detection O ur a na ly si s O bj ec tN et p ap er < Recognizers by year > 13.86 31.19 ObjectNet Top-5 ( box ) ObjectNet Top-1 ( box ) 49.84 61.50 39.48 27.89 25-35 % performance drop A cc ur ac y % A cc ur ac y % 40-45 % performance drop Using our code

Summary: This paper revisits ObjectNet dataset closely and found applying classifiers on object bounding box significantly reduces the gap between ImageNet and ObjectNet. The authors further investigates the robustness of CNNs against image perturbations and adversarial attacks, and found limiting the object area to their segmentation mask significantly improves model accuracy (and robustness). Qualitative evaluation is also performed over confident and less-confident / incorrect model predictions and find it correlates with human perception.