Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Images

1 INTRODUCTION . Deep neural networks ( DNNs ) have reached tremendous success in recognizing and localizing objects in images . The fundamental approach that led to DNNs consists of building artificial systems based on the brain and human vision . Yet in many important aspects , the capabilities of DNNs are inferior to those of human vision . A promising strand of research is to investigate the similarities and differences , and by bridging the gaps , further improve DNNs ( Hassabis et al. , 2017 ) . Studying cognitive biases and optical illusions is particularly revealing of the function of a visual system , whether natural or artificial . Ullman et al . ( 2016 ) present such a striking phenomenon of human vision , called `` minimal images '' ( Ullman et al. , 2016 ; Ben-Yosef et al. , 2018 ; Ben-Yosef & Ullman , 2018 ) . Minimal images are small regions of an image ( e.g . 10x10 pixels ) in which only a part of an object is observed , and a slight adjustment of the visible area produces a sharp drop in human recognition accuracy . Figure 1 provides examples of human minimal images . Ullman et al . ( 2016 ) show that DNNs are unable to recognize human minimal images , and the DNN drop in accuracy for these minimal images is gradual rather than sharp . This begs the question of whether the sharp drop in accuracy for minimal images is a phenomenon exclusive to human vision , or there exist distinct but analogous images that produce a sharp drop in DNN accuracy . In this paper , we provide evidence for the latter hypothesis by showing that there is a large set of analogs to minimal images that affect DNNs . These are different from the minimal images for humans in several aspects , namely region size and location , and frequency and sharpness of the drop in accuracy . We find that a slight adjustment of a one-pixel shift or two-pixel shrink of the visible ∗Equal Contribution image region produces a drop in DNN recognition accuracy in many image regions . Figure 2 shows several examples of minimal image analogs of state-of-the-art DNNs . The adjustments of the visible area that affect DNNs are almost indistinguishable to humans and can occur in larger regions than the “ human minimal ” region . To describe this phenomenon we introduce fragile recognition images ( FRIs ) for DNNs , which are more general than minimal images : Fragile Recognition Image ( FRI ) : A fragile recognition image is a region of an image for which a slight change of the region ’ s size or location in the image produces a large change in DNN recognition output . This definition is more general than the definition of minimal images for humans , as the latter is included in the definition for DNNs . Minimal images are the case of fragile recognition in which the slight change is a reduction in the size of the region , and the minimal image is one of the smallest possible FRIs . In human vision , the more general definition of fragile recognition that we are introducing here is not useful because human minimal images appear only when the visible area of the object is small . Object recognition modelCabbage butterfly Since FRIs are hardly distinguishable to humans but cause DNNs to fail , there is a connection with so-called adversarial examples presented by Szegedy et al . ( 2013 ) . Adversarial examples are images with small synthetic perturbations that are imperceptible to humans , but produce a sharp drop in DNN recognition accuracy . There are several types , eg . ( Goodfellow et al. , 2014 ; Liu et al. , 2016 ) , and the strategies to alleviate them are not able to fully protect DNNs ( Tramèr et al. , 2017 ; Madry et al. , 2017 ) . Furthermore , humans may suffer from adversarial attacks ; human recognition ability is shown to be impaired by perturbations under rapid image presentation ( Elsayed et al. , 2018 ) . Unlike these adversarial examples , fragile recognition arise in natural images without introducing synthetic perturbation . This causes new concerns for use of DNNs in computer vision applications . We evaluate FRIs in ImageNet ( Deng et al. , 2009 ) for state-of-the-art DNNs , specifically VGG16 ( Simonyan & Zisserman , 2015 ) , Inception ( Szegedy et al. , 2014 ) , and ResNet ( He et al. , 2016 ) . Results show that FRIs are abundant and can occur for any region size . Furthermore , we investigate whether fragile recognition is related to the lack of invariance to small changes in the object location , which has been recently reported in the literature to affect DNNs ( Engstrom et al. , 2017 ; Azulay & Weiss , 2018 ) . Our results demonstrate that fragile recognition is independent from this phenomenon : bigger pooling regions reduce most of the lack of invariance to changes in object location , while pooling only marginally reduces fragile recognition . We also show that known strategies to increase network generalization , i.e . adding regularization and data augmentation , reduce the number of FRIs but still leave far more than humans have . These results highlight how much more fragile current DNN recognition ability is than human vision . 2 METHODS : EXTRACTING FRAGILE RECOGNITION IMAGES . In this Section we introduce the method for extracting FRIs for DNNs . The method for extracting human minimal images employs a tree search strategy ( Ullman et al. , 2016 ) . The full object is presented to human subjects and they are asked to recognize it . If at least 50 % of subjects recognize it correctly , smaller crops of the object , called descendants , are tested . If any of the smaller region is recognizable , the crop size is further reduced and tested . Once an image crop is found such that it is recognizable and all of its descendants are not recognizable , it is considered a human minimal image . Our FRI extraction method for DNNs relies on an exhaustive grid search . This is possible in DNNs due to parallelization across multiple GPUs , and would be prohibitively time-consuming to replicate with humans . The grid search consists on a two-step process : first , every possible square region is classified by the DNN and the correctness is annotated in the correctness map . From the correctness map , each region ’ s correctness is compared with the region ’ s slightly changed location or size in order to determine if there has been a change of the correctness . FRIs are regions that are classified correctly and a small change causes failure , as well as regions that are classified incorrectly and a small change causes success . The two step process to detect FRIs is summarized in Figure 3 . We now detail the two steps : 1 . Correctness map generation . An exhaustive grid search is performed to see if each possible square image region of a fixed size is classified correctly by a given DNN . After extracting the region from the image , the region is resized to be of the size required by the network . The region size is parametrized by P , which is defined as the proportion of the image occupied by the region , i.e . P = S/min ( h , w ) where h and w are the height and width of the input image , and S is the region ’ s side length . The results are arranged in a map such that a given map pixel contains the binary correctness for the square region centered at the corresponding pixel in the original image . The resulting map is of dimension ( h− S ) × ( w − S ) due to padding loss . The first two panels of Figure 3 show a visualization of the correctness map generation process . 2 . Fragile recognition image ( FRI ) extraction from correctness maps . We define different variations of FRIs , depending if they are based on changes on the location or size of the image region , and on how strict we are when evaluating the changes in the correctness of DNN : –Shift or Shrink : we define two types of `` small changes '' of the region that affect DNN correctness . `` Shift '' is a one-pixel translation of the region location ; `` shrink '' is a two-pixel reduction of the region side length within the region ’ s original boundaries , in a fixed full image size . –Strict or Loose : the visible region can be shifted in various directions , or shrunk in different ways while remaining within its original boundaries . `` Loose '' FRIs are regions such that there exists a small change that flip network correctness . `` Strict '' FRIs are regions such that network correctness is flipped for all small changes . These definitions yield four fragile recognition types : loose shift , loose shrink , strict shift , and strict shrink . Note that strict shrink is the most analogous to human minimal images . The correctness maps are used to detect fragile recognition due to shifts by comparing neighbouring pixels in a correctness map , and due to shrinks by comparing correctness maps at two slightly different region sizes . We use fragile recognition image ( FRI ) maps to visually represent the result of the grid search that extracts FRIs . As shown in the FRI map of Figure 3 , each pixel of the map indicates whether the corresponding window in the original image is an FRI . The second and third panel show a visualization of the FRI map generation process . Figure 4 shows an ImageNet image and each of its FRI maps . 3 FRAGILE RECOGNITION IMAGES OF STATE-OF-THE-ART DNNS . In this Section , we will first discuss occurrence of fragile recognition for state-of-the-art object recognition models through results based on ImageNet ( Deng et al. , 2009 ) . Then , we analyze if data augmentation and regularization help reduce fragile recognition through experiments in CIFAR-10 Krizhevsky ( 2009 ) . 3.1 FRAGILE RECOGNITION IMAGES FOR STATE-OF-THE-ART DNNS IN IMAGENET . The following experiments are performed on 500 images , randomly sampled from ImageNet ’ s validation set , which consists of 50 , 000 images and ground-truth object bound-in boxes ( Deng et al. , 2009 ) . The images are sampled from 10 supercategories ( dog , snake , monkey , fish , vegetable , musical instrument , boat , land vehicle , drinks , furniture ) , each of which covers a set of ImageNet categories . Correctness is measured as top-5 accuracy , which is commonly used in ImageNet . FRIs are extracted for three architectures : VGG-16 ( Simonyan & Zisserman , 2015 ) , Inception ( Szegedy et al. , 2014 ) , and ResNet ( He et al. , 2016 ) . Experiments are run in eight K80 NVIDIA GPUs . The exhaustive grid search takes about 5 minutes for one image using all eight GPUs . In the following paragraphs , we report the results of the experiments . We evaluate how much a DNN is affected by fragile recognition by quantifying the proportion of possible image regions that are FRIs , i.e . we quantify the amount of regions affected by a shift or shrink . Recall that we consider a region to be an FRI when the classification changes from correct to incorrect , and also from incorrect to correct . As expected , we found that both of these cases are approximately equally probable under all tested conditions . In Figure 5 , we show the percentage of shrink FRIs in the image for a given region size , P ; in Figure A.1 we show the same for shift FRIs . All networks are significantly affected by FRIs , with ResNet being the most : almost one out of 50 regions of a size that covers almost the entire image ( P = 80 % ) can affect ResNet . When the size of the image region is P = 20 % , FRIs are even more frequent ( 8 times more ) , i.e . almost two out of 25 regions in the image is a loose FRI for ResNet . Comparing Figure 5a and Figure 5b , we see that the proportion of strict FRIs is less than that of loose FRIs because of the more stringent definition . The results show that there are many regions in an image for which the network is very sensitive to slight changes in the region , eg . one out of 250 image regions of size P = 20 % will be misclassified by ResNet when the region is slightly shrunk . Recall that strict shrink FRIs are the equivalent case of minimal images in humans . Thus , these results demonstrate the existence of many minimal images analogs in DNNs . Note that smaller FRIs ( lower P ) are much more frequent than larger ones , as observed across the different network architectures and types of FRIs . This trend is expected : one-pixel shifts and two-pixel shrinks are proportionally larger changes for smaller regions , and larger regions generally allow for more high-level features to be included . We verified that FRIs are not an artifact of the algorithm that resizes the region to the size required by the DNN ( 224× 224 pixels for VGG-16 ) . We took regions of side length 224 and removed any resizing before input , and we observed that this procedure produces the same results we reported . Besides the qualitative examples in Figure 1 and Figure 2 , we display a more varied set of qualitative examples in Figure A.10 . We observe that FRIs are usually located within object boundaries but can also be found in the background . This is because DNNs are able to recognize regions that only contain background ( Zhu et al. , 2016 ) , as they have been shown to exploit dataset biases . In Figure A.9 we display the DNN ’ s output confidence in the true class for individual regions in map form . We see that sharp drops in confidence are frequent within an image , providing a basis for minimal images . Finally , in Figure A.11 we show qualitative examples of the activation maps at different layers of the DNN of the correctly classified crop and its shifted version . These examples show what we have observed in all cases : the activation maps are imperceptibly similar at the first layers but are clearly different at the last layers . Finally , we show qualitative examples to illustrate that FRIs are a concern for computer vision systems based on similar DNNs architectures . We conducted a test on detection algorithms , to validate that FRIs dramatically affect both the location and the label of the detected objects . In Figure A.12 we show FRIs for the widely used object detector called “ YOLO ” ( Redmon & Farhadi , 2017 ) .

Summary: Ullman et al. showed that slight changes in location or size of visible regions in minimal recognizable images can significantly impair human ability to recognize objects. This paper is a  follow-up of Ullman et al. paper, with focus on sensitivity of DNNs to certain regions in images. In other words, slight change of such regions’ size or location in the image can significantly affect DNN ability in recognizing them, even-though these changes are not noticeable for humans.