Needles in Haystacks: On Classifying Tiny Objects in Large Images

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) are the current state-of-the-art approach for image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2015 ; Huang et al. , 2017 ) . The goal of image classification is to assign an image-level label to an image . Typically , it is assumed that an object ( or concept ) that correlates with the label is clearly visible and occupies a significant portion of the image ( Lecun et al. , 1998 ; Krizhevsky , 2009 ; Deng et al. , 2009 ) . Yet , in a variety of real-life applications , such as medical image or hyperspectral image analysis , only a small portion of the input correlates with the label , resulting in low signal-to-noise ratio . We define this input image signal-to-noise ratio as Object to Image ( O2I ) ratio . The O2I ratio range for three real-life datasets is depicted in Figure 1 . As can be seen , there exists a distribution shift between standard classification benchmarks and domain specific datasets . For instance , in the ImageNet dataset ( Deng et al. , 2009 ) objects fill at least 1 % of the entire image , while in histopathology slices ( Ehteshami Bejnordi et al. , 2017 ) cancer cells can occupy as little as 10−6 % of the whole image . Recent works have studied CNNs under different noise scenarios , either by performing random input-to-label experiments ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) or by directly working with noisy annotations ( Mahajan et al. , 2018 ; Jiang et al. , 2017 ; Han et al. , 2018 ) . While , it has been shown that large amounts of label-corruption noise hinders the CNNs generalization ( Zhang et al. , 2017 ; Arpit et al. , 2017 ) , it has been further demonstrated that CNNs can mitigate this label-corruption noise by increasing the size of training data ( Mahajan et al. , 2018 ) , tuning the optimizer hyperparameters ( Jastrzębski et al. , 2017 ) or weighting input training samples ( Jiang et al. , 2017 ; Han et al. , 2018 ) . However , all these works focus on input-to-label corruption and do not consider the case of noiseless input-to-label assignments with low and very low O2I ratios . In this paper , we build a novel testbed allowing us to specifically study the performance of CNNs when applied to tiny object classification and to investigate the interplay between input signal-to-noise ratio and model generalization . We create two synthetic datasets inspired by the children ’ s puzzle book Where ’ s Wally ? ( Handford , 1987 ) . The first dataset is derived from MNIST digits and allows us ImageNet MiniMIAS CAMELYON17 . O2I = . Aimage , where Aobject and Aimage denote the area of the object and the image , respectively . Together with O2I range , we display examples of images jointly with the object area Aobject ( in red ) . to produce a relatively large number of datapoints with explicit control of the O2I ratio . The second dataset is extracted from histopathology imaging ( Ehteshami Bejnordi et al. , 2017 ) where we crop images around lesions and obtain small number of datapoints with an approximate control of the O2I ratio . To the best of our knowledge these datasets are the first ones designed to explicitly stress-test the behaviour of the CNNs in the low input image signal-to-noise ratio . We develop a classification framework , based on CNNs , and analyze the effects of different factors affecting the model optimization and generalization . Throughout an empirical evaluation , we make the following observations : – Models can be trained in low O2I regime without using any pixel-level annotations and generalize if we leverage enough training data . However , the amount of training data required for the model to generalize scales rapidly with the inverse of the O2I ratio . When considering datasets with fixed size , we observe an O2I ratio limit in which all tested scenarios fail to exceed random performance . – We empirically observe that higher capacity models show better generalization . We hypothesize that high capacity models learn the input noise structure and , as result , achieve satisfactory generalization . – We confirm the importance of model inductive bias — in particular , the model ’ s receptive field size . Our results suggest that different pooling operations exhibit similar performance , for larger O2I ratios ; however , for very small O2I ratios , the type of pooling operation affects the optimization ease , with max-pooling leading to fastest convergence . The code of our testbed will be publicly available at : https : //anonymous.url allowing to reproduce all data and results ; we hope this work can serve as a valuable resource facilitating further research into the understudied problem of low signal-to-noise classification scenarios . 2 TESTBED FOR LOW SIGNAL-TO-NOISE CLASSIFICATION SCENARIOS . 2.1 DATASETS : IS THERE A WALLY IN AN IMAGE ? . To study the optimization and generalization properties of CNNs , we build two datasets : one derived from the MNIST ( Lecun et al. , 1998 ) dataset and another one produced by cropping large resolution images from the CAMELYON dataset ( Ehteshami Bejnordi et al. , 2017 ) . Each dataset allows to evaluate the behaviour of a CNN-based binary classifier when altering different data-related factors of variation such as dataset size , object size , image resolution and class balance . In this subsection , we describe the data generation process . Digits : needle MNIST ( nMNIST ) . Inspired by the cluttered MNIST dataset ( Ba et al. , 2015 ) , we introduce a scaled up , large resolution cluttered MNIST dataset , suitable for binary image classification . In this dataset , images are obtained by randomly placing a varying number of MNIST digits on a large resolution image canvas . We keep the original 28× 28 pixels digit resolution and control the O2I ratio by increasing the resolution of the canvas 1 . As result , we obtain the following O2I ratios { 19.1 , 4.8 , 1.2 , 0.3 , and 0.075 } % that correspond to the following canvas resolutions 64×64 , 128×128 , 256×256 , 512×512 , and 1024×1024 pixels , respectively . As object of interest , we select digit 3 . All positive images contain exactly one instance of the digit 3 randomly placed within the image canvas , while negative instances do not contain any instance . We also include distractors ( clutter digits ) : any MNIST digit image sampled with replacement from a set of labels { 0 , 1 , 2 , 4 , 5 , 6 , 7 , 8 , 9 } . We maintain approximately constant clutter density over different O2I ratios . Thus , the following O2I ratios { 19.1 , 4.8 , 1.2 , 0.3 , and 0.075 } % correspond to 2 , 5 , 25 , 100 , and 400 clutter objects , respectively . For each value of O2I ratio , we obtain 11276 , 1972 , 4040 of training , validation and test images2 . Fig . 2 depicts example images for different O2I ratios . We refer the interested reader to the supplementary material for details on image generation process as well as additional dataset visualizations . Histopathology : needle CAMELYON ( nCAMELYON ) . The CAMELYON ( Ehteshami Bejnordi et al. , 2017 ) dataset contains gigapixel hystopathology images with pixel-level lesion annotations from 5 different acquisition sites . We use the pixel-wise annotations to extract crops with controlled O2I ratios . Namely , we generate datasets for O2I ratios in the range of ( 100−50 ) % , ( 50−10 ) % , ( 10−1 ) % , and ( 1 − 0.1 ) % , and we crop different image resolutions with the size of 128 × 128 , 256 × 256 , and 512 × 512 pixels . This results in training sets of about 20 − 235 unique lesions per dataset configuration ( see supplementary for a detailed list of dataset sizes ) . More precisely , positive examples are created by taking 50 random crops from every contiguous lesion annotation and rejecting the crop if the O2I ratio does not fall within the desired range . Negative images are taken by randomly cropping healthy images and filtering image crops that mostly contain background . We ensure the class balance by sampling an equal amount of positive and negative crops . Once the crops were extracted , no pixel-wise information is used during training . Figure 2 shows examples of extracted images used in the nCAMELYON dataset experiments . We refer to the supplementary for more detail about the data extraction process , the resulting dataset sizes and more visualizations . 2.2 MODELS . Our classification pipelines follow BagNets ( Brendel & Bethge , 2019 ) backbone , which allows us to explicitly control for the network receptive field size . Figure 3 shows a schematic of our approach . As can be seen , the pipelines are built of three components : ( 1 ) topological embedding extractor in 1Alternatively , we could fix canvas image resolution and downscale MNIST digits ; however , downscaling might reduce the object quality . 2We obtain those numbers by using the original MNIST data , we use every digit 3 only once to generate positive images and we balance the dataset with negative images . See supplementary material for class imbalanced data scenarios . which we can control for embedding receptive field , ( 2 ) global pooling operation that converts the topological embedding into a global embedding , and ( 3 ) a binary classifier that receives the global embedding and outputs binary classification probabilities . By varying the embedding extractor and the pooling operation , we test a set of 48 different architectures . Topological embedding extractor . The extractor takes as input an image I of size [ wimg × himg × cimg ] and outputs a topological embedding Et of shape [ wenc × henc × cenc ] , where w. , h. , and c. represent width , height and number of channels . Due to the relatively large image sizes , we train the pipeline with small batch sizes and , thus , we replace BagNet-used BatchNorm operation ( Ioffe & Szegedy , 2015 ) with Instance Normalization ( Ulyanov et al. , 2016 ) . In our experiments , we test 12 different extractor architectures obtained by adapting embedding extractor receptive field and capacity . For model details , please refer to section B.2 in the supplementary material . Global pooling operation . Global pooling operation takes as an input topological embedding Et of shape [ wenc × henc × cenc ] and outputs global image embedding EI of shape [ 1× 1× cenc ] . In the paper , we experiment with four different pooling operations , namely : max , logsumexp , average , and soft attention . In our experiments , we follow the soft attention formulation of ( Ilse et al. , 2018 ) . The details about global pooling operations can be found in the supplementary material . 3 EXPERIMENTAL RESULTS . In this section , we experimentally test how CNNs ’ optimization and generalization scale with low and very low O2I ratios . First , we provide details about our experimental setup and then we design experiments to provide empirical evidence to the following questions : ( 1 ) Image-level annotations : Is it possible to train classification systems that generalize well in low and very low O2I scenarios ? ( 2 ) O2I limit vs. dataset size : Is there an O2I ratio limit below which the CNNs will experience generalization difficulties ? Does this O2I limit depend on the dataset size ? ( 3 ) O2I limit vs. model capacity : Do higher capacity models generalize better ? ( 4 ) Inductive bias - receptive field : Is adjusting receptive field size to match ( or exceed ) the expected object size beneficial ? ( 5 ) Global pooling operations : Does the choice of global pooling operation affect model generalization ? Finally , we inquire about the optimization ease of the models trained on data with very low O2I ratios . In all our experiments , we used RMSProp ( Tieleman & Hinton , 2012 ) with a learning rate of η = 5 · 10−5 and decayed the learning rate multiplying it by 0.1 at 80 , 120 and 160 epochs 3 . All models were trained with cross entropy loss for a maximum of 200 epochs . We used an effective batch size of 32 . If the batch did not fit into memory we used smaller batches with gradient accumulation . To ensure robustness of our conclusions , we run every experiment with six different random seeds and report the mean and standard deviation . Throughout the training we monitored validation accuracy , and reported test set results for the model that achieved best validation set performance . 3Before committing to a single optimization scheme , we evaluated a variety of optimizers ( Adam , RMSprop and SGD with momentum ) , learning rates ( η ∈ { 1 , 2 , 3 , 5 , 7 , 10 } · 10−5 ) , and 3 learning rate schedules .

Summary: The authors present an empirical study to evaluate the performance of CNN-based object classifiers for situations in which the object of interest is very small relative to the size of the image. Two artificial datasets, based on MNIST and histopathological images are introduced to conduct the experiments. Through empirical evaluation the authors conclude that the size of the dataset required for generalization increases rapidly with the inverse of the O2I ratio, that higher capacity models generalize better, and that accounting for the model's receptive field is key.