Needles in Haystacks: On Classifying Tiny Objects in Large Images
Authors: Nick Pawlowski
Suvrat Bhooshan
Nicolas Ballas
Francesco Ciompi
Ben Glocker
Michal Drozdzal
Venue:  (2020)
Keywords: computer vision
CNNs
small objects
low signal-to-noise image classification

Abstract: In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noise ratios. Our observations indicate that: (1) There exists a limit to signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size - more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-noise ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noise values, all tested global pooling operations exhibit similar performance.

Under review as a conference paper at ICLR 2020
NEEDLES IN HAYSTACKS: ON CLASSIFYING TINY OBJECTS IN LARGE IMAGES
A Anonymous authors Paper under double-bling review
ABSTRACT
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were de- veloped using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one de- rived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN architectures with a broad spectrum of signal-to-noiss ratios. Our observations indicate that: (1) There exists limit signal-to-noise below which CNNs fail to generalize and that this limit is affected by dataset size more data leading to better performances; however, the amount of training data required for the model to generalize scales rapidly with the inverse of the object-to-image ratio (2) in general, higher capacity models exhibit better generalization; (3) when knowing the approximate object sizes, adapting receptive field is beneficial; and (4) for very small signal-to-nowl ratio the choice of global pooling operation affects optimization, whereas for relatively large signal-to-noie values, all tested global pooling operations exhibit similar performance.
INTRODUCTION
Convolutional Neural Networks (CNNs) are the current state-of-the-arr art approach for image classifica- tion (Krizhevsky et al. 2012 Simonyan & Zisserman 2014 He et al. 2015 Huang et al. 2017). The goal of image classification to assign an image-level label to an image. Typically, is assumed that an object (or concept) that correlates with the label is clearly visible and occupies significant portion of the image (Lecun et al. 1998 Krizhevsky, 2009 Deng et al. 2009). Yet, in a variety of real-life applications, such as medical image or hyperspectral image analysis, only small portion of the input correlates with the label, resulting in low signal-to-nois noise ratio. We define this input image signal-to-nois ratio as Object to Image (O2I) ratio. The 021 ratio range for three real-life datasets is depicted in Figure T As can be seen, there exists a distribution shift between standard classification benchmarks and domain specific datasets. For instance, in the ImageNet dataset (Deng al. 2009) objects fill at least 1% of the entire image, while in histopathology slices (Ehteshami Bejnordi et al. 2017) cancer cells can occupy as little 10 10-6% of the whole image.
Recent works have studied CNNs under different noise scenarios, either by performing random input-to-label experiments (Zhang al. 2017, Arpit et al., 2017) by directly working with noisy annotations (Mahajan et al., 2018 Jiang et al. 2017; Han et al., 2018). While, it has been shown that large amounts of label-corruption noise hinders the CNNs generalization (Zhang et al. 2017 Arpit et al. 2017), it has been further demonstrated that CNNs can mitigate this label-corruption noise by increasing the size of training data (Mahajan et al., 2018), tuning the optimizer hyperpa- rameters (Jastrzębski et al.) 2017) or weighting imput training samples (Jiang et al. 2017 Han et al.) 2018). However, all these works focus on input-to-label corruption and do not consider the case of noiseless input-to-label assignments with low and very low 02I ratios.
In this paper, we build a novel testbed allowing us to specifically study the performance of CNNs when applied to tiny object classification and to investigate the interplay between input signal-to-non toiise ratio and model generalization. We create two synthetic datasets inspired by the children's 's puzzle book Where's 's Wally? (Handford 1987). The first dataset is derived from MNIST digits and allows us

---

Under review as a conference paper at ICLR 2020
CAMELYON17
MiniMIAS
2
10P
10
10a
ImageNet
ImageNet
MiniMIAS
CAMELYON17
Figure 1: Range of Object Image (021) ratios [%] for two medical imaging datasets (CAME- LYON17 (Ehteshami Bejnordi et al. 2017) and MiniMIAS (Suckling. 1994)) as well as one standard computer vision classification dataset (ImageNet (Deng al.] 2009h). The ratio is defined as
O21 Aobject Asmage where Aohject and Aimage denote the area the object and the image, respectively,
Together with O2I range, we display examples of images jointly with the object area Abbject (in red).
to produce relatively large number of datapoints with explicit control of the 02I ratio. The second dataset extracted from histopathology imaging (Ehteshami Bejnordi et al. 2017) where we crop images around lesions and obtain small number of datapoints with an approximate control of the 021 ratio. To the best of our knowledge these datasets are the first ones designed explicitly stress-test the behaviour of the CNNs in the low input image signal-to-noise ratio.
We develop a classification framework, based on CNNs, and analyze the effects different factors affecting the model optimization and generalization. Throughout empirical evaluation, we make the following observations:
Models can be trained in low 021 regime without using any pixel-level annotations and generalize if we leverage enough training data. However, the amount of training data required for the model to generalize scales rapidly with the inverse of the 021 ratio. When considering datasets with fixed size, we observe an 021 ratio limit in which all tested scenarios fail exceed random performance.
We empirically observe that higher capacity models show better generalization. We hy- pothesize that high capacity models learn the input noise structure and, result, achieve satisfactory generalization.
We confirm the importance of model inductive bias in particular, the model's 's receptive field size. Our results suggest that different pooling operations exhibit similar performance, for larger 02I ratios; however, for very small 021 ratios, the type of pooling operation affects the optimization ease, with max-pooling leading fastest convergence.
The code our testbed will be publicly available at: /attps://nooymous . .rr allowing to reproduce all data and results; we hope this work can serve as valuable resource facilitating further research into the understudied problem of low signal-to-ne to-ise classification scenarios.
2 TESTBED FOR LOW SIGNAL-TO-NOISE CLASSIFICATION SCENARIOS
2.1 DATASETS: IS THERE A WALLY IN AN IMAGE?
To study the optimization and generalization properties of CNNs, we build two datasets: one derived from the MNIST (Lecun et al. 1998) dataset and another produced by cropping large resolution images from the CAMELYON dataset (Ehteshami Bejnordi et al., (017). Each dataset allows to evaluate the behaviour of CNN- binary classifier when altering different data-related factors
2

---

Under review as a conference paper at ICLR 2020
t
p
B
(a) 02T ratio 0.3%
(b) 02T ratio 0.075%
(c) 02I [1 10/% (d) 021 ratio [0.1 11%
Figure 2: Example images from our nMNIST (a. b) and nCAMELYON (c. d) datasets with different 021 ratios. The object of interest marked by red (nMNIST) or green outlines (nCAMELYON).
of variation such as dataset size, object size, image resolution and class balance. In this subsection, we describe the data generation process.
Digits: needle MNIST (nMNIST). Inspired by the cluttered MNIST dataset (Ba et al. (015). we introduce a scaled up, large resolution cluttered MNIST dataset, suitable for binary image classification. In this dataset, images are obtained by randomly placing a varying number MNIST digits on large resolution image canvas. We keep the original 28 x 28 pixels digit resolution and control the O2I ratio by increasing the resolution of the canvas As result, we obtain the following 02I ratios {19.1. 4.8, 1.2, 0.3, and 0.075}% that correspond to the following canvas resolutions 64 x 64, 128 x 128, 256 256, 512 x 512, and 1024 x 1024 pixels, respectively. As object of interest, select digit 3. All positive images contain exactly one instance of the digit 3 randomly placed within the image canvas, while negative instances do not contain any instance. We also include distractors (clutter digits): any MNIST digit image sampled with replacement from a set of labels {0, 1, 4, 5, 6, 7, 8, 9}. We maintain approximately constant clutter density over different O21 ratios. Thus, the following 02I ratios (19.1, 1.2, 0.3, and 0.075]% % correspond to 2, 5, 25, 100, and 400 clutter objects, respectively, For each value of 02I ratio, we obtain 11276, 1972, 4040 of training, validation and test i Fig. 2 depicts example images for different O2I ratios. We refer the interested reader to the supplementary material for details on image generation process as well as additional dataset visualizations.
Histopathology: needle CAMELYON (RCAMELYON), The CAMELYON (Ehteshami Bejnordi cct al. 2017) dataset contains gigapixel hystopathology images with pixel-level lesion annotations from 5 different acquisition sites. We use the pixel-wise annotations to extract crops with controlled 021 ratios. Namely, we generate datasets for O2I ratios in the range of (100-50) 50)% %. (50-10)6 (10-1)% and (1 0.1)%, and crop different image resolutions with the size of 128 x 128, 256 256, and 512 x 512 pixels. This results in training sets of about 20 - 235 unique lesions per dataset configuration (see supplementary for detailed list of dataset sizes). More precisely, positive examples are created by taking 50 random crops from every contiguous lesion annotation and rejecting the crop if the O2I ratio does not fall within the desired range. Negative images are taken by randomly cropping healthy images and filtering image crops that mostly contain background. We ensure the class balance by sampling an equal amount of positive and negative crops. Once the crops were extracted, no pixel-wise information is used during training. Figure Higure]ssowws []shows examples of extracted images used in the nCAMELYON dataset experiments. We refer to the supplementary for more detail about the data extraction process, the resulting dataset sizes and more visualizations.
2.2 MODELS
Our classification pipelines follow BagNets (Brendel & Bethge 2019) backbone, which allows to explicitly control for the network receptive field size. Figure shows a schematic of our approach. As can be seen, the pipelines are built of three components: (I) topological embedding extractor in
Alternatively, we could fix canvas image resolution and downscale MNIST digits; however, downscaling might reduce the object quality.
We obtain those numbers by using the original MNIST data, we use every digit 3 only once generate positive images and we balance the dataset with negative images. See supplementary material for class imbalanced data scenarios.
3

---

Under review as a conference paper at ICLR 2020
11 www
Giobel
0.9
Classaffar
9
0.2
0.3
007
4.19
ICAMELYON
3-30
021 (%k
Figure 3: Pipeline. Our pipeline is built of three com- ponents: (1) a CNN extracting topological embedding, (2) global pooling operation and (3) a binary classifier. See text for details.
Figure 4: Image-level annotations. Test set accuracy vs. O2I ratio for best models, see text for details.
which we can control for embedding receptive field, (2) global pooling operation that converts the topological embedding into global embedding, and (3) binary classifier that receives the global embedding and outputs binary classification probabilities. By varying the embedding extractor and of 48 different architectures.
the pooling operation,
Topological embedding extractor. The extractor takes as input an image I of size [Wing x himg x Csmgl and outputs topological embedding E' of shape [Wene hene Cenel, where w,, h, and c, represent width, height and number of channels. Due to the relatively large image sizes, we train the pipeline with small batch sizes and, thus, we replace BagNet-used BatchNorm operation (loffe & Szegedy, 2015) with Instance Normalization (Ulyanov et al. (016). In our experiments, we test 12 different extractor architectures obtained by adapting embedding extractor receptive field and capacity, For model details, please refer to section B.2 in the supplementary material.
Global pooling operation. Global pooling operation takes as an input topological embedding E of shape [Wene X hene X Cene] and outputs global image embedding E' of shape [1 x 1 x Cenej. In the paper, we experiment with four different pooling operations, namely: max, logsumexp, average, and soft attention. In experiments, follow the soft attention formulation of (Ilse al. 2018). The details about global pooling operations can be found in the supplementary material.
EXPERIMENTAL RESULTS
In this section, experimentally test how CNNs' optimization and generalization scale with low and very low 021 ratios. First, we provide details about our experimental setup and then we design experiments to provide empirical evidence the following questions: (1) Image-level annotations: Is possible to train classification systems that generalize well in low and very low O2I scenarios? (2) 02I limit vs. dataset size: Is there an 02I ratio limit below which the CNNs will experience generalization difficulties? Does this O2I limit depend on the dataset size? (3) 021 limit vs. model capacity: Do higher capacity models generalize better? (4) Inductive bias receptive field: Is adjusting receptive field size to match (or exceed) the expected object size beneficial? (5) Global pooling operations: Does the choice of global pooling operation affect model generalization? Finally, inquire about the optimization ease of the models trained on data with very low 021 ratios.
In all our experiments, we used RMSProp (Tieleman & Hinton 2012) with learning rate of 5- 10-3 and decayed the learning rate multiplying it by 0.1 at SO, 120 and 160 epochs All models were trained with cross entropy loss for a maximum of 200 epochs. We used an effective batch size of 32. If the batch did not fit into memory To ensure robustness of our conclusions, we run every experiment with six different random seeds and report the mean and standard deviation. Throughout the training monitored validation accuracy, and reported test set results for the model that achieved best validation set performance.
used smaller batches with gradient accumulation.
Before committing to single optimization scheme, evaluated a variety of optimizers (Adam, RMSprop and SGD with momentum), learning rates (17 E {1, (1,2,3, 2, 3, 10) 10) and learning rate schedules.
4

---

Under review as a conference paper at ICLR 2020
Training set size
#Parameters
#Parameters
2819
11276
10
203
1.4
203
10
5638
22552
5.8
92"
5.8+10
92210'
0.9
Accuracy
0.9
Test Accuracy
0.8
0.8
0.8
0.7
0.7
Test
0.7
0.6
0.6
0.6
0.5
0.5
0.30
1.20
4.79
19:14
0.30
1.20
4.79
19.14
0.1-1
1- 110
10-
50
50
100
ratio [16]
021 ratio [%6]
ratio [96]
(a) AMNIST
(b) AMNIST
(c) aCAMELYON
Figure 5: Testing the 021 limit. Subfigure (a) depicts the test set performance as a function of training dataset size for the nMNIST dataset, while subfigures (b) and (c) show the test set performance as function of model capacity for the nMNIST dataset and the RCAMELYON dataset, respectively.
Pooliing Mothod
Pooling Mothod Logsumoxp
20352
100
0.55
0.53
0.53
0.55
1.82
0:81
CP ratt [4]
1:20
0.00
0.75
Valdatice Accursyy
4.99
0.09
0:67
0.76
0.73
7.
0.04
00.0%
503
0.63
0.72
0.11
0.00
0.67
2210
1850
sel
rato
(a) AMNIST
(b) AMNIST
Figure 6: Testing the O2I limit. (a) mean validation set accuracy heatmap for max pooling operation, and (b) minimum required training set size to achieve the noted validation accuracy. We test training set sizes E {1400, 2819, 5638, 7500, 11276, 22552] and report the minimum amount of training examples that achieve specific validation performance pooling over different network capacities.
3.1 RESULTS
In this subsection, we present and discuss the main results of our analysis. Unless stated otherwise, the capacity of the ResNet-50 network is about 2.3 10 parameters. Additional results and analysis are presented in the supplementary material.
Image-level annotations: For this experiment, we vary the 02I ratio on mMNIST and aCAMELYON to test its influence on the generalization of the network. Figure depicts the results for the best configuration according to the validation performance: we max-pooling and receptive field sizes of 33 x 33 and 9 x 9 pixels for the nMNIST and nCAMELYON datasets, respectively. For the nMNIST dataset, the plot represents the mean over 6 random seeds together with the standard deviation; while for the ICAMELYON dataset we report an average over both the 6 seeds and the crop sizes. We find that the tested CNNs achieve reasonable test set accuracies for the 02I ratios larger than 0.3% for the nMNIST datset and the 02I ratios above 1% for the histopathology dataset. For both datasets, smaller 02I ratios lead poor or even random test set accuracies.
021 limit vs. dataset size: We test the influence of the training set size on model generalization for the nMNIST data, to understand the CNNs' generalization problems for very small 021 ratios. We tested six different dataset sizes (1400, 2819, 5638, 7500, 11276, 22552) [ Fig. depicts the results for max pooling and receptive field of 33 x 33 pixels. We observe that larger datasets yield better generalization and this increment is more pronounced for small 02I ratios. For further insights, we plot a heatmap representing the mean validation set results Fl for all considered 02Is and training set sizes (Fig, 6a) well the minimum number of training examples to achieve validation accuracy
"We allow to reuse each digit 3 for larger training sets and select subset for smaller training sets. "oore precisely, we plot the mean of all pipeline configurations that surpassed 70% training accuracy.
5

---

Under review as a conference paper at ICLR 2020
Pooling Method
Pooling
Mean
Logsumexp attontion
Mean
Logsumexp
Soft attention
Receptive Field
177
1.0
150
0.9
Test Accuracy
0.8
0.0
Epochs to
100
0.7
Test
0.7
0.6
0.6
0.5
R
0.07
0.30
1.20
4.79
19.14
0.07
0.30
1.20
4.79
19.14
0.07
0.30
021 ratio [%6
1.20
4.79
19.14
ratio [%6]
[96]
(a) AMNIST
(a) aMNIST
(a) AMNIST
Pooting
Pooling
Mean
Logoumexp Soft attention
Mean
Logsumexp attention
Receptive fwild
1.0
0.9
Test Accuracy
Test Accuracy
0.8
0.7
0.6
0.6
0.5
011-1
1 10
10
50
50
100
011-1
1-10
10
50
100
021 ratio 1%6]
(96
[%)
(b) aCAMELYON
(b) ICCAMELOON
(b) AMNIST
Figure 7: Inductive bias: for
Figure 8: Global pooling op- erations: for (a) the nMNIST dataset and (b) the nCAME- LYON datset. We report only runs that fit the training data. Otherwise we report random accuracy and depict it with texture on the bars.
Figure 9: nMNIST optimiza-
(a) the AMNIST dataset and (b) the aCAMELYON dataset. We report only runs that fit the training data. Otherwise we report random accuracy and depict it with texture on the bars.
tion: (a) number of training epochs needed fit the 11k training data and (b) the num- ber of successful runs. The textured bars indicate that the model did not fit the training data for all random seeds.
of 70% and 85% (Fig. 6b). We observe that in order to achieve good classification generalization the required training set size rapidly increases with the decrease of the 021 ratio.
O2I limit vs. capacity: In this experiment, we train networks with different capacities - by uniformly scaling the initial number of filters in convolutional kernels by [2 2. 1, and 2 We show the CNNs test set performances as function of the 021 ratio and the network capacity in Figures 5b and Se the nMNIST (with 11k training points) and iCAMELYON data, respectively. On nMNIST, we observe a clear trend, where the model test set performance increases with capacity and this boost larger for smaller O2Is. We hypothesize, that this generalization improvement is due the model ability to learn-to-ige carn-to-ignore the input data noise; with smaller 02I there is more noise to ignore and, thus, higher network capacity required to solve the task. However, for the nCAMELYON dataset, this trend is not so pronounced and we attribute this to the limited dataset size (more precisely to the small number of unique lesions). These results suggest that collecting very large histopathology dataset might enable training of CNN models using only image level annotations.
Inductive bias receptive field: We report the test accuracy as a function of the 02I ratio and the receptive field size for MMINIST in Figure and for RCAMELYON in Figure 7b Both plots depict
We chose the maximum scaling factor so that the largest resolution images still fit in available GPU memory. For images with 02I ratio of 0.07, the available GPU memory prevents testing networks with higher capacity.

---

Under review as a conference paper at ICLR 2020
results for the global max pooling operation. For nMNIST, we observe that receptive field that is bigger than the area occupied by one single digit leads to best performances; for example, receptive fields of 33 x 33 and 177 x 177 pixels clearly outperform the smallest tested receptive field of 9 x 9 pixels. However, for the nCAMELYON dataset we observe that the smallest receptive field actually performs best. This suggests that most of the class-relevant information is contained in the texture and that higher receptive fields pick up more spurious correlations, because the capacity of the networks constant.
Global pooling operations: In this experiment, we compare the performance of four different pooling approaches. We present the relation between test accuracy and pooling function for different 02I ratios with a receptive field of 33 x 33 pixels for nMNIST in Figure Ba and 9 x 9 pixels for nCAMELYON in Figure [b) On the one hand, for the nMNIST dataset, we observe that for the relatively large 02T ratios, all pooling operations reach similar performance; however, for smaller O2Is we see that max-pooling the best choice. We hypothesize that the global max pooling operation is best suited to remove nMNIST-type of structured input noise. On the other hand, when using the histopathology dataset, for the smallest O2I mean and soft attention poolings reach best performances; however, these outcomes might be affected by the relatively small nCAMELYON dataset used for training.
Optimization: In our large scale nMNIST experiments (when using x 11k datapoints), observed that some config configurations have problems fitting the training data M In some runs, after significant efforts put into CNNs hyperparamenter selection, the training accuracy was close random. To investigate this issue further, we followed the setup of randomized experiments from (Zhang et al. 2017 Arpit al. 2017) and we substituted the nMNIST datapoints with samples from an isotropic Gaussian distribution. On the one hand, we observed that all the tested setups of our pipeline were able to memorize the Gaussian samples, while, on the other hand, most setups were failing to memorize the same-size, nMNIST datataset for small and very small O2I ratios. We argue that the nMNIST structured noise and its compositionality may be a "harder" type of noise for the CNNs than Gaussian isotropic noise. To provide further experimental evidence, we depict average time-to-fit fit the training data (in epochs) in Fig. 9a as well as number of successful optimizations in Fig. 9b for different 02I ratios and pooling methods We observe that the optimization gets progressively harder with decreasing O2I ratio (with max pooling being the most robust). Moreover, we note that the results are consistent across different random seeds, where all runs either succeed or fail to converge.
4 RELATED WORK
4.1 TINY OBJECT CLASSIFICATION
Reasoning about tiny objects is of high interest in many computer vision areas, such as medical
imaging (Ehteshami Bejnordi al. 2017 Aresta et al. 2018 Setio et al. 2017 Suckling 1994.
Sudre et al. 2018) and remote sensing (Xia et al.) 2018 Pang et al. 2019). To overcome the low signal-to- noisee ratio, most approaches rely on manual dataset "curation" and collect additional lexel-level annotations such as landmark positions (Borovec et al., 2018). bounding boxes (Wei jet al. 2019 Resta et al., 201 or segmentation maps (Ehteshami Bejnordi et al., 2017). This additional annotation allows to transform the original needle-in-a-haystace problem into less noisy but imbalanced classification problem (Wei et al.) 2019: Lee & Paeng 2018 Bándi et al. 2019)).
However, collecting pixel level annotations has significant cost and might require expert knowledge, and as such, is a bottleneck in the data collection process.
Other approaches leverage the fact that resk-relevant information is often not uniformly distributed across input data, e.g. by using attention mechanisms to process very sigh-dimessional inputs (Mnih) et al. 2014 Ba et al. 2015 Almahairi al., 2016 Katharopoulos & Fleuret 2019). However, those approaches are mainly motivated from a computational perspective trying to reduce the computational footprint at inference time.
7 We did not observe optimization problems for small dataset sizes of the nMNIST nor for nCAMELYON. We define an optimization to he successful if it the training accuracy surpassed 99%.
7

---

Under review as a conference paper at ICLR 2020
Some recent research has also studied attention based approaches both in the context of multi-instanee learning (lsse et al. 2018)) and histopathology image classification (Tomita et al. 2018). However, neither of the works report the exact O2I ratio used in the experiments.
4.2 GENERALIZATION OF CNNS
In this subsection, we briefly highlight the dimensions of optimization and generalization of CNN that are handy in low O2I classification scenarios.
Model capacity. For fixed training accuracy, wver-parametrizee CNNs tend to generalize better (No- vak et al. 2018). In addition, when properly regularized and given a fixed size dataset, higher capacity models tend provide better performance (He et al. 2016; Huang al. (017). However, finding proper regularization is not trivial (Goodfellow et al. 2016).
Dataset size. CNN performance improves logarithmically with dataset size (Sun et al. 2017) (). Moreover, in order to fully exploit the data benefit, the model capacity should scale jointly with the dataset size (Mahajan et al. 2018 Sun et al. 2017).
Model inductive biases. Inductive biases limit the space of possible solutions that a neural network can learn (Goodfellow et al. 2016). Incorporating these biases is an effective way to include data (or domain) specific knowledge in the model. Perhaps the most successful inductive bias is the use of convolutions in CNNs (LeCun & Bengiol 1998). Different CNN architectures (e. g. altering network connectivity) also lead to improved model performance (He et al.) [016; Huang et al. (017)) Additionally, it has been shown on the ImageNet dataset that CNN accuracy scales logarithmically with the size of the receptive field (Brendel & Bethge 2019).
DISCUSSION AND CONCLUSIONS
Although low input image signal-to-noise scenarios have been extensively studied in signal processing field (e.g. in tasks such as image reconstruction), less attention has been devoted to low signal-to- noise classification scenarios. Thus, in this paper we identified an unexplored machine learning problem, namely image classification in low and very low signal-to-noiss ratios. In order to study such scenarios, we built two datasets that allowed us to perform controlled experiments by manipulating the input image signal-to- signal-to-noise ratio and highlighted that CNNs struggle show good generalization for low and very low signal-t-nooie ratios even for relatively elementary MNIST-based dataset. Finally, we ran series of controlled experiments thhat that explore both a variety of CNNs' architectural choices and the importance of training data scale for the low and very low signal-to signal-to-noise classification. One of our main observation was that properly designed CNNs can be trained in low 021 regime withour using any pixel-level annotations and generalize if we leverage enough training data; however, the amount of training data required for the model to generalize scales rupidly with the inverse of the 021 ratio. Thus, with our paper (and the code release) we invite the community to work on data-efficient solutions to low and very low signal-to- -onal-to-noise classification.
Our experimental study exhibits limitations: for explicit control of the input signal-to-noise ratios, we were forced to use the synthetically built mMNIST dataset for most of our analysis. As a real life dataset, we used crops from the histopathology CAMELYON dataset; however, due to relatively a small number of unique lesions we were unable to scale the histopathology experiments to the extent the nMNIST experiments, and, result, some conclusions might be affected by the limited dataset size. Other large scale computer vision datasets like MS COCO (Lin et al.) 2014)) exhibit correlations of the object of interest with the image background. For MS COCO, the smallest 02I ratios are for the object category "sports ball" which on average occupies between 0.3% and 0.4% of an image and its presence tends to be correlated with the image background (e. g. presence of sports fields and players). However, future research could examine a setup in which negative images contain objects of the categories "person" " and "baseball bat" and positive images also contain "sports ball". Second, all the tested models improve the generalization with larger dataset sizes; however, scaling datasets such as CAMELYON to tens of thousands of samples might be prohibitively expensive. Instead, further research should be devoted to developing computationally-sctn scalable, data-cfficient inductive biases that can handle very low
First, due to the lack of large scale datasets that allow
We ran more than 750 experiments each with different seeds.
8

---

Under review as a conference paper at ICLR 2020
signal-to-noise ratios with limited dataset sizes. Future work, could explore the knowledge of the low O2I ratio and therefore sparse signal as an inductive bias. Finally, we studied low signal-to- toiee scenarios only for binary classification scenarios further investigation should be devoted to multi- class problems. We hope that this study will stimulate the research in image classification for low signal-to-noise input scenarios.
REFERENCES
Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. Dynamic capacity networks. In International Conference on Machine Learning, pp. 2549-25425 -258,, 2016.
Guilherme Aresta, Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan K. P., Alex Varghese, Bahram Marami, Marcel Prastawa, Monica Chan, Michael J. Donovan, Gerardo Fernandez, Jack Zeinch, Matthias Kohl, Christoph Walz, Florian Ludwig, Stefan Braunewell, Maximilian Baust, Quoe Dang Vu, Minh Nguyen Nhat To, Fal Kim, Jin Tae Kwak, Sameh Galal, Veronica Sanchez-Freire, Nadia Brancati, Maria Frucci, Daniel Riccio, Yaqi Wang, Lingling Sun, Kaiqiang Ma, Jiannan Fang, Ismaël Koné, Lahsen Boulmane, Aurélio Campilho, Catarina Eloy, António Polónia, and Paulo Aguiar. BACH: grand challenge on breast cancer histology images. CoRR, abs/ 04277, 2018
/attt://arxiv. arxiv http://pxxxxxxxxa///////a/aoaaa/ 1808 04277
Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In Proceedings of the 34th Inter- national Conference on Machine Learning, JCML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 233-242, 2017. URL http:: http:::/prcceeeeee ngs ml www.mmm.r.sss......m www.preessvvvvv vv/as//v00aaa wwtp0////pit. aaaairriit tlmpitt1al 7a html
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In ICLR, 2015.
Jiri Borovec, Arrate Munoz- Barrutaaa and Jan Kybic. Benchmarking of image registration methods for differently stained histological slides. 10 2018. doi: 10.1 10.1109/cip..000 http:/iccpp........... 2218.8411044 8451040.
Wieland Brendel and Matthias Bethge. Approximating CNNs with lag-of-local-feature models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019. URL
P. Bándi, O. Geessink, Q. Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B. Ehteshami Bejnordi, B. Lee, K. Paeng, A. Zhong, Q Li, F. G. Zanjani, S. Zinger, K. Fukuta, D. Komura, V. Ovtcharov, S. Cheng, S. Zeng, J. Thagaard, A. B. Dahl, H. Lin, H. Chen, L. Jacobsson, M. Hedlund, M. Çetin, E. Haltei, H. Jackson, R. Chen, F. Both, Franke, H. Küsters- Vandevelde, W. Vreuls, P. Bult, B. van Ginneken, van der Laak, and G. Litjens. From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 17 challenge. IEEE Transactions on Medical Imaging, 38(2):550-560, Feb 2019. ISSN 0278-0062. doi: 10.11009/M122011828666
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Diest, Bram Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen A. W. M. van der Laak, and the CAMELYON16 Con- sortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Machine Learning Detection of Breast Cancer Lymph Node Mathsta Learning Detection of Breast Cancer Lymph Node Metastases. JAMA, doi: 10. 201.0//aam..00...1e6m
31 318922))2199-2211, 12 2017. ISSN 0098-7484.
URL
1001/ 1000/jamaa... 201 14585
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
// www..eepplean.... www.oriiinoooooooro
For experiments with unhalanced binary data, see section C.I of the supplementary material.

---

Under review as a conference paper at ICLR 2020
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems. pp. 8527-8537, 2018.
Martin Handford. Where's 's Wally? Walker, 1987.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image abs/ss1s 03385
recognition. CoRR, abs/ 151/1552.,3385, 2015. URLLhttp///axiitii
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 345.645. Springer. 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Maximilian Ilse, Jakub M Tomezak, and Max Welling. Attention- deep multiple instance learning. arXiv preprint 102.v4180..04712, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32Nd International Conference on International Conference Machine Learning Volume 37, ICML 15, pp. 448-456. JMLR.org. 2015. URL http: /attp://dl.do http://ccaaa.........aea.eea.11. 304516
Stanislaw Jastrzehski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint 17 2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055, 2017.
Angelos Katharopoulos and François Fleuret. Processing megapixel images with deep attention- sampling models. arXiv preprint arXiv:1905.03711, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012. URL
nips wwtpppprrr.nsss.ccppppe...eeee cc/paper/
4824
pdf
Yann LeCun and Yoshua Bengio. The handbook of brain theory and neural networks. chapter Convolutional Networks for Images, Speech, and Time Series, pp. 255-258. MIT Press, Cambridge, MA, USA, 1998. ISBN 0-262-51102-9, URL U /tcttp//cl... acm. hwtp://rgggciiiiieeo cfm
303568. 303704
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324, 1998.
Byungjae Lee and Kyunghyun Paeng. A robust and effective approach towards accurate metastasis detection and pn-stage classification in breast cancer. CoRR, abs/1805.12066 2018. |
/aaxxvvorg//bbs
12067
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European conference on computer vision, pp. 750-755. Springer, 2014.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 181-196, 2018.
10

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-22111 2212, 2014.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Confer- ence on Learning Representations, 2018. URL httph:: http:::///penzeveete net forum?ic
HJC2Sz2CR
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivie. object localization for free?- supervised learning with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 685-694, 2015.
Jiangmiao Pang, Cong Li, Jianping Shi, Zhihai Xu, and Huajun Feng. R2-cnn: Fast tiny object detection large-scale remote sensing images. arXiv preprint (xXiv:992200604: 2019.
S. Resta, N. Acito, M. Diani, G. Corsini, Opsahl, and T. V. Haavardsholm. Detection of small changes in airborne hyperspectral imagery: Experimental results over urban areas. In 2011 / 6th International Workshop on the Analysis of Multi-temporaa Remote Sensing Images (Multi-Temp), pp. 5 8, July 2011. doi: 10.1 109/Multi- WwWMMuuli-Temp-222. Temp.201 201 16005533.
Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S.N. Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, Rob- bert van der Gugten, Pheng Ann Heng, Bart Jansen, Michael M.J. de Kaste, Valentin Kotov, Jack Yu-Hung Lin, Jeroen T.M.C. Manders, Alexander Satora-Mengana, Juan Carlos Garca- Naranjo, Evgenia Pupavasilciou, Mathias Prokop. Marco Saletta, Cornelia M Schacfer-Prokop. Ernst T. Scholten, Luuk Scholten, Miranda M. Snoeren, Ernesto Lopez Torres, Jef Vandemeule- broucke, Nicole Walasek, Guido C.A. Zuidhof, Bram van Ginneken, and Colin Jacobs. Vali- dation, comparison, and combination of algorithms for automatic detection of pulmonary nod- ules in computed tomography images: The lunal6 16 challenge. Medical Image Analysis, 42:1 13, 2017. ISSN 1361-8415. doi: https://bo.orr/11..00.......mm 20...0...1........ URL http: 7/www. wwww.sciieccedrre... rect. com/ http:///ccccnn.aaa.....////////a
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, 0oos//149.1110 1556, 2014.
John Suckling. The mammographic image analysis society digital mammogram database" dxtabase"cxerppta medica. Exerpta Medica International Congress Series, 1069, 01 1994.
Carole H Sudre, Beatriz Gomez Anson, Silvia Ingala, Chris D Lane, Daniel Jimenez, Lukas Haider, Thomas Varsavsky, Lorna Smith, H Rolf Jäger, and M Jorge Cardoso. 3d multirater for multimodal multiclass detection and characterisation of extremely small objects. arXiv preprint arXiv:1812.00044 2018
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017. 2017.
T. Tieleman and G. Hinton. Lecture 6.5 :ss-Roppopp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Naofumi Tomita, Behnaz Abdollahi, Jason Wei, Bing Ren, Arief Suriawinata, and Saeed Hassanpour. Finding needle in the haystack: Attention-based classification of high resolution microscopy images. arXiv preprint arXiv:1811.088 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/ fbss/160..0g 1007000002,, 2016.
Jason W. Wei, Laura J. Tafe, Yevgeniy A. Linnik, Louis J. Vaickus, Naofumi Tomita, and Saced Hassanpour. Pathologist-lewe level classification of histologic patterns resected lung adenocarcinoma slides with deep neural networks. CoRR, abs/1901 1449,, 2019. URL URL //tpxx..gx/////////
abs 1901 11489
11

---

Under review as a conference paper at ICLR 2020
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection aerial images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 2017. URL https: /tttpp://arxiv.ooo org/bbs...g........ 1611. 03530
12

---

Under review as a conference paper at ICLR 2020
Supplementary Material for
Needles in Haystacks: On Classifying Tiny Objects in Large Images
A DATASETS
In this section, we provide additional details about the datasets used in our experiments.
A.1 NEEDLE MNIST
The needle MNIST (nMNIST) dataset designed as a binary classification problem: Is there a in this image?'. To generate nMINST, we use the original training, validation and testing splits of the MNIST dataset and generate different nMINST subsets by varying the object-to-image (O2I) ratio, resulting in 02I ratios of 19.1 1%, 4.8%, 1.2 1.2%, 0.3%, and 0.075%. We define positive images as the ones containing exactly one digit and negative images as images without any instance of We keep the original MNIST digit size and place digits randomly onto clear canvas to generate sample of the nMNIST dataset. More precisely, we adapt the O2I ratio by changing the the canvas size, resulting in nMNIST image resolution being in 64 x 64, 128 128, 256 256, 512 x 512, and 1024 x 1024 pixels. To assign MNIST digits to canvas, we split the MNIST digits into two subsets: digit-3 versus clutter (any digit from set of {0. 1, 2, 4, 5, 6, 8, (.))). For the positive nMNIST images, sample one digit 3 (without replacement) and n digits (with replacement) from the digit-3 and clutter subsets, respectively. For the negative nMNIST images, we sample subset. We adapt n to keep approximately constant object density for all canvas and choose n to be 2, 5, 25, 100, and 400 for canvas resolutions 64 x 64, 128 128, 256 x 256, 512 x 512, and 1024 x 1024, respectively. As result, for each value of O2I ratio, we obtain 11276, 1972, 4040 of training, validation and testing images, out of which 50% are negative and 50% are positive images. We present both positive and negative samples for different 02I ratios in Figure 10
instances from the clutter
3
7
C
GG
35
(a) 021
(b) 4.7%
(c)
1.2%
(d) raso
021 ratio 0.075%
4
2
5
b
2
1
6
0
6
4 8
47
9
8
4
86
4
A6
D2 miio 19.15
(g) 021 4.7%
(h) 021 miin - 1.2%
(i) 021 ratio 0.3%
(j) 021 ratio 0.075%
Figure 10: Example images from our MNIST dataset with different 021 ratios. Top row images represent positive examples- - digit is present (marked with red rectangle), while bottom row depicts negative images. Note that for visualization purposes all images have been rescaled to the same resolution.
A.2 NEEDLE CAMELYON
The needle CAMELYON (#CAMELYON) is designed as a binary classification task: Are there breast cancer metastases in the image not?. We rely the pixel-level annotations within CAMELYON to extract samples for nCAMELYON, We use downsampling level 3 from the original whole slide image using the MultiResolltion Image interface released with the original CAMELYON dataset. For positive examples, we identify contiguous regions within the annotations, and take 50 random crops around each contiguous region ensuring that the full contiguous region is inside the crop, and
13

---

Under review as a conference paper at ICLR 2020
OZIR [50 100/%
(b) 028 E [10 50.%
(C) OZIR [2 10,%
0218 E [0.1 11%
(e) Negatvee Example
(f) O2IR [50 100/% %
(g) O2IR [10 50
(h) 02TR [1 10j%
(i 021R [0.1 11%
Negative Example
(k) OZR [50- 100]%
()) OZIR E [10 50/%
(m) OZIR E 10.%
(n) OZIR E [0.1 1J%
(0) Negative Example
Figure 11: Example images from our CAMELYON dataset for different crop sizes and 021 ratios. We show crops with size 128 x 128, 256 x 256, and 512 512 in the top, middle, and bottom row, respectively. The green outlines show the cancerous regions. Note that for visualization purposes all images have been rescaled same resolution.
total number of lesion pixels inside the crop are in the desired 02I ratio. The negative crops are taken from healthy images randomly filtering for images that are mostly background using a heuristic that the average green pixel value the crop is below 200. Since the CAMELYON dataset contains images acquired 5 different centers, split training. validation and test sets center-wiss wise to avoid any contamination
data across the three sets. All crops coming from center are part of the validation set, and all crops coming from center 4 are part of the test set. All images are generated for resolutions 128 x 128, 256 x 256, 512 x 512, and 1024 1024 and are split into 4 different O21 ratios: (100- 50)%, (50- 10)%. (10 1)%, and (1 0.1) 1.. Figure I1 shows examples of images from ACAMELYON dataset, Table presents number of unique lesions in each dataset, and Table 2 depicts number of dataset images stratified for image resolution and 02I ratios. Because center 3 does not contain lesions of suitable size for crops of with resolution 128 x 128 and 02I ratio (50 100)%. we not include those training runs in our analysis.
Table 1: Number of unique lesions extracted for each set of the nCAMELYON data for differen 021
ratios and crop sizes.
Crop Size
128
256
512
021 ratio
Train
Val
Test
Train
Val
Test
Train
Val
Test
(50 100)% %
20
0
27
13
23
5
13
500-
84
12
16
101
16
15
68
15
17
(1- 10)%
176
17
18
227
17
18
235
21
15
(0.1 - 1)%
5
93
16
9
173
20
11
14

---

Under review as a conference paper at ICLR 2020
Table 2: Number of crops extracted for each set of the nCAMELYON data for differen O2I ratios and crop sizes. Note that the dataset is balanced (e. g. 50% are positive images and 50% are negative). Moreover, for positive images we have relatively small number of unique cancer regions as noted in
Table 1
Crop Size
128
256
512
ratio
Train
Test
Train
Val
Test
Train
Val
Test
(50 100)%
1000
400
1350
100
650
1150
250
650
(10-50)%
4200)
600
800)
5050
800)
750
3400
750
850
(1 10)%
8686
850
900
11270
850
900
11750
1050
750
(0.1 1)%
1488
247
207
4255
800
450
8312
965
negative
19608
6000
6100
19595
6000
6100
19574
6000
6100
B EXPERIMENTAL SETUP
In this section, we provide additional details about the pipeline used in the experiments. More precisely, we formally define global pooling operations and provide detailed description of the different architectures.
B. GLOBAL POOLING OPERATIONS
In our experiments, we are testing four different global pooling functions: max-pooling, mean- pooling, logsumexp and soft attention. The max pooling operation simply returns the maximum value per each channel in the topological embedding. This operation can be formally defined subscript notation denote dimensions of the
E max max; E Bi Note, that embedding. The max pooling operation has spacing effect on gradient backpropagation, during the backward pass through the model all information will be propagated through the embedding position that corresponds the maximal value. order to improve gradient backpropagation, one could apply logsumexp pooling, a soft approximation to max pooling. This pooling operation is defined as:
hene
E
log
exp
E [r,h]
w==1
1=1
Alternatively, one could use an average pooling operation that computes mean value for each channel in the topological embedding. This pooling operation can be formally defined as follows:
hence
E
E
(m,
Wene
Ienn
Finally, attention based pooling include additional weighting tensor a dimension (Wene X hene X Cenc) that rescales each topological embedding before averaging them. This operation can be formally defined as:
hane
[lo.h]
(3)
h=1
Wene hene
s.t.
a [(mnkh]
(4)
In our experiments, following lsse et al. (2018), we parametrize the soft-attention mechanisms as a [| = sof coftmar(] (FE (Psppt))[w.h)) [w.h) where f(.) is modelled by two fully connected layers with lanh-activation and 128 hidden units.
B.2 MODEL ARCHITECTURE DETAILS
We adapt the BagNet architecture proposed (Brendel & Bethge 2019) An overview of the architectures for the tested three receptive field sizes is shown in Table We depict the layers of
15

---

Under review as a conference paper at ICLR 2020
100
066
Accurct
01
0.2
Legnustep
0.6
0.5
0.3
0.99 2.30
0.25
133 0.75 Delance
0.90
0.01
030 025 6bT
0.75
0.90
0.99
003 2.30 025 0.00 0.75
0.90
0.9
Oelance
Dalance
Figure 12: Impact of the training set balance on model accuracy for different pooling operations and receptive field sizes.
residual blocks in brackets and perform downsampling using convolutions with stride within the first residual block. Note that the architectures for different receptive fields differ in the number of 3 x 3 convolutions. The rightmost column shows a regular ResNet-50 model. The receptive field is decreased by replacing 3 x 3 convolutions with x convolutions. We increase the number of convolution filters by a factor of 2.5 if the receptive field is reduced to account for the loss of the trainable parameters. Moreover, when testing different network capacities we evenly scale the number of convolutional filters by multiplying with constant factor of s E 1,,
C ADDITIONAL RESULTS
In this section, we provide additional experimental results as well as additional visualizations of the experiments presented in the main body of the paper.
C.I CLASS-IMBALANCED CLASSIFICATION
In many medical imaging datasets, it is common to be faced with class-imbalanced datasets. Therefore, in this experiment, use our nMNIST dataset and test CNNs generalization under moderate and severe class imbalanced scenario. We alter the training set class balance by altering the proportion of positive images the training dataset and use the following balance values 0.01, 0.1, 0.25, 0.5, 0.75, 0.9 and 0.99, where value of 0.01 means almost no positive examples and 0.99 indicates very low number of negative images available at training time. Moreover, we ensure that the dataset size is constant (zV 11k) and only the class-balance is modified. We run the experiments using the 021 ratio of 1 1%, three receptive field sizes x 9, 33 x and 177 177 pixels) and four pooling operations (mean, max, logsumexp and soft attention). For each balance value, we train 6 models using random seeds and we oversample the underrepresented class. The results are depicted in Figure 12 We observe that the model performance drops the the training data becomes more unbalanced and that max pooling and logsumexp seem to be the most robust to the class imbalance.
C.2 INCREASE OF MODEL CAPACITY FOR SMALL DATASET SIZES.
We also tested the effect of model capacity increase while having access only a small dataset (3k class-balanced images) and contrast it with larger dataset of ~ 11k training images. We run this experiment on the nMNIST dataset using network with 2.3 10" parameters using global max pooling operation and there different receptive field sizes: 9 x 9, 33 x 33 and 177 x 177 pixels. The results are depicted in Figure 13 can be seen that the model's 's capacity increase does not lead to better generalization, for small size datasets of 2s 3k.
C.3 02I LIMIT VS. DATASET SIZE
In this section, we report additional results for all tested global pooling operations on 02I limit vs. dataset size. We plot heatmaps representing the validation set results for all considered 021 and training set sizes (Figure [4p as well as the minimum number of training examples required to achieve validation accuracy of 70% and 85% (Figure 15)
16

---

Under review as a conference paper at ICLR 2020
#Training
ATraining
#Training
69
Twxt Aoruneyy
0.5
201
-Training - 11276 9
aTroining 1 88-7618-22
#Triiing 11276 177
1.9
Accumery
0.8
0.6
Loquumexp
Lagarmerp
Leqaumexp
Pooking
Pooling
Peoling
Impact of the network capacity on the generalization performance dependent on the training set size for AMNIST 02I ratio 1.2%. The improvement based on the increased network capacity shrinks with smaller training set.
Figure 13:
Method Max
Pooling Melhod
Melthd Mean
Method stention
0:53
O21 rato [N]
4.79
0:59
0.75
074
0.2
0.78
0:58
0.76
0.54
0:57
0.62
0.72
18:54
0.00
0.72
0.51
0.02
0.00
0.57
0.63
0.69
0.70
0.7
0.50
0.77
0.60
0.72
0:11
0.02
0.11
0.00
0.00
0.60
0.70
0.76
0.00
0.00
-06
1400
20002
20002
1400
20502
1400
2000
1379
20002
Figure 14: Testing the 02I limit. Validation set accuracy heatmap for max, logsumexp, mean and soft attention poolings. We test training set sizes E (1400, 2819, 5638, 7500, 11276, 22552) ) and report the average validation accuracy.
C.4 WEAKLY SUPERVISED OBJECT DETECTION: NMNIST
We test the object localization capabilities of the trained classification models by examining their saliency maps. Figure 16 shows examples of the nMNIST dataset with the object bounding box in
Prssing Veettel Naxo
Presing Lbetrett
Prening Natnos Mean
Pooling Merboe Seaft atberinn
Wildeles
Of 2%
300
2019
0.0
1.00
470
19.14
6:3
1.20
4.73
14
533
1.00
172
18:55
0.00
1:00
4.75
13
Figure 15: Testing the 02T limit. Minimum required training set size to achieve the noted valida- tion accuracy. We test training set sizes e {1400, 2819, 5638, 7500, 11276, 22552) and report the minimum amount of training examples that achieve a specific validation performance pooling over different network capacities.
17

---

Under review as a conference paper at ICLR 2020
9
5
b
= 10y=1ÿÿ-
4
89 | b
1
3
1
7
3
J
1
88
(e) 1, =1
(i) 0, =1
(g) y 1, =
(h) = 0,9 00
y
9 8
2
0
4 4 0 6
2
5
1 1
y
4
9
8 9
80
7
8
7
O
O
7
4
3 4
69
0
9 4
4
15
24 4
3
1/7
4
2
2
(i) y = 1, 11
(j)
= 0, 1 =
(k) y = 1, 0 0
(1) y = 0, 0 =0
o
2
12%
120
75
(m) y 1,
(n) y 0,
(0) y 1, 0
(p) 0,
Figure 16: Example images from the nMNIST validation set and their corresponding saliency maps in red. We generate the saliency maps by calculating the absolute of the gradients with respect the input image using max max-pooling we show random examples for 02I ratios of [19.14 4.79, 1 1.20, 0.30) %. We annotate the object of interest with a blue outline. The captions show the true label y and the prediction ÿ.
receptive field of 33, and ResNet-50 capacity. From top to bottom,
18

---

Under review as a conference paper at ICLR 2020
Mean
Logsumexp Soft attention
RF
Max
0.7
0.7
Average Precsion
Average Precsion
0.5
0.5
0.3
0.3
0.2
0.2
0.0
0.0
0.30
120
4.79
19.14
0.30
1.20
4.79
19.14
021 ratio [%6]
021 ratio [%]
(a)
(b)
Figure 17: Average precision for detecting the object of interest using the saliency maps for nMNIST. We adapt (Oquab et al., (015) and use the localize object by the maximum magnitude of the saliency. We use the magnitude of the saliency as the confidence of the detection. We count wrongly localised objects both false positive and false negative. For images without object of interest, the we increase the false positive count only. We plot results for max- max-pooling, a receptive field 33, training set with 1 1177 examples and ResNet-50 capacity. (a) shows the dependence of the AP on the pooling method using RF 33 x 33, (b) shows the dependence on the receptive field using max-pooling.
blue and the magnitude of the saliency in red. We rescale the saliency to 00, 1] for better contrast. However, this prevents the comparison of absolute saliency values across different images. In samples containing object of interest, the models correctly assign high saliency to the regions surrounding the relevant object. On negative examples, the network assigns homogenous importance to all objects.
We localise an object of interest the location with maximum saliency. We follow (Oquab et al. 2015) to quantitatively examine the object detection performance using the saliency maps of the models. We plot the corresponding average precision in Figure 17, We find that the detection performance deteriorates for smaller 02I ratios regardless of the method. This is aligned with the classification accuracy. For small O2I ratios, max-pooling achieves the best detection scores. On larger 02I ratios, logsumexp achieves the best scores.
C.5 WEAKLY SUPERVISED OBJECT DETECTION: NCAMELYON
We qualitatively show object detection on aCAMELYON in Figures 19 21 for True Positives, True Negatives, False Positives and False Negatives. We observe weak correlation between segmenta- tion maps and saliency maps, signifying that the classifier was able to focus on the object of interest instead of looking at superficial signals in the data.
19

---

Under review as a conference paper at ICLR 2020
50
50
100
150
150
200
200
100
150
200
250
50
100
150
200
250
(a)
(b)
50
50
100
100
200
250
250
50
100
150
200
250
50
100
150
200
250
(c)
(d)
0
100
100
150
150
200
250
100
150
200
50
100
150
200
250
(c)
Figure 18: Example True Positive Images of 1CAMBLYON validation sets and their corresponding segmentation maps with saliencies overlaid.
20

---

Under review as a conference paper at ICLR 2020
50
50
100
150
150
200
200
150
250
100
150
200
250
(a)
(b)
Figure 19: Example True Negative Image of nCAMELYON validation sets and corresponding saliency map.
50
50
100
100
200
250
250
0
50
100
150
200
250
0
50
100
150
200
250
(a)
(b)
Figure 20: Example False Negative Image of aCAMELYON validation sets and corresponding segmentation map with saliency overlaid.
0
50
100
100
150
150
200
200
250
150
200
100
150
200
(a)
(b)
Figure 21: Example False Positive Image of nCAMELYON validation sets and corresponding saliency map.
21

---

Under review as a conference paper at ICLR 2020
Table Schematic the architecture of the different topological embedding encoders used this paper. The operations and their corresponding parameters of the residual blocks denoted in brackets. The first block within each section performs downsampling using convolutions with stride We use InstanceNorm instead of BatchNorm and test different pooling methods after the topological embeddings.
RF=9
RF=33
RF=177
conv, 1 x 64
conv, 3 64
conv, x 1, 64
conv, x 1, 64
conv, X 1, 64
conv, 3 x
conv, 3 x 3,
conv, x 3, 64
conv, 1,256
conv, x 1,6
conv, 1,226
conv, 1 1,
conv, 6,
conv, x 1,
conv, x 160
conv, 160
conv, x 3,
conv, x 1,6
conv, x 1,,
conv, x 1,26
conv, x 1,
conv, x 1,.64
conv, x 1.1,,00 14
conv; x 1, 160
conv, 1 x 160
conv, x 64
conv, x 1,6
conv, 1,2
conv, 1 x 1,256
conv, 1, 128
conv, x 128
conv, 1, 128
conv, x 3, 128
conv, 3 x x
conv, 3 3,
conv; x 1,512
conv, x 1,512
conv, x 1,52
conv, x 128
conv, x 128
conv, x 128
conv, x 1,320
conv, X 1,,300
conv, 3 x 128
conv, x 1,512
conv, x 1,512
conv, x 1, 1,512
conv, x 128
conv, X 128
conv, x 128
conv, 1,320
conv, 1,,20
conv,
conv, 1 x 1,512
conv, 1 x 1,512
conv, 1 x 512
conv, x 1, 128
conv, x 1,8
conv, x 1 1, 128
conv, X 1,320
conv, X 1,,
conv, 3 x x, 128
conv, x 1,512
conv, x 1,512
conv, 1 x 1,512
conv, 1 x 2,6
conv, x 152
conv, x 256
conv, 140
conv, 3,2
conv, 3, 256
conv, x 1, 1024
conv, 1 1024
conv, 1 x 1024
conv, x 1, 256
conv, x 1,2
conv, 1, 256
conv, x 140
conv, x 1,600
conv, 3 x x, 256
conv, 1 1, 1024
conv, 1 1, 1024
conv, 1 x 10002
conv, x 1, 256
conv, x 256
conv, 1 x 1, 256
conv, 140
conv, 140
conv, x
conv, 1024
conv, 1 1, 1,24
conv, 1 x 1,24
conv, x 2,6
conv, 1, 256
conv, 1, 256
conv, x 140
conv, 1 x 1, 140
conv, 3 x 3, 256
conv, 1 x 1, 1024
conv, 1 1024
conv, 1 x 1024
conv, x 1, 256
conv, x 1,
conv, x 1, 256
conv, x 140
conv, x 140
conv, 3 x 3, 256
conv, 1024
conv, 1, 1024
conv, 1 1024
conv, 1, 1,255
conv, 156
conv, 1, 256
conv, x 140
conv, 140
conv, 3 x 3, 256
cony, 1 x 1, 1024
conv, 1 1,
conv, 1 x 1024
conv, x 512
conv, x 115
conv, 1 x 1, 512
conv, 1 1280
conv, x 3,522
conv, 3 x 512
conv, 1,2018
conv, 1 1,2048
conv, 1 1,2048
conv, 1,122
conv, x 1,512
conv, 1,512
conv, 1 1, 1280
conv, 1 1,1280
conv, 3, 3,212
conv, 1 1,2048
conv, 1
conv, x,2008
conv, 1 x 1,512
conv, x
conv, 1,1,,12
conv, 1 x 1, 1280
couv, 1 x 1,1280
conv, x 3,
conv, 1 x 2048
conv, 1 1,2048
conv, 1 x x, 1,48
conv, 1 x 1,512
conv, 1,512
conv, x 1,512
conv, 1280
conv, 1 1280
conv, 3. 512
conv, 1, 2048
conv, 1,2048
conv, 2048
22


Reviews:
[{"review_id": "SylBTBUTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes an analysis of the impact of object size in images when performing classification tasks using neural networks of the BagNet family. The analysis is performed on two datasets, a large resolution cluttered MNIST and a histopathology dataset named nCAMELYON.\n\nThe paper attack interesting questions and links the size of the object in the image (O2I) to the training dataset size required. Also, showing that max-pooling is the only pooling operation that converges for very low O2I (but is the slowest to converge at higher O2I) is interesting and encourages discussion about the training (optimization) process.\n\nIn my opinion, the main issue about the submission is the limited depth in the contributions, analyzing a single family of network architectures (BagNets) over two datasets, one of which is relatively small. The family of R-CNN and its derivatives were especially designed to counter the impact of object size, it would have been interesting to include them in the analysis. Furthermore, limited insights can be carried out for tasks related to classification such as localization and segmentation. Models such as Single Shot Detectors split the image into grids and variable anchor sizes to perform their inference, are they affected to a lesser extent by object size? The limited insights provided to potential future readers prevents me from recommending the submission for acceptance.\n\nOn p. 7 (and fig. 7 (b) ), it is said that the lower performance of the larger receptive field suggests that class-relevant information is contained in the texture. I am not sure about this remark, I would have expected the network to learn to focus on the right regions, provided the receptive field is big enough to see the whole object of interest. Could the decrease in performance be attributed to the increased amount of learnable parameters that ended up too large for the “relatively small nCAMELYON dataset used for training”? (sec. 3.1, Global pooling operations)\n\nFig. 16 seems to suggest that multiple numbers can overlap significantly in nMNIST, yielding potentially confusing images even for humans. I am not sure if this is a desirable characteristic for such dataset.\n\n\nMinor details\n- Please use \\cdot instead of the asterisk operator to denote multiplication (sec. 3, footnote 3, fig. 5 (b-c) and sec. 3.1;\n- Fig. 4 uses two different (and non-linear) values for the x-axes, giving the impression that both curves can be compared, that might be confusing to the reader;\n- Sec 1 and 5 “pixel-level level annotations”: duplicate “level”;\n- p. 6 “1 and , 2”: extraneous comma.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sye60k9rqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present an empirical study to evaluate the performance of CNN-based object classifiers for situations in which the object of interest is very small relative to the size of the image. Two artificial datasets, based on MNIST and histopathological images are introduced to conduct the experiments. Through empirical evaluation the authors conclude that the size of the dataset required for generalization increases rapidly with the inverse of the O2I ratio, that higher capacity models generalize better, and that accounting for the model's receptive field is key.\n\nThe contributions of the study are limited: i) The artificial datasets generate images that are small and O2Is that are big for the applications of interest, e.g., gigapixels images in digital pathology (see Figure 1). ii) The dataset based on MNIST is perhaps too artificial (too structured), once one compares their results relative to nCAMELYON. iii) The authors only consider ResNet-50. iv) The authors do not consider multi-instance learning pooling functions, e.g., noisy or, noisy and or attention. v) The authors do not consider performance as a function of the positive instances in the image (number occurrences of 3 in the proposed nMNIST). vi) There is no methodological contribution.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HylRTeIAYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a testbed framework to investigate the limitations of CNN at the classification of tiny objects and the effects of signal to noise ratio has in the task. The implemented framework will be made available online upon acceptance.\n\nI believe that the question the authors try to answer is very interesting and worth of exploration. However, I am a bit less excited about the achieved results since I consider them not to be sufficient to drive conclusions. The experiments proposed by the authors are run on two different datasets created ad-hoc: the nMNIST and the nCAMELYON, both modifications of MNIST and CAMELYON datasets. Over all the experiments run, the behaviors observed on the two datasets are not the same. The explanation provided by the authors is that since nCAMELYON is very small, results are different from the ones of nMNIST. While I consider this a valid explanation, this still limits the overall conclusions of this paper. Therefore, my main recommendation to the authors would be to identify other datasets for their experiment.\n\nIn the dicussion section, the authors argue that MS COCO is not a good candidate since the background usually contains information that allows to infer the label. I wonder if it would be possible to generate a new dataset, starting from MS COCO, that avoids this problem. Using the example of the paper, I would argue that it is possible to obtain images of outdoors with people where there are no balls. Having such a dataset would make your paper much stronger.\n\nFinally, to some extent, I consider that your problem is strongly linked to anomaly/detection detection. In the end, this is what a needle in a haystack is. How do you position yourselves with respect to the state of the art on this topic?  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes performs an empirical study to evaluate CNN-based object classifier for the case where the object of interest is very small relative to the size of the image. Two synthetic databases are used to conduct the experiments, through which the authors made a number of observations and conclusions. The reviewers concern that the databases used are too structured or artificial, and one of the two databases is very small as well. On top of that, only one network architecture is used for evaluation. Furthermore, the conclusion from two databases seem inconsistent as well. The authors provided detailed responses to the reviewers' comments but were not able to change the overall rating of the paper. Given these concerns, as well as no methodological contribution, there are general concerns from all reviewers that the contributions of this work is not sufficient for ICLR. The ACs concur the concerns and the paper can not be accepted at its current state.
Decision: Reject