Title: PSEUDOSACCADES: A SIMPLE ENSEMBLE SCHEME FOR IMPROVING CLASSIFICATION PERFORMANCE OF DEEP NETS

Abstract: We describe a simple ensemble approach that, unlike conventional ensembles, uses multiple random data sketches ('pseudosaccades') rather than multiple classifiers to improve classification performance. Using this simple, but novel, approach we obtain statistically significant improvements in classification performance on AlexNet, GoogLeNet, ResNet-50 and ResNet-152 baselines on Imagenet datae.g. of the order of 0.3% to 0.6% in Top-1 accuracy and similar improvements in Top-k accuracy -essentially nearly for free.

1. INTRODUCTION:

Deep Neural Networks (DNN) are state-of-the-art tools for various machine learning tasks (LeCun et al., 2015) and have proved especially useful for image classification tasks. For example, the most recent winners of the Imagenet challenge 1 have all been DNNs. Although it is not at all well understood -at least in terms of formal learning-theoretic guarantees -how and why DNNs perform so well 2 , empirical understanding of how to construct a DNN is substantial and growing, and there are many plausible hypotheses regarding their performance. One striking example of the latter is that not only do DNNs have clear parallels with aspects of human visual processing, but in controlled psychological experiments they also match human performance on visual recognition tasks very closely (Serre et al., 2007) .

Taking inspiration from nature, in this paper we show that an approximate analogue for saccades in human visual processing can improve the performance of a carefully-tuned DNN on an image classification task that it was explicitly designed to solve. More precisely, we use a very simple ensemble-like approach that employs voting but, unlike typical ensemble approaches, rather than learning several similar DNNs and obtaining a weighted combination of votes from that ensemble, instead we use just a single DNN but feed it as input multiple random low-dimensional sketches of an image and take the DNN's vote with itself on these sketches to reach a majority verdict.

Using our simple approach we obtain statistically significant improvements in classification performance on AlexNet, GoogLeNet, ResNet-50 and ResNet-152 baselines on Imagenet data -e.g. of the order of 0.3% to 0.6% in Top-1 accuracy -essentially nearly for free. We carry out a comprehensive empirical exploration of our approach, reporting results using different levels of subsampling and different ensemble sizes, as well as an initial exploration of whether the improvements have any identifiable systematic component (such as occurring disproportionately in the same class).

2. MOTIVATION:

Our approach is inspired by considering saccades in human visual processing, that is eye movements that focus attention on elements in a visual scene. The human eye has only a few degrees of visual arc of high-resolution imaging capability, and saccades are a mechanism by which a scene can be estimated from high-resolution subsampling of parts of it. In human visual processing this subsampling is not uniformly at random -we attend to certain features proportionately more often than othersbut we hypothesised that an evolutionary precursor to saccades could have been something closer to a uniform random sampling of features in a scene and that (if indeed there was such a precursor) this must have conferred some selective advantage in order to propagate. Meanwhile recent theoretical results in Lim & Durrant (2017) show that randomly subsampling rows and columns from an image without replacement results in -with high probability -an approximately affine transformation of the original image. Putting these ideas together, since image labels should remain invariant under affine transformations, we speculated that such subsampling could potentially lead to improved classification performance, perhaps even for an already highly-accurate classifier, by providing the classifier with multiple low-dimensional sketches of the same image in a similar way that saccadic sampling of a scene does. We call this subsampling of rows and columns 'pseudosaccades'. Because image classification inputs can be of varying sizes, while most classification algorithms accept only a fixed size input, a common preprocessing is to convert them to a (usually smaller) standard-sized input prior to classification. However as far as we are aware it has not been much exploited before that such preprocessing offers an opportunity for generating multiple instances of a particular image. By extracting pseudosaccade sketches of an image before applying the standardizing preprocessing, allows the generation -for typical image sizes -of thousands of such instances per image. Moreover unlike cropping and reflection the resulting pseudosaccade images resemble photographs captured following a change of camera angle and position, while still keeping the subject central in scene -see figure 1 . Confirming our conjectures, we found that exploiting several of these pseudosaccade sketches can, indeed, improve classification performance and in the remainder of this paper we present and discuss our experimental results. 

3. EXPERIMENTS AND RESULTS:

In this section we present details of our experimental protocol and the results of our experiments. We show that the classification accuracy on a single pseudosaccade version of an image is similar to the accuracy on the original images, given a suitably high projection dimension. Moreover, using pseudosaccades as a diversity generator, an ensemble classifier employing several pseudosaccade versions of each image can consistently outperform the classification accuracy of the same classifier on the original images.

3.1. DATASET:

We used the validation dataset from the Imagenet Large Scale Visual Recognition (ILSVR) Challenge 2012 as described in Russakovsky et al. (2015) for our experiments. This dataset comprises of 50000 images, ranging in size from 56x54 pixels to 5005x3646 pixels, where each image is an example from one of 1000 distinct classes. The subject of an image (i.e. its class label) is the dominant, and usually the central, object in that image and therefore elements of attentive viewing are already present in these images by virtue of the location of the subject. The classes in this dataset range from broad categories to fine-grained labels -for example one subset of the labels is a classification of 120 different breeds of dogs. 2018) and summarizes the characteristics of these DNNs as well as the baseline accuracies we obtained on the ILSVR challenge validation dataset using them.

3.3. EXPERIMENTAL PROCEDURE:

To obtain baseline accuracies for each of the four DNNs we classified each image in the ILSVR validation set with no preprocessing, other than that implemented by the DNN itself to standardize the image sizes. The preprocessing carried out by the DNNs is noted in table 2 . We measured the top-1, top-3 and top-5 accuracy for each classifier on the full validation set of 50000 imagesthese are also presented in table 2 and we will refer to these results obtained on the original images (without subsampling) as the 'baseline classifier' results.

For our pseudosaccade approach we first fix the 'projection dimension' to be an integer d ∈ {450, 430, 410, 390, 370, 350, 330, 310, 290, 270, 250, 200, 150} and then randomly sample min(d, width) columns and min(d, height) rows from the images without replacement. As in the baseline experiments, we apply no further preprocessing, other than that implemented by the DNN to standardize input size, and we measure the top-1, top-3 and top-5 accuracy for each DNN on all 50000 images in the ILSVR validation dataset. We refer to these results as the 'saccade classifier' results. We also store the scores and the top-5 predicted labels for each combination of sampled projection dimension d, image, and DNN. Since the obtained accuracies, scores, and labels are realizations of random variables we repeated these experiments for each combination of d, image, and DNN a total of twenty-four times, and we calculated the means and standard deviations for the top-1, top-3 and top-5 accuracies.

Keeping d fixed we construct an ensemble of size m ∈ {1, 2, . . . , 15} using the scores of between one and fifteen saccade classifiers by sampling without replacement m sets of top-5 scores from the 24 sets of stored saccade classifier scores. We combine these to obtain the ensemble decision by simply summing scores for each label. For each d, m, k triple and each classifier we repeated this process fifty times and we calculated the corresponding means and standard deviations for the top-1, top-3, and top-5 accuracy. 2 for all values of d, m and k. In figure 2 the orange plane shows the baseline accuracy for each classifier and top-k combination within a sub-figure. The surface plots show the average classification error for a given d, m, k triple using pseudosaccades. From tables 3 and 4 we see that these average outcomes are very stable indeed, and if the projection dimension d is sufficiently high then even a small ensemble can outperform the DNNs working with the original images at the 5% level of significance (or better) on Top-1, Top-3 and Top-5 classification accuracy.

On the other hand we see that using a single pseudosaccade representation of each of the images, although we can match or nearly match the baseline accuracy with a projection dimension as high as d = 350 (see figure 2 ), with a lower projection dimension we obtain far worse accuracy than the baseline. The curve comprising the left-hand boundary of each surface plot shows the average accuracy for a single pseudosaccade plotted against the projection dimension d. Finally we see that the accuracy of the ensemble exceeds that of the baseline classifiers, even for a small ensemble of classifiers and small projection dimension, and this behaviour is consistent across all of the classifier architectures.

3.5. FURTHER EXPERIMENTS:

A natural question, given the improvements from pseudosaccades, is whether an 'ensemble of ensembles' would improve performance further? We started by looking further into the diversity of the saccade classifiers. Following Kuncheva & Whitaker (2003), we calculate the correlation between the saccade classifier errors and the baseline classifier errors using

ρ i,j = N 11 N 00 -N 01 N 10 (N 11 + N 10 )(N 01 + N 00 )(N 11 + N 01 )(N 10 + N 00 )

, where i is the base classifier, and j is the saccade version with, and the definitions for N 00 , N 01 , N 11 and N 10 are given in table 5. However since all of the accuracies are already high it seemed worthwhile to examine where the improvements were coming from -were these for similar class labels for every classifier for example?

Digging deeper we observed that the classification accuracy of the individual classes is not uniformly affected by pseudosaccades. Moreover, at this lower level of granularity we see that the different architectures do tend to be affected by the pseudosaccades differently.

Tables 10,11 and 9 show lists of predicted class labels for a given class label for ResNet-152, with projection dimension 390 and ensemble size 5. Note that there are 50 instances in each of the true class labels, and we omitted predicted labels where there was only a single prediction or two predictions for reasons of space and readability.

In table 10, we present a list of labels for which the ResNet-152 classifier obtained less than 20% recall -considering this and similar tables for AlexNet and GoogLeNet (omitted due to space constraints) we observed that the ensemble of pseudosaccade classifiers performs similarly to the baseline classifier on labels that are also difficult for the baseline classifier to predict accurately, but we also saw that the different classifier architecture have their own sets of 'difficult labels' that are different.

Finally tables 9 and 11 give examples of class labels where the ensemble classifier respectively gives either a large improvement or is much worse (±10%) on classification accuracy for these classes. Comparing with the other architectures we found that the saccade classifiers were affected differently on different classifier architecture.

Thus, although high-level summary statistics seemed to indicate little diversity between the different ensemble classifiers, a more principled investigation reveals that the errors for both the original DNNs and for the corresponding pseudosaccade ensembles arise from different classes and different instances in the dataset.

We therefore constructed two ensemble classifiers -one using the four baseline DNNs and one which combined four pseudosaccade ensembles to see if further improvements were possible. We used five-fold cross-validation on the validation set data to train a shallow neural network with a single hidden layer with ReLU activations on the baseline scores for 40000 images from the validation dataset to learn a weighting function for the ensemble of baseline classifiers. We used the average and maximum scores from pseudosaccade versions of the four DNNs for the same 40000 images to train a similar network to weight the 'ensemble of ensembles'. We evaluated both ensembles using the 10000 remaining held-out images from the validation dataset and estimated the top-1,top-3 and top-5 accuracies for both ensembles with the cross-validation error. We carried out one round of five-fold cross-validation for the baseline classifiers and 50 rounds for the pseudosaccade classifiers, for different d, m, k triples and calculated the mean accuracies and their standard deviations. For both types of ensemble we saw substantial improvements over the original baseline accuracies and, consistent with our earlier experiments, the pseudosaccade ensembles were yet again able to outperform the ensemble of baseline DNN classifiers. Figure 3 , shows the accuracy of the DNN ensemble versus the pseudosaccade ensembles for different d, m, k triples. The horizontal orange plane indicates the (average) accuracy of the DNN ensemble. The pseudosaccade ensembles outperforms the increased Top-1 accuracy baseline of 75.78% by 0.3%, and the accuracy of the best performing classifier Resnet-152 by 3.7%. We conjecture that further, possibly minor, improvements in accuracy may be possible using a more careful approach to learn the weighting function. 

4. CONCLUSIONS AND FUTURE WORK:

We demonstrated that using a very simple, and computationally cheap, 'pseudosaccade' ensemble learning approach can improve the image classification performance of DNNs. This improvement is small but statistically significant at the 5% level and requires no complicated training or optimization, simply selecting two integer parameters d and m. In our experiments setting d ≥ 350 and m ≥ 5 worked well, with the approach much more sensitive to over-small values of d than of m. Following a careful analysis of the sources of error in our classification problem, we showed that these improvements also propagate to a weighted ensemble of pseudosaccade versions of (offthe-shelf) DNNs. An open problem is whether a (simple, or low overhead) non-uniform sampling scheme for constructing pseudosaccade data exists that could improve performance further, possibly mediated by a scene-dependent prior. Human visual processing suggests that such a scheme should be at least a possibility. We are examining non-uniform sampling schemes such as stratified sampling, and also techniques such as seam-carving, with a view to progress in this direction. 

:

http://www.image-net.org/challenges/LSVRC/

For example, while it is known that the VC dimension of a DNN is upper-bounded by the number of nodes in the network(Anthony & Bartlett, 2009), it is not known why DNNs dramatically outperform 'wide' neural networks with the same number of nodes but fewer hidden layers.

Figures and Tables:

Figure 1: Two images incorrectly classified by Alexnet in their original form (left-hand column), but correctly classified in pseudosaccade form (right-hand column).

Figure 2: Accuracy vs ensemble size and projection dimension. Reference plane shows the accuracy for the baseline classifier

Figure 3: Top-1, top-3 and top-5 accuracy of the "ensemble of ensembles" vs ensemble size and projection dimension

We used the winners of ILSVR Challenge from 2012, 2014 and 2015 namely Alexnet(Krizhevsky et al., 2012), GoogLeNet(Szegedy et al., 2015), and ResNet-50 and ResNet-152(He et al., 2016)

Table content: <table><tr><td>summarizes the main characteristics of this dataset.</td></tr></table>

Properties of the Imagenet validation dataset

Table content: <table><tr><td/><td>Min</td><td>Mean</td><td>Max</td><td/></tr><tr><td colspan="2">Image Count</td><td>50000</td><td/><td/></tr><tr><td colspan="2">Label Count</td><td>1000</td><td/><td/></tr><tr><td colspan="2">Fine-Grained Labels</td><td>120</td><td/><td/></tr><tr><td>Height</td><td>56</td><td>430.25</td><td>5005</td><td/></tr><tr><td>Width</td><td>54</td><td>490.37</td><td>4288</td><td/></tr><tr><td>Size</td><td colspan="3">3456 231320 18248230</td><td/></tr><tr><td colspan="4">Table 2: Summary of the DNN classifiers</td><td/></tr><tr><td/><td>AlexNet</td><td>GoogLeNet</td><td colspan="2">ResNet-50 ResNet-152</td></tr><tr><td>Architecture</td><td>CNN</td><td colspan="3">LeNet-Inception Residual Neural Network</td></tr><tr><td># Convolution Layers</td><td>5</td><td>57</td><td>50</td><td>152</td></tr><tr><td># Fully Connected Layers</td><td>3</td><td>7</td><td>1</td><td>1</td></tr><tr><td># Parameters</td><td>61 M</td><td>7M</td><td>25.6M</td><td>60.3M</td></tr><tr><td># Multiply and Accumulates</td><td>724M</td><td>1.43G</td><td>3.9G</td><td>11.3G</td></tr><tr><td>Regularization</td><td>Batch Normalization</td><td>Local Response Normalization</td><td colspan="2">Batch Normalization</td></tr><tr><td>Image Resizing</td><td>bicubic scaling (227x227)</td><td colspan="3">bilinear scaling (224x224)</td></tr><tr><td>Top-1 accuracy</td><td>54.70%</td><td>65.46%</td><td>70.39%</td><td>72.45%</td></tr><tr><td>Top-3 accuracy</td><td>71.68%</td><td>82.22%</td><td>85.55%</td><td>87.05%</td></tr><tr><td>Top-5 accuracy</td><td>77.56%</td><td>86.93%</td><td>89.66%</td><td>90.66%</td></tr><tr><td>3.4 RESULTS</td><td/><td/><td/><td/></tr></table>

Ensemble classifier accuracy for ensemble size m = 5 and projection dimensions d ∈ {450, 410, 350, 310, 250}, with the standard deviation from a sample of 50 ensembles. Values with '*' exceeded the top-k accuracies of the baseline classifiers by at least 2 standard deviations

Table content: <table><tr><td/><td/><td colspan="2">Alexnet (%)</td><td colspan="2">GoogLeNet (%)</td><td>ResNet-50 (%)</td><td>ResNet-152 (%)</td></tr><tr><td colspan="2">Projection Dimensions</td><td>Mean</td><td colspan="2">Std Dev Mean</td><td colspan="2">Std Dev Mean</td><td>Std Dev Mean</td><td>Std Dev</td></tr><tr><td/><td colspan="6">Top-1 55.255 0.044 * 65.918 0.051 * 70.726 0.048 * 72.826 0.043 *</td></tr><tr><td>450</td><td colspan="6">Top-3 72.235 0.033 * 82.526 0.040 * 85.743 0.037 * 87.289 0.050 *</td></tr><tr><td/><td colspan="6">Top-5 77.996 0.038 * 87.207 0.038 * 89.830 0.049 * 90.901 0.032 *</td></tr><tr><td/><td colspan="6">Top-1 55.416 0.044 * 65.910 0.047 * 70.629 0.075 * 72.759 0.065 *</td></tr><tr><td>410</td><td colspan="6">Top-3 72.342 0.041 * 82.490 0.040 * 85.730 0.051 * 87.264 0.043 *</td></tr><tr><td/><td colspan="6">Top-5 78.092 0.038 * 87.194 0.036 * 89.771 0.057</td><td>90.860 0.038 *</td></tr><tr><td/><td colspan="5">Top-1 55.520 0.051 * 65.512 0.060</td><td>69.901 0.074</td><td>72.308 0.047</td></tr><tr><td>350</td><td colspan="5">Top-3 72.294 0.056 * 82.153 0.050</td><td>85.250 0.069</td><td>86.955 0.043</td></tr><tr><td/><td colspan="5">Top-5 78.061 0.045 * 86.918 0.055</td><td>89.307 0.067</td><td>90.613 0.044</td></tr><tr><td/><td colspan="5">Top-1 55.038 0.074 * 64.596 0.068</td><td>68.793 0.103</td><td>71.269 0.064</td></tr><tr><td>310</td><td colspan="5">Top-3 71.989 0.052 * 81.429 0.047</td><td>84.286 0.080</td><td>86.253 0.047</td></tr><tr><td/><td colspan="5">Top-5 77.738 0.057 * 86.261 0.051</td><td>88.512 0.069</td><td>90.072 0.037</td></tr><tr><td/><td colspan="3">Top-1 53.299 0.076</td><td colspan="2">61.188 0.071</td><td>65.511 0.158</td><td>68.588 0.171</td></tr><tr><td>250</td><td colspan="3">Top-3 70.487 0.046</td><td colspan="2">78.555 0.063</td><td>81.596 0.102</td><td>84.113 0.112</td></tr><tr><td/><td colspan="3">Top-5 76.398 0.066</td><td colspan="2">83.878 0.057</td><td>86.247 0.068</td><td>88.283 0.076</td></tr></table>

In table 6, we see that -based on this summary statistic -the accuracy of the saccade classifiers is highly correlated with that of the corresponding baseline classifier, indicating to us that the classifier performance is not greatly reduced by pseudosaccade projection. Table7meanwhile shows that although the saccade classifier errors are correlated with one another, this is to a lesser degree than to the baseline classifiers. These facts suggest that there might be little to gain from combining the Table 5: 2x2 table of the relationship between the classifiers D i and D j D j Correct D j Wrong D

Table content: <table><tr><td>standard deviations</td></tr></table>

The average classifier correlation ρ base,saccade between baseline classifier and saccade

Table content: <table><tr><td>classifiers</td><td/><td/><td/><td/></tr><tr><td>Saccade Dimensions</td><td colspan="4">AlexNet GoogLeNet Resnet-50 Resnet-152</td></tr><tr><td>450</td><td>0.8959</td><td>0.8952</td><td>0.8916</td><td>0.8949</td></tr><tr><td>430</td><td>0.8851</td><td>0.882</td><td>0.877</td><td>0.8811</td></tr><tr><td>410</td><td>0.8753</td><td>0.8694</td><td>0.8615</td><td>0.8681</td></tr><tr><td>390</td><td>0.865</td><td>0.8552</td><td>0.8479</td><td>0.8552</td></tr><tr><td>370</td><td>0.8524</td><td>0.8368</td><td>0.8301</td><td>0.8377</td></tr><tr><td>350</td><td>0.8375</td><td>0.8164</td><td>0.8078</td><td>0.8164</td></tr><tr><td>330</td><td>0.8221</td><td>0.7937</td><td>0.7853</td><td>0.7943</td></tr><tr><td>310</td><td>0.8039</td><td>0.7683</td><td>0.7601</td><td>0.7715</td></tr><tr><td>290</td><td>0.785</td><td>0.7418</td><td>0.7335</td><td>0.7449</td></tr><tr><td>270</td><td>0.764</td><td>0.7119</td><td>0.7068</td><td>0.7174</td></tr><tr><td>250</td><td>0.742</td><td>0.678</td><td>0.676</td><td>0.6889</td></tr><tr><td>200</td><td>0.6727</td><td>0.5729</td><td>0.5806</td><td>0.5977</td></tr><tr><td>150</td><td>0.5626</td><td>0.437</td><td>0.4563</td><td>0.4666</td></tr></table>

The average classifier correlation ρ saccade1,saccade2 between all pairs of saccade classifiers

Table content: <table><tr><td>Saccade Dimensions</td><td colspan="4">AlexNet GoogLeNet ResNet-50 ResNet-152</td></tr><tr><td>450</td><td>0.8847</td><td>0.8830</td><td>0.8809</td><td>0.8876</td></tr><tr><td>430</td><td>0.8744</td><td>0.8702</td><td>0.8682</td><td>0.8754</td></tr><tr><td>410</td><td>0.8657</td><td>0.8597</td><td>0.8562</td><td>0.8648</td></tr><tr><td>390</td><td>0.8564</td><td>0.8484</td><td>0.8456</td><td>0.8543</td></tr><tr><td>370</td><td>0.8448</td><td>0.8344</td><td>0.8337</td><td>0.8419</td></tr><tr><td>350</td><td>0.8320</td><td>0.8202</td><td>0.8196</td><td>0.8266</td></tr><tr><td>330</td><td>0.8196</td><td>0.8060</td><td>0.8059</td><td>0.8121</td></tr><tr><td>310</td><td>0.8077</td><td>0.7917</td><td>0.7938</td><td>0.7991</td></tr><tr><td>290</td><td>0.7956</td><td>0.7771</td><td>0.7792</td><td>0.7849</td></tr><tr><td>270</td><td>0.7835</td><td>0.7621</td><td/><td>0.7700</td></tr><tr><td>250</td><td>0.7717</td><td>0.7466</td><td>0.7528</td><td>0.7572</td></tr><tr><td>200</td><td>0.7369</td><td>0.7027</td><td>0.7138</td><td>0.7211</td></tr><tr><td>150</td><td>0.6942</td><td>0.6473</td><td>0.6666</td><td>0.6750</td></tr></table>

Top-k accuracies for the "ensemble of ensembles" for ensemble size m ∈ {1, 5, 10, 15},projection dimensions d ∈ {450, 410, 350, 310, 250}. Entries with '*' denotes statistically significant improvements over the baseline accuracies (top-1: 75.78%, top-3: 87.65%,

Table content: <table><tr><td colspan="2">top-5: 90.35%)</td><td/><td/><td/><td/></tr><tr><td/><td/><td>One Saccade</td><td>5 Saccades</td><td>10 Saccades</td><td>15 Saccades</td></tr><tr><td colspan="6">Projection Dimensions Mean Std Dev Mean Std Dev Mean Std Dev Mean Std Dev</td></tr><tr><td/><td>Top-1</td><td>75.73 0.12</td><td>76.08 0.09 *</td><td>76.16 0.05 *</td><td>76.17 0.05 *</td></tr><tr><td>450</td><td>Top-3</td><td>87.51 0.10</td><td>87.98 0.07 *</td><td>88.13 0.05 *</td><td>88.21 0.04 *</td></tr><tr><td/><td>Top-5</td><td>90.19 0.08</td><td>90.74 0.07 *</td><td>90.86 0.05 *</td><td>90.92 0.04 *</td></tr><tr><td/><td>Top-1</td><td>75.55 0.14</td><td>75.96 0.08 *</td><td>76.04 0.06 *</td><td>76.10 0.05 *</td></tr><tr><td>410</td><td>Top-3</td><td>87.38 0.11</td><td>87.99 0.07 *</td><td>88.15 0.05 *</td><td>88.22 0.03 *</td></tr><tr><td/><td>Top-5</td><td>90.12 0.08</td><td>90.76 0.07 *</td><td>90.91 0.06 *</td><td>90.97 0.05 *</td></tr><tr><td/><td>Top-1</td><td>74.65 0.15</td><td>75.36 0.11</td><td>75.51 0.08</td><td>75.57 0.06</td></tr><tr><td>350</td><td>Top-3</td><td>86.83 0.09</td><td>87.75 0.07</td><td>87.95 0.06 *</td><td>88.06 0.05 *</td></tr><tr><td/><td>Top-5</td><td>89.61 0.12</td><td>90.57 0.08 *</td><td>90.79 0.07 *</td><td>90.91 0.05 *</td></tr><tr><td/><td>Top-1</td><td>73.58 0.15</td><td>74.56 0.11</td><td>74.73 0.08</td><td>74.80 0.06</td></tr><tr><td>310</td><td>Top-3</td><td>85.91 0.12</td><td>87.06 0.09</td><td>87.31 0.07</td><td>87.46 0.05</td></tr><tr><td/><td>Top-5</td><td>88.92 0.13</td><td>90.13 0.10</td><td>90.50 0.07 *</td><td>90.67 0.06 *</td></tr><tr><td/><td>Top-1</td><td>70.34 0.19</td><td>71.78 0.11</td><td>71.96 0.11</td><td>72.01 0.09</td></tr><tr><td>250</td><td>Top-3</td><td>83.33 0.15</td><td>85.04 0.09</td><td>85.35 0.08</td><td>85.48 0.06</td></tr><tr><td/><td>Top-5</td><td>86.65 0.17</td><td>88.44 0.11</td><td>88.89 0.09</td><td>89.08 0.06</td></tr></table>

Labels where ensemble method performed significantly better (≥ 10%) than the baseline Resnet-152 Imagenet classifier. Number of instances for which the given label was returned by classifier in brackets.

Table content: <table><tr><td>true label</td><td>base label</td><td>saccade label</td><td>ensemble label</td></tr><tr><td/><td>dungeness crab (4)</td><td>rock crab (30)</td><td>dungeness crab (3)</td></tr><tr><td>rock crab</td><td>rock crab (28)</td><td>crayfish (3)</td><td>rock crab (33)</td></tr><tr><td/><td>hermit crab (4)</td><td>hermit crab (3)</td><td>hermit crab (4)</td></tr><tr><td colspan="4">bedlington terrier bedlington terrier (28) bedlington terrier (42) bedlington terrier (43)</td></tr><tr><td/><td>bloodhound (3)</td><td/><td/></tr><tr><td>labrador retriever</td><td>saluki (3) golden retriever (3)</td><td>saluki (3) labrador retriever (36)</td><td>labrador retriever (39)</td></tr><tr><td/><td>labrador retriever (34)</td><td/><td/></tr><tr><td>bell cote</td><td>bell cote (26) chime (3) church (11) monastry (4)</td><td>bell cote (31) church (8) monastry (4)</td><td>bell cote (31) church (9) monastry (4)</td></tr><tr><td>bow</td><td>bow (30)</td><td>bow (32)</td><td>bow (35)</td></tr><tr><td>necklace</td><td>chain (3) necklace (40)</td><td>necklace (46)</td><td>necklace (46)</td></tr><tr><td/><td>pitcher (19)</td><td>pitcher (25)</td><td>pitcher (24)</td></tr><tr><td>pitcher</td><td>vase (4)</td><td>vase(4)</td><td>vase (3)</td></tr><tr><td/><td>water jug (6)</td><td>water jug (5)</td><td>water jug (8)</td></tr><tr><td>plastic bag</td><td>plastic bag (24)</td><td>plastic bag (26)</td><td>plastic bag (29)</td></tr><tr><td>hen of the wood</td><td>coral fungus (3) hen of the wood (35)</td><td>coral fungus (4) hen of the wood (36)</td><td>hen of the wood (40)</td></tr></table>

Labels where Resnet-152 Imagenet classifier achieved ≤ 20% recall. Number of instances for which the given label was returned by classifier in brackets.

Table content: <table><tr><td>true label</td><td>base label</td><td>saccade label</td><td>ensemble label</td></tr><tr><td>cassette player</td><td>cassette player (10) cd player (4) radio (3) tape player (22)</td><td>cassette player (9) cd player (4) tape player (23)</td><td>cassette player (9) cd player (4) radio (3) tape player (21)</td></tr><tr><td>crt screen</td><td>desk (6) desktop computer (8) monitor (4) crt screen (8) television (8)</td><td>desk (6) desktop computer (8) monitor (5) crt screen (9) television (9)</td><td>desk (5) desktop computer (8) laptop computer (3) monitor (6) crt screen (9) television (9)</td></tr><tr><td>sunglass</td><td>sunglass (11) sunglasses (19)</td><td>sunglass (10) sunglasses (19)</td><td>sunglass (11) sunglasses (16)</td></tr></table>

Labels where ensemble method performed significantly worse (≥ 10%) than the baseline Resnet-152 Imagenet classifier. Number of instances for which the given label was returned by classifier in brackets.

Table content: <table><tr><td>true label</td><td>base label</td><td>saccade label</td><td>ensemble label</td></tr><tr><td>mantis</td><td>walking stick (3) mantis (37)</td><td>walking stick (5) mantis (31)</td><td>walking stick (5) mantis (32)</td></tr><tr><td>abaya</td><td>abaya (41)</td><td>abaya (37) cloak (3)</td><td>abaya (36) cloak (3)</td></tr><tr><td>perfume</td><td>perfume (40)</td><td>perfume (35)</td><td>perfume (34)</td></tr><tr><td>wok</td><td>wok (28) hot pot (10)</td><td>dutch oven (4) frying pan (3) wok (22) hot pot (9)</td><td>frying pan (4) wok (23) hot pot (11)</td></tr></table>

Peer review (iclr2019):
 This paper proposes a data ensemble method for image classification: sub sample an image, classify each sub sample, and vote those sub samples to get the final decision. Questions: 1. The validation accuracy of ResNet is much lower than that reported in the original ResNet paper, https://arxiv.org/pdf/1512.03385.pdf. For example, the top-1/5 accuracy of RestNet-50 is 79+/94+, which is only about 70+/89+% in this paper. Similarly, there is a big gap between the results of ResNet-152 reported in this paper and the original ResNet paper. Maybe I misunderstand something; otherwise, the results in this paper are not reliable. 2. This work does not compare with any other data augmentation methods for testing, e.g., the widely used 10-crop test [ref1, ref2]: “At test time, the network makes a prediction by extracting five 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.” 3. A minor issue is the practical value of this work. If one can afford the computational cost of data ensemble test, why not train m (as the data ensemble in this paper) models and ensemble them given that ensemble of models usually brings more accuracy improvement? Note that the training of multiple models is conducted offline and the ensemble of models is of the same computation cost compared with the method in this paper. [ref1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. [ref2] He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.