Laconic Image Classification: Human vs. Machine Performance
Authors: Javier Carrasco
Aidan Hogan
Jorge Pérez
Venue:  (2020)
Keywords: minimal images
entropy
human vs. machine performance

Abstract: We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is to minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given a classifier and a test image, we compute an approximate minimal-entropy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers a unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) on classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in experiments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise to reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting a texture bias in the ILSVRC-trained models used. We also find, in the evaluated setting, that humans classify the minimal-entropy positive images of machine models with higher precision than machines classify those of humans.

Under review as a conference paper at ICLR 2020
LACONIC IMAGE CLASSIFICATION: HUMAN VS. MACHINE PERFORMANCE
Anonymous authors Paper under double-blind review
ABSTRACT
We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given classifier and test image, we compute an approximate minimmll-ntrroy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in periments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise
reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting texture bias in the ILSVRC-trained models used. We also find, the evaluated setting, that humans classify the minimal-entropp positive images of machine models with higher precision than machines classify those of humans.
INTRODUCTION
Deep neural networks now surpass human-level performance on variety of specific tasks and met- rics relating to visual recognition. In a widely-used yardstick for human-level performance, Rus- sakovsky al, 2015) estimated that an expert human with prior training can achieve a top-5 clas- sification error rate of 5.1% on dataset of 1, 1,500 ILSVRC images and 1,000 target classes. Shortly after, He et al. (2015) surpassed numanllevel performance on the same task achieving 4.9% top-5 er- ror with PReLU-net. Later works would further reduce this error rate, including ResNet (3.6%) (He ct al. 2016), Trimps-Sousher (3.0% (Shao et al., 2016). SeNet (2.3%) (Hu et al,, 2019), etc., with contemporary state-of-the-art models more than halving estimated human error for this specific task.
Though such results represent landmark advances, by focusing on classification errors alone, they do not reveal the full story of relative machine performance for image classification. Works adversarial examples (Dalvi et af.) 2004) [Szegedy et al., 2014]] Nguyen et al.) (0055), for instance, establish that human and machine perception diverges greatly for specifically constructed images. Other works have presented bespoke experiments comparing human and machine performance be- yond classification errors, presenting evidence for lack of robustness in the presence of noisy (Rus-) sakovsky et al.) 2015; Dodge& & Karam 2017 2019) or incomplete information (Ullman et al. 2016 2018 Srivastava et al. 2019). sensitivity to spa- tial Falzi & (rossard((011)) Hénaff & Simoncelli 2016) : Xiao et al. (201 8); Engstrom et al. (2019) or colour colour Hosseini Poovendran (2018): Hosseini et al. (2018) transformations, lack of generali- sation (Geirhos et all.) bias towards texture (Geirhos et al. 2019), etc., in evaluated models. By applying specific transformations on test images prior to classification, these latter works provide insights into the specific differences in the types of information that humans and machines rely to perform adequately at this task.
Wick et al.] [016 2016] Linsley al. 20
These latter recent works suggest the need for such issues: a framework within which the performance of classifiers be they human, machine or other can be compared and understood, allowing to quantify, in more fine-grained manner, the
citrmattion-hheoretie framework that generalises

---

Under review as a conference paper at ICLR 2020
type of information in the input which given classifier depends. While previous works address individual or multiple types of information reduction on input images in isolation, a more general framework should allow to combine and compare different types of reduction test inputs.
In this paper, we propose such a framework, based on the principle of computing and analysing minimal entropy positive imputs: inputs with minimal information with respect to yielding correct classification results. The notion of entropy intuitively generalises and allows to compare the rela- tive effects of different reductions on inputs and their combinations on classifier performance; such reductions may include, for example, cropping, downsampling and quantisation. The goal in this framework thus shifts from precise classification to laconic classification: providing clas- sifier that minimises the entropy of input(s) required for correct classification. Being based on a continuous notion of entropy rather than a discrete notion of correct/incorrect, this latter goal thus presents novel challenge heyond minimising classification error-a goal for which state-of-the-ard approaches are close to achieving perfect results on datasets like ILSVRC. In fact, existing datasets- such ILSRVC- can be straightforwardly leveraged for evaluating classifiers under this new goal. Models performing more laconic classification (we currently conjecture) should likewise perform more robustly in practical settings involving incomplete noisy information capture.
Though the framework we propose can be applied to any classifier for any classification task. herein first instantiate the framework the aforementioned problem of image classification. We con- sider three general operations for reducing the entropy of test images: crop, resolution reduction (downsampling) and colour reduction (quantisation). We then propose two methods for finding the minimal entropy positive images under these reductions for two different types of classifier.
Given a pre-trained machine classifier where classification can be separated from learning an input test image, and a set of reduction operations, we use a known search algorithm (Powell) 1964) and apply the given reduction functions to the input image to find (under certain assumptions) the lowest entropy image that the model classifies correctly such that applying any further reduction of entropy leads to incorrect classification. We apply the aforementioned framework find the mini- mal entropy images from a sample ILSVRC test-set for wfae-of-thhear deep neural-network (DNN) models (GoogLeNet, SqueezeNet, ResNet50 and SeNetResNet50), with respect to the three afore- mentioned reduction operations and their combination. Our experiments show that minimal entropy images are considerably smaller than original images for DNNs; for example, we find that with only 2-6% of the information content of the original test-images (on average) the best performing machine model can still produce a correct classification of the reduced image.
We then consider also human classifiers in order to compare their ability to perform laconic classifi- cation with DNN models; applying our framework for humans was not trivial since learning cannot be separated from classification (we cannot start with the full input test-image and reduce it since the human will remember the image) and an automatic optimization algorithm was not possible. To cope with this we designed an experiment reversing the optimization goal: starting from void image, the human evaluator may add information incrementally until they believe that confident classification of the displayed image is possible. We apply this framework with more than 500 human subjects through an online interface. Our results show that the minimal-entros positive images for humans are considerably smaller in the case of colour and resolution reductions 153-62% the size of the corresponding size for the best performing machine model).
Finally we crass-classffied the munimal-entrope positive images among DNNs and humans: given classifier A and B, we presented A's minimal-entropy positive images to B and vice versa, com- puting classification precision. We show that the precision of human classifiers on DNN minimal- entropy positive images was considerably better (0.74 precision in the worst case) than the corre- sponding results for cross-classfficttion among DNN models (0.02 precision in the worst case for the best model). Furthermore, we find that the DNN models on human images give precisions of 0.03-0.43, depending on the reductions used for the images and the model used for classification.
Our results provide insights into how different classifiers both machine and human rely dif- ferently on different types of information, providing further evidence to support, for example, prior claims of the lack of robustness incomplete information relative humans in such models (Dodge) & Karam 2017). a bias towards texture in -Ltaind models (Geirhos et al., 2019). and so forth. As a more general conclusion, we show that humans are capable of performing (much) more
2

---

Under review as a conference paper at ICLR 2020
laconic classification in the evaluated setting than satte-of-theann art machine classifiers. We conclude with open challenges regarding the laconic classification of images using DNNs.
RELATED WORKS
number of previous works have studied the robustness of image classification for DNNs and humans. [Ullman et al. (2016) introduced the notion of "minimal image" as the smallest region of an image that is still recognisable by a human. They show that a small change in these minimal images can have drastic effect on recognition. Moreover. [[ilman et al. (206)) show that DNNs are unable to accurately recognise human minimal images. To compute the minimal images, they started from the complete image and then iteratively cropped it showing the new cropped image to a different human subject every time. The process stopped when the accuracy of recognizing every crop of the current patch dropped from a threshold. Given the difficulty of computing minimal regions recognisable by humans, their experiments were limited to 10 images (Ullman et al. 2016).
Srivastava al. (2199) extended the previous work by showing that the sharp drop in accuracy for minimal images can also be observed in DNNs. They defined the notion of "fragile recognition image" as a region of an image for which small change in size produces a considerable change in accuracy of recognition by DNNs. (Srivastava et al. (2019) showed that fragile recognition images are abundant and can occur for several different sizes. The work by Ullman et al. (2016) and Srivastaval et al. (2019) can be modelled inside our minimal-entropy framework, which considers not only regions of images (i.e., crop), but also reductions resolution, colour, and combinations thereof.
Another line of related works study the robustness of DNN image classification with respect dis-
tortions to the input image (Hendrycks & Dietterich] 2019 Geirhos et al. 2018; 2019 Dodge &
Karam 2017; 2019; Fawzi & Frossard 2015 Hénaff & Simoncelli 2016; Xiao et al. 2018 En-
gstrom et al. 2019 Hosseini & Poovendran, 2018 Hosseini et al., 2018). Dodge & Karam (0077)
2019) also study how humans react to these distortions showing that the performance of DNNs is
much lower than human performance on distorted images. Hendrycks & Dietterich (2019) con-
structed benchmark for image classifier robustness based on several different image distortions. Although all these distortions are designed to make the image recognition task more difficult, not all of them reduce the information the image. For example, two transformations considered by Geirhos et al. (2118) are image rotation and colour inversion that do not alter the volume of informa- tion. Other transformations considered by Geiroos et al. (2018) such as Fidolon and additive noise may even increase the image file size. In contrast we currently only consider transformations that reduce the information content of the input image file.
APPROXIMATING HINIMAI-EENRONT POSITIVE IMAGES
We now discuss the framework we use for evaluating the laconic classification of images. We first discuss the entropy measure and the reductions applied to images in this work. We then discuss the computation of approximate minimal-entropy positive images (MEPIs) for DNNs and for humans.
3.1 ENTROPY MEASURE
Our goal is to find the minimal-entropy positive images (MEPIs) for various classifiers based on various forms of entropy reduction. Although there exist numerous proposed entropy measures (Wul et al., (2013: Larkin 2016), the analysis of entropy for images is not straightforward due to their dimensionality. particularly in the case of multi-channel images where measures of entropy are further complicated by the conditional and joint entropy that must be measured across the channels. We mitigate this issue by adopting the compressed size of an image using a lossless image encoder as estimation of image entropy (Larkin 2016). For these purposes we propose to use the Portable Network Graphics (PNG) encoder proposed as a successor for the Graphics Interchange Format (GIF) which combines an initial predictive filtering step to improve compressibility, followed by a DEFLATE compression. The size PNG images has been shown to outperform more complex measures of entropy (Larkin 2016), while being widely implemented in variety of libraries.
3

---

Under review as a conference paper at ICLR 2020
3.2 ENTROPY REDUCTION
Given an m x n. matrix A, we consider the following forms of entropy reduction:
Alq(s)) is defined as the (nearest-clement) m quantised matrix of A with quantisation factor x, max(A) )² for all d' in Alq(s)-
such that d'y :- arg me
AJD(o) is defined as the x (r $ m, n) downsampled matrix of A with scaling factor such that 0< o s 1, [om]= =r, |on]== =.
A.s(3.3))))) is defined as the (contiguous) pxq submatrix (p S m, q S m) computed from the indices at, B, 7, 8 (ax < k. 3 = a p. 7 < n, 0 = y () such that (l = A[a, a 1,... 3 1:7:7 + 1...., d 1]; in other words, removes rows 1,...,0- a 1, m and columns 1 1..-.-7- o,,,..., n from A. We use Alsp(a) Als(amm.1.n) A Asse AL,(11.,1.m)) Alsv(7) = A.s(1.m....)) and ( == (L.(m.m,111))) to denote removing rows/columns (only) from the left, right, top and bottom of the image, respectively.
In the case of images, these reductions correspond, respectively, to colour reduction (parameter: k), resolution reduction (parameter: σ), and crop (parameters: a, y, 6). We also consider their combination (parameters: a, B, 7, Ks, rr) We thus have six parameters by which to reduce entropy. We adapt these reductions to multi-channel images in the natural way. In the case of crop, we apply the reduction to all channels separately: however, based on initial experiments with DNNs, rather than remove the rows and columns of the image's 's channels, we rather replace them with a constant neutral value, which allowed further entropy reduction in positive images by avoiding distortions once images are internally rescaled by the network (i.e., images with narrow crops being "stretched out"). In the case of colour reduction, the nearest quantisation values are computed in the multi- channel case based on Euclidean distance; we further normalise the output values to fill the colour space after the quantisation, choosing equidistant points. In the case of downsampling, the reduction applied to each channel and maintains the same aspect ratio for square images.
In Appendix A we provide examples of the four types of reduction image of dog.
3.3 MINIMAL-ENTROPY POSITIVE IMAGES
Given an input image, set of entropy reductions, and classifier, define naive minimal- entropy positive image to be the smallest (PNG) image derived from the input image using the given reductions for which the classifier provides a correct classification (i.e., the top predicted class is correct). To compute such images, we apply reductions in discrete steps. More formally, given a matrix A and entropy reduction as (c)(c) such that t= min | | A..(z)) # A}: i.e., is the smallest value that modifies A under AJ.(). For a, 3,7, 6 and k. the step size is one (e 1). In the case of ar, we define e as the smallest value such that [om] [ ccmm] or [on] [(a- - c)n] (i.e., such that the image is rescaled).
(Q, D.Stt SD, S4, $v,S4)
define an atomic reduction step
This definition may lead to undesirable results: consider, for example, the case a classifier making predictions based on the individual (nondescript) pixels of the input image. With varying probability -depending on the number of pixels, images, and classes, the form of classification, etc. a classifier may for some pixel "guess" the correct class, with that pixel potentially becoming naive minimal- entropy positive image. To avoid such cases, we add continuity condition: a minimal-enten -entrppy positive image (MEP!) is then defined to be the smallest image derived from the input image using the given reductions for which the classifier provides correct classification, and for which there is a continuous path of atomic reduction steps from the input image giving correct classifications.
3.4 APPROXIMATING MEPIS FOR DNNS
In the case of DNNs, for a given image and applicable reductions, we apply search for the value(s) of the parameter(s) that correspond to the MEPI of the image. Our general method begins with the input image, incrementally applying atomic reduction steps while the prediction of the classifier remains correct, backtracking in the case of an incorrect prediction. While relatively simple to implement for reductions with a single parameter, in the case of crop or combined reductions, we have multiple parameters with respect to which must search; for example, in the case of crop, for
4

---

Under review as a conference paper at ICLR 2020
an x image, the number of images to check in the worst-case is potentially (²)()) i.e., more than 274 billion contiguous sub-images for a 1024 1024 input image (and this considering crop alone without combination with resolution colour). Approximation is thus sought.
First, our assumption of continuity helps us to prune the search space: if we reach a set of parameter values for which classification is correct but for which any further atomic reduction is incorrect, know we can rule out all further reductions from that point. We may still, however, encounter an infeasible search space when considering multiple parameters; improve performance, we choose to apply a greedy search algorithm: given that the function we wish to minimise is not differentiable but has a fixed number of inputs, we apply Powell's method (Powell 1964) to find local minimum.
3.5 APPROXIMATING MEPIS FOR HUMANS
Humans operate differently to DNNs and thus require specialised methods approximate their MEPIs. First, humans cannot separate classification from learning; hence we cannot show the full input image and apply reductions as the human will (of course) remember the full input image. Sec- ond, humans require (much) more time per classification; hence the search steps must be adjusted.
In particular, given A. rather compute a bottom-up approximation of MEPIS for humans. In what follows, given a set of entropy reductions R S {Q. D,S >, 54, 8v. SS}, we use Ax to denote the set of matrices recursively reachable from A by some sequence of (atomic) reduction steps from R, including A itself. Given R, we can then define () as the inverse of +.(.) under Ar, such that if A' e AR, then A')-(=) A" if and only if A" é AR and A"t.(=) = A' for any value x well-defined for Now, starting from an initial image Ao é AAR, each step i 2 0 the human can then either (1) if A; A, select an atomic step e R and increment the entropy of the image by a constant factor Ç such that A;+1 :- Ac).(c), or (2) select class for the image from list if they believe they can correctly classify the image. If the human guesses correctly, this is considered to be an approximate MEPI (for the image, parameters and human); if not, the image is discarded and a fresh image is considered. To mitigate fatigue, larger atomic steps are defined such that more than 20 are required to reach the input image along a given dimension; i.c., a constant step y is chosen such that A can be reached from Ao by applying (1()) 20 times for each e R. In the case of crop, starting from the central pixel (Ao := [1, [ I. we offer the option to increment all directions at once, while in the combined case, we offer the option to increment all parameters at once. This method makes it feasible to approximate MEPIs for humans, but it is important to note that with the limitations of coarser steps and always starting at the central pixel, the MEPIs approximated using this method are more likely to overestimate the entropy required for correct classification.
EXPERIMENTAL SETTING
We base experiments images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2112) (Russakovsky et al.) 2015). We select four DNN classifiers for evaluation (all trained on the ILSVRC2012 training set): SqueezeNet (landola al., 2016). GoogLeNet (Szegedy) let al.) 2015). ResNet50 (He et 2016) and SeN5tResNet50 (Hu et al. 2019); as discussed in the in- troduction, the latter two classifiers surpass human-level performance in terms of top-5 classification
error on a dataset of 1,500 ILSVRC images and 1,000 target classes.
As acknowledged by Russakovsky al. (2015), the 1,000 detailed classes of ILSVRC (e.g., COU- CAL, SEALYHAM TERRIER) require extensive previous training for humans to perform adequately classification, not only visually distinguish the objects, but also to be able to retrieve their label from the 1,000 possible labels. We thus opt to greatly simplify the classes, selecting the following twenty: BEAR, BIRD, CAT, DOG, FISH, FLOWER, FOX, FRUIT, FUNGUS, HIPPOPOTAMUS, INSECT, LION, MONKEY, REPTILE, SHARK, SPIDER, TIGER, VEGETABLE, VEHICLE, WOLF. We select these classes as they should be generally recognisable to humans without prior training; furthermore, we select mostly plants and animals to provide a more challenging classification task, with visually sim- ilar classes, such as LION/TIGER, FRUIT/VEGETABLE, DOG/WOLF, INSECT/SPIDER, etc., providing non-trivial cases visually distinguish. The results of DNN models then mapped hierarchically to these higher-level classes (e.g., COUCAL + BIRD, TEALYHAM TERRIER + DOG). We sample 15 images for each of the 20 classes from the VLSSVRC2012 test set for experiments.
5

---

Under review as a conference paper at ICLR 2020
0.8
Entropy ratio
0.6
0.4
0.2
0
COLOUR
RESOLUTION
CROP
COMBINED
Figure 1: Box-plots of mean entropy ratio for DNN and human MEPIs across classes
5 EXPERIMENTAL RESULTS
With our experimental results, we address two main questions: (1) How does the entropy required by DNN and human classifiers compare? (2) How do the classifiers perform in terms of precision for each others' MEPIs? We address these two questions the following subsections, along with other underlying questions relating to how different image reductions affect individual models, how the goals of laconic and accurate classification correlation, etc.
5.1 ENTROPY RATIO MEPIS
For the 20 15 300 test images, first compute the top-down MEPIs for the four DNN classifiers using the method described in Section 3.4 For humans, following the bottom-up method described in Section 3.5 we provide web interface that starting with void image allows human in each step to either increment the image by the given parameters, or select the class for the currently displayed image. In order achieve many responses, the interface was shared on university forum as well as social media. The options and instructions were presented in Spanish, corresponding the native language of the country in which the university is based. Screenshots of the interface
are given in Appendix B In total, 423 user sessions were logged; of the 1,722 responses obtained (average 4.07 per session), 1,340 (77.8%) resulted in a correct validation and thus a valid MEPI.
In Figure we present the entropy ratio for four different settings across five different classifiers; entropy ratio defined here as the ratio of (PNG-encoded) size of the original input image versus the size of the extracted MEPI. We take the average entropy ratio for the images of each of the twenty classes. Figure then presents the box-plots displaying the 1 (min), 25th 50th (median), th and th0th (max) percentiles with across the different classes; to illustrate with an example, in the case of SqueezeNet, considering only the COLOUR experiment, the best class had mean entropy ratio of 0. (bottom whisker), the worst class had a mean entropy ratio of 0.24 (top whisker), etc. We present the DNN models in order of their reported performance for top-5 classification error on the ILSVRC dataset, with SqueezeNet having the highest such eтror and SeNetResNet50 having the lowest.
marked diamond for the mean ratio
From this figure we can draw some high-level observations about the DNN classifiers. First, for the DNN classifiers, the entropy ratio is lowest for the COMBINED experiment (as expected), which offers more avenues by which to reduce the entropy while maintaining a correct classification. Sec- ond, the results for DNN classifiers follow the same trend as for performance based on classification error; this suggests that there is a correlation between the goals of laconic classification and pre- cise classification. Third, we see that DNNs are most sensitive reductions resolution, which supports the hypothesis that DNNs trained on ILSVRC images are biased towards texture (Geirhos) al. 2019) (with RESOLUTION being the parameter that most affects the ability distinguish tex- ture). We also see considerable variation across classes, particularly for RESOLUTION; for space reasons, provide the detailed results of mean entropy ratio by class in Appendix

---

Under review as a conference paper at ICLR 2020
SN
GN
RN
SRN
Hum
SN
GN
RN
SRN
Hum
SN
1.000
0.114
0.344
0.382
0.136
SN
1.000
0.221
0.158
0.084
0.039
GN
0.903
1.000
0.632
0.640
0.255
GN
0.760
1.000
0.275
0.121
0.030
RN
0.905
0.880
1.000
0.771
0.432
RN
0.588
0.806
1.000
0.314
0.152
SRN
0.919
0.888
0.765
1,000
0.421
SRN
0.950
0.933
0.840
1.000
0.255
Hum
0.923
0.911
0.891
0.914
1.000
Hum
0.920
0.907
0.918
0.864
1.000
(a) COLOUR
(b) RESOLUTION
SN
GN
RN
SRN
Hum
SN
GN
RN
SRN
Hum
SN
1.000
0.040
0.080
0.019
SN
1.000
0.010
0.027
0.008
0.068
GN
0.597
1.000
0.200
0.041
0:221
GN
0.460
1.000
0.187
0.026
0.002
RN
0.530
0.214
1.000
0.023
0.226
RN
0.490
0.193
1.000
0.057
0.232
SRN
0.753
0.572
0.637
1,000
0.424
SRN
0.867
0.418
0.484
1.000
0.348
Hum
0.867
0.816
0.852
0.700
1.000
Hum
0.847
0.809
0.533
0.740
1.000
(c) CROP
(d) COMBINED
Figure Heatmaps of classification precision on MEPIs for the four types of entropy reduc- tion; rows indicate the classifier, columns indicate the MEPIs classified: SN SqueezeNet;
GN = GoogLeNet; RN ResNet50; SRN SeResNe150; Hum Human
With respect to human classifiers, note that they are less sensitive to reductions in COLOUR and much less sensitive to reductions in RESOLUTION than all DNNs, and more sensitive to reductions in CROP than some state-of-the-an DNNs. We also see unusual result, whereby the reduction ratios for COMBINED are not lower than those for (e.g..) COLOUR; this suggests that the users may have struggled with the interface for the CROP and COMBINED experiments, where multiple options were provided to increment the entropy of the image (versus COLOUR and RESOLUTION, which only permitted increments in one direction). As such, the results for CROP and COMBINED in human classifiers leave ambiguity: are the relatively poor results of humans due in this case to greater sensitivity to such (multi-parametric) reductions, or because of difficulty using the more complex interface in these cases? We require further experiments to address this ambiguity.
5.2 CLASSIFICATION PRECISION FOR MEPIS DIFFERENT CLASSIFIERS
We now look at the precision of (top- classification across the MEPIs of models, again considering COLOUR, RESOLUTION, CROP and COMBINED. To gather results for human classification, we created second, simpler, online interface that presents the MEPI of a particular model under particular reduction and asks the human evaluator to select the class for that MEPI from the list of twenty possible classes. The interface was shared again on a university forum and through social media. We provide a screenshot of this interface in Appendix B We first ran a control group with 25 trusted users, which logged an aggregate precision of 0.875 with standard deviation of 0.061; in the open/online evaluation, we then filter user sessions more than two standard deviations from the control mean precision, giving a lower threshold of 0.753. The public evaluation then logged 531 valid user sessions according to the threshold, resulting in 11,88 valid classifications of machine MEPIs (equating to 26.2 classifications on average per session). A comparison between the different groups of humans users is provided in Appendix D
The results of the precision for cross-llasiffictiio of MEPIS are summarised in Figure 2 Cells shaded darker to indicate better performance. Darker rows indicate better classifier performance while darker columns indicate having MEPIs that are easier for other models to classify. First we observe that all classifiers correctly predict (as expected by definition) all of their own MEPIs.
With DNNs ordered by expected performance, we again see the general trend that fewer reported classification errors in VLSVRC again correlate with better precision in the classification of MEPIs,

---

Under review as a conference paper at ICLR 2020
with, for example, SeNetResNet50 (SRN) (SRN) having darker rows and lighter columns than other DNNs. We also see good cross-classifier performance for COLOUR and RESOLUTION: given that this are one dimensional reductions, the space of possible images is greatly reduced. On the other hand, DNN models struggle in cross-classificttion of the MEPIs under CROP and COMBINED: this is perhaps due to the larger search space for these reductions, but also indicates that these MEPIs for DNNs are characteristic of that particular DNN. In general, the MEPIs of SRN are notably more difficult for other models to classify (as could have been expected from Figure D.
Regarding human classification, we see that in general humans perform best of all classifiers in this experiment, being adaptable enough to classify the MEPIs under all reductions with relatively high precision. The most difficult MEPIs for human are in the CROP and COMBINED configurations for SeResNet50 (SRN), where a relatively high precision of 74-76% correct classifications still seen. With respect to the previously discussed ambiguity in the results of the previous sub-section, we can thus see that although CROP and COMBINED are more difficult cases for humans, issues with the more complex interface for these cases may explain the relatively poor results of humans in Figure D for CROP and COMBINED. Furthermore, we can see that DNNs often struggle to classify the human MEPIs, where the best precision reached was 42% correct classifications for SRN in he case of CROP; Appendix providess sample of three human MEPIs not correctly classified by any DNN model, illustrating some of the most difficult such cases for DNN models.
6 DISCUSSION
Based on our proposed frameworks for computing minimal-entropy positive images (MEPIs) for both machine and human classifiers, our first results were based on the relative entropy of such images for the different classifiers. From these results we can conclude that state-of-the- machine models are relatively sensitive (entropy-wise to reductions in resolution when compared with other forms of reduction based on cropping or colour; such observations appear to independently support previous results indicating a bias in texture for classifiers trained on the ILSVRC dataset (Geirhos) eet al., 2019). On the other hand, humans are relatively sensitive to the cropping of images, but are much less sensitive than machine classifiers to reductions in resolution; this tends to suggest that humans rely more on form and context rather than textures of small regions of the image. In our second set of results, we performed cross- closssllasifiiatiinn of the MEPIs for different models; we saw that humans greatly outperform machine models for classifying the MEPIs of other models, suggesting a generally more robust aptitude in humans for the laconic classification of images. Of particular interest is that there appears to be a strong correlation between the tasks of laconic and accurate classification (per Figure I which suggests that the former may be new take on the latter: working on laconic classification suggests a range of ceeemattiontthhoreee approaches that could be brought to bear in order to improve classifier accuracy.
Our work has a number of limitations that could be addressed for future work. While the method and interface used for computing human MEPIs in bottom-up fashion work well for simpler (i.e., one- dimensional) forms of image enhancement, the unexpected result in Figure I showing an increase in the size of MEPIs in the combined case suggest that the users had some difficulty using the interface for multi- mullii-dimenionall settings; given that we filter incorrect images from consideration, we speculate that the users often "overshoot" the MEPI in particular dimensions, finding it difficult to select the particular dimension that is most likely to help them correctly classify the image. Refinements of this idea would be interesting explore in the future. Furthermore, in order facilitate non-expert humans to participate, we selected high-level classes, where it would be of interest to try to repeat the results for lover-level classes (though again presenting challenges for the interface).
On the other hand, we have performed experiments for pre- trained, off-the-shelf DNN models. An interesting line of research would be to rather train models specifically for the task of laconic classifi- cation. Along these lines, one could consider computing MEPIs in the training set and feeding them (potentially recursively) back into the model; unlike the internal transformations applied in specific DNN models, such a framework for laconic learning could treat a particular model as a black-box. would further be interesting to explore how models trained in such a manner perform in more tra- ditional classification metrics error rates, precision, etc. the original input images, as well as whether or not they might help to address the observed lack of robustness of sfate-of-the t--hhh--rt DNNs in the presence of noisy (Russakovsky et al.] 2015) Dodge & Karam 2017 Hosseini et al. 2018
8

---

Under review as a conference paper at ICLR 2020
Dodge & Karam 2019)
incomplete information (Ullman et al., 2016 Wick et al. 2016 Linsley
et al. 2017 7 Ho- Phuoc 2018 Srivastava et al. 2019). . or their lack of generalisation Geirhos et al. 2018
or their bias towards texture (Geirhos al. 2019). Further exploration is needed.
More generally, we view laconic classification as a potentially fruitful avenue to explore: one that is relatively straightforward to implement, re-use available datasets, connects number of recent works, provides insights into the performance and robustness diverse classifiers, and one that may perhaps reveal new challenges and lead to new developments in image classification.
NOTE:
We plan to publish images and other materials online after double-blind review. Please also see appendices for further details.
REFERENCES
Nilesh N. Dalvi, Pedro M. Domingos, Mausam, Sumit K. Sanghai, and Deepak Verma. Adversarial classification. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 99-108. ACM, 2004.
Samuel Dodge and Lina Karam. Human and DNN Classification Performance Images With Quality Distortions: A Comparative Study. ACM Transactions on Applied Perception (TAP), 16 (2):7:1-7:17, 2019.
Samuel Dodge and Lina Karam. A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions. In International Conference Computer Communication and Networks, (ICCCN), pp. 1-7. IEEE, 2017.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the Landscape of Spatial Robustmess. In International Conference on Machine Learning (ICML), pp. 1802-1811. PMLR, 2019.
Alhussein Fawzi and Pascal Frossard. Manitest: Are classifiers really invariant? British Machine Vision Conference (BMVC), pp. 106.1-106.13 13. BMVA Press, 2015.
Robert Geirhos, Carlos R. Medina Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 7549-7561, 2018.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR). OpenReview.net, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Huvan-Level Performance on ImageNet Classification. In IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034. IEEE Computer Society, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Patfern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEFE Computer Society, 2016.
Olivier J. Hénaff and Eero P. Simoncelli. Geodesics of learned representations. In 4th International Conference on Learning Representations (ICLR), 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: /2261, 2019.
Tien Ho-Phuoc. CIFAR10 to compare visual recognition performance between deep neural networks and humans. arXiV: 181 1.72700 2018.
Hossein Hosseini and Radha Poovendran. Semantic Adversarial Examples. In IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPR), pp. 1614-1619, IEEE Computer Society, 2018

---

Under review as a conference paper at ICLR 2020
Hossein Hosseini, Baicen Xiao, Mayoore Jaiswal, and Radha Poovendran. Assessing Shape Bias Property of Convolutional Neural Networks. In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR), pp. 1923-1931. IEEE Computer Society, 2018.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-Excitatin Networks.
MtXIV:1799001007v 2019.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <IMB model size. CoRR, abs/1602.07360, 2016.
Kieran G. Larkin. Reflections Shannon Information: In search of natural information-entropyy for images. CoRR, abs/1609.011111 2016.
Drew Linsley, Sven Eberhardt, T. Sharma, P. Gupta, and Thomas Serre. What are the Visual Features Underlying Human Versus Machine Vision? IEEE International Conference on Computer Vision (ICCV) Workshops, pp. 2706-2714. 14.. IEEE Computer Society, 2017.
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-436. IEEE Computer Society, 2015.
Michael J. D. Powell. An efficient method for finding the minimum of a function of several variables without calculating derivatives. Computer Journal, 7(2):155-162, 1964.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei Li. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-222 2015.
Jie Shao, Xiaoteng Zhang, Zhengyan Ding, Yixin Zhao, Yanjun Chen, Jianying Zhou, Wenfei Wang, Lin Mei, and Chuanping Hu. Good practices for deep feature fusion. ECCV 2016 Talk, 2016.
Sanjana Srivastava, Guy Ben-Yosef, and Xavier Boix. Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Images. In International Conference on Learning Repre- sentations (ICLR). OpenReview.net, 2019.
Christian Szegedy, Wojeiech Zaremba, Iya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
Shimon Ullman, Liav Assif, Ethan Fetaya, and Daniel Harari. Atoms of recognition in human and computer vision. Proceedings of the National Academy of Sciences (PNAS), 113(10))2744-2749 2016.
Farahnaz Ahmed Wick, Michael I.. Wick, and Mare Pomplun. Filling in the details: Perceiving from low fidelity images. 1.Xi011604.04125, 2016.
Yue Wu, Yicong Zhou, George Saveriades, Sos S. Agaian, Joseph Noonan, and Premkumar Natarajan. Local Shannon entropy measure with statistical tests for image randomness. Inf. Sci., 222:323-342, 2013.
Chaowei Xiao, Jun Yan Zhu, Bo Li, Waren He, Mingyan Liu, and Dawn Song. Spatially Trans- formed Adversarial Examples. In International Conference on Learning Representations (ICLR). OpenReview.net, 2018.
10

---

Under review as a conference paper at ICLR 2020
A EXAMPLE OF REDUCTIONS AND MEPIS
In Figure 3 we present examples for the four reductions considered; these examples correspond to the MEPIs for an instance of DOG given by SeNetResNe150.
ORIGINAL
(b) COLOUR
(c) RESOLUTION
(d) CROP
(e) COMBINED
Figure 3: MEPIs for the four types of entropy reduction generated by SeNetResNer50
11

---

Under review as a conference paper at ICLR 2020
B SCREENSHOTS OF INTERFACES
We present screenshots the online interfaces used to collect human MEPIs and classifications. As discussed in the paper, the options and instructions were provided in Spanish as the target audience were students of university whose native language Spanish. 7 show screenshots
interfaces used to collect human MEPIs for COLOUR,, RESOLUTION, CROP and COMBINED. On the other hand, Figure shows screenshot of the interface to classify machine MEPIs.
Experimento: Clesificer Imégenes
Aumenta hitro que crees idendificar
Achrina que categoria perterece la imagan mosorada adivvnss puntos).
Selecciona la imagen..
imagen.
Colores
Puntaje adivinas correctamente: 295 Puntaje Acumulado: 110
Ave
Flar
Hipopdana
Hongo
Hortzlizz
Insecto
Lwón
Mino
Oso
Repti
Tibr
Tigre
Vetiiculo
Figure 4: Screenshot of interface for humans to specify COLOUR MEPIs
12

---

Under review as a conference paper at ICLR 2020
Experimente: Clasilicar imágenes
Numenra necossrrio para iöter lillz
Adiv'na a que categoria perterece la Imagen mostrnde adivinas incornetamento plandes 20 puntos).
Seleociona calager de
Revokuzain
Puntaje si adivinas correctamente: 300 Puntaje Acumulado:
Araha
Fruta
Gato
Hipopdtamo
Hongo
Insecto
Loón
Parro
Poz
Welicolo
Zorra
Figure 5: Screenshot of interface for humans to specify RESOLUTION MEPIs
Experimento: Clasificar Imágenes
que catlggrr pertanace la imagen mostrada incoreectaee the plerdes 20 puntos)..
necesanio para ideenificar to Imagen.
categoris de Imagen.
Zona Visiale
Puntaje si adivinas correctamente: 255
Araifa
Airiba
Puntaje
Absjj
boqueedda
Dato
Derrcct
Hloopdtamo
inecto
Ladn
Labo
Pemro
Deu
Reptl
Tiburón
Tigre
Vehriuul
Zerro
Figure 6: Screenshot of interface for humans to specify CROP MEPIs
13

---

Under review as a conference paper at ICLR 2020
Experimento: Clasificar Imágenes
Aumenta fitrr que crees
Acfivinw a que oxtegaria per partenwoe ia imagen mostrade arthvruus incornectamente plendes puntos).
Selecciona la categoria de la imagens
necosarto para identificari la Resclución
Puntaje adivinas correctamente: 250 Puntaje Acumulado:
Colores
Flor
Fruta
Gato
Hongo
Hortaltza
Donecha
insecto
Leõn
lodas Direaciones
Aumentar Fitros
Oso
Porro
Tiburón
Tigre
Vehiculo
Figure 7: Screenshot of interface for humans to specify COMBINED MEPIs
Experimento: Imágenes
Correctas: 5 Incorretas: 0| Acierto: mostadds
Adkina
Seeo kina la categooii lmnnages
Biceago
Ferm
Ber
Napii
T4xade
Tym
Figure 8: Screenshot interface for humans to classify machine MEPIs (example using COLOUR)
14

---

Under review as a conference paper at ICLR 2020
C
MEAN ENTROPY RATIO PER CLASS
In Table we present the mean entropy ratio of MEPIs per class.
Table 1: Mean entropy ratio per class for DNN and human MEPIs
bird
fish
Fruit
hippo
SqueezeNet
0.16
0.23
0.17
0.11
0.23
0:15
0.21
0.17
Geer
0.18
0.14
0.17
0.16
0.15
0.1
0.18
0.13
0.19
0.15
Resne:50
0.16
0.12
0.12
0.12
0.12
0.08
0.16
0.11
0.16
0.13
5e-Reme150
Humans
0.07
0.06
0.08
lion
monkey
reptile
spider
tiger
vegetable
vehicle
SqueezeNet
0.15
0.22
0.23
0.19
0.11
0.17
0.25
0:19
0.18
0.21
GoogLaNet
0.12
0.39
0.18
0.15
0.13
0.12
0,17
0:15
0.12
0.17
Resne:50
0.16
0.15
0.13
0.09
0.14
0.15
0.12
St-Resset50
0.12
Humans
0.07
0.07
0.08
0.08
0.05
0.06
0.08
0.09
0.08
0.03
RESDLUTTION
cat
dog.
fungus
hippo
SqueezeNet
0.5
0.51
0.64
0.47
0.57
0.36
0.57
0.43
0.39
0.46
GoogLeNeet
0.35
0.37
0.51
0.3
0.42
0.35
0.42
0.38
0.3
0.45
Resnet50
0.34
0.22
0.4
Humans
0.07
0.05
0.6
0.06
0.14
0.09
0.16
0.06
0.16
0.11
insect
Roon
monkey
reptile
shark
spider
figer 0.55
vegelablee 0:47
vehicle
SqueessNett
0.5
0.53
0.48
0.71
0.5
0.6
0.59
0.54
GoogLaNet
0.37
0.33
0.39
0.56
0.42
0.39
0.4
0.33
0.41
Rosne:50
0.35
So-Rosne150
0.13
Humans
0.09
0.05
0.11
0.14
0.01
0.14
0.1
0.15
0.07
0.11
Caor
cat
dog
fungus
hippo
SqueezeNet
0.2
0.18
0.16
0.12
0.33
0.19
0.1
0.25
0.31
0.25
GoogLaNo
0.12
0.12
0.12
0.09
0.22
0.16
0.1
0.12
0.22
0.16
0.12
0.16
Resmet50
0.08
0.06
0.14
0.13
0.08
0.05
0.12
Humans
0.18
0.16
0.13
0.19
0.23
0.14
0.16
0.14
0.21
0.18
Insect
llon
monkey
reptile
shark
spider
tiger
vegelable
vehicle
wolf
SqueezeNet
0.19
0.26
0.19
00.77
01.4
0.35
0.24
0.2
0.31
0.16
GoogLaNet
0.15
0.24
Resne:50
Se-Resee100
0.11
0.11
0.1
0.13
0.18
0.12
0.09
0.08
0.1
0.1
Humans
0.18
0.34
0.18
0.16
0.34
0.1
0.14
0.14
0.1
0.11
COMPINTD
hear
bird
cat
dog
fish
Bower
fox
fruit
fungus
hippo
SqueezeNet
0.06
0,07
0.66
0.014
03.09
0.04
0.04
0:177
0.12
0.08
GoogLaNet
0.05
0.94
0.05
0.03
0.08
0.05
0.03
0.05
0.08
0.06
Resnet50
0.05
Se Kesset50
0.04
0.04
004
0.02
0.06
0.04
0.04
0.02
0.05
0.03
Humans
0.12
0.07
0.09
0.05
0.15
0.17
0.09
0.1
0.25
0.11
insect
lion
monkey
reptile
shurk
spider
liger
veggelable
vehicle
wulf
SqueereNet
0.06
0.00
0.07
0.11
0.14
0.12
0.1
0.06
0.08
0.05
GoogLeNet
Resne:50
0.04
0.05
0.07
0.03
0.06
Se-Resnet50
0.04
0.04
0.04
0.06
0.06
0.04
0.02
0.03
0.03
0.04
Humans
0.1
0.08
0.09
0.14
0.07
15

---

Under review as a conference paper at ICLR 2020
D HUMAN PRECISION
In Table 2 we offer a characterisation of the human answers obtained from online entries and a control group for the different classes and filters. Used results refer to users in the control group and accepted online entries; Control refers to all users in the control group; Online All refers to all online users; Online Accepted refers the online users that passed the filter.
Table Mean human precision for the different entries grouping per label and filter
COLLUR
bear
bird
cal
dug
fish
[lower
fox
fruit
fungus
hippo
Used
0:96
0:97
0.74
0.35
0.35
0.57
0.96
Online enttles
0.91
0.88
0.86
0.91
0.75
0.09
0.8
0.81
0.54
0.9
Online Accepted
0.96
0.96
0:96
0.96
0.85
0.73
0.87
0.38
0.57
0.95
Inseet
Bion
monkey
reptille
shark
spider
Häger
vegetable
veliele
welf
Used
0.96
0.94
0.95
0.96
0.94
0.89
0.95
0.65
0.97
0.92
Online
0.91
0.88
0.81
0.59
0.83
Online Accepted
0.96
0.94
0.94
0.96
0.94
0.89
0.95
0.65
0.97
0.92
RESOLUTION
dog
flower
fungus 0.56
háppo
Used
0.95
0.95
0.97
(192
0.91
0.78
0.82
0.9
0.96
Cintrol
0.96
0.96
0.99
0.94
0.92
0.79
0.84
0.92
0.57
0.98
0.89
0.85
Online Accepted
0.95
0.97
0.78
0.81
0.56
0.95
monkey
reptiile
spider
tiger
vegetable
Used
0.94
0.95
0.94
0.96
0.91
0.92
0.96
0.68
0.98
Contral
0.96
0.96
0.96
0.90
0.93
0.53
0.588
0.09
1.0
0.82
Online
0.85
0.85
0.83
0.63
0.72
Accepted
bear
bird
fish 0.89
flowe
fox
fruit
Control
0.81
0.55
0.75
0.788
0.76
0.56
0.68
0.98
Online All
0.71
0.77
0.68
0.72
0.8
0.67
0.57
0.8
0.51
0.85
Online Accepted
0.84
0.74
0.75
0.96
insect
lion
monkey
reptile
shark
spider
liger
vegetable
vehicle
welf
Used
0.78
0.86
0.69
0.65
0.79
Control
0.79
0.94
0.87
0.92
0.7
0.97
0.66
1.0
Online All
0.7
0.86
0.79
0.79
0.82
0.62
0.88
0.61
0.89
0.74
Online Accopted
0.78
0.93
0.86
0.87
0.9
0.69
0.96
0.65
0.98
0.79
COMBINED
bear
bird
call
dug
fish
Hlower
fox
fruit
Tungus 0.69
hippo
Used
0.89
0.76
0.8
0165
0.83
0.62
0.75
0.56
0.93
0.7
Online
0.81
0.7
0.76
0.75
0.67
0.81
0.84
Online Accepted
0.89
0.76
0.8
0.65
0.83
0.62
0.75
0.56
0.69
0.92
Inseet
Hon
monkey
reptill
shark
spider
täger
vegetable
vebicle
welf
Used
0:75
0.89
0.9
0.79
0.89
0176
0.96
0.61
(3.55
0.75
Control
0.76
0.91
0.91
0.78
0.98
0.62
0.75
0.85
Online Accepted
0.75
0.88
0.9
0.78
0.89
0.76
0.95
0.61
0.96
0.73
16

---

Under review as a conference paper at ICLR 2020
E HUMAN MEPIS NOT CLASSIFIED CORRECTLY BY ANY DNN MODEL
Here we show three examples of human MEPIs not correctly classified by any of the four evaluated DNN models: Figure of fox, Figure[T0j TO is of a fungus. while Figure[TT]is of a vehicle. All three were generated under COMBINED. The gray border is used to highlight the cropped region.
Figure 9: Human MEPI of fox not classified correctly by any DNN model
Figure 10: Human MEPI of fungus not classified correctly by any DNN model
17

---

Under review as a conference paper at ICLR 2020
Figure 11: Human MEPI of vehicle not classified correctly by any DNN model
18


Reviews:
[{"review_id": "rJxzyUwf9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content - hence the name \"laconic\" image classification. The motivation for this task is to compare human and machine performance in a task that deviates from the standard ImageNet setup. To make different content reductions comparable, the authors measure the (approximate) entropy of an image via its PNG-compressed file size. As image transformations, the authors utilize quantization, downsampling, cropping, and a combination of the above. The authors find that convnets with higher accuracy are also more robust to these perturbations, and that humans perform well on the minimum-entropy examples of the networks (but not vice-versa).\n\nOverall I find the comparison of human and machine performance interesting and hence recommend accepting the paper. However, there are multiple directions that could possibly strengthen the core experiment. Hence I only give a weak accept at this point. Concretely, these directions are:\n\n- Do the results in the paper change if a different entropy measure is used (e.g., JPEG compression)?\n\n- As suggested by the authors, training networks to be robust to the \"laconic\" image perturbations could be an interesting direction. For instance, standard data augmentation with the proposed perturbations would be a relevant baseline.\n\n- To aid replicability and to compare the performance of different human test subjects, it would be interesting to conduct the experiment also on a crowdsourcing platform such as Mechanical Turk (as a complement to, not a replacement for, the university population in the paper).\n\n- Also to add replicability and to make it easier to compare different human accuracy evaluations, it would be good to measure how well the annotators perform on the unperturbed images and on a simple noise transformation (e.g., Gaussian noise).\n\n- It would be good to know how approximate the entropy measures in the paper are, e.g., to understand why humans perform worse in the \"combined\" perturbation setting.\n\n- How did the results from the control group and the open / online evaluation differ?\n\n\nIn addition, I have the following suggestions for improving the paper:\n\n- For the related work section, the authors may find the following papers on robustness of convnets to distortions interesting:\n\n* Manitest: Are classifiers really invariant?\nhttps://arxiv.org/abs/1507.06535\n\n* Exploring the Landscape of Spatial Robustness\nhttps://arxiv.org/abs/1712.02779\n\n* Spatially Transformed Adversarial Examples\nhttps://arxiv.org/abs/1801.02612\n\n* Semantic Adversarial Examples\nhttps://arxiv.org/abs/1804.00499\n\n- It could be helpful for the reader to see some example images of the different transformations in the main text.\n\n- Section 6: \"[...] a bias in texture for images trained on the ILSVRC dataset [...]\" - should this be \"classifiers\" instead of \"images\"?\n\n- Section 6: \"[...] would be interesting to explore in future\" - insert \"the\" before \"future\"?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJxR8XGzFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this empirical study, the authors attempt to identify a minimal entropy version of an image such that the image may be correctly classified by a human or computer. The authors then compare the efficacy of a human and computer to maintain accuracy in the presence of a reduced entropy representation of an image. The authors find that machines are more sensitive to reductions in entropy due to image resolution than humans (as opposed to color or cropping). In addition, the authors find that humans are generally better at identifying minimal entropy images than machines.\n\n1. Corruption results not surprising. \n\nAlthough the authors offer some intriguing methods, I found the results to not be compelling nor improve our understanding of the relative differences between human and machine perception. While identifying that humans are less sensitive to a reduction in resolution, this result is not terribly surprising given that networks are known to suffer from aliasing artifacts, e.g.\n\n  Geodesics Of Learned Representations\n  Olivier J. Henaff & Eero P. Simoncelli\n  https://www.cns.nyu.edu/pub/lcv/henaff16b-reprint.pdf\n\n2. Unclear what we learn from the method.\n\nI am not too clear about what specific insights the methods provide in this paper. Estimating the entropy associated with each image corruption -- while interesting -- does not lead to any substantive analysis nor conclusions as far as I can tell. Given the lack of benefit to analyzing the entropy, I am left to really just consider these methods to be image corruptions that downsample the resolution or desaturate the images. These corruptions are not terribly novel with respect to previous work, e.g.\n\n  Generalisation in humans and deep neural networks\n  https://arxiv.org/pdf/1808.08750.pdf\n\n  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\n  https://arxiv.org/abs/1903.12261\n\nTo summarize, my feedback is the following:\n\n1. Please justify the use of entropy to quantify the distortion. What does entropy provide above and beyond just parameterizing the distortion (e.g. image resolution, color saturation)?\n\n2. Are there other results above-and-beyond sensitivity to image resolution that distinguishes human and machine in these experiments? These results seem to be largely known by just considering corruptions such as low pass filters, etc. presented in the above papers.\n\n\nMinor Comments:\n\n- The authors should provide a figure with example images in the main text showcasing how each method corrupts an image.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1xx7qp5YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to understand and compare the performance of DNNs classifier, which is different from the precise prediction in the notion of correct/wrong. With approximate minimal-entropy of input images, the classifiers can recognize this image so that different classifiers including human and DNNs will need different reduction methods (cropping, downsampling, color reduction )for the same image to give correct prediction and also will give us different performance in a same test dataset. By comparing the results with human’s and DNNs’, the author claims that it will have more challenges for DNNs in this laconic image classification task than human will have.  \nFor the motivation in this paper, the author tries to propose a new perspective to evaluate the robustness of image classifiers. Especially compared with humans’ understanding, this paper would like to rethink the influence of reduction for this task. However, it is not clear that why three reduction methods the author used can help with understanding the difference between humans and DNNs because when training a DNN for image classification, we usually use these methods to augment our training set, but for humans, it is a totally different story that how to recognize an image.\nFor the theoretical demonstration, in this paper, the author uses approximating minimal-entropy to quantify the minimal content of an image DNNs or humans need to give correct category. The intuition of this method is suitable. But in section 3, the author didn’t give a clear demonstration of how to compute the entropy reduction in 3.1. I think if it is better to introduce how to measure 3.2 in detail, then 3.1 may be more clear. And it also makes me confused about the atomic reduction step in the last paragraph in 3.1. For the 3.5, I think the authors should focus on how to demonstrate MEPIs for humans more mathematically so that it will be more reliable.\nFor the experiments, the author tries to answer two questions: 1. How does the entropy required by DNN and human classifiers compare? 2. How do the classifiers perform in terms of precision for each others’ MEPIs? However, the experiments do not provide convincing evidence to existing approaches. First of all, for a single DNN, how different entropy reduction methods influence the classification? Secondly, how different reduction scales in the same model influence the results? At last, the comparison between different models should give a more visualized figure to illuminate the difference. It will be better to provide more ablation study experiments for this paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content. 
All the reviewers found the comparison of human and machine performance interesting and valuable. However the reviewers expressed concerns and noted the following weaknesses: the presented results are not convincing to support our understanding of the differences between human and machine perception (R1), using entropy to quantify the distortion is not well motivated and has been addressed before (R1), lack of empirical evidence (R2). 
AC suggests, in its current state the manuscript is not ready for a publication. We hope the detailed reviews are useful for improving and revising the paper. 
Decision: Reject