Title: LACONIC IMAGE CLASSIFICATION: HUMAN VS. MACHINE PERFORMANCE

Abstract: We propose laconic classification as a novel way to understand and compare the performance of diverse image classifiers. The goal in this setting is to minimise the amount of information (aka. entropy) required in individual test images to maintain correct classification. Given a classifier and a test image, we compute an approximate minimal-entropy positive image for which the classifier provides a correct classification, becoming incorrect upon any further reduction. The notion of entropy offers a unifying metric that allows to combine and compare the effects of various types of reductions (e.g., crop, colour reduction, resolution reduction) on classification performance, in turn generalising similar methods explored in previous works. Proposing two complementary frameworks for computing the minimal-entropy positive images of both human and machine classifiers, in experiments over the ILSVRC test-set, we find that machine classifiers are more sensitive entropy-wise to reduced resolution (versus cropping or reduced colour for machines, as well as reduced resolution for humans), supporting recent results suggesting a texture bias in the ILSVRC-trained models used. We also find, in the evaluated setting, that humans classify the minimal-entropy positive images of machine models with higher precision than machines classify those of humans.

1. INTRODUCTION:

Deep neural networks now surpass human-level performance on a variety of specific tasks and metrics relating to visual recognition. In a widely-used yardstick for human-level performance, Russakovsky et al. (2015) estimated that an expert human with prior training can achieve a top-5 classification error rate of 5.1% on a dataset of 1,500 ILSVRC images and 1,000 target classes. Shortly after, He et al. (2015) surpassed human-level performance on the same task achieving 4.9% top-5 error with PReLU-net. Later works would further reduce this error rate, including ResNet (3.6%) (He et al., 2016) , Trimps-Soushen (3.0%) (Shao et al., 2016) , SeNet (2.3%) (Hu et al., 2019) , etc., with contemporary state-of-the-art models more than halving estimated human error for this specific task.

Though such results represent landmark advances, by focusing on classification errors alone, they do not reveal the full story of relative machine performance for image classification. Works on adversarial examples (Dalvi et al., 2004; Szegedy et al., 2014; Nguyen et al., 2015) , for instance, establish that human and machine perception diverges greatly for specifically constructed images. Other works have presented bespoke experiments comparing human and machine performance beyond classification errors, presenting evidence for a lack of robustness in the presence of noisy (Russakovsky et al., 2015; Dodge & Karam, 2017; 2019) or incomplete information (Ullman et al., 2016; Wick et al., 2016; Linsley et al., 2017; Ho-Phuoc, 2018; Srivastava et al., 2019) , a sensitivity to spatial Fawzi & Frossard (2015) ; Hénaff & Simoncelli (2016) ; Xiao et al. (2018) ; Engstrom et al. (2019) or colour Hosseini & Poovendran (2018) ; Hosseini et al. (2018) transformations, a lack of generalisation (Geirhos et al., 2018) , a bias towards texture (Geirhos et al., 2019) , etc., in evaluated models. By applying specific transformations on test images prior to classification, these latter works provide insights into the specific differences in the types of information that humans and machines rely on to perform adequately at this task. These latter recent works suggest the need for an information-theoretic framework that generalises such issues: a framework within which the performance of classifiers -be they human, machine or other -can be compared and understood, allowing to quantify, in a more fine-grained manner, the type of information in the input on which a given classifier depends. While previous works address individual or multiple types of information reduction on input images in isolation, a more general framework should allow to combine and compare different types of reduction on test inputs.

In this paper, we propose such a framework, based on the principle of computing and analysing minimal entropy positive inputs: inputs with minimal information with respect to yielding correct classification results. The notion of entropy intuitively generalises and allows to compare the relative effects of different reductions on inputs -and their combinations -on classifier performance; such reductions may include, for example, cropping, downsampling and quantisation. The goal in this framework thus shifts from precise classification to laconic classification: providing a classifier that minimises the entropy of input(s) required for correct classification. Being based on a continuous notion of entropy rather than a discrete notion of correct/incorrect, this latter goal thus presents a novel challenge beyond minimising classification error-a goal for which state-of-the-art approaches are close to achieving perfect results on datasets like ILSVRC. In fact, existing datasetssuch as ILSRVC -can be straightforwardly leveraged for evaluating classifiers under this new goal. Models performing more laconic classification (we currently conjecture) should likewise perform more robustly in practical settings involving incomplete or noisy information capture.

Though the framework we propose can be applied to any classifier for any classification task, herein we first instantiate the framework on the aforementioned problem of image classification. We consider three general operations for reducing the entropy of test images: crop, resolution reduction (downsampling) and colour reduction (quantisation). We then propose two methods for finding the minimal entropy positive images under these reductions for two different types of classifier.

Given a pre-trained machine classifier -where classification can be separated from learning -an input test image, and a set of reduction operations, we use a known search algorithm (Powell, 1964) and apply the given reduction functions to the input image to find (under certain assumptions) the lowest entropy image that the model classifies correctly such that applying any further reduction of entropy leads to incorrect classification. We apply the aforementioned framework to find the minimal entropy images from a sample ILSVRC test-set for state-of-the-art deep neural-network (DNN) models (GoogLeNet, SqueezeNet, ResNet50 and SeNetResNet50), with respect to the three aforementioned reduction operations and their combination. Our experiments show that minimal entropy images are considerably smaller than original images for DNNs; for example, we find that with only 2-6% of the information content of the original test-images (on average) the best performing machine model can still produce a correct classification of the reduced image.

We then consider also human classifiers in order to compare their ability to perform laconic classification with DNN models; applying our framework for humans was not trivial since learning cannot be separated from classification (we cannot start with the full input test-image and reduce it since the human will remember the image) and an automatic optimization algorithm was not possible. To cope with this we designed an experiment reversing the optimization goal: starting from a void image, the human evaluator may add information incrementally until they believe that a confident classification of the displayed image is possible. We apply this framework with more than 500 human subjects through an online interface. Our results show that the minimal-entropy positive images for humans are considerably smaller in the case of colour and resolution reductions (53-62% the size of the corresponding size for the best performing machine model).

Finally we cross-classified the minimal-entropy positive images among DNNs and humans: given classifier A and B, we presented A's minimal-entropy positive images to B and vice versa, computing classification precision. We show that the precision of human classifiers on DNN minimalentropy positive images was considerably better (0.74 precision in the worst case) than the corresponding results for cross-classification among DNN models (0.02 precision in the worst case for the best model). Furthermore, we find that the DNN models on human images give precisions of 0.03-0.43, depending on the reductions used for the images and the model used for classification.

Our results provide insights into how different classifiers -both machine and human -rely differently on different types of information, providing further evidence to support, for example, prior claims of the lack of robustness to incomplete information relative to humans in such models (Dodge & Karam, 2017) , a bias towards texture in ILSVRC-trained models (Geirhos et al., 2019) , and so forth. As a more general conclusion, we show that humans are capable of performing (much) more laconic classification in the evaluated setting than state-of-the-art machine classifiers. We conclude with open challenges regarding the laconic classification of images using DNNs.

2. RELATED WORKS:

A number of previous works have studied the robustness of image classification for DNNs and humans. Ullman et al. (2016) introduced the notion of a "minimal image" as the smallest region of an image that is still recognisable by a human. They show that a small change in these minimal images can have a drastic effect on recognition. Moreover, Ullman et al. (2016) show that DNNs are unable to accurately recognise human minimal images. To compute the minimal images, they started from the complete image and then iteratively cropped it showing the new cropped image to a different human subject every time. The process stopped when the accuracy of recognizing every crop of the current patch dropped from a threshold. Given the difficulty of computing minimal regions recognisable by humans, their experiments were limited to 10 images (Ullman et al., 2016) . Srivastava et al. (2019) extended the previous work by showing that the sharp drop in accuracy for minimal images can also be observed in DNNs. They defined the notion of "fragile recognition image" as a region of an image for which a small change in size produces a considerable change in accuracy of recognition by DNNs. Srivastava et al. (2019) showed that fragile recognition images are abundant and can occur for several different sizes. The work by Ullman et al. (2016) and Srivastava et al. (2019) can be modelled inside our minimal-entropy framework, which considers not only regions of images (i.e., crop), but also reductions in resolution, colour, and combinations thereof.

Another line of related works study the robustness of DNN image classification with respect to distortions to the input image (Hendrycks & Dietterich, 2019; Geirhos et al., 2018; 2019; Dodge & Karam, 2017; 2019; Fawzi & Frossard, 2015; Hénaff & Simoncelli, 2016; Xiao et al., 2018; Engstrom et al., 2019; Hosseini & Poovendran, 2018; Hosseini et al., 2018) . Dodge & Karam (2017; 2019) also study how humans react to these distortions showing that the performance of DNNs is much lower than human performance on distorted images. Hendrycks & Dietterich (2019) constructed a benchmark for image classifier robustness based on several different image distortions. Although all these distortions are designed to make the image recognition task more difficult, not all of them reduce the information in the image. For example, two transformations considered by Geirhos et al. (2018) are image rotation and colour inversion that do not alter the volume of information. Other transformations considered by Geirhos et al. (2018) -such as Eidolon and additive noise -may even increase the image file size. In contrast we currently only consider transformations that reduce the information content of the input image file.

3. APPROXIMATING MINIMAL-ENTROPY POSITIVE IMAGES:

We now discuss the framework we use for evaluating the laconic classification of images. We first discuss the entropy measure and the reductions applied to images in this work. We then discuss the computation of approximate minimal-entropy positive images (MEPIs) for DNNs and for humans.

3.1. ENTROPY MEASURE:

Our goal is to find the minimal-entropy positive images (MEPIs) for various classifiers based on various forms of entropy reduction. Although there exist numerous proposed entropy measures (Wu et al., 2013; Larkin, 2016) , the analysis of entropy for images is not straightforward due to their dimensionality, particularly in the case of multi-channel images where measures of entropy are further complicated by the conditional and joint entropy that must be measured across the channels. We mitigate this issue by adopting the compressed size of an image using a lossless image encoder as an estimation of image entropy (Larkin, 2016) . For these purposes we propose to use the Portable Network Graphics (PNG) encoder -proposed as a successor for the Graphics Interchange Format (GIF) -which combines an initial predictive filtering step to improve compressibility, followed by a DEFLATE compression. The size of PNG images has been shown to outperform more complex measures of entropy (Larkin, 2016), while being widely implemented in a variety of libraries.

3.2. ENTROPY REDUCTION:

Given an m ! n matrix A, we consider the following forms of entropy reduction: A£ Qκ is defined as the (nearest-element) m ! n quantised matrix of A with quantisation factor κ, such that a ij ¢ arg min v>1,...,κ v κ aij maxA 2 for all a ij in A£ Qκ .

A£ Dσ is defined as the r ! s (r f m, s f n) downsampled matrix of A with scaling factor σ such that 0 d σ f 1, σm r, σn s. A£ Sα,β,γ,δ is defined as the (contiguous In the case of images, these reductions correspond, respectively, to colour reduction (parameter: κ), resolution reduction (parameter: σ), and crop (parameters: α, β, γ, δ). We also consider their combination (parameters: α, β, γ, δ, κ, σ). We thus have six parameters by which to reduce entropy. We adapt these reductions to multi-channel images in the natural way. In the case of crop, we apply the reduction to all channels separately; however, based on initial experiments with DNNs, rather than remove the rows and columns of the image's channels, we rather replace them with a constant neutral value, which allowed further entropy reduction in positive images by avoiding distortions once images are internally rescaled by the network (i.e., images with narrow crops being "stretched out"). In the case of colour reduction, the nearest quantisation values are computed in the multichannel case based on Euclidean distance; we further normalise the output values to fill the colour space after the quantisation, choosing equidistant points. In the case of downsampling, the reduction is applied to each channel and maintains the same aspect ratio for square images.

) p!q submatrix (p f m, q f n) computed from the indices α, β, γ, δ (α d k, β α p, γ d n, δ γ q) such that A£ Sα,β,γ,δ A α, α 1, ..., β 1; γ, γ 1, . . . ,

In Appendix A we provide examples of the four types of reduction on an image of a dog.

3.3. MINIMAL-ENTROPY POSITIVE IMAGES:

Given an input image, a set of entropy reductions, and a classifier, we define a naive minimalentropy positive image to be the smallest (PNG) image derived from the input image using the given reductions for which the classifier provides a correct classification (i.e., the top predicted class is correct). To compute such images, we apply reductions in discrete steps. More formally, given a matrix A and entropy reduction b Q, D, S, S, S, S, we define an atomic reduction step as A£ such that ¢ minx A£ x x A; i.e., is the smallest value that modifies A under A£ . For α, β, γ, δ and κ, the step size is one ( 1). In the case of σ, we define as the smallest value such that σm x σ m or σn x σ n (i.e., such that the image is rescaled).

This definition may lead to undesirable results: consider, for example, the case of a classifier making predictions based on the individual (nondescript) pixels of the input image. With varying probability -depending on the number of pixels, images, and classes, the form of classification, etc. -a classifier may for some pixel "guess" the correct class, with that pixel potentially becoming a naive minimalentropy positive image. To avoid such cases, we add a continuity condition: a minimal-entropy positive image (MEPI) is then defined to be the smallest image derived from the input image using the given reductions for which the classifier provides a correct classification, and for which there is a continuous path of atomic reduction steps from the input image giving correct classifications.

3.4. APPROXIMATING MEPIS FOR DNNS:

In the case of DNNs, for a given image and applicable reductions, we apply a search for the value(s) of the parameter(s) that correspond to the MEPI of the image. Our general method begins with the input image, incrementally applying atomic reduction steps while the prediction of the classifier remains correct, backtracking in the case of an incorrect prediction. While relatively simple to implement for reductions with a single parameter, in the case of crop or combined reductions, we have multiple parameters with respect to which we must search; for example, in the case of crop, for an m ! n image, the number of images to check in the worst-case is potentially m 2 n 2 , i.e., more than 274 billion contiguous sub-images for a 1024 ! 1024 input image (and this considering crop alone without combination with resolution or colour). Approximation is thus sought. First, our assumption of continuity helps us to prune the search space: if we reach a set of parameter values for which classification is correct but for which any further atomic reduction is incorrect, we know we can rule out all further reductions from that point. We may still, however, encounter an infeasible search space when considering multiple parameters; to improve performance, we choose to apply a greedy search algorithm: given that the function we wish to minimise is not differentiable but has a fixed number of inputs, we apply Powell's method (Powell, 1964) to find a local minimum.

3.5. APPROXIMATING MEPIS FOR HUMANS:

Humans operate differently to DNNs and thus require specialised methods to approximate their MEPIs. First, humans cannot separate classification from learning; hence we cannot show the full input image and apply reductions as the human will (of course) remember the full input image. Second, humans require (much) more time per classification; hence the search steps must be adjusted.

In particular, given A, we rather compute a bottom-up approximation of MEPIs for humans. In what follows, given a set of entropy reductions R Q, D, S, S, S, S, we use A R to denote the set of matrices recursively reachable from A by some sequence of (atomic) reduction steps from R, including A itself. Given b R, we can then define ¡ as the inverse of £ under A R , such that if A b A R , then A ¡ x A if and only if A b A R and A £ x A for any value x well-defined for . Now, starting from an initial image A 0 b A R , at each step i g 0 the human can then either (1) if A i x A, select an atomic step b R and increment the entropy of the image by a constant factor ζ such that A i 1 ¢ A i ¡ ζ , or (2) select a class for the image from a list if they believe they can correctly classify the image. If the human guesses correctly, this is considered to be an approximate MEPI (for the image, parameters and human); if not, the image is discarded and a fresh image is considered. To mitigate fatigue, larger atomic steps are defined such that no more than 20 are required to reach the input image along a given dimension; i.e., a constant step γ is chosen such that A can be reached from A 0 by applying ¡ γ 20 times for each b R. In the case of crop, starting from the central pixel (A 0 ¢ ¡a m 2 , n 2 ¦), we offer the option to increment in all directions at once, while in the combined case, we offer the option to increment all parameters at once. This method makes it feasible to approximate MEPIs for humans, but it is important to note that with the limitations of coarser steps and always starting at the central pixel, the MEPIs approximated using this method are more likely to overestimate the entropy required for correct classification.

4. EXPERIMENTAL SETTING:

We base our experiments on images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) (Russakovsky et al., 2015) . We select four DNN classifiers for evaluation (all trained on the ILSVRC2012 training set): SqueezeNet (Iandola et al., 2016) , GoogLeNet (Szegedy et al., 2015) , ResNet50 (He et al., 2016) and SeNetResNet50 (Hu et al., 2019) ; as discussed in the introduction, the latter two classifiers surpass human-level performance in terms of top-5 classification error on a dataset of 1,500 ILSVRC images and 1,000 target classes.

As acknowledged by Russakovsky et al. (2015) , the 1,000 detailed classes of ILSVRC (e.g., COU-CAL, SEALYHAM TERRIER) require extensive previous training for humans to perform adequately at classification, not only to visually distinguish the objects, but also to be able to retrieve their label from the 1,000 possible labels. We thus opt to greatly simplify the classes, selecting the following twenty: BEAR, BIRD, CAT, DOG, FISH, FLOWER, FOX, FRUIT, FUNGUS, HIPPOPOTAMUS, INSECT, LION, MONKEY, REPTILE, SHARK, SPIDER, TIGER, VEGETABLE, VEHICLE, WOLF. We select these classes as they should be generally recognisable to humans without prior training; furthermore, we select mostly plants and animals to provide a more challenging classification task, with visually similar classes, such as LION/TIGER, FRUIT/VEGETABLE, DOG/WOLF, INSECT/SPIDER, etc., providing non-trivial cases to visually distinguish. The results of DNN models are then mapped hierarchically to these higher-level classes (e.g., COUCAL @ BIRD, SEALYHAM TERRIER @ DOG). We sample 15 images for each of the 20 classes from the ILSVRC2012 test set for experiments. 

Colour:



5. EXPERIMENTAL RESULTS:

With our experimental results, we address two main questions: (1) How does the entropy required by DNN and human classifiers compare? (2) How do the classifiers perform in terms of precision for each others' MEPIs? We address these two questions in the following subsections, along with other underlying questions relating to how different image reductions affect individual models, how the goals of laconic and accurate classification correlation, etc.

5.1. ENTROPY RATIO IN MEPIS:

For the 20!15 300 test images, we first compute the top-down MEPIs for the four DNN classifiers using the method described in Section 3.4. For humans, following the bottom-up method described in Section 3.5, we provide a web interface that -starting with a void image -allows a human user in each step to either increment the image by the given parameters, or select the class for the currently displayed image. In order to achieve many responses, the interface was shared on a university forum as well as on social media. The options and instructions were presented in Spanish, corresponding to the native language of the country in which the university is based. Screenshots of the interface are given in Appendix B. In total, 423 user sessions were logged; of the 1,722 responses obtained (average 4.07 per session), 1,340 (77.8%) resulted in a correct validation and thus a valid MEPI.

In Figure 1 we present the entropy ratio for four different settings across five different classifiers; entropy ratio is defined here as the ratio of the (PNG-encoded) size of the original input image versus the size of the extracted MEPI. We take the average entropy ratio for the images of each of the twenty classes. Figure 1 then presents the box-plots -displaying the 1 st (min), 25 th , 50 th (median), 75 th , and 100 th (max) percentiles with mean marked as a diamond -for the mean ratio across the different classes; to illustrate with an example, in the case of SqueezeNet, considering only the COLOUR experiment, the best class had a mean entropy ratio of 0.11 (bottom whisker), the worst class had a mean entropy ratio of 0.24 (top whisker), etc. We present the DNN models in order of their reported performance for top-5 classification error on the ILSVRC dataset, with SqueezeNet having the highest such error and SeNetResNet50 having the lowest.

From this figure we can draw some high-level observations about the DNN classifiers. First, for the DNN classifiers, the entropy ratio is lowest for the COMBINED experiment (as expected), which offers more avenues by which to reduce the entropy while maintaining a correct classification. Second, the results for DNN classifiers follow the same trend as for performance based on classification error; this suggests that there is a correlation between the goals of laconic classification and precise classification. Third, we see that DNNs are most sensitive to reductions in resolution, which supports the hypothesis that DNNs trained on ILSVRC images are biased towards texture (Geirhos et al., 2019) (with RESOLUTION being the parameter that most affects the ability to distinguish texture). We also see a considerable variation across classes, particularly for RESOLUTION; for space reasons, we provide the detailed results of mean entropy ratio by class in Appendix C. With respect to human classifiers, we note that they are less sensitive to reductions in COLOUR and much less sensitive to reductions in RESOLUTION than all DNNs, and more sensitive to reductions in CROP than some state-of-the-art DNNs. We also see an unusual result, whereby the reduction ratios for COMBINED are not lower than those for (e.g.,) COLOUR; this suggests that the users may have struggled with the interface for the CROP and COMBINED experiments, where multiple options were provided to increment the entropy of the image (versus COLOUR and RESOLUTION, which only permitted increments in one direction). As such, the results for CROP and COMBINED in human classifiers leave an ambiguity: are the relatively poor results of humans due in this case to greater sensitivity to such (multi-parametric) reductions, or because of difficulty using the more complex interface in these cases? We require further experiments to address this ambiguity.

5.2. CLASSIFICATION PRECISION FOR MEPIS OF DIFFERENT CLASSIFIERS:

We now look at the precision of (top-1) classification across the MEPIs of models, again considering COLOUR, RESOLUTION, CROP and COMBINED. To gather results for human classification, we created a second, simpler, online interface that presents the MEPI of a particular model under a particular reduction and asks the human evaluator to select the class for that MEPI from the list of twenty possible classes. The interface was shared again on a university forum and through social media. We provide a screenshot of this interface in Appendix B. We first ran a control group with 25 trusted users, which logged an aggregate precision of 0.875 with a standard deviation of 0.061; in the open/online evaluation, we then filter user sessions more than two standard deviations from the control mean precision, giving a lower threshold of 0.753. The public evaluation then logged 531 valid user sessions according to the threshold, resulting in 11,588 valid classifications of machine MEPIs (equating to 26.2 classifications on average per session). A comparison between the different groups of humans users is provided in Appendix D.

The results of the precision for cross-classification of MEPIs are summarised in Figure 2 . Cells are shaded darker to indicate better performance. Darker rows indicate better classifier performance while darker columns indicate having MEPIs that are easier for other models to classify. First we observe that all classifiers correctly predict (as expected by definition) all of their own MEPIs.

With DNNs ordered by expected performance, we again see the general trend that fewer reported classification errors in ILSVRC again correlate with better precision in the classification of MEPIs, with, for example, SeNetResNet50 (SRN) having darker rows and lighter columns than other DNNs. We also see good cross-classifier performance for COLOUR and RESOLUTION: given that this are one dimensional reductions, the space of possible images is greatly reduced. On the other hand, DNN models struggle in cross-classification of the MEPIs under CROP and COMBINED: this is perhaps due to the larger search space for these reductions, but also indicates that these MEPIs for DNNs are characteristic of that particular DNN. In general, the MEPIs of SRN are notably more difficult for other models to classify (as could have been expected from Figure 1 ).

Regarding human classification, we see that in general humans perform best of all classifiers in this experiment, being adaptable enough to classify the MEPIs under all reductions with relatively high precision. The most difficult MEPIs for human are in the CROP and COMBINED configurations for SeResNet50 (SRN), where a relatively high precision of 74-76% correct classifications is still seen. With respect to the previously discussed ambiguity in the results of the previous sub-section, we can thus see that although CROP and COMBINED are more difficult cases for humans, issues with the more complex interface for these cases may explain the relatively poor results of humans in Figure 1 for CROP and COMBINED. Furthermore, we can see that DNNs often struggle to classify the human MEPIs, where the best precision reached was 42% correct classifications for SRN in he case of CROP; Appendix E provides a sample of three human MEPIs not correctly classified by any DNN model, illustrating some of the most difficult such cases for DNN models.

6. DISCUSSION:

Based on our proposed frameworks for computing minimal-entropy positive images (MEPIs) for both machine and human classifiers, our first results were based on the relative entropy of such images for the different classifiers. From these results we can conclude that state-of-the-art machine models are relatively sensitive (entropy-wise) to reductions in resolution when compared with other forms of reduction based on cropping or colour; such observations appear to independently support previous results indicating a bias in texture for classifiers trained on the ILSVRC dataset (Geirhos et al., 2019) . On the other hand, humans are relatively sensitive to the cropping of images, but are much less sensitive than machine classifiers to reductions in resolution; this tends to suggest that humans rely more on form and context rather than textures of small regions of the image. In our second set of results, we performed cross-classification of the MEPIs for different models; we saw that humans greatly outperform machine models for classifying the MEPIs of other models, suggesting a generally more robust aptitude in humans for the laconic classification of images. Of particular interest is that there appears to be a strong correlation between the tasks of laconic and accurate classification (per Figure 1 ) which suggests that the former may be a new take on the latter: working on laconic classification suggests a range of information-theoretic approaches that could be brought to bear in order to improve classifier accuracy.

Our work has a number of limitations that could be addressed for future work. While the method and interface used for computing human MEPIs in a bottom-up fashion work well for simpler (i.e., onedimensional) forms of image enhancement, the unexpected result in Figure 1 showing an increase in the size of MEPIs in the combined case suggest that the users had some difficulty using the interface for multi-dimensional settings; given that we filter incorrect images from consideration, we speculate that the users often "overshoot" the MEPI in particular dimensions, finding it difficult to select the particular dimension that is most likely to help them correctly classify the image. Refinements of this idea would be interesting to explore in the future. Furthermore, in order to facilitate non-expert humans to participate, we selected high-level classes, where it would be of interest to try to repeat the results for lower-level classes (though again presenting challenges for the interface).

On the other hand, we have performed experiments for pre-trained, off-the-shelf DNN models. An interesting line of research would be to rather train models specifically for the task of laconic classification. Along these lines, one could consider computing MEPIs in the training set and feeding them (potentially recursively) back into the model; unlike the internal transformations applied in specific DNN models, such a framework for laconic learning could treat a particular model as a black-box. It would further be interesting to explore how models trained in such a manner perform in more traditional classification metrics -error rates, precision, etc. -on the original input images, as well as whether or not they might help to address the observed lack of robustness of state-of-the-art DNNs in the presence of noisy (Russakovsky et al., 2015; Dodge & Karam, 2017; Hosseini et al., 2018; Dodge & Karam, 2019) or incomplete information (Ullman et al., 2016; Wick et al., 2016; Linsley et al., 2017; Ho-Phuoc, 2018; Srivastava et al., 2019) , or their lack of generalisation (Geirhos et al., 2018) , or their bias towards texture (Geirhos et al., 2019) . Further exploration is needed.

More generally, we view laconic classification as a potentially fruitful avenue to explore: one that is relatively straightforward to implement, can re-use available datasets, connects a number of recent works, provides insights into the performance and robustness of diverse classifiers, and one that may perhaps reveal new challenges and lead to new developments in image classification.

NOTE::

We plan to publish images and other materials online after double-blind review. Please also see appendices for further details.

B SCREENSHOTS OF INTERFACES:

We present screenshots of the online interfaces used to collect human MEPIs and classifications. As discussed in the paper, the options and instructions were provided in Spanish as the target audience were students of a university whose native language is Spanish. Figures 4 5 6 7 show screenshots of the interfaces used to collect human MEPIs for COLOUR, RESOLUTION, CROP and COMBINED. On the other hand, Figure 8 shows a screenshot of the interface to classify machine MEPIs. 

C MEAN ENTROPY RATIO PER CLASS:

In Table 1 we present the mean entropy ratio of MEPIs per class. 

D HUMAN PRECISION:

In Table 2 we offer a characterisation of the human answers -obtained from online entries and a control group -for the different classes and filters. Used results refer to users in the control group and accepted online entries; Control refers to all users in the control group; Online All refers to all online users; Online Accepted refers to the online users that passed the filter. 

:



Figures and Tables:

Figure 1: Box-plots of mean entropy ratio for DNN and human MEPIs across classes

Figure 2: Heatmaps of classification precision on MEPIs for the four types of entropy reduction; rows indicate the classifier, columns indicate the MEPIs classified: SN = SqueezeNet; GN = GoogLeNet; RN = ResNet50; SRN = SeResNet50; Hum = Human

Figure 4: Screenshot of interface for humans to specify COLOUR MEPIs

Figure 7: Screenshot of interface for humans to specify COMBINED MEPIs

HUMAN MEPIS NOT CLASSIFIED CORRECTLY BY ANY DNN MODEL Here we show three examples of human MEPIs not correctly classified by any of the four evaluated DNN models: Figure 9 is of a fox, Figure 10 is of a fungus, while Figure 11 is of a vehicle. All three were generated under COMBINED. The gray border is used to highlight the cropped region.

Figure 9: Human MEPI of fox not classified correctly by any DNN model

δ 1¥; in other words, it removes rows 1, . . . , α 1, β, . . . , m and columns 1, . . . , γ 1, δ, . . . , n from A. We use A£ Sα ¢ A£ Sα,m,1,n , A£ Sβ ¢ A£ S1,β,1,n , A£ Sγ ¢ A£ S1,m,γ,n and A£ Sδ ¢ A£ S1,m,1,δ to denote removing rows/columns (only) from the left, right, top and bottom of the image, respectively.

Table content: <table/>

Mean entropy ratio per class for DNN and human MEPIs

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">COLOUR</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>SqueezeNet</td><td>0.24</td><td>0.16</td><td>0.22</td><td>0.23</td><td>0.17</td><td>0.11</td><td>0.23</td><td>0.15</td><td>0.21</td><td>0.17</td></tr><tr><td>GoogLeNet</td><td>0.18</td><td>0.14</td><td>0.17</td><td>0.16</td><td>0.15</td><td>0.1</td><td>0.18</td><td>0.13</td><td>0.19</td><td>0.15</td></tr><tr><td>Resnet50</td><td>0.16</td><td>0.12</td><td>0.12</td><td>0.12</td><td>0.12</td><td>0.08</td><td>0.16</td><td>0.11</td><td>0.16</td><td>0.13</td></tr><tr><td>Se-Resnet50</td><td>0.17</td><td>0.11</td><td>0.12</td><td>0.14</td><td>0.13</td><td>0.09</td><td>0.14</td><td>0.15</td><td>0.14</td><td>0.11</td></tr><tr><td>Humans</td><td>0.1</td><td>0.07</td><td>0.07</td><td>0.06</td><td>0.08</td><td>0.09</td><td>0.08</td><td>0.07</td><td>0.15</td><td>0.08</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>SqueezeNet</td><td>0.15</td><td>0.22</td><td>0.23</td><td>0.19</td><td>0.18</td><td>0.17</td><td>0.23</td><td>0.19</td><td>0.18</td><td>0.21</td></tr><tr><td>GoogLeNet</td><td>0.12</td><td>0.19</td><td>0.18</td><td>0.15</td><td>0.13</td><td>0.12</td><td>0.17</td><td>0.15</td><td>0.12</td><td>0.17</td></tr><tr><td>Resnet50</td><td>0.09</td><td>0.16</td><td>0.15</td><td>0.1</td><td>0.13</td><td>0.09</td><td>0.16</td><td>0.14</td><td>0.15</td><td>0.12</td></tr><tr><td>Se-Resnet50</td><td>0.11</td><td>0.12</td><td>0.15</td><td>0.12</td><td>0.12</td><td>0.1</td><td>0.13</td><td>0.14</td><td>0.09</td><td>0.12</td></tr><tr><td>Humans</td><td>0.07</td><td>0.07</td><td>0.08</td><td>0.08</td><td>0.05</td><td>0.06</td><td>0.08</td><td>0.09</td><td>0.08</td><td>0.05</td></tr><tr><td/><td/><td/><td/><td colspan="2">RESOLUTION</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>SqueezeNet</td><td>0.5</td><td>0.51</td><td>0.64</td><td>0.47</td><td>0.57</td><td>0.36</td><td>0.57</td><td>0.43</td><td>0.39</td><td>0.46</td></tr><tr><td>GoogLeNet</td><td>0.35</td><td>0.37</td><td>0.51</td><td>0.3</td><td>0.42</td><td>0.35</td><td>0.42</td><td>0.38</td><td>0.3</td><td>0.45</td></tr><tr><td>Resnet50</td><td>0.28</td><td>0.34</td><td>0.21</td><td>0.11</td><td>0.28</td><td>0.14</td><td>0.39</td><td>0.29</td><td>0.32</td><td>0.4</td></tr><tr><td>Se-Resnet50</td><td>0.18</td><td>0.27</td><td>0.17</td><td>0.05</td><td>0.17</td><td>0.11</td><td>0.22</td><td>0.17</td><td>0.19</td><td>0.25</td></tr><tr><td>Humans</td><td>0.07</td><td>0.05</td><td>0.06</td><td>0.06</td><td>0.14</td><td>0.09</td><td>0.16</td><td>0.06</td><td>0.16</td><td>0.11</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>SqueezeNet</td><td>0.5</td><td>0.53</td><td>0.48</td><td>0.71</td><td>0.5</td><td>0.6</td><td>0.55</td><td>0.47</td><td>0.59</td><td>0.54</td></tr><tr><td>GoogLeNet</td><td>0.37</td><td>0.33</td><td>0.39</td><td>0.56</td><td>0.42</td><td>0.39</td><td>0.4</td><td>0.33</td><td>0.4</td><td>0.41</td></tr><tr><td>Resnet50</td><td>0.33</td><td>0.14</td><td>0.25</td><td>0.23</td><td>0.35</td><td>0.21</td><td>0.36</td><td>0.19</td><td>0.44</td><td>0.26</td></tr><tr><td>Se-Resnet50</td><td>0.22</td><td>0.13</td><td>0.17</td><td>0.3</td><td>0.2</td><td>0.26</td><td>0.18</td><td>0.13</td><td>0.18</td><td>0.15</td></tr><tr><td>Humans</td><td>0.09</td><td>0.05</td><td>0.11</td><td>0.14</td><td>0.04</td><td>0.14</td><td>0.1</td><td>0.15</td><td>0.07</td><td>0.11</td></tr><tr><td/><td/><td/><td/><td/><td>CROP</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>SqueezeNet</td><td>0.2</td><td>0.18</td><td>0.16</td><td>0.12</td><td>0.33</td><td>0.19</td><td>0.1</td><td>0.23</td><td>0.31</td><td>0.25</td></tr><tr><td>GoogLeNet</td><td>0.12</td><td>0.12</td><td>0.12</td><td>0.09</td><td>0.22</td><td>0.16</td><td>0.1</td><td>0.12</td><td>0.22</td><td>0.16</td></tr><tr><td>Resnet50</td><td>0.18</td><td>0.12</td><td>0.11</td><td>0.1</td><td>0.16</td><td>0.17</td><td>0.11</td><td>0.11</td><td>0.2</td><td>0.17</td></tr><tr><td>Se-Resnet50</td><td>0.09</td><td>0.08</td><td>0.08</td><td>0.06</td><td>0.14</td><td>0.13</td><td>0.08</td><td>0.05</td><td>0.12</td><td>0.13</td></tr><tr><td>Humans</td><td>0.18</td><td>0.16</td><td>0.13</td><td>0.19</td><td>0.23</td><td>0.14</td><td>0.16</td><td>0.14</td><td>0.21</td><td>0.18</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>SqueezeNet</td><td>0.19</td><td>0.26</td><td>0.19</td><td>0.27</td><td>0.41</td><td>0.35</td><td>0.24</td><td>0.2</td><td>0.31</td><td>0.16</td></tr><tr><td>GoogLeNet</td><td>0.18</td><td>0.18</td><td>0.14</td><td>0.16</td><td>0.24</td><td>0.19</td><td>0.17</td><td>0.16</td><td>0.2</td><td>0.1</td></tr><tr><td>Resnet50</td><td>0.17</td><td>0.11</td><td>0.15</td><td>0.13</td><td>0.22</td><td>0.15</td><td>0.15</td><td>0.11</td><td>0.17</td><td>0.11</td></tr><tr><td>Se-Resnet50</td><td>0.11</td><td>0.11</td><td>0.1</td><td>0.13</td><td>0.18</td><td>0.12</td><td>0.09</td><td>0.08</td><td>0.1</td><td>0.1</td></tr><tr><td>Humans</td><td>0.18</td><td>0.14</td><td>0.18</td><td>0.16</td><td>0.14</td><td>0.1</td><td>0.14</td><td>0.14</td><td>0.1</td><td>0.11</td></tr><tr><td/><td/><td/><td/><td colspan="2">COMBINED</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>SqueezeNet</td><td>0.06</td><td>0.07</td><td>0.06</td><td>0.04</td><td>0.09</td><td>0.04</td><td>0.04</td><td>0.07</td><td>0.12</td><td>0.08</td></tr><tr><td>GoogLeNet</td><td>0.05</td><td>0.04</td><td>0.05</td><td>0.03</td><td>0.08</td><td>0.05</td><td>0.03</td><td>0.05</td><td>0.08</td><td>0.06</td></tr><tr><td>Resnet50</td><td>0.05</td><td>0.04</td><td>0.05</td><td>0.04</td><td>0.06</td><td>0.05</td><td>0.04</td><td>0.03</td><td>0.06</td><td>0.06</td></tr><tr><td>Se-Resnet50</td><td>0.04</td><td>0.04</td><td>0.04</td><td>0.02</td><td>0.06</td><td>0.04</td><td>0.04</td><td>0.02</td><td>0.05</td><td>0.05</td></tr><tr><td>Humans</td><td>0.12</td><td>0.07</td><td>0.09</td><td>0.05</td><td>0.15</td><td>0.17</td><td>0.09</td><td>0.1</td><td>0.25</td><td>0.11</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>SqueezeNet</td><td>0.06</td><td>0.09</td><td>0.07</td><td>0.11</td><td>0.14</td><td>0.12</td><td>0.1</td><td>0.06</td><td>0.08</td><td>0.05</td></tr><tr><td>GoogLeNet</td><td>0.04</td><td>0.06</td><td>0.05</td><td>0.07</td><td>0.08</td><td>0.07</td><td>0.07</td><td>0.05</td><td>0.05</td><td>0.04</td></tr><tr><td>Resnet50</td><td>0.04</td><td>0.04</td><td>0.06</td><td>0.05</td><td>0.07</td><td>0.05</td><td>0.05</td><td>0.03</td><td>0.06</td><td>0.04</td></tr><tr><td>Se-Resnet50</td><td>0.04</td><td>0.04</td><td>0.04</td><td>0.06</td><td>0.06</td><td>0.04</td><td>0.02</td><td>0.03</td><td>0.03</td><td>0.04</td></tr><tr><td>Humans</td><td>0.11</td><td>0.1</td><td>0.1</td><td>0.2</td><td>0.08</td><td>0.09</td><td>0.12</td><td>0.14</td><td>0.07</td><td>0.09</td></tr></table>

Mean human precision for the different entries grouping per label and filter

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">COLOUR</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>Used</td><td>0.96</td><td>0.97</td><td>0.97</td><td>0.97</td><td>0.85</td><td>0.74</td><td>0.88</td><td>0.88</td><td>0.57</td><td>0.96</td></tr><tr><td>Control</td><td>0.98</td><td>0.99</td><td>1.0</td><td>0.99</td><td>0.87</td><td>0.75</td><td>0.89</td><td>0.89</td><td>0.58</td><td>0.98</td></tr><tr><td>Online entries</td><td>0.91</td><td>0.88</td><td>0.86</td><td>0.91</td><td>0.75</td><td>0.69</td><td>0.8</td><td>0.81</td><td>0.54</td><td>0.9</td></tr><tr><td>Online Accepted</td><td>0.96</td><td>0.96</td><td>0.96</td><td>0.96</td><td>0.85</td><td>0.73</td><td>0.87</td><td>0.88</td><td>0.57</td><td>0.95</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>Used</td><td>0.96</td><td>0.94</td><td>0.95</td><td>0.96</td><td>0.94</td><td>0.89</td><td>0.95</td><td>0.65</td><td>0.97</td><td>0.92</td></tr><tr><td>Control</td><td>0.98</td><td>0.96</td><td>0.97</td><td>0.97</td><td>0.96</td><td>0.9</td><td>0.97</td><td>0.66</td><td>0.99</td><td>0.93</td></tr><tr><td>Online All</td><td>0.91</td><td>0.89</td><td>0.88</td><td>0.89</td><td>0.88</td><td>0.81</td><td>0.81</td><td>0.59</td><td>0.89</td><td>0.83</td></tr><tr><td>Online Accepted</td><td>0.96</td><td>0.94</td><td>0.94</td><td>0.96</td><td>0.94</td><td>0.89</td><td>0.95</td><td>0.65</td><td>0.97</td><td>0.92</td></tr><tr><td/><td/><td/><td/><td colspan="2">RESOLUTION</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>Used</td><td>0.95</td><td>0.95</td><td>0.97</td><td>0.92</td><td>0.91</td><td>0.78</td><td>0.82</td><td>0.9</td><td>0.56</td><td>0.96</td></tr><tr><td>Control</td><td>0.96</td><td>0.96</td><td>0.99</td><td>0.94</td><td>0.92</td><td>0.79</td><td>0.84</td><td>0.92</td><td>0.57</td><td>0.98</td></tr><tr><td>Online All</td><td>0.9</td><td>0.89</td><td>0.89</td><td>0.84</td><td>0.86</td><td>0.7</td><td>0.77</td><td>0.83</td><td>0.51</td><td>0.85</td></tr><tr><td>Online Accepted</td><td>0.95</td><td>0.94</td><td>0.97</td><td>0.92</td><td>0.91</td><td>0.78</td><td>0.81</td><td>0.9</td><td>0.56</td><td>0.96</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>Used</td><td>0.94</td><td>0.95</td><td>0.94</td><td>0.96</td><td>0.91</td><td>0.92</td><td>0.96</td><td>0.68</td><td>0.98</td><td>0.8</td></tr><tr><td>Control</td><td>0.96</td><td>0.96</td><td>0.96</td><td>0.97</td><td>0.93</td><td>0.93</td><td>0.98</td><td>0.69</td><td>1.0</td><td>0.82</td></tr><tr><td>Online All</td><td>0.85</td><td>0.87</td><td>0.85</td><td>0.87</td><td>0.83</td><td>0.83</td><td>0.9</td><td>0.63</td><td>0.88</td><td>0.72</td></tr><tr><td>Online Accepted</td><td>0.94</td><td>0.95</td><td>0.94</td><td>0.95</td><td>0.91</td><td>0.92</td><td>0.96</td><td>0.67</td><td>0.97</td><td>0.8</td></tr><tr><td/><td/><td/><td/><td>CROP</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>Used</td><td>0.8</td><td>0.84</td><td>0.74</td><td>0.77</td><td>0.88</td><td>0.75</td><td>0.64</td><td>0.88</td><td>0.66</td><td>0.96</td></tr><tr><td>Control</td><td>0.81</td><td>0.85</td><td>0.75</td><td>0.78</td><td>0.89</td><td>0.76</td><td>0.66</td><td>0.9</td><td>0.68</td><td>0.98</td></tr><tr><td>Online All</td><td>0.71</td><td>0.77</td><td>0.68</td><td>0.72</td><td>0.8</td><td>0.67</td><td>0.57</td><td>0.8</td><td>0.61</td><td>0.86</td></tr><tr><td>Online Accepted</td><td>0.8</td><td>0.84</td><td>0.74</td><td>0.77</td><td>0.87</td><td>0.75</td><td>0.64</td><td>0.87</td><td>0.66</td><td>0.96</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>Used</td><td>0.78</td><td>0.93</td><td>0.86</td><td>0.87</td><td>0.91</td><td>0.69</td><td>0.96</td><td>0.65</td><td>0.98</td><td>0.79</td></tr><tr><td>Control</td><td>0.79</td><td>0.94</td><td>0.87</td><td>0.89</td><td>0.92</td><td>0.7</td><td>0.97</td><td>0.66</td><td>1.0</td><td>0.8</td></tr><tr><td>Online All</td><td>0.7</td><td>0.86</td><td>0.79</td><td>0.79</td><td>0.82</td><td>0.62</td><td>0.88</td><td>0.61</td><td>0.89</td><td>0.74</td></tr><tr><td>Online Accepted</td><td>0.78</td><td>0.93</td><td>0.86</td><td>0.87</td><td>0.9</td><td>0.69</td><td>0.96</td><td>0.65</td><td>0.98</td><td>0.79</td></tr><tr><td/><td/><td/><td/><td colspan="2">COMBINED</td><td/><td/><td/><td/><td/></tr><tr><td/><td>bear</td><td>bird</td><td>cat</td><td>dog</td><td>fish</td><td>flower</td><td>fox</td><td>fruit</td><td>fungus</td><td>hippo</td></tr><tr><td>Used</td><td>0.89</td><td>0.76</td><td>0.8</td><td>0.65</td><td>0.83</td><td>0.62</td><td>0.75</td><td>0.86</td><td>0.69</td><td>0.93</td></tr><tr><td>Control</td><td>0.91</td><td>0.78</td><td>0.81</td><td>0.66</td><td>0.85</td><td>0.63</td><td>0.76</td><td>0.88</td><td>0.7</td><td>0.94</td></tr><tr><td>Online All</td><td>0.81</td><td>0.7</td><td>0.76</td><td>0.62</td><td>0.75</td><td>0.58</td><td>0.67</td><td>0.81</td><td>0.63</td><td>0.84</td></tr><tr><td>Online Accepted</td><td>0.89</td><td>0.76</td><td>0.8</td><td>0.65</td><td>0.83</td><td>0.62</td><td>0.75</td><td>0.86</td><td>0.69</td><td>0.92</td></tr><tr><td/><td>insect</td><td>lion</td><td>monkey</td><td>reptile</td><td>shark</td><td>spider</td><td>tiger</td><td>vegetable</td><td>vehicle</td><td>wolf</td></tr><tr><td>Used</td><td>0.75</td><td>0.89</td><td>0.9</td><td>0.79</td><td>0.89</td><td>0.76</td><td>0.96</td><td>0.61</td><td>0.96</td><td>0.73</td></tr><tr><td>Control</td><td>0.76</td><td>0.91</td><td>0.91</td><td>0.81</td><td>0.91</td><td>0.78</td><td>0.98</td><td>0.62</td><td>0.98</td><td>0.75</td></tr><tr><td>Online All</td><td>0.69</td><td>0.84</td><td>0.85</td><td>0.72</td><td>0.84</td><td>0.71</td><td>0.87</td><td>0.55</td><td>0.89</td><td>0.66</td></tr><tr><td>Online Accepted</td><td>0.75</td><td>0.88</td><td>0.9</td><td>0.78</td><td>0.89</td><td>0.76</td><td>0.95</td><td>0.61</td><td>0.96</td><td>0.73</td></tr></table>

Peer review (iclr2020):
This paper proposes a method to understand and compare the performance of DNNs classifier, which is different from the precise prediction in the notion of correct/wrong. With approximate minimal-entropy of input images, the classifiers can recognize this image so that different classifiers including human and DNNs will need different reduction methods (cropping, downsampling, color reduction )for the same image to give correct prediction and also will give us different performance in a same test dataset. By comparing the results with human’s and DNNs’, the author claims that it will have more challenges for DNNs in this laconic image classification task than human will have. For the motivation in this paper, the author tries to propose a new perspective to evaluate the robustness of image classifiers. Especially compared with humans’ understanding, this paper would like to rethink the influence of reduction for this task. However, it is not clear that why three reduction methods the author used can help with understanding the difference between humans and DNNs because when training a DNN for image classification, we usually use these methods to augment our training set, but for humans, it is a totally different story that how to recognize an image. For the theoretical demonstration, in this paper, the author uses approximating minimal-entropy to quantify the minimal content of an image DNNs or humans need to give correct category. The intuition of this method is suitable. But in section 3, the author didn’t give a clear demonstration of how to compute the entropy reduction in 3.1. I think if it is better to introduce how to measure 3.2 in detail, then 3.1 may be more clear. And it also makes me confused about the atomic reduction step in the last paragraph in 3.1. For the 3.5, I think the authors should focus on how to demonstrate MEPIs for humans more mathematically so that it will be more reliable. For the experiments, the author tries to answer two questions: 1. How does the entropy required by DNN and human classifiers compare??2. How do the classifiers perform in terms of precision for each others’ MEPIs? However, the experiments do not provide convincing evidence to existing approaches. First of all, for a single DNN, how different entropy reduction methods influence the classification? Secondly, how different reduction scales in the same model influence the results? At last, the comparison between different models should give a more visualized figure to illuminate the difference. It will be better to provide more ablation study experiments for this paper.