Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content - hence the name "laconic" image classification. The motivation for this task is to compare human and machine performance in a task that deviates from the standard ImageNet setup. To make different content reductions comparable, the authors measure the (approximate) entropy of an image via its PNG-compressed file size. As image transformations, the authors utilize quantization, downsampling, cropping, and a combination of the above. The authors find that convnets with higher accuracy are also more robust to these perturbations, and that humans perform well on the minimum-entropy examples of the networks (but not vice-versa).

Overall I find the comparison of human and machine performance interesting and hence recommend accepting the paper. However, there are multiple directions that could possibly strengthen the core experiment. Hence I only give a weak accept at this point. Concretely, these directions are:

- Do the results in the paper change if a different entropy measure is used (e.g., JPEG compression)?

- As suggested by the authors, training networks to be robust to the "laconic" image perturbations could be an interesting direction. For instance, standard data augmentation with the proposed perturbations would be a relevant baseline.

- To aid replicability and to compare the performance of different human test subjects, it would be interesting to conduct the experiment also on a crowdsourcing platform such as Mechanical Turk (as a complement to, not a replacement for, the university population in the paper).

- Also to add replicability and to make it easier to compare different human accuracy evaluations, it would be good to measure how well the annotators perform on the unperturbed images and on a simple noise transformation (e.g., Gaussian noise).

- It would be good to know how approximate the entropy measures in the paper are, e.g., to understand why humans perform worse in the "combined" perturbation setting.

- How did the results from the control group and the open / online evaluation differ?


In addition, I have the following suggestions for improving the paper:

- For the related work section, the authors may find the following papers on robustness of convnets to distortions interesting:

* Manitest: Are classifiers really invariant?
https://arxiv.org/abs/1507.06535

* Exploring the Landscape of Spatial Robustness
https://arxiv.org/abs/1712.02779

* Spatially Transformed Adversarial Examples
https://arxiv.org/abs/1801.02612

* Semantic Adversarial Examples
https://arxiv.org/abs/1804.00499

- It could be helpful for the reader to see some example images of the different transformations in the main text.

- Section 6: "[...] a bias in texture for images trained on the ILSVRC dataset [...]" - should this be "classifiers" instead of "images"?

- Section 6: "[...] would be interesting to explore in future" - insert "the" before "future"?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to understand and compare the performance of DNNs classifier, which is different from the precise prediction in the notion of correct/wrong. With approximate minimal-entropy of input images, the classifiers can recognize this image so that different classifiers including human and DNNs will need different reduction methods (cropping, downsampling, color reduction )for the same image to give correct prediction and also will give us different performance in a same test dataset. By comparing the results with human’s and DNNs’, the author claims that it will have more challenges for DNNs in this laconic image classification task than human will have.  
For the motivation in this paper, the author tries to propose a new perspective to evaluate the robustness of image classifiers. Especially compared with humans’ understanding, this paper would like to rethink the influence of reduction for this task. However, it is not clear that why three reduction methods the author used can help with understanding the difference between humans and DNNs because when training a DNN for image classification, we usually use these methods to augment our training set, but for humans, it is a totally different story that how to recognize an image.
For the theoretical demonstration, in this paper, the author uses approximating minimal-entropy to quantify the minimal content of an image DNNs or humans need to give correct category. The intuition of this method is suitable. But in section 3, the author didn’t give a clear demonstration of how to compute the entropy reduction in 3.1. I think if it is better to introduce how to measure 3.2 in detail, then 3.1 may be more clear. And it also makes me confused about the atomic reduction step in the last paragraph in 3.1. For the 3.5, I think the authors should focus on how to demonstrate MEPIs for humans more mathematically so that it will be more reliable.
For the experiments, the author tries to answer two questions: 1. How does the entropy required by DNN and human classifiers compare? 2. How do the classifiers perform in terms of precision for each others’ MEPIs? However, the experiments do not provide convincing evidence to existing approaches. First of all, for a single DNN, how different entropy reduction methods influence the classification? Secondly, how different reduction scales in the same model influence the results? At last, the comparison between different models should give a more visualized figure to illuminate the difference. It will be better to provide more ablation study experiments for this paper.

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary:
In this empirical study, the authors attempt to identify a minimal entropy version of an image such that the image may be correctly classified by a human or computer. The authors then compare the efficacy of a human and computer to maintain accuracy in the presence of a reduced entropy representation of an image. The authors find that machines are more sensitive to reductions in entropy due to image resolution than humans (as opposed to color or cropping). In addition, the authors find that humans are generally better at identifying minimal entropy images than machines.

1. Corruption results not surprising. 

Although the authors offer some intriguing methods, I found the results to not be compelling nor improve our understanding of the relative differences between human and machine perception. While identifying that humans are less sensitive to a reduction in resolution, this result is not terribly surprising given that networks are known to suffer from aliasing artifacts, e.g.

  Geodesics Of Learned Representations
  Olivier J. Henaff & Eero P. Simoncelli
  https://www.cns.nyu.edu/pub/lcv/henaff16b-reprint.pdf

2. Unclear what we learn from the method.

I am not too clear about what specific insights the methods provide in this paper. Estimating the entropy associated with each image corruption -- while interesting -- does not lead to any substantive analysis nor conclusions as far as I can tell. Given the lack of benefit to analyzing the entropy, I am left to really just consider these methods to be image corruptions that downsample the resolution or desaturate the images. These corruptions are not terribly novel with respect to previous work, e.g.

  Generalisation in humans and deep neural networks
  https://arxiv.org/pdf/1808.08750.pdf

  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
  https://arxiv.org/abs/1903.12261

To summarize, my feedback is the following:

1. Please justify the use of entropy to quantify the distortion. What does entropy provide above and beyond just parameterizing the distortion (e.g. image resolution, color saturation)?

2. Are there other results above-and-beyond sensitivity to image resolution that distinguishes human and machine in these experiments? These results seem to be largely known by just considering corruptions such as low pass filters, etc. presented in the above papers.


Minor Comments:

- The authors should provide a figure with example images in the main text showcasing how each method corrupts an image.
review_title: Official Blind Review #1

Meta-review: The paper proposes and studies a task where the goal is to classify an image that has been intentionally degraded to reduce information content. 
All the reviewers found the comparison of human and machine performance interesting and valuable. However the reviewers expressed concerns and noted the following weaknesses: the presented results are not convincing to support our understanding of the differences between human and machine perception (R1), using entropy to quantify the distortion is not well motivated and has been addressed before (R1), lack of empirical evidence (R2). 
AC suggests, in its current state the manuscript is not ready for a publication. We hope the detailed reviews are useful for improving and revising the paper. 
Decision: Reject