How hard are computer vision datasets? Calibrating dataset difficulty to viewing time
Authors: David Mayo
Jesse Cummings
Xinyu Lin
Dan Gutfreund
Boris Katz
Andrei Barbu
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
computer vision
cognitive science
datasets

Abstract: Humans outperform object recognizers despite the fact that models perform well on current datasets. Numerous attempts have been made to create more challenging datasets by scaling them up from the web, exploring distribution shift, or adding controls for biases. The difficulty of each image in each dataset is not independently evaluated, nor is the concept of dataset difficulty as a whole well-posed. We develop a new dataset difficulty metric based on how long humans must view an image in order to classify a target object. Images whose objects can be recognized in 17ms are considered to be easier than those which require seconds of viewing time. Using 133,588 judgments on two major datasets, ImageNet and ObjectNet, we determine the distribution of image difficulties in those datasets, which we find varies wildly, but significantly undersamples hard images. Rather than hoping that distribution shift or other approaches will lead to hard datasets, we should measure the difficulty of datasets and seek to explicitly fill out the class of difficult examples. Analyzing model performance guided by image difficulty reveals that models tend to have lower performance and a larger generalization gap on harder images. Encouragingly for the biological validity of current architectures, much of the variance in human difficulty can be accounted for given an object recognizer by computing a combination of prediction depth, c-score, and adversarial robustness. We release a dataset of such judgments as a complementary metric to raw performance and a network’s ability to explain neural recordings. Such experiments with humans allow us to create a metric for progress in object recognition datasets, which we find are skewed toward easy examples, to test the biological validity of models in a novel way, and to develop tools for shaping datasets as they are being gathered to focus them on filling out the missing class of hard examples from today’s datasets. Dataset and analysis code can be found at https://github.com/image-flash/image-flash-2022.

whole well-posed. Under review as a conference paper at ICLR 2023
HOW HARD ARE COMPUTER VISION DATASETS? CALIBRATING DATASET DIFFICULTY TO VIEWING TIME
Anonymous authors Paper under double-bling review
ABSTRACT
Humans outperform object recognizers despite the fact that models perform well on current datasets. Numerous attempts have been made to create more challenging datasets by scaling them up from the web, exploring distribution shift, or adding controls for biases. The difficulty of each image in each dataset is not indepen- dently evaluated, nor is the concept of dataset difficulty We develop a new dataset difficulty metric based on how long humans must view an image in order to classify a target object. Images whose objects can be rec- ognized in 1ms are considered to be casier than those which require seconds of viewing time. Using 133,588 judgments on two major datasets, ImageNet and ObjectNet, we determine the distribution of image difficulties in those datasets, which we find varies wildly, but significantly undersamples hard images. Rather than hoping that distribution shift other approaches will lead to hard datasets, we should measure the difficulty of datasets and seek to explicitly fill out the class of difficult examples. Analyzing model performance guided by image difficulty reveals that models tend to have lower performance and a larger generalization gap on harder images. Encouragingly for the biological validity of current archi- tectures, much of the variance in human difficulty can be accounted for given an object recognizer by computing combination of prediction depth, c-score, and adversarial robustness. We release a dataset of such judgments as a complementary metric raw performance and network's ability to explain neural recordings. Such experiments with humans allow us to create metric for progress in object recognition datasets, which we find are skewed toward easy examples, to test the biological validity of models in a novel way, and to develop tools for shaping datasets as they are heing gathered to focus them on filling out the missing class hard examples from today's is datasets. Dataset and analysis code can be found at https: / wit.hb.hhuuug.gggg com/ image- flash/ /image-meeeee flash-2022.
INTRODUCTION
Numerous efforts exist to build better evaluations for object recognizers. Broadly, these fall into four categories. Those that probe distribution shift, like ImageNetV2 (Recht al., 2019). Those that add scale like OpenImages (Kuznetsova et al., 2020). Those that explicitly attempt to make images more difffcullt for models by adversarially selecting them, like ImageNet-A (Hendrycks al., 2021) or adding artificial corruptions, like ImageNet-C (Hendrycks & Dietterich, 2019). And those that attempt to explicitly control for biases like ObjectNet (Barbu al., 2019). These are responses the fact that performance on standard benchmarks does not translate well to real-world conditions; 90% accuracy for one class ImageNet does not mean that the detector will achieve 90% accuracy for that class in one's home or on frames of movie. In all four cases, these efforts have no objective guide, no metric that evaluates how far they have progressed towards enabling models to generalize.
We set out measure an orthogonal quantity how difficult images in these datasets are for humans. Distribution shift and bias control won't 't on their own address this problem datasets are overwhelmingly easy compared to what humans capable of recognizing. And while scale helps, datasets are heavily skewed toward images that are easy for humans, the statistics of performance on such datasets may hide the real underlying performance trends of models on harder images.

---

Under review as a conference paper at ICLR 2023
Missing hard examples
Missing hard examples
1.0
100
Number
50
50
Difficuly
Score
Difliculty Score
(a) ImageNet image difficulty distribution
(b) ObjectNet image difficulty distribution
Figure I: The image difficulty distribution in ObjectNet and ImageNet. Difficulty here defined as how many participants failed to recognize given image across viewing times; easy images were almost always recognized even at short viewing times, while hard images were rarely recognized at short presentation times. Note that the difficulty of both datasets is roughly the same, and that hard images are under-sampled. Compared to what the human visual system can recognize, ImageNet and ObjectNet largely only test what can be recognized with short viewing times.
An objective metric by which to measure the difficulty of computer vision datasets has several advantages. First, we can determine if there are gaps in our datasets; perhaps certain difficulties are systematically undersampled. We find that this is the case: hard images are essentially missing. Moreover, merely aiming for distribution shift, even by changing how the dataset is gathered doesn't 't meaningfully change the difficulty distribution; ObjectNet and ImageNet were gathered from different sources (captured by Mechanical Turk vs the web), with different goals and additional controls for ObjectNet, yet their difficulty distributions are remarkably similar. Second, we can evaluate model scaling function of difficulty. We find that most model families scale poorly, performing well on the easy images but hardly improving on the hard images, with the exception of CLIP (Radford et al.). Third, it provides new kind of metric for biological plausibility, orthogonal to raw performance the error distribution, or how well networks predict neural activity. If a network is to be a model of the human visual system, not just an engincering model, then some quantity computed from that network should explain the observed difficulty scores. About half of the variance in the difficulty results is accounted for by a combination of c-score (Jiang et al., 2021), prediction depth (Baldock et al., 2021), and adversarial robustness (Goodfellow et al., 2015). Fourth, tools that measure difficulty could be incorporated into dataset collection and into how we report datasets and the overall progress of our community. In the long term, we intend to establish a dashboard giving perspective on object recognition from the point of view of difficulty.
To build this difficulty metric, we choose as proxy the minimum viewing time (followed by backward mask) that human viewer requires before being able to recognize the object in an image. Earlier readout is likely an indication that fewer mental resources were needed to recognize the image. After viewing the image, subjects have unlimited time to respond to a 1-out-of-50 forced choice task where they must identify the object class in the image that was shown. This metric is related to object solution time (OST) (Kar et al., 2018) explored in the neuroscience literature. We are of course not the first to carry out such viewing time experiments (Rajalingham et al., 2018). But, we do scale. with images from modern datasets, turn these results into difficulty metric with practical applications, then predict this difficulty metric from quantities computed from current networks, and show the scaling of current models. We hope that in the future, benchmarks will regularly report their difficulty distribution (they can do so for only a few hundred dollars with the tools we provide) and that collections of benchmarks will seek out datasets based on their difficulty distribution. Additional attention may need to be paid while collecting datasets to not eliminate hard examples; any quick consensus-ben based process with multiple annotators is likely to be heavily biased against including hard examples. Practically, when collecting datasets for domains where the cost of errors is high (the medical domain, the military, etc.). being mindful of the difficulty distribution and actively shaping it to fill out harder images may be critical to building confidence in the resulting models.
Our contributions are:
1. a dataset of 133,588 human object recognition judgments as a function of viewing time for 4,771 images from ImageNet and ObjectNet,
2. the distribution of image difficulties for ImageNet and ObjectNet relative to what humans can recognize, shown in fig. 1,
2

---

Under review as a conference paper at ICLR 2023
What did you see?
IMAGENET
ObjectNet
5,000 images
image durations
28 human responses image per image duration
2,500 per dataset
17ms, 50ms, 150ms, and 10s
Figure Overview of the experiment. (left) 50 images from object classes were randomly selected from both ImageNet and ObjectNet total 5,000 images; of which we analyze 4771. Images were cropped in a square around the object of interest and then shown to human subjects on Amazon Mechanical Turk and in controlled laboratory setting. (middle) Participants first cross for 500ms, then the image for either 17ms, 50ms, 150ms, or 10s, followed by a mask. After each image, subjects were given l-out-of-50) forced-choice task to identify the correct object class. (right) Each image was seen by 28 subjects, seven for each of the four image durations. No subjects saw the same image twice.
fixation
3. new analysis of model performance function of image difficulty,
4. a new metric for validating the biological plausibility of object recognizers, predicting image difficulty, and
5. new subset of images from ObjectNet and ImageNet sorted by difficulty for use in neuroscientific and behavioral experiments.
EXPERIMENT
We performed an experiment with human subjects on Mechanical Turk and in the lab in order to determine the minimum amount of viewing time required before subjects could identify an object present in an image. See fig. 2 for details. In what follows, we describe the stimuli, procedures, and validation of the online experiment with in-lab experiments.
2.1 STIMULI
selected 2,500 images from the ImageNet validation set (which contains 50 images per class; we selected all 50 for 50 classes) and 2,500 images from the ObjectNet dataset (ObjectNet is only test set). These images were evenly distributed among 50 object classes shared between ObjectNet and ImageNet. Since ObjectNet was gathered by participants taking pictures of objects in their homes, all of the object classes are household objects (see right panel on fig. 2 for a full list of the object classes) and all participants were likely familiar with all object classes used. The 50 classes hand-picked to minimize similarity between classes that could be confusing for experiment subjects.
At short presentation times, subjects do not have time to fixate on multiple locations. Off-center objects would be recognized by peripheral vision. To climinate this effect, we cropped all images around the target object using the ImageNet validation bounding boxes and new bounding box annotations for ObjectNet. In each case, we produced square 224 by 224 image, padding with black needed. Details about cropping can be found the appendix. Note that this eliminates clutter and focuses only on the appearance of the object; we discuss this limitation in the conclusion.
2.2 PROCEDURES
Subjects consented and provided warning about photosensitivity flashes. Only subjects with normal corrected vision were included in the experiment. Written instructions and sample video of the experiment were then provided. Subjects then read through list of the 50 object categories.
3

---

Under review as a conference paper at ICLR 2023
The Mechanical Turk experiments had additional calibration steps to determine the user's screen size (utilizing the fact that credit cards have a standardized size throughout the world) and distance from the screen (using a blind spot test) Li et al. (2020). Images were all shown at size that would take up 8 degrees of visual angle in the participant's field of view. Further details about the setup steps and calibration are available in the appendix.
For an overview of the procedures, see the middle panel of fig. 2. Each trial had a randomly selected presentation time, either 17 ms, 50 ms, 150 ms, 10 seconds; timings were selected to account for participant's 's use of a 60Hz screen and to probe fine-grained distinctions between image difficulty most apparent short timings. image. for one of the four timings. Immediately after the image, phase gradient backward mask (Breitmeyer, 2007) was shown for 500 ms to disrupt further processing of the image. Details of the mask generation are described in the appendix. Participants were then presented with grid of the names of 50 object categories and were asked to click on the object that they saw.
each trial, a fixation cross was shown for 500 ms followed by the
After each presentation, participants had to choose the identity of the object that was presented. As this was forced-choice experiment, subjects were told to select their best guess they are unsure. The order of the object classes in the grid was randomized at each trial. Subjects were given an unlimited amount of time to make a choice, although this time was recorded and appears to be anti-correlated with performance (quick decisions were likely to be more accurate than slow ones).
Experiments were counterbalanced. total, each image was viewed by the same number of subjects at the same number of viewing times. Any one subject did not the same image twice any duration. Each participant viewed 50 images, one from each class, 25 from ImageNet and 25 ObjectNet, evenly spread across the four viewing times. The order of images, viewing times, and classes were randomized for each participant. Participants were allowed to complete several experiments; for those who did, we ensured that they saw disjoint images each time. total, 1,495 workers took part in the experiment. Subjects on Mechanical Turk were compensated at over $ $10 per hour. In total, we collected 140,000 trials, after which we discovered that a small number of images (229 images) were either incorrectly annotated, incorrectly cropped, or workers circumvented our automated means to ensure that no two workers saw the same image twice. This resulted in 133,588 trials for 4,771 images (28 presentations of each image, each of 7 subjects seeing each image at one of four timings).
2.3 IN-LAB VALIDATION
Great variation exists between monitors and browsers, with some showing the same images for nearly twice as long at short presentation times. Subjects as well can vary in their attention alertness. To account for this, we validated the results with experiments in the lab. These followed the same procedures described above using the same interface with a gaming monitor (27 inch LG-27GN950-B monitor, with a Ims GTG response time), on a machine with an NVIDIA 3060TI, using Chrome 102, running 144Hz. We recorded the screen at 200fps and found that the presentation times accurately reflected the time periods during which this monitor showed each image.
We selected 200 images from the 5,000 used in the previous experiment, evenly distributed across the 50 classes and the two datasets (ImageNet and ObjectNet). Each participant saw all 200 images, random presentation times in random order; they never saw the same image twice, and could only participate in the experiment once. Subjects were instructed to stay at fixed distance from the screen. An experimenter was present throughout to ensure that subjects were not distracted. Lighting was kept constant by closing the blinds. In total, 12 subjects (6 male, 6 female) participated in this experiment, and as can be seen in the next section, there was widespread agreement between the in-lab and Mechanical Turk experiments. It took subjects just short of an hour to complete the 200 images. Subjects were MIT graduate and undergraduate students compensated $20 for their participation.
RESULTS
We considered an image as recognized at some viewing time when half of the participants could classify it Chance on the I-out-of-50 task is 2%; even a single correct response is an indication that something could be recognized. Typical images by minimum presentation time are shown in fig. 3.
4

---

Under review as a conference paper at ICLR 2023
Increasing difficulty
17 ms
s 50 ms
150
10
Figure Images as function difficulty, using the minimum viewing time before they were reliably recognized, i.e., when more than half of subjects were correct. Harder images are more atypical, have more difficult lightning, more occlusions, and are sometimes more blurry. Additional examples are available in the appendix and online. Most datasets contain images like those recognized at short timings.
1.0
1.0
Dataset
Dataset
ImageNet
ImageNet
Accuracy
0.6
ObjectNet
Accuracy
0.6
ObjectNet
0.4
0.4
0.2
0.2
17
50
150
10000
17
150
10000
Presentation time (ms)
Presentation time (ms)
(a) Mechanical Turk experiments
(b) In-lab experiments
Figure 4: Accuracy as function of presentation time. were presented at timings. Results for both conditions were similar, although in-lab experiments achieved nearly 100% accuracy with 10 second viewing times, while halving the performance 17ms compared to MTurk. The accuracy improvement at long times is likely because of the more controlled conditions with fewer distractors in lab. The lower accuracies short timings likely due to issues with displaying images at short timings: some monitors are very slow and can fade the image in and out effectively displaying it for twice the intended 17ms.
The same images (5,000 online, 200 in lab)
Images that are quickly recognized by humans, easy images, are most similar those seen in current datasets, while more difficult images include occlusion or difficult lighting.
overview of accuracy experiments broadly agree with one another. In-lab experiments have higher variances due to having 20x fewer images and half as many subjects per image. In lab, the performance on short timings was significantly worse, half of that seen online. We believe this is largely due to slow monitors which display the image for significantly longer than 17ms, roughly twice as long. Recording screens with high-speed cameras supports this hypothesis. At the high end, subjects in lab were nearly 100% accurate, 10% higher than online. We believe this likely has to do with how distracted subjects were. In what follows, we focus on the Mechanical Turk experiments due to their scale.
function of viewing time online and in the lab shown in fig. 4. Both
Human performance drops off steeply when difficult images are shown for shorter viewing times; see minor variations in difficulty.
fig. 5. This makes the experiment sensitive
5

---

Under review as a conference paper at ICLR 2023
10
Accuracy
Presentation Time (mS5)
0.4
Subset
Figure 5: Human accuracy on Mechanical Turk as function of the difficulty subset. Colors denote presentation time, while the x-axis denotes subsets of images that require least the stated viewing time to be recognized reliably. Accuracy drops off steeply when hard images are displayed at shorter time intervals.
0.6
0.4
0.4
0.4
0.4
Human Accuracy
Accuracy
Accuracy
Accuraty
57
ResNet-50
SWSL (ResNet-50)
MAE (VIT-B/16)
CLIP (ViT-L/14)
Figure 6: Accuracy four recent models on ImageNet (solid) and ObjectNet (dashed) as a function of image difficulty (extensive experiments with numerous additional models are available in the appendix). Note that this understates human performance, as it includes the Mechanical Turk, not the in-lab experiment results. Even for the hardest images, humans have nearly perfect performance in lab. Model performance drops off significantly for the harder images. Harder images also show a much larger gap when testing on ObjectNet.
3.1 How HARD ARE TODAY'S OBJECT RECOGNITION DATASETS?
The minimum viewing time required for reliable recognition is a proxy for image difficulty. Datasets today not gathered to control for difficulty and, indeed, when plotting the difficulty of images in ImageNet and ObjectNet, we find that the difficulty curve for these datasets highly skewed; see fig. 1. Rather than plotting four bins, one for each viewing time, we plot more fine-grained quantity: the total number of incorrect responses out of the 28 presentations of each image (7 participants at timings). Images with few incorrect responses are easy: all participants at all timings could recognize them, even the short timings. Images with many incorrect responses are hard: few participants could recognize the images only a few longer timings.
Hard images are vastly underrepresented in today's datasets. The difficulty distribution of ImageNet and ObjectNet are very similar to one another. This is despite the fact that the latter was collected in a manner intentionally intended to highlight more diverse and less biased images. Merely collecting more images does not appear to help. Instead, we must develop new tools to guide dataset collection toward more difficult exemplars; we describe such tool and workflow below.
3.2 WHAT WE CAN LEARN ABOUT MODELS FROM HARD IMAGES
Machine accuracy varies as function of the minimum viewing time required to recognize the object an image. Most models, but not all, see a significant performance dropoff between the easy images and the hard images, fig. 6. See the appendix for extensive results for dozens of models broken down by image difficulty. Note that this understates human performance as it shows the Mechanical Turk results. In-lab, even for the hardest images, humans have nearly perfect performance.
These results also show that the gap between ImageNet and ObjectNet performance increases as image difficulty increases. Likely, many phenomena such as distribution shift much more acute for harder rather than easier images. Datasets intended to challenge object recognizers would benefit from sampling hard images, rather than vastly oversampling easy images they do today.

---

Under review as a conference paper at ICLR 2023
roooggh and na lably by faaturre when vianed for $55
Cupp becovonn tmagpoheee and chopech accuracy
2 pascbol
Orjectiec accurady recognzzed raliatly by humans when viewed for 17mm
e1
Figure 7: Model robustness trends as a function of image difficulty. The horizontal axis shows the accuracy gap between ImageNet and ObjectNet for easy images. The vertical axis shows the accuracy gap between ImageNet and ObjectNet for hard images. Each point is model. Arrows connect models that are part of the same family, starting with smaller models pointing to larger ones. Points on the diagonal have the same gap in performance between ObjectNet and ImageNet. Humans hover around zero with some small noise, they are robust with respect to the distribution shift between ObjectNet and ImageNet. Models that are part of the same family with increasing number of parameters are connected by lines. As model size increases, plain ResNets improves on easy images, but increase their gap on harder images. SimCLR with increasing model size is horizontal, it narrows the gap on easy images, but hardly improves on hard images. CLIP stands out. Larger CLIP models tend toward having little no gap between ImageNet and ObjectNett, although, this is not monotonic and appears to reverse itself as the network grows in size.
Image difficulty can tease apart differences between recognition models that would otherwise be lost because of the skewed underlying difficulty distributions in current datasets. In fig. 7, we plot numerous object recognition models contrasting their performance the casiest and hardest images. Rather than computing the absolute performance of models, that would naturally favor larger models with larger training sets, we measure the gap in performance between ImageNet and ObjectNet. A more robust detector is one that has a smaller gap, even if its performance is lower, scaling models up (both in parameter size and training set size) is well understood. Models that are part of the same family are connected by arrows starting with the smaller variants pointing the larger ones.
Humans hover around zero; they are robust with respect to the distribution shift between ObjectNet and ImageNet. Some model families are horizontal, like SimCLR, or even have negative slope. This shows that as they scale, their performance is increasing on the easy images, but not on the harder images, in the case of negative slopes, it is widening the gap harder images. CLIP stands out, and most closely approaches the human results. Extrapolating the performance of models on such a graph shows that many current model families likely to stop or radically slow down performance improvements on hard images, much less optimistic story than if one merely considers aggregate performance. At the same time, CLIP shows promise, although it too appears eventually begin to move away from being robust to distribution shifts as model size increases. Note that these experiments use the largest CLIP models available to the public; other large models described in the CLIP publication may have even more optimistic results.
3.3 EXPLAINING IMAGE DIFFICULTY AND TESTING HOW SIMILAR MODELS ARE TO HUMANS
If models are to not just perform well, but to also process images in ways that are similar to how the human visual system does, then they should contain a proxy for difficulty. Most models today not recurrent, so no direct analog exists to viewing time (although, even if they were recurrent, is unclear if directly taking the number of iterations of the model is the correct analog of viewing time).
7

---

Under review as a conference paper at ICLR 2023
1.00
Prodiction depth
14
ImageNet ObjectNet
ImageNet ObjectNet
Cscore
igaming speed
0.75
12
Epsilon
0.10
0.50
10
0.05
ImageNet ObjectNet
0.00
0.00
50
150
10000
17
50
150
10000
17
50
150
10000
Subset
Subset
Subset
C-Score
Prediction Depth
Perturbation
Figure 8: The correlation between three metrics and image difficulty. On the x axis images subset by the minimum amount of viewing time, in ms, required for the majority of participants before they recognized. All three metrics are correlated with difficulty. Hard images are learned later in model training. predicted by later layers, and need much smaller perturbations attack.
Minimum viewing time judgments provide an independent and complementary metric by which to evaluate how similar models are to human brains, in addition to their behavior, error pattern, and ability to explain neural recordings.
We investigate three quantities computed from models that could be used to explain the difficulty judgments: c-score (Jiang et al., 2021), prediction depth (Baldock et al., 2021), and adversarial robustness (Goodfellow et al., 2015). C-score is consistency score computed while the network is training; we use the learning speed proxy to compute it. Prediction depth measures the earliest layer at which network produces its final prediction. And adversarial robustness refers to the perturbation, needed to fool the network on a given example. See fig. 8 for an overview of the correlation between these metries and difficulty. The same network with ResNet-50 architecture trained on ImageNet was used for all experiments presented here. Note that e only computed over the correctly classified images.
This analysis reveals that images that require more viewing time for humans are harder for networks in several ways. They are learned much later in the training process. They are predicted by later layers in the network. And they require much smaller perturbation to create adversarial examples. This underscores that human minimum viewing time has many practical consequences for how networks process images. At the same time, these metrics alone do not explain all of the difficult judgments. Logistic regression including all three metries classifies images into the four difficulty subsets with 48.5% accuracy, far above chance, but leaving plenty of room to do more. New metrics for understanding how networks process images could be validated against this data.
4 RELATED WORK
Image difficulty Image difficulty has previously been investigated primarily from the perspective of models. Jiang and Zhang et al. constructed a metric c-score (Jiang et al., 2021), to measure how much model was required memorize to learn particular image instance. It also proposed learning speed proxy for image difficulty; more difficult images are classified correctly later in the course of training. Baldock et al. measured image difficulty by measuring the model layer depth at which a model makes its final prediction for images (Baldock et al., 2021). More difficult examples can only be decoded the model final prediction from later layers. Agarwal al. introduced the variance of gradients (VoG) difficulty metric (Agarwal et al., 2022). VoG is computed over the course of training, similar to the c-score learning speed proxy. but it uses the variance in the gradient updates over training instead of the models final prediction. More difficult examples have higher variance. Since each of these methods depends on model performance rather than human performance they find ObjectNet significantly more difficult than ImageNet, unlike our human presentation time metric as show in fig. 1.
Dataset design As field, we have measured our progress by models' performance on tests created by splitting a random subset of images from large-scale image datsets. Almost all of the most influential datasets were created by web scraping mubicly-available images and labeling them based on a consensus among human annotators (Russakovsky et al., 2015). More recently, researchers have realized that there is significant value to testing models' abilities recognize objects out of
8

---

Under review as a conference paper at ICLR 2023
domain. This has included attempting to recreate existing test sets from new data, ImageNetV2 (Recht et al., 2019), constructing adversarial test sets for models, ImageNet-A (Hendrycks et al., 2021), well as adding corruptions to existing test sets, ImageNet-C (Hendrycks & Dietterich, 2019). The ObjectNer (Barbu et al., 2019) test set took this one step further by collecting a new set of images sampling uniformly across set of visual controls. In this work we find that the viewing time difficcltyy distributions between ImageNet and ObjectNet are similar, but models demonstrate larger robustness performance gap on hard images.
OOD model performance Prior work studying object recognition models performance has found a linear trend in performance improvement on ImageNet and other OOD datasets (Recht al., 2019; Taori et al., 2020; Barbu et al., 2019). This linear trend can be beaten slightly when massively scaling up the quantity of training data used. Recently, the multimodal models CL.IP (Radford et al.) and LiT (Zhai et al., 2022) have demonstrated a break from this linear trend, greatly increasing out of distribution generalization performance by using multimodal learning.
Limited viewing time Geirhos 2018 et al. (Geirhos et al., 2017) studied human and machine performance under limited presentation time with test set images made more difficult by adding various image corruptions. Finding that humans are far more robust to these difficulty changes than machines. Rajalingham and Isaa et al. (Rajalingham et al., 2018) studied human and primate object recognition performance under limited presentation time by presenting 100ms images finding models unable to match image-level behavioral patterns of primates.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
Rather than focusing on scaling, distribution shift, or control for biases alone, we should also focus on dataset difficulty explicitly. Today's s datasets skew toward being too easy by undersampling hard images, ObjectNet was designed for distribution shift and bias control and was not collected from the web, yet its distribution of image difficulties is remarkably similar to that of ImageNet. By focusing on ways to measure dataset difficulty as datasets are collected, we can better calibrate the entire community, and create the resources needed to push object recognition forward. addition to just creating better datasets, understanding performance as a function of difficulty reveals radically different scaling curves for different models and approaches. can also provide subsets of images that highlight different types of processing for neuroscientific or behavioral experiments.
Our approach to measuring the difficulty of object recognition tasks focuses specifically on object instances, rather than other factors that also add complexity such as saliency and clutter. By cropping images around the object, we make gooure segmentation far easier and separate out the issue of the difficulty of this specific instance of an object opposed to the difficulty of finding this object within a cluttered background.
The particular notion of image difficulty derived from this experiment is just one metric of difficulty. As we describe in the introduction, other metrics exist which can be computed automatically given models that recognize objects. But those metrics rely on the models and change models change. The metric we present here is both absolute, it is calibrated to what humans can achieve, and model- agnostic. Models change over time, far faster than datasets do, there is real danger when designing datasets that their difficulty will be tuned to specific models, which can be avoided by using measures related to humans.
While we performed extensive experiments to validate the approach presented here, that having been established, measuring the difficulty of any one dataset is now easy and cheap. One can sample few hundred images and run an experiment on Mechanical Turk. This only costs on the order of hundreds of dollars per dataset and can be carried out quickly. The more critical the dataset, and the cost of object recognition failures, the more important it is to do so. Of course, while doing so experimenters should ensure that participants are treated fairly and paid appropriately.
Roughly 4 weeks of compute time on 2 machines with 8 TITAN RTX were used to generate the results, largely in computing c-score, prediction depth, adversarial robustness, and finetuning models. We release our experiments under the Creative Commons Attribution 4.0 license.
This paradigm for measuring difficulty could be adapted to other vision tasks such as segmentation or optical flow. Other tasks are out of scope, like visual search, they literally require multiple saccades. Calibrating our field to what humans can do across a wide range of tasks, datasets, conditions, remains significant challenge, but one that think can now be addressed.
9

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We received an IRB exemption for this study as poses minimal risk to participants. No personally identifiable data was collected from any subject. Both in person study participants and Mechanical Turk participants were compensated appropriately. We hope that more objective measures of dataset difficulty will lead to more dataset equity, As it appears that gathering more difficult datasets requires more deliberate effort, an objective metric could hold creators to account. For example, datasets for training autonomous cars could represent different socioeconomic conditions equally, but more time might be invested in gathering higher-quality data in more affluent conditions. This can now be brought to light by computing the difficulty of each subset of the data, ultimately we hope leading to fairer datasets.
REPRODUCIBILITY STATEMENT
We publicly release all data and analysis code used in this study at the URL https: /tt///gitte github. com/ mage--a f a ash/ mage- aasee-flash-2200 trials used in our analysis and code for reproducing the results and figures presented in this paper. We also included scripts for ImageNet and preprocessed images for ObjectNet to reproduce the stimuli images used in this study.
This includes the responses from all experiment
We provided a detailed description of our studies procedures, including screenshots, the main text and the appendix. After publication, plan fully release the web tool we developed for performing these experiments.
REFERENCES
Chirag Agarwal, Daniel D' D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern abs //0./.0000000000 11600.
Recognition (CVPR), 2022. URL https: /ttpps:///rxiv..o
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example diffieulty. In Advances in Neural Information Processing Systems (NeurIPS), 2021. URL /ettps://rooceedeee deebbaasdd3ceeebdddeee 7addee2dd03llll Paper.
neurips cct cc/papprr/222//iL
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale cias-controlled dataset for push- ing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL https: http://ppoceeee...//ee / www.rdiiinnnnnnnununeeeee neurips....... www.bbaaaeeeaaaaa11aaeaaa
wwtpurpss.ccppppeeeeee cc/paper/2019/
Bruno Breitmeyer. Visual masking: Past accomplishments, present status, future developments.
Advances in Cognitive Psychology (ACP), 2007. URL https: http:s://wwwwwe nebi mnm. nih. gov/
ppte//articiless/MM228111
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceddiggs of the 37th International Conference Machine Learning (ICMI), 2020. URL https::t////...... /ptpceee/ppceeeeeeeeee ngs ml 1r...... w.r.ppes//vv.... v chen20j html.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021. URL https: httpa://openaccess http://cppccccceeeeeeeeescc.ssse theevf. com wttmm//ccneet//cc//22221 papers/ Chenn An Empiricaiiii... https//eeheeeeeaaaeaaaaaaaaaaaae www.ooffo_rrinnn_e__e__e__eee stw.rriiinnnnellff-eeeeseeeeeee
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robust- image perturbations. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL attpa://proceecngg ietiigs.nuuri... httpp/ppps.cppppppe////////////e 98b1 98b17f066d5d997666 9fb8ae4 4708884470811111 waw.eee1.---pppeeeap
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations (ICLR), 2021. URL https: httgg/abs/2200000 2010 11929.
Robert Geirhos, David H. Janssen, Heiko H. Schütt, Jonas Rauber, Matthias Bethge, and Felix A. Wichmann. Comparing deep neural networks against humans: object recognition when the signal gets weaker, 2017. URL https: arxiv. wwt./o///////.ooooooo..oooso.sas wwwp////....///o..//./// 06969.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations (ICLR), 2015. URL /stp://arxil http://aaxxxv...aa////os/ss
6572.
Kaiming He, X. Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016. URL https: http:////ppaccccss/sm thecvf...... wwt/coo////////oo httc//tnntttttceeeeeeeeeeeeeeeee cctpr_2016/papeeeeeeeeeo He_ He_Deep_RRet Res iesaadalla Leaiiaan__errnnn____eeee Cttpe__CeE________2_22 1w101_pppe...pp.eeee pdf.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019. URL https: http//iiivv..////// org aaa//b/////// abs/1807.... 01697.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEFF/CVE Conference on Computer Vision and Puttern Recogni- tion (CVPR), 2021. URL https: htpp///eeccccccseeeeeeeee wtwrseccsesssttteeeeeeeeeeeeeeee thecvf com/ content///////ooo CVVR20221 Natural Adversariaa Cec 2021 pappp_____2ppppprr.r.....
pdf.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification, 2017a. URL https: 09844,
//doi orr/ 0. 48550/arxiv
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017b. URL httph:ttt//// wwwpoccess.cs...ppoppopc........ thecvf com cttp//ttttttccceeeeeeeeeeeeeeett Huang waweelly__ooneeee___oooooooo
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data overparameterized models. In Proceedings of the 38th Interna- tional Conference on Machine Learning (PMLR), 2021. URL https::/pprocedd:: press/ /ooss/v139 1919////30 jiang2kk. html.
mlr.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nature Neuroscience, 2018. URL https: /www nature com/ articles/
s41593-019- 100--009--39----
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience, 2019. URL https: / /wwww//wwwww/////////////ww///// nature. wwwaaaeeeteeeaeaaaaaaaaeaaaaaaae com/articles....e
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong. Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescout-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L Yamins, and James DiCarlo. Brain-like object recognition with high- performing shallow recurrent (NeurIPS), 2019. URL httppp:/ppooceeeiii.. .euriipeeuspp.. httpsccc/ppppppe cc/paper//0...///// 9/ file/ 781 2813d15 72l.
In Advances in Neural Information Processing Systems
11

---

Under review as a conference paper at ICLR 2023
Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuses Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 2020. URL wttpt::p///// /ttp/a.xvvvoooo/.o pdf /11111111 00982. .df.
Qisheng Li, Sung Jun Joo, Jason D. Yeatman, and Katharina Reinecke, Controlling for partici- pants' viewing distance in large-scale, psychophysical online experiments using virtual chinrest. Scientific Reports, 2020. URL https: //parr/paaa///// nsf. gov http.//...///bbbb1///111111.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, (ICML). URL http:::/prooceediiiige mlr. press/ v1 3 radford21e .ttml.
Rishi Rajalingham, Elias Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James Dicarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the the--hhean art deep artificial neural networks. The Journal of Neuroscience, 2018.
URL wttph:tpt//.oo....p. /www jneurosci orgg hwtp///on/////////o///oo/ooooctt /38/ 7255..
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (PMLR), 2019. URL https: http///poooeeeeneeeeeeeeee/cesee mlr. press/v97 recht 9a. html.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015. URL https: www. image--aae........ net. org.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015. URL
http://arxiv> org/ abs /1409 1556.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. URL https: http://rooocdddeee iig.eeddiii....d.eeoe ngs mll www.pprsssvvve.. / ttan 9a html.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL https: https:s//pocccepigtttge////////t http:///pccceeccceeeeee///c neurips. pdf.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL https://oeenaccesa http://pwwwacccss..e..eecceece www.cf. wttp:/f..oo///oeettttttme content_
http://ppps.t..aaaaaaaaaiaaaiiia pdf.
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, 2019. URL http://arxiv. 00546.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image image text tuning. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. URL https: // //axxvxiv../////////////////aav/ org abs/
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 IMAGE CROPPING PROCEDURE
1. We draw bounding box around the object (we use existing bounding boxes for the ImageNet validation set and collect our own bounding boxes for ObjectNet from MTurk).
2. We initialize the cropping box to be the bounding box.
3. If the cropping box does not form a square, we extend the shorter side of the rectangular cropping box to form square. If the image is not large enough to extend the shorter side of the cropping box, we pad it with black pixels to form a square.
4. We crop using the cropping box for the image. The cropped image will be square.
5. We resize the cropped image to be 224x224 pixels.
A.2 MASK GENERATION
The masks were generated following the procedure used by (Kar et al., 2019). Specifically, Fourier transform was applicd to each image to obtain the magnitude and phase components. Then, a random array with elements sampled uniformly from [0, 1] was added to the image phase component after which the magnitude and phase components were recombined via an inverse Fourier transform to produce the mask. Each image was paired with its particular mhase-scrambles mask in the experiments.
A.3 EXPERIMENT PROCEDURE AND PAYMENT
Participants both in the lab and on Mechanical Turk were presented with a document informing them of the purpose, privacy, and risks associated with the experiment and soliciting their consent to participate (see fig. 9). Participants were then instructed as to how to carry out the experiment and were shown an example video as well the list of image classes for their review before beginning. They were informed that they would not need to memorize the classes as the classes would be shown to them after each video. Participants were also encouraged to take breaks should they feel fatigued or otherwise uncomfortable. Example instructions are shown in fig. 10
After giving consent and reading the experiment overview. participants then completed two calibration steps for to estimate the size of their monitor and their distance from the screen for us to then size the videos appropriately to degrees of visual angle. First, the participants are shown an image of a credit card and are asked to card of their own to adjust slider to change the size of the card on the screen to the size of their card. Since credit cards are the same size around the world, this allows us measure the pixel-to-inche ratio of the participant's monitor. Next, the participant completes a blind-spot test (Li et al., 2020) that allows us to estimate the distance they are sitting from their screen. Together, these two measurements See fig. 11 for images of the calibration steps.
sufficient compute the desired video eccentricity.
The estimated hourly wage for participants on Mechanical Turk and in the lab $10/hr and $20/hr respectively with approximately $15,000 spent in total on participant compensation.
A.4 IN-LAB EXPERIMENT RESULTS
To corroborate our Amazon Mechanical Turk results, we selected 200 images shown to Turk workers conduct the same experiment in a controlled laboratory setting. 12 individuals came to participate in the experiment in which they viewed and responded to all 200 images on our 144Hz refresh rate monitor with 1 ms gray-to-gray time. After conducting the experiment, 3 individuals had seen each image at each of the 4 presentation times. When compared to the MTuurk results for those same 200 images, the comparison is much as we would expect. The In-Lab accuracy with shortest image duration (17ms) is less than on MTurk which can likely be contributed to the use of our new, high refresh-rate -aate monitor in the controlled environment. It is likely that MTurk workers' personal computers differ in their graphics presentation abilities which may result in the image being visible for slightly greater than 17ms on some monitors. On the other end, the in-lab experiments reported higher accuracy at the longest image duration (10s) which is also unsurprising as the in-lab participants
13

---

Under review as a conference paper at ICLR 2023
Table 1: Dataset statistics
Number of responses
133,588
Number of images
4,771
Number of presentaiton durations
4
Number of response per image
28
Number of ObjectNet images
2,415
Number of ImageNet images
2,356
Number of participants
1,495
Informed consent to participate in this study
of adlentticc ressarch project decli complete voluntary. There no
Mientilyy you
The only infor motiiciion will have, aciitin your resparisa, the time which you comna snod the surviry gersolo van ider oiriitttee alsuut your computer such sn mncok and browio vendin unbber
The reule of researc vay the prnoented ae edientifc mertingg--r
puslaned in macientticc purmai..
collected experiment released actentirc comminnite the public.
not have photoss aptapay serittivity Haashing ligtts
Figure 9: Informed consent page shown to participants before beginning the experiment.
completed the task in controlled environment with distractions and are likely more inclined to take the task seriously and stay focused. The results show no significant differences in accuracy at the intermediate image durations. See fig. for side-by-side comparison between MTurk and In-Lab results.
A.5 DATASET STATISTICS
We collected 28 human responses for each of 5,000 images (2,500 from ImageNet and 2,500 from ObjectNet). Alter reviewing response, 229 images were removed due either being unrecognizable, mislabeled, or having been seen by the same worker twice despite safeguards in place to disallow it. Additional dataset statisties listed in table 1.
A.6 FINETUNED MODELS
Here we list details regarding training//inetuningg procedures for the model results reported in the paper.
A.6.1 MODEL. TRAINING PROCEDURE
Pretrained models weights were instantiated using publicly available model checkpoints, either using torchvision found on the model's source repository. The models-with the exception of CLIP- were then finetuned using subsets of the ImageNet training and validation sets containing only the 50 classes chose use in the psychophysics experiments. The models were finetuned for 90 epochs with an SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and weight decay coefficient of 0.0001. The learning rate decayed by factor of 2 every 9 epochs.
14

---

Under review as a conference paper at ICLR 2023
Task Overview
the multe choice st of eluects will the same for the inegeaa (ake
minute read these object names you are uniarnal for with category
contustngg do cuick search web ROSORES, they
ecamp imagee..
owing:
You tertipuuiing exper mant whilch stuatying people's image
beckpack
image np
bench
butcher's lestte chenver
computer ricASO
docomet
plower dell
included
exompole plays,
ask binefly change
Ofhict opt
hal dryor
hammee
croes When
clodhee trnn
tenpattede
lemon
mage on object and ihen change to image random petfern.
lppedic
match (La, matcherto))
mag
After you sen the imagge wes will you what object aru saxo in the imaae.. and focused intetdiattely
meciklion
paciloos
You will only abila to the once,
tored
butte
Row
preasing play, object that you thar's okay,
plaetic hag
Eortab
muligie cholke of poesiole otdects, think sbout you sow, and takk your best quees There is time limtt for your response, but you have to respord before move
printer
rocket lennia raclec))
Wabte (aanner bent
selavbiicn [TV]
sailles
watch
Water
example
whiictle
these oprions aod. The der the oprioos the rable randomized but al 160 oprions will always be vis ble Please take the time find the correct nover you know what it you
the sespones page, you
what
guess. We will
know if aru nort trying to answerr correctly and your submissiinn will rejected,
seriounly.
olt acts you now Wi resect taiee about 60 minutees You take brea los time. you feel like you are lited or having troubll focuding please take a break and
exparimnnt you well at images
after submitting a resporse for an imrage and totfore clicking play the nest Reagpo the same link, will be ablle pick up where you left off,
IP hesse
you
long
We would sise Boe poesible.
roperiment
weal--i room
Thank you for parti partoppsting
Figure 10: Instructions given to participants before beginning the experiment.
15

---

Under review as a conference paper at ICLR 2023
Let's find what monitor size
Coeas samo
Pilas can janail datal groniy dendd acinc
THE BANK BANK
5678 9101
1121
12/20
YOUR NONE
Now: lat's quickly
You right wik eacue's cvelce
Aakkul y allagge foan
Mesiiie dictt trae sereer pur
Socl the the arrurinniaion
Ohew your
starthe
The bell descpeer florr you dilonis toves from
Press dieee" key unee the cooeppenn frre rechhe truck theist. moir choppen ingh- uall
lop
Figure 11: Images of the experiment calibration steps. The credit card task was used to measure the pixel-to-inches ratio the subject's screen. The blind spot task provided an estimate of the subjects distance from their screen.
16

---

Under review as a conference paper at ICLR 2023
Table 2: Model accuracy per recognition time subset. Models are named to include architecture, training objective, and training dataset where appropriate. ResNet-X-Y indicates ResNet with depth X and trained random Y% subset of the ImageNet-Il 1k dataset. Model names ending in 21k were pretrained on ImageNet-2Ik. All other models with the exception of SWSL and CLIP models pre-trained on the full ImageNet- Ik dataset.
Subser
Datieet
Image- Net
Dijjct- Net
Tmage- Net
Objjct- Net
Image- Net
Oijject- Net
Iriago- Net
Objert-
Net
ResNet-18 He al. (2016)
92.7
73.77
89.8
63.0
76.2
41.7
55.7
23.3
ResNet -0%
$7.6
85,0
71.2
38.3
51.6
20.8
Rs-Ne-18-400
70.2
ResNes-18-40s
K3.1
58.4
85.1
52.0
63.5
36.3
48.7
18.8
ResNe: 18-20%
73.7
46.7
74.1
44.5
60.7
26.0
38.3
13.5
RosNes-50
93.8
34
RosNee300-305
ResNes: 30-00%
89.
89.0
67.4
75,6
46.7
61.4
25.
RsSNe.5004005
92.7
68.6
86.2
59.5
72.3
40.5
53.7
25.3
KesNet.30-205
85.4
ResNes-10--805
92.1
74.5
92.5
729
81,9
51.7
63.5
3000
ResNet-00 (01 60%
91.0
75.9
88.3
67.1
78.8
46.7
61.7
27.6
ResNes-101-009
87.9
22.2
ResNe--111220
RosNes:15 152 He et (2016)
93.3
83.4
95.5
80.9
89,9
62.9
75.2
36.3
RosNe-152--005
90.4
75.2
9016
71.3
82.7
51.1
67.2
29.2
1ss.es-55.660%
8900
73.9
RosNes-15--30%
85.4
60.6
81.3
50.8
69.5
35.1
48.9
16.3
CORNet& Kubilius (2019)
92.7
73.7
9016
68.7
78.3
47.0
55.4
26.1
VOneNer-RoseeS Dapello
VOneNas-CORNesS
90.4
78.8
VGG-19 Simonyan an Zisserman (2015)
91.0
74.3
88.6
65.7
78.2
45.9
60.0
22.
Noiey Stadent (FrficienNNer1.. 1,22 Xic (2020)
94.9
75.9
92.5
66.6
85.3
45.8
67.7
25.1
DenseNer-12
(2017b)
MSDNet
Huang (20176)
55.7
MSDNet Classifer
87.6
64-2
84.6
$4.6
70,0
37.1
$1.1
19.0
MSDNet Clossifler
89.
68.6
88.7
62.2
75.8
45.3
57.8
25.9
Clossifer
MSDNet
92.7
89.5
25.2
SunCL ResNer50l Chen et (2020)
88.2
6016
84.5
59.3
70,0
46.1
56.9
21.4
SimCL ResNer101
92.7
76.6
88.3
68.8
80.0
52.9
64.3
288
SiaCC RexNeil/22
75.7
CLIP-VYT-8/16
97.2
90.5
93.2
86.2
81.3
73.1
62.7
53.1
CLIP-VVT-L.114
98.3
9T.1
96.6
92.9
91.5
85.4
78.8
73.3
CLI--VT--/14--333
50
91.0
81A
55.0
93.8
R5.4
85.4
75.1
70.0
58.4
49.6
36.9
CLIP- Res-eeeNNe--- 50cl
93.3
B5.4
86.4
77.5
73.6
51.1
40.8
CT.IP-ReNNe-SOO.. 50x16
93.8
CLIP-ResNet.S004
EfficienNaaSS Tin & (2019)
89.1
67.3
92.0
63.8
78,0
44,3
63.9
23.5
EMlicienNaa-M
90.4
7018
89.9
63.4
75.4
30.9
63.9
2200
EffoieeNNas...
96.6
95.2
89,1
63.7
75.2
97.8
88.3
96.4
84.7
90.1
64.6
77.6
40.
EilicientNeeL--11
96.6
69.1
96.6
91.2
67.0
78.1
Duswitskty
92.8
65.1
VIT-B/16
91.1
R5.4
95.3
76.3
85.2
59,3
66.7
32.7
VIT-1116
98.1
94.2
97.5
90.4
96.4
79.4
$3.9
518
McCo-V3 Chen al. (2021)
91.6
91.6
84.5
53,3
69.1
Seatt-ReNNet Yalni etal. (2019)
96.1
92.0
97.4
87.8
95.1
76.3
81.7
51.2
SWSL R01 32x16d
97.2
97.9
95,9
86.7
610
17

---

Under review as a conference paper at ICLR 2023
Table 3: Model accuracy on the ImageNet and ObjectNet subsets of our 4,771 images.
Model
ImageNet
ObjectNet
ResNet-18
79.3
47.1
ResNet-18-80%
74.5
44.1
ResNet-18-60%
74.3
42.8
ResNet-18-40%
70.5
39.3
ResNet-18-20%
63.5
31.0
ResNet-50
86.6
59.3
ResNet-50-80%%
81.7
52.2
ResNet-50-660%
79.6
50.5
ResNet-50-40%%
76.3
45.1
ResNet-50-200%
70.4
36.9
ResNet-101
88.8
62.5
ResNet-101-80%
84.0
55.6
ResNet-101-60%
80.5
51.4
ResNet-101-40%
78.0
44.9
Res2et-101-20%
72.2
37.2
ResNet-152
89.7
64.4
ResNet-152-80%
83.7
54.7
ResNet- 152-660
81.1
51.7
ResNet-152-40%
77.5
46.5
ResNet-152-209%
71.8
37.9
CORNet-S
80.2
51.6
YOneNet-Resnet50
83.4
53.8
VOneNet-CORNNt-S
80.5
47.4
VGG-19
80.1
49.4
Noisy Student (EfficientNerr-I.))
85.7
50.3
DenseNet-121
83.7
55.2
MSDNet Classifier
61.7
28.9
MSDNet Classifier
74.0
40.9
MSDNet Classifier
78.3
48.3
MSDNet Classifier
78.3
50.6
MSDNet Classifier
79.4
51.0
SimCLR ResNet50
75.1
45.8
SimCLR ResNet101
81.5
54.4
SimCLR ResNet152
83.4
55.7
CLIP-VTT-B/32
79.1
64.0
CLIP-VIT-B/1
84.0
74.1
CLIP-ViT-L/14
91.8
86.0
CLIP-ViT-LI440336p
91.6
86.7
CLe5-ResNet-50
69.8
56.5
CLIP-ResNet-1O0
74.5
61.0
CLIP-ResNet-500
76.3
64.9
SL.P-RResNet-50x 16
79.6
72.9
CLPP-ResNett55064
86.6
80.3
EfficientNet-S
82.1
47.9
EllicientNet-M
80.6
46.2
EfficientNet-L
85.5
52.2
EfficiemtNet-S-21
89.6
65.2
EfficientNet-M-211
90.9
67.7
EfficientNet-L-211
91.4
68.8
VIT-T/16
58.1
28.9
ViT-S/16
84.4
56.1
ViT-B/16
86.6
60.7
ViT-L/16
94.6
77.6
MoCo-V3
85.1
55.9
SWSL-RessNex101-32x116
95.0
83.2
SWSL-ResNet50
93.5
75.3
MAE-ViT-B/16
89.6
65.0
18

---

Under review as a conference paper at ICLR 2023
Cscore learning speed proxy
Cscore learning speed proxy
1.00
1.00
Incorrect Correct
Incorrect
0.75
0.75
Correct
0.50
0.50
0.25
0.25
0.00
0.00
17
50
150
10000
17
50
150
10000
Subset
Subset
15.00
Prediction depth
Prediction depth
14.00
12.50
10.00
Incorrect Correct
12.00
Incorrect Correct
7.50
17
= 50
150
10000
17
150 10000 Subset
Subset
Figure Top: left and right are average c-score over subsets for experiment ImageNet and ObjectNet images respectively. Orange shows the images that are correctly predicted by the ResNet- 50 while blue shows the images that are incorrectly predicted. Bottom: prediction depth plots shown in the same way top.
A.6.2 MODEL PERFORMANCE
We evaluate finetuned models the same cropped images used in our psychophysics experiments. See table 3 for model accuracy reports on the image difficulty reported in the paper and table 2 for model performance on the full ImageNet and ObjectNet subsets of the experiment images.
A.7 METRIC CALCULATION PROCEDURE
In this section, we go through the details in computing c-score, prediction depth, and adversarial robustness for our experiment images.
A.7.1 C-SCORE
C-score (Jiang et al., 2021) identifies individual image difficulty by characterizing the expected accuracy or a held-out image given training sets of varying size sampled from the data distribution. In particular, c-score the frequency of classifying an example correctly when is omitted from the training set. However, computing c-score for each image by brute force is computationally infeasible since must train separate model for each image. Instead, as recommended by the authors. Learning speed measures the epoch at which an image is correctly classified by model. Intuitively, a training example that consistent with the training learned quickly because the gradient step for all consistent examples should be similar. The authors found high Spearman rank correlation between c-score and cumulative learning speed based proxies.
computed the learning speed proxy
should be
We trained a ResNet-50 (He et al., 2016) from scratch on ImageNet (Russakovsky et al., 2015) for 90 epochs with an SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and weight decay coefficient of 0.0001. The learning rate decayed by a factor of 2 every 9 epochs and the batch size was 256. The standard ImageNet transforms were applied to all images, and the network was initialized randomly. We then evaluated our experiment images at each epoch and used the average of correct predictions as an estimated c-score for each image. fig. 12 shows the average c-scores for ImageNet and ObjectNet experiment images split whether the ResNet-50 correctly predicted the image. C-score serves as an efficient predictor for human recognition difficulty only for images classified by the model in both ImageNet and ObjectNet. C-scores for images misclassified by
19

---

Under review as a conference paper at ICLR 2023
the model do not reveal information about the human recognition difficulty and remain consistently low across all difficulty subsets.
A.7.2 PREDICTION DEPTII
Prediction depth (Baldock et al., 2021) represents the number of hidden layers after which the network's final prediction is already determined. The authors showed that prediction depth is larger for examples that visually appear to be more difficult and is consistent between architectures and random seeds.
We trained a linear decoder at the end of each block of ResNet-50 on the 50 experiment classes using the ImageNet training and validation set. We used the same ResNet-50 used to calculate c-scores to ensure consistency of our results. There are 16 convolutional layers in a ResNet-50; and each linear decoder follows a convolution layer and consists of a pooling layer, flatten layer, and fully-connected layer. We use the same hyperparameters appendix A.7.1 and only updated the weights of the linear decoder.
A prediction is defined to be made depth L if the linear classifier after layer -= is different from the network's ' final prediction, but the classification of the linear decoder after every layer L are equal to the final classification of the network. Images classified by all decoders are said to be predicted at layer O. Note that prediction depth independent of whether the final prediction is correct not. It measures the layer at which an image's 's prediction converges.
Figure 12 shows the average c-scores for ImageNet and ObjectNet experiment images split by whether the ResNet-50 correctly predicted the image. Like c-score, prediction depth serves an efficient predictor for human recognition difficulty only for images classified by the model in both ImageNet and ObjectNet.
A.7.3 ADVERSARIAL ROBUSTNESS
We measured an image's distance to the decision boundary of a network using fast gradient sign method (FGSM) (Goodfellow et al., 2015). FGSM creates modified example that maximizes the loss using the gradients of loss with respect to the input image:
modg = x + e sign V V 20, x, v))
where adv, is the modified image, x is the original image, y is the original input label, e is multiplier adjusted accordingly to control the size of modification step, is the model parameters, and J is the loss function. Note that gradients are taken with respect to the input image, and model parameters remain constant.
For an image classified by a model, we define its distance to the closest decision boundary of the model the minimum needed for the model misclassify the modified image. On the other hand, for an image misclassified by model, we define its distance to the closest decision boundary of the model the minimum needed for the model classify the modified image.
We used the same ResNet-50 used to calculate c-scores to ensure consistency of our results. We finetuned the ResNet-50 on the 50 experiment classes using the ImageNet training and validation set. We used the same hyperparameters as appendix A.7.1 and only updated the weights of the final pooling, flatten, and fully-connected layer. We used this finetuned ResNet-50 as the backbone for adversarial perturbation and correction.
While perturbing each classified image. we searched for the smallest c, from 0 to 0.02 incrementing by 1.25e-5 and from 0.02 to 2.5 incrementing by 0.005, that would result in misclassification. We only applied only one gradient step when perturbing. While correcting each misclassified image, searched for the smallest from 0 to 0.001 incrementing by 1.25e-6 and from 0.001 to 0.05 incrementing by 1.25e-5. We applied two gradient steps when correcting because correction requires finer and more steps.
Note that the search range depends on the backbone model and the dataset. One must choose them through manual andl--and-rroors yield interesting and significant results. Recall that after removing images that were incorrectly annotated, incorrectly cropped, etc section 2.2, we reduced to 4,771 images from the original 5,000. Of these, 3,296 and 1,475 images were classified and misclassified by
20

---

Under review as a conference paper at ICLR 2023
the finetuned ResNet-50 respectively. We were not able to find an for every image while perturbing and correcting in the corresponding search range. We omitted these images our analysis. We were able to successfully perturb 2,815 out of 3,296 classified images and correct 1, .114 out of 1,475 misclassified images.
We hypothesized that difficult images that are classified and misclassified would be closer and further from the decision boundary respectively. fig. confirms the prior hypothesis. We could not confirm the latter hypothesis due to the smaller number of misclassified images across all subsets, as shown through the higher error bars in fig. 13
ObjectNet
0.008
ImageNet
0.007
Epsilon
0.006
0.005
0.004
0.003
17
s 50
150
10000
Subset
Figure 13: Average magnitude required to correct misclassified images back to their correct class per subset
A.8 CONSTRUCTING A METRIC FOR IMAGE DIFFICULTY
We propose two metries:
1. Difficulty score which provides exact ranking from most difficult recognize to least difficult to recognize based on each response
2. four minimum recognition time (MRT) subsets that quantify the minimum amount of time required for the majority of participants to reliably recognize image.
Difficulty score is a value from 0 to 28 that represents the number of incorrect predictions given by participants in our experiment across all timings for particular image. Each image in our experiment was seen an equal number of times per timing and and only rarely were images that were recognizable shorter timings also recognizable longer timings. This results in a low difficulty score indicating that an image is easy to recognize and high difficulty score indicating that an image is hard to recognize. These scores correlate well with the MRT difficulty subsets as shown in fig. 14. Difficulty score varies significantly by object class well (see fig. 15).
A.9 DIFFICLTTY SCORE DISTRIBUTION BY OBJECT CLASS
21

---

Under review as a conference paper at ICLR 2023
Samuit
8.30
Setuut
10
25
8:30
18
Liftouly
12
Figure 14: Distribution of difficulty score for each MRT subsets in ImageNet (left) and ObjectNet (right).
22

---

Under review as a conference paper at ICLR 2023
recklece
dightal MazEH
phane
Tahin
coffee
Water bettle
clothss
all bettin
plllw
ppeiics
telvidoon
dayer
tockpack
platic beg
doomat
park bench
trniter
pribter
shrve
Rosto
fand-Aid
B staber
Figure 15: Difficulty distribution by object class sorted in order of increasing mean
23

---

Under review as a conference paper at ICLR 2023
SmcA Reclies
vi128 tnured Imagerec228
SWSL model:
NSDNst dasatier layers
models
mmgeleetl
humen Nerorrance
Imagehecti
Azcitonnl Data
Moll Staden deflieettee
Pffieeteett
Rolte-10
HGONet Centler
Astrrcc Classter
0.3
VSONe Gaaifer
Realish
VILTE
Phonsi9
MAC
Sincc NoNev51
StnCcL NenMet522
Effiaartinnn 10
0.2
VITU26X
LIVITB222
Redkvento3
GUI 50
21 PMITA/16
CUP Webhi Soold
CLPW TUS4g333pp
Hamooo
Random Subset Distrbuttonnl Accuracy Sap
Figure 16: Robustness gap for finetuned models two randomly sampled subsets of our experiment data, balanced between ImageNet and ObjectNet. Lines connect model families with arrows pointing in direction of increasing model capacity. Compare with fig. 7.
24

---

Under review as a conference paper at ICLR 2023
0.5
Gap between Imagelvet and ObjectNet accuracy an images recognized reliably by humans viewed S
Mofel Aedee 101
Teales 50
0.3
5920%
0.2
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Gap between ImageNet and ObjectNet accuracy on Images recognized reliably humans when viewed for 17ms
Figure 17: Robustness gap for finetuned ResNets trained varying percentages of the ImageNet training set. Lines connect the same architectures with arrows pointing direction increasing dataset percentage. Compare with fig. 7.
0.8
0.2
10a
10a
Figure 18: Human accuracy Image presentation time from Mechanical Turk results. Time is log-scale with sigmoid fit
25


Reviews:
[{"review_id": "sFg26mcrx2U", "reviewer": "Reviewer_y1Qy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduce a novel dataset difficulty metric based on how long humans have to view an image in order to classify it correctly. The authors release the difficulty metrics for ImageNet and ObjectNet datasets as well as distribution of image difficulties in those datasets. The paper also introduce a new metric predicting object difficulty. ", "summary_of_the_review": "The paper address the important topic of evaluating the dataset quality and difficulty and mining the hard samples. The authors introduce the way to manually compute the difficulty of the sample based on how ling it takes for a human to classify it. Since the process can not be automated for other tasks and it is not fully proven that more hard samples by introduced metric lead the the better model performance the benefit of the suggested metric is debatable. The additional experiments can prove the main concept and increase the usability of the suggested solution", "strength_and_weaknesses": "Strength:\n- Novel metric for evaluating image difficulty.\n- Valuable and deep analysis of the difficulty, easy and hard samples of ImageNet and ObjectNet datasets. \n- The analysis of various model performance on the samples of different difficulty\n- Promising results that can help constructing the new datasets for various Computer Vision Tasks\n\nWeaknesses:\n- The process of determining the difficulty of the sample is manual and tedious. The authors did not suggest the way to automate the process for the other datasets. It would be interesting to experiment whether a neural network can learn the sample difficulty function from the labeled data and whether this can be used to automatically select samples for labeling\n- The correlation of the model performance and the sample difficulty is expected but it is not proved whether additional \"hard\" samples would improve the model performance. As shown in the paper many of the \"hard\" samples do contain various degradations (occlusions, blur, bad illumination, etc.) The authors might perform a test training a model on 2 versions of the dataset: with less and more \"hard\" samples and evaluating the performance on a fair large scale test dataset.\n- The work does not introduce the updated version of the dataset and does not elaborate what the optimal distribution of the sample difficulty should be.\n- Some of the plots (Figure 7) are hard to follow, authors can change the format or the quantity of the models on the plot to improve the readability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JJJ6aIq1Az", "reviewer": "Reviewer_aKwB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a behavioral approach for identifying challenging images in large image datasets. They use their method to find thousands of challenging images in ImageNet and ObjectNet, and find that CLIP has to a great extent matched humans on these images. Through further analyses they elaborate on the failures seen by different classes of models. They release their datasets and code for the community to build off.", "summary_of_the_review": "I am totally borderline on this. I think the method for curating datasets is important but I have some quibbles about its details (50-way classification potentially introduces issues in the data — totally non-ecological). I think some of the analyses are interesting but there's no key finding here I can take to my group to say this is changing the way we think about AI or human vision. I look forward to seeing what the other reviewers say. The work is high quality but the impact of the presented results is what keeps me from recommending acceptance.", "strength_and_weaknesses": "\nStrengths:\n1. Timely investigation of challenging images for humans, with the goal of using those to drive the development of better models.\n2. The authors introduce a toolkit for mining existing datasets for these challenging images.\n3. A combination of web-based and in-lab experimental validation. Very nice agreement between these experiments.\n\n\n\nWeaknesses:\n\n1. Can these images be used to develop better models of AI or human cognition? There's a forward inference/positive result missing here in my opinion. It's not news that CLIP is a good model, so restating that isn't all that interesting to me. It doesn't look like there's any open challenge for AI revealed in these experiments.\n\n2. Does the introduced method identify challenging images consistently, or are some of these images merely poorly labeled exemplars that humans have trouble categorizing?\n\n3. I think the behavioral method is very complicated. Specifically, giving individuals the ability to make a 50-way classification judgement. We don't know the psychological distances between those categories so it's difficult to say whether or not a preponderance of certain categorical choices could makes it easier/harder to find a given category (i.e., it might be harder to distinguish between species of dogs )\n\n4. Make sure you ref [1] as that is most similar to this work for OOD classification.\n\n5. Measuring difficulty in humans is tough -- I agree that there should be images that are unrecognizable in fast presentations, but these same images should be recognizable with enough time. Otherwise, I am worried that the objects are just mislabeled. What do you think?\n\n6. Figure 7 is really cool but also complicated. Would it be possible to add some additional plots to highlight the insights you describe in the text about SimCLR etc?\n\n7. \"We are of course not the first to carry out such viewing time experiments\" I think you should cite the much longer history of rapid visual classification experiments: [2], [3], [4] to give a sampling of works over the years that investigate performance on natural image databases (ImageNet in the case of [4]) as a function of stimulus exposure time. The goal of these works is to limit viewing time to better interogate visual system mechanisms associated with the \"feedforward\" sweep.\n\n8. I suspect the findings of c-score and Figure 8 could be idiosyncratic for different architectures. For instance, \"This analysis reveals that images that require more viewing time... are predicted by later layers in the network.\" This is consistent with the plethora of studies analogizing ResNets and RNNs, which say that the skip connections enable stronger non-linearities and as a result the ability to process more complicated stimuli. What happens when you try other architectures like the ViT in CLIP?\n\n[1] Geirhos et al. Partial success in closing the gap between human and machine vision. 2021.\n\n[2] Fabre-Thorpe. The Characteristics and Limits of Rapid Visual Categorization. 2011.\n[3] Serre et al. A feedforward architecture accounts for rapid categorization. 2007.\n[4] Eberhardt et al. How Deep is the Feature Analysis underlying Rapid Visual Categorization? 2016.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4QTwpAhF-Hv", "reviewer": "Reviewer_KToy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to conduct psychophysical experiments using a combination of in-lab and\ncrowdsourced subjects to derive an 'objective' (that is, one decoupled from any particular model)\nmeasure of the distribution of sample-difficulty in two widely-used object recognition datasets,\nImageNet and ObjectNet. This differs from recent work that conducts similar experiments to assay\nthe gap between existing models and humans on OOD benchmarks and derive a benchmark using the\ncollected traits. The authors propose to use 'minimum viewing time needed to recognise an object\nwithin an image' as a proxy for sample difficulty, with an object considered 'recognised' if more\nthan half of participants could classify it. Four presentation times are considered for\ncategorisation, with objects recognisable within 17ms constituting the easiest samples, objects\nrecognisable within 10s constituting the hardest samples. The authors find that ImageNet and\nObjectNet are greatly skewed towards easier samples according to the proposed metric and suggest\nincreasing a representation of harder samples to be an important avenue for future work on creating\nmore robust models. The crowdsourced data is compared the data obtained under controlled lab\nconditions and found to be reasonably-well aligned, with discrepancies mainly occurring at the\nshorter presentation times. The authors show that the proposed measure of difficulty correlates\nwith performance-degradation in a variety of models and furthermore that this measure can be\npredicted to some degree using model-based difficulty measures such as c-score and prediction\ndepth.\n", "summary_of_the_review": "The paper conducts an impressive array of both psychophysical -- involving human subjects -- and\ncomputational experiments in order to derive an validate an objective measure of sample-difficulty\nfor ImageNet and ObjectNet. While I'm not convinced about the extensibility of the method\nto problems beyond ImageNet-style datasets, the paper addresses and attempts to quantify an\nincreasingly-apparent problem; proposes and empirically justifies, through comprehensive analysis, an\nintuitive proxy metric for the datasets in question (which are among the most-used benchmarks in\nML); provides interesting insights into the datasets in question using the derived metric; is generally\nwell-written (consistently clear and concise); and does well in outlining the procedures for the psychophysical trials\nand addressing the ethics and limitations involved.\n", "strength_and_weaknesses": "### Strengths\n- The motivation and its distinction from prior work is clearly established -- while similar\n  undertakings have been conducted before, the paper does explore a novel avenue in attempting to\n  quantify sample difficulty in a way decoupled from any given model via psychophysical trials.\n- The scale of the psychophysical experiments is commendable as is the rigor invested in trying to\ncontrol them and validate those conducted under less-controlled conditions (Mechanical Turk).\n  While the results for easier samples may not be entirely admissible, for the hard samples that\n  are of primary interest the two groups do seem to align reasonably well.\n- Figures are mostly clear and germane, with descriptive captions, though some figures are not\nobvious without reading of said captions (namely, Figure 5 and 7).\n- Discussing limitations in the experimental procedure is not shied away from -- the authors, for\n  instance, identify and concede issues with the shorter presentation times, stemming from \n  the use of crowdsourcing. \n- The authors are able to convincingly show correlation between their psychophysics-derived\n  metric and the performance of neural networks across a wide range of architectures. They make a\n  sound case, consistent with other recent literature, that current benchmark datasets for object\n  recognition are lacking in their ability to capture real-world diversity.\n- The authors show with large-scale experimentation that viewing time can serve as a valid\nmetric for sample difficulty across a variety of architectures.\n- The appendices are in their detailing of experimental procedure and materials and in presentation\n  of additional results.\n\n### Weaknesses\n- There is some important related work missing. Most notably, while the authors cite Geirhos et al.\n2018, they neglect to follow up the follow-up work from 2021 [1] which extends the suite of psychophysical\ntrials and shows a closing gap in the OOD performance of humans and machines. Although I understand\nthe goals of said work and the present work to be complementary (the former aiming to evaluate the\ndifferences in robustness, the latter aiming to devise an 'objective' measurement for sample\ndifficulty), given that the works employ similar methodologies to establish a benchmark with which\nto contrast human vs. machine performance (albeit under different regimes), I think some comparison\nis warranted.\n- As alluded to in the above section, while I understand the rationale behind the design choices\nfor Figure 7, it's not an easy figure to navigate visually. Similarly, Figure 5 is confusing for\nits use of presentation time both as a categoriser and as an axis (though I realise an\nexplanation of this is given in the accompanying caption).\n- The proposed method for evaluating sample difficulty is only obviously applicable to datasets of the same\ntype as ImageNet, that is those that are focused on single objects -- there's no obvious way to extend\nthe procedure to instance segmentation datasets, for instance, in which many objects may be\npresent in a scene and the task goes beyond simple image-level recognition.\n- Much of the introduction reads rather informally rather than as academic prose.\n\n(very minor) gripe: The reference associated with Geirhos et al. 2018 is an older (arxiv) version of the\ntext, not the version published at NeurIPS in 2018\n\n[1] Geirhos R, Narayanappa K, Mitzkus B, Thieringer T, Bethge M, Wichmann FA, Brendel W. Partial\nsuccess in closing the gap between human and machine vision. Advances in Neural Information\nProcessing Systems. 2021 Dec 6;34:23885-99.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "13H6v_fJtie", "reviewer": "Reviewer_ZZZd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a new way of measuring difficulty of computer vision datasets. The authors propose to use humans’ viewing time as a difficulty measure and show that longer viewing time correlates with difficulty. They also show that the measured difficulty can be computationally modeled with a combination of prediction depth, c-score, and adversarial robustness. Analysis of SOTA models reveals that there exists larger gap on harder images.", "summary_of_the_review": "This paper studies an important problem for the computer vision community. The experiments and analysis are well organized and provides new insights. Given the overall strengths of the paper, I recommend acceptance. ", "strength_and_weaknesses": "Strength\n\n* Paper is very well written and easy to follow. Experiments are well designed and systematically executed. Analyses and questions asked in the paper are thought provoking and high quality.\n* Demonstrated the feasibility of this metric as a difficulty measure and identified key next steps we need to work towards as a community.\n* The proposed approach is much more scalable than gaze tracking and can be potentially adopted by others with low cost.\n\n\nWeakness\n\n* Further discussion on what would be a good way to incorporate this metric intro a new data collection protocol  would be useful; how can researchers plan ahead and target at collecting more harder samples in the beginning?\n* It remains unclear whether viewing time would generalize well and serve as a good difficulty metric for other vision tasks such as optical flow\n\n\nI have a minor question on the study design; a set of words are displayed at the end of the viewing session - what if the viewer is not familiar with certain word and fails to answer it correctly because they do not know how the object looks like, but not because they do not recognize the image? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject