How hard are computer vision datasets? Calibrating dataset difficulty to viewing time
Authors: David Mayo
Jesse Cummings
Xinyu Lin
Dan Gutfreund
Boris Katz
Andrei Barbu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Deep learning
Representation learning
Cognitive science
Computer vision datasets

Abstract: Humans outperform object recognizers despite the fact that models perform well on current datasets, including those explicitly designed to challenge machines with debiased images or distribution shift. This problem persists, in part, because we have no guidance on the absolute difficulty of an image or dataset making it hard to objectively assess progress toward human-level performance, to cover the range of human abilities, and to increase the challenge posed by a dataset. We develop a dataset difficulty metric MVT, Minimum Viewing Time, that addresses these three problems. Subjects view an image that flashes on screen and then classify the object in the image. Images that require brief flashes to recognize are easy, those which require seconds of viewing are hard. We compute the ImageNet and ObjectNet image difficulty distribution, which we find significantly undersamples hard images. Nearly 90% of current benchmark performance is derived from images that are easy for humans. Rather than hoping that we will make harder datasets, we can for the first time objectively guide dataset difficulty during development. We can also subset recognition performance as a function of difficulty: model performance drops precipitously while human performance remains stable. Difficulty provides a new lens through which to view model performance, one which uncovers new scaling laws: vision-language models stand out as being the most robust and human-like while all other techniques scale poorly. We release tools to automatically compute MVT, along with image sets which are tagged by difficulty. Objective image difficulty has practical applications – one can measure how hard a test set is before deploying a real-world system – and scientific applications such as discovering the neural correlates of image difficulty and enabling new object recognition techniques that eliminate the benchmark-vs- real-world performance gap.

How hard are computer vision datasets? Calibrating dataset difficulty to viewing time
David Mayo CSAIL & CBMM
Jesse Cumming CSAIL & CBMM MIT
Xinyu Lin CSAIL & CBMM MIT
Dan Gutfreund MIT-IBM Watson Al Lab IBM
MIT
Boris Katz
Andrei Barbu CSAIL & CBMM
CSAIL & CBMM MIT
MIT
Abstract
Humans outperform object recognizers despite the fact that models perform well current datasets, including those explicitly designed to challenge machines with debiased images or distribution shift. This problem persists, in part, because we have guidance on the absolute difficulty of an image or dataset making it hard to objectively assess progress toward human-level performance, to cover the range of human abilities, and to increase the challenge posed by dataset. We develop a dataset difficulty metric MVT, Minimum Viewing Time, that addresses these three problems. Subjects view image that flashes on screen and then classify the object in the image. Images that require brief flashes to recognize are easy, those which require seconds of viewing ImageNet and ObjectNet image difficulty distribution, which we find significantly undersamples hard images. Nearly 90% of current benchmark performance derived from images that are easy for humans. Rather than hoping that we will make harder datasets, we can for the first time objectively guide dataset difficulty during development. We can also subset recognition performance as a function of difficulty: model performance drops precipitously while human performance remains stable. Difficulty provides new lens through which to view model performance, one which uncovers new scaling laws: vision-language models stand out as being the most robust and human-like while all other techniques scale poorly. We release tools to automatically compute MVT, along with image sets which are tagged by difficulty. Objective image difficulty has practical applications can measure how hard a test set is before deploying a real-world system and scientific applications such discovering the neural correlates of image difficulty and enabling new object recognition techniques that eliminate the benchmark-vs. real-world performance gap.
hard. We compute the
Introduction
Numerous efforts exist to build better evaluations for object recognizers. Broadly, these fall into four categories. Those that probe distribution shift, like ImageNetV2 [I]. Those that add scale like OpenImages (2]. Those that explicitly attempt to make images more difficult for models by adversarially selecting them, like ImageNet-A [ or adding artificial corruptions, like ImageNet-C C [.. And those that attempt to explicitly control for biases like ObjectNet [3]. These are responses to the fact that performance standard benchmarks does not translate well to real-world conditions;
Equal contribution. Website http:///////eeeteeedddee//////// Cortrbttiii.oootigggrrr/////oooe
Correpoodiing author author dwwyo22mit. edu
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Missing exansiues
Missing examples
Number of images
Number of images
75
50
50
25
0
10
20
30
40
0
20
30
40
Difficulty Score
Difficulty Score
(a) ImageNet image difficulty distribution
(b) ObjectNet image difficulty distribution
Figure I: ImageNet and ObjectNet image difficulty distributions. Difficulty here is defined as how many participants failed to recognize given image across viewing times. The difficulty of both datasets is roughly the same, significantly under-sampling hard images. Slightly under 90% of performance metries are derived from easy images. Today, we largely as a community only lest what easiest for the human visual system.
90% accuracy for one class in ImageNet does not mean that the detector will achieve 90% accuracy for that class in one's home on frames of a movie. In all four cases, these efforts have objective guide, no metric that evaluates how far they have progressed towards enabling models to generalize.
We set out measure orthogonal quantity how difficult images in these datasets for humans. Distribution shift and bias control won't 't on their own address this problem datasets are overwhelmingly easy compared to what humans are capable of recognizing. And while scale helps, datasets are heavily skewed toward images that are easy for humans, the statistics of performance on such datasets may hide the real underlying performance trends of models on harder images.
An objective metric by which to measure the difficulty of computer vision datasets has several advantages. First, we can determine if there are gaps in our datasets; perhaps certain difficulties are systematically undersampled. We find that this is the case: hard images are essentially missing. Moreover, merely aiming for distribution shift, even by changing how the dataset is gathered, doesn't 't meaningfully change the difficulty distribution; ObjectNet and ImageNet were gathered from different sources (captured by Mechanical Turk vs the web), with different goals and additional controls for ObjectNet, yet their difficulty distributions are remarkably similar. Second, we can evaluate model scaling as function of difficulty. We find that most model families scale poorly in terms of out-of- distribution robustness, generalizing well on the easy images but hardly improving on the hard images, with the exception of CLIP ( Third, it provides new kind of metric for biological plausibility, orthogonal to raw performance, error distribution, or how well networks predict neural activity. If network to be model of the human visual system, not just an engineering model, then some quantity computed from that network should explain the observed difficulty scores. About half of the variance the difficulty results is accounted for by a combination of c-score (7), prediction depth [8. and adversarial robustness [9]. Fourth, tools that measure difficulty could be incorporated into dataset collection and into how we report datasets and the overall progress of our community, In the long term, we intend to establish a dashboard giving a perspective on object recognition from the point of view of difficulty.
To build this difficulty metric, we choose as a proxy the minimum viewing time (followed by a backward mask) that human viewer requires before being able recognize the object in an image. Earlier readout is likely an indication that fewer mental resources were needed to recognize the image. 1-out-of-50 forced choice task
After viewing the image, subjects have unlimited time respond This metric is related to object where they must identify the object class in the image that was shown.
solution time (OST) 10 explored in the neuroscience literature. We are of course not the first to carry out such viewing time experiments [II) but we do so at scale and with images from modern datasets. Further, we turn these results into difficulty metric with practical applications, predict this difficulty metric from quantities computed from current networks, and show the scaling of current models. We hope that in the future, benchmarks will regularly report their difficulty distribution (they can do so for only few hundred dollars with the tools we provide) and that collections of benchmarks will seek out datasets based on their difficulty distribution. Additional attention may need to be paid while collecting datasets to not eliminate hard examples; any quick consensus-based process with multiple annotators likely to be heavily biased against including hard examples. Practically, when collecting datasets for domains where the cost of errors is high (the medical domain, autonomous vehicles, etc.),

---

being mindful of the difficulty distribution and actively shaping it to fill out harder images may be critical building confidence in the resulting models.
Our contributions are:
1. A dataset of 200,382 human object recognition judgments as a function of viewing time for 4,771 images from ImageNet and ObjectNet
2. The first objective image difficulty metric, minimum viewing time, MVT
3. The difficulty distribution for ImageNet and ObjectNet
4. An evaluation of model performance function of image difficulty and new scaling law
5. A new metric for validating the biological plausibility of models, predicting image difficulty
6. A new subset of images from ObjectNet and ImageNet sorted by difficulty for use in neuroscientific and behavioral experiments.
2 Related work
Image difficulty Image difficulty has previously been investigated primarily from the perspective of models. Jiang and Zhang al. 7 constructed new metric for image difficulty, c-score, which is well approximated by its learning speed proxy; more difficult images are classified correctly later in the course of training. Agarwal et al. [12] introduced variance of gradients (VoG), method similar to the c-score learning speed proxy. but using the variance in the gradient updates. Baldock et al. [8] identify the first layer at which model produces the same classification its final output. These methods find ObjectNet significantly more difficult than ImageNet, unlike our human presentation time metric as shown in fig. | because ObjectNet harder for models; they are fundamentally not objective metrics of difficulty but they depend on the training set, architecture, optimizer, etc. of specific model. Other work [33] 14 has found that image difficulty estimates can improve task performance or inform compute trade-offs. Such approaches are promising and could likely benefit from more objective, model-free difficulty measures like ours, Meding et al. [15] investigated image classification accuracy as function of model type, finding that many images are solved by all models, while some are never solved. Humans predicted which images were easy vs. hard. This is an overt binary judgment that requires introspection, while work provides graded difficulty metric which is implicit to the operation of the visual system.
Dataset design As field, we have measured our progress by models' performance on tests created by splitting random subset of images from large-scale image datasets. Most datasets were created by web-scraping images and labeling them according to consensus of human annotatators [16]. More challenging test sets were developed by recreating existing test sets from new data (ImageNetV2 (1). constructing adversarial test sets (ImageNet-A 3), adding corruptions to existing test sets, (ImageNet-C 4)), and eliminating spurious correlations (ObjectNet (5)) Others have improved labeling errors in existing datasets to demonstrate, much like our own work, that model performance gains are mislcading and that we should not declare victory too soon [8]. Our image difficulty metric can aid all of these datasets by calibrating their difficulties and providing a roadmap for how to collect better datasets.
OOD model performance Prior work studying object recognition models performance has found a linear trend in performance improvement on ImageNet and other OOD datasets [ 19). This linear trend can be beaten slightly when massively scaling up the quantity of training data used. Recently, the multimodal models CLIP (6) and LiT 20 have demonstrated break from this linear trend, greatly increasing out of distribution generalization performance by using multimodal learning.
Human Judgments Collecting human behavioral data has long been integral to computer vision research. Human psychophysics has been used to improve model robustness [21] and representational power (22]. to motivate evaluation metries (23 and develop challenging datasets with rich
annotations 28 29.. Modeling human behavioral data is at times an end in itself (22 33) These endeavors often find that while models have made progress toward parity with humans, they continue to lag behind in many important dimensions [34]. Especially relevant to our work is ImageNet-X. labeling of ImageNet with human judgment annotations across 16 dimensions (pose, lighting, etc.) designed to aid in discovering and explaining failure modes of models with higher resolution. While these dimensions are a great contribution, they fail to explain the more complex phenomenon of human recognition difficulty.
3

---

What did see?
pPane
IMAGENET
ObjectNett
Mask
1080
Stimull
Fixation
Isave
5,000 images
image presentation times: 17ms, 50ms, 100ms, 150ms, 250ms, and
42 human responses per image, per presentation time
2,500 per dataset
50 object categories
Figure 2: Overview of the experiment. (left) 50 images from 50 object classes randomly selected from both ImageNet and ObjectNet to total 5,000 images; of which we analyze 4,771. Images were cropped in square around the object of interest and then shown to human subjects on Amazon Mechanical Turk and in controlled laboratory setting. (middle) Participants first saw a fixation cross for 500ms, then the image for either 17ms, 50ms, 100ms, 150ms, 250ms, 10s, followed foreed-choice task identify the correct object class. (right) Each image was seen by 42 subjects, seven for each of the six image durations. No subjects saw same image twice.
mask. After each image, subjects given l-out-of-50
Limited viewing time Geirhos et al. 35 studied human and machine performance under limited presentation time with images altered by corruptions, finding that humans more robust than machines. Rajalingham and Isaa et al. [[]] presented images at 100ms finding that models are unable match image-level behavioral patterns of primates. While our experiments agree with these results, they provide new capabilities: measuring the difficulty of entire datasets, calibrating dataset difficulty to human abilities, discovering revealing a new understanding of difficulty as a function of image viewing time. Geirhos et al. [36] investigated the humans-machin gap as a function of architecture, objectives, and dataset sizes using a 16-way classification experiment with 200ms presentations. In this work, we investigate an orthogonal quantity, the numan-machinn gap as a function of viewing time, and derive new metric from it. Our results agree with this prior work and expand upon it.
scaling laws, etc. All of these are derived from
3 Experiment
We performed experiment with human subjects Mechanical Turk and in the lab in order to determine the minimum amount of viewing time required before subjects could identify an object present in image. See fig. for details. In what follows, describe the stimuli, procedures, and validation of the online experiment with in-lab experiments.
Stimuli We selected 2,500 images from the ImageNet validation set (which contains 50 images per class; selected all 50 for 50 classes) and 2,500 images from the ObjectNet dataset (ObjectNet is only a test set). These images were evenly distributed among 50 object classes shared between ObjectNet and ImageNet. Since ObjectNet was gathered by participants taking pictures of objects in their homes, all of the object classes are household objects (see right panel on fig. 2 for full list of the object classes) and all participants were likely familiar with all object classes used. The 50 classes were picked to minimize similarity between classes
At short presentation times, subjects do not have time fixate multiple locations. Off-center objects would be recognized by peripheral vision. To eliminate this effect, we cropped all images around the target object using the ImageNet validation bounding boxes and new bounding box annotations for ObjectNet. In each case, we produced square 224 by 224 image, padding with black needed. Details about cropping can be found the appendix. Note that this eliminates clutter and focuses only on the appearance of the object; we discuss this limitation in the conclusion.
Procedures The MIT IRB oversaw the experiment. Subjects were consented and provided warning about image flashing. Only subjects with normal corrected vision were included. Written instructions and sample vidco of the experiment were provided. Subjects then read through list of

---

the 50 object categories. The Mechanical Turk experiments had calibration steps to determine the screen size (utilizing credit cards since they have standardized size) and distance from the screen using a blind spot test (37]. Images were shown at 8 degrees of visual angle. No private information was recorded and no offensive images were shown. Details about setup and calibration are available in the appendix.
For an overview of the procedures, see the middle panel of fig. 2 Each trial had randomly selected presentation time, either 17 ms, 50 ms. 100ms, 150 ms, 250ms, or 10 seconds; timings were selected to account for participant's is use of 60Hz screen and to probe fine fine-grained distinctions between image difficulty most apparent at short timings. We selected timings shorter than the typical human pre-saccade latency (250ms) so participants had time for only one fixation with the exception of our 10s control accuracy timing, In each trial, a fixation cross was shown for 500 ms followed by the image, for one of the six timings. Immediately after the image, a phase gradient backward mask [38] was shown for 500 ms to disrupt further processing of the image. Details of the mask generation are described in the appendix.
Participants were then presented with grid of the names of 50 object categories and were asked to click on the class of the object that they saw. As this was a forced-choice experiment, subjects were told to select their best guess if they were unsure. The order of the object classes in the grid was randomized at each trial. Subjects were given an unlimited amount of time to make a choice although this time was recorded and appears to be correcoreel with performance (quick decisions were likely to be more accurate than slow ones).
Experiments were counterbalanced. In total, each image was viewed by the same number of subjects at the same number of viewing times. Any one subject did not the same image twice any duration. Each participant viewed 50 images, one from each class, 25 from ImageNet and 25 ObjectNet, evenly spread across the six viewing times. The order of images, viewing times, and classes were randomized for each participant. Participants were allowed to complete several experiments; for those who did, we ensured that they saw disjoint images each time. In total, 2.647 workers took part in the experiment. Subjects on Mechanical Turk were compensated at over $ $10 per hour. In total, we collected 210,000 trials, after which we discovered that a small number of images (229 images) were either incorrectly annotated, incorrectly cropped, or workers circumvented our automated means for ensuring that no two workers saw the same image twice. This resulted 200,382 trials for 4,771 images (42 presentations of each image, each of 7 subjects seeing each image at one of six timings).
In-lab validation Great variation exists between monitors and browsers, with some showing the same images for nearly twice long at short presentation times. Subjects as well can vary in their attention or alertness. To account for this, we validated the results with experiments in the lab. These followed the same procedures described above with I 144Hz gaming monitor (27 inch LG-27GN950-B; Ims GTG response time). on a machine with an NVIDIA 3060Ti, using Chrome 102. We recorded the screen at 200fps and found that the presentation times accurately reflected the time periods during which this monitor showed each image.
We selected 200 images from the 5,000 used in the previous experiment, evenly distributed across the 50 classes and the two datasets (ImageNet and ObjectNet). Each participant saw all 200 images, random presentation times in a random order; they never saw the same image twice, and could only participate in the experiment once. Subjects were instructed to stay at fixed distance from the screen. An experimenter was present throughout to ensure that subjects were not distracted. Lighting was kept constant by closing the blinds. In total, 12 subjects (6 male, 6 female) participated in this experiment, and as can be seen in the next section, there was widespread agreement between the in-lab and Mechanical Turk experiments. It took subjects just short of an hour to complete the 200 images. Subjects were compensated $20 for their participation.
4 Results
We considered an image as recognized at some viewing time when half of the participants could classify it. Chance on the l-out-of-50 task is 2%; even a single correct response is an indication that something could be recognized. MVT is the minimum duration at which an image is recognized (MVT of 100ms means the image is not recognized by the majority of participants at < 100ms and correctly recognized by the majority of participants at 100ms). Typical images by MVT are shown
5

---

11
=50 $100
$150
$250
S <10 sec
Ruler
Lampshade
Hammer
Speaker
Clothes
Iron
Figure 3: Images as function of difficulty, using the minimum viewing time (MVT) before they were reliably recognized (columns), i.c., when more than half subjects were correct. Harder images to the right are more atypical, have more difficult lighting, more occlusions, and are sometimes more blurry. Easier images to the left are more prototypical instances of their respective object class. Additional examples are available in the appendix and online. Our experiments indicate that many datasets oversample easy images.
Dataset
Dataset
Imagetett
OsjectMet
octective
0.6
Image-ket
0.6
0.2
0.2
13
50
100
155
252
10020
17
153
10000
Image Preventation Time (ms))
Image Presentation Time (ms)
(a) Mechanical Turk experiments
(b) In-lab experiments
Figure 4: Accuracy as function of presentation time: the left are results for Mechanical Turk experiments and the right are results for in-lab experiments. The same images (5,000 online, 200 lab) presented 4 timings. Results both conditions were similar, although in-lab experiments achieved nearly 100% accuracy with 10 second viewing times, while halving the performance at 17ms compared to MTurk. The accuracy improvement at long times is likely because of the more controlled conditions with fewer distractors in lab. The lower accuracies at short timings are likely due to issues with displaying images at short timings: some monitors very slow and fade the image in and out effectively displaying it for twice the intended 17ms.
in fig.3) Images that are quickly recognized by humans-easy images-are most similar to those in datasets, while harder images include occlusion difficult lighting.
An overview of accuracy as function of viewing time online and in the lab shown in fig. 4 Both experiments broadly agree with one another. In-lab experiments have higher variances due to having 20x fewer images and half as many subjects per image. In lab, the performance on short timings was significantly worse, half of that seen online. We believe this is largely due to slow monitors which display the image for significantly longer than 17ms, roughly twice as long. Recording screens with high-speed cameras supports this hypothesis. At the high end, subjects in lab accurate, 10% higher than online. We believe this likely has to do with how distracted subjects were. In what follows, focus the Mechanical Turk experiments due to their scale.
nearly 100%
Human performance drops off steeply when difficult images are shown for shorter viewing times; see fig. 5 This makes the experiment sensitive minor variations in difficulty.

---

Acaincy
Presortation tims
150250
100359
390
230
230
Officity Suzse
199999
9:290
5.99
Figure Human accuracy a function MVT image difficulty subset. Colors denote presentation time, shown helow each har, collected by minimum viewing time, shown at the bottom. Accuracy drops steeply when hard images are displayed at short time intervals. These results are derived from Mechanical Turk experiments; in-lab experiments have exactly the same trend but higher absolute accuracies for correctly recognized images.
Alts Alll AR
ResNet-50
SWSL (ResNer-50)
MAE (ViT-B/16)
CLIP (ViT-L/14)
Figure 6: Accuracy of models ImageNet (solid) and ObjectNet (dashed) function of image difficulty; many more models are reported in the appendix. Note how in-lab experiments, human accuracy barely declines as a function of difficulty. while model performance drops off significantly. Subsetting difficult images provides much more overhead for model improvement. The in-lah human accuracy is computed over the subset of images shown in the in-lab experiments.
4.1 How hard are today's object recognition datasets?
The minimum viewing time required for reliable recognition is a proxy for image difficulty. Datasets today not gathered to control for difficulty and, indeed, when plotting the difficulty of images in ImageNet and ObjectNet, we find that the difficulty curve for these datasets highly skewed; see fig. [ Rather than plotting six bins, one for each viewing time, plot more fine-grained quantity: the total number of incorrect responses out of the 42 presentations of each image (7 participants at timings). Images with few incorrect responses them, even at short timings. Images with many incorrect responses are hard: few participants could recognize the images only few longer timings.
easy: all participants all timings could recognize
Our results indicate that hard images are vastly underrepresented today's datasets. The difficulty distribution of ImageNet and ObjectNet are very similar to one another. This is despite the fact that the latter was collected in a manner intentionally intended to highlight more diverse and less biased images. Merely collecting more images does not appear to help which has strong implications for how this difficulty distribution looks for other datasets that are not measured here. Instcad, we must develop new tools guide dataset collection toward more difficult exemplars; describe such tool and workflow below.
4.2 What we can learn about models from hard images
Machine accuracy varies as a function of the minimum viewing time required to recognize the object in an image. Most models-bol not all all-see significant performance dropoff between the easy images and the hard images, see fig. 6 See the appendix for extensive results for dozens of models broken down by image difficulty. Note that this understates human performance it shows the Mechanical Turk results. In-lab, even for the hardest images, humans have nearly perfect performance.
These results also show that the gap between ImageNet and ObjectNet performance increases as image difficulty increases. Likely, many more phenomena much more acute for harder rather

---

0.5
Sudker (FRoeetee))))
0.4
Bipcoti 2
Virto
A 10s Subset Distributional Accuracy Gap
Whbad
corRee
SIn Rnstact01
0.3
TEGReT Cassiter
ReoMoet00 Sheted
Restet-
yinta
ImageNet1k
Additional Data
SimCLR ResNets
Imagenet21k
SWSL models
MSDNet classifier layers
CUPV 2WW.M 1o00
Ci Pursery
CLIP VIT models
CLIP ResNet models
Sumanc
EfficientNet trained ImageNet21k
0.0
EfficientNet trained on Imagenetlk
jurrans R8 Yarss
Resnet
Human Performance
0.0
0.1
0.2
0.3
Q.4
0.5
17ms Subset Distributional Accuracy Gap
Figure 7: Robustness function of image difficulty comparing the ImageNet-ObjectNe performance gap on easy images (horizontal) and hard images (vertical) determined by the MTurk MVT experiment. Each point model. Arrows connect model families, pointing from small to large variants. Horizontal model families are undesirable: they scale by improving on easy images. Diagonal model families are desirable: they scale by improving on both hard and casy images. Humans show no gap between ObjectNet and ImageNet. ett. All model familics are horizontal, regardless of dataset, supervision, training regime, or architecture, Only CLIP models stand out by scaling a more human way. This scaling law has been observed before and the reasons behind it are unknown.
than easier images just as distribution shift. Datasets intended to challenge object recognizers would benefit from sampling hard images, rather than vastly oversampling easy images as they do today.
Image difficulty can tease apart differences between recognition models that would otherwise be lost because of the skewed underlying difficulty distributions in current datasets. In fig. 7 we plot numerous object recognition models contrasting their performance on the easiest and hardest images. Rather than computing the absolute performance of models, which would naturally favor larger models with larger training sets, we measure the gap in performance between ImageNet and ObjectNet. A more robust detector is one that has smaller gap, even if its performance is lower, as scaling models up (both in parameter size and training set size) is well understood. Models that are part of the same family are connected by arrows starting with the smaller variants pointing to the larger ones.
Humans hover around zero: they are robust with respect to the distribution shift between ObjectNet ImageNet. Some model families are roughly horizontal, like SimCLR, or even have negative slope. This shows that they scale, their performance is increasing on the easy images, not on the harder images, or in the case of negative slopes, widening the gap harder images. CLIP stands out, and most closely approaches the human results.

---

1.00
545
120
114
0-12
02:22
3.61
0-32
1202
C-Score
Prediction Depth
Perturbation
Figure 8: The correlation between three metries and image difficulty. On the x axis images subset by the minimum amount viewing time, in ms, required for the majority of participants before they are recognized. All three metrics are correlated with difficulty. Hard images are learned later in model training, predicted by later layers, and need much smaller perturbations to attack.
Extrapolating the performance of models such graph shows that many current model families are likely to stop or radically slow down performance improvements on hard images, a much less optimistic story than if one merely considers aggregate performance. At the same time, CLIP shows promise, although it too appears to eventually begin to move away from being robust to distribution shifts model size increases. Note that these experiments models available to the public; other large models described in the CLIP publication may have even more optimistic results. SimCLR also stands from the non-CLIP models, demonstrating a mild productive scaling trend although it is not clear whether this trend continues with larger models.
the largest CLIP
Four weeks of compute on two machines with 8 TITAN RTX were used to generate the results in this paper. We release our data under the Creative Commons BY-SA license.
4.3 Explaining image difficulty and testing how similar models are to humans
If models are not just perform well, but to also process images in ways that are similar to how the human visual system does, then they should contain a proxy for difficulty. Most models today are not recurrent, no direct analog exists viewing time (although, even if they were recurrent, is unclear directly taking the number of iterations of the model is the correct analog of viewing time). Minimum viewing time judgments provide independent and complementary metric by which to evaluate how similar models are to human brains, in addition to their behavior, error pattern, and ability to explain neural recordings.
investigate three quantities computed from models that could used to explain the difficulty judgments: C-score 7l. prediction depth 8 and adversarial robustness [9] C-score is a consistency score computed while the network training; we use the learning speed proxy to compute Prediction depth measures the earliest layer at which network produces its final prediction. And adversarial robustness refers to the perturbation, e, needed to fool the network on given example. See fig. X for overview of the correlation between these metrics and difficulty. The same network with a ResNet-55 architecture trained on ImageNet was used for all experiments presented here. Note that is only computed over the correctly classified images.
This analysis reveals that images that require more viewing time for humans are harder for networks in several ways. They are learned much later in the training process. They are predicted by later layers in the network. And they require much smaller perturbation to create adversarial examples. This underscores that human minimum viewing time has many practical consequences for how networks process images. At the same time, these metries alone do not explain all the difficult judgments. Logistic regression including all three metrics classifies images into the three difficulty bins (short: (17, 50), medium: (100, 150, 250), and long (10s)) with 47.7% accuracy. Above chance, but leaving plenty of room to do more. New metrics for understanding how networks process images could be validated against this data.
We also explored more intuitive explanations of image difficulty by leveraging ImageNet-X,, the human-anmotated dimensions of the ImageNet validation set 22). However, we found no correlation between these annotations and MVT. Though ImageNet-X is useful for analyzing model failure modes, fails to meaningfully explain human psychophysics table Low-MVT and high-MVT images are qualitatively distinct fig. and, empirically, humans able to distinguish between hard

---

and easy images 139, but more specific explanation eludes us at this time. Just like it is hard to pinpoint in language what exactly is difficult about computer vision in general, may also prove challenging to establish semantic explanations of image difficulty.
One high level qualitative trend that we can observe from our data is that images which are easy for humans are more prototypical while difficult images are atypical in innumerable different ways. We do intend to investigate this topic further in future work.
5 Conclusion, limitations, and future work
Rather than focusing on scaling, distribution shift, or control for biases alone, we should also focus explicitly on dataset difficulty. Today's 's datasets skew toward being too easy by undersampling hard images. ObjectNet was designed for distribution shift and bias control and was not collected from the web, yet its distribution of image difficulties is remarkably similar to that of ImageNet. By focusing on ways to measure dataset difficulty as datasets are collected, we can better calibrate the entire community, and create the resources needed to push object recognition forward. In addition to just creating better datasets, understanding performance as a function of difficulty reveals radically different scaling curves for different models and approaches. It can also provide subsets of images that highlight different types of processing for neuroscientific or behavioral experiments.
Our approach to measuring the difficulty of object recognition tasks focuses specifically on object instances, rather than other factors that also add complexity such as saliency and clutter. By cropping images around the object, we make figure- gooue-- segmentation far easier and separate out the issue of the difficulty of this specific instance of an object as opposed to the difficulty of finding this object within a cluttered background.
The particular notion of image difficulty derived from this experiment is just one metric of difficulty. As we describe in the introduction, other metrics exist which can be computed automatically given models that recognize objects. But those metrics rely on models and therefore change as models change. The metric we present here is both absolute - it is calibrated to what humans can achieve and model-agnostic. Models change over time, far faster than datasets do; there is a real danger when designing datasets that their difficulty will be tuned to specific models, which can be avoided by using measures related to humans.
Having performed extensive experiments to validate this approach, measuring the difficulty of any dataset now easy and cheap. One can sample a few hundred images and run an experiment on Mechanical Turk. This only costs on the order of hundreds of dollars per dataset and can be carried out quickly. The more critical the dataset, and the cost of object recognition failures, the more important it is to do so. We provide a toolkit with code for creating experiment stimuli, hosting the experiment, posting tasks to MTurk, and collecting and analyzing data at htttps://gitiiiii com/ Miiyoo//VT-diffiie fu All the user needs to do is provide the images.
Collecting datasets without considering image difficulty results in datasets that are skewed toward easy images which overstates model performance as we have shown. We encourage the community to develop novel dataset collection pipelines to tune datasets to desired difficulty distributions. This could take the form of posthoc filtering of webscraped images, or more on-line approaches which update collection parameters to discover ideal procedures for collecting difficult images. As shown here, there is promise in developing MVT proxies so that difficulty can be determined for each image efficiently.
Since posting this work online, MVT-based image difficulty has already been used in novel ways, for example, to improve performance by more efficiently allocating limited resources in embedded settings 40) This highlights how useful, reproducible, and widely-applicabe MVT is.
With modifications to our experiment, an MVT difficulty metric could be created for multi-objer objecc clas- sification-for example, by evaluating class recall rather than single-class optical flow. Other tasks are out of scope, like visual search, as they literally require multiple saccades. Calibrating our field to what humans can do across a wide range of tasks, datasets, conditions, remains significant challenge, but one that we think can now be addressed.
or
10

---

5.1 Acknowledgements
We would like to thank Jim DiCarlo, Guy Gaziv, Michael Lee, Dan Yamins, Colin Conwell, Martin Schrimpf and Ko Kar for their helpful feedback and discussion about experiments. We would also like to thank David Lu for his contributions to our early experiments and directions.
This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF- 1231216, the NSF award 2 2124052, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19 19-2- 10000, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-00
and the Office of Naval Research under award number N00014-20-1 1--20-1--258- -2589 and award number N00014- 4- 20-1-2644 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference Machine Learning (PMLR), 2019.
[2] Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 2020.
[3] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[4] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. International Conference on Learning Representations (ICLR), 2019.
[5] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang. Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, (ICML).
[7] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing struc- tural regularities of labeled data overparameterized models. In Proceedings of the 38th International Conference on Machine Learning (PMLR), 2021.
[8] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[9] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing ad- versarial examples. In 3rd International Conference on Learning Representations (ICIR), 2015.
[10] Kohitij Kar. Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience, 2018

---

[11] Rishi Rajalingham, Elias Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James Dicarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and tate-offthheart deep artificial neural networks. The Journal of Neuroscience, 2018.
[12] Chirag Agarwal, Daniel D' souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition ( (CVPR), 2022.
[13] Petru Soviany and Radu Tudor Ionescu. Optimizing the trade-off between single-stage and two-stage deep object detectors using image difficulty prediction. In 2018 20th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pages 214, IEFE, 2018
[14] Dingding Liu, Yingen Xiong, Kari Pulli, and Linda Shapiro. Estimating image segmentation difficulty. In Machine Learning and Data Mining in Pattern Recognition: 7th International Conference, MLDM 2011, New York, NY, USA, August 30 30-September 3, 2011. Proceedings 7, pages
Springer, 2011.
[15] Kristof Meding, Luca M. Schulze Buschoff. Robert Geirhos, and Felix Wichmann. Trivial or impossible dichotomous data difficulty masks model differences (on imagenet and beyond). ICLR, 2022.
[16] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jaanttaaaaoouuaaa of computer vision, 2015.
[17] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23500, 2021.
[18] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprini arXiv:2006.07159 2020.
[19] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[20] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[21] Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617-9626, 2019.
[22] Maria Attarian, Brett D Roads, and Michael C Mozer. Transforming neural network visual representations predict human judgments similarity. arXiv preprint arXiv.2010.06512. 2020.
[23] Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao. Pipal: large-scale image quality assessment dataset for perceptual image restoration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XI 16, pages 633-651, Springer, 2020.
[24] Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. Learning to evaluate image captioning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5804-5812, 2018
[25] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. European conference on computer vision, pages 382-398. Springer, 2016.
12

---

[26] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104.08718, 2021.
[27] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-bad based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-457, 2015.
[28] Kate Sanders, Reno Kriz, Anqi Liu, and Benjamin Van Durme, Ambiguous images with human judgments for robust visual event classification. Advances in Neural Information Processing Systems, 35:2637-2650, 2022.
[29] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazir- bas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. arXiv preprint arXiv:2211.01866 2022.
[30] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885- 23899, 2021.
[31] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Modeling human categoriza- tion of natural images using deep feature representations. arXiv preprint arXiv:1711.04833 2017.
[32] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott- Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[33] Radu Tudor Ionescu, Bogdan Alexe, Marius Leordeanu, Marius Popescu, Dim P Papadopoulos, and Vittorio Ferrari. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2157-2166, 2016.
[34] Lukas S Huber, Robert Geirhos, and Felix A Wichmann. The developmental trajectory of object recognition robustness: children are like small adults but unlike big deep neural networks. arXiv preprint a2X0v:2205. 2022.
[35] Robert Geirhos, David H. J. Janssen, Heiko H. Schütt, Jonas Rauber, Matthias Bethge, and Felix Wichmann. Comparing deep neural networks against humans: object recognition when the signal gets weaker. ArXiv, 0as/1706606666) 2017.
[36] Robert Geirhos, Kantharaju Narayanappa. Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. NeurlPS, 2021.
[37] Qisheng Li, Sung Jun Joo, Jason D. Yeatman, and Katharina Reinecke. Controlling for participants' viewing distance large-scale, psychophysical online experiments using a virtual chinrest. Scientific Reports, 2020.
[38] Bruno Breitmeyer. Visual masking: Past accomplishments, present status, future developments. Advances in Cognitive Psychology (ACP), 2007.
[39] Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann. Trivial or impossible- dichotomous data difficulty masks model differences (on imagenet and beyond). In International Conference on Learning Representations (ICLR), 2022.
[40] Vasileios Pentsos, Ourania Spantidi, and Iraklis Anagnostopoulos. Dynamic image difficulty- aware dnn pruning. Micromackines, 14(5):908, 2023.
[41] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience, 2019.
13

---

[42] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, al. massive fmni dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[43] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
14

---

Appendix
6.1
Hosting and maintenance plan
This dataset will be hosted on MIT servers in perpetuity https://objeccneeddewvw// with a backup on dropbox. Our dataset collection toolbox is hosted publicly on github at A datacard for this dataset will be available at
http://bbbeecnet.dee//vvvatttttt
Object classes
Test subjects were presented with images from 50 possible object classes and asked select which object they saw. The 50 classes were hand-picked to minimize similarity between classes that could be confusing for experiment subjects. The object classes were:
Band-Aid, T-shirt, backpack, banana, cleaver, clothes iron, coffee mug, computer mouse, digital watch, doormat, dumbbell, envelope, hair dryer, hammer, lampshade, lemon, lipstick, match, mobile phone, necklace, padlock, paintbrush, paper towel, park bench, pill bottle, pillow, plastic bag, plunger, power drill, printer, racket, ruler, safety pin, salt shaker, sandal, screw, shovel, space heater, spatula, speaker, strainer, sunglasses, teddy bear, television, umbrella, vase, wallet, waste container, water bottle, whistle
6.3 Image Selection
After choosing object classes, we selected images for the experiment. We used all 50 images belonging to a class in the ImageNet Validation set with no additional selection step. For ObjectNet, collected bounding box data for the images, and then randomly selected 50 images per class such that when cropped to the bounding box, the object in the image was centered and clear.
6.4 Image cropping procedure
1. We draw bounding box around the object (we existing bounding boxes for the ImageNet validation set and collect our own bounding boxes for ObjectNet from MTurk).
2. We initialize the cropping box to be the bounding box.
3. If the cropping box does not form a square, we extend the shorter side of the rectangular cropping box to form square. If the image is not large enough extend the shorter side of the cropping box, we pad with black pixels to form a square.
4. We crop using the cropping box for the image. The cropped image will be a square.
5. We resize the cropped image to be 224x224 pixels.
6.5 Mask generation
The masks were generated following the procedure used by Specifically, Fourier transform was applied to each image to obtain the magnitude and phase components. Then, random array with elements sampled uniformly from [0, 1] was added to the image phase component after which the magnitude and phase components were recombined via an inverse Fourier transform to produce the mask. Each image was paired with particular phase-scrambled mask in the experiments.
6.6 Experiment Procedure and Payment
Participants both in the lab and Mechanical Turk were presented with document informing them of the purpose, privacy, and risks associated with the experiment and soliciting their consent to participate (see fig. TOP. Participants were then instructed to how to carry out the experiment and were shown an example video as well as the list of image classes for their review before beginning. They were informed that they would not need to memorize the classes the classes would be shown to them after each video. Participants were also encouraged to take breaks should they feel fatigued otherwise uncomfortable. Example instructions are shown in fig.
After giving consent and reading the experiment overview. participants then completed two calibration steps for to estimate the size of their monitor and their distance from the screen for us to then size
15

---

Table I: Dataset statistics
number of responses
200,382
number of images
4,771
number of presentaiton durations
6
number of response per image
42
number of objectnet images
2415
number of imagenet images
2356
number of participants
2647
the videos appropriately to 8 degrees of visual angle. First, the participants are shown image of a credit card and are asked to use a card their own to adjust a slider to change the size of the card the screen the size of their card. Since credit cards are the same size around the world, this allows us to measure the pixel-to-inche ratio the participant's monitor. Next, the participant completes blind-spot test (37) that allows us to estimate the distance they are sitting from their screen. Together, these two measurements are sufficient to compute the desired video eccentricity. See fig. 12 for images of the calibration steps.
The estimated hourly wage for participants on Mechanical Turk and the lab was $10/hr and $20/hr respectively with approximately $15,000 spent in total on participant compensation.
6.7 In-Lab Experiment Results
To corroborate our Amazon Mechanical Turk results, we selected 200 images shown to Turk workers to conduct the same experiment in a controlled laboratory setting. 12 individuals came to participate in the experiment in which they viewed and responded to all 200 images on our 144Hz refresh rate monitor with Ims gray-to-gray time. After conducting the experiment, 3 individuals had seen each image at each of the 4 presentation times. When compared to the MTurk results for those same 200 images, the comparison is much as we would expect. The In-Lab accuracy with shortest image duration (17ms) is less than MTurk which likely be contributed the of our new, high refresh-rate monitor the controlled environment. It is likely that MTurk workers' personal computers differ in their graphics presentation abilities which may result in the image being visible for slightly greater than 17ms on some monitors. On the other end, the in-lab experiments reported higher accuracy the longest image duration (10s) which is also unsurprising as the in-lab participants completed the task in a controlled environment with no distractions and are likely more inclined to take the task seriously and stay focused. The results show significant differences in accuracy the intermediate image durations. See fig. for side-by-side comparison between MTurk and In-Lab results.
6.8 Dataset statistics
We collected 42 human responses for each of 5,000 images (2,500 from ImageNet and 2,500 from ObjectNet). After reviewing response, 229 images were removed due to either being unrecognizable, mislabeled, or having been seen by the same worker twice despite safeguards in place disallow Additional dataset statistics are listed in table [[
6.9 Preliminary COCO MVT Results
To bolster our claims about the difficulty of current datasets, we conducted the MTTrrk MVT exper- iment on small subset of the COCO dataset. As COCO a more visually complex dataset than many single object classification datasets, it provides good litmus test for how our conclusions generalize other kinds of datasets. See See/foo for the results.
6.9.1 Image selection
To maximize the utility our results for both computer science and neuroscience research we selected 732 images from the Natural Scenes Dataset subset of COCO for which fMRI data was collected from human participants. We used the image crops used in the NSD experiments. These crops ensure that the image is square.
16

---

Human accuracy on COCO vs image presentation time
1.0
COCO
0.8
40
33
Accuracy
0.6
Kumber tmwaans
0.4
20
15
0.2
0.0
17
100
150
250
10000
Presentation time (ms)
Difficutty Score
(a) Human accuracy per presentation time
(b) Difficulty score for COCO images normalized by number of collected responses per image. A difficulty score of 1.0 here correspondes 42 in figure
COCO.
Figure 9: MVT experiment results subset of the COCO dataset.
Informed consent to participate in this study
only intor
acaitin resporssss abouut
which
moseth now prnsenttd at exiertifo meeningg.e puplited iuccannticc purmat..
The respormue collectadin this experiment wiil be releared to actenatti. community and the public
Clicking SSBMIT buttom the button
indioater that
are less yeurs and agree comaiicoe this arturity
Thik oppeitinnt faschigg Adooss, photosmnaliv ransbiilly flaeking
and
rriddpata
Figure Informed consent page shown to participants before beginning the experiment.
6.9.2
Image classes
We selected set of 41 classes such that no image contained more than one of the classes.
6.9.3 Experimental procedure
We conducted the MVT experiment as described the text, asking participants to perform 1-of-41 forced-choice single-object recognition task.
Below, we present preliminary results computed on 340 of the images. The final manuscript will include all 732 images. The results in fig. Dare striking in their similarity to those presented for ImageNet and ObjectNet in the main text. The accuracy of human workers at each of the presentation times while performing the COCO classification task is almost the same that of the ImageNet experiments. Given that both ImageNet and COCO originate from the same online pool of images, this is to be expected.
Similarly, the difficulty scores of COCO images (the counterpart of fig. [ skewed toward easy im- ages, perhaps more so than either ImageNet or ObjectNet. These results indicate that our conclusions about the difficulty distributions of individual object recognition tasks in vision datasets generalizes. Of course, COCO has images where multiple object classes are present, which involves visual search addition to recognizing individual image instances, but, for the quantity that we measure here, how hard are objects themselves to recognize, it COCO and ImageNet essentially the same.
17

---

Task Overview
fnd contusing, do cuick the untari for You DG NOT NEEO To MEWORLIE THESe GA ReOOORE, they will shown you again after yoursatch each video.
minute med
Please read the fol owinng
You centicquee perticqueingg expert mentt whith we stualyying pwopler' seility sessagn on what objedt in an imagge given coeying armounte of mage viewing lime.
beckkpack
bamare
barn phone
bench
butcher's lesta, cleaver
computerr
dill
included below is on cxamrile of what we ask you view, Look at the center of the cross. Whnc the video plays, the crees bricftly change to object and then
mag
imouge
you cejecc
imagee
necklodd
paotookk
You only adiee bhy chees orcw,, ac be reacly wen focs beklow preasing play.. the amage may dia ploappeer tso quniddyy for you innme distelyy that's ekay, Look mu ligre chocks posalole sblects, riiik abouut
paper tonne
bott
plotte bag
plunger
portalle teater
abtie raamo the otject that
per
ruter Duo menourrny storu
saffery
salt shaker
sundal
NoW
guess, There time your response, but you before you can mowe on the next image.
showel
spester
vivir er
stuffed unemmal
samplaysees
9-ahirt
rockett torria radior))
container trish ben)
seleiiion [TV]
Limbrelle
ay snamplia video
On the rasponda you will ise table with these S0 oprions digp aod. The order eprions the table be nandoriized but 160 opriions will always Please take time find the
you know what itis. you do not know what you san, take your best guess. We will submission
know if trying rejected. Plaase talke this task seriously.
correctlly
this expariment you udl liook at seme imagess and hall us what oit sets saw. ocpect ta about 60 miiuuses You take
time. you feel
fined or hraving trouble focusing
take
breek ar resu the expetiment lster PP hesee try tave your reaks after submitting
image before diccking play
image.. Resparraas rass are sau as you subonit then as long ruturn you off,
souul also Boo you our axpariment qu. wval-81 room possiole.
you pantiiiiinggg
Figure 11: Instructions given to participants before beginning the experiment.
6.10
Finetuned Models
Here we list details regarding training/finetuning procedures for the model results reported in the paper.
6.10.1 Model training procedure
Pretrained models weights were instantiated using publicly available model checkpoints, either using torchvision or found on the model's source repository. The models-with the exception of CLIP- were then finetuned using subsets of the ImageNet training and validation sets containing only the 50 classes we chose to use in the psychophysics experiments. The models were finetuned for 90 epochs with SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and weight decay coefficient of 0.0001. The learning rate decayed by a factor of 2 every 9 epochs. Training, finetuning, and inference were performed a cluster of Nvidia TITAN RTX graphics cards.
6.10.2 Model Performance
We evaluate our finetuned models on the same cropped images used in our psychophysics experiments. See table for model accuracy reports on the image difficulty reported in the paper and table and table 3 for model performance the full ImageNet and ObjectNet subsets of the experiment images.
18

---

Let's find out what your monitor size is.
prin
Frews (unall
ahas unth scleen
They doddin shape lrrnge
THE BANK
1234
5678
9101
1121
12/20
YOUR NAME
Now, let's quickly test how for away you are sitting. tes
dinanca
Instructions
Paslice yoar tree peoiico
din prie kinch lareslmak
pou scuam
diicc the lu bokou startio anvesiio mai kilk. bell descome your rélen toves
ise1 the bull villnd actaiiy caapper fromy you scwers
bel rechhe truck in theistt ints the
Hosse tie wep ey cased and tthe "space"
Figure 12:
Images of the experiment calibration steps. The credit card task was used to measure the pixel to- inches ratio of the subject's screen. The blind spot task provided an estimate of the subjects distance from their screen.
6.11
Metric calculation procedure
In this section, we go through the details in computing c-score, prediction depth, and adversarial robustness for experiment images.
6.11.1 C-score
C-score (7) identifies individual image difficulty by characterizing the expected accuracy or held-out image given training sets of varying size sampled from the data distribution. In particular, c-score is the frequency of classifying an example correctly when it is omitted from the training set. However, computing c-score for each image by brute force is computationally infeasible since we must train separate model for each image. Instead, we computed the learning speed proxy as recommended by the authors. Learning speed measures the epoch at which an image is correctly classified by model. Intuitively, a training example that is consistent with the training set should be learned quickly because the gradient step for consistent examples should be similar. The authors found high Spearman rank correlation between c-score and cumulative learning speed based proxies.
We trained ResNet-50 43 from scratch on ImageNet (16) for 90 epochs with an SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and weight decay coefficient of 0.0001. The learning rate decayed by factor of every 9 epochs and the batch size was 256. The standard ImageNet transforms were applied to all images, and the network was initialized randomly. We then evaluated our experiment images at each epoch and used the average of correct predictions as an estimated c-score for each image. fig. 13 shows the average c-scores for ImageNet and ObjectNet experiment images split by whether the ResNet-50 correctly predicted the image. C-score serves as an efficient predictor for human recognition difficulty only for images classified by the model in both ImageNet and ObjectNet. C-scores for images misclassified by the model not reveal information about the human recognition difficulty and remain consistently low across all MVT difficulty subsets.
19

---

Table 2: Model accuracy ImageNet per MVT subset. Models named include architecture, training objective, and training dataset where appropriate. ResNet-X-Yss indicates a ResNet with depth and trained on a random Y% subset of the ImageNet-1k dataset. Model names ending in 21k were pretrained on ImageNet-211 All other models with the exception of SWSL and CLIP models were pre-trained on the full ImageNet 1k dataset.
MVT subset
50
100
150
250
10000
R8sNer-18
94.4
91.8
81.1
77.2
61.3
49.0
ResNer-88-219
77.2
Ro-Na-18-402%
85.0
67.8
63.5
50.5
46.
ResNa--8-6008
87.5
88.3
72.6
70.4
53.8
46.2
18-97--
88.1
86.5
76.7
68.8
5488
48.6
ReNNcs50
82.9
65.1
ResNet-50-404
93.8
89.1
74.1
74,1
60.2
48.
ResNet-50-65
91.2
90.2
78.5
79.4
60.2
59
RosNo-500809
ResNee--10 101
95.0
95.2
90.0
$7.8
79.6
71.5
ResNe-101-200%
86.2
85.4
70.0
66.1
90.5
48.2
ResNer-101-409
90.6
90.0
78.1
77.8
63.4
50.6
ResNec-101-6i0
Ro-Na-101-80%
92.5
94.1
82.5
64.5
61.8
ResNec-155 152
93.8
96.4
93.7
36.8
78,5
72.7
ResNec-152 152-20022 20%
86.9
84.4
73.0
71.4
52.7
44.
ResNa-152-400
ResNect1122 602-60%
93.1
90.4
82.2
78.8
66.7
59.4
152-8015
90.6
91,9
86.3
85.7
76.3
60.6
CORNei-S
93.8
92.6
81.9
78.8
58.1
52.
Resnet50
VOnoNei-COORe CORNee-S
91.9
92.3
82.2
77.2
61.4
53.4
VGG-19
91.9
90.2
80.7
79.4
62.4
$
Noisy Student (EffccienNNt-L)
95.0
93.3
87.8
$6.8
68.8
65.5
DenseNet-1
MSDNer Classifer
78.8
76.0
00.0
54.0
40.9
33.7
MSDNet Classiferr
89.4
86.2
23.7
67.7
53.8
45.8
MSDNet Chasifer
91.9
89,9
77.8
72.5
62.4
51.4
MSDNet Classfier
91.3
74.3
78.8
62.4
52.2
SimCLR ResNe150
88
86.3
73.7
69.8
60.2
$4.6
SimCLR ResNet101
89.5
ResNei152
92.1
CLI--VI--B/12
95.6
90.8
79.3
74.6
67.1
4%.6
97.5
94.7
83.5
$1.0
$0.6
52.
CLIP L/T-L/1
98.1
97.1
92.6
91.0
$6.0
72.3
CLIP-YIEL/14485000
CLIP ResNet:50
92.5
84,4
69.6
67.7
55.9
34.5
CLIP: CLIP-Re<Net: Re<Net-101
94.4
88.1
71.9
68.8
67.7
41.0
CLIP-ReeNNi-3004
88.8
50x16:-50015
CLiP.ResNen.3006
98.8
95.9
87.0
85.2
77.4
59.0
E'ficienNNerS
91.2
92.5
84.1
78.3
64.5
62.
ElficienNNtt M
91.5
73.5
61.4
EmocenNer-LL.
EffciinNer-S-- S-221
96.9
95.6
92.6
$7.8
$1.7
71.5
ElficeenNeer M
97.5
97.0
93.7
$8.9
$1.9
72.3
E1heienNer-L-21
72.3
VIT-S/16
95.0
94.5
82.6
85.2
68.8
58.6
VIE-B/16
96.2
95.6
85.2
87.3
67.7
63.5
VIF-L/16
98.8
97.5
MoCo-VS
92.6
64.7
96.9
98.1
97.4
95.8
S7.1
85.5
SWSL ResNet50
96.2
97.7
96.7
95.2
$49
77.9
MAE-VIT-B16
6.11.2 Prediction depth
Prediction depth represents the number of hidden layers after which the network's final prediction already determined. The authors showed that prediction depth is larger for examples that visually appear to be more difficult and is consistent between architectures and random seeds.
We trained a linear decoder at the end of each block of ResNet-50 on the 50 experiment classes using the ImageNet training and validation set. We used the same ResNet-50 used to calculate c-scores to ensure consistency of our results. There are 16 convolutional layers in a ResNet-50; and each linear decoder follows convolution layer and consists of a pooling layer, flatten layer, and colneccondetted layer. We use the same hyperparameters as section 1. and only updated the weights of the linear decoder.
prediction is defined to be made depth I. if the linear classifier after layer = l different from the network's final prediction, but the classification of the linear decoder after every
20

---

Table 3: Model accuracy ObjectNet per MVT subset.
MVT subset
50
100
250
10000
ResNec:
76.1
65.1
49.1
41.2
25.3
20.6
18-2298
27.5
ResNec-18-4077
58.1
53.8
43.0
25.2
20.7
16.2
RosNet 18 60%
67.5
60.0
13.3
37.9
19.5
17.4
ResNec-18-300 18-30%
634
RosNec-30
ResNee: 30-20%
58.
31.1
36.4
25.2
15.7
RosNec-500-40
70.1
61.7
45.4
42.9
29.9
20.3
ResNee-50
70.1
19.7
RcsNer-50-9955
70,4
49.5
ResNer-101
86.3
81.0
08.7
54,9
47.1
29.6
RosNec 101 20%
50.4
53.3
41.2
33.0
21.8
13.6
ResNec-101-409
ResiNes 101 101-80%
75.2
75.2
62.9
46.2
34,5
25.2
ResNet-152
85.5
83.8
68.7
60.4
46.0
50.7
ResNar-152-409
152N6--152-6008
72.6
684
57.0
41.8
35.6
226
RosNec-152-8009 152-80%6
74.4
73.5
25.7
20.0
35.6
24.6
CORNes-5
VOneNot-Resaee00
75.7
VOleNer-CONNeeS
72.6
67.0
31.9
42.9
31.0
16.3
VGGG-19
76.1
66.3
50.9
46.7
34.5
18.3
Student (#ffcceenNee-1.22
DemscNat-121
74,9
49.5
22.3
MSDNer Classifferr
45.3
39,0
31.3
28.0
17.2
10.7
MSDNer Classiteer
62.4
56.4
41.9
36,3
23.0
15.9
Classifee
Classifeer
MSDNer Classifier
75.5
70.9
31.5
47.3
37.9
22.3
SimCI RedNe150
60.7
61.0
49.8
45.6
27.6
17.4
SinCER ResNet101
SimCLR 55
78.6
70.3
57.0
35.5
28.7
CLIP-VIT-R/93
88.9
80.5
61.2
61.0
43.7
33.6
CLIP-WIT-B11
02.3
88.2
78.0
60.8
S0.6
484
CLIP-VIT-L14
CLPP-VTA//144 336px
CLIP.Re/Net-50
78.6
73.5
58.1
54,9
34.5
27.5
CLIP-Re:Nei-101
86.3
76,9
63.6
58.2
41.4
32.5
CTJP.Rc:NNe--5044
79.3
CLIP.ResNes-50016
CLJP-ReoNNn-300664
89.7
90.3
84.2
$1.9
60.0
$5.9
EMicienNNr--S
68.4
66.2
52.9
40.1
23.0
20.6
EMicienNeetL.
75.2
71.5
45.1
23.2
2MokkmNer-S-21
94.0
83.9
72.9
64.8
41.4
30.4
EMiceeiNet--22
87.5
67.0
46.0
34.2
EMicienNee--221
VIT-S/16
83.8
76,8
57.7
52.2
31.0
23.5
VIT-R/16
86.3
80.2
65.6
58.2
37.9
264
McCo-V3
73.3
34,9
34,5
24.6
SWSI 101-Naa100-32
94.0
94.5
89.3
85.7
62.
574
SWSL-ReNee50
89.4
83.8
72.0
52.9
472
MAE-VT-BV16
layer L 2 are equal to the final classification the network. Images classified by all decoders are said to be predicted at layer O. Note that prediction depth is independent of whether the final prediction is correct or not. measures the layer at which an image's prediction converges.
Figure 1 shows the average c-scores for ImageNet and ObjectNet experiment images split by whether the ResNet-50 correctly predicted the image. Like e-score, prediction depth serves an efficient predictor for human recognition difficulty only for images classified by the model in both ImageNet and ObjectNet.
6.11.3 Adversarial robustness
We measured image's distance the decision boundary of network using fast gradient sign method (FGSM) (9] FGSM creates an modified example that maximizes the loss using the gradients of loss with respect the input image:
mod, + c sign ( VV.J
where adv, is the modified image, is the original image, y the original input label, is multiplier adjusted accordingly to control the size of modification step, is the model parameters, and J is the
21

---

Cerrec.
Corect
2.8
0.6
Hours Heursing www.al grosoo
0.
Tearthn Cooor
3.8
06
Subset
Corret
Donec
14
24
Predecton Deeth
Prodictioo
10
Figure 13: Top: left and right are average c-score over subsets for experiment ImageNet and ObjectNet images respectively. Orange shows the images that correctly predicted by the ResNet-50 while blue shows the images that are incorrectly predicted, Bottom: prediction depth plots shown the same way as top.
loss function. Note that gradients are taken with respect to the input image, and model parameters remain constant.
For an image classified by model, we define its distance to the closest decision boundary of the model as the minimum needed for the model to misclassify the modified image. On the other hand, for image misclassified by model, we define distance the closest decision boundary of the model as the minimum needed for the model to classify the modified image.
We used the same ResNet-50 used calculate c-scores ensure consistency of our results. We finetuned the ResNet-50 on the 50 experiment classes using the ImageNet training and validation set. We used the same hyperparameters as section 6.. only updated the weights of the final pooling, flatten, and fully-connected layer. We used this finetuned ResNet-50 as the backbone for adversarial perturbation and correction.
While perturbing each classified image, we searched for the smallest c, from 0 to 0.02 incrementing by 1.25e-5 and from 0.02 to 2.5 incrementing by 0.005, that would result in misclassification We only applied only one gradient step when perturbing. While correcting each misclassified image, searched for the smallest c, from 0.001 incrementing by 1.25e-6 and from 0.001 to 0.05 incrementing by 1.25e-5. We applied two gradient steps when correcting because correction requires finer and more steps.
Note that the search range depends on the backbone model and the dataset. One must choose them through manual trial-and-errors to yield interesting and significant results. Recall that after removing images that were incorrectly annotated, incorrectly cropped, etc section 3 we reduced to 4,771 images from the original 5,000. Of these, 3,296 and 1,475 images were classified and misclassified by the finetuned 5e respectively. We were not able to find an e for every image while perturbing and correcting in the corresponding search range. We omitted these images in our analysis. We were able to successfully perturb 2,815 out of 3,296 classified images and correct 1,1 out of 1,475 misclassified images.
We hypothesized that difficult images that are classified and misclassified would be closer and further from the decision boundary respectively, fig. g confirms the prior hypothesis. We could not confirm
22

---

dataset ObjertNet
ImageNet
0.007
epsioon
0.006
0.005
0.004
0.003
0.002
17
100
150
<= 250
100000
subset
Figure 14: Average magnitude required correct misclassified images back to their correct class per subset
0.24
0.300
8:10
646
LVT cully Scors
Eiftiuuly Score
Figure 15: Distribution of difficulty score for each MVT subsets ImageNet (left) and OhjectNet (right).
the latter hypothesis due to the smaller number of misclassified images across all subsets, shown through the higher error bars in fig. 14
6.12 What factors effect MVT? Imagenet-x analysis
We found no clear trends across MVT subsets for the 16 dimensions labeled in the imagenet-x dataset. The results of our analysis can be found in table
6.13
Constructing a metric for image difficulty
We propose two metrics:
1. Difficulty score which provides exact ranking from most difficult to recognize to least difficult to recognize based on each response
2. six minimum viewing time (MVT) subsets that quantify the minimum amount of time required for the majority of participants to reliably recognize an image.
Difficulty score is a value from 0 to 42 that represents the number of incorrect predictions given by participants in our experiment across all timings for a particular image. Each image in our experiment was seen an equal number of times per timing and and only rarely were images that were recognizable shorter timings also recognizable longer timings. This results in a low difficulty score indicating that an image easy to recognize and high difficulty score indicating that an image is hard to recognize. These scores correlate well with the MVT difficulty subsets as shown fig. 15] Difficulty score varies significantly by object class well (see fig. 16
23

---

boddy
Radlnch
soffee 2148
Water bottls
dintbal
banner
4:1
prister
apaster
Vate
Rind
shoee
Drfiuby Seare dicittution
Figure 16: Difficulty distribution by object class sorted in order of increasing mean
6.14 Difficulty score distribution by object class
24

---

0.5
Random Subset 2 Distributional Accuracy Gap
0.4
Jolly
ceethes
Alddrthha M
Denontee 323
0.3
VOUNE Castter
GORUEE
CICLAA Redkes58
Barber-1l
Redtei123
thfeettee MIT
0.2
SWSL AadWett
ImageNet1k
ATtuSe
C.P-Panteee-11
Additional Data
SimCLR ResNets
VIT/16 trained on Imagenet2 1k
0.1
CLP Nopfoct 2014
SWSL models
MSDNet classifier layers
Hamere 2hteno
Lsam
CLIP models
Humons Jtcms
CUPMIUAA
CLIP ResNet models
EfficientNet trained on ImageNet21K
Homsag
0.0
EfficientNet trained Imagenet1k
Resnet
Human Performance
0.0
0.1
0.2
0.3
0.4
0.5
Random Subset Distributional Accuracy Gap
Figure 17: Robustness gap for our finetuned models on two randomly sampled subsets of our experiment data, ObjectNet. Lines connect model families with arrows pointing in direction of balanced between ImageNet and increasing model capacity. Compare with tig.
25

---

0.5
ImageNet1k
ResNet-18
ResNet-50
ResNet-101
ResNet-152
ResNet-152
0.4
ResNet-50-60%
A 10s Subset Distributional Accuracy Gap
ResNet-50
ResNet-152-60%%
ResNet-101-80%
ResNet-50-80%
80%
ResNet-101--22
ResNet-101-609%
ResNet-152-440%
ResNet-18-80%
ResNet-101-40%
hesNet-18-409
ResNet-18 8-60%
ResNet-152-20%
ResNet-18
ResNet-50-20%
ResNet-18-20%
0.2
0.0
0.2
0.5
17ms Subset Distributional Accuracy Gap
Figure 18: Robustness gap for our finetuned ResNets trained on varying percentages of the ImageNet training set. Lines connect the same architectures with arrows pointing in direction of increasing dataset percentage. Compare with fig. 7
26

---

1.0
0.8
Accuracy
0.6
0.4
0.2
0.0
103 Presentation Time (ms)
102
10a
Figure 19: Human accuracy vs Image presentation time from Mechanical Turk results. Time log-scale
27

---

Table 4: Model accuracy on the ImageNet and ObjectNet subsets of our 4,771 images.
ResNet-118
80.4
48.8
ResNet-18-20%
65.0
31.1
ResNet--18-40
71.5
39.9
ResNec-11-600%
75.3
43.6
Re-8et-11-800%
75.6
45.2
ResNet-50
87.1
60.0
ResNet-50-20%
71.0
38.3
ResNet-500440%
77.7
46.4
ResNet-50-60%
80.4
51.6
ResNet-50-80%
82.9
52.6
ResNet-101
89.2
62.7
ResNet-101-20%
73.2
37.8
ResNct-101-40%%
79.5
45.0
ResNct-101-60%%
81.9
52.7
RcsNet-101-80%
84.6
56.7
ResNet-152
90.3
64.5
ResNet-152-20%%
73.4
38.5
ResNet-152-40%%
78.7
47.9
ResNet-152-60%
82.3
51.4
ResNet-152-80%
84.6
54.8
CORNet-S
81.6
52.6
VOneNet-Resnet50
84.0
54.8
VOneNet-CORNettS
81.3
48.8
VGG-19
81.1
50.2
Noisy Student (Efficien(Net-L2)
86.2
51.2
DenseNet-122 121
83.7
55.7
MSDNet Classifier 0
62.8
29.3
MSDNet Classifier 1
74.6
41.6
MSDNet Classifier 2
78.6
49.6
MSDNet Classifier 3
78.9
51.6
MSDNet Classifier 4
81.0
52.8
SimCLR ResNet50
76.4
46.1
SimCLR ResNet101
82.1
55.4
SimCLR ResNet152
84.3
56.0
CLIP-VI-BB32
80.0
63.3
CLIP-ViT-B/16
84.5
73.6
CLIP-VIT-LAA
92.1
85.8
CLIP-VIT-L/14@336GX
92.0
86.9
CLIP-ResNet-50
71.6
56.8
CL.P--ReNNet-011
75.3
61.4
CLIP-RRsSNet-50x4
77.5
64.6
CLIP-ResNet-50x16
80.1
72.4
CLIP-ResNet-50x64
87.1
79.4
EfficientNet-S
83.1
48.6
EfficientNet-M
81.7
47.2
EfficientNet-L.
85.8
52.7
EfficientNet---21
90.2
66.4
EfficientNet-M-21
91.4
68.2
EfficientNet-L-211
91.5
68.8
VIT-T/16
59.4
29.9
VIT-S/16
84.4
57.1
VIT-B/16
86.5
61.3
VIT-L/16
94.1
78.1
MoCo-V3
85.5
56.6
SW.SL-ResNex1001-224
95.1
82.3
SWSL-ResNet50
93.4
75.4
MAE-V VIT-VVT-
89.5
65.2
28

---

Table 5: ImageNet-x factors % of MVT subset. Each table entry represents the percentage of the images
MVT subset (row) were labeled containing feature (column). This analysis over the ImageNet images our dataset.
multiple objects
background
color
brighter
darker
style
larger
smaller
MVT subset
17 ms
0.00
20.69
22.76
0.69
0.69
0.00
0.00
0.00
50 ms
0.15
25.23
20.39
0.15
0.15
0.15
3.32
ms
28.23
16.13
0.00
0.00
5.65
150 ms
0.00
25.56
16.67
0.00
1.11
0.00
0.56
4.44
250 ms
0.00
29.89
12.64
0.00
0.00
0.00
0.00
3.45
10
0.00
27.27
16.94
0.00
1.24
0.00
0.41
4.13
object blocking
person blocking
partial view
pattern
pose
shape
subcategory
texture
MVT subset
17 ms
0.00
0.00
0.69
27.59
21.38
3.45
1.38
0.69
50
0.00
0.00
1.06
23.26
21.75
1.36
2.27
0.76
ms
0.40
0.00
20.56
20.97
2.42
3.63
0.40
150 ms
0.56
0.00
1.67
22.78
20.56
3.89
1.11
1.11
250 ms
0.00
1.15
4.60
22.99
19.54
1.15
3.45
1.15
10 sec
0.00
0.00
2.07
19.42
22.73
4.13
0.83
0.83
29


Reviews:
[{"review_id": "dOZnkXCzsr", "reviewer": "Reviewer_HVkw", "summary": "", "questions": "", "limitations": "The authors note the limitations of the Mechanical Turk experiments, especially at low time intervals. The limitation in control over the users environment and screen lead to an over-estimation of human accuracy.  The note the limitations in number of participants in the in lab experiment. The authors address the safety issues that may arise running human experiments, and state that the MIT IRB oversaw the experiment.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed MVT metric is a convincing way to measure difficulty of an image. The authors seek to better understand model performance and the model human performance gap better. The number of methods and models analyzed is very thorough. The paper convincingly shows a gap in performance - that humans are still able to classify the hardest ImageNet images consistently where as models cannot. With such high accuracy on ImageNet overall, focussing on high MVT images could help measure meaningful gains in model performance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dd5SCZlmGH", "reviewer": "Reviewer_cj8A", "summary": "", "questions": "", "limitations": "The authors properly describe their paper' limitations in the conclusions.", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper is very well written, ideas flow very clearly, and concepts are presented very well.\n- This paper fills an important gap in the state of the art, difficulty estimation methods are evaluated mostly qualitatively, as there is no ground truth difficulty labels, and this paper provides a human-based difficulty metric, so a new dataset of images to with human-based ground truth difficulty is built.\n- The experiment for obtaining a ground truth difficulty makes sense to me, it is base on neuroscience background knowledge, and overall it seems to produce meaningful results. And it is very positive to capture data from both mechanical turk (online) and in-person lab subjects, to remove possible biases in both.\n- The viewing time metric seems to be novel as applied to difficulty estimation of machine learning models, and it is used to evaluate the difficulty distribution of ImageNet and ObjectNet, with important conclusions for the literature and future research: Both ImageNet and ObjectNet are skewed in their difficulty distribution towards easy images and there is less hard images. We should aim to obtain more difficult datasets and not scale datasets with easy images.\n- A major result from this paper is that ML models like ResNet and Vision Transformers seem to have lower accuracy on more difficult images, while humans do not seem to have this problem, with almost constant accuracy for varying image difficulty. The best performing model according to accuracy vs difficulty plots seem to be CLIP.\n- Three difficulty estimation methods from the state of the art are compared with the ground truth difficulty obtained in the human trials, and there is a good degree of correlation, showing that there is some relation between model difficulty and human difficulty, but still new metrics and models are needed that mimic human difficulty closely.\n- I believe the experimental setup, dataset building and labeling, and evaluation are correctly done and valid, and I tend to trust these results.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wBpATHApdy", "reviewer": "Reviewer_ZsVb", "summary": "", "questions": "", "limitations": "Yes, the limitations section is appropriate.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "-\tThe paper identifies a relevant problem in the community and provides a well thought-out solution. A quantitative metric for dataset difficulty would be helpful for characterizing such benchmarks.\n-\tThe paper details a rigorous procedure for collecting the MVT annotations from humans that avoids a range of possible design flaws.\n-\tMeasuring the gap between ImageNet and ObjectNet performance is a good way to avoid explicit model accuracy comparisons.\n- The paper is generally well-written.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "s9eeCXRvDB", "reviewer": "Reviewer_L4sU", "summary": "", "questions": "", "limitations": "This work promotes greater interaction between computer vision and object recogntion psychophysics and I can't see any severe limitations. It would be interesting to identify images where difficulty is inverse correlated for humans and machines.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Proposes to bridge the gap between human recognition performance and machine recognition performance by incorporating minimum viewing time based difficulty as a criteria.\n2. Demonstrate the loss of performance of the models in case of samples that are hard to recognize for humans.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vMMGIMeaHr", "reviewer": "Reviewer_NiF3", "summary": "", "questions": "", "limitations": "The comparison of models to the human brain seems quite unsophisticated, and it is potentially concerning if the analysis in the paper is used to make conclusions about the similarity between the two. The crowdsource and in-lab human experiments done in the paper seem like they were properly conducted.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The experimental design, as shown in Figure 2, is efficient and logical.\n2. The analysis of ImageNet and ObjectNet, which are widely used datasets, is very important and can significantly impact further research that is done with these datasets.\n3. Figure 7, which depicts image difficulty vs. robustness, is very insightful and provides useful information about popular models.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The main contribution of this paper is a new dataset difficulty metric, MVT (Minimum Viewing Time). There were several interesting findings from this paper such as the existing image datasets generally consist of easy images, the human-level difficulty and the machine accuracy have a correlation, and most model families don't scale equally on hard/easy images (Figure 7).

After the discussion period, this paper has mixed opinions. Three out of five reviewers (cj8A, L4sU and NiF3) recommended acceptance, while two reviewers (HVkw and ZsVb) recommended borderline rejection.

The main concerns raised by the reviewers include the unreliability of the proposed metric, no investigation of the source of the difficulty, the generalizability to the other datasets rather than object datasets, and the fact that MVT judgment always requires expensive human labor.

During the reviewer-author discussion period, most concerns are resolved by the discussions. As far as I understood, the authors did provide additional materials for the concerns, but the reviewers did not respond to their last materials. When I checked the additional materials, such as the authors' rebuttal and the COCO experiments, I think the reviewers' concerns were generally well addressed.

Although some reviewers still stand for a negative decision, considering the materials provided by the authors during the rebuttal period, I recommend acceptance.
Decision: Accept (Poster)