SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David Mayo*  
CSAIL & CBMM  
MIT
Jesse Cummings*  
CSAIL & CBMM  
MIT
Xinyu Lin* CSAIL & CBMM MIT
Dan Gutfreund  
MIT-IBM Watson AI Lab  
IBM
Boris Katz  
CSAIL & CBMM  
MIT
Andrei Barbu  
CSAIL & CBMM  
MIT
Humans outperform object recognizers despite the fact that models perform well on current datasets, including those explicitly designed to challenge machines with debiased images or distribution shift. This problem persists, in part, because we have no guidance on the absolute difficulty of an image or dataset making it hard to objectively assess progress toward human-level performance, to cover the range of human abilities, and to increase the challenge posed by a dataset. We develop a dataset difficulty metric MVT, Minimum Viewing Time, that addresses these three problems. Subjects view an image that flashes on screen and then classify the object in the image. Images that require brief flashes to recognize are easy, those which require seconds of viewing are hard. We compute the ImageNet and ObjectNet image difficulty distribution, which we find significantly undersamples hard images. Nearly  $90\%$  of current benchmark performance is derived from images that are easy for humans. Rather than hoping that we will make harder datasets, we can for the first time objectively guide dataset difficulty during development. We can also subset recognition performance as a function of difficulty: model performance drops precipitously while human performance remains stable. Difficulty provides a new lens through which to view model performance, one which uncovers new scaling laws: vision-language models stand out as being the most robust and human-like while all other techniques scale poorly. We release tools to automatically compute MVT, along with image sets which are tagged by difficulty. Objective image difficulty has practical applications – one can measure how hard a test set is before deploying a real-world system – and scientific applications such as discovering the neural correlates of image difficulty and enabling new object recognition techniques that eliminate the benchmark-vs-real-world performance gap.
Numerous efforts exist to build better evaluations for object recognizers. Broadly, these fall into four categories. Those that probe distribution shift, like ImageNetV2 [1]. Those that add scale like OpenImages [2]. Those that explicitly attempt to make images more difficult for models by adversarially selecting them, like ImageNet-A [3] or adding artificial corruptions, like ImageNet-C [4]. And those that attempt to explicitly control for biases like ObjectNet [5]. These are responses to the fact that performance on standard benchmarks does not translate well to real-world conditions;
*Equal contribution. Website https://objectnet.dev/mvt Corresponding author dmayo2@mit.edu
(a) ImageNet image difficulty distribution
(b) ObjectNet image difficulty distribution
Figure 1: ImageNet and ObjectNet image difficulty distributions. Difficulty here is defined as how many participants failed to recognize a given image across viewing times. The difficulty of both datasets is roughly the same, significantly under-sampling hard images. Slightly under  $90\%$  of performance metrics are derived from easy images. Today, we largely as a community only test what is easiest for the human visual system.
90% accuracy for one class in ImageNet does not mean that the detector will achieve 90% accuracy for that class in one's home or on frames of a movie. In all four cases, these efforts have no objective guide, no metric that evaluates how far they have progressed towards enabling models to generalize.
We set out to measure an orthogonal quantity - how difficult images in these datasets are for humans. Distribution shift and bias control won't on their own address this problem if datasets are overwhelmingly easy compared to what humans are capable of recognizing. And while scale helps, if datasets are heavily skewed toward images that are easy for humans, the statistics of performance on such datasets may hide the real underlying performance trends of models on harder images.
An objective metric by which to measure the difficulty of computer vision datasets has several advantages. First, we can determine if there are gaps in our datasets; perhaps certain difficulties are systematically undersampled. We find that this is the case: hard images are essentially missing. Moreover, merely aiming for distribution shift, even by changing how the dataset is gathered, doesn't meaningfully change the difficulty distribution; ObjectNet and ImageNet were gathered from different sources (captured by Mechanical Turk vs the web), with different goals and additional controls for ObjectNet, yet their difficulty distributions are remarkably similar. Second, we can evaluate model scaling as a function of difficulty. We find that most model families scale poorly in terms of out-of-distribution robustness, generalizing well on the easy images but hardly improving on the hard images, with the exception of CLIP [6]. Third, it provides a new kind of metric for biological plausibility, orthogonal to raw performance, error distribution, or how well networks predict neural activity. If a network is to be a model of the human visual system, not just an engineering model, then some quantity computed from that network should explain the observed difficulty scores. About half of the variance in the difficulty results is accounted for by a combination of c-score [7], prediction depth [8], and adversarial robustness [9]. Fourth, tools that measure difficulty could be incorporated into dataset collection and into how we report datasets and the overall progress of our community. In the long term, we intend to establish a dashboard giving a perspective on object recognition from the point of view of difficulty.
To build this difficulty metric, we choose as a proxy the minimum viewing time (followed by a backward mask) that a human viewer requires before being able to recognize the object in an image. Earlier readout is likely an indication that fewer mental resources were needed to recognize the image. After viewing the image, subjects have unlimited time to respond to a 1-out-of-50 forced choice task where they must identify the object class in the image that was shown. This metric is related to object solution time (OST) [10] explored in the neuroscience literature. We are of course not the first to carry out such viewing time experiments [11], but we do so at scale and with images from modern datasets. Further, we turn these results into a difficulty metric with practical applications, predict this difficulty metric from quantities computed from current networks, and show the scaling of current models. We hope that in the future, benchmarks will regularly report their difficulty distribution (they can do so for only a few hundred dollars with the tools we provide) and that collections of benchmarks will seek out datasets based on their difficulty distribution. Additional attention may need to be paid while collecting datasets to not eliminate hard examples; any quick consensus-based process with multiple annotators is likely to be heavily biased against including hard examples. Practically, when collecting datasets for domains where the cost of errors is high (the medical domain, autonomous vehicles, etc.),
being mindful of the difficulty distribution and actively shaping it to fill out harder images may be critical to building confidence in the resulting models.
1. A dataset of 200,382 human object recognition judgments as a function of viewing time for 4,771 images from ImageNet and ObjectNet
2. The first objective image difficulty metric, minimum viewing time, MVT
3. The difficulty distribution for ImageNet and ObjectNet
4. An evaluation of model performance as a function of image difficulty and a new scaling law
5. A new metric for validating the biological plausibility of models, predicting image difficulty
6. A new subset of images from ObjectNet and ImageNet sorted by difficulty for use in neuroscientific and behavioral experiments.
Image difficulty Image difficulty has previously been investigated primarily from the perspective of models. Jiang and Zhang et al. [7] constructed a new metric for image difficulty, c-score, which is well approximated by its learning speed proxy; more difficult images are classified correctly later in the course of training. Agarwal et al. [12] introduced variance of gradients (VoG), a method similar to the c-score learning speed proxy, but using the variance in the gradient updates. Baldock et al. [8] identify the first layer at which a model produces the same classification as its final output. These methods find ObjectNet significantly more difficult than ImageNet, unlike our human presentation time metric as shown in fig. 1 because ObjectNet is harder for models; they are fundamentally not objective metrics of difficulty but they depend on the training set, architecture, optimizer, etc. of a specific model. Other work [13, 14] has found that image difficulty estimates can improve task performance or inform compute trade-offs. Such approaches are promising and could likely benefit from more objective, model-free difficulty measures like ours. Meding et al. [15] investigated image classification accuracy as a function of model type, finding that many images are solved by all models, while some are never solved. Humans predicted which images were easy vs. hard. This is an overt binary judgment that requires introspection, while our work provides a graded difficulty metric which is implicit to the operation of the visual system.
Dataset design As a field, we have measured our progress by models' performance on tests created by splitting a random subset of images from large-scale image datasets. Most datasets were created by web-scraping images and labeling them according to consensus of human annotators [16]. More challenging test sets were developed by recreating existing test sets from new data (ImageNetV2 [1]), constructing adversarial test sets (ImageNet-A [3]), adding corruptions to existing test sets, (ImageNet-C [4]), and eliminating spurious correlations (ObjectNet [5]). Others have improved labeling errors in existing datasets to demonstrate, much like our own work, that model performance gains are misleading and that we should not declare victory too soon [17, 18]. Our image difficulty metric can aid all of these datasets by calibrating their difficulties and providing a roadmap for how to collect better datasets.
OOD model performance Prior work studying object recognition models performance has found a linear trend in performance improvement on ImageNet and other OOD datasets [1, 5, 19]. This linear trend can be beaten slightly when massively scaling up the quantity of training data used. Recently, the multimodal models CLIP [6] and LiT [20] have demonstrated a break from this linear trend, greatly increasing out of distribution generalization performance by using multimodal learning.
Human Judgments Collecting human behavioral data has long been integral to computer vision research. Human psychophysics has been used to improve model robustness [21] and representational power [22], to motivate evaluation metrics [23-27] and to develop challenging datasets with rich annotations [28, 29]. Modeling human behavioral data is at times an end in itself [22, 30-33]. These endeavors often find that while models have made progress toward parity with humans, they continue to lag behind in many important dimensions [34]. Especially relevant to our work is ImageNet-X, a labeling of ImageNet with human judgment annotations across 16 dimensions (pose, lighting, etc.) designed to aid in discovering and explaining failure modes of models with higher resolution. While these dimensions are a great contribution, they fail to explain the more complex phenomenon of human recognition difficulty.
Figure 2: Overview of the experiment. (left) 50 images from 50 object classes were randomly selected from both ImageNet and ObjectNet to total 5,000 images; of which we analyze 4,771. Images were cropped in a square around the object of interest and then shown to human subjects on Amazon Mechanical Turk and in a controlled laboratory setting. (middle) Participants first saw a fixation cross for  $500\mathrm{ms}$ , then the image for either  $17\mathrm{ms}$ ,  $50\mathrm{ms}$ ,  $100\mathrm{ms}$ ,  $150\mathrm{ms}$ ,  $250\mathrm{ms}$ , or  $10\mathrm{s}$ , followed by a mask. After each image, subjects were given a 1-out-of-50 forced-choice task to identify the correct object class. (right) Each image was seen by 42 subjects, seven for each of the six image durations. No subjects saw the same image twice.
Limited viewing time Geirhos et al. [35] studied human and machine performance under limited presentation time with images altered by corruptions, finding that humans are more robust than machines. Rajalingham and Isaa et al. [11] presented images at 100ms finding that models are unable to match image-level behavioral patterns of primates. While our experiments agree with these results, they provide new capabilities: measuring the difficulty of entire datasets, calibrating dataset difficulty to human abilities, discovering new scaling laws, etc. All of these are derived from revealing a new understanding of difficulty as a function of image viewing time. Geirhos et al. [36] investigated the humans-machine gap as a function of architecture, objectives, and dataset sizes using a 16-way classification experiment with 200ms presentations. In this work, we investigate an orthogonal quantity, the human-machine gap as a function of viewing time, and derive a new metric from it. Our results agree with this prior work and expand upon it.
We performed an experiment with human subjects on Mechanical Turk and in the lab in order to determine the minimum amount of viewing time required before subjects could identify an object present in an image. See fig. 2 for details. In what follows, we describe the stimuli, procedures, and validation of the online experiment with in-lab experiments.
Stimuli We selected 2,500 images from the ImageNet validation set (which contains 50 images per class; we selected all 50 for 50 classes) and 2,500 images from the ObjectNet dataset (ObjectNet is only a test set). These images were evenly distributed among 50 object classes shared between ObjectNet and ImageNet. Since ObjectNet was gathered by participants taking pictures of objects in their homes, all of the object classes are household objects (see right panel on fig. 2 for a full list of the object classes) and all participants were likely familiar with all object classes used. The 50 classes were picked to minimize similarity between classes.
At short presentation times, subjects do not have time to fixate on multiple locations. Off-center objects would be recognized by peripheral vision. To eliminate this effect, we cropped all images around the target object using the ImageNet validation bounding boxes and new bounding box annotations for ObjectNet. In each case, we produced a square 224 by 224 image, padding with black if needed. Details about cropping can be found in the appendix. Note that this eliminates clutter and focuses only on the appearance of the object; we discuss this limitation in the conclusion.
Procedures The MIT IRB oversaw the experiment. Subjects were consented and provided a warning about image flashing. Only subjects with normal corrected vision were included. Written instructions and a sample video of the experiment were provided. Subjects then read through a list of
the 50 object categories. The Mechanical Turk experiments had calibration steps to determine the screen size (utilizing credit cards since they have a standardized size) and distance from the screen using a blind spot test [37]. Images were shown at 8 degrees of visual angle. No private information was recorded and no offensive images were shown. Details about setup and calibration are available in the appendix.
For an overview of the procedures, see the middle panel of fig. 2. Each trial had a randomly selected presentation time, either  $17\mathrm{ms}$ ,  $50\mathrm{ms}$ ,  $100\mathrm{ms}$ ,  $150\mathrm{ms}$ ,  $250\mathrm{ms}$ , or 10 seconds; timings were selected to account for a participant's use of a  $60\mathrm{Hz}$  screen and to probe fine-grained distinctions between image difficulty most apparent at short timings. We selected timings shorter than the typical human pre-saccade latency (250ms) so participants had time for only one fixation – with the exception of our 10s control accuracy timing. In each trial, a fixation cross was shown for  $500\mathrm{ms}$  followed by the image, for one of the six timings. Immediately after the image, a phase gradient backward mask [38] was shown for  $500\mathrm{ms}$  to disrupt further processing of the image. Details of the mask generation are described in the appendix.
Participants were then presented with a grid of the names of 50 object categories and were asked to click on the class of the object that they saw. As this was a forced-choice experiment, subjects were told to select their best guess if they were unsure. The order of the object classes in the grid was randomized at each trial. Subjects were given an unlimited amount of time to make a choice although this time was recorded and appears to be anti-correlated with performance (quick decisions were likely to be more accurate than slow ones).
Experiments were counterbalanced. In total, each image was viewed by the same number of subjects at the same number of viewing times. Any one subject did not see the same image twice at any duration. Each participant viewed 50 images, one from each class, 25 from ImageNet and 25 ObjectNet, evenly spread across the six viewing times. The order of images, viewing times, and classes were randomized for each participant. Participants were allowed to complete several experiments; for those who did, we ensured that they saw disjoint images each time. In total, 2,647 workers took part in the experiment. Subjects on Mechanical Turk were compensated at over $10 per hour. In total, we collected 210,000 trials, after which we discovered that a small number of images (229 images) were either incorrectly annotated, incorrectly cropped, or workers circumvented our automated means for ensuring that no two workers saw the same image twice. This resulted in 200,382 trials for 4,771 images (42 presentations of each image, each of 7 subjects seeing each image at one of six timings).
In-lab validation Great variation exists between monitors and browsers, with some showing the same images for nearly twice as long at short presentation times. Subjects as well can vary in their attention or alertness. To account for this, we validated the results with experiments in the lab. These followed the same procedures described above with a  $144\mathrm{Hz}$  gaming monitor (27 inch LG-27GN950-B; 1ms GTG response time), on a machine with an NVIDIA 3060Ti, using Chrome 102. We recorded the screen at 200fps and found that the presentation times accurately reflected the time periods during which this monitor showed each image.
We selected 200 images from the 5,000 used in the previous experiment, evenly distributed across the 50 classes and the two datasets (ImageNet and ObjectNet). Each participant saw all 200 images, at random presentation times in a random order; they never saw the same image twice, and could only participate in the experiment once. Subjects were instructed to stay at a fixed distance from the screen. An experimenter was present throughout to ensure that subjects were not distracted. Lighting was kept constant by closing the blinds. In total, 12 subjects (6 male, 6 female) participated in this experiment, and as can be seen in the next section, there was widespread agreement between the in-lab and Mechanical Turk experiments. It took subjects just short of an hour to complete the 200 images. Subjects were compensated $20 for their participation.
We considered an image as recognized at some viewing time when half of the participants could classify it. Chance on the 1-out-of-50 task is  $2\%$ ; even a single correct response is an indication that something could be recognized. MVT is the minimum duration at which an image is recognized (MVT of 100ms means the image is not recognized by the majority of participants at  $< 100$  ms and correctly recognized by the majority of participants at  $\geq 100$  ms). Typical images by MVT are shown
Figure 3: Images as a function of difficulty, using the minimum viewing time (MVT) before they were reliably recognized (columns), i.e., when more than half of subjects were correct. Harder images to the right are more atypical, have more difficult lighting, more occlusions, and are sometimes more blurry. Easier images to the left are more prototypical instances of their respective object class. Additional examples are available in the appendix and online. Our experiments indicate that many datasets oversample easy images.
(a) Mechanical Turk experiments
(b) In-lab experiments
Figure 4: Accuracy as a function of presentation time: on the left are results for Mechanical Turk experiments and on the right are results for in-lab experiments. The same images (5,000 online, 200 in lab) were presented at 4 timings. Results for both conditions were similar, although in-lab experiments achieved nearly  $100\%$  accuracy with 10 second viewing times, while halving the performance at 17ms compared to MTurk. The accuracy improvement at long times is likely because of the more controlled conditions with fewer distractors in lab. The lower accuracies at short timings are likely due to issues with displaying images at short timings: some monitors are very slow and can fade the image in and out effectively displaying it for twice the intended 17ms.
in fig. 3. Images that are quickly recognized by humans—easy images—are most similar to those seen in datasets, while harder images include occlusion or difficult lighting.
An overview of accuracy as a function of viewing time online and in the lab is shown in fig. 4. Both experiments broadly agree with one another. In-lab experiments have higher variances due to having  $20\mathrm{x}$  fewer images and half as many subjects per image. In lab, the performance on short timings was significantly worse, half of that seen online. We believe this is largely due to slow monitors which display the image for significantly longer than  $17\mathrm{ms}$ , roughly twice as long. Recording screens with high-speed cameras supports this hypothesis. At the high end, subjects in lab were nearly  $100\%$  accurate,  $10\%$  higher than online. We believe this likely has to do with how distracted subjects were. In what follows, we focus on the Mechanical Turk experiments due to their scale.
Human performance drops off steeply when difficult images are shown for shorter viewing times; see fig. 5. This makes the experiment sensitive to even minor variations in difficulty.
Figure 5: Human accuracy as a function of MVT image difficulty subset. Colors denote presentation time, shown below each bar, collected by minimum viewing time, shown at the bottom. Accuracy drops off steeply when hard images are displayed at short time intervals. These results are derived from Mechanical Turk experiments; in-lab experiments have exactly the same trend but higher absolute accuracies for correctly recognized images.
SWSL (ResNet-50)
Figure 6: Accuracy of models on ImageNet (solid) and ObjectNet (dashed) as a function of image difficulty; many more models are reported in the appendix. Note how in in-lab experiments, human accuracy barely declines as a function of difficulty. while model performance drops off significantly. Subsetting difficult images provides much more overhead for model improvement. The in-lab human accuracy is computed over the subset of images shown in the in-lab experiments.
MAE (ViT-B/16)
CLIP (ViT-L/14)
4.1 How hard are today's object recognition datasets?