SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The minimum viewing time required for reliable recognition is a proxy for image difficulty. Datasets today are not gathered to control for difficulty and, indeed, when plotting the difficulty of images in ImageNet and ObjectNet, we find that the difficulty curve for these datasets is highly skewed; see fig. 1. Rather than plotting six bins, one for each viewing time, we plot a more fine-grained quantity: the total number of incorrect responses out of the 42 presentations of each image (7 participants at 6 timings). Images with few incorrect responses are easy: all participants at all timings could recognize them, even at short timings. Images with many incorrect responses are hard: few participants could recognize the images at only a few longer timings.
Our results indicate that hard images are vastly underrepresented in today's datasets. The difficulty distribution of ImageNet and ObjectNet are very similar to one another. This is despite the fact that the latter was collected in a manner intentionally intended to highlight more diverse and less biased images. Merely collecting more images does not appear to help which has strong implications for how this difficulty distribution looks for other datasets that are not measured here. Instead, we must develop new tools to guide dataset collection toward more difficult exemplars; we describe such a tool and workflow below.
4.2 What we can learn about models from hard images
Machine accuracy varies as a function of the minimum viewing time required to recognize the object in an image. Most models—but not all—see a significant performance dropoff between the easy images and the hard images, see fig. 6. See the appendix for extensive results for dozens of models broken down by image difficulty. Note that this understates human performance as it shows the Mechanical Turk results. In-lab, even for the hardest images, humans have nearly perfect performance.
These results also show that the gap between ImageNet and ObjectNet performance increases as image difficulty increases. Likely, many more phenomena are much more acute for harder rather
Figure 7: Robustness as a function of image difficulty comparing the ImageNet-ObjectNet performance gap on easy images (horizontal) and hard images (vertical) as determined by the MTurk MVT experiment. Each point is a model. Arrows connect model families, pointing from small to large variants. Horizontal model families are undesirable: they scale by improving on easy images. Diagonal model families are desirable: they scale by improving on both hard and easy images. Humans show no gap between ObjectNet and ImageNet. All model families are horizontal, regardless of dataset, supervision, training regime, or architecture. Only CLIP models stand out by scaling in a more human-like way. This scaling law has not been observed before and the reasons behind it are unknown.
than easier images just as distribution shift. Datasets intended to challenge object recognizers would benefit from sampling hard images, rather than vastly oversampling easy images as they do today.
Image difficulty can tease apart differences between recognition models that would otherwise be lost because of the skewed underlying difficulty distributions in current datasets. In fig. 7, we plot numerous object recognition models contrasting their performance on the easiest and hardest images. Rather than computing the absolute performance of models, which would naturally favor larger models with larger training sets, we measure the gap in performance between ImageNet and ObjectNet. A more robust detector is one that has a smaller gap, even if its performance is lower, as scaling models up (both in parameter size and training set size) is well understood. Models that are part of the same family are connected by arrows starting with the smaller variants pointing to the larger ones.
Humans hover around zero; they are robust with respect to the distribution shift between ObjectNet and ImageNet. Some model families are roughly horizontal, like SimCLR, or even have a negative slope. This shows that as they scale, their performance is increasing on the easy images, but not on the harder images, or in the case of negative slopes, it is widening the gap on harder images. CLIP stands out, and most closely approaches the human results.
C-Score
Prediction Depth
Figure 8: The correlation between three metrics and image difficulty. On the x axis images subset by the minimum amount of viewing time, in ms, required for the majority of participants before they are recognized. All three metrics are correlated with difficulty. Hard images are learned later in model training, predicted by later layers, and need much smaller perturbations to attack.
Perturbation  $\epsilon$
Extrapolating the performance of models on such a graph shows that many current model families are likely to stop or radically slow down performance improvements on hard images, a much less optimistic story than if one merely considers aggregate performance. At the same time, CLIP shows promise, although it too appears to eventually begin to move away from being robust to distribution shifts as model size increases. Note that these experiments use the largest CLIP models available to the public; other large models described in the CLIP publication may have even more optimistic results. SimCLR also stands out from the non-CLIP models, demonstrating a mild productive scaling trend although it is not clear whether this trend continues with larger models.
Four weeks of compute on two machines with 8 Titan RTX were used to generate the results in this paper. We release our data under the Creative Commons BY-SA license.
4.3 Explaining image difficulty and testing how similar models are to humans
If models are to not just perform well, but to also process images in ways that are similar to how the human visual system does, then they should contain a proxy for difficulty. Most models today are not recurrent, so no direct analog exists to viewing time (although, even if they were recurrent, it is unclear if directly taking the number of iterations of the model is the correct analog of viewing time). Minimum viewing time judgments provide an independent and complementary metric by which to evaluate how similar models are to human brains, in addition to their behavior, error pattern, and ability to explain neural recordings.
We investigate three quantities computed from models that could be used to explain the difficulty judgments: c-score [7], prediction depth [8], and adversarial robustness [9]. C-score is a consistency score computed while the network is training; we use the learning speed proxy to compute it. Prediction depth measures the earliest layer at which a network produces its final prediction. And adversarial robustness refers to the perturbation,  $\epsilon$ , needed to fool the network on a given example. See fig. 8 for an overview of the correlation between these metrics and difficulty. The same network with a ResNet-50 architecture trained on ImageNet was used for all experiments presented here. Note that  $\epsilon$  is only computed over the correctly classified images.
This analysis reveals that images that require more viewing time for humans are harder for networks in several ways. They are learned much later in the training process. They are predicted by later layers in the network. And they require much smaller perturbation to create adversarial examples. This underscores that human minimum viewing time has many practical consequences for how networks process images. At the same time, these metrics alone do not explain all of the difficult judgments. Logistic regression including all three metrics classifies images into the three difficulty bins (short: (17, 50), medium: (100, 150, 250), and long (10s)) with  $47.7\%$  accuracy. Above chance, but leaving plenty of room to do more. New metrics for understanding how networks process images could be validated against this data.
We also explored more intuitive explanations of image difficulty by leveraging ImageNet-X, the human-annotated dimensions of the ImageNet validation set [29]. However, we found no correlation between these annotations and MVT. Though ImageNet-X is useful for analyzing model failure modes, it fails to meaningfully explain human psychophysics table 5. Low-MVT and high-MVT images are qualitatively distinct fig. 3, and, empirically, humans are able to distinguish between hard
and easy images [39], but a more specific explanation eludes us at this time. Just like it is hard to pinpoint in language what exactly is difficult about computer vision in general, it may also prove challenging to establish semantic explanations of image difficulty.
One high level qualitative trend that we can observe from our data is that images which are easy for humans are more prototypical while difficult images are atypical in innumerable different ways. We do intend to investigate this topic further in future work.
Rather than focusing on scaling, distribution shift, or control for biases alone, we should also focus explicitly on dataset difficulty. Today's datasets skew toward being too easy by undersampling hard images. ObjectNet was designed for distribution shift and bias control and was not collected from the web, yet its distribution of image difficulties is remarkably similar to that of ImageNet. By focusing on ways to measure dataset difficulty as datasets are collected, we can better calibrate the entire community, and create the resources needed to push object recognition forward. In addition to just creating better datasets, understanding performance as a function of difficulty reveals radically different scaling curves for different models and approaches. It can also provide subsets of images that highlight different types of processing for neuroscientific or behavioral experiments.
Our approach to measuring the difficulty of object recognition tasks focuses specifically on object instances, rather than other factors that also add complexity such as saliency and clutter. By cropping images around the object, we make figure-ground segmentation far easier and separate out the issue of the difficulty of this specific instance of an object as opposed to the difficulty of finding this object within a cluttered background.
The particular notion of image difficulty derived from this experiment is just one metric of difficulty. As we describe in the introduction, other metrics exist which can be computed automatically given models that recognize objects. But those metrics rely on models and therefore change as models change. The metric we present here is both absolute - it is calibrated to what humans can achieve - and model-agnostic. Models change over time, far faster than datasets do; there is a real danger when designing datasets that their difficulty will be tuned to specific models, which can be avoided by using measures related to humans.
Having performed extensive experiments to validate this approach, measuring the difficulty of any dataset is now easy and cheap. One can sample a few hundred images and run an experiment on Mechanical Turk. This only costs on the order of hundreds of dollars per dataset and can be carried out quickly. The more critical the dataset, and the cost of object recognition failures, the more important it is to do so. We provide a toolkit with code for creating experiment stimuli, hosting the experiment, posting tasks to MTurk, and collecting and analyzing data at https://github.com/dmayo/MVT-difficulty. All the user needs to do is provide the images.
Collecting datasets without considering image difficulty results in datasets that are skewed toward easy images which overstates model performance as we have shown. We encourage the community to develop novel dataset collection pipelines to tune datasets to desired difficulty distributions. This could take the form of posthoc filtering of webscraped images, or more on-line approaches which update collection parameters to discover ideal procedures for collecting difficult images. As shown here, there is promise in developing MVT proxies so that difficulty can be determined for each image efficiently.
Since posting this work online, MVT-based image difficulty has already been used in novel ways, for example, to improve performance by more efficiently allocating limited resources in embedded settings [40]. This highlights how useful, reproducible, and widely-applicable MVT is.
With modifications to our experiment, an MVT difficulty metric could be created for multi-object classification—for example, by evaluating class recall rather than single-class accuracy—segmentation or optical flow. Other tasks are out of scope, like visual search, as they literally require multiple saccades. Calibrating our field to what humans can do across a wide range of tasks, datasets, conditions, remains a significant challenge, but one that we think can now be addressed.
5.1 Acknowledgements
We would like to thank Jim DiCarlo, Guy Gaziv, Michael Lee, Dan Yamins, Colin Conwell, Martin Schrimpf and Ko Kar for their helpful feedback and discussion about our experiments. We would also like to thank David Lu for his contributions to our early experiments and directions.
This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF-1231216, the NSF award 2124052, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19-2-1000, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0014, and the Office of Naval Research under award number N00014-20-1-2589 and award number N00014-20-1-2643. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
[1] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning (PMLR), 2019.
[2] Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 2020.
[3] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[4] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
[5] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, (ICML).
[7] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. In Proceedings of the 38th International Conference on Machine Learning (PMLR), 2021.
[8] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[9] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations (ICLR), 2015.
[10] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience, 2018.
[42] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[43] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
6.1 Hosting and maintenance plan
This dataset will be hosted on MIT servers in perpetuity at https://objectnet.dev/mvt/ with a backup on dropbox. Our dataset collection toolbox is hosted publicly on github at https://github.com/dmayo/MVT-difficulty. A datacard for this dataset will be available at https://objectnet.dev/mvt/datcard.
6.2 Object classes
Test subjects were presented with images from 50 possible object classes and asked to select which object they saw. The 50 classes were hand-picked to minimize similarity between classes that could be confusing for experiment subjects. The object classes were:
Band-Aid, T-shirt, backpack, banana, cleaver, clothes iron, coffee mug, computer mouse, digital watch, doormat, dumbbell, envelope, hair dryer, hammer, lampshade, lemon, lipstick, match, mobile phone, necklace, padlock, paintbrush, paper towel, park bench, pill bottle, pillow, plastic bag, plunger, power drill, printer, racket, ruler, safety pin, salt shaker, sandal, screw, shovel, space heater, spatula, speaker, strainer, sunglasses, teddy bear, television, umbrella, vase, wallet, waste container, water bottle, whistle
6.3 Image Selection
After choosing object classes, we selected images for the experiment. We used all 50 images belonging to a class in the ImageNet Validation set with no additional selection step. For ObjectNet, we collected bounding box data for the images, and then randomly selected 50 images per class such that when cropped to the bounding box, the object in the image was centered and clear.
6.4 Image cropping procedure
1. We draw a bounding box around the object (we use existing bounding boxes for the ImageNet validation set and collect our own bounding boxes for ObjectNet from MTurk).
2. We initialize the cropping box to be the bounding box.
3. If the cropping box does not form a square, we extend the shorter side of the rectangular cropping box to form a square. If the image is not large enough to extend the shorter side of the cropping box, we pad it with black pixels to form a square.
4. We crop using the cropping box for the image. The cropped image will be a square.
5. We resize the cropped image to be  $224 \times 224$  pixels.
6.5 Mask generation
The masks were generated following the procedure used by [41]. Specifically, a Fourier transform was applied to each image to obtain the magnitude and phase components. Then, a random array with elements sampled uniformly from  $[0, 1]$  was added to the image phase component after which the magnitude and phase components were recombined via an inverse Fourier transform to produce the mask. Each image was paired with its particular phase-scrambled mask in the experiments.
6.6 Experiment Procedure and Payment
Participants both in the lab and on Mechanical Turk were presented with a document informing them of the purpose, privacy, and risks associated with the experiment and soliciting their consent to participate (see fig. 10). Participants were then instructed as to how to carry out the experiment and were shown an example video as well as the list of image classes for their review before beginning. They were informed that they would not need to memorize the classes as the classes would be shown to them after each video. Participants were also encouraged to take breaks should they feel fatigued or otherwise uncomfortable. Example instructions are shown in fig. 11
After giving consent and reading the experiment overview, participants then completed two calibration steps for to estimate the size of their monitor and their distance from the screen for us to then size
Table 1: Dataset statistics
number of responses	200,382
number of images	4,771
number of presentaiton durations	6
number of response per image	42
number of objectnet images	2415
number of imagenet images	2356
number of participants	2647
the videos appropriately to 8 degrees of visual angle. First, the participants are shown an image of a credit card and are asked to use a card of their own to adjust a slider to change the size of the card on the screen to the size of their card. Since credit cards are the same size around the world, this allows us to measure the pixel-to-inches ratio of the participant's monitor. Next, the participant completes a blind-spot test [37] that allows us to estimate the distance they are sitting from their screen. Together, these two measurements are sufficient to compute the desired video eccentricity. See fig. 12 for images of the calibration steps.
The estimated hourly wage for participants on Mechanical Turk and in the lab was  $10/hr and$ 20/hr respectively with approximately $15,000 spent in total on participant compensation.
6.7 In-Lab Experiment Results
To corroborate our Amazon Mechanical Turk results, we selected 200 images shown to Turk workers to conduct the same experiment in a controlled laboratory setting. 12 individuals came to participate in the experiment in which they viewed and responded to all 200 images on our  $144\mathrm{Hz}$  refresh rate monitor with 1ms gray-to-gray time. After conducting the experiment, 3 individuals had seen each image at each of the 4 presentation times. When compared to the MTurk results for those same 200 images, the comparison is much as we would expect. The In-Lab accuracy with shortest image duration (17ms) is less than on MTurk which can likely be contributed to the use of our new, high refresh-rate monitor in the controlled environment. It is likely that MTurk workers' personal computers differ in their graphics presentation abilities which may result in the image being visible for slightly greater than 17ms on some monitors. On the other end, the in-lab experiments reported higher accuracy at the longest image duration (10s) which is also unsurprising as the in-lab participants completed the task in a controlled environment with no distractions and are likely more inclined to take the task seriously and stay focused. The results show no significant differences in accuracy at the intermediate image durations. See fig. 4 for a side-by-side comparison between MTurk and In-Lab results.
6.8 Dataset statistics
We collected 42 human responses for each of 5,000 images (2,500 from ImageNet and 2,500 from ObjectNet). After reviewing response, 229 images were removed due to either being unrecognizable, mislabeled, or having been seen by the same worker twice despite safeguards in place to disallow it. Additional dataset statistics are listed in table 1.
6.9 Preliminary COCO MVT Results
To bolster our claims about the difficulty of current datasets, we conducted the MTurk MVT experiment on a small subset of the COCO dataset. As COCO is a more visually complex dataset than many single object classification datasets, it provides a good litmus test for how our conclusions generalize to other kinds of datasets. See 9 for the results.
6.9.1 Image selection
To maximize the utility of our results for both computer science and neuroscience research we selected 732 images from the Natural Scenes Dataset[42], a subset of COCO for which fMRI data was collected from human participants. We used the image crops used in the NSD experiments. These crops ensure that the image is square.