SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kushin Mukherjee $^{1*}$ , Holly Huey $^{2*}$ , Xuanchen Lu $^{2*}$ , Yael Vinker $^{3}$ , Rio Aguina-Kang $^{2}$ , Ariel Shamir $^{4}$ , and Judith E. Fan $^{2,5}$
University of Wisconsin-Madison<sup>1</sup>
Tel-Aviv University3
Reichman University
Sketching is a powerful tool for creating abstract images that are sparse but meaningful. Sketch understanding poses fundamental challenges for general-purpose vision algorithms because it requires robustness to the sparsity of sketches relative to natural visual inputs and because it demands tolerance for semantic ambiguity, as sketches can reliably evoke multiple meanings. While current vision algorithms have achieved high performance on a variety of visual tasks, it remains unclear to what extent they understand sketches in a human-like way. Here we introduce SEVA, a new benchmark dataset containing approximately 90K human-generated sketches of 128 object concepts produced under different time constraints, and thus systematically varying in sparsity. We evaluated a suite of state-of-the-art vision algorithms on their ability to correctly identify the target concept depicted in these sketches and to generate responses that are strongly aligned with human response patterns on the same sketch recognition task. We found that vision algorithms that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but there remains a sizable gap between model and human response patterns. To explore the potential of models that emulate human visual abstraction in generative tasks, we conducted further evaluations of a recently developed sketch generation algorithm [91] capable of generating sketches that vary in sparsity. We hope that public release of this dataset and evaluation protocol will catalyze progress towards algorithms with enhanced capacities for human-like visual abstraction.
Abstraction is key to how humans understand the external world. Abstraction enables distillation of individual sensory experiences into compact latent representations that support learning of new concepts [80, 45, 66, 30] and efficient communication about these concepts with others [23, 85, 34, 28]. For example, while no two roses are identical, people can rapidly infer what properties make a flower a rose and not some other kind of flower from just a few examples [99, 50], especially when these examples are selected to support such strong inferences [32, 77].
1/\*Equal contribution
1.1 Human Visual Abstraction as Key Target for AI
Visual abstraction enables humans to express what they know about the visual world by creating external representations that highlight the information they judge to be most relevant in any given context—for instance, pictures that highlight the visual features that are diagnostic of a rose in a botanical field guide [24, 23, 92]. Critically, there are many different ways to depict even the same object—from a detailed illustration to a simple sketch. The Spanish artist Pablo Picasso famously demonstrated this point in *The Bull* (1945), a series of 11 lithographs of bulls, each sparser than the last (Fig. 1). While some of the drawings in this series look more realistic and others more stylized, all of these images remain evocative of a bull (and perhaps other similar animals, such as a moose or buffalo) to most human viewers.
Drawing is one of the most accessible, enduring, and versatile techniques that humans in many cultures use to encode ideas and emotions in visual form [41, 1, 31]. Even without special training, humans can robustly produce and understand simple line drawings or sketches of familiar visual concepts [24, 81, 43]. The ability to leverage drawings to understand and convey key aspects of the visual world emerges early in childhood [62, 5, 44] and improves throughout development with children's expanding conceptual knowledge [19, 59, 42]. Moreover, failures to produce and recognize drawings of objects are associated with semantic dementia [10, 73], suggesting links between a robust capacity for visual abstraction and the organization of semantic knowledge in the brain.
Figure 1: Pablo Picasso. The Bull, 1945.
In addition to drawings that represent objects and scenes, other
abstract human-made visualizations (e.g., maps, diagrams, charts, graphs) serve important functions in many domains, including all branches of science and engineering [89, 90, 39, 13, 11]. Given the ubiquity and importance of such visualizations in modern life, developing computational models that achieve human-like understanding of freehand sketches is an important milestone. Such computational models of human visual abstraction stand to not only advance our understanding of human intelligence, but to also make AI systems more robust and general [40, 63, 27]. For example, prior work has found that incorporating principles based on the structure and function of the human visual system have led to vision models that are more robust (e.g., to adversarial attacks) [6, 55, 25].
1.2 Desiderata for Evaluating Alignment Between Human and Machine Visual Abstraction
The past several years have seen remarkable progress in the development of increasingly performant general-purpose vision algorithms [78, 36, 20, 70], with some of the most prominent algorithms also capable of emulating key aspects of how the primate brain encodes natural visual inputs [102, 48, 110, 47]. Over the same period, artificial vision systems have also been steadily achieving higher performance on tasks involving abstract visual inputs, including sketch categorization [21, 108, 4, 106], sketch segmentation [54, 104], sketch-based image/shape retrieval [22, 74, 105, 82, 9], among others [88, 100, 60, 16]. Moreover, these models have been found to predict human behavior on sketch recognition tasks to some degree [24, 23]. However, these otherwise high-performing vision algorithms struggle to simultaneously achieve robust understanding of visual inputs across multiple levels of abstraction [3, 79, 23]. Moreover, one study found that current vision models trained on natural images still fall short of the representational capabilities of the inferotemporal cortex, a key brain region supporting object categorization, in generalizing to new image distributions, including sketches [2]. Nevertheless, more recently developed models trained on substantially larger and varied datasets show promise on both tasks involving images with different visual styles [70, 107, 76] and tasks that go beyond recognition, including sketch generation [91, 69, 97] and sketch-guided image generation [61, 109, 96, 93, 56].
At present, it remains unclear to what degree any state-of-the-art models achieve human-like understanding of line drawings that vary in their degree of abstraction, much less the full range of abstract images that humans regularly engage with. Gaining further clarity on this question requires meeting two key challenges: first, creating a dataset containing drawings of a wide variety of object concepts that also systematically vary in their degree of abstraction; and second, developing evaluation protocols that can be used to estimate the degree to which any model emulates human-like understanding of this suite of drawn images.
Dataset. Meeting the first challenge requires going beyond existing sketch datasets [21, 74, 43, 18, 22, 54, 105, 82, 26, 95, 75, 51-53, 67, 111, 57]. While most of these datasets span a reasonably wide range of visual concepts (i.e., ranging from 125 concepts in Sketchy to 345 in Quickdraw) and some of them contain fine-grained information (e.g., stroke information and photo-sketch pairing), none of them systematically varied how detailed individual sketches could be, one of the most straightforward ways of inducing variation in semantic abstraction [8, 23, 103]. Our paper addresses the gap by providing sketches with controlled levels of detail, while encompassing the variety and granularity present in existing datasets (Table 1).
Evaluation protocol. Meeting the second challenge requires going beyond simple accuracy-based model performance metrics alone. Instead, it is critical to measure detailed patterns of human behavior on the same sketch understanding tasks to evaluate how well any model emulates these behavioral patterns, following recent work in the computational neuroscience of vision [71, 7, 68].
1.3 SEVA: A Novel Sketch Benchmark for Evaluating Visual Abstraction in Humans and Machines
In recognition of the above desiderata, here we introduce SEVA (Sketch-based Evaluations of Visual Abstraction), a new sketch dataset and benchmark for evaluating alignment between human and machine visual abstraction.
Dataset. Our dataset contains approximately 90K human-generated sketches of a wide variety of visual objects that also systematically vary in their level of detail, and thus the variety of meanings they evoke. Each sketch is associated with one of 2,048 object instances belonging to one of 128 object categories selected from the THINGS dataset [38]. We achieved variation in sketch detail by imposing constraints on how much time humans ( $N = 5,563$  participants) had to produce each sketch (i.e., 4s, 8s, 16s, 32s).
Evaluation protocol. Leveraging these human-generated sketches, we systematically evaluated how well a diverse suite of 17 state-of-the-art vision models generate classification responses that align with those produced by humans ( $N = 1,709$  participants) tasked with identifying the most appropriate concept label for each sketch. Of these participants, 579 participants also participated in the sketch production study but were not shown any of their own sketches during this study. We evaluated human-model alignment using three different metrics: (1) top-1 classification accuracy, reflecting raw sketch recognition performance; (2) Shannon entropy of the response distribution, reflecting the degree of uncertainty about the target label; and (3) semantic neighbor preference, reflecting the degree to which models and humans generated off-target responses that were semantically related to the target label.
Summary of key findings. We found that sparser human sketches produced under more severe time pressure (e.g., 4 seconds) exhibited greater semantic ambiguity—in other words, both humans and models assigned a greater variety of labels to them than to the more detailed sketches that took more time to make (e.g., 32 seconds). Furthermore, we found that models that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but none of the models came close to approximating human response patterns to human-generated sketches at any level of detail. To explore the potential of models that emulate human visual abstraction in generative tasks, we conducted further evaluations of CLIPasso, a recently developed sketch generation algorithm [91] capable of generating sketches that vary in sparsity (measured by number of constituent strokes in a sketch). We discovered that the most detailed CLIPasso-generated sketches converged with human sketches of the same object concepts, as measured by the distribution of labels humans assigned to sketches made by both agent types; however, sparser CLIPasso sketches diverged from human sketches of the same object concepts, reflecting a gap between how CLIPasso and human participants attempted to preserve sketch meaning under more severe production constraints.
2.1 Human Sketch Production
A core contribution of this work is a new dataset containing human-generated sketches of a wide range of visual object concepts that also systematically span multiple levels of semantic abstraction. We created this dataset by crowdsourcing these sketches online, following prior work [24, 105, 74, 23, 34, 43]. Each sketch in the dataset was recorded as a bitmap image as well as a collection of stroke coordinates, thus preserving the precise cursor movements a participant enacted to create the sketch.
Figure 2: Humans and CLIPasso generated approximately 90K sketches under various production constraints.
Participants. 5,563 participants (2,870 male;  $M_{age} = 36.7$  years) were recruited from Prolific and compensated  $\$15.50/$ hour for their participation. Data from 104 of these sessions were excluded from subsequent analyses due to technical issues (e.g., images did not load). All participants provided informed consent in accordance with the UC San Diego IRB.
Object concepts. We included 128 concrete real-world object categories (e.g., "lion", "banjo", "car") sourced from the THINGS dataset. We used the THINGS dataset [38, 37] because it is a well validated set of concrete, real-world visual object categories designed to support interoperability among large-scale studies in human visual cognition and cognitive neuroscience. For each of these 128 object concepts, we randomly sampled 16 object instances represented by color photographs, which served to visually ground the human sketch production task. As such, each sketch in our dataset is uniquely associated with one of these 2,048 object instances, and our final sample size was determined by our predefined goal of obtaining at least 10 human sketches of each of these instances.
Sketch production task. In each session, participants produced sketches of 16 different object categories, randomly sampled from the full set of 128 object categories. On each trial, they were cued with a color photograph  $(500\mathrm{px} \times 500\mathrm{px})$  of an object paired with its concept label. Each participant was randomly assigned to one of four conditions, defined by the maximum amount of time participants could take to produce their sketches: 4 seconds, 8 seconds, 16 seconds, or 32 seconds (Fig. 2, left). Such random assignment of participants to condition ensures that estimates of differences between conditions will not, in expectation, be biased by individual differences in sketching behavior. Participants drew on a digital drawing canvas  $(500\mathrm{px} \times 500\mathrm{px})$  using whatever input device they already had available (e.g., mouse, stylus) and were able to undo their most recent
Dataset Name	Dataset Contents	# Classes	Stroke Info?	Photo Cue?	Abstraction?
TU-Berlin [21]	20K sketches	250	✓
QuickDraw [43]	50M sketches	345	✓
QuickDrawExtended [18]	330K sketches, 204K photos	110
SPG [54]	20K sketches w/ stroke grouping	25	✓
SBSR [22]	1.8K sketches, 1.8K 3D models	161
QMUL [105, 82]	1.3K sketches, 1.3K photos	3	✓
Sketchy [74]	75K sketches, 12K photos	125	✓	✓
SketchyCOCO [26]	14K sketches, 14K photos	17	✓	✓
SEVA	90K sketches, 2048 photos	128	✓	✓	✓
Table 1: Comparison between SEVA and prior sketch datasets.
stroke or completely clear their canvas if needed. They were encouraged to make their drawings as recognizable as they could at the concept level and to use the photograph only to remind them of what individual objects belonging to that category generally look like. A countdown timer indicated how many seconds they had left to produce their drawing. Each trial ended either when time ran out or when the participant indicated that they wished to continue to the next trial, but participants were encouraged to use the full time available to produce as recognizable of a drawing as they could. At the beginning of the session, participants were explicitly instructed not to include any background context (e.g., grass in a drawing of a "horse"), arrows, or text. Participants also completed one practice trial (that we did not include in analyses) at the beginning of the session to familiarize themselves with the drawing interface. Our final dataset contains 89,797 sketches after filtering out invalid sketches (e.g., blank canvases).
2.2 Human Sketch Understanding
A key component of any evaluation of how well current vision algorithms emulate human visual abstraction is measurement of human behavior in tasks relying on visual abstraction. Here we focus on characterizing what meanings humans extract from the collected sketches, providing the basis for our subsequent empirical evaluation of how well any state-of-the-art vision model approximates human response patterns when presented with the same sketches.
Participants. 1,709 participants (776 male;  $M_{age} = 39.2$  years) were recruited from Prolific and compensated  $\$ 15.50/$ hour for their participation. Data from 21 of these sessions were excluded from subsequent analyses due to technical issues. Our predefined criterion for stopping data collection was acquisition of at least 12 recognition judgments for each sketch.
Sketch recognition task. In each session, participants provided labels for 64 sketches randomly sampled from a fixed set of 8,192 sketches, approximately  $10\%$  of the full human sketch dataset. The specific set of 8,192 sketches included in this experiment was determined by randomly sampling one sketch cued by each object instance from each drawing-time condition (i.e., 16 instances/concept  $\times$  128 categories  $\times$  4 drawing-time conditions  $= 8,192$ ). On each trial, participants were presented with a single sketch  $(300\mathrm{px} \times 300\mathrm{px})$  and a text field where they could provide their best guess concerning the concept the sketch was intended to convey. As soon as they began typing, a drop-down menu appeared with suggested word completions. This drop-down menu contained the entire set of 1,854 labels in the THINGS dataset and only responses that matched one of these 1,854 labels were accepted. Because many words have multiple meanings, the labels contained in this dropdown menu were also accompanied by disambiguating text (e.g., to distinguish mouse (animal) from mouse (computer)). If participants were unsure of which label best applied to the sketch, they were encouraged to provide additional guesses (up to 5 per sketch). Collecting multiple labels on each drawing trial was important because it enabled us to more thoroughly sample the distribution of meanings that each sketch evoked for human participants (i.e., which labels came to mind and how often they did so). At the beginning of the session, participants completed a practice trial (that we did not include in analyses) to familiarize themselves with the labeling interface.
2.3 Machine Sketch Understanding
We propose a generic protocol for evaluating machine sketch understanding that can be applied to any vision algorithm using our sketch dataset. In this paper, we conduct evaluations of a wide range of state-of-the-art vision models with the goal of demonstrating the feasibility of our protocol and guiding future model development.
Model Suite. Specifically, we evaluated 17 vision models spanning a wide range of architectures and training methods (Table 2), all of which have been demonstrated to achieve high performance on object recognition on standard datasets, such as ImageNet [17]. We also made sure to include variants of standard ConvNet and Transformer models that have gained traction within the field of computational cognitive neuroscience for their potential to close the gap between biological and artificial vision [46, 25, 49, 65].
Model	Architecture	Training Paradigm	Dataset
VGG-19 [78]	VGG-19	supervised	ImageNet
Inception-V3 [84]	Inception-V3	supervised	ImageNet
ResNet-50 [36]	ResNet-50	supervised	ImageNet
ViT-B [20]	ViT-B	supervised	ImageNet
Swin-B [58]	Swin-B	supervised	ImageNet
MLPMixer-B [87]	MLPMixer-B	supervised	ImageNet
CORnet-S [49]	CORnet-S	supervised	ImageNet
Harmonization [25]	ViT-B	supervised	ImageNet + Human Feature Importance [25]
ECOSET [65]	ResNet-50	supervised	ECOSET [65]
SimCLR [14]	ResNet-50	self-supervised	ImageNet
MoCo-v3 [15]	ViT-B	self-supervised	ImageNet
DINO [12]	ViT-B	self-supervised	ImageNet
MAE [35]	ViT-B	self-supervised	ImageNet
CLIP [70]	ViT-B	self-supervised	WebImageText [70]
IPCL [46]	AlexNet	self-supervised	ImageNet
Noisy Student [98]	EfficientNet-b4	semi-supervised	ImageNet + JFT [83]
SWSL [101]	ResNet-50	semi-supervised	ImageNet + YFCC-100M [86] + IG-1B-Targeted [101]
Table 2: Model suite annotated by backbone architecture, training paradigm, and training dataset.
Evaluation Protocol. The goal of our evaluation protocol was to measure how well any of these vision models approximated human sketch recognition behavior when presented with the same sketches. Because these models contain different latent representations of widely varying dimensionalities, we measured machine sketch-recognition behavior by extracting activation patterns from each model's final convolutional or attention block and training linear classification-based readouts on these activation patterns. That is, for each model we independently fit 1,854-way logistic regression classifiers using 5-fold stratified cross-validation to predict the "ground-truth" concept label associated with each sketch.
These predicted labels were aggregated across the 16 sketches of the same concept (e.g., lion) and from the same drawing-time condition (e.g., 4 seconds) to yield a model's response distribution for that type of sketch. The top-1 classification accuracy was determined by computing the relative frequency of the "ground-truth" concept label in this response distribution. We also computed the Shannon entropy of this response distribution to estimate the degree of semantic ambiguity exhibited by this type of sketch. Further, we derived a measure of the degree to which even the non-ground-truth labels generated by each model were semantically related to the ground-truth label, which we term the semantic neighbor preference score. This semantic neighbor preference score falls in the range [0, 1] and is highest when labels that are more semantically related to the ground-truth label appear more frequently than more semantically distant labels, is close to 0.5 when labels appear with uniform probability, and is minimized when labels that are more semantically distant appear most frequently.
To compare model classification outputs with human responses, we derived an analogous response distribution from the human labels obtained in the human sketch recognition experiment. That is, we aggregated all labels assigned by human participants to all 16 sketches of the same concept (e.g., lion) and from the same drawing-time condition (e.g., 4 seconds) to construct a response distribution for each type of sketch. We then computed the same three metrics above (i.e., top-1 classification accuracy, response entropy, semantic neighbor preference) using the human response distributions.
<sup>1</sup>Because these classification-based readouts were only trained on sketches of 128 object concepts subsetted from the THINGS dataset, the probabilities assigned to the remaining 1,726 labels were set to zero during training. As such, none of these models produced any of these other labels at test time, but humans in the sketch recognition experiment could select these other labels. This difference between how sketch recognition behavior was elicited from humans and models led us to focus on relative measures of performance when evaluating human-model alignment.
time	accuracymean	accuracysem	entropymean	entropysem	SNPmean	SNPsem
4 seconds	.031	.003	1.958	.011	.628	.008
8 seconds	.082	.004	1.814	.013	.701	.009
16 seconds	.139	.006	1.690	.014	.750	.010
32 seconds	.199	.007	1.555	.015	.787	.010
Table 3: Human sketch understanding under each draw duration constraint. Columns represent means and standard errors of the mean for top-1 accuracy, response entropy, and semantic neighbour preference (SNP).
2.4 Machine Sketch Production
To explore the potential of models that emulate human visual abstraction in generative tasks, we also include evaluations of CLIPasso, a recently developed sketch generation algorithm [91] capable of generating sketches that vary in sparsity.