SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: sketch understanding
visual abstraction
vision
cognitive science
TL;DR: We evaluate a suite of state-of-the-art vision models in their ability to understand human sketches at varying levels of abstraction

Abstract: Sketching is a powerful tool for creating abstract images that are sparse but meaningful. Sketch understanding poses fundamental challenges for general-purpose vision algorithms because it requires robustness to the sparsity of sketches relative to natural visual inputs and because it demands tolerance for semantic ambiguity, as sketches can reliably evoke multiple meanings. While current vision algorithms have achieved high performance on a variety of visual tasks, it remains unclear to what extent they understand sketches in a human-like way. Here we introduce $\texttt{SEVA}$, a new benchmark dataset containing approximately 90K human-generated sketches of 128 object concepts produced under different time constraints, and thus systematically varying in sparsity. We evaluated a suite of state-of-the-art vision algorithms on their ability to correctly identify the target concept depicted in these sketches and to generate responses that are strongly aligned with human response patterns on the same sketch recognition task. We found that vision algorithms that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but there remains a sizable gap between model and human response patterns. To explore the potential of models that emulate human visual abstraction in generative tasks, we conducted further evaluations of a recently developed sketch generation algorithm (Vinker et al., 2022) capable of generating sketches that vary in sparsity. We hope that public release of this dataset and evaluation protocol will catalyze progress towards algorithms with enhanced capacities for human-like visual abstraction.

SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction
Kushin Mukherjee"; Holly H Xuanchen Lu Yacl Vinker", Rio Aguina-Kang",
Ariel Shamir', and Judith E. Fan
University of Wisconsin- Madison
University of California, San Diego?
Tel-Aviv University
Reichman University
Stanford University
Abstract
Sketching a powerful tool for creating abstract images that are sparse but meaningful. Sketch understanding poses fundamental challenges for general- purpose vision algorithms because it requires robustness to the sparsity of sketches relative to natural visual inputs and because demands tolerance for semantic ambiguity, as sketches can reliably evoke multiple meanings. While current vision algorithms have achieved high performance on variety of visual tasks, it remains unclear to what extent they understand sketches in human- way. Here we introduce SEVA, a new benchmark dataset containing approximately 90K human- generated sketches of 128 object concepts produced under different time constraints, and thus systematically varying in sparsity. We evaluated a suite of state-of-the-art vision algorithms on their ability to correctly identify the target concept depicted in these sketches and generate responses that are strongly aligned with human response patterns on the same sketch recognition task. We found that vision algorithms that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but there remains sizable gap between model and human response patterns. To explore the potential models that emulate human visual abstraction in generative tasks, conducted further evaluations a recently developed sketch generation algorithm [91] capable of generating sketches that vary in sparsity. We hope that public release of this dataset and evaluation protocol will catalyze progress towards algorithms with enhanced capacities for human-like visual abstraction.
Introduction
Abstraction is key to how humans understand the external world. Abstraction enables distillation of individual sensory experiences into compact latent representations that support learning of new
concepts 80 45 66 30) and efficient communication about these concepts with others [23 35 R4
28 For example, while no two roses are identical, people can rapidly infer what properties make flower and not some other kind of flower from just few examples [99] (5),, especially when these examples are selected to support such strong inferences [32 77
1/- Equal contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

BT. Even without 1.1 Human Visual Abstraction as Key Target for AI
Visual abstraction enables humans to express what they know about the visual world by creating external representations that highlight the information they judge to be most relevant in any given context- for instance. pictures that highlight the visual features that are diagnostic of rose in botanical field guide [24] 23 192. Critically, there are many different ways to depict even the same object-from a detailed illustration to simple sketch. The Spanish artist Pablo Picasso famously demonstrated this point in The Bull (1945), series of 11 lithographs of bulls, each sparser than the last (Fig. D While some of the drawings this series look more realistic and others more stylized, all of these images remain evocative of bull (and perhaps other similar animals, such moose or buffalo) to most human viewers.
Drawing is one of the most accessible, enduring, and versatile techniques that humans in many cultures use to encode ideas and emotions in visual form
special training. humans can robustly produce and understand simple line drawings or sketches of familiar visual concepts [2 81
The ability to leverage drawings to understand and convey key aspects of the visual world emerges early in childhood [62 44] and improves throughout development with children's 's expanding conceptual knowledge [19 59 42)
Moreover, failures to produce and recognize drawings of objects associated with semantic dementia [10 73). suggesting links between a robust capacity for visual abstraction and the organization of semantic knowledge in the brain.
Figure 1: Pablo Picasso. The Bull, 1945.
In addition to drawings that represent objects and scenes, other abstract -aae visualizations (e.g., maps, diagrams, charts, graphs) serve important functions in many domains, including all branches of science and engineering (89 90 39
13 11]. Given the ubiquity and importance of such visualizations in modern life, developing computational models that achieve human-like understanding of freehand sketches is an important milestone. Such computational models of human visual abstraction stand to not only advance our understanding of human intelligence, but to also make Al systems more robust and general [40 63 27]. For example, prior work has found that incorporating principles based on the structure and function of the human visual system have led to vision models that are more robust (c.g., to adversarial attacks) 5655 25
1.2 Desiderata for Evaluating Alignment Between Human and Machine Visual Abstraction
The past several years have seen remarkable progress in the development of increasingly performant general-purposs vision algorithms 778 36 20, 70), with some of the most prominent algorithms also capable of emulating key aspects of how the primate brain encodes natural visual inputs
[102 48 110 47). Over the same period, artificial vision systems have also been steadily achieving higher performance on tasks involving abstract visual inputs, including sketch categorization 21
108 106), sketch segmentation (54] 104), sketch-based image/shape retrieval [22 77 105] 82
among others [88] 100, 60, [6]. Moreover, these models have been found to predict human behavior on sketch recognition tasks to some degree (44 23 However, these otherwise high-performing vision
algorithms struggle to simultaneously achieve robust understanding of visual inputs across multiple levels of abstraction [ 79 23). Moreover, one study found that current vision models trained on natural images still fall short of the representational capabilities of the inferotemporal cortex, key brain region supporting object categorization, in generalizing to new image distributions, including sketches 2 Nevertheless, more recently developed models trained on substantially larger and varied datasets show promise on both tasks involving images with different visual styles 107] (66 and tasks that go beyond recognition, including sketch generation [9] 69 97 and betch-guidedd image
generation [6] 109 96 93 56].
At present, it remains unclear to what degree any state-of-the-ar models achieve human-like understanding of line drawings that vary in their degree of abstraction, much less the full range of abstract images that humans regularly engage with. Gaining further clarity on this question requires meeting two key challenges: first, creating a dataset containing drawings of a wide variety of object concepts that also systematically vary in their degree of abstraction; and second, developing evaluation protocols that can be used to estimate the degree to which any model emulates human-like understanding of this suite of drawn images.
2

---

Dataset. Meeting the first challenge requires going beyond existing sketch datasets 21 74 43 18
22 [4 [OS]
67 III [57. While most of these datasets span reasonably wide range of visual concepts (i.e., ranging from 125 concepts in Sketchy to 345 in Quickdraw) and some of them contain fine-gran -ne-ggained information (e.g., stroke information and photo-sketch pairing), none of them systematically varied how detailed individual sketches could be, one of the most straightforward ways of inducing variation in semantic abstraction [8] 23 [03]. Our paper addresses the gap by providing sketches with controlled levels of detail, while encompassing the variety and granularity present in existing datasets (Table
82
26
95
75 53
Evaluation protocol. Meeting the second challenge requires going beyond simple accuracy- -eseracy-based model performance metries alone. Instead, it is critical measure detailed patterns of human behavior on the same sketch understanding tasks to evaluate how well any model emulates these behavioral patterns, following recent work in the computational neuroscience of vision 77 68
1.3 SEVA: A Novel Sketch Benchmark for Evaluating Visual Abstraction in Humans and Machines
In recognition of the above desiderata, here we introduce SEVA (Sketch-based Evaluations of Visual Abstraction), a new sketch dataset and benchmark for evaluating alignment between human and machine visual abstraction.
sketches of a wide variety of visual objects that also systematically vary in their level of detail, and thus the variety of meanings they evoke. Each sketch is associated with one of 2,048 object instances belonging to one of 128 object categories selected from the THINGS dataset (38. We achieved variation in sketch detail by imposing constraints on how much time humans (N65,5563 participants) had produce each sketch (i.e., 4s, 8s, 16s, 32s).
Dataset. Our dataset contains approximately 90K
Evaluation protocol. Leveraging these human-generated sketches, we systematically evaluated how well a diverse suite of 17 state-of-the-art vision models generate classification responses that align with those produced by humans (N=1,709 participants) tasked with identifying the most appropriate concept label for each sketch. Of these participants, 579 participants also participated in the sketch production study but were not shown any of their own sketches during this study. We evaluated human-model alignment using three different metrics: (1) top-1 classification accuracy, reflecting raw sketch recognition performance; (2) Shannon entropy of the response distribution, reflecting the degree of uncertainty about the target label; and (3) semantic neighbor preference, reflecting the degree to which models and humans generated off-target responses that were semantically related to
the target label.
Summary of key findings. We found that sparser human sketches produced under more severe time pressure (e.g.. 4 seconds) exhibited greater semantic ambiguity in other words, both humans and models assigned a greater variety of labels them than to the more detailed sketches that took more time to make (e.g., ......3. 32 seconds). Furthermore, we found that models that better predicted human sketch recognition performance also better approximated human uncertainty about sketch meaning, but none of the models came close to approximating human response patterns to numan-generae sketches at any level of detail. To explore the potential of models that emulate human visual abstraction in generative tasks, we conducted further evaluations of CLIPasso, recently developed sketch generation algorithm [9T] capable of generating sketches that vary in sparsity (measured by number of constituent strokes in a sketch). We discovered that the most detailed CLIPasso- aassoogeegeee sketches converged with human sketches of the same object concepts, as measured by the distribution of labels humans assigned to sketches made by both agent types; however, sparser CLIPasso sketches diverged from human sketches of the same object concepts, reflecting a gap between how CLIPasso and human participants attempted to preserve sketch meaning under more severe production constraints.
2 Methods
2.1
Human Sketch Production
core contribution of this work is a new dataset containing human- gennan-eeeeraed sketches of wide range of visual object concepts that also systematically span multiple levels of semantic abstraction. We created this dataset by crowdsoureing these sketches online, following prior work [24] 105] 74
23 34 43). Each sketch in the dataset was recorded as a bitmap image as well as a collection of stroke coordinates, thus preserving the precise cursor movements a participant enacted to create the sketch.
3

---

Human sketches
animals
clothing
food
seconds
seconds
16 seconds
seconds
objects
vehicles
seconds
seconds
16 seconds
seconds
Machine sketches
animals
clothing
food
strokes
117
strokes
16 strokes
objects
vehicles
4 strokes
V7
strokes
32 strokes
Figure 2: Humans and CLIPasso generated approximately 90K sketches under various production constraints.
Participants. 5,563 participants (2,870 male; Mage 36.7 years) were recruited from Prolific and compensated $15.50/hour for their participation. Data from 104 of these sessions were excluded from subsequent analyses due to technical issues (c.g.. images did not load). All participants provided informed consent in accordance with the UC San Diego IRB.
Object concepts. We included 128 concrete real-world object categories (c.g., "lion", "banjo", "car") sourced from the THINGS dataset. We used the THINGS dataset 33 because a well validated set of concrete, real-world visual object categories designed to support interoperability among large-scale studies in human visual cognition and cognitive neuroscience. For each of these 128 object concepts, we randomly sampled 16 object instances represented by color photographs, which served to visually ground the human sketch production task. As such, each sketch in our dataset is uniquely associated with one of these 2,048 object instances, and our final sample size was determined by our predefined goal of obtaining at least 10 human sketches of each of these instances.
Sketch production task. In each session, participants produced sketches of 16 different object categories, randomly sampled from the full set of 128 object categories. On each trial, they were cued with a color photograph (500px x 500px) of an object paired with its concept label. Each participant was randomly assigned to one of four conditions, defined by the maximum amount of time participants could take to produce their sketches: 4 seconds, 8 seconds, 16 seconds, or 32 seconds (Fig. - left). Such random assignment of participants to condition ensures that estimates of differences between conditions will not, in expectation, be biased by individual differences in sketching behavior. Participants drew on digital drawing canvas (500px x 500px) using whatever input device they already had available (c.g., mouse, stylus) and were able to undo their most recent

---

Dataset Name
Dataset Contents
Classes
Stroke Info?
Photo Cue?
Abstraction?
TU-Berlin 21
20K sketches
250
QuickDraw
[k]
SOM sketches
345
QuickDrave Extended
18
330K sketches, 204K photos
110
SPG 54
20K sketches w/ stroke grouping
25
v
SBSR
1.8K sketches, 1.8K 3D models
161
QMUL 105 82
1.3K sketches, 1.3K photos
Sketchy 74
75K sketches, 12K photos
125
SketchyCOCCO
26
14K sketches, 14K photos
17
SEVA
90K sketches, 2048 photos
128
Table 1:
Comparison between SEVA and prior sketch datasets.
stroke or completely clear their canvas if needed. They were encouraged to make their drawings as recognizable as they could the concept level and use the photograph only to remind them of what individual objects belonging to that category generally look like. A countdown timer indicated how many seconds they had left produce their drawing. Each trial ended either when time ran out or when the participant indicated that they wished to continue to the next trial, but participants were encouraged to use the full time available to produce as recognizable of a drawing as they could. At the beginning of the session, participants were explicitly instructed not to include any background context (e.g., grass grass in a drawing of "horse"), arrows, or text. Participants also completed one practice trial (that we did not include in analyses) at the beginning of the session to familiarize themselves with the drawing interface. Our final dataset contains 89,797 sketches after filtering out invalid sketches (e.g.. blank canvases).
2.2 Human Sketch Understanding
A key component of any evaluation of how well current vision algorithms emulate human visual abstraction is measurement of human behavior in tasks relying on visual abstraction. Here we focus on characterizing what meanings humans extract from the collected sketches, providing the basis for our subsequent empirical evaluation of how well any state-of-the-art vision model approximates human response patterns when presented with the same sketches
Participants. 1,709 participants (776 male; Mage = 39.2 years) were recruited from Prolific and compensated $15.50/hour for their participation. Data from 21 of these sessions were excluded from subsequent analyses due to technical issues. Our predefined criterion for stopping data collection was acquisition of at least recognition judgments for each sketch.
Sketch recognition task. In each session, participants provided labels for 64 sketches randomly sampled from fixed set of 8,192 sketches, approximately 10% of the full human sketch dataset. The specific set of 8,192 sketches included in this experiment was determined by randomly sampling one sketch cued by each object instance from each drawing-time condition (i.e., 16 instances/concept x 128 categories x drawing-time conditions = 8, 192). On each trial, participants were presented with single sketch (300px 300px) and text field where they could provide their best guess concerning the concept the sketch was intended to convey. As soon as they began typing, drop-down menu appeared with suggested word completions. This drop-down menu contained the entire set of 1,854 labels in the THINGS dataset and only responses that matched one of these 1,854 labels were accepted. Because many words have multiple meanings, the labels contained in this dropdown menu were also accompanied by disambiguating text (e.g., to distinguish mouse (animal) from mouse (computer)). If participants were unsure of which label best applied the sketch, they were encouraged to provide additional guesses (up to 5 per sketch). Collecting multiple labels on each drawing trial was important because it enabled us to more thoroughly sample the distribution of meanings that each sketch evoked for human participants (i.e., which labels came to mind and how often they did so). At the beginning of the session, participants completed a practice trial (that we did not include in analyses) to familiarize themselves with the labeling interface.
2.3 Machine Sketch Understanding
We propose a generic protocol for evaluating machine sketch understanding that can be applied to any vision algorithm using our sketch dataset. In this paper, we conduct evaluations of wide range of state-of-the-a vision models with the goal of demonstrating the feasibility of our protocol and guiding future model development.
5

---

Model Suite. Specifically, we evaluated 17 vision models spanning a wide range of architectures and training methods (Table all of which have been demonstrated achieve high performance on object recognition on standard datasets, such as ImageNet [17]. We also made sure to include variants of standard ConvNet and Transformer models that have gained traction within the field of
computational cognitive neuroscience for their potential to close the gap between biological and
artificial vision [46 25
65
Model
Architecture
Training Paradigm
Dataset
VGG-19 18
VGG-19
supervised
ImageNNtt
Inception-1 V3
Inception-V3
supervised
ImageNet
ResNet-50
36
ResNet-50
supervised
ImageNet
VTT-B 80
VIT-B
supervised
ImageNet
Swin-B B
58
Swin-B B
supervised
ImageNet
MLPMMixe-B 87
ML.Miixr-
supervised
ImageNet
CORnet-S
CORnet-S
supervised
ImageNet
Harmonization
25
VIT-B
supervised
InageNet + -umman Feature Importance 25
ECOSET
ResNet-50
supervised
ECOSET
SimCLR
14
ResNet-S0
self supersueee
ImageNet
MoCo-s3
15
VIT-B
self supervised supervidd
ImageNet
DINO
12
ViT-B
ImageNet
MAE 35
VIT-B
self supervised
ImageNet
CLIP
VIT-B
self- superviee
WeblmageTest 70
JPCL
40
AlexNet
self-supervised semi supervised
ImageNet
Noisy Student
98
EfficientNethhl h4
ImageNet + JFT
K3
SWSL
101
ResNet-50
semi-supervised
ImageNet 100M 86
IG--I--Tugeeed 101
Table 2: Model suite annotated by backbone architecture, training paradigm, and training dataset.
Evaluation Protocol. The goal of our evaluation protocol was to measure how well any of these vision models approximated human sketch recognition behavior when presented with the same sketches Because these models contain different latent representations of widely varying dimensionalities, we measured machine sketch-recognitton behavior by extracting activation patterns from each model's final convolutional or attention block and training linear classification based readouts on these activation patterns. That is, for each model we independently fit 1,854-way logistic regression classifiers using 5-fold stratified cross-validation to predict the "ground-truth" concept label associated with each sketch.
These predicted labels were aggregated across the 16 sketches of the same concept (e.g., lion) and from the same drawing-time condition (e.g., 4 seconds) to yield model's response distribution for that type of sketch. The top-1 classification accuracy was determined by computing the relative frequency of the "ground-truth" concept label in this response distribution. We also computed the Shannon entropy of this response distribution to estimate the degree of semantic ambiguity exhibited by this type of sketch. Further, we derived measure of the degree to which even the non-ground -ruth labels generated by each model were semantically related to the ground-truth label, which we term the semantic neighbor preference score. This semantic neighbor preference score falls in the range [0, 1] and is highest when labels that are more semantically related to the ground-truth label appear more frequently than more semantically distant labels, close to 0.5 when labels appear with uniform probability, and is minimized when labels that are more semantically distant appear most frequently.
To compare model classification outputs with human responses, we derived analogous response distribution from the human labels obtained in the human sketch recognition experiment. That is, we aggregated all labels assigned by human participants all 16 sketches of the same concept (e.g.. lion) and from the same drawing-tinn condition (e.g., 4 seconds) to construct response distribution for each type of sketch. We then computed the same three metrics above (i.c., top-1 classification accuracy, response entropy, semantic neighbor preference) using the human response distributions.
'Because these classification- based readouts were only trained on sketches of 128 object concepts subsetted from the THINGS dataset, the probabilities assigned the remaining 1,726 labels were training. such, none of these models produced any of these other labels at test time, but humans in the sketch recognition experiment could select these other labels. This difference between how sketch recognition behavior was elicited from humans and models led us to focus on relative measures of performance when evaluating human-man model alignment.
zero during

---

time
accuracy meen
accuracy sem
entropy meen
entropy sem
SNP mean
SNP scm
4 seconds
.031
.003
1.958
.011
.628
.008
8 seconds
.082
.004
1.814
.013
.701
.009
16 seconds
.139
.006
1.690
.014
.750
.010
32 seconds
.199
.007
1.555
.015
.787
.010
Table 3: Human sketch understanding under each draw duration constraint. Columns represent
means and standard errors of the mean for top-1 accuracy, response entropy, and semantic neighbour preference (SNP).
2.4 Machine Sketch Production
To explore the potential of models that emulate human visual abstraction in generative tasks, we also include evaluations of CLIPasso, recently developed sketch generation algorithm [91] capable of generating sketches that vary in sparsity.
Generating machine sketches. Specifically, we leveraged CLI to generate 8, 8,192 sketches conditioned on the same 2,048 object instances we used in the human sketch production experiment such that each sketch was constrained to consist of either 4, 8, 16, or 32 pen strokes (Fig. 2 right). CLIPasso generates sketches by optimizing the parameters of set of curves (i.e., start/end points; control points), each representing single pen stroke, to be similar target image. This optimization guided by a pretrained implementation of CLIP [70], a large model trained using contrastive learning vast quantities of text-image pairs. Similarity to the target image is defined based on the distance between CLIP's embedding of the target image and embedding of the sketch, where these embeddings reflect combinations of feature activations from multiple intermediate layers of CLIP.
Measuring human understanding of machine sketches. We evaluated human recognition performance on these CLIPasso CLIPasso-generated sketches by recruiting 1,481 participants (730 male; Mage 41.05 years) on Prolific to complete the same sketch recognition task described earlier. Data from 7 of these sessions were excluded from subsequent analyses due to technical issues.
3 Results
Humans produce sparser sketches under stronger time constraints. We first sought to validate the effect of manipulating the maximum time that human participants had draw on how detailed their sketches were. We estimated how detailed a sketch was by counting the number of strokes it contained (Fig. 3 and then fit mixed- effects linear regression model predicting the number of strokes a function of drawing- -imm condition (i.e., 4s, 8s, 16s, 32s), with random intercepts for object concept. We found that drawings produced under the 4s limit contained the fewest strokes produced under the 32s limit contained the greatest number of strokes (3 .29. SE 4.95 x 10-3 3, p < .001). These results confirm that restricting the amount of time human participants had to produce their sketches led to systematic differences how detailed their sketches were.
0.54
8 seconds seconds
0.10
seconds
density
0.08
0.08
0.02
0.00
6
10
15
20
25
30
35
40
number of strokeess
Figure 3: Distributions of number of strokes for each drawing time condition in the human sketch production task. Vertical lines indicate means.
average, whereas those
Sparser sketches are more semantically ambiguous for models and humans. Having verified that we had successfully manipulated the level of detail in human sketches, we next sought to evaluate how well current vision models extract semantic information from them at each level of detail. Our general approach was to fit mixed-effect linear regression models to estimate the effect of sparsity on key metrics: (1) top-1 1 classification accuracy, (2) entropy of response distributions, and (3) semantic neighbor preference. Regression models included random intercepts and slopes for object concepts. Figure 4 shows the performance of each vision model with respect to these metrics for sketches produced under different time constraints.
found that models generally achieved higher top-1 classification accuracy for more detailed sketches than sparser ones (3 9.57 10
20.25, p .001).

---

top-1 dlssiiication accuracy mosm
We further found the entropy of the models' response distribution was lower for detailed sketches than for sparser sketches (8 = 033 t -30.19, p .001), suggesting greater uncertainty about the best label to apply to sparser sketches. Even when sketches were more ambiguous, however, models generated labels that were semantically related to the ground-truth label, as measured by our semantic neighbor preference score, with more detailed sketches eliciting greater proportion of semantically related labels (3 2.5 x 10
0.8
CLIP
0.6
RAA PcOReT SINSL
0.4
[PCL. footoonicion
Inception-V3
VIT-B
0.2
Mt. MMMMice
10
20
25
drawing time (6)
t 8.43, < .001). These patterns were mirrored in human sketch recognition behavior, with more detailed sketches being associated with higher top-1 classification performance (3 = 6.08 x 10-2, t 9.57. p < .001), a tighter distribution of responses (lower entropy) (3 = -.14. f - 12.299, p < .001), and greater semantic neighbor preference (3 = 5.41 10-2. t = 20.24, < .001).
disiitutton artroyy B
SInCLR
Hamonization
IPCL
VID10-13
DIND RealNet-50
Coljout NAE
Student
CLDF
Inception-V3
Different models display distinct patterns of sketch recognition behavior. Although all vision models were sensitive to the effect of our drawing- time time manipulation, we found that there were reliable differences in classification accuracy between models (x2(16) Moreover, some models generated a greater diversity of responses than others, as measured by the entropy of their response distribution (Fig. [B. (16) = 89698, p < .001).
10
20 25
drawing lime (6)
semertic neightor gelfeercc
3455.3, p < .001).
0.9
CUP
0.8
Human
Baann MAE
DIKO Noley
0.7
Ecotes
VIG0-11 Inception-V3 MLPMioor-B
Finally, models varied in the degree to which they generated non-ground-truth labels that were semantically related to the ground-truth label (x6(16) = 318.46, p < .001). Taken together, these results indicate that these models, all high- performing, display systematic differences in how they extract semantic information from sketches
0.5
chance
Figure 4: Effect of drawing time constraints on sketch understanding in different vision models.
large gap remains between human and model sketch understanding. While both humans and models are affected by the amount detail in sketches, it is not yet clear to what degree their response patterns are well aligned. We evaluated human-model alignment scores using the same three metrics (i.e., top-1 classification accuracy, entropy, semantic neighbor preference) by estimating the degree to which model performance on different types of sketches (c.g.. lions drawn in 4 seconds or less) covaried systematically with human performance on the same types of sketches. For example, model is considered well aligned with humans with respect to recognition performance it achieves high top-1 classification accuracy on the same types of sketches that humans succeed in classifying and if it achieves low accuracy the types of sketches that humans fail to classify. Similarly, a model is considered well aligned with humans with respect to semantic ambiguity if produces response distribution with high entropy for the same types of sketches that humans are also highly uncertain about and if it produces a low-entropy response distribution for the types of sketches that humans systematically agree on (regardless of whether this agreement is concentrated on the correct label). We found that models generaly displayed some degree of alignment to humans on top- classification accuracy (3- = 7.70 x 10 2, 6.77, p .001). response entropy (8 1.17 10 t = 28. 28.52, p < .001). and semantic neighbor preference (3 = 6.66 x 10 10-2. 2.
6.21, p < .001).
Moreover, we found that different vision models aligned with humans varying degrees (top-1 classification accuracy: (16) 134.81, p < .001; response entropy: x²61) 725.78, p .001; semantic neighbor preference: x² x161) 5.05. p .99). Nevertheless, a sizable gap remains between the most aligned models and top-1 classification accuracy:
-uman consistency baseline for all metrics (Fig
952. 19, p < .001: response entropy: = 184.21, p < .00I: semantic neighbor preference: L = 389.56, p .001). Finally, we observe that while examining classification accuracy and entropy yield similar rankings over which models are best aligned to humans, these metrics appear to capture non-redundant sources of information about human and model sketch understanding 5D).
8

---

top-1 classification accuracy
response entropy
human-model alignment (R)
human-model alignment (R²
0.6
0.6
0.4
0.4
0.2
0.2
Homen
Reatier boreei
DNO
Skeni
ster PMMbor
0.0
Homen Holay Stoden
Hoetter Swri Ioo
Holey
human-mode alignment (R)) o
semantic neighbor preference
D
1.0
accuracy
0.8
SNP entropy
0.75
0.2
Holey Humen
0.0
Inorption Shm
DIOO
RaMMet 89
OfF
MLPRbbe dree
Hamm ECOIEE
0.19
-0.01
Co
accuracy entropy
SNP
Figure 5: Human-mode model alignment on (A) top-1 classification accuracy, (B) response entropy, and (C) semantic neighbour preference. Leftmost red bars in each plot correspond baseline human-human consistency each metric. Error bars indicate bootstrapped 95% confidence intervals. (D) Spearman p correlations between the rank-ordering of vision models with respect to their alignment human performance on each metric.
A CLIP-based sketch generation algorithm emulates human sketches under conditions.
While sketch understanding a critical aspect of visual abstraction, the ability to produce sketches spanning different levels of abstraction is no less important. Although there remains gap between human and model sketch understanding, CLIIP-based vision model 770 was among the most performant and best aligned to human sketch understanding. Insofar as the major bottleneck to being able to generate more human-like sketches is achieving more human- understanding sketches and other images [24), generative model leveraging CLIP's latent representation may be promising approach. Consistent with this possibility, we found that CLIPasso generated sketches of concepts each abstraction level were about recognizable as the human human-generated sketches of those same concepts at the same abstraction level (Fig. 6A; adjusted R² .64). Moreover, we found that the more detailed CLIPasso sketches were especially human-lii like in that they evoked similar set of meanings to their human-generated counterparts.
To measure the degree to which human and CLIPasso sketches converged with respect to the object labels that they elicited from human viewers, we computed the Sonsen-Shannor distance (JSD) between the label distributions of human and CLIPasso sketches for each concept at each of the levels of abstraction. We found that humans and CLIPasso sketches were least divergent in terms of their perceived -6.86,
Fig. B. we show the average label divergence at each level of abstraction.
meaning when they were depicted in greater detail or were less abstract (3 = -0.69, p .001). At lower levels of detail and visual fidelity, human and CLIPasso sketches elicited more diverging responses. Thus, while CLIPasso more closely approximates human sketch production behavior at greater levels of detail, there remains large gap between how CLIPassO and human participants attempted to preserve sketch meaning under more severe production constraints. Taken together, while CLIPasso marks significant progress towards human-like sketch production its ability to produce highly abstract sketches human-like manner remains limited.

---

4 Conclusion
Recent advances in machine vision have enabled new opportunities
A.
.25
to understand the computational mechanisms that underlie human-like visual abstraction-how humans create and interpret a wide variety of images (from detailed illustrations to schematic diagrams) to convey what
top-1 dessificaioo accuracy (185
numan-CLIPasse CLPasso divergence
20
human
-15
CLiPasso
25
20
-10
15
they perceive and know about the world. Here we introduce SEVA, a new
10
.05
dataset and benchmark containing approximately 90K human and machine generated sketches spanning multiple levels of abstraction, to evaluate progress towards alignment between human and machine visual abstraction. Critically, our model evaluation protocol is focused not just on performance but on how well these models approximate human-like sketch production and understanding. Initial benchmarking of a set of 17 vision models on SEVA following this protocol revealed that even the most performant and well-aligned models deviate from human behavior in systematic ways. We also find that current generative models of sketching are able to sketch in human-like ways, but only in limited settings.
0
loweel
highess skatch production budget
lowest
hig
skotch production budget
Figure 6: (A) Human top-1 recognition performance CLIPasso and human sketches at different production budgets (draw duration for human sketches and number of strokes for CLIPasso sketches). (B) Divergence in human-
responses to the same type of human and CLIPasso sketches as a function of sketch production budget.
We hope that SEVA will accelerate progress towards unified computational theories that explain how humans capable of generating and understanding such wide variety of abstract visual representations. Specifically, our dataset could be used to adjudicate between vision models that are high-performing tasks involving natural images [64| [29] by characterizing their alignment with how humans understand images generated in very different manner, including freehand sketches. In addition, SEVA is distinctive among existing human sketch datasets in that it contains detailed measurements of how humans make moment-to tomeentttt-mment decisions about where to place each stroke, when aiming to produce a sketch of visible object, under different constraints. As such, expect SEVA to be a key resource for advancing the state-of-the-art in sketch generation and thus lead computational models of generalized visual abstraction.
669 94 91
33 72
5 Limitations
We note several limitations of the current study that would be important to address in future work. First, our sample of participants was limited to English-speaking individuals based in the United States. As such, the current study cannot speak to potential differences in aketch-production behavior across geographical and cultural contexts. However, future work that recruits from broader cross-section of individuals, including those located in the "Majority World," will be vital for understanding those potential sources of variation in human sketch production and comprehension. Second, we recruited participants via Prolific, a widely used crowdsourcing platform human behavioral research, without regard to any previous artistic training they had received. As such, our study generally reflects sketch production behavior among individuals without substantial expertise in the visual arts. Third, following prior work (24 23
43 [9], the human sketches in our dataset were obtained using a web-based digital drawing interface, where most participants used a mouse or trackpad to produce their sketches (89. (9.9 and some participants used a touchscreen (6.94%) or stylus (1.67%). Investigation of the impact of input device on sketch production would be a fruitful avenue for follow-up work. Fourth, because we did not obtain non-digital sketches (e.g., drawn on paper with a pen/pencil), our data cannot speak to differences between digital and non-digital sketches. Fifth, our study of machine sketch production included just one model, CLIPasso, limiting the conclusions that can be drawn about sketch generation algorithms in general. Future work that evaluates a broader suite of algorithms would thus be valuable.
34
105
10

---

Acknowledgements
We thank members of the Cognitive Tools Lab at Stanford and the Stanford Neuro AI Lab for helpful feedback, discussions, and support. This work was supported by NSF CAREER Award #2047191 to J.E.F. J.E.F is additionally supported by an ONR Science Autonomy award.
All code and materials available at:
https://github. http:://thhh..coooogollaaaaaa/ hatp://laa.aataaaaaaaaaaattotona haw.rractttosssbbmmcmrrriinnnnn
Broader Impacts
Visual abstraction as ubiquitous as central to our understanding of the visual world. Humans recognize depictions of objects at varying levels of fidelity to their real-world counterparts. This ability to recognize and render concepts abstract forms is crucial for visual knowledge transmission in the form of graphs, diagrams, and symbols. While computer vision has steadily made progress towards algorithms that can recognize objects in scenes, distinguish among different instances of an object, even answer questions about those objects in natural language, it remains unclear if these systems have the capability to understand visual concepts at multiple levels of abstraction in human-like manner. SEVA marks a step towards evaluating current vision algorithms on their sensitivity to depictions of wide variety of common object concepts at varying levels of abstraction. Critically, also provide results human performance of varying sparsity, setting a clear benchmark for future vision algorithms. Our initial study of state-of-the-as vision algorithms shows that even the most nigh-performing of models still fall short of human consistency baselines. With these vision algorithms increasingly being deployed in many aspects of everyday life, crucial to measure how human-like they are in this most natural of human abilities. We hope that SEVA will be used by the community to help close the gaps in current vision algorithms' alignment human behavior that we have identified in this paper.
sketch understanding for sketches
References
[1] Maxime Aubert, Adam Brumm, Muhammad Ramli, Thomas Sutikna, E Wahyu Saptomo, Budianto Hakim, Michael J Morwood, Gerrit van den Bergh, Leslie Kinsley, and Anthony Dosseto. Pleistocene cave art from sulawesi, indonesia. Nature, 514(7521):223-22) 2014.
[2] Ayu Marliawaty I Gusti Bagus, Tiago Marques, Sachi Sanghavi, James J DiCarlo, and Martin Schrimpf. Primate inferotemporal cortex neurons generalize better to novel image distributions than analogous deep neural networks units. In SVRHM 2022 Workshop@ @ NewrIPS.
[3] Nicholas Baker and Philip J Kellman. Abstract shape representation human visual perception. Journal of Experimental Psychology: General, 147(9): 1295, 2018.
[4] Pedro Ballester and Ricardo Araujo. On the performance of googlenet and alexnet applied to sketches. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[5] MD Barrett and PH Light. Symbolism and intellectual realism in children's 's drawings. British Journal of Educational Psychology, 198-222,, 1976.
[6] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Capturing human categorization of natural images by combining deep networks and cognitive models. Nature communications, 11(1):5418, 2020.
[7] Daniel M Bear, Elias Wang. Damian Mrowca, Felix Binder, Hsiao- Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.0820 2021.
[8] Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. Style and abstraction in portrait sketching. ACM Transactions Graphics (TOG), 32-4):1-12, 2013.
11

---

[9] Ayan Kumar Bhunia, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Sketch less for more: On-the-fly fly one-grained sketch-base image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 978899788. 2020.
[10] Sasha Bozeat, Matthew A Lambon Raiph, Kim S Graham, Karalyn Patterson, Helen Wilkin, Josephin Rowland, Timothy T Rogers, and John Hodges. duck with four legs: Investigating the structure of conceptual knowledge using picture drawing in semantic dementia. Cognitive neuropsychology. 20(1):27-47, 2003.
[11] Mackinlay Card. Readings in information visualization: using vision to think. Morgan Kaufmann, 1999.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[13] Min Chen, Helwig Hauser, Penny Rheingans, and Gerik Scheuermann. Foundations of data visualization. Springer, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning. pages 1697-1607, PMLR, 2020.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[16] Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. What can human sketches do for object detection? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15083-15094, 2023.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[18] Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. Doodle to search: Practical sero sketch-based image retrieval. In Proceedings of the IEEE/CVE conference computer vision and pallern recognition, pages 2179-2188, 2019.
[19] Moira R Dillon. Rooms without walls: Young children draw objects but not layouts. Journal of Experimental Psychology: General, 150(6): 2021.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[21] Mathias Eitz, James Hays, and Mare Alexa. How do humans sketch objects? ACM Transactions graphics (TOG), (4): 2012.
[22] Mathias Eitz, Ronald Richter, Tamy Boubekeur, Kristian Hildebrand, and Mare Alexa. Sketch- based shape retrieval. ACM Transactions on gruphics (TOG), 31 2012.
[23] Judith E Fan, Robert Hawkins, Mike Wu, and Noah Goodman. Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. Computational Bruin & Behavior, 3:16-101, 2020.
[24] Judith E Fan, Daniel LK Yamins, and Nicholas B Turk-Browne. Common object representations for visual production and recognition. Cognitive science, 2288)226702 2018.
12

---

[25] Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. Sketchycoco: Image generation from freehand scene sketches. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5 5174-5183, 2020.
[27] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schüitt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
[28] Dedre Gentner and Christian Hoyos. Analogy and abstraction. Topics cognitive science, 9(3):672-693, 2017.
[29] Tal Golan, Prashant C Raju, and Nikolaus Kriegeskorte. Controversial stimuli: Pitting neural networks against each other as models of human cognition. Proceedings of the National Academy of Sciences, 117(47):29330-293)
2020.
[30] Robert I. Goldstone, Alan Kersten, and Paulo F Carvalho. Concepts and categorization. 2013.
[31]
Ernst Hans Gombrich. The story of art, volume 12. Phaidon London, 1995.
[32] Hyowon Gweon, Joshua B Tenenbaum, and Laura E Schulz. Infants consider both the sample and the sampling process in inductive generalization. Proceedings of the National Academy of Sciences, 107(00):906- 00016-97111... 2010.
[33] David Ha and Douglas Eck. A neural representation of sketch drawings. arXiv preprint wrXiv:1704.03477 2017.
[34] Robert D Hawkins, Megumi Sano, Noah D Goodman, and Judith E Fan. Visual resemblance and interaction history jointly constrain pictorial meaning. Nature Communications, 14(1) (4(1))2199, 2023.
[35] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[37] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkann and Chris I Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. Elife, 12:e82580, 2023.
[38] Martin N Hebart, Adam H Dickter, Alexis Kidder, Wan Y Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris Baker. Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PloS one, 14(10):e0233792, 2019.
[39] Mary Hegarty. The cognitive science of visual-spatial displays: Implications for design. Topics in cognitive science, 3(3):446-474, 2011.
[40] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: / /903. /2261, 2019.
[41] D. L. Hoffmann, C. D. Standish. M. García-Diez. P. B. Pettitt, J. A. Milton, J. Zilhão, J. J. P. Cantalejo-Duarte, H. Collado, R. de Balbín, M. Lorblanchet, J. Ramos- Muñoz, G.CCh. Weniger, and A. W. G. Pike. U-th dating of carbonate crusts reveals neandertal origin of iberian cave art. Science, 359(6378):912 915, 2018
[42] Holly Huey and Bria Long. Developmental changes in the semantic part structure of drawn objects. In Proceedings of the 44th Annual Meeting of the Cognitive Science Society, 2022.
13

---

[43] J Jongejan, H Rowley, T Kawashima, J Kim, and N Fox-Gieg. The quick, draw! dataset, 2017.
[44] Annette Karmiloff-Smitt Constraints representational change: Evidence from children's drawing. Cognition, 34(1):57-83, 1990.
[45] Charles Kemp, Andrew Perfors, and Joshua B Tenenbaum. Learning overhypotheses with hierarchical bayesian models. Developmental science, 10(3):07-321, 2007.
[46] Talia Konkle and George A Alvarez. Beyond category-supervision: instance-level contrastive learning models predict human visual system responses objects. bioRxiv, pages 2021-05, 2021.
[47] Talia Konkle and George A Alvarez. A self-supervised domain-genente learning framework for human ventral stream representation. Nature Communications, 13(1):1-12, 2022.
[48] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modelling biological vision and brain information processing. biorxiv, page 029876, 2015.
[49] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019.
[50] Brenden M Lake and Steven Piantadosi. People infer recursive visual concepts from just a few examples. Computational Brain & Behavior, 3:54-65, 2020.
[51] Bo Li, Yijuan Lu, Afzal Godil, Tobias Schreck, Benjamin Bustos, Alfredo Ferreira, Takahiko Furuya, Manuel J Fonseca, Henry Johan, Takahiro Matsuda, et al. A comparison of methods for sketch-basee 3d shape retrieval. Computer Vision and Image Understanding. 119:57-80, 2014.
[52] Bo Li, Yijuan Lu, Chunyuan Li, Afzal Godil, Tobias Schreck, Masaki Aono, Martin Burtscher, Hongbo Fu, Takahiko Furuya, Henry Johan, et al. Shrec' 14 track: Extended large scale sketch-based 3d shape retrieval. In Eurographics workshop 3D) object retrieval, volume 2014, pages 121-130, 2014.
[53] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[54] Ke Li, Kaiyue Pang. Jifei Song, Yi-Zhe Song. Tao Xiang, Timothy M Hospedales, and Honggang Zhang. Universal sketch perceptual grouping. In Proceedings of the european conference on computer vision (ECCV), pages 582-597, 2018.
[55] Xiao Li, Ziqi Wang, Bo Zhang, Fuchun Sun, and Xiaolin Hu. Recognizing object by components with human prior knowledge enhances adversarial robustness of deep neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[56] Gao Lin, Liu Feng-Lin, Chen Shu- Shu-Yu, Jiang Kaiwen, Li Chunpeng, Yukun Lai, and Fu Hongbo. Skotchfacenerf: Sketch-based facial generation and editing in neural radiance fields. ACM Transactions Graphics, 2023.
[57] Fang Liu, Changqing Zou, Xiaoming Deng, Ran Zuo, Yu-Kun Lai, Cuixia Ma. Yong- Liu, and Hongan Wang. Scenesketcher: Fine-grained image retrieval with scene sketches. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pages 718-734. Springer, 2020.
[58] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[59] Bria Long, Judith Fan, Zixian Chai, and Michael Frank. Parallel developmental changes in children's 's drawing and recognition of visual concepts. 2021.
14

---

[60] Xuanchen Lu, Xiaolong Wang, and Judith E Fan. Learning dense correspondences between photos and sketches. 2023.
[61] Yongyi Lu, Shangzhe Wu, Yu-Wing Tai, and Chi-Keung Tang. Image generation from sketch constraint using contextual gan. In Proceedings of the European conference computer vision (ECCV), pages 205-220, 2018.
[62] Georges-Henri Luquet. Le dessin aulantin.(bibliothhee de psychologie de I" enfant et de pédagogie.). 1927.
[63] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arviv: (tll::70600833 2017.
[64] Nesma Mahmoud, Hanna Antson, Jaesik Choi, Osamu Shimmi, and Kallol Roy. Stress and adaptation: Applying anna karenina principle in deep learning for image classification. arXiv preprint arXiv:2302.1 2023.
[65] Johannes Mehrer, Courtney Spoerer, Emer C Jones, Nikolaus Kriegeskorte, and Tim C Kietzmann. An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences, 201))))111141111.. 2021.
[66] Gregory Murphy. The big book of concepts. MIT press, 2004.
[67] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[68] Joshua Peterson, Ruairidh M Battleday, Thomas Griffiths, and Olga Russakovsky. Human uncertainty makes classification Conference on Computer Vision, pages 9617-9626, 2019.
robust. In Proceedings of the IEEF/CVF International
[69] Shuwen Qiu, Sirui Xie, Lifeng Fan, Tao Gao, Jungseock Joo, Song-Chun Zha, and Yixin Zhu. Emergent graphical conventions in visual communication game. Advances in Neural Information Processing Systems, 35:13119-1111 19-13111,11111111111111111111131 2022.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-8763. PMIR, 2021.
[71] Rishi Rajalingham, Elias Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior humans, monkeys, and state-of-the-art deep artificial neural networks. Journal of Neuroscience, 38(33):7255-7266 2018.
[72] Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer- representation for sketched structure. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14153-14166 2020.
[73] Timothy T Rogers and Karalyn Patterson. Object categorization: reversals and explanations of the basic-level advantage. Journal of Experimental Psychology: General, 136(3):451, 2007.
[74] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 35 2016.
[75] Ravi Kiran Sarvadevabhatla, Sudharshan Suresh, and R Venkatesh Babu. Object category understanding via eye fixations on freehand sketches. IEEE Transactions on Image Processing, 26(5):2508-2511 2017.
[76] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 400m: Open dataset of fil-filteeed 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
15

---

[77] Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55-89, 2014.
[78] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprimt arXiv: 1409. 1556, 2014.
[79] Johannes JD Singer, Katja Seeliger, Tim C Kietzmann, and Martin N Hebart. From photos to sketches-how humans and deep neural networks process objects across different levels of visual abstraction. Journal of vision, 22(2):4-4, 2022.
[80] Linda B Smith, Susan S Jones, Barbara Landau, Lisa Gershkoff-Stovee and Larissa Samuelson. Object name learning provides on-the-job training for attention. Psychological science, 13(1):13-19, 2002.
[81] Joan G Snodgrass and Mary Vanderwart. A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. Journal of experimental psychology: Human learning and memory, 6(2):174, 1980.
[82] Jifei Song, Qian Yu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Deep spatial- semantic attention for fine- fine-grained sketch-basee image retrieval. In Proceedings of the IEEE international conference on computer vision, pages 5551-5560, 2017.
[83] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness data in deep learning era. In Proceedings of the IEEE international conference computer vision, pages 843-852, 2017.
[84] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[85] Michael Henry Tessler and Noah D Goodman. The language of generalization. Psychological review, 126(3):395, 2019.
[86] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):4473, 2016.
[87] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp- mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261-24272, 2021.
[88] Aditay Tripathi, Rajath R Dani, Anand Mishra, and Anirban Chakraborty. Eketch-guided object localization in natural images. Computer Vision-ECCe 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part VI 16, pages 532-547. Springer, 2020.
[89] Edward R. Tufte. The Visual Display of Quantitative Information. Graphies Press, USA, 1986.
[90] Barbara Tversky. Some ways that maps and diagrams communicate. In Spatial cognition II: Integrating abstract theories, empirical studies, formal methods, and practical applications, pages 72-79. Springer, 2000.
[91] Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-awann object sketching. arXiv preprint arXiv:2202.05822 2022.
[92] Ivan Viola and Tobias lsenberg. Pondering the concept of abstraction in (illustrative) visualization. IEEE transactions on visualization and computer graphics, 24(9):253-2558 2017.
[93] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
16

---

[94] Alexander Wang, Mengye Ren, and Richard Zemel. Sketchembednet: Learning novel concepts by imitating drawings. International Conference on Machine Learning, pages 10870-10881. PMLR, 2021.
[95] Fei Wang. Shujin Lin, Hefeng Wu, Hanhui Li, Ruomei Wang, Xiaonan Luo, and Xiangjian He. Spfusionnet: Sketch segmentation using multi-modal data fusion. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1654-1659. IEEE, 2019.
[96] Jiayun Wang, Jierui Lin, Qian Yu, Runtao Liu, Yubei Chen, and Stella X Yu. 3d shape reconstruction from free-hand sketches. In European Conference Computer Vision, pages 184-202 Springer, 2022.
[97] Qiang Wang, Haoge Deng, Yonggang Qi, Da Li, and Yi-Zhe Song. Sketchknitter: Vectorized sketch generation with diffusion models. In The Eleventh International Conference Learning Representations, 2022.
[98] Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoe Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10687-10698, 2020.
[99] Fei Xu and Joshua Tenenbaum. Word learning as bayesian inference. Psychological review,
1 14((2)2245, 2007.
[100] Peng Xu, Timothy M Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. Deep learning for free-hand sketch: A survey. IEEE transactions on pattern analysis and machine intelligence, 45(1))285-311 2022.
[101] Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546 2019.
[102] Daniel I.K Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):6119-6222 2014.
[103] Justin Yang and Judith E Fan. Visual communication of object concepts at different levels of abstraction. arXiv preprint Sikiv:2106.0277: 2021.
[104] Lumin Yang, Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, and Youyi Zheng. Sketchgnn: Semantic sketch segmentation with graph neural networks. ACM Transactions on Graphics (TOG), 40(3): 2021.
[105] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen-Change Loy. Sketch me that shoe. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 799-807, 2016.
[106] Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-net: A deep neural network that beats humans. International journal of computer vision, 122:411 1-525, 2017.
[107] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-1213,, 2022.
[108] Hua Zhang, Si Liu, Changqing Zhang, Wenqi Ren, Rui Wang, and Xiaochun Cao. Sketchnet: Sketch classification with web images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1105-1113, 2016.
[109] Song-Hat Hai Zhang, Yuan-Chen Guo, and Qing Qing-Wen Gu. Sketch2model: View-aware 3d modeling from single free-hand sketches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6012-6021. 2021.
[110] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3)c201449611 18, 2021.
17

---

[111] Changqing Zou, Qian Yu, Ruofei Du Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang. Sketchyscene: annhttann scene sketches. In Proceedings of the european conference on computer vision (ECCV), pages 421-436 2018.
18


Reviews:
[{"review_id": "SnmhTlO40a", "reviewer": "Reviewer_Jrh3", "summary": "", "questions": "", "limitations": "NA. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset is of large-scale. \n- Sketch understanding especially for different-level of abstractions is also a very important topic.  \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "o70Dye2hvU", "reviewer": "Reviewer_d6px", "summary": "", "questions": "", "limitations": "I believe the Clipasso is trained to mimic sketches of professional artist, where the crowdsourcing sketches are of amateare artist, I wonder if the drawing process of professionals is different and will resemble clipasso more closely \n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset collected is very large and will be very useful for followup works\nSketch recognition benchmark is comprehensive and contains 17 different recognition methods\n\nThe evaluation and comparison of both recognition and generation abilities are thorough and contain multiple metrics. Author did a good job in highlighting where vision models and humans agree and where a clear difference is presence. I especially appreciate the discussion and graphs presented in Fig. 7.\n\nObservations regarding the similarity and the gap between human sketch generation and recognition are interesting and pave the motivation for future algorithms that will better follow human perception.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "X7hwYXh45e", "reviewer": "Reviewer_p23D", "summary": "", "questions": "", "limitations": "The authors did not adequately address the limitations and potential negative societal impact of their work, though they mentioned broader impacts in a more positive way.\nPossible limitations were mentioned a bit in the above sections, e.g. the way the human-generated sketches were collected.\nThe potential societal impact could be the bias built in the data collection part and the corresponding influence on the findings and algorithm design.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ This paper contributes a large-scale human-generated sketch dataset with a large number of participants.\n\n+ The key findings from the benchmark study are very interesting, such as the \"none of the models came close to approximating human response patterns\", \"a gap between how sketch generation algorithm CLIPasso [62] and human participants attempted to preserve sketch meaning\", etc. These findings could be useful and inspire corresponding research.\n\n+ The findings concluded in the Results section are interesting but intuitively make sense. These might be useful for following-up sketch-related research.\n\n+ The paper is well-written and clearly motivated. The reviewer enjoyed reading it, and believes it will inspire the audience of the conference.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "54iJOKjgG7", "reviewer": "Reviewer_K3MZ", "summary": "", "questions": "", "limitations": "**[Drawing tool might impact the human sketch abstraction]** Participants may use any device for drawing sketches with different abstraction levels. Sometimes it might be hard to draw a sketch if a participant only uses a computer mouse. Then, the drawn sketch is not representing human visual abstraction level but it might also get influenced by the drawing device. For example, if all participants were provided with an electric pen and a tablet, the drawn sketches might look completely different than ones which are only generated by a computer mouse.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "I appreciate the authors efforts for this paper and found multiple strengths as follow:\n\n1. **[Data gathering]** The data gathering process was clearly explained.\n2. **[Motivation & insight]** The motivation behind understanding how humans perceive different levels of abstraction is a valuable research direction. This research direction might help to better understand human abstraction and perception. It might also help to develop algorithms that might encourage machines to have similar abstraction levels as humans or human-like deep learning models. \n3. **[Providing multiple labels for ambiguous sketches]** It was a smart direction to provide multiple labels for ambiguous sketches because it can help to design better classifiers in future for sketch recognition, which can model these ambiguities.  \n4. **[Various baselines]** Authors incorporated a considerable number of baselines (17 models), which is great in understanding each model's capabilities to deal with various abstraction levels and ambiguities.  \n5. **[Valuable data]** It is hard to gather human drawn sketches in a large scale because it requires lots of effort and participants. Therefore, I appreciate the effort used to gather this data.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "A9QJJc7HYr", "reviewer": "Reviewer_81P5", "summary": "", "questions": "", "limitations": "See the limitations and suggestions below.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is generally well-written.\n2. The problem of measuring sketch alignment and recognition between ai and humans is an important and interesting problem.\n3. Multiple aspects of alignments are covered and assessed rigorously. This includes introduction to multiple evaluation metrics, and benchmarking 17 models.\n4. The dataset design is well-structured and well-organized (see some suggestions and questions below).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper introduces a new dataset of human sketches with varying levels of abstraction and proposes a benchmark study comparing human and machine visual abstraction in sketch recognition. The dataset is of large-scale and contains valuable information for follow-up research. The evaluation is comprehensive, with multiple recognition methods and metrics used. The findings are interesting and relevant to better understanding human perception and guiding future algorithms. However, there are opportunities for improvement, including providing more visualizations of the dataset and addressing concerns raised by reviewers regarding the dataset collection process, the need for human-like models, and potential overlaps in participant recruitment. The limitations and ethical considerations also need to be addressed, and additional experiments and comparisons, such as with recurrent models, could be considered.
Decision: Accept (Poster)