SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generating machine sketches. Specifically, we leveraged CLIPasso to generate 8,192 sketches conditioned on the same 2,048 object instances we used in the human sketch production experiment such that each sketch was constrained to consist of either 4, 8, 16, or 32 pen strokes (Fig. 2, right). CLIPasso generates sketches by optimizing the parameters of a set of curves (i.e., start/end points; control points), each representing a single pen stroke, to be similar to target image. This optimization is guided by a pretrained implementation of CLIP [70], a large model trained using contrastive learning on vast quantities of text-image pairs. Similarity to the target image is defined based on the distance between CLIP's embedding of the target image and its embedding of the sketch, where these embeddings reflect combinations of feature activations from multiple intermediate layers of CLIP.
Measuring human understanding of machine sketches. We evaluated human recognition performance on these CLIPasso-generated sketches by recruiting 1,481 participants (730 male;  $M_{age} = 41.05$  years) on Prolific to complete the same sketch recognition task described earlier. Data from 7 of these sessions were excluded from subsequent analyses due to technical issues.
Humans produce sparser sketches under stronger time constraints. We first sought to validate the effect of manipulating the maximum time that human participants had to draw on how detailed their sketches were. We estimated how detailed a sketch was by counting the number of strokes it contained (Fig. 3) and then fit a mixed-effects linear regression model predicting the number of strokes as a function of drawing-time condition (i.e., 4s, 8s, 16s, 32s), with random intercepts for object concept. We found that drawings produced under the 4s limit contained the fewest strokes on average, whereas those produced under the 32s limit contained the greatest number of strokes  $(\beta = .29, SE = .05)$ .
Figure 3: Distributions of number of strokes for each drawing time condition in the human sketch production task. Vertical lines indicate means.
$4.95 \times 10^{-3}, p < .001$ . These results confirm that restricting the amount of time human participants had to produce their sketches led to systematic differences in how detailed their sketches were.
Sparser sketches are more semantically ambiguous for models and humans. Having verified that we had successfully manipulated the level of detail in human sketches, we next sought to evaluate how well current vision models extract semantic information from them at each level of detail. Our general approach was to fit mixed-effect linear regression models to estimate the effect of sparsity on 3 key metrics: (1) top-1 classification accuracy, (2) entropy of response distributions, and (3) semantic neighbor preference. Regression models included random intercepts and slopes for object concepts. Figure 4 shows the performance of each vision model with respect to these metrics for sketches produced under different time constraints.
We found that models generally achieved higher top-1 classification accuracy for more detailed sketches than sparser ones ( $\beta = 9.57 \times 10^{-2}$ ,  $t = 20.25$ ,  $p < .001$ ).
We further found the entropy of the models' response distribution was lower for detailed sketches than for sparser sketches ( $\beta = -.03$ ,  $t = -30.19$ ,  $p < .001$ ), suggesting greater uncertainty about the best label to apply to sparser sketches. Even when sketches were more ambiguous, however, models generated labels that were semantically related to the ground-truth label, as measured by our semantic neighbor preference score, with more detailed sketches eliciting a greater proportion of semantically related labels ( $\beta = 2.5 \times 10^{-2}$ ,  $t = 8.43$ ,  $p < .001$ ). These patterns were mirrored in human sketch recognition behavior, with more detailed sketches being associated with higher top-1 classification performance ( $\beta = 6.08 \times 10^{-2}$ ,  $t = 9.57$ ,  $p < .001$ ), a tighter distribution of responses (lower entropy) ( $\beta = -.14$ ,  $t = -12.29$ ,  $p < .001$ ) and greater semantic neighbor preference ( $\beta = 5.41 \times 10^{-2}$ ,  $t = 20.24$ ,  $p < .001$ ).
Different models display distinct patterns of sketch recognition behavior. Although all vision models were sensitive to the effect of our drawing-time manipulation, we found that there were reliable differences in classification accuracy between models  $(\chi^2(16) = 3455.3, p < .001)$ . Moreover, some models generated a greater diversity of responses than others, as measured by the entropy of their response distribution (Fig. 4B,  $\chi^2(16) = 89698, p < .001$ ). Finally, models varied in the degree to which they generated non-ground-truth labels that were semantically related to the ground-truth label  $(\chi^2(16) = 318.46, p < .001)$ . Taken together, these results indicate that these models, all high-performing, display systematic differences in how they extract semantic information from sketches.
A large gap remains between human and model sketch understanding. While both humans and models are affected by the amount of detail in sketches, it is not yet clear to what
degree their response patterns are well aligned. We evaluated human-model alignment scores using the same three metrics (i.e., top-1 classification accuracy, entropy, semantic neighbor preference) by estimating the degree to which model performance on different types of sketches (e.g., lions drawn in 4 seconds or less) covaried systematically with human performance on the same types of sketches. For example, a model is considered well aligned with humans with respect to recognition performance if it achieves high top-1 classification accuracy on the same types of sketches that humans succeed in classifying and if it achieves low accuracy on the types of sketches that humans fail to classify. Similarly, a model is considered well aligned with humans with respect to semantic ambiguity if it produces a response distribution with high entropy for the same types of sketches that humans are also highly uncertain about and if it produces a low-entropy response distribution for the types of sketches that humans systematically agree on (regardless of whether this agreement is concentrated on the correct label). We found that models generally displayed some degree of alignment to humans on top-1 classification accuracy ( $\beta = 7.70 \times 10^{-2}$ ,  $t = 6.77$ ,  $p < .001$ ), response entropy ( $\beta = 1.17 \times 10^{-1}$ ,  $t = 28.52$ ,  $p < .001$ ), and semantic neighbor preference ( $\beta = 6.66 \times 10^{-2}$ ,  $t = 6.21$ ,  $p < .001$ ). Moreover, we found that different vision models aligned with humans to varying degrees (top-1 classification accuracy:  $\chi^2(16) = 134.81$ ,  $p < .001$ ; response entropy:  $\chi^2(16) = 725.78$ ,  $p < .001$ ; semantic neighbor preference:  $\chi^2(16) = 5.05$ ,  $p = .99$ ). Nevertheless, a sizable gap remains between the most aligned models and a human-human consistency baseline for all metrics (Fig 5; top-1 classification accuracy:  $t = 952.19$ ,  $p < .001$ ; response entropy:  $t = 184.21$ ,  $p < .001$ ; semantic neighbor preference:  $t = 389.56$ ,  $p < .001$ ). Finally, we observe that while examining classification accuracy and entropy yield similar rankings over which models are best aligned to humans, these metrics appear to capture non-redundant sources of information about human and model sketch understanding (Fig 5D).
Figure 4: Effect of drawing time constraints on sketch understanding in different vision models.
Figure 5: Human-model alignment on (A) top-1 classification accuracy, (B) response entropy, and (C) semantic neighbour preference. Leftmost red bars in each plot correspond to baseline human-human consistency on each metric. Error bars indicate bootstrapped  $95\%$  confidence intervals. (D) Spearman  $\rho$  correlations between the rank-ordering of vision models with respect to their alignment to human performance on each metric.
A CLIP-based sketch generation algorithm emulates human sketches under some conditions.
While sketch understanding is a critical aspect of visual abstraction, the ability to produce sketches spanning different levels of abstraction is no less important. Although there remains a gap between human and model sketch understanding, a CLIP-based vision model [70] was among the most performant and best aligned to human sketch understanding. Insofar as the major bottleneck to being able to generate more human-like sketches is achieving more human-like understanding of sketches and other images [24], a generative model leveraging CLIP's latent representation may be a promising approach. Consistent with this possibility, we found that CLIPasso generated sketches of concepts at each abstraction level were about as recognizable as the human-generated sketches of those same concepts at the same abstraction level (Fig. 6A; adjusted  $R^2 = .64$ ). Moreover, we found that the more detailed CLIPasso sketches were especially human-like in that they evoked a similar set of meanings to their human-generated counterparts.
To measure the degree to which human and CLIPasso sketches converged with respect to the object labels that they elicited from human viewers, we computed the Jensen-Shannon distance (JSD) between the label distributions of human and CLIPasso sketches for each concept at each of the 4 levels of abstraction. In Fig. 6 B, we show the average label divergence at each level of abstraction. We found that humans and CLIPasso sketches were least divergent in terms of their perceived meaning when they were depicted in greater detail or were less abstract ( $\beta = -0.69$ ,  $t = -6.86$ ,  $p < .001$ ). At lower levels of detail and visual fidelity, human and CLIPasso sketches elicited more diverging responses. Thus, while CLIPasso more closely approximates human sketch production behavior at greater levels of detail, there remains a large gap between how CLIPasso and human participants attempted to preserve sketch meaning under more severe production constraints. Taken together, while CLIPasso marks significant progress towards human-like sketch production its ability to produce highly abstract sketches in a human-like manner remains limited.
Recent advances in machine vision have enabled new opportunities to understand the computational mechanisms that underlie human-like visual abstraction—how humans create and interpret a wide variety of images (from detailed illustrations to schematic diagrams) to convey what they perceive and know about the world. Here we introduce SEVA, a new dataset and benchmark containing approximately 90K human and machine generated sketches spanning multiple levels of abstraction, to evaluate progress towards alignment between human and machine visual abstraction. Critically, our model evaluation protocol is focused not just on performance but on how well these models approximate human-like
Figure 6: (A) Human top-1 recognition performance on CLIPasso and human sketches at different production budgets (draw duration for human sketches and number of strokes for CLIPasso sketches). (B) Divergence in humanresponses to the same type of human and CLIPasso sketches as a function of sketch production budget.
sketch production and understanding. Initial benchmarking of a set of 17 vision models on SEVA following this protocol revealed that even the most performant and well-aligned models deviate from human behavior in systematic ways. We also find that current generative models of sketching are able to sketch in human-like ways, but only in limited settings.
We hope that SEVA will accelerate progress towards unified computational theories that explain how humans are capable of generating and understanding such a wide variety of abstract visual representations. Specifically, our dataset could be used to adjudicate between vision models that are high-performing on tasks involving natural images [64, 29] by characterizing their alignment with how humans understand images generated in a very different manner, including freehand sketches. In addition, SEVA is distinctive among existing human sketch datasets in that it contains detailed measurements of how humans make moment-to-moment decisions about where to place each stroke, when aiming to produce a sketch of a visible object, under different constraints. As such, we expect SEVA to be a key resource for advancing the state-of-the-art in sketch generation [69, 94, 91, 33, 72], and thus lead to computational models of generalized visual abstraction.
We note several limitations of the current study that would be important to address in future work. First, our sample of participants was limited to English-speaking individuals based in the United States. As such, the current study cannot speak to potential differences in sketch-production behavior across geographical and cultural contexts. However, future work that recruits from a broader cross-section of individuals, including those located in the "Majority World," will be vital for understanding those potential sources of variation in human sketch production and comprehension. Second, we recruited participants via Prolific, a widely used crowdsourcing platform in human behavioral research, without regard to any previous artistic training they had received. As such, our study generally reflects sketch production behavior among individuals without substantial expertise in the visual arts. Third, following prior work [24, 23, 34, 105, 43, 59], the human sketches in our dataset were obtained using a web-based digital drawing interface, where most participants used a mouse or trackpad to produce their sketches (89.96%) and some participants used a touchscreen (6.94%) or a stylus (1.67%). Investigation of the impact of input device on sketch production would be a fruitful avenue for follow-up work. Fourth, because we did not obtain non-digital sketches (e.g., drawn on paper with a pen/pencil), our data cannot speak to differences between digital and non-digital sketches. Fifth, our study of machine sketch production included just one model, CLIPasso, limiting the conclusions that can be drawn about sketch generation algorithms in general. Future work that evaluates a broader suite of algorithms would thus be valuable.
We thank members of the Cognitive Tools Lab at Stanford and the Stanford Neuro AI Lab for helpful feedback, discussions, and support. This work was supported by an NSF CAREER Award #2047191 to J.E.F.. J.E.F is additionally supported by an ONR Science of Autonomy award.
All code and materials available at: https://github.com/cogtoolslab/visual_abstractions_benchmarking_public2023/
Visual abstraction is as ubiquitous as it is central to our understanding of the visual world. Humans can recognize depictions of objects at varying levels of fidelity to their real-world counterparts. This ability to recognize and render concepts in abstract forms is crucial for visual knowledge transmission in the form of graphs, diagrams, and symbols. While computer vision has steadily made progress towards algorithms that can recognize objects in scenes, distinguish among different instances of an object, or even answer questions about those objects in natural language, it remains unclear if these systems have the capability to understand visual concepts at multiple levels of abstraction in a human-like manner. SEVA marks a step towards evaluating current vision algorithms on their sensitivity to depictions of a wide variety of common object concepts at varying levels of abstraction. Critically, we also provide results on human performance on sketch understanding for sketches of varying sparsity, setting a clear benchmark for future vision algorithms. Our initial study of state-of-the-art vision algorithms shows that even the most high-performing of models still fall short of human consistency baselines. With these vision algorithms increasingly being deployed in many aspects of everyday life, it is crucial to measure how human-like they are in this most natural of human abilities. We hope that SEVA will be used by the community to help close the gaps in current vision algorithms' alignment to human behavior that we have identified in this paper.
[1] Maxime Aubert, Adam Brumm, Muhammad Ramli, Thomas Sutikna, E Wahyu Saptomo, Budianto Hakim, Michael J Morwood, Gerrit D van den Bergh, Leslie Kinsley, and Anthony Dosseto. Pleistocene cave art from sulawesi, Indonesia. Nature, 514(7521):223-227, 2014.
[2] Ayu Marliawaty I Gusti Bagus, Tiago Marques, Sachi Sanghavi, James J DiCarlo, and Martin Schrimpf. Primate inferotemporal cortex neurons generalize better to novel image distributions than analogous deep neural networks units. In SVRHM 2022 Workshop@ NeurIPS.
[3] Nicholas Baker and Philip J Kellman. Abstract shape representation in human visual perception. Journal of Experimental Psychology: General, 147(9):1295, 2018.
[4] Pedro Ballester and Ricardo Araujo. On the performance of googlenet and alexnet applied to sketches. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[5] MD Barrett and PH Light. Symbolism and intellectual realism in children's drawings. British Journal of Educational Psychology, 46(2):198-202, 1976.
[6] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths. Capturing human categorization of natural images by combining deep networks and cognitive models. Nature communications, 11(1):5418, 2020.
[7] Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.08261, 2021.
[8] Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, and Jessica Hodgins. Style and abstraction in portrait sketching. ACM Transactions on Graphics (TOG), 32(4):1-12, 2013.
[111] Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang. Sketchyscene: Richly-annotated scene sketches. In Proceedings of the European conference on computer vision (ECCV), pages 421–436, 2018.