Deep Learning for Identifying Potential Conceptual
Shifts for Co-creative Drawing

Pegah Karimi
University of North Carolina at CharlotteCharlotte, NC 28223 USApkarimi@uncc.edu
Kazjon Grace
The University of Sydney
Sydney, NSW 2006 Australia
kazjon.grace@sydney.edu.au

Nicholas Davis
University of North Carolina at CharlotteCharlotte, NC 28223 USA
ndavis64@uncc.edu
Mary Lou Maher
University of North Carolina at CharlotteCharlotte, NC 28223 USAm.maher@uncc.edu

Abstract
We present a system for identifying conceptual shifts between visual categories,
which will form the basis for a co-creative drawing system to help users draw morecreative sketches. The system recognizes human sketches and matches them tostructurally similar sketches from categories to which they do not belong. This
would allow a co-creative drawing system to produce an ambiguous sketch thatblends features from both categories.

1

Introduction

Creative sketching is important in a wide variety of domains including engineering, art, and architecture [1]. We propose a co-creative system that plays a free association drawing game thatfacilitates more creative sketches by introducing conceptual shifts. In co-creative systems, an AI
and a user collaborate on a creative task [2]. Our proposed AI agent recognizes objects in humansketches and responds by suggesting a conceptual shift. In this context, a conceptual shift entails
recognizing an object from a different conceptual category with which the current object shares
structural characteristics. The goal is to respond with another sketch that leverages this conceptual
shift, such as by drawing a conceptual blend [3] of the two objects. The resulting free association
game would encourage the fluent expression of ideas by the user. The system described here forms
the basis of a co-creative tool that helps the user to learn creative sketching.

2Training the co-creative agent with sketches

We use clustering on deep features to train the co-creative agent to generate a visual representation ofsketches that enable a conceptual shift. This training has two steps:
Learning the visual representation of sketches. Deep learning architectures are able to extract
highly detailed visual representations from images leading to high classification accuracy and highquality generated images [4]. We use the Google Quick, Draw! dataset [5] in place of user input andtrain our system on a subset of 35 categories with up to 110,000 images in each. We represent eachsketch as a high-level feature vector obtained using VGG-16 [6], a standard convolutional neuralnetwork (CNN) architecture. We started with a model that was pre-trained on the ImageNet dataset[7] and then fine-tuned the weights by training on the sketches. The model contains 15 convolutional
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

layers and two fully connected layers. We extract the features from the first fully connected layer
which produces 4096 features per image, and use these features to represent each sketch.
Clustering sketches into sets. As the objects of each category have high variance (e.g. different
shape, viewpoint and angle), we separate each category using clustering on the VGG-16 representation,
which identifies structurally similar sub-categories that we use for identifying potential conceptual
shifts. We used k-means algorithm and selected k via the elbow method.

Figure 1: Clustering samples (a) and visualization using LargeVis [8] (b).

3

The Co-creative System for drawing

In our proposed co-creative drawing system a user and an agent take turns to sketch. The agent
has to perceive the user input and identify opportunities for conceptual shifts. To identify potential
conceptual shifts we match objects from one category to structurally similar objects in anothercategory. We compare the Euclidean distances between the centroids of the cluster (i.e. sub-category)
to which the sketch belongs and clusters from other categories. We identify the cluster with theminimum distance to the current object’s cluster and use this to propose a conceptual shift. Our
approach involves 3 steps:
Recognizing. The agent takes the user’s sketch (currently a randomly selected sketch from thedataset), extracts its feature vector, and determines its cluster.
Matching. The user’s sketch is matched to the most similar cluster in another category.
Contributing. The agent contributes (currently by random selection from the dataset, but in futureby (neural) drawing) a sketch from the cluster identified as a potential conceptual shift.
In Figure 2 we show examples of this process from 6 categories. The three on the left have identified
good potential conceptual shifts (with a low distance between clusters), while the three on the righthave less potential (and higher distance).

Figure 2: Agent responses to user sketches from six categories.

2

4Conclusions and Future WorkIn this paper we described a co-creative drawing system that plays a free association game. We described a framework that identifies a conceptual shift through recognizing, matching and contributing.
This system will be used as the basis for a drawing agent that creates conceptual blends that encourage
user creativity. In the future we will investigate alternative architectures for the encoder to improve
the sketch representation for clustering. In addition, we would like to develop a more comprehensive
matching algorithm for conceptual shifts, and incorporate a neural approach to drawing.

References[1] Ullman, D. G., Wood, S. & Craig, D. (1990). The importance of drawing in the mechanical design process.
Computers and graphics, 14(2), 263–274.
[2] Davis, N., Hsiao, C. P., Yashraj Singh, K., Li, L. & Magerko, B. (2016, March). Empirically studying
participatory sense-making in abstract drawing with a co-creative cognitive agent. In Proceedings of the 21stInternational Conference on Intelligent User Interfaces (pp. 196-207). ACM.
[3] Fauconnier, G. & Turner, M. (2008). The way we think: Conceptual blending and the mind’s hidden
complexities. Basic Books.
[4] LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
[5] Ha, D. & Eck, D. (2017). A Neural Representation of Sketch Drawings. arXiv preprint arXiv:1704.03477.
[6] Simonyan, K. & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition.
arXiv preprint arXiv:1409.1556.
[7] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K. & Fei-Fei, L. (2009, June). Imagenet: A large-scale
hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conferenceon (pp. 248-255). IEEE.
[8] Tang, J., Liu, J., Zhang, M. & Mei, Q. (2016, April). Visualizing large-scale and high-dimensional data. In
Proceedings of the 25th International Conference on World Wide Web (pp. 287-297). International World Wide
Web Conferences Steering Committee.

3