DeepFaceDrawing: Deep Generation of Face Images from Sketches
SHU-YU CHEN† , Institute of Computing Technology, CAS and University of Chinese Academy of Sciences
WANCHAO SU† , School of Creative Media, City University of Hong Kong
LIN GAO∗ , Institute of Computing Technology, CAS and University of Chinese Academy of Sciences
SHIHONG XIA, Institute of Computing Technology, CAS and University of Chinese Academy of Sciences
HONGBO FU, School of Creative Media, City University of Hong Kong

(a)

(b)

(c)

(d)

(e)

Fig. 1. Our DeepFaceDrawing system allows users with little training in drawing to produce high-quality face images (Bottom) from rough or even incomplete
freehand sketches (Top). Note that our method faithfully respects user intentions in input strokes, which serve more like soft constraints to guide image
synthesis.
Recent deep image-to-image translation techniques allow fast generationof face images from freehand sketches. However, existing solutions tend to
overfit to sketches, thus requiring professional sketches or even edge maps
as input. To address this issue, our key idea is to implicitly model the shapespace of plausible face images and synthesize a face image in this space toapproximate an input sketch. We take a local-to-global approach. We firstlearn feature embeddings of key face components, and push correspondingparts of input sketches towards underlying component manifolds definedby the feature vectors of face component samples. We also propose anotherdeep neural network to learn the mapping from the embedded component
features to realistic images with multi-channel feature maps as intermediate
results to improve the information flow. Our method essentially uses input
sketches as soft constraints and is thus able to produce high-quality face
images even from rough and/or incomplete sketches. Our tool is easy touse even for non-artists, while still supporting fine-grained control of shapedetails. Both qualitative and quantitative evaluations show the superior
generation ability of our system to existing and alternative solutions. The
usability and expressiveness of our system are confirmed by a user study.
† Authors contributed equally.
∗ Corresponding author.
Webpage: http://geometrylearning.com/DeepFaceDrawing/This is the author’s version of the work. It is posted here for your personal use. Not forredistribution.

CCS Concepts: • Human-centered computing → Graphical user interfaces; • Computing methodologies → Perception; Texturing; Image
processing.
Additional Key Words and Phrases: image-to-image translation, featureembedding, sketch-based generation, face synthesis

1

INTRODUCTION

Creating realistic human face images from scratch benefits various applications including criminal investigation, character design,
educational training, etc. Due to their simplicity, conciseness andease of use, sketches are often used to depict desired faces. The
recently proposed deep learning based image-to-image translation
techniques (e.g., [19, 38]) allow automatic generation of photo images from sketches for various object categories including humanfaces, and lead to impressive results.
Most of such deep learning based solutions (e.g., [6, 19, 26, 38]) for
sketch-to-image translation often take input sketches almost fixed
and attempt to infer the missing texture or shading information
between strokes. To some extent, their problems are formulated
more like reconstruction problems with input sketches as hard constraints. Since they often train their networks from pairs of real

2 •Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

images and their corresponding edge maps, due to the data-driven
nature, they thus require test sketches with quality similar to edge
maps of real images to synthesize realistic face images. However,such sketches are difficult to make especially for users with little
training in drawing.
To address this issue, our key idea is to implicitly learn a spaceof plausible face sketches from real face sketch images and findthe closest point in this space to approximate an input sketch. In
this way, sketches can be used more like soft constraints to guide
image synthesis. Thus we can increase the plausibility of synthesized images even for rough and/or incomplete input sketches while
respecting the characteristics represented in the sketches (e.g., Figure 1 (a-d)). Learning such a space globally (if exists) is not very
feasible due to the limited training data against an expected highdimensional feature space. This motivates us to implicitly modelcomponent-level manifolds, which makes a better sense to assume
each component manifold is low-dimensional and locally linear [32].
This decision not only helps locally span such manifolds using alimited amount of face data, but also enables finer-grained controlof shape details (Figure 1 (e)).
To this end we present a novel deep learning framework forsketch-based face image synthesis, as illustrated in Figure 3. Our
system consists of three main modules, namely, CE (Component
Embedding), FM (Feature Mapping), and IS (Image Synthesis). The
CE module adopts an auto-encoder architecture and separately
learns five feature descriptors from the face sketch data, namely,for “left-eye”, “right-eye”, “nose”, “mouth”, and “remainder” for locally spanning the component manifolds. The FM and IS modules
together form another deep learning sub-network for conditional
image generation, and map component feature vectors to realistic
images. Although FM looks similar to the decoding part of CE, by
mapping the feature vectors to 32-channel feature maps insteadof 1-channel sketches, it improves the information flow and thus
provides more flexibility to fuse individual face components forhigher-quality synthesis results.
Inspired by [25], we provide a shadow-guided interface (implemented based on CE) for users to input face sketches with proper
structures more easily (Figure 8). Corresponding parts of input
sketches are projected to the underlying facial component manifolds
and then mapped to the corresponding feature maps for conditions
for image synthesis. Our system produces high-quality realistic
face images (with resolution of 512 × 512), which faithfully respect
input sketches. We evaluate our system by comparing with the existing and alternative solutions, both quantitatively and qualitatively.
The results show that our method produces visually more pleasing face images. The usability and expressiveness of our system
are confirmed by a user study. We also propose several interesting
applications using our method.

2

RELATED WORK

Our work is related to existing works for drawing assistance andconditional face generation. We focus on the works closely related
to ours. A full review on such topics is beyond the scope of ourpaper.

2.1

Drawing AssistanceMultiple guidance or suggestive interfaces (e.g., [17]) have been
proposed to assist users in creating drawings of better quality. For
example, Dixon et al. [7] proposed iCanDraw, which provides corrective feedbacks based on an input sketch and facial features extracted
from a reference image. ShadowDraw by Lee et al. [25] retrieves
real images from an image repository involving many object categories for an input sketch as query and then blends the retrieved
images as shadow for drawing guidance. Our shadow-guided interface for inputting sketches is based on the concept of ShadowDraw
but specially designed for assisting in face drawing. Matsui et al. [29]proposed DrawFromDrawings, which allows the retrieval of reference sketches and their interpolation with an input sketch. Our
solution for projecting an input sketch to underlying component
manifolds follows a similar retrieval-and-interpolation idea but we
perform this in the learned feature spaces, without explicit correspondence detection, as needed by DrawFromDrawings. Unlike theabove works, which aim to produce quality sketches as output, ourwork treats such sketches as possible inputs and we are more interested in producing realistic face images even from rough and/or
incomplete sketches.
Another group of methods (e.g., [1, 18]) take a more aggressive
way and aim to automatically correct input sketches. For example, Limpaecher et al. [27] learn a correction vector field from acrowdsourced set of face drawings to correct a face sketch, with theassumption that such face drawings and the input sketch is for asame subject. Xie et al. [41] and Su et al. [36] propose optimizationbased approaches for refining sketches roughly drawn on a reference image. We refine an input sketch by projecting individual facecomponents of the input sketch to the corresponding component
manifolds. However, as shown in Figure 5, directly using such refined component sketches as input to conditional image generation
might cause artifacts across facial components. Since our goal issketch-based image synthesis, we thus perform sketch refinement
only implicitly.

2.2

Conditional Face Generation

In recent years, conditional generative models, in particular, conditional Generative Adversarial Networks [11] (GANs), have been
popular for image generation conditioned on various input types.
Karras et al. [22] propose an alternative for the generator in GAN
that separates the high level face attributes and stochastic variations in generating high quality face images. Based on conditional
GANs [30], Isola et al. [19] present the pix2pix framework for various image-and-image translation problems like image colorization,
semantic segmentation, sketch-to-image synthesis, etc. Wang etal. [38] introduce pix2pixHD, an improved version of pix2pix to generate higher-resolution images, and demonstrate its application toimage synthesis from semantic label maps. Wang et al. [37] generate
an image given a semantic label map as well as an image exemplar.
Sangkloy et al. [34] take hand-drawn sketches as input and colorize them under the guidance of user-specified sparse color strokes.
These systems tend to overfit to conditions seen during training, andthus when sketches being used as conditions, they achieve quality
results only given edge maps as input. To address this issue, insteadDeepFaceDrawing: Deep Generation of Face Images from Sketches • 3

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. The comparisons of different edge extraction methods. (a): Input
real image. (b): Result by HED [42]. (c): Result by APDrawingGAN [43]. (d):Canny edges [4]. (e): the result by the Photocopy filter in Photoshop. (f):Simplification of (e) by [35]. Photo (a) courtesy of © LanaLucia.

of training an end-to-end network for sketch-to-image synthesis,
we exploit the domain knowledge and condition GAN on featuremaps derived from the component feature vectors.
Considering the known structure of human faces, researchers
have explored component-based methods (e.g., [15]) for face image
generation. For example, given an input sketch, Wu and Dai [40]first retrieve best-fit face components from a database of face images, then compose the retrieved components together, and finally
deform the composed image to approximate a sketch. Due to theirsynthesis-and-deforming strategy, their solution requires a welldrawn sketch as input. To enable component-level controllability,
Gu et al. [12] use auto-encoders to learn feature embeddings forindividual face components, and fuse component feature tensors
in a mask-guided generative network. Our CE module is inspiredby their work. However, their local embeddings learned from real
images are mainly used to generate portrait images with high diversity while ours learned from sketch images are mainly for implicitly
refining and completing input sketches.
Conditional GANs have also been adopted for local editing offace images, via interfaces either based on semantic label masks[12, 24, 37] or sketches [20, 31]. While the former is more flexible forapplications such as component transfer and style transfer, the latterprovides a more direct and finer control of details, even within facecomponents. Deep sketch-based face editing is essentially a sketchguided image completion problem, which requires the completionof missing parts such that the completed content faithfully reflects
an input sketch and seamlessly connects to the known context.
It thus requires different networks from ours. The SN-patchGAN
proposed by Jo and Park [20] is able to produce impressive details
for example for a sketched earring. However, this also implies that
their solution requires high-quality sketches as input. To tolerate
the errors in hand-drawn sketches, Portenier et al. [31] propose touse smoothed edge maps as part of the input to their conditional
completion network. Our work takes a step further to implicitly
model face component manifolds and perform manifold projection.
Several attempts have also been made to generate images fromincomplete sketches. To synthesize face images from line maps possibly with some missing face components, Li et al. [26] proposed a conditional self-attention GAN with a multi-scale discriminator, wherea large-scale discriminator enforces the completeness of global structures. Although their method leads to visually better results thanpix2pix [19] and SkethyGAN [5], due to the direct condition onedge maps, their solution has poor ability to handle hand-drawn
sketches. Ghosh et al. [10] present a shape generator to completea partial sketch before image generation, and present interesting
auto-completion results. However, their synthesized images still
exhibit noticeable artifacts, since the performance of their image
generation step (i.e., pix2pixHD [38] for single-class generation and
SkinnyResNet [10] for multi-class generation) heavily depends on
the quality of input sketches. A similar problem exists with the progressive image reconstruction network proposed by You et al. [45],which is able to reconstruct images from extremely sparse inputs
but still requires relatively accurate inputs.
To alleviate the heterogeneity of input sketches and real face
images, some researchers resort to the unpaired image-to-image
methods (e.g., [16, 44, 48]). These methods adopt self-consistentconstraints to solve the lack of paired data. While the self-consistentmechanism ensures the correspondence between the input and thereconstructed input, there is no guarantee for the correspondence
between the input and the transformed representation. Since ourgoal is to transform sketches to the corresponding face images, theseframeworks are not suitable for our task. In addition, there are someworks leveraging the image manifolds. For example, Lu et al. [28]learn a fused representation from shape and texture features toconstruct a face retrieval system. In contrast, our method not only
retrieves but also interpolates the face representations in generation.
Zhu et al. [47] first construct a manifold with the real image dataset,then predict a dense correspondence between a projected source
image and an edit-oriented “feasible” target in the manifold, andfinally apply the dense correspondence back to the original source
image to complete the visual manipulation. In contrast, our method
directly interpolates the nearest neighbors of the query and feeds theinterpolation result to the subsequent generation process. Comparedto Zhu et al. [47], our method is more direct and efficient for thesketch-based image generation task.

3

METHODOLOGY

The 3D shape space of human faces has been well studied (see theclassic morphable face model [3]). A possible approach to synthesize
realistic faces from hand-drawn sketches is to first project an input
sketch to such a 3D face space [13] and then synthesize a face image
from a generated 3D face. However, such a global parametric model
is not flexible enough to accommodate rich image details or support
local editing. Inspired by [8], which shows the effectiveness of a
local-global structure for faithful local detail synthesis, our method
aims for modeling the shape spaces of face components in the image
domain.

4•

Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

Fig. 3. Illustration of our network architecture. The upper half is the Component Embedding module. We learn feature embeddings of face components using
individual auto-encoders. The feature vectors of component samples are considered as the point samples of the underlying component manifolds and are used
to refine an input hand-drawn sketch by projecting its individual parts to the corresponding component manifolds. The lower half illustrates a sub-network
consisting of the Feature Mapping (FM) and the Image Synthesis (IS) modules. The FM module decodes the component feature vectors to the corresponding
multi-channel feature maps (𝐻 × 𝑊 × 32), which are combined according to the spatial locations of the corresponding facial components before passing themto the IS module.

To achieve this, we first learn the feature embeddings of facecomponents (Section 3.2). For each component type, the points
corresponding to component samples implicitly define a manifold.
However, we do not explicitly learn this manifold, since we are moreinterested in knowing the closest point in such a manifold given a
new sketched face component, which needs to be refined. Observing
that in the embedding spaces semantically similar components areclose to each other, we assume that the underlying component
manifolds are locally linear. We then follow the main idea of theclassic locally linear embedding (LLE) algorithm [32] to project thefeature vector of the sketched face component to its component
manifold (Section 3.3).
The learned feature embeddings also allow us to guide conditionalsketch-to-image synthesis to explicitly exploit the information in thefeature space. Unlike traditional sketch-to-image synthesis methods(e.g., [19, 38]), which learn conditional GANs to translate sketches toimages, our approach forces the synthesis pipeline to go through thecomponent feature spaces and then map 1-channel feature vectors
to 32-channel feature maps before the use of a conditional GAN(Section 3.2). This greatly improves the information flow and benefits component fusion. Below we first discuss our data preparation
procedure (Section 3.1). We then introduce our novel pipeline forsketch-to-image synthesis (Section 3.2), and our approach for manifold projection (Section 3.3). Finally present our shadow-guided
interface (Section 3.4).

3.1

Data Preparation

To train our network, it requires a reasonably large-scale dataset offace sketch-image pairs. There exist several relevant datasets like the
CUHK face sketch database [39, 46]. However, the sketches in suchdatasets involve shading effects while we expect a more abstract
representation of faces using sparse lines. We thus contribute to a
new dataset of pairs of face images and corresponding synthesized
sketches. We build this on the face image data of CelebAMask-HQ[24], which contains high-resolution facial images with semantic
masks of facial attributes. For simplicity, we currently focus on front
faces, without decorative accessories (e.g., glasses, face masks).
To extract sparse lines from real images, we have tried the following edge detection methods. As shown in Figure 2 (b) and (d),DeepFaceDrawing: Deep Generation of Face Images from Sketches • 5

the holistically-nested edge detection (HED) method [42] and the
traditional Canny edge detection algorithm [4] tend to produce edge
maps with discontinuous lines. APDrawingGAN [43], a very recent
approach for generating portrait drawings from face photos leads
to artistically pleasing results, which, however, are different fromour expectation (e.g., see the regional effects in the hair area andmissing details around the mouth in Figure 2 (c)). We also resorted
to the Photocopy filter in Photoshop, which preserves facial details
but meanwhile brings excessive noise (Figure 2 (e)). By applying thesketch simplification method by Simo-Serra et al. [35] to the result
by the Photocopy filter, we get an edge map with the noise reduced
and the lines better resembling hand-drawn sketches (Figure 2 (f)).
We thus adopt this approach (i.e., Photocopy + sketch simplification)
to prepare our training dataset, which contains 17K pairs of sketchimage pairs (see an example pair in Figure 2 (f) and (a)), with 6247
for male subjects and 11456 for female subjects. Since our dataset is
not very large-scale, we reserve the data in the training process asmuch as possible to provide as many samples as possible to span
the component manifolds. Thus we set a training/testing ratio to
20:1 in our experiments. It results in 16,860 samples for training and
842 for testing.

3.2

Sketch-to-Image Synthesis Architecture

Fig. 4. Two examples of generation flexibility supported by using separate
components for the left and right eyes.

As illustrated in Figure 3, our deep learning framework takes as
input a sketch image and generates a high-quality facial image ofsize 512×512. It consists of two sub-networks: The first sub-network
is our CE module, which is responsible for learning feature embeddings of individual face components using separate auto-encoder
networks. This step turns component sketches into semanticallymeaningful feature vectors. The second sub-network consists of
two modules: FM and IS. FM turns the component feature vectors
to the corresponding feature maps to improve the information flow.
The feature maps of individual face components are then combined
according to the face structure and finally passed to IS for face image
synthesis.

We experimented with different numbers of dimensions for the latent representation (128, 256, 512) – we found that 512 dimensions
are enough for reconstructing and representing the sketch details.
Instead, lower-dimensional representations tend to lead to blurry
results. By trial and error, we append a residual block after every
convolution/deconvolution operation in each encoding/decoding
layer to construct the latent descriptors instead of only using convolution and deconvolution layers. We use Adam solver [23] in the
training process. Please find the details of the network architectures
and the parameter settings in the supplemental materials.

Component Embedding Module. Since human faces share a clear
structure, we decompose a face sketch into five components, denoted
as 𝑆 𝑐 , 𝑐 ∈ {1, 2, 3, 4, 5} for “left-eye", “right-eye", “nose", “mouth",and “remainder", respectively. To handle the details in-between components, we define the first four components simply by using four
overlapping windows centered at individual face components (derived from the pre-labeled segmentation masks in the dataset), as
illustrated in Figure 3 (Top-Left). A “remainder” image corresponding to the “remainder” component is the same as the original sketch
image but with the eyes, nose and mouth removed. Here we treat“left-eye” and “right-eye” separately to best explore the flexibility inthe generated faces (see two examples in Figure 4). To better controlof the details of individual components, for each face component
type we learn a local feature embedding. We obtain the featuredescriptors of individual components by using five auto-encoder
networks, denoted as {𝐸𝑐 , 𝐷𝑐 } with 𝐸𝑐 being an encoder and 𝐷𝑐 a
decoder for component 𝑐.
Each auto-encoder consists of five encoding layers and five decoding layers. We add a fully connected layer in the middle to ensure
the latent descriptor is of 512 dimensions for all the five components.

Feature Mapping Module. Given an input sketch, we can project
its individual parts to the component manifolds to increase its plausibility (Section 3.3). One possible solution to synthesize a realistic
image is to first convert the feature vectors of the projected manifold
points back to the component sketches using the learned decoders{𝐷𝑐 }, then perform component-level sketch-to-image synthesis (e.g.,based on [38]), and finally fuse the component images together to
get a complete face. However, this straightforward solution easily
leads to inconsistencies in synthesized results in terms of both local
details and global styles, since there is no mechanism to coordinate
the individual generation processes.
Another possible solution is to first fuse the decoded component
sketches into a complete face sketch (Figure 5 (b)) and then perform
sketch-to-image synthesis to get a face image (Figure 5 (c)). It can beseen that this solution also easily causes artifacts (e.g., misalignment
between face components, incompatible hair styles) in the synthesized sketch, and such artifacts are inherited to the synthesizedimage, since existing deep learning solutions for sketch-to-image
synthesis tend to use input sketches as rather hard constraints, as
discussed previously.

6 •Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

f sc
w1c

w2c

c
f proj

wkc

Mc
(a)

(c)

(b)

(d)

Fig. 5. Given the same input sketch (a), image synthesis conditioned on thefeature vectors after manifold projection achieves a more realistic result (d)than that (c) by image synthesis conditioned on an intermediate sketch (b).
See the highlighted artifacts in both the intermediate sketch (b) and thecorresponding synthesized result (c) by pix2pixHD [38].

We observe that the above issues mainly happen in the overlapping regions of the cropping windows for individual components.
Since sketches only have one channel, the incompatibility of neighboring components in the overlapping regions is thus difficult toautomatically resolve by sketch-to-image networks. This motivates
us to map the feature vectors of sampled manifold points to multichannel feature maps (i.e., 3D feature tensors). This significantly
improves the information flow, and fusing the feature maps insteadof sketch components helps resolve the inconsistency between facecomponents.
Since the descriptors for different components bear different semantic meanings, we design the FM module with five separate
decoding models converting feature vectors to spatial feature maps.
Each decoding model consists of a fully connected layer and five
decoding layers. For each feature map, it has 32 channels and isof the same spatial size as the corresponding component in thesketch domain. The resulting feature maps for “left-eye”, “right-eye”,“nose”, and “mouth” are placed back to the “remainder” feature maps
according to the exact positions of the face components in the input
face sketch image to retain the original spatial relations between
face components. As illustrated in Figure 3 (Bottom-Center), we
use a fixed depth order (i.e., “left/right eyes" > “nose" > “mouth” >“remainder") to merge the feature maps.

Kc

Fig. 6. Illustration of manifold projection. Given a new feature vector 𝑓𝑠˜𝑐 , we𝑐replace it with the projected feature vector 𝑓𝑝𝑟𝑜
𝑗 using K nearest neighborsof 𝑓𝑠˜𝑐 .

Image Synthesis Module. Given the combined feature maps, theIS module converts them to a realistic face image. We implement
this module using a conditional GAN architecture, which takes thefeature maps as input to a generator, with the generation guided
by a discriminator. Like the global generator in pix2pixHD [38],our generator contains an encoding part, a residual block, and adecoding unit. The input feature maps go through these units sequentially. Similar to [38], the discriminator is designed to determine
the samples in a multi-scale manner: we downsample the input tomultiple sizes and use multiple discriminators to process different
inputs at different scales. We use this setting to learn the high-level
correlations among parts implicitly.
Two-stage Training. As illustrated in Figure 3, we adopt a twostage training strategy to train our network using our dataset ofsketch-image pairs (Section 3.1). In Stage I, we train only the CE module, by using component sketches to train five individual autoencoders for feature embeddings. The training is done in a selfsupervised manner, with the mean square error (MSE) loss between
an input sketch image and the reconstructed image. In Stage II, we
fix the parameters of the trained component encoders and train
the entire network with the unknown parameters in the FM andIS modules together in an end-to-end manner. For the GAN in the IS,
besides the GAN loss, we also incorporate a 𝐿1 loss to further guidethe generator and thus ensure the pixel-wise quality of generated
images. We use the perceptual loss [21] in the discriminator to compare the high-level difference between real and generated images.
Due to the different characteristics of female and male portraits, wetrain the network using the complete set but constrain the searching
space into the male and female spaces for testing.

3.3

Manifold Projection

Let S = {𝑠𝑖 } denote a set of sketch images used to train the featureembeddings of face components (Section 3.2). For each component 𝑐,
we can get a set of points in the 𝑐-component feature space by using
the trained encoders, denoted as F 𝑐 = {𝑓𝑖𝑐 = 𝐸𝑐 (𝑠𝑖𝑐 )}. Althougheach feature space is 512-dimensional, given that similar component
images are placed closely in such feature spaces, we tend to believethat all the points in F 𝑐 are in an underlying low-dimensional
manifold, denoted as M𝑐 , and further assume each component
manifold is locally linear: each point and its neighbors lie on orclose to a locally linear patch of the manifold [32].

DeepFaceDrawing: Deep Generation of Face Images from Sketches • 7that as we change the interpolation weight continuously, it results insmooth changes between the consecutive reconstructed component
sketches from a pair of selected sketches. This shows the feasibilityof our descriptor interpolation.

Fig. 7. Illustration of linear interpolation between pairs of randomly selected neighboring component sketches (Leftmost and Rightmost) in thecorresponding feature spaces. The middle three images are decoded fromthe uniformly interpolated feature vectors.

˜ to increase its plausibility as a human
Given an input sketch 𝑠,face, we project its 𝑐-th component to M𝑐 . With the locally linear
assumption, we follow the main idea of LLE and take a retrievaland-interpolation approach to project the 𝑐-th component feature
vector of 𝐸𝑐 (𝑠˜𝑐 ), denoted as 𝑓𝑠˜𝑐 to M𝑐 , as illustrated in Figure 3.
As illustrated in Figure 6, given the 𝑐-th component feature vector
𝑓𝑠˜𝑐 , we first find the 𝐾 nearest samples in F 𝑐 under the Euclidean
space. By trial and error, we found that 𝐾=10 is sufficient in providing face plausibility while maintaining adequate variations. Let
K 𝑐 = {𝑠𝑘𝑐 } (with {𝑠𝑘 } ⊂ S) denote the resulting set of 𝐾 nearest
samples, i.e., the neighbors of 𝑠˜𝑐 on M𝑐 . We then seek a linear combination of these neighbors to reconstruct 𝑠˜𝑐 by minimizing thereconstruction error. This is equivalent to solving for the interpolation weights through the following minimization problem:ÕÕmin ||𝑓𝑠˜𝑐 −𝑤𝑘𝑐 · 𝑓𝑘𝑐 || 22, 𝑠.𝑡 .
𝑤𝑘𝑐 = 1,
(1)
𝑘 ∈K 𝑐𝑘 ∈K𝑐where 𝑤𝑘 is the unknown weight for sample 𝑠𝑘𝑐 . The weights can befound by solving a constrained least-squares problem for individualcomponents independently. Given the solved weights {𝑤𝑘𝑐 }, theprojected point of 𝑠˜𝑐 on M𝑐 can be computed asÕ𝑐𝑓𝑝𝑟𝑜
𝑤𝑘𝑐 · 𝑓𝑘𝑐 .
(2)
𝑗 =𝑘 ∈K 𝑐𝑐
𝑓𝑝𝑟𝑜 𝑗 is the feature vector of the refined version of 𝑠˜𝑐 , and can bepassed to the FM and IS modules for image synthesis.
To verify the local continuity of the underlying manifolds, wefirst randomly select a sample from S, and for its 𝑐-th component
randomly select one of its nearest neighbors in the correspondence
feature space. We then perform linear interpolation between such apair of component sketches in the 𝑐-th feature space, and reconstruct
the interpolated component sketches using the learned 𝑐-th decoder
𝐷𝑐 . The reconstructed results are shown in Figure 7. It can be seenFig. 8. A screenshot of our shadow-guided sketching interface (Left) for
facial image synthesis (Right). The sliders at the up-right corner can be used
to control the degree of interpolation between an input sketch and a refinedversion after manifold project for individual components.

3.4

Shadow-guided Sketching Interface

To assist users, especially those with little training in drawing, inspired by ShadowDraw [25], we provide a shadow-guided sketching
˜ we first find 𝐾 (𝐾 = 10 in ourinterface. Given a current sketch 𝑠,
implementation) most similar sketch component images from S
according to 𝑠˜𝑐 by using the Euclidean distance in the feature space.
The found component images are then blended as shadow andplaced at the corresponding components’ positions for sketchingguidance (Figure 8 (Left)). Initially when the canvas is empty, theshadow is more blurry. The shadow is updated instantly for every
new input stroke. The synthesized image is displayed in the window
on the right. Users may choose to update the synthesized image
instantly or trigger an “Convert” command. We show two sequencesof sketching and synthesis results in Figure 18.
Users with good drawing skills tend to trust their own drawings
more than those with little training in drawing. We thus provide
a slider for each component type to control the blending weights
between a sketched component and its refined version after manifold
projection. Let 𝑤𝑏𝑐 denote the blending weight for component 𝑐.
The feature vector after blending can be calculated as:𝑐𝑐𝑓𝑏𝑙𝑒𝑛𝑑= 𝑤𝑏𝑐 × 𝑓𝑠˜𝑐 + (1 − 𝑤𝑏𝑐 ) × 𝑓𝑝𝑟𝑜
𝑗.

(3)

𝑐
Feeding 𝑓𝑏𝑙𝑒𝑛𝑑
to the subsequent trained modules, we get a new

synthesized image.
Figure 9 shows an example of synthesized results under different values of 𝑤𝑏𝑐 . This blending feature is particularly useful for
creating faces that are very different from any existing samples ortheir blending. For example, for the female data in our training set,most of the subjects have long hairstyles. Always pushing our input
sketch to such samples would not allow us to create short-hairstyle8 •Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

Input Sketch

𝑤𝑏 5 = 0.50

𝑤𝑏 5 = 0.00

𝑤𝑏 5 = 0.75

𝑤𝑏 5 = 0.25

Input Sketch

Without Refinement 𝑤𝑏 = 1.0

Full Refinement 𝑤𝑏 = 0.0

𝑤𝑏 1,2,4 = {0.7, 0.4, 0.3}

𝑤𝑏 5 = 1.00

Fig. 9. Interpolating an input sketch and its refined version (for the “remainder” component in this example) after manifold projection under different
blending weight values. 𝑤𝑏𝑐 = 1 means a full use of an input sketch forimage synthesis, while by setting 𝑤𝑏𝑐 = 0 we fully trust the data for interpolation.

effects. This is solved by trusting the input sketch for its “remainder”
component by adjusting its corresponding blending weight. Figure
10 shows another example with different blending weights for different components. It can be easily seen that the result with automaticrefinement (lower left) is visually more realistic than that without
any refinement (upper right). Fine-tuning of the blending weights
leads to a result better reflecting the input sketch more faithfully.

4

EXPERIMENTS

We have done extensive evaluations to show the effectiveness of oursketch-to-image face synthesis system and its usability via a pilot
study. Below we present some of the obtained results. Please refer tothe supplemental materials for more results and an accompanying
video for sketch-based image synthesis in action.
Figure 11 shows two representative results where users progressively introduce new strokes to add or stress local details. As shownin the demo video, running on a PC with an Intel i7-7700 CPU, 16GB
RAM and a single Nvidia GTX 1080Ti GPU, our method achieves
real-time feedback. Thanks to our local-to-global approach, generally more strokes lead to new or refined details (e.g., the nosein the first example, and the eyebrows and wrinkles in the second
example), with other areas largely unchanged. Still due to the combination step, local editing might still introduce subtle but globalchanges. For example, for the first example, the local change oflighting in the nose area leads to the change of highlight in thewhole face (especially in the forehead region). Figure 18 shows twomore complete sequences of progressive sketching and synthesis,
with our shadow-guided interface.

4.1

Fig. 10. Blending an input sketch and its refined version after manifold
projection for the “left-eye”, “right-eye”, and “mouth” components. Upper
Right: result without any sketch refinement; Lower Left: result with fulldegree sketch refinement; Lower Right: result with partial-degree sketchrefinement.

Usability Study

We conducted a usability study to evaluate the usefulness and effectiveness of our system. 10 subjects (9 male and 1 female, aged from

20 to 26) were invited to participate in this study. We first asked
them to self-assess their drawing skills through a nine-point Likert scale (1: novice to 9: professional), and divided them into threegroups: 4 novice users (drawing skill score: 1 – 3), 4 middle users(4 – 6), and 2 professional users (7 – 9). Before the drawing session,
each participant was given a short tutorial about our system (about
10 minutes). The participants used an iPad with iPencil to remotely
control the server PC for drawing. Then each of them was asked
to create at least 3 faces using our system. The study ended with aquestionnaire to get user feedbacks on ease-of-use, controllability,
variance of results, quality of results, and expectation fitness. The
additional comments on our system were also welcome.
Figure 12 gives a gallery of sketches and synthesized faces by theparticipants. It can be seen that our system consistently produce
realistic results given input sketches with different styles and levelsof abstraction. For several examples, the participants attempted to
depict beard styles via hatching and our system captured the users’
intention very well.
Figure 13 shows a radar plot, summarizing quantitative feedbacks
on our system for participant groups with different levels of drawing
skills. The feedbacks for all the groups of participants were positivein all the measured aspects. Particularly, the participants with good
drawing skills felt a high level of controllability, while they gave

DeepFaceDrawing: Deep Generation of Face Images from Sketches • 9original ShadowDraw paper [25]. One of the professional users mentioned that automatic synthesis of face images given sparse inputs
saved a lot of efforts and time compared to traditional painting
software. One professional user mentioned that it would be better
if our system could provide color control.

4.2

Fig. 11. Representative results through progressive sketching for adding
details (1st example) and stressing local details (2nd example).

slightly lower scores for the degree of result variance. Using oursystem, the average time needed for drawing a face sketch among
the participants with different drawing abilities are: 17 ′ 14 ′′ (professional), 3 ′ 17 ′′ (middle) and 2 ′ 26 ′′ (novice). It took much longer for
professionals, since they spent more time sketching and refining
details. For the refinement sliders, the most frequently used slider
was for the “remainder” component (56.69%), which means for morethan half of the results, the “remainder” slider was manipulated. In
contrast, for the other components we have 21.66% for “left-eye”,
12.74% for “right-eye”, 12.10% for “nose” and 19.75% for “mouth”.
For all the adjustments made in the components, participants trust
the “remainder” component most, with the averaged confidence
0.78; The least trusted component is “mouth” (0.56); other component confidences are 0.70 (“left-eye”), 0.61 (“right-eye”) and 0.58(“nose”). The averaged confidences implied the importance of sketchrefinement in creating the synthesized faces in this study.
All of the participants felt that our system was powerful to create
realistic faces using such sparse sketches. They liked the intuitive
shadow-guided interface, which was quite helpful for them to construct face sketches with proper structures and layouts. On the other
hand, some users, particularly those with good drawing skills, feltthat the shadow guidance was sometimes distracting when editing details. This finding is consistent with the conclusions in the

Comparison with Alternative Refinement Strategies

To refine an input sketch, we essentially take a component-level
retrieval-and-interpolation approach. We compare this method with
two alternative sketch refinement methods by globally or locally
retrieving the most similar sample in the training data. For faircomparison, we use the same FM and IS modules for image synthesis.
For the local retrieval method, it is the same as our method exceptthat for manifold projection we simply retrieve the closest (i.e., top1 instead of top-𝐾) component sample in each component-levelfeature space without any interpolation. For the global retrieval
method, we replace the CE module with a new module for the featureembeddings of entire face sketches. Specifically, we first learn thefeature embeddings of the entire face sketch images, and given a
new sketch we find the most similar (i.e., top-1) sample in the wholeface feature space. For each component in the globally retrieved
sample image, we then encode it using the corresponding trained
component-level encoder (i.e., 𝐸𝑐 ), and pass all the component-levelfeature vectors to our FM and IS for image synthesis. Note that we
do not globally retrieve real face images, since our goal here is for afair comparison of the sketch refinement methods.
Figure 14 shows comparison results. From the overlay of input
sketches and the retrieved or interpolated sketches, it can be easily
seen that the component-level retrieval method returns samples
closer to the input component sketches than the global-retrieval
method, mainly due to the limited sample data. Thanks to the interpolation step, the interpolated sketches almost perfectly fit the
input sketches. Note that we show the decoded sketches after interpolation here only for the comparison purpose, and our conditional
image synthesis sub-network takes the interpolated feature vectors
as input (Section 3.2).

4.3

Perceptive Evaluation StudyAs shown in Figure 14 (Right), the three refinement methods alllead to realistic face images. To evaluate the visual quality and thefaithfulness (i.e., the degree of fitness to input sketches) of synthesize
results, we conducted a user study.
We prepared a set of input sketches, containing in total 22 sketches,
including 9 from the usability study (Section 4.1) and 13 from theauthors. This sketch set (see the supplementary materials) covered
inputs with different levels of abstraction and different degrees ofroughness. We applied the three refinement methods to each input
sketch to generate the corresponding synthesized results (see two
representative sets in Figure 15).
The evaluation was done through an online questionnaire. 60
participants (39 male, 21 female, aged from 18 to 32) participated inthis study. Most of them had no professional training in drawing.
We showed each participant four images including input sketch andthe three synthesized images, placed side by side in a random order.
Each participant was asked to evaluate the quality and faithfulness

10•

Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

Fig. 12. Gallery of input sketches and synthesized results in the usability study.

DeepFaceDrawing: Deep Generation of Face Images from Sketches •(a)11Professional
NoviceMiddle

(e)

(b)

(c)

Fig. 13. The summary of quantitative feedback in the usability study. (a)Ease of use. (b) Controllability. (c) Variance of results. (d) Quality of results.
(e) Expectation fitness.

Quality Perception
Score
Faithfulness
Perception
Score

(a)

Fig. 15. Two representative sets of input sketches and synthesized results
used in the perceptive evaluation study. From left to right: input sketch, theresults by sketch refinement through global retrieval, local retrieval, andlocal retrieval with interpolation (our method).

(b)
Input Sketch

5.8
5
5.6

Faithfulness
Perception
Score
Quality Perception
Score

(d)

5.4
4.5
5.2
5
4
4.8
4.6
3.5
4.4
4.2
3

(c)
Part 2

Part 3

Part 4

5.4
4.5
4.5
5.2
5
4
4
4.8
4.6
3.5
3.5
4.4
4.2
3

GlobalPart 1

5.5
5.8
5
5.6
5

Local

Interpolation

GlobalLocal

Interpolation
Interpolation

Image
Fig. 16. Box plots of the average quality and faithfulness perception scores
over the participants for each method.

(a)Fig. 14. Comparisons of using global retrieval (a), component-level retrieval(b), and our method (essentially component-level retrieval followed by interpolation) (c) for sketch refinement. The right column shows the corresponding synthesized results. For easy comparison we overlay input sketches(in light blue) on top of the retrieved or interpolated sketches by different
methods.

further confirmed that our method (mean: 4.85) led to significantly
more faithful results than both the global (mean: 3.65; [𝑡 = −29.77,
𝑝 < 0.001] and local (mean: 4.23; [𝑡 = −16.34, 𝑝 < 0.001]) retrieval
methods. This is consistent with our expectation, since our method
provides the largest flexibility to fit to input sketches.
In terms of visual quality our method (mean: 5.50) significantly
outperformed the global retrieval method (mean: 5.37; [𝑡 = −3.94,
𝑝 < 0.001]) and the local retrieval method (mean: 4.68; [𝑡 = −24.60,
𝑝 < 0.001]). It is surprisingly to see that the quality of results by
our method was even higher than the global retrieval method, since
we had expected that the visual quality of the results by the global
method and ours would be similar. This is possibly because some
information is lost after first decomposing an entire sketch into
components and then recombining the corresponding feature maps.

both in a seven-point Likert scale (1 = strongly negative to 7 =strongly positive). In total, to evaluate either the faithfulness or thequality, we got 60 (participants) × 22 (sketches) = 1,320 subjective
evaluations for each method.
Figure 16 plots the statistics of the evaluation results. We performed one-way ANOVA tests on the quality and faithfulness scores,
and found significant effects for both quality (𝐹 (2,63) = 47.26, 𝑝 <0.001) and faithfulness (𝐹 (2,63) = 51.72, 𝑝 < 0.001). Paired t-tests

We compare our method with the state-of-the-art methods for image
synthesis conditioned on sketches, including pix2pix [19], pix2pixHD[38] and Lines2FacePhoto [26] and iSketchNFill [10] in terms of visualquality of generated faces. We use their released source code but for
fair comparisons we train all the networks on our training dataset(Section 3.1). The (input and output) resolution for our method andpix2pixHD is 512 × 512, while we have 256 × 256 for pix2pix andLines2FacePhoto according to their default setting. In addition, for(b)
Input Sketch(c)Part 1

Part 2

Part 3

Part 4Image

4.4

Comparison with Existing Solutions

•Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

Ours

iSketchNFill

Pix2pixHD

Lines2FacePhoto

Pix2pix

Input Sketch

12Fig. 17. Comparisons with the state-of-the-art methods given the same input sketches (Top Row).

Lines2FacePhoto, following their original paper, we also convert eachsketch to a distance map as input for both training and testing. For
iSketchNFill, we train their shape completion module before feeding
it to pix2pix [19] (acting as the appearance synthesis module). The
input and output resolutions in their method are 256 × 256 and128 × 128, respectively.

Figure 17 shows representative testing results given the same
sketches as input. It can be easily seen that our method produces
more realistic synthesized results. Since the input sketches are roughand/or incomplete, they are generally different from the training
data, making the compared methods fail to produce realistic faces.
Although Lines2FacePhoto generates a relatively plausible result

DeepFaceDrawing: Deep Generation of Face Images from Sketches •given an incomplete sketch, its ability to handle data imperfections
is rather limited. We attempted to perform quantitative evaluation as
well. However, none of the assessment metrics we tried, including
Fréchet Inception Distance [14] and Inception Score [33], could
faithfully reflect visual perception. For example, the averaged valuesof the Inception Score were 2.59 and 1.82 (the higher, the better) for
pix2pixHD and ours, respectively. However, it is easily noticeable
that our results are visually better than those by pix2pixHD.

5

APPLICATIONS

Our system can be adapted for various applications. In this section
we present two applications: face morphing and face copy-paste.

5.1

Face Morphing

Traditional face morphing algorithms [2] often require a set of
keypoint-level correspondence between two face images to guidesemantic interpolation. We show a simple but effective morphing approach by 1) decomposing a pair of source and target face sketches inthe training dataset into five components (Section 3.2); 2) encoding
the component sketches as feature vectors in the corresponding feature spaces; 3) performing linear interpolation between the source
and target feature vectors for the corresponding components; 4)
finally feeding the interpolated feature vectors to the FM and IS module to get intermediate face images. Figure 19 shows examples offace morphing using our method. It can be seen that our method
leads to smoothly transforming results in identity, expression, andeven highlight effects.

5.2

Face Copy-Paste

Traditional copy-paste methods (e.g., [9]) use seamless stitching
methods on colored images. However, there will be situations where
the hue of local areas is irrelevant. To address this issue, we recombine face components for composing new faces, which can maintainthe consistency of the overall color and lighting. Specifically, it canbe achieved by first encoding face component sketches (possibly
from different subjects) as feature vectors and then combining themas new faces by using the FM and IS modules. This can be used
to either replace components of existing faces with corresponding
components from another source, or combining components frommultiple persons. Figure 20 presents several synthesized new faces
by re-combining eyes, nose, mouth and the remainder region fromfour source sketches. Our image synthesis sub-network is able toresolve the inconsistencies between face components from different
sources in terms of both lighting and shape.

6

CONCLUSION AND DISCUSSIONSIn this paper we have presented a novel deep learning frameworkfor synthesizing realistic face images from rough and/or incomplete
freehand sketches. We take a local-to-global approach by first decomposing a sketched face into components, refining its individualcomponents by projecting them to component manifolds defined by
the existing component samples in the feature spaces, mapping therefined feature vectors to the feature maps for spatial combination,and finally translating the combined feature maps to realistic images. This approach naturally supports local editing and makes the13

involved network easy to train from a training dataset of not verylarge scale. Our approach outperforms existing sketch-to-image
synthesis approaches, which often require edge maps or sketcheswith similar quality as input. Our user study confirmed the usabilityof our system. We also adapted our system for two applications:face morphing and face copy-paste.
Our current implementation considers individual components
rather independently. This provides flexibility (Figure 4) but also
introduces possible incompatibility problems. This issue is more
obvious for the eyes (Figure 21), which are often symmetric. Thismight be addressed by introducing a symmetry loss [15] or explicitly
requiring two eyes from the same samples (similar to Figure 20).
Our work has focused on refining an input sketch component-bycomponent. In other words our system is generally able to handle
errors within individual components, but is not designed to fix theerrors in the layouts of components (Figure 21). To achieve proper
layouts, we resort to a shadow-guided interface. In the future, we areinterested in modeling spatial relations between facial components
and fixing input layout errors.
Our system takes black-and-white rasterized sketches as input
and currently does not provide any control of color or texture insynthesized results. In a continuous drawing session, small changes
in sketches sometimes might cause abrupt color changes. This mightsurprise users and is thus not desirable for usability. We believe
this can be potentially addressed by introducing a color control
mechanism in generation. For example, we might introduce color
constraints by either adding them in the input as additional hints
or appending them to the latent space as additional guidance. In
addition, adding color control is also beneficial for applications suchas face morphing and face copy-and-paste.
Like other learning-based approaches, the performance of oursystem is also dependent on the amount of training data. Althoughcomponent-level manifolds of faces might be low dimensional, due
to the relatively high-dimensional space of our feature vectors, ourlimited data only provides very sparse sampling of the manifolds. In
the future we are interested in increasing the scale of our training
data, and aim to model underlying component manifolds moreaccurately. This will also help our system to handle non-frontal
faces, faces with accessories. It is also interesting to increase thediversity of results by adding random noise to the input. Explicitly
learning such manifolds and providing intuitive exploration tools
in a 2D space would be also interesting to explore.
Our current system is specially designed for faces by making useof the fixed structure of faces. How to adapt our idea to support
the synthesis of objects of other categories is an interesting but
challenging problem.

ACKNOWLEDGMENTS
This work was supported by Beijing Program for International
S&T Cooperation Project (No. Z191100001619003), Royal Society
Newton Advanced Fellowship (No. NAF\R2\192151), Youth Innovation Promotion Association CAS, CCF-Tencent Open Fund andOpen Project Program of the National Laboratory of Pattern Recognition (NLPR) (No. 201900055). Hongbo Fu was supported by anunrestricted gift from Adobe and grants from the Research Grants

14•

Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu

Fig. 18. Two sequences of progressive sketching (under shadow guidance) and synthesis results.

Fig. 19. Examples of face morphing by interpolating the component-level feature vectors of two given face sketches (Leftmost and Rightmost are corresponding
synthesized images).

Council of the Hong Kong Special Administrative Region, China (No.
CityU 11212119, 11237116), City University of Hong Kong (No. SRG
7005176), and the Centre for Applied Computing and Interactive
Media (ACIM) of School of Creative Media, CityU.

REFERENCES[1] James Arvo and Kevin Novins. 2000. Fluid Sketches: Continuous Recognition andMorphing of Simple Hand-Drawn Shapes. In Proceedings of the 13th annual ACM
symposium on User interface software and technology. ACM, 73–80.
[2] Martin Bichsel. 1996. Automatic interpolation and recognition of face images bymorphing. In Proceedings of the Second International Conference on Automatic Face

DeepFaceDrawing: Deep Generation of Face Images from Sketches •15

Fig. 20. In each set, we show color image (Left) of the source sketches (not shown here), a new face sketch (Middle) by directly recombining the cropped
source sketches in the image domain, and a new face (Right) synthesized by using our method with the recombined sketches of the cropped components(eyes, nose, mouth, and remainder) as input.

Fig. 21. A less successful example. The eyes in the generated image are ofdifferent colors. For the sketched mouth, it is slightly below an expected
position, leading to a blurry result for this component.

and Gesture Recognition. IEEE, 128–135.
[3] Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of
3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics andInteractive Techniques. ACM, 187–194.
[4] John Canny. 1986. A computational approach to edge detection. IEEE Transactions
on Pattern Analysis and Machine Intelligence PAMI-8, 6 (1986), 679–698.
[5] Wengling Chen and James Hays. 2018. Sketchygan: Towards diverse and realistic
sketch to image synthesis. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). IEEE, 9416–9425.
[6] Tali Dekel, Chuang Gan, Dilip Krishnan, Ce Liu, and William T Freeman. 2018.
Sparse, smart contours to represent and edit images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3511–3520.
[7] Daniel Dixon, Manoj Prasad, and Tracy Hammond. 2010. iCanDraw: using sketch
recognition and corrective feedback to assist a user in drawing human faces. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
ACM, 897–906.
[8] Lin Gao, Jie Yang, Tong Wu, Yu-Jie Yuan, Hongbo Fu, Yu-Kun Lai, and Hao(Richard)
Zhang. 2019. SDM-NET: Deep Generative Network for Structured Deformable
Mesh. ACM Trans. Graph. 38, 6 (2019), 243:1–243:15.
[9] Shiming Ge, Xin Jin, Qiting Ye, Zhao Luo, and Qiang Li. 2018. Image editingby object-aware optimal boundary searching and mixed-domain composition.
Computational Visual Media 4 (01 2018). https://doi.org/10.1007/s41095-017-01028[10] Arnab Ghosh, Richard Zhang, Puneet K Dokania, Oliver Wang, Alexei A Efros,Philip HS Torr, and Eli Shechtman. 2019. Interactive Sketch & Fill: Multiclass
Sketch-to-Image Translation. In IEEE International Conference on Computer Vision(ICCV). IEEE, 1171–1180.
[11] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial

Nets. In Proceedings of the 27th International Conference on Neural Information
Processing Systems - Volume 2 (Montreal, Canada) (NIPS’14). MIT Press, Cambridge,
MA, USA, 2672–2680.
[12] Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. 2019.
Mask-Guided Portrait Editing with Conditional GANs. In IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). IEEE, 3436–3445.
[13] Xiaoguang Han, Chang Gao, and Yizhou Yu. 2017. DeepSketch2Face: a deeplearning based sketching system for 3D face and caricature modeling. ACM Trans.
Graph. 36, 4, Article 126 (2017), 12 pages.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, andSepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge toa local nash equilibrium. In Advances in Neural Information Processing Systems.
Curran Associates, Inc., 6626–6637.
[15] Rui Huang, Shu Zhang, Tianyu Li, and Ran He. 2017. Beyond face rotation:
Global and local perception gan for photorealistic and identity preserving frontal
view synthesis. In IEEE International Conference on Computer Vision (ICCV). IEEE,
2439–2448.
[16] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal
unsupervised image-to-image translation. In European Conference on Computer
Vision (ECCV). 172–189.
[17] Emmanuel Iarussi, Adrien Bousseau, and Theophanis Tsandilas. 2013. The drawing assistant: Automated drawing guidance and feedback from photographs. In
Proceedings of the 26th Annual ACM Symposium on User Interface Software andTechnology. ACM, 183–192.
[18] Takeo Igarashi, Satoshi Matsuoka, Sachiko Kawachiya, and Hidehiko Tanaka.
1997. Interactive Beautification: A Technique for Rapid Geometric Design. In
Proceedings of the 10th Annual ACM Symposium on User Interface Software andTechnology (UIST ’97). Association for Computing Machinery, 105–114.
[19] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-toimage translation with conditional adversarial networks. In IEEE Conference on
Computer Vision and Pattern Recognition (CVPR). IEEE, 1125–1134.
[20] Youngjoo Jo and Jongyoul Park. 2019. SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color. In IEEE International Conference
on Computer Vision (ICCV). IEEE, 1745–1753.
[21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for realtime style transfer and super-resolution. In European Conference on Computer
Vision (ECCV). Springer-Verlag, 694–711.
[22] Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision
and Pattern Recognition (CVPR). 4401–4410.
[23] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. http://arxiv.org/abs/1412.6980 cite arxiv:1412.6980Comment: Published
as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
[24] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2019. MaskGAN: Towards
Diverse and Interactive Facial Image Manipulation. arXiv preprint arXiv:1907.11922(2019).
[25] Yong Jae Lee, C Lawrence Zitnick, and Michael F Cohen. 2011. Shadowdraw:
real-time user guidance for freehand drawing. ACM Trans. Graph. 30, 4, Article
27 (2011), 10 pages.

16•

Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu[26] Yuhang Li, Xuejin Chen, Feng Wu, and Zheng-Jun Zha. 2019. LinesToFacePhoto:
Face Photo Generation From Lines With Conditional Self-Attention Generative
Adversarial Networks. In Proceedings of the 27th ACM International Conference on
Multimedia. ACM, 2323–2331.
[27] Alex Limpaecher, Nicolas Feltman, Adrien Treuille, and Michael Cohen. 2013.
Real-time drawing assistance through crowdsourcing. ACM Trans. Graph. 32, 4,
Article 54 (2013), 8 pages.
[28] Zongguang Lu, Yang Jing, and Qingshan Liu. 2017. Face image retrieval basedon shape and texture feature fusion. Computational Visual Media 3, 4 (12 2017),
359–368. https://doi.org/10.1007/s41095-017-0091-7[29] Yusuke Matsui, Takaaki Shiratori, and Kiyoharu Aizawa. 2016. DrawFromDrawings: 2D drawing assistance via stroke interpolation with a sketch database. IEEE
Transactions on Visualization and Computer Graphics 23, 7 (2016), 1852–1862.
[30] Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784 (2014).
[31] Tiziano Portenier, Qiyang Hu, Attila Szabo, Siavash Arjomand Bigdeli, Paolo
Favaro, and Matthias Zwicker. 2018. Faceshop: Deep sketch-based face image
editing. ACM Trans. Graph. 37, 4, Article 99 (2018), 13 pages.
[32] Sam T Roweis and Lawrence K Saul. 2000. Nonlinear dimensionality reduction
by locally linear embedding. Science 290, 5500 (2000), 2323–2326.
[33] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford,
and Xi Chen. 2016. Improved techniques for training gans. In Advances in neural
information processing systems. Curran Associates, Inc., 2234–2242.
[34] Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference
on Computer Vision and Pattern Recognition (CVPR). IEEE, 5400–5409.
[35] Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016.
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup.
ACM Trans. Graph. 35, 4, Article 121 (2016), 11 pages.
[36] Qingkun Su, Wing Ho Andy Li, Jue Wang, and Hongbo Fu. 2014. EZ-sketching:
three-level optimization for error-tolerant image tracing. ACM Trans. Graph. 33,
4, Article 54 (2014), 9 pages.
[37] Miao Wang, Guo-Ye Yang, Ruilong Li, Run-Ze Liang, Song-Hai Zhang, Peter M
Hall, and Shi-Min Hu. 2019. Example-guided style-consistent image synthesis fromsemantic labeling. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR). 1495–1504.
[38] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan
Catanzaro. 2018. High-resolution image synthesis and semantic manipulation withconditional gans. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR). IEEE, 8798–8807.
[39] Xiaogang Wang and Xiaoou Tang. 2008. Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 11 (2008),
1955–1967.
[40] Di Wu and Qionghai Dai. 2009. Sketch realizing: lifelike portrait synthesis fromsketch. In Proceedings of the 2009 Computer Graphics International Conference.
ACM, 13–20.
[41] Jun Xie, Aaron Hertzmann, Wilmot Li, and Holger Winnemöller. 2014. PortraitSketch: face sketching assistance for novices. In Proceedings of the 27th annual
ACM symposium on User interface software and technology. ACM, 407–417.
[42] Saining Xie and Zhuowen Tu. 2015. Holistically-Nested Edge Detection. In IEEE
International Conference on Computer Vision (ICCV). IEEE, 1395–1403.
[43] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin. 2019. APDrawingGAN:
Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,
10743–10752.
[44] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised
dual learning for image-to-image translation. In IEEE International Conference on
Computer Vision (ICCV). 2849–2857.
[45] Sheng You, Ning You, and Minxue Pan. 2019. PI-REC: Progressive Image Reconstruction Network With Edge and Color Domain. arXiv preprint arXiv:1903.10146(2019).
[46] Wei Zhang, Xiaogang Wang, and Xiaoou Tang. 2011. Coupled informationtheoretic encoding for face photo-sketch recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 513–520.
[47] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. 2016.
Generative Visual Manipulation on the Natural Image Manifold. In European
Conference on Computer Vision (ECCV).
[48] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired
image-to-image translation using cycle-consistent adversarial networks. In IEEE
International Conference on Computer Vision (ICCV). 2223–2232.