Learning to Simplify:
Fully Convolutional Networks for Rough Sketch Cleanup
Edgar Simo-Serra∗

Satoshi Iizuka∗

Kazuma Sasaki

Hiroshi Ishikawa

Waseda University

Figure 1: Example of our sketch simplification results on two different images. Our approach automatically converts the rough pencil
sketches on the left to the clean vector results on the right.

Abstract

1

In this paper, we present a novel technique to simplify sketch drawings based on learning a series of convolution operators. In contrast
to existing approaches that require vector images as input, we allow
the more general and challenging input of rough raster sketchessuch as those obtained from scanning pencil sketches. We convert
the rough sketch into a simplified version which is then amendable for vectorization. This is all done in a fully automatic way
without user intervention. Our model consists of a fully convolutional neural network which, unlike most existing convolutional
neural networks, is able to process images of any dimensions andaspect ratio as input, and outputs a simplified sketch which has thesame dimensions as the input image. In order to teach our model tosimplify, we present a new dataset of pairs of rough and simplified
sketch drawings. By leveraging convolution operators in combination with efficient use of our proposed dataset, we are able to train
our sketch simplification model. Our approach naturally overcomesthe limitations of existing methods, e.g., vector images as input andlong computation time; and we show that meaningful simplifications can be obtained for many different test cases. Finally, we
validate our results with a user study in which we greatly outperform similar approaches and establish the state of the art in sketch
simplification of raster images.

Sketching is the fundamental first step for expressing artistic ideas
and beginning an iterative process of design refinement. It allows
artists to quickly render their ideas on paper. The priority is to express concepts and ideas quickly, rather than exhibit fine details,
which leads to coarse and rough sketches. After an initial sketch,feedback is used to iteratively refine the design until the final pieceis produced. This iterative refinement forces artists to have to continuously clean up their rough sketches into simplified drawings
and thus implies an additional workload. The process of manuallytracing the rough sketch to produce a clean drawing, as one would
expect, is fairly tedious and time-consuming.

Keywords: sketch simplification, convolutional neural network
Concepts: •Applied computing → Fine arts; •Computing
methodologies → Neural networks;∗ The authors assert equal contribution and joint first authorship.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on

Introduction

In this work we aim at automatically converting rough sketches into
simplified clean drawings. Unlike existing methods, we are able
to directly simplify rough raster sketches, which is fundamental as a large segment of the artist population uses traditional tools
such as pencil-and-paper rather than digital tablets. Our approach,based on Convolutional Neural Networks (CNN), consists of processing the image with a series of convolution operations that areable to group the rough sketch lines and output a simplification directly. The kernels used for the convolutions are learnt from a novel
dataset of rough images with their associated simplifications which
we also present in this work. This data-driven approach has two
import advantages: first of all, it learns all the necessary heuristics
necessary for sketch simplification automatically from the training
data, and secondly, convolutions can be implemented efficiently on
the GPU allowing for processing times of under a second for mostimages. Unlike most standard CNN architectures used for processservers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org. c 2016 Copyrightheld by the owner/author(s). Publication rights licensed to ACM.
SIGGRAPH ’16 Technical Paper,, July 24-28, 2016, Anaheim, CA,
ISBN: 978-1-4503-4279-7/16/07
DOI: http://dx.doi.org/10.1145/2897824.2925972

ing images which use layers that are fully connected to the previous
layer, ours uses only convolutional layers, based on sparse connections, which allow our approach to process images of any resolution
or aspect ratio efficiently.

Rough

TargetRoughTargetOnce a rough sketch is processed by our model to obtain a simplified sketch, it is then possible to use existing vectorization approaches to convert the raster output image to a vector image. As
we will show, directly vectorizing the rough sketch leads to very
noisy images, while vectorizing the output of our approach leads toclean images that can be used as is. We show several examples ofcomplicated scenes drawn with pencil converted to vector images
with our approach in Fig. 1.
In summary, we present:• The first sketch simplification approach that is optimized to directly operate on raster images of rough sketches.
• A novel fully Convolutional Neural Network architecture thatcan simplify sketches directly from images of any resolution.
• An efficient approach to learn the sketch simplification network.
• A dataset for large-scale learning of sketch simplification.

2

Related Work

Various approaches have been proposed to simplify sketch drawings of vector images. One of the strategies for simplification isprogressive modification during sketching. In this approach, several drawing tools assist the user in adjusting the shapes of thestrokes using: mark-based reparametrization [Baudel 1994], geometric constraints among strokes [Igarashi et al. 1997], cubic Bézier
curve fitting [Bae et al. 2008], and progressive merging based on
proximity and topology [Grimm and Joshi 2012]. Fišer et al. [2015]
proposed a system for beautification of freehand sketches basedon various rules of geometric relationships between strokes, whichworks with general Bézier curves. These progressive drawing tools
generally depend on the stroke ordering and thus are not easily
adapted to non-progressive applications. In contrast, our approachis independent of the stroke order and works on general images.
Other approaches simplify line drawings by removing unnecessary
strokes. Preim and Strothotte [1995] enable the user control over
the amount of lines based on the length, screen position, and density. Deussen and Strothotte [2000] used depth information to draw
simplified foliage of trees. Depth and silhouette information obtained from 3D models is often utilized to evaluate the significanceof input strokes [Wilson and Ma 2004; Grabli et al. 2004]. Cole etal. [2006] proposed item and priority buffers that determine line visibility and line density respectively. The main problem with thesemethods is that they are only able to remove existing strokes andare unable to add new ones. This is a severe limitation as usuallylong strokes consist of a series of short strokes in sketch drawings;
the best solution is not necessarily any of the strokes that have beendrawn, but a new stroke that would be consistent with the smaller
ones. Our approach can both remove and add strokes.
In contrast to the stroke reduction that only removes the less significant strokes, several methods to generate new meaningful strokes
by grouping drawn strokes have been proposed. Roshin [1994]
grouped strokes based on their three aspects: continuation, parallelism, and proximity. Lindlbauer et al. [2013] added appearance
similarities (e.g., thickness) to the above features to improve theperceptual grouping. Barla et al. [2005] proposed a morphologicalproperty on simplified strokes that prevents them from folding onto
themselves. This method was later improved by exploiting the extent of overlapping [Shesh and Chen 2008]. Pusch et al. [2007] presented subdivision-based line simplification that recursively subdivide an input image until each sub-box has a single stroke. The

Figure 2: Examples of the complexity of simplifying rough raster
images. We show small examples of rough sketch patches and theircorresponding sketch simplifications taken from our dataset. Note
how it is common for multiple lines to have to be collapsed intoa single line and how the intensity of the different input lines vary
greatly even within the same image. Our approach is able to learn
how to tackle these extremely challenging using our dataset to thensimplify general rough sketches.
sub-boxes are then connected and B-spline curve fitting is used
to generate smooth simple strokes. Orbay and Kara [2011] proposed a sketch beautification method that converts digitally-createdsketches into beautified curve segments. They use a supervised
stroke clustering algorithm based on geometric relationships between strokes of training sketches. Liu et al. [2015] proposed a
closure-aware sketch simplification that utilizes closed regions ofstrokes for semantic analysis of input drawings. However, thesestroke reduction approaches still require vector images as input,
while our approach can be applied on raster images.
Although the simplification methods of vector images reasonably
succeed to generate meaningful simple drawings, the sketch simplification of raster images remains a challenging problem, as neither geometric continuities nor the ordering of vectorized strokes
cannot be used. Traditional vectorization approaches are basedon line tracing [Freeman 1974], thinning [Zhang and Suen 1984],
straight line fitting to anchor points [Janssen and Vossepoel 1997],
and cubic Bézier curves fitting [Chang and Yan 1998]. Hilaire andTombre [2006] proposed a vectorization method that segments line
drawings into the most probable graphical primitives such as arcs.
These methods use binary images as input and are not suitable forfree-hand rough sketches. Bartolo et al. [2007] described a simplification and vectorization technique for scribble drawings using Gabor and Kalman filtering. Chen et al. [2013] proposed a gradientbased technique for coherence-enhancing filtering, which generates
simplified smooth lines via non-oriented gradient fields. However,their method cannot generate detailed structures of sketches suchas pencil-and-paper drawings where gradients are subtle and noisy.
Noris et al. [2013] proposed a vectorization technique for clean
drawings, which solves ambiguities near junctions of strokes basedon gradient-based pixel clustering and a reverse drawing approachthat determines the most suitable stroke configurations. However,unlike our method, this method is not applicable for rough sketch
simplification as it cannot convert multiple rough lines to a single
clean line. Furthermore, none of these approaches have been used
on input images as challenging as the ones we present in this work.
While neural networks learnt with back-propagation have been×

Up-convolution
2

×2

Flat-convolution
4

×

4

8

×

8

4

×

4

2

×

×

2

Down-convolution
Figure 3: Overview of our model. Our model is based on convolutional layers of three types: down-convolution, with a stride of 2 that halves
the image size; flat-convolution, with a stride of 1 that maintains the image size; and up-convolution, with a stride of 1/2 that doubles theimage size. Initially we decrease the image size with down-convolutions to reduce the data bandwidth and increase the spatial support ofsubsequent layers, afterwards up-convolutions are used to restore the image to its original size. The depth of each of the convolutional-layer
blocks in the figure is proportional to the number of filters it has.
around for several decades [Rumelhart et al. 1986], only recently
has the computational power and data been available to more fully
exploit the technique [Krizhevsky et al. 2012]. Originally focusing
on classification, in the last few years there have been many different networks proposed for particular tasks. Related to the model
we present in this paper are the approaches that output images,
such as super-resolution [Dong et al. 2016], semantic segmentations [Long et al. 2015; Noh et al. 2015], contour detection [Shenet al. 2015], and optical flow [Fischer et al. 2015]. Out of these approaches, we can distinguish those that rely on fixed-size image
patches [Shen et al. 2015; Dong et al. 2016], and those that rely
on up-convolutions [Long et al. 2015; Noh et al. 2015; Fischer
et al. 2015]. Our model is inspired by the up-convolutions-based
approaches [Zeiler and Fergus 2014; Long et al. 2015; Dosovitskiy et al. 2015], which allow designing networks that downsample
to spatially compress the information, and then upsample the databack to the original image size. This also allows training everything
in a single end-to-end system unlike the patch-based approaches.
In contrast with other methods that use natural images [Long et al.
2015; Noh et al. 2015], we are unable to exploit existing networks
as they both require RGB image inputs and are optimized for natural images rather than rough sketches; so we train our network
entirely from scratch.
The deep network architecture of Noh et al. [2015] is the most similar to our approach: it relies on a fully-convolutional architecturewith up-convolutions for semantic segmentation. Yet it still has significant differences due to building off a VGG16 network architecture [Simonyan and Zisserman 2015] and conserving all the poolinglayers and the fully-connected layers except the last (treated as convolutions with 1 × 1 kernels). This results in a network that can
only deal with resolutions in 224 × 224 pixel increments due tousing an accumulated 224 × 224 pixel pooling in their architecture, i.e., images between 224 × 224 and 448 × 448 pixels without
padding will have outputs with 224 × 224 pixels. In contrast, ourarchitecture uses an accumulated 8×8 pixel pooling (in the form of
down-convolutions instead of max-pooling) which allows a muchlarger range of output image resolutions. By not relying on existing
pre-trained networks and designing our architecture from scratch,
we are able to completely adapt our network to the rough sketch
simplification problem. Furthermore, in order to simplify sketches,
we have carefully created a dataset and use a new training procedure which is essential for performance and allows the training of

networks from scratch. In particular, without the inverse dataset
creation technique we present, it is not possible to train a successful sketch simplification model at all.
In this paper, we overcome the strong limitation of vector input images that existing approaches to sketch simplification have. We areable to handle a variety of practical rough sketches such as scanned
pencil-and-paper drawings and detailed sketches with complicatedstructures as shown in Fig. 2, which cannot be directly vectorizedusing existing methods. Note how multiple lines are used to represent single lines. Our approach overcomes the difficulty of theseimages to provide realistic sketch simplifications.

3

Learning to Simplify

We base our model on very deep Convolutional Neural Networks(CNNs) [Krizhevsky et al. 2012; Simonyan and Zisserman 2015]that have a large capacity to learn from data to perform sketch
simplification. In order to be able to simplify sketch images,
we leverage a large set of recent improvements, e.g., batch normalization [Ioffe and Szegedy 2015], ADADELTA [Zeiler 2012],
3 × 3 convolution kernels [Simonyan and Zisserman 2015], upconvolutions [Long et al. 2015], no explicit pooling [Springenberg
et al. 2015], etc., and heavily tailor both the model and the learning
approach for the task of sketch simplification. Our contributions include both a novel method for learning and model architecture. An
overview of our model can be seen in Fig. 3.

3.1

Convolutional Neural Networks

Convolutional Neural Networks are an extension to Artificial Neural Networks (ANNs) in which the weights are shared across layers [Fukushima 1988; LeCun et al. 1998]. ANNs and their derivatives are a method of approximating a complex unknown function.
In our case, this consists of the operation of converting a roughsketch into a simplified drawing. The network consists of several
layers of units that can hold real numbers. Each layer can be seen
as a multichannel image of the size h × w, where h and w are theheight and the width. Let C denote the number of channels, so thatthe multichannel image is a vector in RC·h·w . The first layer is the
input layer, thus its size coincides with the size (H × W ) of the input grayscale image, i.e., h = H, w = W, C = 1. Similarly, thestride1/2 will double the resolution of the output layer [Long et al. 2015],

as input pixels will be linearly interpolated before being convolved
with the convolutional kernel. Our model will use both downscaling and upscaling convolutional layers to increase the spatial resolution with a decreased number of layers, while maintaining an
output the same size as the input. An overview of using strides to
up- and downsample images is shown in Fig. 4.

strideFlat-convolution
strideUp-convolution
Down-convolution
Figure 4: Upsampling and downsampling using convolutions. We
show how using different strides with convolutions allows us todownsample (down-convolution), perform a non-linear mapping(flat-convolution), and upsample the input (up-convolution).
last layer is the output layer, which also has the same size.
Successive layers are connected by a convolution-with-bias map
00convadd : RC·h·w −→ RC ·h ·w

0

,

(1)

where (C, h, w) and (C 0 , h0 , w0 ) are the number of channels, theheight, and the width of a layer (L) and the next (L0 ). For eachchannel C 0 of layer L0 , the map is defined as a convolution witha kernel of the size C × kh × kw followed by the addition of ac0
constant “bias” image. Let Wc,i,j
be the components of the kernel
0and bc the constant bias for channel c0 of the layer L0 , respectively.
Then, the value yc0 ,u,v of a specific pixel at (u, v) in channel c0 of
the layer L0 is given by:0

yc0 ,u,v = bc0 +

kh
X

0

kw
C
X
X

0 j=−k0 c=1
i=−kh
w

0

c
0 ,j+k0 xc,u+i,v+j , (2)Wc,i+kwhwhere (xc,s,t ) is the multichannel image of layer L, kh0 = (kh −1)/2,0
and kw= (kw −1)/2.
In the ANN view, this can be seen as synapses connecting the layers where the weights W are independent of the spatial location(u, v) and thus can be seen as shared between synapses related bya parallel translation. Conversely, we can learn the kernel and thebias by back-propagation [Rumelhart et al. 1986] while fixing theshared weights to each other. Thus, the kernel and the bias together
give rise to C · C 0 · kh · kw + C 0 learnable weights for each pair of
successive layers. Note the number of weights only depends on the
kernel size, and the number of channels in the layers.
It is possible to use an increased “stride” to lower the resolution of
the output layer. That is, only a subset of positions (u, v) are computed for yc0 ,u,v . For example, a stride of 2 would decrease theresolution of the output volume by two as it would only compute
yc0 ,u,v for every other pixel. By decreasing the spatial resolution,subsequent convolutions will have an increased spatial support, i.e.,the “pixels” in the feature maps will be computed using a larger
patch of the original input image. For example, a 3 × 3 convolution on the original image has a spatial support of 3 × 3 input pixels
for each output pixel. However, if the original image is resized tohalf the size, the same 3 × 3 convolution will actually be looking
at a 5 × 5 image patch in the original image. We will construct ourmodel by using increased strides for the first layers to increase thespatial support of subsequent layers. However, increasing the stride
decreases the image resolution. In order for the output image tobe the same size as the input, we utilize fractional strides to effectively increase the resolution. As an example, using a stride of

After each convolution-with-bias map, a non-linear operation is
performed, with the most common one being the Rectified Linear
Unit (ReLU) [Nair and Hinton 2010]:
σReLU (x) = max (0, x) .

(3)

Our model also uses the Sigmoid operation for the final layer tohave an output in the range [0, 1]:
σSigmoid (x) =1.
1 + e−x(4)

The weights of an ANN are learned using backpropagation [Rumelhart et al. 1986] in which given the errorof a network, the partial derivative of each weight with respect tothe error is computed and used to update the weight by gradientdescent. The error of the network is determined by the loss function and the resulting optimization is highly non-convex. Due tothe large amount of data used to train these models in combination
with a large amount of parameters or weights, stochastic variantsof gradient descent are used for optimization, in which each step of
the gradient descent algorithm is computed using only a subset ofthe data known as a batch.

3.2

Model

In contrast with the common CNN models that have fullyconnected layers, which do not allow processing images of arbitrary
resolution, we focus on exploiting the convolution operation, whichallows sharing parameters and processing images of arbitrary resolution. This is inspired by recent approaches [Long et al. 2015; Nohet al. 2015]; however, we opt for designing our architecture fromscratch instead of using a pre-trained existing model, as sketch images differ drastically from photographies. We design our modelwith sketch simplification in mind by having three parts: the firstpart acts as an encoder and spatially compresses the image, the second part processes and extracts the essential lines from the image,afterwards the third and last part acts as a decoder which converts
the small more simple representation to an grayscale image of thesame resolution as the input. This is all done using convolutions.
The down- and up-convolution architecture may seem similar toa simple filter banks. However, it is important to realize that the
number of channels is much larger where resolution is lower, e.g.,1024 where the size is 1/8. This ensures that information that leads
to clean lines is carried through the low-resolution part; the network
is trained to choose which information to carry by the encoderdecoder architecture.
For our convolutional layers, we use padding to compensate for the
kernel size and ensure the output is the same size as the input whena stride of 1 is used, although the number of channels may change.
Instead of using pooling layers, we use convolutional layers withincreased stride to lower the resolution from the previous layer. In
order for the output of the model to be of the same dimension as themodel input, we rely on fractional strides to increase the resolution.
Our model is formed by convolutional layers with stride of 1 (flatconvolution), 2 (downsampling convolution or down-convolution),and 1/2 (upsampling convolution or up-convolution). An overview
of our model can be seen in Fig. 3.

strideoutput size

input
down-convolution
flat-convolution
flat-convolution

5×5
3×3
3×3

2×2
1×1
1×1

1×H ×W
48 × H/2 × W/2
128 × H/2 × W/2
128 × H/2 × W/2

down-convolution
flat-convolution
flat-convolution

3×3
3×3
3×3

2×2
1×1
1×1

256 × H/4 × W/4
256 × H/4 × W/4
256 × H/4 × W/4

down-convolution
flat-convolution
flat-convolution
flat-convolution
flat-convolution
flat-convolution
flat-convolution
flat-convolution

3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3

2×2
1×1
1×1
1×1
1×1
1×1
1×1
1×1

256 × H/8 × W/8
512 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
512 × H/8 × W/8
256 × H/8 × W/8

up-convolution
flat-convolution
flat-convolution

4×4
3×3
3×3

1/2 × 1/2

256 × H/4 × W/4
256 × H/4 × W/4
128 × H/4 × W/4

up-convolution
flat-convolution
flat-convolution

4×4
3×3
3×3

1/2 × 1/2

up-convolution
flat-convolution
flat-convolution

4×4
3×3
3×3

1/2 × 1/2

1×1
1×1
1×1
1×1
1×1
1×1

128 × H/2 × W/2
128 × H/2 × W/2
48 × H/2 × W/2
48 × H × W
24 × H × W
1×H ×W

Target image

Loss map

Iteration 50

Iteration 100

Iteration 200

w/ loss map

type kernel size

Input image

w/o loss map

Table 1: Sketch simplification Convolutional Neural Network architecture. After each convolutional layer, except the last one, there
is a rectified linear unit. In the case of the last convolutional layer,
there is a Sigmoid layer instead to normalize the output to the [0, 1]
range. We pad all convolutional layers with zeros such that the output size is the same as the input size when using a stride of 1, i.e.,
2 pixel padding for 5 × 5 kernels and 1 pixel padding for 3 × 3
kernels. All output sizes reference the original image width W andheight H, as the model can process images of any resolution.

Figure 5: Visualization of the optimization process with and without the loss map. The main purpose of the loss map is to decrease
the importance of thick lines when training to speed-up the learning
process. If we train the model using the single image without theloss map, the optimization focuses on the thicker lines while ignoring the other thinner lines, as we can see in the top row. However,when we add the loss map, a balance is struck between the thickand the thin lines as shown in the bottom row. We can see how theeyebrows that have very thick lines are detected and modulated tohave a weaker loss. We use values of α = 6, β = −2, dh = 2, andbh = 10 for computing the loss map.
weighted mean square error criterion

The basic building block of our model is a convolutional layer(Eq. (2)) followed by a rectified linear unit layer (Eq. (3)). The lastlayer is special in that it is followed by a Sigmoid unit layer (Eq. (4))
in order to output a grayscale image. We downsample the model
three times using convolutional layers with a stride of 2 (downconvolution) and upsample three times using convolutional layerswith a stride of 1/2 (up-convolution). This fully-convolutional approach allows our model to work with any resolution and aspectratio in contrast to the standard CNN models with fully-connected
layers that require fixed input sizes.
We reduce the number of parameters in the full model by relying
primarily on 3 × 3 convolution kernels except for the first convolutional layer, which uses a 5 × 5 kernel, and the upsampling layers,which use 4 × 4 kernels. The reasoning behind this is that a 5 × 5
convolution can be approximated by two consecutive 3 × 3 convolutions with only 18/25 = 72% the amount of parameters. Furthermore, using two 3 × 3 convolutions allow better approximation of
non-linearities [Simonyan and Zisserman 2015]. However, whenupsampling, a 4 × 4 kernel is used instead of a 3 × 3 kernel sothat the output size is exactly twice the input size. The full detailsof our architecture can be seen in Table 1.

3.3

Model Loss

We train the model using training pairs of rough and simplified
sketches as input and target, respectively. As a loss, we use thel(Y, Y ∗ , M ) = kM

(Y − Y ∗ ) k2FRO ,(5)∗

where Y is the model output, Y is the target output, M is the loss
map,
is the matrix element-wise multiplication or Haddamard
product, and k · kFRO is the Frobenius norm. Note that a perfectmodel (Y = Y ∗ ) will have a loss of 0 regardless of the loss map
M chosen.
We experimentally tested various loss maps and found that, while
they do not change the final performance substantially, the one we
describe here can speed-up the learning. We chose a loss map that
reduces the loss on the thicker lines, in order to avoid having themodel focus on the thicker lines and forego the thinner lines. We
construct our loss maps by looking at histograms around each pixelin the ground truth (target) label. The loss map is defined as:1
if I(u, v) = 1
M (u, v) =min (α exp (−H(I, u, v)) + β, 1) else(6)
where H(I, u, v) is the value of the bin of the local normalized
histogram in which the pixel I(u, v) falls into. The histogram isconstructed using all pixels within dh pixels from the center using
bh bins.
An example of the optimization with and without the loss map canbe seen in Fig. 5. We can see how the loss map attenuates theeyebrows of the figure in the sketch to allow the model to learn
all parts of the image in a more equal fashion. Notice how, afterInput Image

Initialization

Iteration 100

Input Image
Output Image
Vectorization
Figure 7: Effect of vectorization on the output of our model. We
vectorize our model using automatic publically available tools withdefault parameters. As the output of our model is a clean simplified
image, such a simple approach yields excellent results. The vectorization process consists of a high-pass filter followed by a binary
thresholding. Afterwards polygons are fitted to the binary image
and converted to Bézier curves. An example result of vectorizationis shown on the right.

Iteration 1k

Iteration 10k Iteration 100k Iteration 500k Target Image
Figure 6: Visualization of the training of the model. It is initialized randomly and as training proceeds we can see how the figure
becomes clear and polished. The model initially focuses on joining
the lines into a single blurry line (Iteration 1000) and then progressively learns to refine the simplified line until it converges.

the square of the gradients and the square of the updates, which areused to determine the learning rate. An update of the parameters ofthe model θ then becomes,
θt+1 = θt + ∆θt = θt −200 iterations, the output lines in the model optimized with the loss
map can be clearly seen, in comparison with the case when it is not
used, where the output is still a set of blurry blobs.

3.4

Learning

One of the main recent innovations that have allowed the trainingof such deep models as the one we present from scratch are batchnormalization layers [Ioffe and Szegedy 2015]. They consist of
simply keeping a running mean and standard deviation of the input
data, and using them to normalize the input data. The output ofthese layers roughly has a mean of 0 and a standard deviation of
1. The running mean is only updated during training and is kept
fixed during evaluation. Batch normalization layers also have two
additional learnable parameters that serve to scale the normalized
output and add a bias:
x−µyBN (x) = √γ+η ,s2 + 

(7)

where µ is the running mean, s is the running standard deviation, 
is a constant for numerical stability, and γ and η are learnable parameters. We use these layers after all convolutional layers except
for the last one during training. Once the model is trained, these additional layers can be folded into the previous convolutional layer
to not add any overhead during inference. This is done by simply
reducing Eq. (7) to a linear transformation (note that everything except x is constant during evaluation), and merging this linear transformation with the linear transformation of the preceding convolutional layer. Thatis, the weightsof the convolutional layer getppmultiplied by γ/ s2 + and γµ/ s2 + − η is subtracted from thebias. Without these temporary layers, learning is not possible in areasonable amount of time.
For learning the weights of the models, we rely on the ADADELTA
algorithm [Zeiler 2012]. The main advantage of this approach isthat it does not require explicitly setting a learning rate, which isa non-trivial task. ADADELTA has been shown to generally converge to similar solutions as other algorithms, however, it will takea longer time to converge in comparison with optimally tuned learning rates. For training a sketch model, we tried various other optimizers and found that the result did not change significantly, whileother optimizers have the added complexity of choosing a learning
rate scheduler. ADADELTA consists of keeping a running mean ofRMS[∆θ]t−1δθt ,RMS[δθ]t(8)

where ∆θt is the parameter update, and δθt is the gradient of theparameters given the loss for a given iteration t. The update is doneby computing the Root Mean Square (RMS) of the running averages. Note that this approach automatically adjusts the learning
rate independently for all the different weights of the model.
We perform extensive data augmentation in order to combat overfitting of the model and improve the generalization. We train withconstant size 424×424 image patches extracted randomly from thefull image. We first extend the dataset by downscaling it by 7/6,
8/6, 9/6, 10/6, 11/6, 12/6, 13/6, and 14/6. Note that we do not use
the downscaled images if they are smaller than the training image
patch size. This results in roughly nine times the original amount ofimage pairs, although they are heavily correlated. We then threshold the simplified sketch images so that all pixels, which are in the[0, 1] range, with a value below 0.9 are set to 0. This normalization is critical for learning as all output images will have similar
tones. The resulting images are randomly rotated in the range of[−180, 180] degrees and also randomly flipped horizontally. Whentraining, we sample larger images more frequently based on thenumber of pixels in comparison with smaller images to compensate
that smaller images contribute more to the learning when extracting patches. Thus, patches from a 1024 × 1024 image will be fourtimes more likely to appear than patches from a 512 × 512 image.
Furthermore, with a probability of 10%, we change the input image
to be the same as the target image, i.e., we try to teach the modelthat clean images should not be modified. Training is done until theconvergence of the loss.

3.5

Vectorization

We employ simple techniques to vectorize our model output as it
is already a clean simplified line drawing in order for the result tobe directly usable by graphical artists. We automate the approachby performing a simple high-pass filter and thresholding using the
publicly available Potrace software [Selinger 2003] with default parameters. We show the result of vectorizing the output of our modelin Fig. 7. Note that this vectorization is, like our model, fully automated and requires no user intervention.

Sketch dataset

Extracted patches

・・・

・・・

Figure 8: Examples from our sketch simplification dataset used for training our model. The left of each pair shows the rough sketch while
the right shows the corresponding simplified sketch. We use the rough sketches as the input of our model and the simplified sketches as thetarget output when training our model. Note that these patches are randomly extracted during training and not fixed.

Input
s=1
s = 0.5
s = 0.25
Figure 9: Sketch simplification on scaled versions of the input image. The scaling done with respect to the input image on the left isdenoted with s, where s = 1 denotes no scaling. By down-scaling
the input image, it is possible to obtain more simplified sketches.

3.6

Controlling Simplicity by Scaling

While our approach is fully automatic and requires no userintervention, it is possible to tweak the results in various ways.
The most straight-forward way is to scale the input image. Downscaling the input images will result in more simple output images.
On the other hand, up-scaling the images will result in more conservation of fine details. By changing the amount of scaling, it ispossible for the user to control the degree of simplification of the algorithm. An example of the effect of scaling can be seen in Fig. 9.

(a) Direct(b) Inverse
Figure 10: Comparison of direct and inverse dataset construction
approaches. For both cases, the input image is shown in originalgrayscale, while the target image is shown overlaid in red. If we
attempt to create rough images and their simplifications in a directway, we get results such as the one shown on the left. As we canclearly see, even when aligned, the artist has taken various liberties to change different parts of the original sketch. If we attempt touse this for training, our model will not be able to learn this mapping. However, if we ask the artist to once again create a roughsketch based on the clean sketch obtained in the direct approach,
we get the result shown on the right. Notice how the input andtarget images are very well aligned. We call this dataset construction approach the inverse dataset construction approach, as we are
generating input images from target images. Data created with thisapproach is suitable for training deep neural networks.

4.1

4Inverse Dataset Construction

Rough Sketch Dataset

To teach our model to simplify sketches, we build a dataset using
the inverse dataset construction which consists of creating roughsketches from clean sketches and results critical to be able to train asuccessful sketch simplification model. Our dataset is formed by 68
pairs of training images drawn by 5 different artists. These images
consist of pairs of rough and simplified images and have different
resolutions with an average of 1280.0×1662.7 pixels. The smallest
image is 630 × 630 pixels and the largest is 2416 × 3219 pixels.
Some examples from the dataset and patches used for learning ourmodel can be seen in Fig. 8.

Dataset quality and quantity is critical for the performance of Deep
Convolutional Neural Networks such as the one proposed in thiswork. We found that the standard approach, which we denote as
direct dataset construction, of asking artists to draw a rough sketch
and then produce a clean version of the sketch ended up with a
lot of changes in the figure, i.e., output lines are greatly changed
with respect to their input lines, or new lines are added in the output. This results in very noisy training data that does not perform
well. In order to avoid this issue, we found that the best approachis the inverse dataset construction approach, that is, given a clean
simplified sketch drawing, the artist is asked to make a rough version of that sketch. While this does result in additional overhead

Table 2: Results of our user study comparing our approach withcommercial vectorization software. We processed 15 images with
our model, Potrace, and Adobe Live Trace. For Potrace and Adobe
Live Trace, we manually set the threshold for each image to obtain
the best results, while our approach is fully automatic. We show theabsolute score on the scale of 1 to 5 for each model and relative
comparisons, i.e., which model is better in the last three rows. We
can see that our approach significantly outperforms the vectorization approaches.
(a) input(b) tone(c) slur(b) noise
Figure 11: We further augment the dataset by changing the toneof the input image (b), slurring the input image (c), and adding
random noise (d). By using these additional training images, we
can make our model generalize better to other images.
to the artist, the quality of the dataset is significantly superior andtraining using this data results in much better models. An exampleof the difference between the traditional direct and inverse dataset
construction approach can be seen in Fig. 10. We use this approachin the creation of the 68 pairs of training images.

4.2

Data Augmentation

Due to the relatively low number of training images and the highdiversity of rough sketches found in the wild, we further augment
the dataset to have four times the images. We employ Adobe Photoshop to perform this data augmentation and perform three augmentations: tone change, image slur, and noise. For an example ofthese augmentations, refer to Fig. 11.
Tone change is done by using the Auto Tone tool with default parameters. This automatically sets the exposure, contrast, highlights,
shadows, whites and blacks of the image as shown in Fig. 11-(b). It
requires no parameters and is done in a fully automated approach.
Image slur is done by using the Fragment tool with default parameters. This blurs the image, duplicates it, and puts the duplicates
together with an offset. The resulting image has a more rough,although somewhat blurry, appearance in comparison with the raw
input. This can be seen in Fig. 11-(c).
Noise is done by the Noise-Uniform tool with default parameters.
This adds noise to all the pixels in the image using a uniform distribution. An example can be seen in Fig. 11-(d). This gives a
result that is similar to that of low-quality digital scans of paperand-pencil drawings and helps our model be robust to those.
By manually augmenting all the training images, we obtain a training dataset with four times more images than the original, which
improves the quality of the results.

Ours

Live Trace

Potrace

Score

4.53

2.94

2.80

vs Oursvs Live Trace
vs Potrace

97.5%
97.2%

2.5%
69.7%

2.8%
30.3%
-

draft the sketch with pencil-and-paper. Afterwards, the sketch isscanned and vectorized manually using a digital tablet. Our approach intends to replace this manual step and allow the sketch to
be directly imported as a vector image which the artist can thenmodify and colorize. In order to test our model, we directly input
rough sketches drawn with pencil and visualized the vector image
results. We evaluated on various rough sketches obtained from different artists in Fig. 12. Note that we did not perform any sort ofpreprocessing on the input images. We directly input them into ourmodel and performed simple vectorization on the result to obtain
vector images as output.
We can see that, despite the complexity and differences between thevarious images, our model in general is able to perform accurate andmeaningful line simplifications. We note that other existing sketch
simplification approaches require vector inputs and vectorization
approaches are unable to handle such complicated rough sketches
as the ones we consider in this work.

5.2 Comparison with the State of the Art
We compare against the state of the art [Liu et al. 2015] in sketch
simplifications on several images. However, note that the state of
the art requires vector images as input, while our approach does not.
For the purpose of evaluation, we fed a vector image to [Liu et al.
2015] and its rasterized version to our model. The comparison canbe seen in Fig. 13. We can see that, in general, our performance ison par despite not being limited to vector image inputs. We further
note that, as these images were rasterized from the original vector
images and thus are fairly different from the images on which wetrain our model, i.e., they already have much cleaner dark lines incomparison to the dirty real sketches from Fig. 12.

We have performed extensive analysis of our model and showedthat it is robust and suitable for all types of rough sketches. For
all images, we first subtracted the mean gray value of the training
dataset. For computing the loss map, we used the values of α = 6,
β = −2, dh = 2, and bh = 10 as depicted in Fig. 5. We trained
our model for 600,000 iterations with a batch size of 6. This takes
roughly three weeks using a Nvidia TITAN X GPU. We use thesame model for all the experiments in the rest of the section.

We also performed comparisons against commercial vectorization
software which can process raster input images. In particular, we
compared with the Potrace [Selinger 2003] and Adobe Live Trace.
We used the default parameters except we manually set the threshold of both approaches to 0.9 in order to obtain the best results. We
show the results in Fig. 14. We can see that, due to the complexity of the images we evaluate on, vectorization approaches failed togive good results. This is especially visible on parts of the sketchthat have overlapping multiple lines. Our approach was able to elegantly fuse these lines into a single clean line, while performing
vectorization directly either conserves multiple lines or, in the casethey are faint, fails to conserve them at all.

5.1

5.3 User Study

5

Experimental Results and Discussion

From Pencil and Paper to Vector Images

While digital sketching has taken force with the appearance ofmany high-quality digital tablets, many artists still prefer to initially

We also performed a user study to evaluate our model. We compared our model against Potrace and Adobe Live Trace. We se-(a) Animals(b) Kimono(c) Matsuri(d) Masks(e) Book

Figure 12: Results of our approach on different pencil-and-paper images. Note the variety and the coarseness of the different sketches.
Despite the complexity, our approach is able to obtain reliable line simplifications. We note in particular how clean the result of (d) is despite
the very challenging dirty input image. All sketches come from three different artists.
lected 15 images for evaluation and processed them with all three
approaches. In the case of Potrace and Adobe Live Trace, we manually set the threshold for each image to obtain best results. Comparison was done in two formats: (a) comparing two processed images to see which is better, and (b) ranking a processed image on ascale of 1 to 5.
We used 19 users for both cases, 10 of whom had significant experience in sketch drawing. Results are shown in Table 2. We can see
that, when compared to the other approaches, our model was considered better in over 97% of the cases. Furthermore, in absoluteterms, our approach was ranked 4.53 on a scale of 1 to 5. We foundno significant differences between the naı̈ve and expert users.

5.4

Computation Time

Evaluation time depends heavily on the resolution of the input image. Our model can be run both on the GPU and CPU, althoughbest performance is obtained on the GPU, allowing for near realtime performance. In comparison, [Liu et al. 2015] take various
minutes depending on the number of strokes. We test for various

Table 3: Analysis of computation time for our model. We noticea significant speedup when using the GPU that drives computation
times to under a second even for large input images.
Image Size

PixelsCPU (s)GPU (s)Speedup

320 × 320
640 × 640
1024 × 1024

102,400
409,600
1,048,576

2.014
7.533
19.463

0.047
0.159
0.397

42.9×
47.4×
49.0×

square images of different sizes initialized randomly and show themean results for 100 evaluations in Fig. 3. For evaluation, we usean Intel Core i7-5960X CPU at 3.00 GHz with 8 cores and NVIDIA
GeForce TITAN X GPU. We note that using a GPU gives nearly
a 50× speedup. As we can see, our approach is suitable for realworld usage.

5.5

Limitations

The main limitation of our approach is that it has a strong dependency on the quality and quantity of the training data. However, we[Liu et al. 2015]

Input

BARTOLO , A., C AMILLERI , K. P., FABRI , S. G., B ORG , J. C.,
AND FARRUGIA , P. J. 2007. Scribbles to vectors: Preparation of scribble drawings for cad interpretation. In Eurographics
Workshop on Sketch-based Interfaces and Modeling, 123–130.
BAUDEL , T. 1994. A mark-based interaction paradigm for freehand drawing. In ACM Symposium on User Interface Software
and Technology, 185–192.

Ours

C HANG , H.-H., AND YAN , H. 1998. Vectorization of handdrawn image using piecewise cubic bézier curves fitting. Pattern
Recognition 31, 11, 1747 – 1755.

(a) Fairy(b) Mouse(c) Duck(d) CarC HEN , J., G UENNEBAUD , G., BARLA , P., AND G RANIER , X.
2013. Non-oriented mls gradient fields. Computer Graphics
Forum 32, 8, 98–109.

Figure 13: Comparison with the state of the art. Note that, while[Liu et al. 2015] uses fairly clean vector images as input, our model
directly uses raster images.

C OLE , F., D E C ARLO , D., F INKELSTEIN , A., K IN , K., M ORLEY,
K., AND S ANTELLA , A. 2006. Directing gaze in 3d models
with stylized focus. In Eurographics Conference on Rendering
Techniques, 377–387.

show that with a small dataset we are still able to generalize fairly
well to many different images. Given additional training data, it islikely that we would be able to obtain better performance and generalization. Additionally, while the inference of the proposed model
is very fast, the learning process is computationally very expensive and relies on high-end GPUs in order to finish in a reasonableamount of time.

D EUSSEN , O., AND S TROTHOTTE , T. 2000. Computer-generated
pen-and-ink illustration of trees. In Conference on Computer
Graphics and Interactive Techniques, 13–18.

6

Conclusions

We have presented a novel automated end-to-end system that takes
rough raster sketches and outputs high quality vectorized simplifications. Our model is based on stacked convolution operations for
efficiency, and is able to handle very challenging pencil-and-paper
scanned images from various sources. Furthermore, our proposed
fully-convolutional architecture is optimized for the simplification
task and can process images of any resolution. We also present
a novel dataset carefully designed for the task that, in combination with our learning method, can be used to teach our model tosimplify sketches. Our approach is fully automatic and requires nouser intervention. Our results show that our approach is able to outperform the state of the art in sketch simplification despite not sharing the severe limitations of only being able to process vector images while maintaining a computation time of under a second. We
also corroborate with a user study that processing images with ourmodel gives significantly better results in comparison with commercial vectorization software. We believe our proposed approach is an
important step towards being able to integrate sketch simplification
into artist’s everyday work flow.

7

Acknowledgements

This work was partially supported by JST CREST.

References
BAE , S.-H., BALAKRISHNAN , R., AND S INGH , K. 2008. Ilovesketch: As-natural-as-possible sketching system for creating 3d
curve models. In ACM Symposium on User Interface Software
and Technology, 151–160.
BARLA , P., T HOLLOT, J., AND S ILLION , F. X. 2005. Geometricclustering for line drawing simplification. In ACM SIGGRAPH
2005 Sketches.

D ONG , C., L OY, C. C., H E , K., AND TANG , X. 2016. Image
super-resolution using deep convolutional networks. PAMI 38,
2, 295–307.
D OSOVITSKIY, A., S PRINGENBERG , J. T., AND B ROX , T. 2015.
Learning to generate chairs with convolutional neural networks.
In CVPR.
F ISCHER , P., D OSOVITSKIY, A., I LG , E., H ÄUSSER , P., H AZIR BAS , C., G OLKOV, V., VAN DER S MAGT, P., C REMERS , D.,
AND B ROX , T. 2015. Flownet: Learning optical flow with convolutional networks.
F I ŠER , J., A SENTE , P., AND S ÝKORA , D. 2015. Shipshape: A
drawing beautification assistant. In Workshop on Sketch-Based
Interfaces and Modeling, 49–57.
F REEMAN , H. 1974. Computer processing of line-drawing images.
ACM Comput. Surv. 6, 1, 57–97.
F UKUSHIMA , K. 1988. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks 1,
2, 119–130.
G RABLI , S., D URAND , F., AND S ILLION , F. 2004. Density measure for line-drawing simplification. In Pacific Conference on
Computer Graphics and Applications, 2004, 309–318.
G RIMM , C., AND J OSHI , P. 2012. Just drawit: A 3d sketchingsystem. In nternational Symposium on Sketch-Based Interfaces
and Modeling, 121–130.
H ILAIRE , X., AND T OMBRE , K. 2006. Robust and accurate vectorization of line drawings. PAMI 28, 6, 890–904.
I GARASHI , T., M ATSUOKA , S., K AWACHIYA , S., AND TANAKA ,H. 1997. Interactive beautification: A technique for rapid geometric design. In ACM Symposium on User Interface Software
and Technology, 105–114.
I OFFE , S., AND S ZEGEDY, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate
shift. In ICML.
JANSSEN , R. D., AND VOSSEPOEL , A. M. 1997. Adaptive vectorization of line drawing images. Computer Vision and Image
Understanding 65, 1, 38 – 56.

(a) Input(b) Potrace(c) Adobe Live Trace(d) Ours
Figure 14: Comparison with commercial tools for vectorization. We used the default parameters when possible for all tools. However, as
Adobe Live Trace requires manually setting a threshold parameter, we set it to the best visual result value of 0.9. We also set the potrace
threshold to 0.9 otherwise most of the input image gets erased. In general, directly vectorizing the image gives poor results, not fully
simplifying the lines or erasing vital parts of the image. In contrast, our approach gave the most accurate simplification of the input image.
K RIZHEVSKY, A., S UTSKEVER , I., AND H INTON , G. E. 2012.
Imagenet classification with deep convolutional neural networks.
In NIPS.
L E C UN , Y., B OTTOU , L., B ENGIO , Y., AND H AFFNER , P. 1998.
Gradient-based learning applied to document recognition. Proceedings of the IEEE 86, 11, 2278–2324.
L INDLBAUER , D., H ALLER , M., H ANCOCK , M. S., S COTT,
S. D., AND S TUERZLINGER , W. 2013. Perceptual grouping:selection assistance for digital sketching. In International Conference on Interactive Tabletops and Surfaces, 51–60.
L IU , X., W ONG , T.-T., AND H ENG , P.-A. 2015. Closure-aware
sketch simplification. ACM Trans. Graph. 34, 6, 168:1–168:10.
L ONG , J., S HELHAMER , E., AND DARRELL , T. 2015. Fully convolutional networks for semantic segmentation. In CVPR.
NAIR , V., AND H INTON , G. E. 2010. Rectified linear units improve restricted boltzmann machines. In ICML, 807–814.
N OH , H., H ONG , S., AND H AN , B. 2015. Learning deconvolution
network for semantic segmentation. In ICCV.
N ORIS , G., H ORNUNG , A., S UMNER , R. W., S IMMONS , M.,
AND G ROSS , M. 2013. Topology-driven vectorization of cleanline drawings. ACM Trans. Graph. 32, 1, 4:1–4:11.
O RBAY, G., AND K ARA , L. 2011. Beautification of design
sketches using trainable stroke clustering and curve fitting. IEEE
Trans. on Visualization and Computer Graphics 17, 5, 694–708.

ROSIN , P. L. 1994. Grouping curved lines. In Machine Graphics
and Vision 7, 625–644.
RUMELHART, D., H INTON , G., AND W ILLIAMS , R. 1986. Learning representations by back-propagating errors. In Nature.
S ELINGER , P. 2003. Potrace: a polygon-based tracing algorithm. Potrace (online), http://potrace. sourceforge. net/potrace.
pdf (2009-07-01).
S HEN , W., WANG , X., WANG , Y., BAI , X., AND Z HANG , Z.
2015. Deepcontour: A deep convolutional feature learned bypositive-sharing loss for contour detection. In CVPR.
S HESH , A., AND C HEN , B. 2008. Efficient and dynamic simplification of line drawings. Computer Graphics Forum 27, 2,
537–545.
S IMONYAN , K., AND Z ISSERMAN , A. 2015. Very deep convolutional networks for large-scale image recognition. In ICLR.
S PRINGENBERG , J. T., D OSOVITSKIY, A., B ROX , T., AND R IED MILLER , M. A. 2015. Striving for simplicity: The all convolutional net. In ICLR Workshop Track.
W ILSON , B., AND M A , K.-L. 2004. Rendering complexity
in computer-generated pen-and-ink illustrations. In International Symposium on Non-photorealistic Animation and Rendering, 129–137.
Z EILER , M. D., AND F ERGUS , R. 2014. Visualizing and understanding convolutional networks. In ECCV.

P REIM , B., AND S TROTHOTTE , T. 1995. Tuning rendered linedrawings. In Winter School in Computer Graphics, 227–237.

Z EILER , M. D. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701.

P USCH , R., S AMAVATI , F., NASRI , A., AND W YVILL , B. 2007.
Improving the sketch-based interface. The Visual Computer 23,
9-11, 955–962.

Z HANG , T. Y., AND S UEN , C. Y. 1984. A fast parallel algorithm
for thinning digital patterns. Commun. ACM 27, 3, 236–239.