General Virtual Sketching Framework for Vector Line Art
HAORAN MO, Sun Yat-sen University, China
EDGAR SIMO-SERRA, Waseda University, JapanCHENGYING GAO∗ , Sun Yat-sen University, China
CHANGQING ZOU, Huawei Technologies Canada, Canada
RUOMEI WANG, Sun Yat-sen University, China
Clean line drawing image (1024px)

Output vector line drawing

Stroke order(dynamic window & longer strokes)

Region (i)

(ii)
(i)
…

…

…

(window scaling & sliding to undrawn area)…

(a) Vectorization
Rough sketch (384px)

Output vector line drawing…

Region (ii)…

(b) Drawing canvas for vectorization
Stroke order

Photograph (256px)

Output vector line drawing

Stroke order

Drawing order(c) Rough sketch simplification(d) Photograph to line drawing

Fig. 1. Given clean line drawings, rough sketches or photographs of arbitrary resolution as input, our framework generates the corresponding vector line
drawings directly. As shown in (b), the framework models a virtual pen surrounded by a dynamic window (red boxes), which moves while drawing the strokes.
It learns to move around by scaling the window and sliding to an undrawn area for restarting the drawing (bottom example; sliding trajectory in blue arrow).
With our proposed stroke regularization mechanism, the framework is able to enlarge the window and draw long strokes for simplicity (top example).
Vector line art plays an important role in graphic design, however, it istedious to manually create. We introduce a general framework to produce
line drawings from a wide variety of images, by learning a mapping fromraster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable
rasterization module allows for training with only supervised raster data.
We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules.
Furthermore, we develop a stroke regularization loss that encourages the∗ Corresponding author.

Authors’ addresses: Haoran Mo, Sun Yat-sen University, China, mohaor@mail2.sysu.
edu.cn; Edgar Simo-Serra, Waseda University, Japan, ess@waseda.jp; Chengying Gao,Sun Yat-sen University, China, mcsgcy@mail.sysu.edu.cn; Changqing Zou, Huawei
Technologies Canada, Canada, aaronzou1125@gmail.com; Ruomei Wang, Sun Yat-sen
University, China, isswrm@mail.sysu.edu.cn.
© 2021 Association for Computing Machinery.
This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The definitive Version of Record was published in ACM Transactions on
Graphics, https://doi.org/10.1145/nnnnnnn.nnnnnnn.

model to use fewer and longer strokes to simplify the resulting vector image.
Ablation studies and comparisons with existing methods corroborate theefficiency of our approach which is able to generate visually better resultsin less computation time, while generalizing better to a diversity of images
and applications.
CCS Concepts: • Computing methodologies → Parametric curve andsurface models; Neural networks.
Additional Key Words and Phrases: vector line art generation, virtual sketching, dynamic window mechanism, stroke regularization
ACM Reference Format:
Haoran Mo, Edgar Simo-Serra, Chengying Gao, Changqing Zou, and Ruomei
Wang. 2021. General Virtual Sketching Framework for Vector Line Art. ACM
Trans. Graph. 1, 1 (April 2021), 15 pages. https://doi.org/10.1145/nnnnnnn.
nnnnnnn1

INTRODUCTION

Vector images play a fundamental role in graphic design giventhat they can be rendered at arbitrary resolutions without loss of
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

2•

Mo et al.

information, and are widely used in engineering design [Egiazarian
et al. 2020], 2D animation [Su et al. 2018], and 3D printing [Liu et al.
2017]. Instead of modifying the pixels directly, vector graphics arebuilt using basic shapes that are described with few parameters, e.g.,control points of a bézier curve or vertices of a polygon. This allows
much more natural and flexible editing in comparison with raster
images, which are described by pixel values.
Line art is often represented using vector images, and in particularparametrized curves, which can be directly generated with illustration software. However, many line drawings are generated as raster
images, such as scanned paper drawings, and it is necessary to convert them to vector drawings. For clean line drawings, vectorization
approaches [Bessmeltsev and Solomon 2019; Favreau et al. 2016;
Noris et al. 2013; Stanko et al. 2020] can directly produce vector line
drawings. In most cases, it is not clean line drawings, but roughsketches that need to be cleaned up and vectorized. Image-to-imagetranslation algorithms [Isola et al. 2017; Li et al. 2019; Simo-Serra
et al. 2018a] are able to generate clean line raster drawings, although
they require post-processing to convert the resulting line drawing
to a vector image. We propose a framework to directly convert arbitrary input images to clean line drawings, which is applicable to adiversity of image types as shown in Fig. 1.
Our approach is based on a recurrent neural network that learns
a raster image to vector image mapping directly, while not requiring
vector images for training. We achieve this with a differentiable
rendering module, which is able to render line drawing raster images from a vector parametrization, and is amenable to integrationin an end-to-end training framework. To overcome the issue ofprocessing images of arbitrary resolution, which is a challenge formost learning-based vector graphics generation algorithms [Huang
et al. 2019; Kim et al. 2018; Zheng et al. 2019], we propose modelinga virtual pen using a dynamic window that updates its positionand size at every frame as shown in Fig. 1-(b). The model learns
to enlarge the window when moving the pen around and drawing
long strokes, while it shrinks the window when drawing fine details
without any direct supervision. Naive implementations of cropping
and pasting operations necessary for the window suffer either from
non-differentiability or quantization artifacts. We overcome bothissues by using an aligned cropping and differentiable pasting modulethat both avoid discretization and allow for gradient propagation.
Another important aspect of vector images is that while manydifferent vector images can be rendered to the same raster image, in
general, we are interested in the simplest vector representation, thatis, the one with the fewest parameters. For line drawings, this consists of representing a long stroke with a single parametrized curve
instead of multiple shorter curves. To integrate this concept into ourmodel, we propose a stroke regularization mechanism which encourages the model to use the minimum vector parameters necessary torepresent a line drawing.
We evaluate for our approach through comprehensive ablation
studies, and comparisons with the existing methods on line drawing
vectorization, rough sketch simplification and photograph to line
drawing. These experiments demonstrate that our approach is able

to generate visually pleasing results while taking less computation
time, and generalizes better to different types of images 1 .
The main contributions of this work are summarized as follows:(1) A general framework for vector line drawing generation thatworks with a wide variety of images, dependent exclusively
on raster training data.
(2) A dynamic window mechanism that allows processing imagesof arbitrary resolution and high complexity.
(3) Stroke regularization mechanism that controls the simplicityof the output vector images.
(4) In depth comparison with existing approaches in a diversityof tasks.

2 RELATED WORK
2.1 Vector Graphics Generation
There are two main lines of work on learning-based vector graphics
generation: data-driven or independent on vector training data.
A number of works on learning with vector training data havebeen proposed in recent years, e.g., sketch reconstruction [Das et al.
2020; Graves 2013; Ha and Eck 2018], and image-based drawing
generation [Egiazarian et al. 2020; Song et al. 2018]. While, in general,
it is more straightforward and easier to learn with direct vector
supervision, it is not always feasible to collect vector training data.
To avoid this issue, another line aims to get around the vector data.
They firstly transform the predicted vector parameters into rasterdrawing images, and then optimize the model at raster level. The
transformation is achieved by using an external black-box rendering
simulator [Ganin et al. 2018; Mellor et al. 2019], or a differentiable
rendering module [Huang et al. 2019; Li et al. 2020; Nakano 2019;
Zheng et al. 2019].
Among works that do not require training vector data, LearningTo-Paint [Huang et al. 2019] is the approach closest to our framework, albeit with some noticeable differences. First, we adopt a
continuous stroke representation instead of a discrete one for morenatural stroke continuousness, and we use a virtual pen which can
avoid the redundant drawing problem in Learning-To-Paint. Second,
Learning-To-Paint is limited to a small fixed image size, while ours
handles images of arbitrary resolution.

2.2

Vectorization

Vectorization approaches can be divided into two lines: optimizationbased and learning-based approaches. Optimization-based algorithms have been widely studied and are still under active development [Bessmeltsev and Solomon 2019; Favreau et al. 2016; Noris
et al. 2013; Stanko et al. 2020]. Due to the high computational complexity of the optimization process, these approaches take a longtime to generate the vector images. In contrast, our approach isfaster. Recently learning-based approaches have also been proposed
as an alternative. VectorNet [Kim et al. 2018] combines neural networks and optimization algorithms to segment the raster image into
a set of paths, and then uses the existing vectorization techniques,e.g., Potrace, to vectorize each path. Guo et al. [2019] use neuralnetworks to subdivide the lines and reconstruct the topology for
1 The source code can be found at https://github.com/MarkMoHR/virtual_sketching.

ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

General Virtual Sketching Framework for Vector Line Art

each junction. Strokes are then traced by curve least-square fitting
method. The learning of both these works lies in the pixel level
rather than the vectorization stage. Thus, they require third-party
vectorization techniques in contrast to our work. Furthermore, bothlines of works are designed for clean line drawings and have difficulties in sketches with rough textures. On the other hand, ourmethod can generalize to a wide variety of images.

2.3

Line Generation from Other Domains

Line drawings can also be generated from images in other domains,such as rough sketches and photographs. For rough sketches (a.k.a.
sketch simplification or sketch clean-up), ClosureAware [Liu et al.
2015] and StrokeAggregator [Liu et al. 2018b] use vector images as
input and output the clean vector sketches. Models in [Simo-Serra
et al. 2018a,b, 2016; Xu et al. 2019] work with raster rough sketches
and output the clean ones in raster format. Line extraction fromphotographs includes widely-known edge detection techniques [Xieand Tu 2015], and GAN-based image translation methods such as
Photo-Sketching [Li et al. 2019]. All these approaches consist ofeither vector to vector or pixel to pixel mappings. In contrast, ourapproach is able to learn a pixel to vector mapping directly.

2.4

Image and Feature Sampling

In object detection and instance segmentation [Girshick 2015; Heet al. 2017], image or feature sampling based on the Region-ofInterests (RoIs) with floating point number position and size isfundamental. RoIPool proposed in [Girshick 2015] performs quantization for the RoIs twice to extract fixed-size feature maps. However,this introduces misalignment and breaks the gradients. RoIWarp
proposed in [Dai et al. 2016] and RoIAlign in Mask R-CNN [Heet al. 2017] try to resolve these problems with RoIAlign performing
better in practice. RoIAlign subdivides each RoI into spatial bins andperforms bilinear feature interpolation within each bin to computethe feature maps. This operation avoids quantization, and thus permits gradient propagation. We employ this approach in our aligned
cropping operation that requires accurate alignment and our differentiable pasting module that needs to preserve the gradients for the
position and window size.

3 LINE DRAWING GENERATION FRAMEWORK
3.1 Overview
Our approach is a general framework for vector line drawing generation given an arbitrary image as input. As illustrated in Fig. 2,
the framework is based on a recurrent neural network-based model
which predicts the drawing strokes step by step based from the
input image. It is designed with a dynamic window of a square thatmoves around while drawing the lines. This window, with size 𝑊
a cursor 𝑄 (i.e., the position), is able to move and scale after eachtime step. There are two advantages of using a dynamic window.
First, it allows us to model images directly at full resolution, i.e., theresolution of the input image, while avoiding the sharp increase intraining difficulty. Second, it enables our model to scale up to an arbitrary resolution even though the model is trained on low-resolution
images.

•3

As shown in Fig. 2-(a), the recurrent model consists of four main
phases at each time step 𝑡:(1) Aligned Cropping: given an image 𝐼 with size 𝑊𝐼 and a samesize canvas 𝐶𝑡 −1 as inputs, this module crops the patches according
to the current window (𝑄𝑡 −1,𝑊𝑡 −1 ), and resamples them to imagesof fixed size 𝑊𝑟 .
(2) Stroke Generation: taking the cropped patches as input, thestroke generator (Fig. 2-(b)) predicts the parameters 𝑎𝑡 of the next
stroke. The generator consists of a Convolutional Neural Network(CNN) encoder which models the image-level information, and a
Recurrent Neural Network (RNN) decoder which takes in the image
features and outputs the stroke parameters. The RNN decoder also
receives a latent vector from previous time step and passes along a
new latent vector to the next time step.
(3) Differentiable Rendering: a neural renderer is then employed
to approximate the stroke image 𝑆𝑡 based on the Bézier curve stroke
parameters 𝑞𝑡 derived from the predicted parameters 𝑎𝑡 while being
fully differentiable. This enables raster-level supervision during an
end-to-end training without the requirement of paired vector images
that are not trivial to collect.
(4) Differentiable Pasting: the rendered stroke image in a fixed rendering size 𝑊𝑟 is then pasted to the full-resolution canvas based on
cursor 𝑄𝑡 −1 and window size 𝑊𝑡 −1 . This pasting process is done in adifferentiable manner, which enables the gradients to be propagatedto the 𝑄𝑡 −1 and 𝑊𝑡 −1 .
Afterwards, a predicted indicator 𝑝𝑡 is used to decide whether
a stroke is drawn or not. Each pasted stroke image to be drawn isinserted into the canvas to form a full-resolution line drawing image
for the comparison with the target image.

3.2

Stroke Generation

3.2.1 Stroke Representation. In our framework, strokes are represented in a relative manner, i.e., continuous strokes, which issimilar to the stroke-3 format (Δ𝑥, Δ𝑦, 𝑝) in Sketch-RNN [Ha andEck 2018], where (Δ𝑥, Δ𝑦) ∈ [−1, +1] 2 is the offset of the next position and 𝑝 ∈ [0, 1] is the pen state to control whether to lift the pen.
Given that such representation is limited to straight lines and fixedline thickness, we augment it with an intermediate control point(𝑥𝑐 , 𝑦𝑐 ) ∈ [−1, +1] 2 to form a quadratic Bézier curve, and a width
factor 𝑤 ∈ [0, 1] to provide varying thickness. We also add a scaling
factor Δ𝑠 ∈ [0, 𝑘] (𝑘 > 1) into the stroke representation which controls the window size at each time step, allowing the model to learn
the best window size for different situations. In summary, the stroke
𝑎𝑡 at time step 𝑡 within a coordinate system [−1, +1] is formulated
as follows:𝑎𝑡 = (𝑥𝑐 , 𝑦𝑐 , Δ𝑥, Δ𝑦, 𝑤, Δ𝑠, 𝑝) 𝑡 ,𝑡 = 1, 2, ...,𝑇 .

(1)

A quadratic Bézier curve specified by three control points 𝑃 0 =(𝑥 0, 𝑦0 ), 𝑃1 = (𝑥 1, 𝑦1 ) and 𝑃 2 = (𝑥 2, 𝑦2 ) is formulated as:
𝐵(𝜏) = (1 − 𝜏) 2 𝑃0 + 2(1 − 𝜏)𝜏𝑃 1 + 𝜏 2 𝑃2,𝜏 ∈ [0, 1].

(2)

Following the parameter design in [Huang et al. 2019], we define thestroke parameters 𝑞𝑡 = (𝑥 0, 𝑦0, 𝑥 1, 𝑦1, 𝑥 2, 𝑦2, 𝑟 0, 𝑟 2 ) 𝑡 for a quadratic
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

4•

Mo et al.

𝑊𝐼

𝑊𝑡−1

ℎ𝑡−1

𝑊𝑟

𝑄𝑡−1

StrokeGeneration

Cropping

Input 𝐼 Canvas 𝐶𝑡−1

𝑎𝑡
Strokeparametersℎ𝑡𝑊𝑟

Bézier curveparameters

𝑞𝑡

Window 𝑄𝑡−1 , 𝑊𝑡−1

𝐶𝑡−1

Pasting

Rendering𝑝𝑡

Stroke 𝑆
image 𝑡Pen state

Canvas 𝐶𝑡(a) Pipeline at each time step
CNN Encoder

64
128

256

Hiddenstate ℎ𝑡−1
128

512

FC

…

32

7

256

𝑧𝑡

RNN
Decoder

FC

𝑆1

𝑆2

𝑆3

𝑆4

𝑆5

𝑆6

𝐶1

𝐶2

𝐶3

𝐶4

𝐶5

𝐶6

𝑎𝑡
…

ℎ𝑡
CoordConv
Conv + Instance Norm + ReLU(b) Architecture of stroke generator(c) Sequential strokes and canvas

Fig. 2. Our framework generates the parametrized strokes step by step in a recurrent manner. It uses a dynamic window (dashed red boxes) around a virtualpen to draw the strokes, and can both move and change the size of the window. (a) Four main modules at each time step: aligned cropping, stroke generation,differentiable rendering and differentiable pasting. (b) Architecture of the stroke generation module. (c) Structural strokes predicted at each step; movement
only is illustrated by blue arrows during which no stroke is drawn on the canvas.

Bézier curve based on 𝑎𝑡 :(𝑥 0, 𝑦0 ) 𝑡 = (0, 0), (𝑥 1, 𝑦1 ) 𝑡 = (𝑥𝑐 , 𝑦𝑐 ) 𝑡 , (𝑥 2, 𝑦2 ) 𝑡 = (Δ𝑥, Δ𝑦) 𝑡 ,(𝑟 0 ) 𝑡 = 𝑤𝑡 −1 𝑎𝑛𝑑 (𝑟 2 ) 𝑡 = 𝑤𝑡 .
(3)Here, the starting control point (𝑥 0, 𝑦0 ) should always be exactly at
the middle of the dynamic window, (𝑥 1, 𝑦1 ) and (𝑥 2, 𝑦2 ) are derived
from the stroke 𝑎𝑡 , and 𝑟 0 and 𝑟 2 denote the widths of 𝑃 0 and 𝑃 2 .
After pasting, the pen state 𝑝 is used to decide whether this strokeis drawn or not.
Relative Moving and Scaling. At each time step, the virtual pen
along with the dynamic window moves to the ending position of
the predicted stroke. If it is not necessary for a stroke to be done,e.g., sliding to a different part of the image, the model can choose
not to draw the stroke by setting the pen state to 𝑝𝑡 = 0. We call this“movement only”. Furthermore, the model can choose to change thesize of the dynamic window at each time step by setting Δ𝑠𝑡 , whichallows enlarging or shrinking the window size. Given the cursor
movement Δ𝑄𝑡 = (Δ𝑥, Δ𝑦) 𝑡 ∈ [−1, +1] 2 and scaling factor Δ𝑠𝑡 , wecan define the dynamic window update rule as:c𝑡 = Δ𝑄𝑡 × 𝑊𝑡 −1 /2 + 𝑄𝑡 −1,
𝑄c𝑡 )),
𝑄𝑡 = max(0, min(𝑊𝐼 , 𝑄

c𝑡 = Δ𝑠𝑡 × 𝑊𝑡 −1,
𝑊c𝑡 )),
𝑊𝑡 = max(𝑊𝑚𝑖𝑛 , min(𝑊𝐼 , 𝑊(4)
where 𝑊𝐼 is the size of the input image and 𝑊𝑚𝑖𝑛 is the pre-definedminimum value for the dynamic window size. Value clipping is used
to avoid potential out-of-bounds issues. In the experiments, we set
𝑊𝑚𝑖𝑛 = 32 × 32, the initial values 𝑊0 = 128 × 128, 𝑄 0 a random
position and 𝑘 = 2 for Δ𝑠. In fact, an endpoint is the junction of two
adjacent strokes belonging to two windows of different sizes, so its
width factor value used in the previous window should also update
to adapt to the next window at every time step. Please refer to the
supplemental materials for in-depth details.
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

Differentiable Pen State Binarizing. The pen state 𝑝 ∈ [0, 1] isa continuous value during training, but is expected to be a discrete binary value with 0 corresponding to movement only and1 corresponding to drawing a stroke. Direct discretization with anon-differentiable operation like argmax does not allow gradientpropagation. To address this problem, we adopt softargmax [Luvizon
et al. 2018] to approximate the argmax function while preserving
the gradients. This differentiable operation can be formulated as:
Õ 𝑒 𝛽𝑥𝑖
softargmax(𝑥) =(5)
Í 𝛽𝑥 𝑗 𝑖.
𝑗𝑒𝑖
Essentially, the softargmax operation outputs a continuous value
close to the discrete index value produced by argmax. In our case,softargmax pushes pen state 𝑝 ∈ [0, 1] closer to 0 or 1 when applied
to a vector [1 − 𝑝, 𝑝] that has index values 0 and 1 only. We set
𝛽 = 10 in our experiments.
3.2.2 Stroke Generator. Figure 2-(b) shows the architecture of thestroke generator, which consists of a CNN encoder and a RNN
decoder built with LSTM cells [Hochreiter and Schmidhuber 1997].
The CNN encoder takes the patches cropped from the input image
and canvas as input. Due to the dynamic window-based design,
the model tends to get into a situation where the cropped patch
has been fully drawn, as shown in the case in Fig. 7. To facilitate
the learning of moving to an undrawn region outside the window,
we resize the entire image and canvas to 𝑊𝑟 = 128 × 128, i.e., thesame size as the cropped patches, and feed them to the generator
as additional global guidance that tells the model where undrawn
strokes may be.
The CNN encoder employs CoordConv [Liu et al. 2018a] at thefirst layer, followed by convolution layers with instance normalization [Ulyanov et al. 2016] and ReLU activation function. A fully
connected (FC) layer is then employed to project the image features
to image embedding 𝑧𝑡 . The RNN decoder takes 𝑧𝑡 and the previous

General Virtual Sketching Framework for Vector Line Art
0

1

2

3

4

5

0

1

2

3

4

5
0

1

1

2

2

3

3

4

4

5

5

1

2

•

5

coordinate change, the bilinear image interpolation operation undercontinuous value space enables differentiation. We introduce detailsof the coordinate system change in the supplemental materials.

1(a) Misaligned Cropping

3.42(b) Aligned Cropping(c) Differentiable PastingFig. 3. Different types of cropping and pasting operations. (a) The image
represented in a dashed grid is cropped by a window in black solid lines witha cursor and window size that are not aligned with pixels. Naive cropping
operation with a quantized discrete window produces a misaligned patchin the light blue area. In pasting phase, this area is padded with pixels
in white to form the pasted frame. (b) The aligned cropping works on awindow that is not necessarily aligned with pixels. It relies on RoIAlignoperation [He et al. 2017], which resamples the cropped patch into a fixed
size 𝑊𝑟 (2 × 2 in this example). (c) The differentiable pasting essentially has
a similar working mechanism to the aligned cropping operation except forrequiring a coordinate system change. Here the black solid lines indicate
the cropping window, and the dashed grid indicates the rendered patch tobe cropped. The pasted frame within the black solid lines is amenable todifferentiation. Dots in the bins indicate the sampling points for RoIAlign.
We omit dots in the other bins for brevity.

hidden state ℎ𝑡 −1 as inputs and predicts the stroke parameters 𝑎𝑡 ,and a new hidden state ℎ𝑡 .

3.3

Aligned Cropping and Differentiable Pasting

The cropping and pasting modules play important roles in the dynamic window-based framework. As illustrated in Fig. 2-(a), at time
step 𝑡, the cropping and pasting are done based on cursor 𝑄𝑡 −1 andwindow size 𝑊𝑡 −1 both of which are floating values. Quantizing thefloating numbers (Fig. 3-(a)) is a straightforward way to performspatial cropping and pasting, but this could lead to two problems:
the window and the cropped or pasted patches may not be aligned,
and the gradients from the pasted canvas cannot be propagated to𝑄𝑡 −1 and 𝑊𝑡 −1 .
To address the misalignment problem in the cropping phase, we
introduce an aligned cropping operator, which is able to cope withthe cursor and window size that are not necessarily aligned withpixels. This operation is based on the RoIAlign module proposedin [He et al. 2017]. As shown in Fig. 3-(b), this operation first subdivides the window into spatial bins based on the window size 𝑊𝑡 −1
and resampling size 𝑊𝑟 = 128 × 128. Then, inside each bin, several
sampling points are set and their values are computed by bilinear
image interpolation. The average of these values is set to the final
value of each bin finally. Afterwards, cropped patches in a fixed size
𝑊𝑟 are obtained without the need for quantization, and they arespatially aligned with the window.
In the pasting phase, to guarantee alignment and enable gradients propagation to the cursor and window size, we introduce a
differentiable pasting module. As shown in Fig. 3, like the aligned
cropping, the differentiable pasting operation is based on bilinear
image interpolation. The main different between both modules isthat the 2D interpolations are under different coordinate systems,
which requires a coordinate system change. After performing theDifferentiable Rendering

The differentiable renderer is to render the 2D stroke raster image
from the 1D stroke vector parameters, which facilitates an end-toend training with a raster-level loss and avoids the requirement ofvector training images. To enable gradients to be propagated fromthe rendered output to the stroke parameters, we follow a similar
approach to Learning-To-Paint [Huang et al. 2019] and use a neuralnetwork to approximate the stroke image 𝑆𝑡 given the quadratic
Bézier curve parameters 𝑞𝑡 mentioned in §3.2.1. The neural renderer
has a similar architecture as the one in Learning-To-Paint. In our
experiments, the rendering window size is set to 128 × 128, whichis equal to the fixed image size 𝑊𝑟 in the stroke generator. Pleaserefer to the supplemental materials for more details about the neuralrenderer.

4 TRAINING
4.1 Overall Loss Function
Our training loss function is made up of three components: (1) araster loss L𝑟𝑎𝑠 for visual supervision, (2) an out-of-bounds penalty
loss L𝑜𝑢𝑡 to avoid out-of-bounds issues of the stroke parameterswith relative moving and scaling, and (3) a stroke regularization
loss L𝑟𝑒𝑔 that can encourage the model to simplify the resulting
stroke vector images. The total loss is formulated as below:
L𝑡𝑜𝑡𝑎𝑙 = L𝑟𝑎𝑠 + 𝜆𝑜𝑢𝑡 L𝑜𝑢𝑡 + 𝜆𝑟𝑒𝑔 L𝑟𝑒𝑔 ,(6)where 𝜆𝑜𝑢𝑡 and 𝜆𝑟𝑒𝑔 are scalars.

4.2

Raster-level Supervision

The differentiable rendering module allows us to adopt a raster-levelloss for end-to-end training without the need for vector images. It
is straightforward to use an 𝐿1 or 𝐿2 loss to calculate the pixel-wisedifference between the target line drawing image and the rendered
output. However, our experiments in §5.5 show that they are notsuitable for our task because they focus largely on the local details
and this results in poor global completeness.
To this end, we seek for a raster loss which is able to guarantee
both details and completeness of line drawings. Inspired by thestudy of perceptual similarity [Zhang et al. 2018], we employ theperceptual difference [Johnson et al. 2016], a kind of structural loss,
as a raster-level loss. In particular, we use VGG-16 [Simonyan andZisserman 2015] as the perceptual model, and we fine-tune it ona sketch dataset QuickDraw [Ha and Eck 2018] to make it moresensitive to line drawings.
Given a rendered line drawing image 𝑦ˆ (the last canvas), a target
line drawing image 𝑦, and a perceptual network 𝜙, we define 𝜙 𝑗 (·)
as the activation map ∈ R𝐷 𝑗 ×𝐻 𝑗 ×𝑊 𝑗 of layer 𝑗. The perceptual lossof layer 𝑗 can be defined as:
1
𝑗L𝑝𝑒𝑟𝑐 =𝜙 𝑗 (𝑦)ˆ − 𝜙 𝑗 (𝑦) 1 .
(7)
𝐷 𝑗 × 𝐻 𝑗 × 𝑊𝑗
Loss Value Normalization. As a raster loss, we use a combinationof perceptual losses from a set of layers 𝐽 . However, as loss values
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

6•

Mo et al.
Drawing order

(a)

(b)
Input image

Output vector drawing

Stroke orderFig. 4. Example of redundancy and incompactness. (a) The redundant
strokes (shown in the black box) are overlapped with the drawn ones. (b) Orange dots indicate the endpoints of each stroke. The result is that a straightline is divided into many short segments.

from different layers can have different orders of magnitude, directly
summing the original loss values as in [Johnson et al. 2016] mightlead to an undesired imbalance of layers. Furthermore, a simple
weighted summation is not practical because it is hard to decide thebest weights. To address this problem, we normalize the loss value
for each layer. Practically, we divide the loss value of layer 𝑗 by themean loss calculated from all the previous training iterations, and𝑗then obtain the normalized perceptual loss L𝑝𝑒𝑟𝑐−𝑛𝑜𝑟𝑚 . Finally, ourraster loss is computed as:Õ𝑗L𝑟𝑎𝑠 =
L𝑝𝑒𝑟𝑐−𝑛𝑜𝑟𝑚 .
(8)
𝑗 ∈𝐽

4.3 Out-of-Bounds Penalty
The stroke offset (Δ𝑥, Δ𝑦) and scaling factor Δ𝑠 can be theoreticallylearnt from only the raster loss. However, we notice the out-ofbounds issue caused by the relative moving and scaling, as mentioned in §3.2.1 and Eq. (4), should also be penalized in order to better
teach the model to predict the relative values inside the boundary.
c𝑡 and window size 𝑊
c𝑡 afterGiven the original values of cursor 𝑄
relative moving and scaling, and the clipping values 𝑄𝑡 and 𝑊𝑡 in
Eq. (4), we then define the out-of-bounds penalty loss for moving
factors by directly penalizing the out-of-bounds distance as:𝑇𝑚𝑜𝑣𝑖𝑛𝑔

L𝑜𝑢𝑡

=

1Õ
c𝑡 .
𝑄𝑡 − 𝑄𝑇 𝑡 =1(9)

The penalty for the scaling factor is then formulated as the normalized outer distance to the upper (𝑊𝐼 ) and bottom (𝑊𝑚𝑖𝑛 ) bounds:c𝑡 − 𝑊𝐼 , 0)/𝑊𝐼 ,L𝑢𝑝 = max(𝑊c𝑡 , 0)/𝑊𝑚𝑖𝑛 ,L𝑏𝑜𝑡𝑡𝑜𝑚 = max(𝑊𝑚𝑖𝑛 − 𝑊
1𝑠𝑐𝑎𝑙𝑖𝑛𝑔
L𝑜𝑢𝑡
=
𝑇

𝑇
Õ

(L

𝑢𝑝

+L

𝑏𝑜𝑡𝑡𝑜𝑚

(10)

).

𝑡 =1

The total out-of-bounds penalty L𝑜𝑢𝑡 is the combination of losses
𝑚𝑜𝑣𝑖𝑛𝑔𝑠𝑐𝑎𝑙𝑖𝑛𝑔
for moving and scaling factors. That is, L𝑜𝑢𝑡 = L𝑜𝑢𝑡
+ L𝑜𝑢𝑡 .
We show the effectiveness of this loss in the supplemental materials.

ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

128px

176px

224px

278px

Fig. 5. Training examples for vectorization and rough sketch simplification.

4.4

Stroke Regularization Mechanism

Any raster image can be rendered by different, yet visually equivalent, vector images. While they may be visually equivalent, thevector images can have varying degrees of complexity. Given that
our approach is trained with a raster loss, we need some mechanism
to encourage the vector representation to be as simple as possible,
otherwise redundant strokes may appear as shown in Fig. 4-(a), or
long strokes may be represented by many shorter strokes as in Fig. 4(b). In graphic design, both redundancy and incompactness increase
the difficulty of editing and we desire the simplest representationof the vector images.
The higher the simplicity, the fewer vector parameters are necessary to represent a line drawing. To this end, we introduce a stroke
regularization mechanism by restricting the number of strokes thatare related to the pen state 𝑝𝑡 ∈ [0, 1] (1 for drawing and 0 for lifting). The stroke regularization term is formulated as the proportionof drawn strokes:
𝑇
1Õ
(11)
L𝑟𝑒𝑔 =𝑝𝑡 .
𝑇 𝑡 =1This term is differentiable and added to the total loss function L𝑡𝑜𝑡𝑎𝑙
in Eq. (6) during end-to-end training. When minimizing the total loss,
ideally the model will learn to use fewer strokes while producing
a better line drawing simultaneously. As a result, the redundantand incompact strokes can be avoided. As with most regularization
terms, weighting the stroke regularization term too strong can leadthe model to use an insufficient number of strokes and thus worse
results. We analyze the effect of this term in §5.6.

5

EXPERIMENTS

We evaluate our approach in a diversity of image-to-line drawing
tasks such as line drawing vectorization, rough sketch simplification,and photograph to line drawing to show the generalness of ourapproach.

5.1

Dataset and Implementation Details

Dataset. Our model is able to process images of any resolution due
to our dynamic window-based framework. For training efficiency,
we train the model with low-resolution images, and then evaluate
its performance on higher resolution images. For the evaluation ofGeneral Virtual Sketching Framework for Vector Line Art

vectorization and rough sketch simplification, we use the QuickDraw [Ha and Eck 2018] dataset that contains vector stroke data, andrender them as raster images of varying resolutions from 128 px to
278 px as shown in the top row of Fig. 5. In the case of rough sketch
simplification, we utilize the pencil art generation and rough augmentation techniques from [Simo-Serra et al. 2018a,b] to synthesize
the corresponding rough sketches (bottom row in Fig. 5) from theclean line drawings. We additionally evaluate for photograph to line
drawing task using the face image dataset CelebAMask-HQ [Leeet al. 2020], where we can generate the facial sketches from theannotated segmentation masks. The face images and sketches are
rendered at a resolution of 256 px.
We also collect separate test sets for quantitative evaluation. In thecase of vectorization and rough sketch simplification, we generate
examples from the test set of QuickDraw. Raster images are rendered
at four sizes: 128, 256, 384 and 512. Such test sets can be used to evaluate the generalization ability on higher resolutions. For photographto line drawing task, we use the test split in CelebAMask-HQ.
Training Details. For vectorization task, we compute perceptual
loss at layers relu1_2, relu2_2, relu3_3 and relu5_1 (short for∪(12, 22, 33, 51)) of the VGG-16 model. For rough sketch simplification and photograph to line drawing, we adopt ∪(22, 33, 51) and∪(22, 33, 42, 51), respectively. The loss weight 𝜆𝑜𝑢𝑡 in Eq. (6) is set
to 10. For the stroke regularization mechanism, we adopt a linearly
increasing loss weight 𝜆𝑟𝑒𝑔 instead of a constant value. We resize
the images using area interpolation when inputting the full image
to the CNN to provide global guidance. We train for 75𝑘/90𝑘/90𝑘
iterations for the three tasks, respectively, with a batch size 20.
Optimization is done with Adam [Kingma and Ba 2014] using an
initial learning rate 1e-4. The maximum number of time steps of therecurrent neural network during training is set to 48.
Testing Details. The testing is done step by step in a fully automatic manner, with the cropping phase, the stroke generation, theexplicit rasterization and the pasting phase performed sequentiallyat each step as in training. When evaluating on images of high resolutions and complexity, we are able to theoretically use an infinite
number of strokes because our model learns to lift the pen with the
pen state. To avoid the prediction of unnecessary pen lifting after
the model finishes drawing early, we design an early-stop strategy.
Specifically, we set 𝑁𝑟𝑜𝑢𝑛𝑑 rounds of drawing, each with a maximum number of strokes 𝑁𝑠𝑡𝑟𝑜𝑘𝑒 . In each round, the moving is done
automatically. When 𝑁𝑏𝑟𝑒𝑎𝑘 continuous pen states corresponding
to lifting the pen (𝑝 = 0) are predicted, we end the round. After eachround, we randomly move the pen to slide the window to a distantundrawn area, which is especially useful with high-resolution images. We do this by randomly moving the cursor to a position far
away from the drawn region. This strategy maximizes the completeness of the resulting vectorization, and manual intervention, i.e.,
random movement, is performed only when the model cannot draw
anymore inside a round.
We set 𝑁𝑏𝑟𝑒𝑎𝑘 = 12 for the three tasks. The values of 𝑁𝑟𝑜𝑢𝑛𝑑 =10/10/1 and 𝑁𝑠𝑡𝑟𝑜𝑘𝑒 = 500/96/100 are found to work well for vectorization, rough sketch simplification and photograph to line drawing, respectively. For the vectorization task, we also design a method
to stop the drawing early, i.e., before reaching the maximum round.

Target image•

Chamfer distance: 3.966
Perceptual score: 1.822

Chamfer distance: 43.734
Perceptual score: 0.966

Chamfer distance: 3.590
Perceptual score: 0.252

Output A

Output B

Output C

7

Fig. 6. Examples to show how the quantitative evaluation metrics work.
Chamfer distances (↓) in 10−5 and perceptual scores (↓) in 10−2 . For both
metrics, a lower value is better.

Please refer to the supplemental materials for more details of thismethod, as well as the datasets and the implementation.

5.2

Quantitative Evaluation Metrics

Our goal of quantitative evaluation is to measure the pixel-level
similarity between target line drawing image and rendered output.
This still remains an open problem as it is hard to define similarity
metrics which are fully consistent with human perception. Chamferdistance [Fan et al. 2017] adopted in [Yan et al. 2020] is able tomeasure sketch-to-sketch similarity effectively, however, we noticesome issues of this metrics. As highlighted in Fig. 6, output A has alower chamfer distance value than B, but the details in A are visuallyworse.
From a rough inspection, we notice chamfer distance is moresensitive to the completeness of the drawing, but not to fine-grained
details. However, fine-grained details play a fundamental role inline drawings. We notice the perceptual score derived from the perceptual loss in §4.2 is more sensitive to the details, and employ it asanother metric. As shown in Fig. 6, output A has a lower chamferdistance corresponding to higher completeness, but a higher perceptual score corresponding to worse details compared with B. Output
C is visually more complete than B, so both chamfer distance andperceptual score are lower.
Different from the perceptual loss during training (§4.2), there
is no history value to perform loss normalization during testing.
Therefore, we use only the middle layer relu3_3 in the VGG-16 to
calculate the perceptual score. For test sets with images of different
resolutions, we average the values.

5.3

Effectiveness of Moving and Scaling

Moving. Our model learns to move the virtual pen along withthe dynamic window around by breaking the continuous strokes
or sliding to an undrawn area with pen state 𝑝 = 0. Figure 7 shows
two representative examples. The top row demonstrates an easily
encountered case: the window moves to where no undrawn pixel
is inside. Next, it slides while not drawing from position A in theupper right direction to position B containing undrawn pixels andthen the model restarts drawing. Bottom row shows a number ofnecessary breakings among the isolated strokes.
Scaling. Despite the lack of ground-truth scaling values for directsupervision, the model does learn some common and meaningfulrules as shown in Fig. 8. Most of the time, the model uses small windows to draw fine details. However, when getting into the situation
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

8•

Mo et al.
Drawing order

B

A

Input image

Output vector drawing

Stroke orderMoving trajectory

Fig. 7. Results of moving. The last column shows the drawing order of thestrokes together with the trajectory of movement only that is not drawn in
the final output. The dashed red box represents a window without undrawn
pixel inside it. Black arrows indicate the moving direction.

the supplemental materials for the scaling value distributions andvisual results.
Figure 9 shows visual comparisons between differentiable andnon-differentiable pasting. We can see that the model with nondifferentiable pasting suffers from bad completion of line drawings,
and the stroke order shows the window fails to slide to undrawn pixels. Quantitative results in Table 1 are in line with the visual results,
showing a rapid deterioration in both metrics (“Non-diff.” in “Pasting” column), especially in chamfer distance which is more sensitive
to completeness. These results also indicate the misalignment, and inparticular, the gradient blocking caused by non-differentiable pasting prevent the model from learning to update the stroke position
well.

5.5
where the window does not contain undrawn pixels (A1), the model
attempts to enlarge the window to find the undrawn pixels (A1-A3),
and slides there with a long stride simultaneously (A3-A4). Afterwards, the model shrinks the window quickly to restart drawing(A5). These results imply that the model is capable of learning tochoose a reasonable window size to adapt to different situations.
We also compare our model with one that uses a fixed-size window. For a fair comparison, we use the same neural render withrendering size set to 128 px, and set the fixed size to 128 px. Quantitative results in Table 1 show that such a model (“Fixed” in “Window”
column) has worse performance for both metrics. We believe that
this is probably because the 128 px window size is too large forthis model to draw details well, after looking at the statistics ofwindow size for our model. Most windows used for drawing strokes
have sizes ranging from 30 px to 70 px at both low and high resolutions. It seems like smaller windows can draw shorter strokes andhelp to recover details. However, overusing small windows tends tocause worse completeness, given that we allow only a finite number of strokes during training. This also explains why our model
learns to use a larger window for movement only. Visual results anddistributions of window sizes are in the supplemental materials.

5.4

Effectiveness of Differentiable Pasting

As discussed in §3.3, misaligned cropping and non-differentiablepasting suffer from misalignment and non-differentiability. The
latter issue in pasting phase results in the loss of gradient withrespect to stroke position and scaling factor in the stroke parameters,
and is detrimental to end-to-end training. We thus mainly study theeffectiveness of differentiable pasting.
We compute statistics for the distributions of scaling factor values, and find that for both low and high resolutions, the model withdifferentiable pasting predicts scaling factors concentrated in theinterval [0.8, 1.2]. However, without differentiable pasting, there
appears to be an abnormal distribution with peaks around 0.5 and2.0. Visual results show the scaling values change alternately between ∼ 0.5 and ∼ 2.0, leading to severe jitter in the window size
rather than meaningful changes as in the differentiable-pastingbased model. The abnormal scaling factors are due to the failure inits learning without normal gradient propagation. Please refer to
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

Ablation Study of Raster-level Loss

Other Raster Losses. We first compare the adopted perceptual loss
with a pixel-wise difference loss, and in particular, the 𝐿1 distance(the 𝐿2 distance has similar performance). Figure 10 shows thatmodel with 𝐿1 loss draws only a small part of the sketches with
dense short strokes. This is probably due to the limited ability the
pixel-wise difference loss has when considering the overall structureof the drawings. Significantly worse chamfer distance in Table 1 also
indicates the failure in completeness. We also study adversarial loss
which is broadly used in image generation tasks with GAN [Goodfellow et al. 2014]. However, we did not observe any significant
improvement.
Different Perceptual Layer Combinations. We use both shallow(relu1_2 and relu2_2) and deep layers (relu3_3 and relu5_1) of
the VGG-16 model for the perceptual loss and evaluate their respective performance. Figure 10 shows that models with shallow layers
Perc.12_22 suffer from worse completeness, although they areable to draw details well. Models based on deep layers Perc.33_51
draw complete sketches but lack in fine details. These results reflect
the different strengths of the different layers. In particular, shallow
layers focus mainly on low-level information, and are thus able torecover the local details but have an issue of incomplete drawings.
On the contrary, deep layers primarily store high-level information,
so they benefit the global completeness but suffer from poor finedetails. Quantitative comparisons further confirm these, showing
a better perceptual score that is sensitive to details but a worse
chamfer distance that is sensitive to completeness with ∪(12, 22)22), incomparison to ∪(33, 51)
51). Our proposed approach is superior in both
metrics. This demonstrates that combining both shallow and deep
layers to form the perceptual loss can balance their performanceand have the advantages of both.
Loss Value Normalization. We notice from a rough observationthat loss values from deeper layers have higher orders of magnitude(i.e., larger loss values). Hence, deeper layers play a more importantrole in the perceptual loss without value normalization. Quantitativeevaluation in Table 1 shows that models without loss normalization
×” in “Loss Norm.” column) perform worse than ours on both(“×
metrics. These confirm that loss value normalization is necessary tobalance the abilities of different layers. Please refer to the supplemental materials for more results of the ablation studies.

General Virtual Sketching Framework for Vector Line Art

68

Input

A1

111

127

A2

75

A3

A4

39

A5

46

•

9

38

A6

A7

Fig. 8. Results of window scaling. Red boxes represent the windows. A1 to A7 are the sequential frames. Numbers on top right corner indicate the window size.
We can see in A1 to A3 how the model enlarges the dynamic window in order to find an undrawn area to move to it. A blue arrow highlights in A4 shows amovement that does not draw a line.
Table 1. Ablation studies. For fair comparisons, we change only one factor (in bold type) while remaining others the same as those of our model at last row.
Numbers in “Perceptual Layers” indicate the relu layers. Values of perceptual score are in e-2 and those of chamfer distance in e-3.
Raster Loss

Pasting

Window

Loss Norm.

Perceptual Layers

Perceptual score(↓)

Chamfer distance(↓)

𝐿1
Perceptual
Perceptual
Perceptual
Perceptual
Perceptual

Diff.
Non-diff.
Diff.
Diff.
Diff.
Diff.

Scalable
Scalable
Fixed
Scalable
Scalable
Scalable

X
X
×
X
X

∪(12, 22, 33, 51)∪(12, 22, 33, 51)∪(12, 22, 33, 51)∪(33, 51)∪(12, 22)

13.616
6.064
1.938
1.789
1.513
1.399

249.050
60.723
4.598
2.597
2.995
3.242

Perceptual

Diff.

ScalableX∪(12, 22, 33, 51)

1.053

1.577

Drawing order

Input image

Output (diff.)

Drawing order

Output (non-diff.) Stroke order (non-diff.)

Fig. 9. Comparisons between differentiable (“diff.”) and non-differentiable(“non-diff.”) pasting.
Input image (640px)

𝜆𝑟𝑒𝑔 = 0.0

𝜆𝑟𝑒𝑔 = 0.1

𝜆𝑟𝑒𝑔 = 0.2

Fig. 11. Removal of redundant strokes by modifying the stroke regularization term weight 𝜆𝑟𝑒𝑔 . Red strokes in the grayscale vector outputs represent
the redundant or overlap strokes.
Input image

L1

Perc.12_22

Perc.33_51

Ours

Stroke number: 82
Average length: 15.67

Stroke number: 66
Average length: 18.70

Stroke number: 54
Average length: 22.41

𝜆𝑟𝑒𝑔 = 0.0

𝜆𝑟𝑒𝑔 = 0.05

𝜆𝑟𝑒𝑔 = 0.2

Fig. 10. Comparisons of different raster losses. The numbers after “Perc.”
indicate the relu layer combination.
Input image

5.6

Stroke Regularization

Redundancy Removal. One function of the stroke regularization
mechanism is to avoid the redundant strokes because it forces themodel to use fewer strokes to draw while maintaining a similar
appearance. Figure 11 shows how a model without regularization(𝜆𝑟𝑒𝑔 = 0.0) produces a number of redundant strokes superimposed
onto the strokes already drawn. In contrast, redundant strokes canbe largely avoided in a model with regularization. For instance, only
4 overlapped strokes (blue boxes) are produced in the model withweight 𝜆𝑟𝑒𝑔 = 0.1 and no overlap with a larger weight 𝜆𝑟𝑒𝑔 = 0.2.

Fig. 12. Improving stroke compactness with different stroke regulation
weights 𝜆𝑟𝑒𝑔 . Orange dots represent the endpoints of each stroke.

Compactness Improvement. Another function of the stroke regularization mechanism is to improve the compactness. Figure 12
shows as the stroke regularization weight 𝜆𝑟𝑒𝑔 increases, the modeldraws the sketches with fewer but longer strokes while maintaining
reasonably high fidelity. For instance, the leftmost part (blue boxes)of the bus consists of 4, 3 and 2 strokes for weights 0.0, 0.05 and 0.2,
respectively, resulting in a more compact representation.
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

10•

Mo et al.

Perceptual score (↓)

Chamfer distance (↓)

Average stroke number

Average stroke length
45.6

70.9
60.3

15.9

19.3

2.043

2.691

1.051

1.081

0.631

0.447

0.473

0.0

0.02

0.1

128px

40.6
1.416

1.053

3.112

54.0
25.3

18.4

34.9

29.0
0.883

19.2

0.5

1.0

0.418

𝜆𝑟𝑒𝑔

0.2

256px

Fig. 13. Quantitative evaluation of the stroke regularization with different
weights 𝜆𝑟𝑒𝑔 . We rescale the values for better visualization of the trends.

Statistics in Fig. 13 also show a downward tendency in average
stroke number and an upward one in average stroke length as theweight 𝜆𝑟𝑒𝑔 increases. When 𝜆𝑟𝑒𝑔 = 0.1, the average stroke number
is about 76% of that in a model without stroke regularization, while
the former model still maintains a comparable performance in quantitative evaluation without significant degradation. This confirmsthat with stroke regularization mechanism, our model does learn touse fewer vector strokes necessary to represent a line drawing.
Parameter Sensibility. From Fig. 13, when adding stroke regularization with weight 𝜆𝑟𝑒𝑔 = 0.02, both perceptual score and chamferdistance have a slight drop, and the model uses fewer (85%) strokes
to draw. This indicates that a small amount of stroke regularization
can improve both fidelity and simplicity. However, as described in§4.4, it reduces the overall performance when imposing an excessive
constraint. Figure 13 shows that when 𝜆𝑟𝑒𝑔 > 0.2, the model uses
much fewer strokes to draw, but suffers a significant performancedegradation in quantitative measurements simultaneously. Pleaserefer to the supplemental materials for visual results.

5.7

Vectorization: Comparison with Existing Approaches

Evaluation Settings. We first compare with the learning-based
method Learning-To-Paint [Huang et al. 2019], which is closest to
our work. Since it works at a fixed resolution, we train different
models for different image sizes. We use the same hyperparameters
as the official implementation2 except for the number of strokes inan action bundle, which is set 1 for stroke-by-stroke prediction as
ours. Comparison is made only on QuickDraw sketches because theLearning-To-Paint training procedure does not seem to converge
on images of higher resolutions.
We then compare with two representative vectorization methods, Fidelity-vs-Simplicity [Favreau et al. 2016] and PolyVectorization [Bessmeltsev and Solomon 2019], on real clean line drawings ofdifferent high resolutions. For Fidelity-vs-Simplicity, we try two setsof parameters 3 as in [Bessmeltsev and Solomon 2019] and selectthe visually best results. For PolyVectorization, we use the default
parameters.
Qualitative Results. Comparison with Learning-To-Paint is shownin Fig. 14. At a low resolution (128 px), Learning-To-Paint is able todraw the sketch with a small stroke number (16). However, because

Input image

Learning-To-Paint

Learning-To-Paint(Stroke number= 16)(Stroke number=48)

Fig. 14. Comparison with Learning-To-Paint [Huang et al. 2019].

of the discrete strokes it employs, the stroke continuousness is visually worse than ours with continuous strokes. Moreover, there
appear redundant drawings due to the lack of a sign for lifting pen.
When applied with a larger number of strokes (48) or a higher resolution (256 px), it fails to reconstruct the sketches. This reflects its
limitation on higher resolutions and instability in modeling longer
sequences, likely due to using reinforcement learning.
Figure 15 shows the comparisons on clean line drawings. Fidelityvs-Simplicity easily produces connected strokes (green boxes), but
fails to draw isolated strokes (e.g., the eyebrows). The connected
strokes are caused by its region segmentation-based skeleton extraction algorithm, which tends to add a connected stroke between close
strokes for better subdividing of a large complex region. Furthermore, this algorithm is not sensitive to isolated strokes. PolyVectorization produces results with artifacts in regions with fine-grained
details and complex junctions (red boxes). The reason is that thismethod mainly disambiguates X- and T-junctions by tracing the
pixel orientations with frame field, but is not robust to more complex
junctions with sharp turns or fine details. In contrast, our model
works better on both completeness and details. With the proposed
dynamic window, our model learns to find and slide to the undrawn area, and is thus able to draw the isolated strokes. Unlike theoptimization-based methods above, which fail in situations where
the pre-defined curve parameterization principles do not work, ourmodel learns to recover the undrawn pixels as best as it can, so thatthe details can be guaranteed.
Computation Time. For all methods, we test the examples under the same environment on a Windows PC with an Intel i7-8700@ 3.2GHz CPU, 48GB RAM, and an NVIDIA GeForce RTX 2070
GPU. For comparison, we show the running time of our model withand without GPU. Table 2 shows that under the same environment
without GPU, our model is much faster than Fidelity-vs-Simplicity.
Compared with PolyVectorization, our approach takes comparable
computation time on low-resolution images while is faster on complicated high-resolution images. When running with a GPU, ourmethod is even faster.

2 https://github.com/megvii-research/ICCV2019-LearningToPaint

3 A default parameter set: maxNumOpenCurves=0, minLengthOpenCurves=30, minRe-

gionSize=7. Another manually selected set: maxNumOpenCurves=30, minLengthOpenCurves=5, minRegionSize=3. Both use a ‘fidelity-simplicity’ weight of 0.5.
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

Ours

5.8

Other Applications

5.8.1

Rough Sketch Simplification.

General Virtual Sketching Framework for Vector Line Art•

11

Drawing order

Input

Fidelity-vs-Simplicity[Favreau et al. 2016]

PolyVectorization[Bessmeltsev et al. 2019]

Our results (vector)

Stroke order

Puppy (640px)

Kitten (780px)

Dracolion (1024px)

Mouse (1024px)

Fig. 15. Comparisons with existing vectorization approaches on real clean line drawings. Our results are from model with stroke regularization of weight
𝜆𝑟𝑒𝑔 = 0.02. More examples can be seen in supplemental materials.

Evaluation Settings. We compare with Learning-To-Paint [Huang
et al. 2019] on the fixed-resolution dataset QuickDraw with synthetic
rough sketches. To the best of our knowledge, there is no existing
work on rough sketch (with textured background) simplification
which outputs vector lines directly. Therefore, we combine the pixellevel sketch simplification [Simo-Serra et al. 2018a] and a vectorization method as a baseline. A vectorization approach is also applied
to rough sketches directly as another baseline. Here PolyVectorization [Bessmeltsev and Solomon 2019] is employed because of its
better visual quality compared with Fidelity-vs-Simplicity [Favreau
et al. 2016]. We evaluate their performance on complex roughsketches using both rough sketches synthesized from real cleanline drawings through pencil art generation technique [Simo-Serra
et al. 2018a], and rough sketches in the wild from a recent benchmark dataset [Yan et al. 2020].
For the quantitative evaluation on [Yan et al. 2020], we follow the
proposed protocol and use the best chamfer distance score among
all input variants compared to all ground truth cleanings for eachrough sketch. In contrast with vector approaches such as Fidelity-vsSimplicity and PolyVectorization that produce fixed stroke thickness,
our approach is able to output lines of varying width. Following
the evaluation protocol for the raster results from [Simo-Serra et al.

ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

12•

Mo et al.

Table 2. Computation time for different vectorization methods on real line
drawings. ‘F-vs-S’ denotes Fidelity-vs-Simplicity [Favreau et al. 2016] and‘PolyVec’ PolyVectorization [Bessmeltsev and Solomon 2019].

Elephant
Puppy
Hippo
Penguin
KittenBanana Tree
Muten
Mouse
Dracolion
Sheriff

Res. (px)F-vs-S

PolyVec

Ours (CPU)Ours (GPU)

640
640
700
720
780
872
1024
1024
1024
1024

55s
70s
55s
96s
83s
97s
89s
89s
75s
69s

17s
16s
12s
8s
22s
22s
39s
61s
69s
47s

20s
20s
17s
11s
19s
21s
33s
37s
46s
47s

11s
10s
9s
6s
10s
12s
21s
23s
29s
29s

2018a], we use a morphological dilation operation for the groundtruth lines with a kernel size of 3 to match the thickness of ourrendered outputs in order to not have the varying widths penalize
the results.
Qualitative Results. On the QuickDraw dataset, Learning-To-Paint
does not seem to perform well at any resolution nor number ofstrokes, which is similar to the results in the second row in Fig. 14(see supplemental materials for the full results). This might be because Learning-To-Paint, which is originally designed for reconstruction tasks, is not suitable to domain translation.
On complex rough sketches, PolyVectorization is sensitive to thestroke intensity, as shown in Fig. 16. As strokes in synthetic roughsketches are lightly drawn, PolyVectorization tends to have low
completion, and on sketch from the wild, it has an inclination tomiss thin and light strokes. This is due to its built-in thresholding
operation, which attempts to keep useful pixels but often discards
important pixels incorrectly. The pixel-level sketch simplification
method [Simo-Serra et al. 2018a] outputs simplified sketches withsmooth lines, but there also appear some artifacts, e.g., the redundant parallel strokes (red boxes) resulting from the noise in roughsketches, and some missing vital parts (eye of Bird, forehead ofPenguin and sleeve of Hand). Subsequent vectorization is required
to convert the pixel-wise images to vector images, but such artifacts
are repeated inevitably in the vector outputs. In contrast, our model
is able to produce comparable simplified results of vector format ina single step given rough sketch images as input. Furthermore, itgets rid of the noise to a large extent and distinguishes vital lines
even though they are in light color.
Quantitative Results. We compute the best chamfer distance foreach rough image and obtain an average value of 0.001696. Figure 17 shows the comparison with other approaches, which indicates that our method obtains comparable performance to otheralgorithms. Our approach is worse than MasteringSketching [SimoSerra et al. 2018a], because MasteringSketching is trained on realrough sketches and thus tends to work better on most wild sketches
in the benchmark. Our model, which is trained on synthetic roughACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

sketches with textured background, works better than most vectorization approaches. We believe that training on a supervised datasetof real sketches could likely improve performance.
5.8.2

Photograph to Line Drawing.

Evaluation Settings. We use the test set of CelebAMask-HQ [Leeet al. 2020] dataset with images of 256 px resolution for evaluation. We use Learning-To-Paint [Huang et al. 2019] as one of thebaselines. Similar to rough sketch simplification, we combine pixellevel image translation method Photo-Sketching [Li et al. 2019]and PolyVectorization [Bessmeltsev and Solomon 2019] as anotherbaseline.
Qualitative Results. Figure 18 shows the comparisons between thebaseline methods and our approach. Learning-To-Paint fails to drawplausible sketches as in rough sketch simplification, which further
confirms that it has difficulty in the domain translation task. PhotoSketching has a good performance in the facial sketch generationtask, but it relies on additional vectorization techniques to obtain
vector results. In contrast, our approach generates comparable facial
sketches and more importantly, works with such domain translationand vectorization simultaneously.

6

LIMITATIONS AND DISCUSSION

Our dynamic window-based framework learns to slide to an undrawn area, and thus is applicable to images of an arbitrary resolution. However, it may still fail to reproduce all the lines in somehighly complicated cases. Figure 19 shows some strokes on the left
and bottom of the resulting vector line drawing are missing eventhough the maximum stroke number is set to a considerably large
number of 5,000 (for reference, around 1,200 strokes are drawn for
Dracolion in Fig. 15). This is because when the CNN encoder downscales the complicated image for a global guidance, it might result inthe vanishing of thin lines. Athough we use the random movement
method during testing as mentioned in §5.1, the missing strokes
can be too small to be noticed. Therefore, resizing full-size images
directly and random movement can be seen as feasible but not optimal solutions to detecting undrawn thin lines globally. To better
capture global information and not necessitate random movements,
training with higher-resolution images, alternative encoding methods (e.g., pyramid views) with the global guidance, or a different
global guidance could be considered to address this problem.
Another limitation is that it is still difficult for our framework to
generalize well on complex rough sketches or photographs and it
may produce artifacts in the results. As an example, in Fig. 16, thelines in the Penguin in our result are not as smooth as those from
Sketch Simplification, and in Fig. 18, the noses in our results areconnected with the eyebrows while they are clearly separated inthose from Photo-Sketching. The non-smoothness problem could be
addressed by using a curve refinement technique [Das et al. 2020] as
post-processing. In addition, the performances on both tasks could
be improved by combining the pixel-level sketch simplification or
image-to-sketch model and our approach in a single end-to-end
model, which may be a promising future direction.
Although the virtual pen-based approach is efficient when drawing short and long strokes, it can perform less than satisfactory inGeneral Virtual Sketching Framework for Vector Line Art•

13

Drawing order
PolyVectorization[Bessmeltsev et al. 2019]

Input

Sketch Simplification (pixel)+ PolyVectorization

Our results (vector)

Stroke order

Bird (384px)

Penguin (640px)

Hand (433px)

Fig. 16. Comparisons with existing approaches on rough sketch simplification. The Bird and Penguin are synthetic rough sketches, while the Hand isfrom a rough sketch benchmark dataset [Yan et al. 2020]. Small figures in the third column are the pixel-level outputs from the sketch simplification
method [Simo-Serra et al. 2018a].
Drawing order

0.007

Input

0.006

Learning-To-Paint Photo-Sketching (pixel)[Huang et al. 2019] + PolyVectorization Our results (vector)

Stroke order

0.005
0.004
0.003
0.002
0.001
0
DelaunayTriangulation MasteringSketching

PolyVector→StrokeAggregator StrokeAggregator

FidelitySimplicity

RealTimeInking

PolyVector

TopologyDriven→StrokeAggregator

TopologyDriven

Ours

Fig. 17. Quantitative evaluation with chamfer distance (↓) on the roughsketch benchmark [Yan et al. 2020].

Fig. 18. Comparisons with existing approaches on photograph to line drawing. Small figures at third column are from Photo-Sketching [Li et al. 2019].

ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

14•

Mo et al.
Drawing orderMovingtrajectory

Input image (1000px)

Our result (vector)

Fig. 19. Limitation of our approach on a highly complicated example. Someshort strokes in the output are missing (red boxes) even when the maximum
number of strokes is set to 5,000.

some types of junctions. For instance, looking at the stroke orderfor the Puppy in Fig. 15, there are some T-junctions (black boxes) inwhich horizontal curves are not drawn together (from left to rightor the opposite). The reason is that our model is not intended forrecovery of topology or meaningful drawing order while vectorizing line drawings. We think this could be an issue rather than alimitation of our approach, because there is no general consensus
in drawing order (e.g., how squares are drawn in east-Asia vs. thewest). Furthermore, in reality different people have various ways ofdrawing. Some artists prefer to draw the silhouette first and then fillin the inner details with shorter segments, while others draw partby part. While correct topology and specific drawing orders might
be beneficial for certain applications, post-processing, pre-definedprinciples as prior or constraint information can be incorporated toform a future extension of our approach.

ACKNOWLEDGMENTS
This work was supported by the Natural Science Foundation ofGuangdong Province, China (Grant No. 2019A1515011075) and the
National Key R&D Program of China (2018AAA0100300).

REFERENCES
Mikhail Bessmeltsev and Justin Solomon. 2019. Vectorization of line drawings via
polyvector fields. ACM Transactions on Graphics (TOG) 38, 1 (2019), 1–12.
Jifeng Dai, Kaiming He, and Jian Sun. 2016. Instance-aware Semantic Segmentation via
Multi-task Network Cascades. In CVPR.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. 2020.
BézierSketch: A generative model for scalable vector sketches. In The European
Conference on Computer Vision (ECCV).
Vage Egiazarian, Oleg Voynov, Alexey Artemov, Denis Volkhonskiy, Aleksandr Safin,
Maria Taktasheva, Denis Zorin, and Evgeny Burnaev. 2020. Deep Vectorization ofTechnical Drawings. arXiv preprint arXiv:2003.05471 (2020).
Haoqiang Fan, Hao Su, and Leonidas J Guibas. 2017. A point set generation network for
3d object reconstruction from a single image. In Proceedings of the IEEE conferenceon computer vision and pattern recognition. 605–613.
Jean-Dominique Favreau, Florent Lafarge, and Adrien Bousseau. 2016. Fidelity vs.
Simplicity: a Global Approach to Line Drawing Vectorization. ACM Transactions on
Graphics (SIGGRAPH Conference Proceedings) (2016).
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals.
2018. Synthesizing Programs for Images using Reinforced Adversarial Learning. In
ICML.
Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on
computer vision. 1440–1448.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In
Advances in neural information processing systems. 2672–2680.
Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv
preprint arXiv:1308.0850 (2013).
Yi Guo, Zhuming Zhang, Chu Han, Wenbo Hu, Chengze Li, and Tien-Tsin Wong. 2019.
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction.
In Computer Graphics Forum, Vol. 38. Wiley Online Library, 81–90.
ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.

David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In
International Conference on Learning Representations.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In
Proceedings of the IEEE international conference on computer vision. 2961–2969.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neuralcomputation 9, 8 (1997), 1735–1780.
Zhewei Huang, Wen Heng, and Shuchang Zhou. 2019. Learning to paint with modelbased deep reinforcement learning. In Proceedings of the IEEE International Conference on Computer Vision. 8709–8718.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-imagetranslation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for realtime style transfer and super-resolution. In European conference on computer vision.
Springer, 694–711.
Byungsoo Kim, Oliver Wang, A. Cengiz Öztireli, and Markus Gross. 2018. Semantic
Segmentation for Line Drawing Vectorization Using Neural Networks. Computer
Graphics Forum (Proc. Eurographics) 37, 2 (2018), 329–338.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980 (2014).
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020. MaskGAN: Towards
Diverse and Interactive Facial Image Manipulation. In IEEE Conference on Computer
Vision and Pattern Recognition (CVPR).
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. 2019. Photosketching: Inferring contour drawings from images. In 2019 IEEE Winter Conference
on Applications of Computer Vision (WACV). IEEE, 1403–1412.
Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on
Graphics (TOG) 39, 6 (2020), 1–15.
Chenxi Liu, Enrique Rosales, and Alla Sheffer. 2018b. Strokeaggregator: Consolidating
raw sketches into artist-intended curve drawings. ACM Transactions on Graphics(TOG) 37, 4 (2018), 1–15.
Lingjie Liu, Duygu Ceylan, Cheng Lin, Wenping Wang, and Niloy J Mitra. 2017. Imagebased reconstruction of wire art. ACM Transactions on Graphics (TOG) 36, 4 (2017),
1–11.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev,
and Jason Yosinski. 2018a. An Intriguing Failing of Convolutional Neural Networks
and the CoordConv Solution. In Advances in Neural Information Processing Systems.
Xueting Liu, Tien-Tsin Wong, and Pheng-Ann Heng. 2015. Closure-aware sketch
simplification. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1–10.
Diogo C Luvizon, David Picard, and Hedi Tabia. 2018. 2D/3D Pose Estimation and Action
Recognition Using Multitask Deep Learning. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition. 5137–5146.
John FJ Mellor, Eunbyung Park, Yaroslav Ganin, Igor Babuschkin, Tejas Kulkarni,
Dan Rosenbaum, Andy Ballard, Theophane Weber, Oriol Vinyals, and SM Eslami.
2019. Unsupervised Doodling and Painting with Improved SPIRAL. arXiv preprint
arXiv:1910.01007 (2019).
Reiichiro Nakano. 2019. Neural painters: A learned differentiable constraint for generating brushstroke paintings. arXiv preprint arXiv:1904.08410 (2019).
Gioacchino Noris, Alexander Hornung, Robert W Sumner, Maryann Simmons, andMarkus Gross. 2013. Topology-driven vectorization of clean line drawings. ACM
Transactions on Graphics (TOG) 32, 1 (2013), 1–11.
Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018a. Mastering sketching:adversarial augmentation for structured prediction. ACM Transactions on Graphics(TOG) 37, 1 (2018), 1–13.
Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018b. Real-time data-driven
interactive rough sketch inking. ACM Transactions on Graphics (TOG) 37, 4 (2018),
1–14.
Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning
to simplify: fully convolutional networks for rough sketch cleanup. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1–11.
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks forLarge-Scale Image Recognition. In International Conference on Learning Representations.
Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. 2018.
Learning to sketch with shortcut cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 801–810.
Tibor Stanko, Mikhail Bessmeltsev, David Bommes, and Adrien Bousseau. 2020. IntegerGrid Sketch Simplification and Vectorization. In Computer Graphics Forum, Vol. 39.
Wiley Online Library, 149–161.
Qingkun Su, Xue Bai, Hongbo Fu, Chiew-Lan Tai, and Jue Wang. 2018. Live sketch:
Video-driven dynamic deformation of static drawings. In Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems. 1–12.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization:
The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).

General Virtual Sketching Framework for Vector Line Art•

15

Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Proceedingsof the IEEE international conference on computer vision. 1395–1403.
Xuemiao Xu, Minshan Xie, Peiqi Miao, Wei Qu, Wenpeng Xiao, Huaidong Zhang,
Xueting Liu, and Tien-Tsin Wong. 2019. Perceptual-aware Sketch Simplification
Based on Integrated VGG Layers. IEEE Transactions on Visualization and Computer
Graphics (2019).
Chuan Yan, David Vanderhaeghe, and Yotam Gingold. 2020. A benchmark for roughsketch cleanup. ACM Transactions on Graphics (TOG) 39, 6 (2020), 1–14.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The
unreasonable effectiveness of deep features as a perceptual metric. In Proceedings ofthe IEEE conference on computer vision and pattern recognition. 586–595.
Ningyuan Zheng, Yifan Jiang, and Dingjiang Huang. 2019. StrokeNet: A Neural Painting
Environment. In International Conference on Learning Representations.

ACM Trans. Graph., Vol. 1, No. 1, Article . Publication date: April 2021.