Deep Sketch Vectorization via
Implicit Surface Extraction

Chuan Yan1, Yong Li2,1, Deepali Aneja3, Matthew Fisher3, Edgar Simo-Serra4, Yotam Gingold1
1.

2.

3.

4.

Hi everyone, my name is Chuan Yan and today I will introduce our work: Deep
Sketch Vectorization via Implicit Surface Extraction. This work is a collaboration
with Yong Li from south China university of technology, Deepali Aneja. Matthew
Fisher from Adobe, Edgar Simo-serra from Waseda university and Yotam Gingold
from George Mason university

1Why is extracting vector lines important?

Accurately extract the vector lines from raster sketches is a beneficial step formany aspects.
First, vector lines and junctions could preserve important information that isdifficult to extract from bitmaps only such as the artist’s drawing intent, the 2D or
3D shape .
Also, it is easy to only choose the target strokes and adjust them without
touching other neighbor strokes.
A lot of downstream sketch processing applications usually requires vector lines
as input.
At last, vector lines are necessary for many human vison abstraction andexpression researches such as sketch retrieval, sketch recognition, etc.

2

Why is extracting vector lines important?

Accurately extract the vector lines from raster sketches is a beneficial step formany aspects.
First, vector lines and junctions could preserve important information that isdifficult to extract from bitmaps only such as the artist’s drawing intent, the 2D or
3D shape .
Also, it is easy to only choose the target strokes and adjust them without
touching other neighbor strokes.
A lot of downstream sketch processing applications usually requires vector lines
as input.
At last, vector lines are necessary for many human vison abstraction andexpression researches such as sketch retrieval, sketch recognition, etc.

3

Challenges• Vectorization fidelity at dense stroke regions[Puhachov et al. 2021]

However, there are still challenges for existing sketch vectorization methods.
The frame field-based methods or fully end-to-end data driven based methods
are all struggle at extracting accurate vector paths on the sketch regions wherecontains dense strokes.

4Challenges• Vectorization fidelity at dense stroke regions[Puhachov et al. 2021][Bessmeltsev et al. 2019][Mo et al. 2021]

However, there are still challenges for existing sketch vectorization methods.
The frame field-based methods or the fully end-to-end data driven based
methods all struggle to extract accurate vector paths on the sketch regions
containing dense strokes.

5Challenges

384 × 51279s

• Vectorization fidelity at dense stroke regions• Running time
[Puhachov et al. 2021]

69s

42s

[Bessmeltsev et al. 2019][Mo et al. 2021]

And it will be difficult to further reduce the runtime if the raster sketches containcomplex structures with numerous junctions.
For example, for a 384 x 512 raster sketch, the current methods will take at least
more than 40 seconds.

6

Our Solution

79s• High vectorization fidelity• Fast running time
[Puhachov et al. 2021][Ours]

69s

42s[Bessmeltsev et al. 2019][Mo et al. 2021]

We have proposed a new sketch vectorization method that provides much higher
vectorization fidelity and faster execution time.

7

Our Solution

384 × 51279s

• High vectorization fidelity• Fast running time
[Puhachov et al. 2021]

4.5s

69s

42s

[Ours][Bessmeltsev et al. 2019][Mo et al. 2021]

We have proposed a new sketch vectorization method that provides much higher
vectorization fidelity and faster execution time.

8Our Solution

384 × 512

4.5s

• High vectorization fidelity• Low algorithm complexity• Supports interactive Refinement[Ours]

Additionally, our intermediate vector path representation naturally support
further interactive topology Refinement if necessary.

But, all results shown in this slides are fully
automatic unless stated otherwise

9Overview• Center line encoding• Line reconstruction• Post processing
Initial
Vectorization

Line R ec ons tructi on

Raster Sketch
Key Po in ts
D ista nce
Fie l dPre dic tion

Und erSampling
Map

Incorrectly Pred icted Top ology
A uto Top ology
Error De tectio n St ep 1

Cen ter L in e

Man ual Lab el
To polo gy Errors St ep 2 ( opt ional)

Refined
Vectorization

Final Vectorization

D ual Contour ing D ow nsa mpl ing & Line g rouping

Re fine dTopolog yThre shol dU ser
Edi tsLoca l
Ma x im um

We designed our vectorization framework into 3 stages: center line encoding,
vector line reconstruction and post Refinements

10Overview• Center line encoding: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Down Sampling & Line Grouping
Distance Field Prediction

Line Reconstruction

Raster Sketch
Key Points

Under-Sampling
Map

Center Line

Threshold
Local Maximum

At the first center line encoding stage, we trained a distance field prediction
network that extract information such as key point, under sampling map andcenter line from the given raster sketch.
All this information are encoded as Unsigned Distance Fields.

11Overview• Center line encoding: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Down Sampling & Line GroupingLine Reconstruction

Initial
Vectorization
Key Points

Under-Sampling
Map

Incorrectly Predicted Topology
Auto Topology E rror
Detection - Step 1

Center Line

Threshold

Manual Label Topology
Errors - Step 2 (optional)

User Edits

Local Maximum

Then we trained another network to reconstruct initial vector paths from theencoded center line which is predicted in our first stage.
After getting the initial vector lines, its junctions will be automatically refined
based on the predicted key point and under sampling maps.

12

Overview• Center line extraction: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Downsampling & Line Grouping
Dual Contouring Downsampling & Line grouping

Initial
Vectorization

Incorrectly Predicted Topology
Auto Topology E rror
Detection - Step 1

Refined
Vectorization

Final Vectorization

Manual Label Topology
Errors - Step 2 (optional)

RefinedTopology

User Edits

At last, to further refine the output.
we proposed a dual contouring down sampling method and a simple line
grouping method to turn vector segments into vector strokes with optimized filesize.
Now let’s dive into the technical details

13

Dual Contouring with Under-Sampled Map

Target Vector Sketch

Center Line

Under-Sampled Map

Edge Flags

Key Points

Edge Vertex

Reconstructed Vector Sketch

One of our core contribution is that we formulate the sketch vectorization as animplicit surface reconstruction task which is Dual Contouring in our method.
Therefore, we will briefly review its reconstruction logic in 2D first.

Given a target vector sketch, we sampled the canvas into a bunch of evenly
distributed grids.
For each grid, we compute the closest distance from the grid center to the vector
paths as an Unsigned Distance field.
Since this field implicitly encoded the vector path as a 2D surface, we can use
dual contouring to reconstruct it as vector segments based on this field.
To be more specific, we could infer the edge flags and edge vertex from it.
The edge flag records the intersection between the target vector sketch and gridedge.
And the edge vertex in each grid is set onto the vector sketch if the grid contains
single vector path.
Each grid could only have one edge vertex.

14

Then the reconstruction will be simply connecting any 2 neighbor vertices if theirshared grid edge has a flag.
However, this reconstruction could be inaccurate since our sampling rate in thevector space is not enough.

And it is theoretically impossible to reconstruct the high-valence

junctions with dual contouring. We termed those regions as
under sampled regions.
To solve this problem, we detect those under sampled regions and extract thejunction key points

14

Dual Contouring with Under-Sampled Map
Ground Truth for DistanceField Prediction Network
Target Vector Sketch≠

Center Line

Under-Sampled Map

Key Points

Ground Truth for Line
Reconstruction Network
Edge Flags

Edge Vertex

Reconstructed Vector Sketch

Then we can make correction by dropping all the line segments in the undersampled regions and rebuild the junction by connecting the key point to the all
the truncations.

To sum up, to get the reconstructed vector sketch, we need derive this 5
information which are our training ground truth from a given target vector sketch.
The upper 3 are the ground truth for our distance filed prediction network, thelower 2 is the ground truth for our line reconstruction network.
It is also worth noting that although the reconstructed vector sketch is not equal
to the target vector sketch, this design of intermediate vector representation
could greatly reduce the training difficulty and naturally support interactive
junction Refinement.

15

Distance Field Prediction• Dataset• 10,000 and 53,000 vector sketches from the Quick
Draw! and Creative Sketch [Ge et al. 2021]• 441,000 raster sketches in total(a) Basic (b) Calligraphy (b) Marker A(c) Marker B(d) Ink A(e) Ink B(f) Pencil

We then create our dataset with 63k vector sketches from public dataset thenrasterize them with 7 different brush styles. This ends up with 441k rastersketches as our training data. One group of the raster sketch examples is shown
below.

16

Distance Field Prediction• Dataset• Loss function• 10,000 and 53,000 vector sketches from the Quick• Masked L1 distance

Draw! and Creative Sketch [Ge et al. 2021]• 441,000 raster sketches in total(a) Basic (b) Calligraphy (b) Marker A(c) Marker B(d) Ink A(e) Ink B(f) Pencil

we formulated this distance field prediction as an image-to-image translation
task and trained a U-Net like network based on the created data.
We using the masked L1 distance as our training loss to predict 3 type of
unsigned distance fields that encoding the centerlines, under sampled maps
and key points, respectively.

17

Distance Field Prediction• Sub-Pixel Sampling• 2x size of prediction output and ground truth• Fewer under-sampled region• Higher precision

To reduce the number of under sampled regions and improve the smoothness of
the reconstructed curve, we doubled the resolution of the output unsigned
distance field which equivalent to double the sampling rate in the vector domain.
This will also lead the resolution increase for both corresponding edge flags andedge vertex.

18

Neural Dual Contouring & Topology Refinement• Network [Chen et al. 2022]• Multi-Resolution convolution branches
Input
UDF

Multi size gridconvolution

Domain Conversion

3x 3

5x 5

Neural DualContouring

2 x 2 Convolution
3 x 3 Residual Convolution
1 x 1 Convolution

OutputCross Entropy Loss +Sketch Skeleton Loss+

L1 Loss

7x 7

Then we adapt recent Neural Dual Contouring and applied a series ofmodifications which particularly optimized for 2D vectorization task.
Please see our paper for more details.

19

Neural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key Point

Now I’m going to give more details about the logic how we refine the topology.

Given a target vector sketch with sampled 4 by 4 grids, we could easily find thatthe reconstructed vector lines can’t preserve the x junction even all edge flags
are correctly predicted. Because there exists under sampled issue

21

Neural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key Point

Now I’m going to give more details about the logic how we refine the topology.

Given a target vector sketch with sampled as 4 by 4 grids, we could easily findthat the reconstructed vector lines can’t preserve the x junction even all edge
flags are correctly predicted.

22

Neural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key Point

But we can still fix this by predicting a under sampling map along with the xjunction key point.

23

Neural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key Point

We remove all the line segments that inside the under-sampling map, thenconnect the all truncations to the predicted key point

24

Neural Dual Contouring & Topology Refinement• Reconstruct star junction(a) Input(b) [Mo et al. 2021](c) [Bessmeltsev et al. 2019](d) [Puhachov et al. 2021](e) Our Result(f) USM & Key point

While reconstructing the high valence junction, or star junction in other word, is ahistorically challenging problem.
Our Refinement method provides one simple solution to it.

25

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper

26

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper

27

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper

28

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper

29

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper30

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively fining shortest paths.
For more details, please check out our paper

31

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively finding shortest paths.
For more details, please check out our paper

32

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively finding shortest paths.
For more details, please check out our paper

33

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling• Line Grouping• Start with each end point• Iteratively find shortest path
• Repeat until all lines have been visited

Then to further reduce the number of unnecessary vector segments, we
proposed an under-sampled map based dual contouring down sampling method
To turn vector segments into strokes, we proposed a simple line grouping
method that iteratively finding shortest paths.
For more details, please check out our paper

34

Comparison• Benchmark [Yan et al. 2020]• Chamfer Distance (lower is better)• 2x–10x lower

We used our sketch clean up benchmark to create a vectorization test set. The
test set contains 369 clean raster sketches with their vector

ground truth.
Then we compared our method with others using
metrics such as chamfer distance, running time under 6different input sketch resolution levels.
Please note that all metrics along the y–axis are log
scaledThe experiment show our method consistently performs better and run fasterthan all other methods.

52Comparison• Benchmark [Yan et al. 2020]• Chamfer Distance (lower is better)• 2x–10x lower• Running Time• 2x–10x faster

We used our sketch clean up benchmark to create a vectorization test set. The
test set contains 369 clean raster sketches with their vector ground truth.
Then we compared our method with others using metrics such as chamfer distance,
running time under 6 different input sketch resolution levels.
The experiment show our method consistently performs better and run fasterthan all other methods.

53

Results on Clean Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (Stroke
Visualization)

Here we show a comparison of vectorization results
As we can see our method faithfully captured much more detail for even a tiny
portion of the raster sketch

54

Results on Clean Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (Stroke
Visualization)

Here we show a comparison of vectorization results
As we can see our method faithfully captured much more detail for even a tiny
portion of the raster sketch

55

Results on Rough Sketch
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (Stroke
Visualization)

We also tested all methods with another test set which contains 112 roughsketches in the wild, this comparison further shows our method’s capacity ofcapturing stroke details.

56

Results on Rough Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (Stroke
Visualization)

It is also worth mentioning that our method could successfully vectorize allrough sketches with faster speed compared to the frame filed based methods

57

Results on Rough Sketches
Input

Our Result

Stroke Visualization

Here are two more vectorization results on complex rough sketches that only our
method could vectorize. Please note that the zoom in region is only about 1/50 to
1/25 of the full sketch.

58

Results on Rough Sketches
Input

Our Result

Stroke Visualization

Here are two more vectorization results on complex rough sketches that only our
method could vectorize. Please note that the zoom in region is only about 1/50 to
1/25 of the full sketch.

59

Limitations(a) Input(b) Output from raw
size(c) Output from 2x downscaling(d) Output from lineextraction preprocessing

Here is one failure case that shows the limitations of our method.
Our methods will output broken and messy lines when the input sketch contains
densely repeated strokes or very thick strokes.
Although this vectorization result could be improved by either down sample theinput raster sketch or apply pre-processing such as line extraction.
We believe the fundamental reason for this limitation should be our pair-wised
training strategy which makes our network struggling when predicting the center
lines and junctions for those type of strokes.

60Future Work• Adaptive sampling rate
• Enforce prediction naturalness at junctions• Vectorize more stroke attributions• Stroke thickness• Color• Texture

In the future, we can consider an adaptive sampling
rate, enforce more natural junctions, and vectorize
more stroke attributes.

61

Searching for faculty positions in 2025
Automatic
Refinemen t

(a)

(b)
Raster line draw ing(c)Dual contouring,
Key points,
Neural inference
Under-sampl ing map

Post-processing(d)Vector line draw ing

Downstream appl ications(e.g., flatting)Thanks for listening!
At last, I want to say thanks for all my collaborators and thank you for yourlistening.

62