Deep Sketch Vectorization via
Implicit Surface Extraction

Chuan Yan1, Yong Li2,1, Deepali Aneja3, Matthew Fisher3, Edgar Simo-Serra4, Yotam Gingold1
1.

2.

3.

4.

Why is extracting vector lines important?

Why is extracting vector lines important?

Challenges• Vectorization fidelity at dense stroke regions[Puhachov et al. 2021]

Challenges• Vectorization fidelity at dense stroke regions[Puhachov et al. 2021][Bessmeltsev et al. 2019][Mo et al. 2021]

Challenges

384 × 51279s• Vectorization fidelity at dense stroke regions• Running time
[Puhachov et al. 2021]
69s

42s

[Bessmeltsev et al. 2019][Mo et al. 2021]

Our Solution

79s• High vectorization fidelity• Fast running time
[Puhachov et al. 2021][Ours]

69s

42s[Bessmeltsev et al. 2019][Mo et al. 2021]

Our Solution

384 × 51279s

• High vectorization fidelity• Fast running time
[Puhachov et al. 2021]

4.5s

69s

42s

[Ours][Bessmeltsev et al. 2019][Mo et al. 2021]

Our Solution

384 × 512

4.5s

• High vectorization fidelity• Low algorithm complexity• Supports interactive Refinement[Ours]

Overview• Center line encoding• Line reconstruction• Post processing
InitialVectorization

Line Reconstruction

Raster Sketch
Key PointsDistanceFieldPredictio
nUnderSampling
Map

Incorrectly Predicted Topology
Auto Topology
Error Detect ion Step 1Center Line

Manu al LabelTopology Errors Step 2 (opt ional)

Refined
Vectorization

Final Vectorization

Dual Contouring Downsampling & Line grouping

RefinedTopology

Threshold
Local
Maximum

UserEdits

Overview• Center line encoding: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Down Sampling & Line Grouping
Distance Field Prediction
Line Reconstruction

Raster Sketch
Key Points

Under-Sampling
MapCenter LineThresh
Local MaximumOverview• Center line encoding: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Down Sampling & Line GroupingLine Reconstruction

Initial
Vectorization
Key Points

Under-Sampling
Map

Auto Topology Error
Detection - Step 1Center Line

Threshold
Local Maximum

Incorrectly Predicted Topology
Manual Label Topology
Errors - Step 2 (optional

User Edits

Overview• Center line extraction: (a) Distance Field Prediction• Line reconstruction: (b) Neural Dual Contouring & Topology Refinement• Post processing: (c) Dual Contouring Downsampling & Line Grouping

InitialVectorization

Dual Contouring Downsampling & Line grouping
Incorrectly Predicted Topology
Auto Topology Error
Detection - Step 1

Refined
Vectorization

Final Vectorization

Manual Label Topology
Errors - Step 2 (optional)

RefinedTopology

User EditsDual Contouring with Under-Sampled Map

Target Vector Sketch

Center LineUnder-Sampled Map

Edge Flags
Reconstructed Vector Sketch

Key Points

Edge Vertex

Dual Contouring with Under-Sampled Map
Ground Truth for DistanceField Prediction NetworkTarget Vector Sketch≠

Center Line

Under-Sampled Map

Key Points

Ground Truth for Line
Reconstruction Network
Edge Flags
Reconstructed Vector Sketch

Edge Vertex

Distance Field Prediction• Dataset• 10,000 and 53,000 vector sketches from the Quick
Draw! and Creative Sketch [Ge et al. 2021]• 441,000 raster sketches in total(a) Basic (b) Calligraphy (b) Marker A(c) Marker B(d) Ink A(e) Ink B(f) Pencil

Distance Field Prediction• Dataset• Loss function• 10,000 and 53,000 vector sketches from the Quick• Masked L1 distance

Draw! and Creative Sketch [Ge et al. 2021]• 441,000 raster sketches in total(a) Basic (b) Calligraphy (b) Marker A(c) Marker B(d) Ink A(e) Ink B(f) Pencil

Distance Field Prediction• Sub-Pixel Sampling• 2x size of prediction output and ground truth• Fewer under-sampled region• Higher precisionNeural Dual Contouring & Topology Refinement• Network [Chen et al. 2022]• Multi-Resolution convolution branchesInputUDF

Multi size gridconvolution

Domain Conversion

3x35x5

Neural DualContouring

2 x 2 Convolution
3 x 3 Residual Convolution
1 x 1 Convolution

OutputCross Entropy Loss +Sketch Skeleton Loss+

L1 Loss

7x7

Neural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key PointNeural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key PointNeural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key PointNeural Dual Contouring & Topology Refinement• Under-Sampled Map-based
Topology Refinement• Refine with Key Point• Refine without Key PointNeural Dual Contouring & Topology Refinement• Reconstruct star junction(a) Input(b) [Mo et al. 2021](c) [Bessmeltsev et al. 2019](d) [Puhachov et al. 2021](e) Our Result(f) USM & Key point

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled resultDual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling

Raw reconstruction

Downsampled result

Dual Contouring Downsampling & Line Grouping• Under-Sampled Map-based Dual Contouring Downsampling• Line Grouping• Start with each end point• Iteratively find shortest path
• Repeat until all lines have been visitedComparison• Benchmark [Yan et al. 2020]• Chamfer Distance (lower is better)• 2x–10x lowerComparison• Benchmark [Yan et al. 2020]• Chamfer Distance (lower is better)• 2x–10x lower• Running Time• 2x–10x fasterResults on Clean Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (StrokeVisualization)

Results on Clean Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (StrokeVisualization)

Results on Rough Sketch
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

Ours

Ours (StrokeVisualization)

Results on Rough Sketches
Input[Puhachov et al.
2021][Bessmeltsev et al.
2019][Mo et al. 2021]

OursOurs (StrokeVisualization)

Results on Rough Sketches
Input

Our Result

Stroke Visualization

Results on Rough Sketches
Input

Our Result

Stroke Visualization

Limitations(a) Input(b) Output from rawsize(c) Output from 2x downscaling(d) Output from lineextraction preprocessing

Future Work• Adaptive sampling rate

• Enforce prediction naturalness at junctions• Vectorize more stroke attributions• Stroke thickness• Color• TextureSearching for faculty positions in 2025
AutomaticRefinement

(a)

(b)
Raster line drawing(c)

Dual contouring,
Key points,
Neural inference
Under-sampling map

Post-processing(d)Vector line drawing

Thanks for listening!

Downstream applications(e.g., flatting)