Learning to Simplify:
Fully Convolutional Networks for Rough Sketch Cleanup

Edgar Simo-Serra∗ ， Satoshi Iizuka∗ ， Kazuma Sasaki， Hiroshi Ishikawa
July 27th, 2016
Waseda University(*equal contribution)

Sketch Simpliﬁcation

1

Sketch Simpliﬁcation
Input: Rough Sketch

Output: Line Art

2

Sketch Simpliﬁcation
RoughTargetRoughTarget3

Related Work

Related Work
1. Sketch Simpliﬁcation
1.1 Progressive Online Modiﬁcation
1.2 Stroke Reduction
1.3 Stroke Grouping

Liu et al. 2015
4

Related Work
1. Sketch Simpliﬁcation
1.1 Progressive Online Modiﬁcation
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

Liu et al. 2015
4

Related Work
1. Sketch Simpliﬁcation
1.1 Progressive Online Modiﬁcation
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

2. Vectorization
2.1 Model Fitting (Bezier, …)
2.2 Gradient-based approaches

Noris et al. 2013
4

Related Work
1. Sketch Simpliﬁcation
1.1 Progressive Online Modiﬁcation
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

2. Vectorization
2.1 Model Fitting (Bezier, …)
2.2 Gradient-based approaches
2.3 Require fairly clean input sketches

Noris et al. 2013
4

Related Work
1. Sketch Simpliﬁcation
1.1 Progressive Online Modiﬁcation
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

2. Vectorization
2.1 Model Fitting (Bezier, …)
2.2 Gradient-based approaches
2.3 Require fairly clean input sketches

3. Deep LearningLong et al. 2015

3.1 Fully Convolutional Network4

Proposed ApproachDeep Learning• Modern Neural Networks• Computational efﬁciency with GPU• Large scale datasets

5

Deep Learning• Modern Neural Networks• Computational efﬁciency with GPU• Large scale datasets• Learns input to output mapping• Basic building block layer: f (x) = σ(Wx)5Deep Learning• Modern Neural Networks• Computational efﬁciency with GPU• Large scale datasets• Learns input to output mapping• Basic building block layer: f (x) = σ(Wx)• Parameters (W . . .) are learnt• Hyper-parameters are set by hand5

Fully Convolutional Network• Uses only convolutional layers• Each layer convolves many ﬁlters• Layer hyperparameters: kernel, padding, and stride• Weights expressed with kernels• Padding conserves the image size
• Stride can change the output resolution

OutputInput
6

Fully Convolutional Network
We create three building blocks by modifying the stride:
1. Flat-convolution
1.1 3 × 3px kernel， 1 × 1px padding， 1px stride

2. Down-convolution
2.1 3 × 3px kernel， 1 × 1px padding， 2px stride

3. Up-convolution
3.1 4 × 4px kernel， 1 × 1px padding， 1/2px stride
stridestrideFlat-convolution
strideUp-convolution
Down-convolution7Model• 23 convolutional layers• Output has the same resolution as the input• Encoder-Decoder architecture• Reduces memory usage• Increases spatial resolution
×Up-convolution
2×2

Flat-convolution
4×48×84×

42××

2

Down-convolution
8

Learning• Trained from scratch• Using 424 × 424px patches• Weighted Mean Square Error loss
• Batch Normalization [Ioffe and Szegedy 2015] is critical• Optimized with ADADELTA [Zeiler 2012]
Input

OutputTarget9Vectorization and Simpliﬁcation• Vectorization with potrace• Open source software• High pass ﬁlter and binarization

Input

OutputVector10

Vectorization and Simpliﬁcation• Vectorization with potrace• Open source software• High pass ﬁlter and binarization• Scaling input changes simpliﬁcation degree10Sketch Dataset

Sketch dataset• 68 pairs of rough and target sketches• 5 illustrators
Sketch dataset

Extracted patches・・・

・・・

11

Inverse Dataset Creation• Data quality is critical• Creating target sketches from rough sketches has misalignments• Creating rough sketches from target sketches properly aligns
Standard

Inverse Creation

12

Data Augmentation• 68 pairs is insufﬁcient• Scaling training data
• Random cropping, ﬂipping and rotation• Additional augmentation: tone, slur, and noise
input

toneslurnoise

13

Results and Comparisons

Results• Intel Core i7-5960X CPU (3.00GHz)• NVIDIA GeForce TITAN X GPU• 3 weeks training time
Image Size

PixelsCPU (s)GPU (s)Speedup

320 × 320
640 × 640
1024 × 1024

102,400
409,600
1,048,576

2.014
7.533
19.463

0.047
0.159
0.397

42.9×
47.4×
49.0×14

Comparison

Input

Adobe Live Trace

PotraceOurs

15

Comparison

Input

Adobe Live Trace

Potrace

Ours

15

User Study• Comparison with 15 images• 19 users participated (10 with illustration experience)• Absolute rating (1 to 5 scale)• Relative evaluation (best of two)
Ours

Live Trace

Potrace

Score

4.53

2.94

2.80

vs Oursvs Live Trace
vs Potrace

97.5%
97.2%

2.5%
69.7%

2.8%
30.3%
16

Ours

[Liu et al. 2015]

Input

Comparison

17

Results

18

Results

18

Results18

Results

18

Conclusions• Automatic Sketch Simpliﬁcation Approach• Convolutional networks are suited to image processing• Proper data is crucial for training

19

Conclusions• Automatic Sketch Simpliﬁcation Approach• Convolutional networks are suited to image processing• Proper data is crucial for training

Try Online: http://hi.cs.waseda.ac.jp:8081/

19Modelkernel size

strideoutput size

input
down-convolution
ﬂat-convolution
ﬂat-convolution

type

5×5
3×3
3×3

2×2
1×1
1×1

1×H×W
48 × H/2 × W/2
128 × H/2 × W/2
128 × H/2 × W/2

down-convolution
ﬂat-convolution
ﬂat-convolution

3×3
3×3
3×3

2×2
1×1
1×1

256 × H/4 × W/4
256 × H/4 × W/4
256 × H/4 × W/4

down-convolution
ﬂat-convolution
ﬂat-convolution
ﬂat-convolution
ﬂat-convolution
ﬂat-convolution
ﬂat-convolution
ﬂat-convolution

3×3
3×3
3×3
3×3
3×3
3×3
3×3
3×3

2×2
1×1
1×1
1×1
1×1
1×1
1×1
1×1

256 × H/8 × W/8
512 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
1024 × H/8 × W/8
512 × H/8 × W/8
256 × H/8 × W/8

up-convolution
ﬂat-convolution
ﬂat-convolution

4×4
3×3
3×3

1/2 × 1/2

256 × H/4 × W/4
256 × H/4 × W/4
128 × H/4 × W/4

up-convolution
ﬂat-convolution
ﬂat-convolution

4×4
3×3
3×3

1/2 × 1/2

128 × H/2 × W/2
128 × H/2 × W/2
48 × H/2 × W/2

up-convolution
ﬂat-convolution
ﬂat-convolution

4×4
3×3
3×3

1/2 × 1/2

48 × H × W
24 × H × W
1×H×W

1×1
1×1
1×1
1×1
1×1
1×1