Mastering Sketching: Adversarial Augmentation for
Structured Prediction

Edgar Simo-Serra*, Satoshi Iizuka*, Hiroshi Ishikawa (*equal contribution)
Wednesday, August 15, 2018
Waseda University

Sketch Simplification
RoughTarget

RoughTarget3

Contributions• Semi-supervised framework for sketch simplification• Pencil generation results• Single-Image Optimization

Input[Simo-Serra+ 2016]©Eisaku KubonouchiOurs

4

Related Work

Related Work

1. Sketch Simplification
1.1 Progressive Online Modification
1.2 Stroke Reduction
1.3 Stroke Grouping

Liu et al. 2015
5

Related Work

1. Sketch Simplification
1.1 Progressive Online Modification
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

Liu et al. 2015
5

Related Work

1. Sketch Simplification
1.1 Progressive Online Modification
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

2. Vectorization
2.1 Model Fitting (Bezier, …)
2.2 Gradient-based approaches

Noris et al. 2013
5

Related Work

1. Sketch Simplification
1.1 Progressive Online Modification
1.2 Stroke Reduction
1.3 Stroke Grouping
1.4 Vector input

2. Vectorization
2.1 Model Fitting (Bezier, …)
2.2 Gradient-based approaches
2.3 Require fairly clean input sketches
Noris et al. 2013
5

Related work [Simo-Serra+ 2016]• 23 layer fully convolutional neural network• Encoder-Decoder shape×Up-convolution
2×2×

Flat-convolution
4×2

48×

84××

24

Down-convolution
6

Related work [Simo-Serra+ 2016]• 23 layer fully convolutional neural network• Encoder-Decoder shape• Dataset construction is critical• Expert knowledge is important
Standard Dataset Creation

Inverse Dataset Creation

6

Dataset Bias
Training pairs

Rough sketches “ in the wild”

7Dataset Bias
Training pairs

Rough sketches “ in the wild”• Supervised dataset (rough sketch and line drawing pairs): ρx,y
• Rough sketch dataset:ρx• Line drawing dataset: ρy7

Generative Adversarial Networks

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)

8

Generative Adversarial Network (GAN)• D(·): maximize classification prediction

max Ey ∗ ∼ρy log D(y ∗ ) + Ez∼N (0,1) log(1 − D(G(z)))
D| {z }| {z }
Real dataRandom

8

Generative Adversarial Network (GAN)• D(·): maximize classification prediction• G(·): minimize to fool D(·)

minGEz∼N (0,1) log(1 − D(G(z)))| {z }
Random8

Generative Adversarial Network (GAN)• D(·): maximize classification prediction• G(·): minimize to fool D(·)
• Alternate optimization
min max Ey ∗ ∼ρy log D(y ∗ ) + Ez∼N (0,1) log(1 − D(G(z)))
GD| {z }| {z }
Real dataRandom

8FrameworkModel• S(·): Sketch simplification model• 23 layer fully convolutional neural network [Simo-Serra+ 2016]• D(·): Discriminator model• 6 layer convolutional neural network
×Up-convolution
2×2×

Flat-convolution
4×2

48×84××2

4

Down-convolution9

Proposed framework
minS

入力

SupervisedStandard Lossz }| {E(x,y∗ )∼ρx,y}|{zkS(x) − y ∗ k2

Standard Loss10Proposed frameworkStandard Loss
Adversarial LossSupervised}|{ z}|{z }| {  zmin max E(x,y∗ )∼ρx,y  kS(x) − y ∗ k2 + α log D(y ∗ ) + α log(1 − D(S(x))) S入力

DStandard Loss+Adversarial

10

Proposed frameworkStandard Loss
Adversarial LossSupervised}|{ z}|{z }| {  zmin max E(x,y∗ )∼ρx,y  kS(x) − y ∗ k2 + α log D(y ∗ ) + α log(1 − D(S(x))) SDRoughLinez }| {z }| {+ β Ey∼ρy [ log D(y) ] + β Ex∼ρx [ log(1 − D(S(x))) ]|{z}
Unsupervised Adversarial Loss

入力

Standard Loss+Adversarial+Unsupervised

10

Training• Supervised data: standard loss + adversarial loss
• Unsupervised data: adversarial loss
Supervised Data

MSEAdversarial

11

Training• Supervised data: standard loss + adversarial loss
• Unsupervised data: adversarial loss

Rough SketchesLine Drawings

Adversarial

11

Results

(lack of) Post-processing• MSE loss requires post-processing to avoid blurring• Adversarial loss avoids blurring

Input

LtS (no PP)

LtS (PP)

Ours (no PP)
12Results

Input[Favreau et al. 2016] [Simo-Serra et al. 2016]©Eisaku Kubonouchi

Ours

13

Perceptual User Study• Comparison against LtS [Simo-Serra et al. 2016]• 99 images, 15 users• 94 images from artists not in training set• 60 images come from twitter
5LtS

Ours

absolute

2.77

3.60

vs LtSvs Ours

11.1%

88.9%
-4321

LtS

Ours

Absolute rating.

Mean results for all users.
14

Extensions

Extensions - Pencil Drawing Generation

15

Extensions - Pencil Drawing Generation

Input

MSE

Illustrator A

Illustrator B

15

Extensions - Single-Image Optimization

Induction
Training
ExamplesParametric
Model

Deduction

Transduction

Testing
Examples• Inductive Machine Learning: Learning a parametric function/model fromtraining data and applying it on new data
• Transductive Machine Learning: Using training data to predict test data
(model not necessary)16Extensions - Single-Image Optimization

Input

Ours
©David Revoy www.davidrevoy.com

Optimized
16

Extensions - Single-Image Optimization

Input

Ours
©David Revoy www.davidrevoy.com

Optimized
16

Limitations• Still a strong dependency on labelled data
• Results not perfect and require manual fixing

Input

Only UnsupervisedOurs

17

Limitations• Still a strong dependency on labelled data
• Results not perfect and require manual fixing

Input

Output17

To conclude
http://hi.cs.waseda.ac.jp/~esimo/research/sketch_master/• Semi-supervised Sketch Simplification Framework• Pencil Drawing Generation• Single-Image Optimization©Edgar Simo-Serra18