Real-Time Data-Driven Interactive Rough Sketch Inking

Edgar Simo-Serra, Satoshi Iizuka, Hiroshi Ishikawa
Wednesday, August 15, 2018
Waseda UniversityMotivation

Input

Previous Approach2

Motivation

Input

Previous Approach

User Input

Standard Eraser

2Motivation

Input

Previous Approach

User Input

Standard Eraser

Proposed2

Motivation“1. The inker’s main purpose is to translate the penciller’s graphite pencil lines
into reproducible, black, ink lines.
2. The inker must honor the penciller’s original intent while adjusting any obviousmistakes.
3. The inker determines the look of the finished art.”— Gary Martin, The Art of Comic Book Inking [1997]2

Interactive Neural Networks• Feed-forward fully convolutional neural network

Input

ModelOutput

3

Interactive Neural Networks• Feed-forward fully convolutional neural network• Input rough sketch and user edit are concatenated channel-wise+
ModelInput

OutputUser
User Edit

3

Interactive Neural Networks - Related Work
• User input is treated as an additional image channel• Training user input is sampled from ground truth• Grayscale image colorization [Sangkloy+ 2017, Zhang+ 2017]

Real-Time User-Guided Image Colorization with Learned Deep Priors. Richard Zhang et al. SIGGRAPH 20174

Interactive Neural Networks - Related Work
• User input is treated as an additional image channel• Training user input is sampled from ground truth• Grayscale image colorization [Sangkloy+ 2017, Zhang+ 2017]• Not directly applicable to the rough sketch inking problem

Input

User Edit

Baseline

Ours

4Interactive Neural Networks - Related Work
• User input is treated as an additional image channel• Training user input is sampled from ground truth• Grayscale image colorization [Sangkloy+ 2017, Zhang+ 2017]• Not directly applicable to the rough sketch inking problem• How to train an interactive network for inking?

Input

User Edit

Baseline

Ours

4Proposed Framework• Main contributions• Line width normalization• Simulation of user edits• Three different smart tools• Evaluation with a perceptual user study
Smart Inker

Input rough sketch
Inker Pen
Inker Brush
Smart Eraser©Krenz Cushart

Canvas

Output5

Training Framework
1. Line width normalization
2. Simulation of user edits

0LineNormalization
User Edit
Simulation

Train

SmartInker0Dataset

Training Data6

Training Framework - Line width normalization
1. Line width normalization
2. Simulation of user edits

0Line
Normalization
User Edit
Simulation

Train

SmartInker

0

DatasetTraining Data7

Training Framework - Line width normalization
Rendered vector images

Training

7

Training Framework - Line width normalization

QuickDraw

TUD-Berlin

KanjiVG

Synthetic

7

Training Framework - Line width normalization

Input[Zhang and Suen 1984]

Ours

7

Training Framework - Line width normalization

Input

No normalization

Normalization
7

Training Framework - Simulation of user edits
1. Line width normalization
2. Simulation of user edits

0LineNormalization
User Edit
Simulation

Train

SmartInker

0

DatasetTraining Data8

Rough Sketch

Line Drawing

Training Framework - Simulation of user edits

Input Data Pair

8

Rough Sketch

Line Drawing

Training Framework - Simulation of user edits

Input Data Pair

Sampled Regions

8

Rough Sketch

Line Drawing

Training Framework - Simulation of user edits

Input Data Pair

Sampled Regions

Add Edits and Noise8

Rough Sketch

Line Drawing

Training Framework - Simulation of user edits

Input Data Pair

Sampled Regions

Add Edits and Noise8

Rough Sketch

Line Drawing

Training Framework - Simulation of user edits

Input Data Pair

Sampled Regions

Add Edits and Noise8

Smart Tools• Inker Pen• Allows for fine-grained line control

Input

Automatic

Edit

Ours

9

Smart Tools• Inker Brush• Sloppy and fast line manipulation

Input

Automatic

Edit

Ours

9

Smart Tools• Smart Eraser• Takes into account rough sketch when erasing

Input

Automatic

Edit

Ours

9

Training
L(y, y ∗ ) = |(y − y ∗ ) ⊙ (1 + γ (1 − y ∗ ))|1{z}| {z } |L1 loss
• Using L1 loss
• Change weight of lines with γ

Weight lines with γ

105

105

105

105

104

104

104

104

103

103

103

Input

[Simo-Serra+ 2016]

Baseline©David Revoy www.davidrevoy.com

103

Ours

10Model• Similar to model of [Simo-Serra+ 2016]• 24 layer fully convolutional neural network• Number of filters optimized for real-time performance• Roughly three times the performance
Approach Parameters 10242 px

15122 px

20482 px

25602 px

[Simo-Serra+] 2016 44,551,425 238.8ms 562.4ms 984.7ms
1.59s
Ours 12,795,169 89.9ms 225.5ms 382.7ms 592.9ms

11

Dataset• 288 rough sketch and line drawing pairs• More challenging than previous works

12Results

Input

Automatic

EditOurs©Eisaku Kubonouchi

14Results

Input

Automatic

EditOurs©Eisaku Kubonouchi

14

User Study• Comparison (time) with proposed approach vs Clip Studio Pro• Total of 10 users and 10 unique images• Each user processes random 5 images with each tool
• Total average time of 2.8 hours per user
• Overall 1.8× speed-up with ours

Some of the images used in the user study
15

User Study

Time (s)3000
2000
10000PT Ours

1000

Time (s)800
600
4002000

PT Ours

Time

Input

AmateurExperienced©David Revoy www.davidrevoy.com

EditOurs
15

User Study©David Revoy www.davidrevoy.com

1000

Time (s)800
600
4002000PT Ours

1000

Time (s)800
600
4002000

PT Ours

Time

Input

AmateurExperienced©Krenz Cushart

Edit

Ours

15

Limitation

Input

AutomaticEdit

Ours

16

To conclude
http://hi.cs.waseda.ac.jp/~esimo/research/inking/• Interactive rough sketch inking framework• Line width normalization• User edit simulation©Edgar Simo-Serra

17