Real-Time Data-Driven Interactive Rough Sketch Inking— SUPPLEMENTAL MATERIAL —EDGAR SIMO-SERRA, Waseda University
SATOSHI IIZUKA, Waseda University
HIROSHI ISHIKAWA, Waseda University

We present an in-depth report of our user study and additional results of our Smart Inker approach. This document isintended as a complement to the technical paper.
1 USER STUDY REPORTWe report on the user study conducting during the evaluation of our approach.
1.1 Design
We designed the user study such that all participants would ink 10 images, 5 with Clip Studio Paint Ex and 5 with
Smart Inker, and measure the time it takes each user to ink each rough sketch. Furthermore, we designed it such that
all images are inked the same amount of times with both Professional Tools (PT) and Smart Inker. With these two
restrictions and a ﬁxed number of participants decided ahead of time, for each user, we randomly choose what images
they would ink with which approach, and the order in which they would be inked. A total of 10 users participated andan overview of the order they inked the rough sketches in is shown in Table 1. All the users are remunerated for their
participation in the user study. After the user inks all 10 images, they are given a survey to ﬁll out with ten questions.
We did not opt for a pairwise design as we found that repeating the same image twice with a single user led to aconsiderable bias depending on the order they were inked in, i.e., the ﬁrst time inking an image takes signiﬁcantly moretime with respect to the second time inking the same image.
1.2 Clip Studio Paint Ex
Clip Studio Paint Ex1 is software specialized for illustration and, in particular, creation of comics. It contains extensive
features for inking, and it particular, removes jitters from the digital pen strokes with improved responsiveness. This
professional software package is what we compared Smart Inker to and we abbreviate as Professional Tools (PT).
1.3 Hardware
We perform the user test using a Wacom MobileStudio Pro 16, which is a professional mobile pen computer designed
for illustration. It comes both with a pen with pressure sensitivity and employs a large 16" touch screen. We note thatwhile Clip Studio Paint Ex uses this pressure sensitivity, our approach does not.
1 https://www.clipstudio.net/en

Authors’ addresses: Edgar Simo-Serra, Waseda University, esimo@aoni.waseda.jp; Satoshi Iizuka, Waseda University, iizuka@aoni.waseda.jp; Hiroshi
Ishikawa, Waseda University, hfs@waseda.jp.
Manuscript submitted to ACM

1

Table 1. Order in which each of the users performed the user test. Asterisk indicates the particular image was inked using Clip Studio
Paint Ex.

User

Order

User 1
User 2User 3User 4User 5User 6
User 7User 8User 9User 10

2*, 9, 3, 4, 7, 1, 8*, 6*, 5*, 10*
7*, 5, 4*, 8, 9, 10*, 2, 3*, 6*, 1
10, 3*, 2*, 9*, 6, 7*, 5, 1*, 8, 4
9, 5*, 8*, 6, 2*, 7, 10*, 4*, 1, 3
10*, 5*, 2, 3, 1*, 4, 8*, 9, 7, 6*
2, 1*, 4*, 3, 7, 9*, 5*, 10, 8*, 6
1*, 9*, 6*, 10, 4, 2, 3*, 8, 5, 7*
8, 4*, 1, 5, 6, 9*, 10, 2*, 7*, 3*
7*, 2, 3*, 6, 1*, 8, 9*, 5, 4*, 10
6*, 5*, 4, 2*, 8*, 10*, 3, 1, 7, 9

1.4 User Explanation
For each user, we brieﬂy explain both the usage of Clip Studio Paint Ex and our approach. For Clip Studio Paint Ex,
we set up the image, and select the “G-pen” inking brush with a default size of 10, although we allow the users tochange the brush and brush size. Additionally, the users are able to rotate, zoom, and translate the illustration easily
with their ﬁngers, making it very easy to ink with. We also show them how to do other standard operations like undo
and redo. For our approach, we show them the basic usage of the three main tools: inker pen, inker brush, and smart
eraser, and teach them how to change some visualization options such as the transparency of the rough sketch. Users
are also taught how to undo and redo actions. Finally, for both approaches, we give the users 2 minutes to practice
before starting the evaluation.
As guidelines for inking, we tell the users to maintain ﬁdelity to the original drawing, while correcting obvious
mistakes, e.g., connect disconnected lines or removing scaﬀolding lines. We also mention explicitly that tone and pencil
shading should not be conserved.
1.5 Results
To complete the user study, including explanation, preparation and inking the 10 images, each user took an averageof 2.8 hours, with the fastest user being a bit over 2 hours, and the slowest user being near 5 hours. The full resultsof the survey are summarized in Fig. 1. We can see that out of the 10 individuals, three profess to have signiﬁcantdrawing experience, three have some drawing experience, and four are complete amateurs. Almost all of the users
ﬁnd our approach easier, with one user preferring Clip Studio Paint Ex, and one ﬁnding both equally easy. Half of theusers ﬁnd their results with our tool good and found the tool easy to use. Of the three tools, the inker brush tool was
found the easiest to use with the inker pen tool being the hardest, although even then 4 people found it easy to use. Allusers believe with more time they could master Smart Inker and that it would be a beneﬁcial addition to professional
software such as Clip Studio Paint Ex.
The amount of time taken by each user for each image is shown in Table 2. A graphical representation is shown in 2.
We can see that, as expected, there is large variation between users and images due to the nature of the task.
We compare the timings between Clip Studio Paint Ex and Smart Inker with the Mann-Whitney U test [1], a nonparametric statistical test, with the null hypothesis being that timing with both approaches have equivalent medians,
and the alternative hypothesis being that they do not (two-tailed test). Unlike the commonly used t-Student test, it
2

10

8

8

8

6

Significant
Some
None

4
2
0

6

Number of Users

10

Number of Users

Number of Users

10

Smart Ink
Same
Clip Studio Paint Ex

4
2
0

1

Question #6

Positive
Neutral
Negative

4
2
0

2

3

Question #

4

5

6

7

8

Question #910Fig. 1. Summary of the answers of the user survey for each of the ten questions which are the following:(1) How much experience do you have with drawing and illustration? (none, some, significant)(2) Which tool is was easier to use? (Clip Studio Paint Ex, same, Smart Inker)(3) What do you think of the results you obtained with Smart Inker? (poor, fair, good)(4) How easy was Smart Inker to use? (hard, fair, easy)(5) How easy was the Smart Inker inker pen tool to use? (hard, fair, easy)(6) How easy was the Smart Inker inker brush tool to use? (hard, fair, easy)(7) How easy was the Smart Inker smart eraser tool to use? (hard, fair, easy)(8) Given enough time to practice, do you feel that Smart Inker is a tool you could become eﬀicient with? (no, maybe with eﬀort,
yes)(9) If yes, how long do you think it would take? (long time, some time, little time)(10) Should Smart Inker be added to programs like Clip Studio Paint Ex, would it help people with their work? (no, I don’t know,
likely)Table 2. Full timing details of the user study. Time is shown in seconds.

Clip Studio Paint Ex
2

3

4

5

6

7

Smart Inker

Image #

1

8

9

10

1

2

3

4

5

6

7

8

9

10

User 1
User 2User 3User 4User 5User 6
User 7User 8User 9User 10301 539
199 18873
579 518
567
395
697
545
564 4452699
1225
730 1969
778
448
779
408 1137
399 2375 202
1172
3410 198 1054 782
786
875
794 6571
2841
589
614 489
1745 396
455
453
1769
978 473
1098 1638
602 699
309

82

148
127 546435 200 256
540
613
455
236
223
163 487213447
446 572677 757
283 301
347 748 367
338 734266
241
413
209 345
158 376 125
325 603390 500
318
165
226
223
231 230

medianmean

579 518 1172 567 489 399 2699 202 1225 697
590 611 1291 614 502 495 3364 246 1568 625

236 325 603 226 390 367 376 241 231 455
242 306 593 280 445 345 333 299 300 480

158

367
357 917249

does not make the assumption of normally distributed data and is thus applicable to data with outliers such as the data
we have obtained in our user study. We obtain a U statistic of 1995.0, which corresponds to a p-value of 2.589 × 10−7 ,indicating that with very high certainty (over 99.999% probability) we can reject the null hypothesis, and accept thealternate hypothesis that the median values are diﬀerent. The global speed-up between all images is of 1.8×.
We also show an image-by-image breakdown in Table 3. We can see that for individual images, 6 of the 10 images
shown statistical diﬀerence in distributions with a signiﬁcance of α = 0.1. Of the tests with statistical signiﬁcance we
see speed-ups of between 1.6× and 7.2×.
3

3500

Clip Studio Paint Ex
Smart Ink

3000

Time (s)

2500
2000
1500
1000
500
0

1

2

3

4

5

6

Image #

7

8

9

10

Mean

Fig. 2. Timing information for inking each image with diﬀerent approaches. We plot the results using box plots with the medianhighlighted in dark red. Outliers are shown in dark gray. We show individual results for each image and the mean results for all theimages.
Table 3. Break down of the timing comparison by image. We compare the median interaction times between Clip Studio Paint Ex
and Smart Inker and show the speed-up obtained by our approach. We perform a Mann-Whitney U test for each image, comparing
whether the two medians are significantly diﬀerent and show in bold the p-values with a significance of α = 0.1.

Median Value (s)Mann-Whitney U Test

Image

Clip Studio Paint Ex

Smart Inker

Speed-up

U statistic

p-value

Image 1
Image 2
Image 3
Image 4
Image 5
Image 6
Image 7
Image 8
Image 9Image 10

579
518
1172
567
489
399
2699
202
1225
697

236
325
603
226
390
367
376
241
231
455

2.5×
1.6×
1.9×
2.5×
1.3×
1.1×
7.2×
0.8×
5.3×
1.5×

25.0
21.5
22.0
25.0
15.0
18.0
25.0
10.0
25.0
18.0

1.219e-02
7.491e-02
6.010e-02
1.219e-02
6.761e-01
2.963e-01
1.219e-02
6.761e-01
1.219e-02
2.963e-01

mean

608

331

1.8×

1995.02.859e-07

Finally, we show the images used in the user test and example results of inking both with Clip Studio Paint Ex andSmart Inker in Figures 3 to 12.

4

1000

Time (s)

800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 3. Image 1 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in
the top row, and two examples done by users with our approach. Image is copyrighted by David Revoy (www.davidrevoy.com) andlicensed under CC-by 4.0.

5

1200

Time (s)

1000
800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 4. Image 2 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach.

6

2000

Time (s)

1500
1000
500
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 5. Image 3 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach. Image is copyrighted by Eisaku Kubonouchi.

7

1000

Time (s)

800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 6. Image 4 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach.

8

1000

Time (s)

800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 7. Image 5 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in
the top row, and two examples done by users with our approach. Image is copyrighted by David Revoy (www.davidrevoy.com) andlicensed under CC-by 4.0.

9

1000

Time (s)

800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 8. Image 6 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach. Image is copyrighted by Eisaku Kubonouchi.

10Time (s)

3000
2000
1000
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 9. Image 7 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach.

11

500

Time (s)

400
300
200
100
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 10. Image 8 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach.

12

2000

Time (s)

1500
1000
500
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 11. Image 9 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex in the
top row, and two examples done by users with our approach. Image is copyrighted by Eisaku Kubonouchi.

13

1000

Time (s)

800
600
400
200
0

PT Ours
Time

Input

AmateurExpertUser A Edit

User A Ours

User B Edit

User B Ours

Fig. 12. Image 10 of the user study. We show an example inking result by an amateur and expert user using Clip Studio Paint Ex
in the top row, and two examples done by users with our approach. Image is copyrighted by Krenz Cushart and is part of Krenz’s
Artwork Sketch Collection 2004-2013.

14

Rough Sketch

Edit Map

Ground Truth

Fig. 13. Example of training data used to train our baseline model. This data is generated by randomly using patches of the groundtruth as “pen” or “eraser” inputs. The user edit shows the lines in white, eraser in black, and the gray values indicate no edit.

2 BASELINE APPROACH
We train a model following a similar approach to those of [2] and [5]. Their approach consists of training interactive
colorization networks by providing color hints to the model during training. This color hints consist of patches orpoints of the ground truth. Given that it is not very practical to ink rough sketches using points, we opt for the patch
approach. In particular, we randomly sample 10 to 30 pixel patches of the ground truth, and randomly show either theline information or the white space information to the model. Up to 8 patches are sampled per image, similar to how
our approach can have up to 8 simulated user edits. An example of the data used to train the baseline model is shownin Fig. 13. This baseline approach is unable to properly learn to use the eraser tool, and in particular has a tendency todraw lines around erased areas as shown in Fig. 14. Our proposed user edit simulation avoids this problem entire, while
also allowing us to use the inker brush tool, which is the most appreciated by the users as shown by our user test. It is
not possible to train a inker brush tool using simple approaches based on showing only the ground truth to the model.
3 IMAGE GENERATION
While our approach focuses on inking images, it can also be used as a drawing tool, in which the inker pen and inker
brush tools will help auto-complete and connect strokes. A simple illustration is shown in Fig. 15. In this example, the15InputOur Automatic Approach

[4]

[3]

User Edit

Standard Eraser

Baseline

Our Interactive Approach

Fig. 14. Comparison of our approach with automatic and interactive approaches. The standard eraser is the result of using thestandard eraser tool found in most software with the provided user edits. The baseline consists in a model in which parts of theground truth are shown as user edits during training.

User Edit

OutputFig. 15. Example of stroke continuity preserving by our approach. We input a completely white image as a rough sketch and show
the eﬀect of diﬀerent additive edits. It is able to conserve the continuity of strokes in primitives as shown on the left and right, whilestill being able to distinguish between nearby parallel lines as shown in the middle. Furthermore, it is possible to seamlessly combine
the inker pen tool and inker brush tool to create a single continuous line as shown below.

inputted rough sketch is white, containing no information. We provide some drawings as the user input, and see how
our approach is able to complete curved lines, straight lines and crossings, while also avoiding joining parallel lines.
4 LINE NORMALIZATION
4.1 Models
We provide the full details of both the wide and nimble line normalization models in Tables 4 and 5, respectively.
16

Table 4. Overview of the line normalization wide model architecture. We specify layer irregularities in the notes column. When thesame layer is repeated several times consecutively, we indicate this with the number of times in parenthesis.

Layer Type

Output Resolution

Notes

Input

1 ×W × H

Line drawing

Convolution
Convolution (×7)
Convolution

64 × W × H
64 × W × H
1 ×W × H

9 × 9 kernel, reﬂection padding
No batch normalization, sigmoid

Table 5. Overview of the line normalization nimble model architecture. We specify layer irregularities in the notes column. When thesame layer is repeated several times consecutively, we indicate this with the number of times in parenthesis.

Layer Type

Output Resolution

Notes

Input

1 ×W × H

Line drawing

Convolution
Convolution (×3)
Convolution

32 × W × H
32 × W × H
1 ×W × H

9 × 9 kernel, reﬂection padding
No batch normalization, sigmoid

4.2 Additional Results
We show some examples of full line drawings that have been normalized for use in training our model in Fig. 16. We
can see how our approach is able to accurately normalize the thickness of all the lines and produce clean anti-aliased
outputs.

17

Ground Truth

Normalized

Fig. 16. Examples of normalized line drawings amenable for training.

18

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 17. Example result. Image is copyrighted by David Revoy (www.davidrevoy.com) and licensed under CC-by 4.0.

5 ADDITIONAL RESULTS
We present additional results and comparisons of our approach with existing approaches [3, 4]. The results of [4] areshown using the post-processing they proposed, while the results of [3] are shown without post-processing as they
point out it is not necessary. For our approach, we show the automatic results and the results of the interactive user
editing.
We can see that the approach of [4] tends to break down on complicated images. This is a result of blurry predictions
in combination with the post-processing, which lead to low quality inking results. The approach of [3] tends to conserve
more thinner lines, but breaks down with heavy texture or shading. Smart Inker already shows better automatic inking
result, especially on challenging rough sketches, which can be complemented with the interactive user editing toachieve results that are amenable to colorizing without further processing of the line drawing.
REFERENCES[1] Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals ofmathematical statistics (1947), 50–60.
[2] Patsorn Sangkloy, Jingwan Lu, Chen Fang, FIsher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis with Sketch and Color. In
IEEE Conference on Computer Vision and Pattern Recognition.
[3] Edgar Simo-Serra, Satoshi Iizuka, and Hiroshi Ishikawa. 2018. Mastering Sketching: Adversarial Augmentation for Structured Prediction. ACM
Transactions on Graphics 37, 1 (2018).
[4] Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hiroshi Ishikawa. 2016. Learning to Simplify: Fully Convolutional Networks for Rough Sketch
Cleanup. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 35, 4 (2016).
[5] Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-Time User-Guided Image
Colorization with Learned Deep Priors. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 9, 4 (2017).

19

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 18. Example result. Image is copyrighted by Eisaku Kubonouchi.

Input

[4]

[3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 19. Example result. Image is copyrighted by David Revoy (www.davidrevoy.com) and licensed under CC-by 4.0.

20Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 20. Example result. Image is copyrighted by David Revoy (www.davidrevoy.com) and licensed under CC-by 4.0.

Input

[4]

[3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 21. Example result. Image is copyrighted by David Revoy (www.davidrevoy.com) and licensed under CC-by 4.0.
21InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 22. Example result. Image is copyrighted by Eisaku Kubonouchi.

22Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 23. Example result.

23

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 24. Example result. Image is copyrighted by Eisaku Kubonouchi.

24

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 25. Example result. Image is copyrighted by Krenz Cushart and is part of Krenz’s Artwork Sketch Collection 2004-2013.

25

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 26. Example result. Image is copyrighted by Eisaku Kubonouchi.

26

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 27. Example result. Image by Vincent Van Gogh.

27

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 28. Example result.

28

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 29. Example result. Image is copyrighted by Krenz Cushart and is part of Krenz’s Artwork Sketch Collection 2004-2013.

29

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 30. Example result. Image is copyrighted by David Revoy (www.davidrevoy.com) and licensed under CC-by 4.0.

Input

Our Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 31. Example result.

30InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 32. Example result. Image by Vincent Van Gogh.

31

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 33. Example result.

32

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 34. Example result. Image is copyrighted by Krenz Cushart and is part of Krenz’s Artwork Sketch Collection 2004-2013.

33

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 35. Example result. Image is copyrighted by Krenz Cushart and is part of Krenz’s Artwork Sketch Collection 2004-2013.

Input

Our Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 36. Example result.

34

Input[4][3]

Our Automatic Output

User Edit

Our Interactive Approach

Fig. 37. Example result. Image is copyrighted by Krenz Cushart and is part of Krenz’s Artwork Sketch Collection 2004-2013.

35

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 38. Example result. Image is copyrighted by Eisaku Kubonouchi.

36InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 39. Example result.

37

InputOur Automatic Output

[4]

[3]

User Edit

Our Interactive Approach

Fig. 40. Example result. Image is copyrighted by Eisaku Kubonouchi.

38