CONFIDENTIAL

Towards Fully AutomatedManga Translation
R. Hinami*1, S. Ishiwatari*1, K. Yasuda2, Y. Matsui2
1 Mantra Inc. 2 The University of Tokyo

CONFIDENTIAL

About mev Research interests- 2014-2016 Semantic Repr. & SMT- 2016-2017 Chunk-based NMT- 2017-2019 Definition Generation- 2019-Manga Translation- 2020-Manga Colorization- 2021-Manga-based Language Learning

2

CONFIDENTIAL

3

CONFIDENTIAL

Mantraのミッション
たくさん世界の言葉で、マンガを届ける。

漫画家

読まれたい
買われたい

いち早く
母語で世界のマンガ好き

読みたい

4

CONFIDENTIAL

Mantraの事業
高速翻訳でマンガを多言語展開

マンガで外国語学習

Mantra Engine

Langaku5CONFIDENTIAL

Problems: Slow & expensive translation

2+ weeks

per episode2-3x cost
vs. plaintext

6

CONFIDENTIAL

Why is transla3on so ineﬃcient?

Translator

Type-setter

Proof-reader

7

CONFIDENTIAL

Why is translation so inefficient?

Translator

Type-se8er

Proof-reader• Difficult translation + lettering• Back-and-forth file sharing via email• Various file formats for various software

8CONFIDENTIAL

Product: CAT* tool for comic translation* Computer assisted translation

ü Machine translation &type-setting for comics
ü No email communication
ü No format conversion2x faster transla6on!
10

CONFIDENTIAL

Topics
1. Motivation

Why comic translation?

2. Challenges

How difficult is (automated) comic translation?

3. Approaches

Towards machine translation for comics translation

4. Future directions

11

CONFIDENTIAL

Is comic translation difficult?
Reference

Subject “I” is omi0ed

I just wanted to...
MT©Mitsuki Kuchitaka

A sentence is divided into two balloons

Referencepick her up...

Context is required for accurate translation

12

CONFIDENTIAL

What makes comic translation difficult?
v Challenges for comic translaJon- High dependence on context -> Context-aware MT- MulK-modality-> Multi-modal MT• e.g., Speaker-speciﬁc wordings- Low-resource-> Low-resource MT• No comic-domain training/evaluaDon data available
13

CONFIDENTIALTopics
1. Motivation

Why comic translation?

2. Challenges

How difficult is (automated) comic translation?

3. Approaches

Towards machine translation for comics translation

4. Future directions

14

CONFIDENTIAL

Approaches to automaJc comic translaJon

1. Evaluation dataset/framework
2. Context-aware comic translation
3. Visual-aware comic translation
15

CONFIDENTIAL

OpenMantra: Evaluation dataset for manga translationv Five manga (Japanese comics) titles
w/ manual annotation & translation
Romance

Mystery©Mitsuki Kuchitaka, Ito Kira, Nako Nameko

Fantasy

Slice of life

Battle16

CONFIDENTIAL

OpenMantra: Evaluation dataset for manga translationv Five manga (Japanese comics) Jtles
w/ manual annotaJon & translaJon
t1

t2

t5

t6

t4

t10
t11

Ja:

(#1) こちらにいるレーネさんを(#2) 迎えにあがったのですが...

En:(#1) I just wanted to...
(#2) pick her up...

Zh:(#1) 我是来接(#2) 蕾娜⼩姐的...

t3

t7
t9
t8

17

CONFIDENTIAL

How to evaluate translated texts?
v Challenge: Texts are oSen swapped aSer
translaJon
Ja:Zh:(balloon #1)(balloon #2)

こちらにいるレーネさんを迎えにあがったのですが...

我是来接蕾娜⼩姐的...

Balloon-by-balloon evaluation does not work with comics

18

CONFIDENTIAL

EvaluaJon toolkit for page-by-page evaluaJon
Score↑

5
4
Aggregate

3
2
1

#1

#2

#3

Method#4#5

19

CONFIDENTIAL

Approaches to automatic comic translation

1. Evaluation dataset/framework
2. Context-aware comic translation
3. Visual-aware comic translation
20

CONFIDENTIAL

Importance of context
2

1

A sentence consists oftwo balloonsJa:

こちらにいるレーネさんを迎えにあがったのですが...

Reference:

I just wanted to...

pick her up...

MT:

Renee-san here...

I was picked up...

Which balloons should be used as “context”?

21

CONFIDENTIAL

Proposed: Scene-based MTv Idea: Assuming each frame as a “scene” andconsidering all the texts in it as context
t1

t2

Page 8 (en)Page 8 (ja)Scene 1

Scene 1

t4

t10
t11

t3

Scene 2t3 t4 t5 t6

t7

Scene 3t7 t8

Scene 4t9
t8t9

Scene 5t10 t11

Context-aware NMTt5t6

t'1 t'2

t1 t2

Scene 2t'3 t'4 t'5 t'6Scene 3t'7 t'8

Scene 4t'9

Scene 5t'10 t'11

22

CONFIDENTIAL

Frame detection and Text ordering
Frame detec@ons1

Text ordering

˙

t5t6

s4v Frame detecDon- Faster R-CNN [Ren+ 15] trained on

s2

s5

t1

t2

t4

t10

s3
t11

t3

t7
t9
t8

Manga109 dataset [Matsui+ 17]v Text ordering- First order frames then balloons- A simple heurisBc: order balloons

by the distance from the upper
right point of each frame

23

CONFIDENTIAL

Context-aware NMT v. sentence-level NMTv Experimental se.ngs- Baseline (sentence-level NMT)
• Transformer (big) [Vaswani+ 17]

v Manual evalua>on results
Score↑
5• Trained data: manga corpus (4M)

4- 2+2 [Tiedemann & Scherrer 17]

3• Simply concatenate 2 sentences

2- Scene-based NMT (Proposed)1• Concatenate all the text in eachframe

Sentence-level
NMT
Context-aware NMT

2.76

2.85

Baseline

2+2

2.98

Scene24

CONFIDENTIAL

How context works in the scene-NMT
Ja:

こちらにいるレーネさんをReference:

I just wanted to...

pick her up...

MT:

Renee-san here...

I was picked up...

I’ve come to get

Lena-san here, but...

Scene-MT:

迎えにあがったのですが...

Two balloons to compose
a meaningful sentence25

CONFIDENTIAL

Approaches to automatic comic translation

1. Evaluation dataset/framework
2. Context-aware comic translation
3. Visual-aware comic translation
26

CONFIDENTIAL

Importance of visual information
No subject nor object!
Ja: 近づきget close toたくwant toないnot

MT: I... I don’t want to go near him!
©Satoshi Arai“him” ?

Can we combine visual information into scene-NMT?

27

CONFIDENTIAL

Proposed: Visual-aware Scene-based NMT
v Idea: Extracting visual information for eachscene and input the extracted tags to NMT
t1t2

t5

t6t4

t10t11Page 8 (ja)<BLACK EYES><1BOY><1GIRL><LONG HAIR><1GIRL>

Scene 1
<1GIRL>t3

Scene 2DetectTOP-N<1BOY>

Scene 3t7

<1GIRL><1GIRL><LONG HAIR>

t9<1GIRL>t8<1BOY><1GIRL>

Scene 4
<1GIRL>

t1 t2
t3 t4 t5 t6
t7 t8t9

Scene 5Illustration2Vec [Saito & Matsui 17]<LONG HAIR><1GIRL>t10 t1128

CONFIDENTIAL

Example of visual-aware MT
Extracted
visual tags
No subject
nor object!

©Satoshi Arai

w/o visual tags: I... I don’t want to go near him!
w/ visual tags: I-I don’t want to get close to her!

29

CONFIDENTIAL

Textual-NMT v. Visual-aware NMTv Result: The visual tags don’t contribute tooverall performance
Score
54

Textual-NMT

Visual-aware NMT

3

2

1

2.76

2.85

2.98

2.91

Baseline

2+2

Scene

Scene
w/ object tags

30

CONFIDENTIAL

Analysis: NegaJve impact of visual tags
Input

Reference2112©Nako Nameko

Predicted tag<MULTIPLE_GIRLS>

Maybe he was tired.
I'll wake him later.

Scene-based NMT w/o visual tags
1

2

Maybe he was just tired.
We’ll wake him up later.

Scene-based NMT w/ visual tags
1

2

Maybe she’s Dred.
We’ll wake her up later.
31

CONFIDENTIAL

Analysis: NegaJve impact of visual tags
Input
1＋

2

Maleface©Nako Nameko

Predicted tag

Predicted tag<1BOY><MULTIPLE_GIRLS>

Scene-based NMT w/ visual tags
1

2

Maybe she’s Dred.
We’ll wake her up later.

Scene-based NMT w/ visual tags
12Maybe he was just tired.
We’ll wake him up later.

BeEer image encoder/visual representa>on are required

32

CONFIDENTIAL

Approaches to automaJc comic translaJon

1. Evalua6on dataset/framework
2. Context-aware comic transla6on
3. Visual-aware comic transla6on33

CONFIDENTIAL

Topics
1. Motivation

Why comic translation?

2. Challenges

How difficult is (automated) comic translation?

3. Approaches

Towards machine translation for comics translation

4. Future directions

34

CONFIDENTIAL

Limita6on of current scene-NMT model
Page 8 (ja)Scene 1
<1GIRL>

Scene 2
<1BOY>Scene 3<1GIRL>

Scene 4
<1GIRL>t1 t2

vVisual informaJon- Discrete tags of the detected objects

t3 t4 t5 t6
t7 t8
t9

Scene 5<LONG HAIR><1GIRL>t10 t11v Textual informaJon- Limited length of context• i.e., texts in a single frame

35

CONFIDENTIAL

Richer visual repr. & contextual encoders
v Visual informaJon- Discrete tags of the detected objects
Continuous distributions/embeddings for comicv Textual informaJon
images should be explored- Limited length of context
Document-level MT architectures• i.e., texts in a single frame that capture longer/hierarchicalcontext can be applied36

CONFIDENTIAL

BeUer use of mul6-modality
v Emo>on predic>onAngerPoliteness

SurprisingAnxiety

v Speaker/listener detec>onspeaker[Yamanishi+ 17]

listener[Rigaud+ 15]

37

CONFIDENTIAL

Online/incremental domain adaptationv Real-Jme domain adaptaJon improves
producJvity of human translators- e.g., [Turchi+ 17; Kothur+ 18; Karimova+ 18]
Adaptedtransla@ons
Model update

38

CONFIDENTIAL

Machine transla6on for entertainments
v Mul>-modality of entertainment contentsComics• Text
• Images

Movies• Video• AudioVideogames• Text• Images• Video• Audio

Challenge:
How to deﬁne, extract, and model mul>-modal context?

39

CONFIDENTIALTopics
1. Mo6va6on

Why comic translaDon?

2. Challenges

How diﬃcult is (automated) comic translaDon?

3. Approaches

Towards machine translaDon for comics translaDon

4. Future direc8ons

40