Video
3.0 x 4.0

Wed-A-V-6-7:
Transcribing Paralinguistic Acoustic Cues
to Target Language Text in Transformer-based
Speech-to-Text Translation
Hirotaka Tokuyama 1 , Sakriani Sakti 1,2
Katsuhito Sudoh 1,2 Satoshi Nakamura 1,2
1 Nara Institute of Science and Technology, Japan
2 RIKEN, Center for Advanced Intelligence Project (AIP), Japan

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

1

Background: Speech-to-Speech Translation
SourcespeechTargettext
Source
text

ASR

MT

3.0 x 4.0

Sourcetext
Targettext→ 今日は暑いです → It is hot today

Video

TTS

Targetspeech→•Speech-to-Speech (S2S) Translation has 3 systems◦ ASR : Automatic Speech Recognition◦ MT : Machine Translation◦ TTS : Text-to-Speech

Yes we can•But, conventional system has the same problems◦ These systems based on linguistic information◦ What is being said and the actual words

Yes we can!

◦ They cannot reflect Paralinguistic information◦ Emphasis, Emotion etc…

◦ Different intonation or emphasis can bring different
meaning to the linguistic contents
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

2

Background: Previous Cascade S2S Structure

Video•Acoustic-to-Acoustic Emphasis translation [1]3.0 x 4.0

◦ Emphasis in input speech reflects output speech

MT

ASR

TTS
it is hot today .

kyou ha atsui desu .
Emphasis
Estimation

Emphasis
Translation•This framework is complicated and suboptimal◦ Estimate and translate emphasis separately◦ Neural MT with HMM-based ASR-TTS◦ Paralinguistic information was kept in acoustic form

•Is emphasis expressed in only acoustic (intensity) ?
◦ Some language may be expressed in other ways[2]9/1/2021[1] Sequence-to-Sequence Models for Emphasis Speech Translation [Q. T. Do 2018][2] Toward Multi-Features Emphasis Speech Translation: Assessment of Human
Emphasis Production and Perception with Speech and Text Clues [Q. T. Do 2018]

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

3

Background: Paralinguistic Researches

Video•Court transcription of evidentiary audio recordings [3]3.0 x 4.0

◦ the contribution of prosodic and paralinguistic cues to the translation of evidentiary audio recordings was
critical◦ It suggests thatcreating “written to be read as if spoken” text, instead of “written to be read”◦ “written to be read” text cannot reflect speakers' intentions, moods, power and attitudes•It is important to constructing speech-to-text translation systems thatconvey acoustic information from speech into text-based communication•The contributions of this topic◦ Expressing paralinguistic acoustic cues to text
◦ Constructing cascade and direct neural speech-to-text translation◦ Performing Japanese-to-English linguistic and paralinguistic speech-to-text translation framework[3] Contribution of prosodic and paralinguistic cues to the translation of evidentiary audio recordings [R. Chakhachiro 2016}

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan4

Propose: “Emphasis” of speech and text

Video•“Emphasis” is different between spoken and written form

3.0 x 4.0

◦ Spoken: changed pitches, durations, and levels of power◦ Written: manifested by intensifier in text

•Ex. “It is hot today” with emphasis◦ Spoken: linguistic contexts are the same, but acoustic form is different◦ Written: adding intensifiers
Level

Emphasized Speech

Emphasized Text

0: Normal

It is hot today.

1: Light

It is a little bit hot today.

2: Medium

It is quite hot today.

3: Strong

It is very hot today.

4: Very strong

It is extremely hot today.

•Need to transcribe paralinguisitic acoustic cues to text
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan5

Propose: Transcribing Paralinguistic Acoustic Cues

Video

•Emphasis acoustic cues to embedded emphasis text

3.0 x 4.0

MethodDescription

Ex. When emphasis level of “hot” is 3

No Emphasis

Original text

it is hot today .

original text and emphasis symbols
Emphasis Separated
separately as in the previous method

it is hot today .

Emphasis Tags

Add emphasis tags to each word

it0 is0 hot3 today0 .0

Emphasis 1-Token

Add a token before emphasized word

it is <to3> hot today .

Emphasis All Token

Add a token before every word<to0> it <to0> is <to3> hot <to0> today <to0> .

00300•Emphasis embedding text to natural text
it0 is0 hot3 today0 .0
it is <to3> hot today .

It is very hot today.

<to0> it <to0> is <to3> hot <to0> today <to0> .

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

6

Propose Structure1 : Cascade Neural-S2T

Video•Simplify by using Neural Network in all systems

3.0 x 4.0

◦ Emphasis Estimation and Translation are done in ASR and MT◦ Neural ASR and MT systems are trained separately earlier in each embedded emphasis text
きょう は あつい です 。

HMM
ASR

NeuralMT

Emphasis
Estimation

Emphasis
Translation
00300

Neural ASR+ Emphasis
Estimation
9/1/2021

Neural MT+ Emphasis
Translation

It is hot today
00300

It0 is0 hot3 today0 .0

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

7

Propose Structure2 : Direct Neural-S2T

Video•Direct Speech-to-Text (S2T) translation using Neural Network

3.0 x 4.0

◦ Combining ASR and MT systems including emphasis processes as single model◦ Translation that reflects the emphasis of speech◦ Using Jia et al.’s direct model as a reference [4]きょう0 は0 あつい3 です0 。0Neural ASR+ Emphasis
Estimation

Neural MT+ Emphasis
Translation

Neural ASR + MT
+Emphasis Estimation &Translation

It0 is0 hot3 today0 .0

It0 is0 hot3 today0 .0[4] Direct speech-to-speech translation with a sequence-to-sequence model [Ye Jia 2019]

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

8

Propose : Acoustic-to-Linguistic Emphasis

Video•Acoustic Emphasis to Linguistic Emphasis (NMT-insertion)

3.0 x 4.0

◦ Transform embedded emphasis text into natural text with intensifiers
きょう0 は0 あつい3 です0 。0Neural ASR+ Emphasis
Estimation

Neural MT+ Emphasis
Translation

Cascade Neural S2TOR

It0 is0 hot3
today0 .0

NMT
insertion

it is very hot
today .

Direct Neural S2T
Neural ASR + MT
+Emphasis Estimation& Translation

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

9

Datasets and Models

Video•Dataset

3.0 x 4.0

◦ Natural Speech Dataset◦ Japanese-English natural speech and text dataset including emphasis◦ Text is prepared with emphasis part and intensifiers◦ 5,145 utterances (1,029 sentences x 5 levels)◦ 20x over sampling for training dataset

Training Dataset

Data Amount

Natural Speech

5,145

BTEC

465,466

APP-BLA

143,171◦ BTEC (Basic Travel Expressions Corpus)◦ Over 400k English-Japanese Parallel Corpus◦ ATR Speech Database of Many Speakers (APP-BLA)◦ 3,700 Japanese speakers reading 503 ATR’s phonetically balanced sentences (without emphasis)◦ 127 speech hours in total•Data Augmentation◦ Natural speech with emphasis is too smaller than BTEC and APP-BLA◦ Augment emphasized speech and text dataset by BTEC

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

10

Create Synthetic Emphasized Speech
1.
2.
3.

Video
3.0 x 4.0

Split before and after the first adjective
Generate speech each part by google TTS (gTTS)
Make emphasized speeches only adjective part for 5 levels◦ Normal, Light, Medium, Strong and Very strong

4.

Concatenate each generated speech
今日 は 暑い です。
kyou ha atsui desuPre
きょうはkyou ha

Emp
あつい
atsui

Post
です
desu

Synthetic Speechkyou ha atsui desu (level 0)

kyou ha atsui desu (level 2)
Generate adjective
speeches with
5 levels of emphasis

9/1/2021

kyou ha atsui desu (level 4)

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

11

Create Synthetic Emphasized Text
1.
2.
3.
4.

Split before and after the first adjective
Add emphasis information in each word (5 levels just like speech)
The same process in English text
Align English and Japanese textVideo
3.0 x 4.0

◦ Some adjectives may not be used only in either Japanese or English text
Emphasis Tag
今日 は 暑い です。
kyou ha atsui desuPre
きょうはkyou ha

Emp
あつい
atsui

Post
です
desu

きょう0 は0 あつい3 です0 。0It0 is0 hot3 today0 .0
Emphasis Token
きょう は <to3> あつい です 。
It is <to3> hot today .
All Token<to0> きょう <to0> は <to3> あつい<to0> です <to0> 。
<to0> it <to0> is <to3> hot <to0> today <to0> .

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

12

Create Synthetic “Inserted” Text

Video

•Analyze and extract how many adverbs using in natural speech texts

3.0 x 4.0

◦ Use top 5 of them based on respective emphases
Natural Speech Text (it is existed in advance)
Ex.1
Normal (level 0): This is loose. (plain text)
Light (level 1): This is a tad loose.
Medium (level 2): This is noticeably loose.
Strong (level 3): This is so loose.
Very strong (level 4) : This is completely loose.

Analyze the inserted word

Ex.2
My cholesterol is high.
My cholesterol is a bit high.
My cholesterol is quite high.
My cholesterol is certainly high.
My cholesterol is extremely high.

Light: a tad, a bit …
Medium : noticeably, quite …
Strong: so, certainly …
Very strong : completely, extremely …

•Insert one of them before the adjectives in target data randomly
we feel sick .

9/1/2021

Normal : we feel sick. (plain text)Light: we feel a bit sick.
Medium : we feel quite sick.
Strong: we feel so sick.
Very strong: we feel completely sick.

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

13

Experimental Set-up

Video•Dataset

3.0 x 4.0
Training Dataset◦ Train Dataset◦ ASR: 718,142 speeches and texts◦ MT / S2T : 574,971 text pairs◦ Natural speech dataset is 20x oversampling◦ Test Dataset

ASR MT S2T Data Amount

BTEC (emphasis)

O

O

O

184,620

BTEC (no emphasis)

O

O

O

297,451

Natural Speech

O

O

O

92,900APP-BLAO

X

X

143,171◦ Natural : 500 sentences (100 sentences x 5 levels with Natural speeches and multi-speaker task)◦ Synthetic : 510 sentences with 5 levels (from BTEC test set with Synthetic speeches and single-speaker task)•Model◦ Implement OpenNMT-py using Transformer◦ Parameter is the same each model•Evaluation◦ ASR : word error rate (WER)◦ MT, S2T: multi-bleu, sacrebleu◦ Emphasis: F-score
9/1/2021

Transformer parameter

Layer

3

RNN size

512

Word vec size

512

Transformer_ff

2048

Heads

8

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan14

Evaluation : ASR & MT (Independently)

Video

Emphasis Tags
•ASR

3.0 x 4.0

Emphasis Token
Emphasis All Token•MTit is hot today .

: it0 is0ASR
hot3 today0 .0
Neural:+ itEmphasis
is <to3> hot today
Text .
<to0> it <to0> is <to3> hot <to0> today <to0> .
:Estimation

Text

00300

Linguistic evaluation
Neural MT
Text
Text+ Emphasis
Emphasis
evaluation
Translation

Linguistic Evaluation

Test data

Data typePrevious
Synthetic(BTEC)

Proposed

Previous
NaturalProposed

MT

WER⇓

Multi-bleu⇑

Sacrebleu⇑

ASR

MT

No Emphasis

0.91

44.84

47.80

---

---

Emphasis Separated

0.91

43.99

46.71

90.91

23.18

Emphasis Tags

0.80

42.64

45.57

100.0064.52

Emphasis 1-Token

0.76

42.31

45.64

93.93

70.77

Emphasis All Token

0.83

44.84

47.71

96.97

74.63

18.11

31.81

34.68

---

---

Emphasis Separated No Emphasis18.88

it is hottoday .
28.74

31.76

60.45

34.31

Emphasis Tags

it is hottoday .
34.12

36.54

71.90

36.76

00300

38.71

64.95

49.48

36.91

63.90

44.22

No Emphasis

Emphasis 1-Token
Emphasis All Token

9/1/2021

ASR

Emphasis Evaluation(F-score)

Method17.05
Emphasis Separated

22.31

36.05

Emphasis Separated (Previous)

Emphasis Tags

it0 is0 hot3 today0 .0

Emphasis 1-Token

it is <to3> hot today .

Emphasis All Token<to0> it <to0> is <to3> hot <to0> today <to0> .

19.87

34.35

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan15

Evaluation : Speech-to-Text Translation•Direct Neural S2T•Cascade Neural S2T
Neural ASR+ Emphasis
Estimation

Video

Neural MT+ Emphasis
Translation

3.0 x 4.0

Neural ASR + MT
+ Emphasis Estimation& Translation

Text

Linguistic Evaluation
Test data

Previous

Synthetic(BTEC)
Proposed

Previous
NaturalProposed

9/1/2021

CascadeData type

Text
Emphasis Evaluation(F-score)

DirectMulti-bleu

Sacrebleu

Multi-bleu

Sacrebleu

CascadeDirectNo Emphasis

42.98

45.92

44.62

47.94

---

---

Emphasis Separated

42.46

45.40

---

---

22.22

---

Emphasis Tags

41.76

44.67

44.55

47.28

58.06

47.13

Emphasis 1-Token

44.13

46.77

44.82

47.84

67.69

49.01

Emphasis All Token

43.88

46.93

39.79

43.09

69.70

41.58

No Emphasis

19.53

21.72

7.56

9.54

---

---

Emphasis Separated

18.41

20.55

---

---

34.81

---

Emphasis Tags

15.28

16.43

9.07

9.88

61.54

15.28

Emphasis 1-Token

15.12

18.07

12.00

13.71

58.46

15.12

Emphasis All Token

17.66

20.31

5.93

7.62

57.97

17.66

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan16

Evaluation : NMT insertion

Video
3.0 x 4.0

Text

Test data

Synthetic(BTEC)NaturalMT+insertion

Inserted
Text

SystemEmphasis Level

Total

0

1

2

3

4

Emphasis Tags

98.18

87.88

93.94

94.94

90.91

96.81

Emphasis 1-Token

99.20

90.91

100.00100.00100.0098.90

Emphasis All Token

94.33

93.94

96.97

90.90

93.94

94.25

Emphasis Tags

84.53

67.03

81.11

80.90

75.28

77.85

Emphasis 1-Token

93.62

82.98

92.55

91.30

91.30

90.34

Emphasis All Token

88.04

73.91

79.35

81.11

82.22

80.92

•Evaluation◦ Calculate the percentage of Inserted word is matched each emphasis level without changing the original text

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan17

Evaluation Summary

Video•Recognition and Translation of paralinguistic informationis possible by using emphasized text data

3.0 x 4.0

◦ Compared to no emphasis, the score is almost about the same or better•Considering S2T and insertion, using tokens is better performance◦ Especially 1-token works the best in proposed method•But, there is a gap between Natural and Synthetic data evaluation◦ The performance of ASR affects S2T◦ Synthetic evaluation is a single speaker task while natural is a multi speaker task
◦ A large portion of training data is synthetic data
◦ More natural data is necessary

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

18

Conclusion and Future WorkVideo

•Purpose

3.0 x 4.0

◦ Translate with acoustic and linguistic emphasis◦ Use neural Transformer-based model in Cascade and Direct Speech-to-Text Translation•Method :◦ Transcribe paralinguistic acoustic cues to text
◦ Generate synthetic speech dataset with emphases by google TTS◦ Perform neural Transformer-based paralinguistic speech-to-text translation•Results◦ Our propose performed better than the baseline in terms of linguistic and paralinguistic evaluation•Future Work :◦ Analyze how humans change acoustic cues to emphasize words◦ Evaluate results and speech data subjectively◦ Implement TTS for paralinguistic speech-to-speech translation

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan19

Video3.0 x 4.0

Appendix

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

20

Vocabulary SizeVideo

•Difference of vocabulary in each method

3.0 x 4.0

MethodJapanese

EnglishNo emphasis

25471

29091

Emphasis Tag

27767

37503

Emphasis 1-Token

25475

29095

Emphasis All-Token

25476

29096

•In Emphasis Tag, emphasized word are different from original word
◦ Ex. “hot0” and “hot1” are recognized as different word
◦ Lighter structure is indispensable because of larger model

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

21

Adverb List from Natural Speech Text

Video

•Light (Level 1)

3.0 x 4.0

◦ 'a little', 'slightly', 'a tad', 'a bit', 'kind of', 'sort of', 'vaguely', 'almost', 'marginally', 'mildly', 'passably', 'faintly','a tiny', 'a touch', 'something of', 'just a', 'bit of', 'a slightly', 'more or', 'perceptibly', 'hardly', 'even a','imperceptibly', 'a mildly’ …

•Medium (Level 2)◦ 'pretty', 'fairly', 'rather', 'quite', 'somewhat', 'relatively', 'moderately', 'reasonably', 'significantly', 'tolerably','noticeably', 'considerably', 'uncommonly', 'more than’, 'unusually', 'admittedly', 'comparatively', 'visibly','partly', 'mostly', 'probably’, …

•Strong (Level 3)◦ 'so', 'very', 'really', 'truly', 'super’, 'largely', 'definitely', 'clearly', 'much', 'considerably', 'decidedly', 'certainly','undeniably', 'positively', 'deeply', 'greatly', 'undoubtedly', 'unquestionably', 'unmistakably', 'so very', 'such','vastly’, 'prohibitively', 'assuredly', 'mostly’ ….

•Very strong (Level 4)◦ 'seriously', 'terribly', 'extremely', 'completely', 'impossibly', 'perfectly', 'supremely', 'obviously', 'shockingly','totally', 'awfully', 'exceptionally', 'horribly', 'dreadfully', 'unbearably', 'wildly', 'powerfully', 'entirely','amazingly', 'wonderfully', 'hideously’ …
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

22

Analysis : ASR Results

Video•Input speech (Natural Speech)

3.0 x 4.0

◦ この きかく は せいさん の めん で まずい 。(この企画は生産の面でまずい。)◦ “まずい” has emphasis level 3
Data type
No Emphasis
Previous

Proposed

Output
この きたく は せいさん の えん で まずい 。
この きかく は せいさん よん ねん で まずい 。

Emphasis Separated(Emphasis Estimation ) 0 0 0 0 0 0 0 3 0

Emphasis Tags

この0 きたく0 は0 せいふ0 の0 ねんれい0 まずい3 。0Emphasis 1-Token

この きかく は それ で さん ねん まえ で <to3> まずい 。

Emphasis All Token

この <to0> ちかく <to0> は <to0> せーるすまん <to0> の <to0> ねんれい <to0> が <to3> まずい <to0> 。

•Emphasis recognition is mostly correct but language recognition is often different◦ More natural speech data is needed because the percentage of natural speech is small•When testing natural speech data, sometimes recognize different emphasis level

9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

23

Analysis : MT Results

Video•Input text -> Correct output

3.0 x 4.0

◦ この きかく は せいさん の めん で まずい 。 -> This scheme is clumsy production wise .
◦ Red word is emphasis part (level 3)
Data type
No Emphasis
Previous

Emphasis Separated

Emphasis Tags
Proposed

OutputThis project lacks control .
This project is going to be tasteless(Emphasis Translation) 0 0 3 3 0 0 0

Something0 is0 wrong3 with0 this0 plan0 .0

Emphasis 1-Token

This project is <to3> tasteless .

Emphasis All Token<to0> This <to0> project <to0> is <to0> completely <to3> wrong <to0> .

•Translate depends on train dataset, but emphasis part is translated even if used different word
•Separated depends on Attention and sometimes translate emphasis towrong emphasis part or other token (ex. <unk>)•Emphasized “clumsy” word is not in train data
•Including Emphasis translation, proposed is better
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

24

Analysis : S2T Task with Synthetic Speech

Video•Input speech (Synthetic Speech) -> Correct output text

3.0 x 4.0

◦ やすい れすとらん を しょうかい し て いただけ ます か 。
-> could you recommend an inexpensive restaurant ?
◦ Red word is emphasis part (level 3)
Data typeOutputNo Emphasis
CascadePrevious

Emphasis Separated
Emphasis Tags

Proposed

Previous

Directcould you recommend a cheap restaurant ?

Proposed

could you recommend a cheap restaurant ?
(Emphasis Translation) 
 
 
 3 3 0 0 0
could0 you0 recommend0 a0 inexpensive3 restaurant0 ?0

Emphasis 1-Token

could you recommend a <to3> cheap restaurant ?

Emphasis All Token<to0> could <to0> you <to0> recommend <to0> an <to3> inexpensive <to0> restaurant <to0> ?

No Emphasis

could you recommend an inexpensive restaurant ?

Emphasis Tags

could0 you0 recommend0 a0 cheap3 restaurant0 ?0

Emphasis 1-Token

could you recommend a <to3> cheap restaurant ?

Emphasis All Token<to0> could <to0> you <to0> recommend <to0> an <to3> inexpensive <to0> restaurant <to0> ?

•When ASR has good results, S2T Task works well
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan25

Analysis : S2T Task with Natural Speech

Video•Input speech (Natural Speech) -> Correct output text

3.0 x 4.0

◦ この きかく は せいさん の めん で まずい 。-> This scheme is clumsy production wise .
◦ Red word is emphasis part (level 3)
Data typeOutputNo Emphasis

CascadePrevious

Emphasis Separated

Emphasis Tags
Proposed
Previous

Directi have to pay my bill now .

Proposedi have to pay my bill three times before i go home .
(emphasis translation) 0 0 0 0 0 0 0 0 0 0 0 0

the0 neighbors0 are0 tasteless3 .0

Emphasis 1-Token

these are the <to3> tastes bad around here .

Emphasis All Token<to0> salesman <to0> ‘s <to3> events <to0> in <to0> this <to0> neighborhood <to0> .

No Emphasis

this lamp sends out a powerful beam .

Emphasis Tags

this0 question0 is0 difficult3 to0 answer0 .0

Emphasis 1-Token

this tea is <to3> unusual .

Emphasis All Token<to0> this <to0> question <to0> is <to3> difficult <to0> to <to> answer <to0> .

•S2T task depends on ASR results◦ When ASR outputs wrong text, MT outputs worse texts …

•We need more natural speech dataset for ASR
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

26

Analysis - NMT insertion

Video•Input text (English text) -> Correct output text (English inserted text)

3.0 x 4.0

◦ this scheme is clumsy production wise . -> this scheme is so clumsy production wise .
◦ Inserted words are 'so', 'very', 'really', 'undeniably', 'positively’, ‘unquestionably’ and etc…
Data typeOutputEmphasis Tags

this scheme is detail production wise.

Emphasis 1-Token

this scheme is so clumsy production wise .

Emphasis All Token

this scheme is so clumsy production wise .

•Almost good results when using token◦ Guessing there are correspondence between the token and the word to be inserted•Sometimes other words occurs when using tags
◦ Emphasized “clumsy” is not in train data, so it may make to replace other word
◦ Ex. “clumsy0” is in train data, but “clumsy1” is not in train data

◦ But, some emphasized word works well
◦ Ex. this0 is0 loose4 .0 -> this is stupendously loose. (呆気なく,凄まじく)
9/1/2021

INTERSPEECH 2021 Wed-A-V-6-7 @ Hirotaka Tokuyama AHC LAB, NAIST, Japan

27