1

Towards Incremental ASR and TTS
for Real-time Interaction
Satoshi Nakamura1,2, with
Sakriani Sakti1,2, Tomoya Yanagita1, Novitasari Sashi1
1Nara Institute of Science and Technology, Japan
2RIKEN, Advanced Intelligence Project AIP, Japan

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

2

TopicsIncremental Speech Processing for Real-time Interaction– Incremental ASR– Incremental TTS

Application– Simultaneous Speech Translation– Spoken Dialogue System

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

Real-time Machine Speech Interpreter3▪ Traditional Speech Translation
SpeechRecognition(ASR)

I go toschoolMachineTranslation
(MT)

私は
学校に行く

SpeechSynthesis(TTS)▪ Real-time Machine Speech Interpreter

SpeechRecognition
(ASR)

I go

SpeechRecognition(ASR)

MachineTranslation
(MT)

to
shool

私は

MachineTranslation(MT)

SpeechSynthesis
(TTS)

学校
に行く

SpeechSynthesis(TTS)

Sakriani
Sakti @ AHC Labs,
Japan | Speaker Odyssey– Tutorial| November
1st-5th,2020
無限の可能性、ここが最先端 －Outgrow yourlimits－AIST 2020©SatoshiNakamura,
AHC Lab,NAIST, Japan
Nov.NAIST,
19. 2020

http://www.naist.jp/

4

Dialogue System

Incremental
interaction

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

5

Neural Incremental Speech Recognition

Neural Incremental Speech Recognition

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan5
http://www.naist.jp/

6

Neural Incremental Speech Recognition
Automatic Speech Recognition System●●

ASR system transcribes speech into text
Task examples:o Spoken dialog systemo Speech translationo Closed-caption generation, etc.

(speech)ASR

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

“Hello”

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan6
http://www.naist.jp/

7

Neural Incremental Speech Recognition
Automatic Speech Recognition System (2)●High delay speech recognition(Standard seq2seq ASR)

State-of-the-art
Sequence-to-sequence neural ASR (end-to-end)o Standard encoder-decoder with a global attention mechanismo Output prediction starts after the input speech finish→
High accuracy but high delaye.g. a 5 minutes speech requires more than 5 minutes to berecognizedoUnsuitable for real-time tasks− Real-time speech translation− Live video closed-caption generation− Real-time meeting transcription generation, etc.

Today I am going …
Time

Delay

Low delay speech recognition(ISR)

Today I am going to ….

●

Incremental ASR (ISR) for low-delay speech recognition
Delay

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, JapanTime

7
http://www.naist.jp/

8

Neural Incremental Speech Recognition
Incremental Speech Recognition● ISR begins the speech recognition without waiting the speech to finish (low delay)o Recognize the speech part-by-part in several incremental steps
o Input: a short part of the speech● Challenge: How to do an incremental step?
1)

Input boundary decision
When the transcription of a short speech partcan be produced?
Give
output?

2)

Output boundary decision
When to stop the output prediction of the current
speech part and and move to the next?
ABCD…

SpeechWhen tostop?

SpeechGive
output?

When to
stop?

KLMN…

Need to learn short input-short output alignments
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan8
http://www.naist.jp/

9Neural Incremental Speech Recognition
Incremental Speech Recognition

Related Works

A. Statistical approach (Pipeline)❖ Hidden Markov model (HMM) ASR [Rabiner, 1989; Juang and Rabiner, 1991]❖ 3 parts: Acoustic model, lexicon, language model

Featureextraction

Acoustic
model

Input speech
HMM

Lexicontoday: /təˈdeɪ/
going: /ˈɡəʊɪŋ/
Pronunciation
labelsLanguagemodel
N-gram model“Today I am going totalk about speech
recognition”
Output text

❖ Low delay speech recognition by performing left-to-right input processing (unidirectional)❖ Not end-to-end Young, 2008
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan9
http://www.naist.jp/

10

Neural Incremental Speech Recognition
How to achieve an ISR system that can:
1. reduce delay,
2. keep the system complexity, and3. maintain a close performanceof the standard neural ASR system?
Proposal

Neural ISR construction by employing sources (architecture, knowledge)
from standard neural ASR.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan10
http://www.naist.jp/

11

Neural Incremental Speech Recognition

Attention-Transfer Incremental Speech Recognition
Sashi Novitasari, Andros Tjandra, Sakriani Sakti, Satoshi Nakamura,“Sequence-to-sequence Learning via Attention Transfer for Incremental
Speech Recognition”, Interspeech 2019, Graz, Austria, DOI:
10.21437/Interspeech.2019-2985, 3835-3839, Sep. 2019
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan11
http://www.naist.jp/

12

Neural Incremental Speech Recognition

Attention-Transfer Incremental Speech Recognition
Attention-Transfer Incremental Speech Recognition (AT-ISR)[Novitasari et al., 2019]●

AT-ISR Training

Aim

This isboundary!

ISR (student) learns to mimic the attention-based alignment generated
by a standard seq2seq ASR (teacher)oo

ISR architecture
Incremental step

: Same as the teacher (seq2seq): Learn through attention transfer from theteacher ASR● Attention transfer : Attention knowledge transfer from teacher tostudent modelo Prev. works → image recognition tasks▪ Teach another model [Zaguruyko and Komodakis, 2017]▪ Domain transfer (image to video) [Li et al., 2017]o Has not been utilized for ISR construction yet

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

ABCDEFG

ABCD|

EFG|

ASR

ISR

ISR

Att

This isboundary!

Standard ASR(TEACHER)AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, JapanISR(STUDENT)

12

http://www.naist.jp/

13

Neural Incremental Speech Recognition

Attention-Transfer Incremental Speech Recognition

Attention Matrix

Overview

Seq2seq ASR: Encoder-Decoder with Attention

Dec.

Output: Character (basic model)
Components● Encoder (recurrent network)
Encode input features sequence (𝑿) into hidden states (𝒉𝑒 )● Decoder (recurrent network)
Predict token sequence (𝒀) based on input context
information (𝑐𝑡 ) and prev. text (𝒀<𝑡 ) for each t-th token● Attention (linear network)
For each t-th token prediction, compute 𝑐𝑡 based onalignment scores of 𝒉𝑒 , ℎ𝑡𝑑

Decoder

Enc.

Attention

EncoderStandard Seq2seq ASR[Bahdanau et al, 2015],
courtesy of [Tjandra et al.,2017]

𝑠 = Encoding timestep; 𝑡 = Decoding timestep
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan13
http://www.naist.jp/

14

Neural Incremental Speech Recognition

Attention-Transfer Incremental Speech Recognition

AT-ISR Recognition Method</m>Given: Full speech (𝑿), length 𝑆●

Recognize the speech segment-by-segment sequentially basedon a fix-sized input window
For each incremental recognition step n:

Decoder

1.

Encode 𝑿𝑛 , a 𝑊 speech frames from 𝑿 (𝑊 < 𝑆)

Attention2.

Decode for 𝒀𝑛 that aligns with 𝑿𝑛 , until an end-of-block</m> token is predicted or max. length is reached
o Attend the input 𝑿𝑛

Shift the input window 𝑊 frames by keeping the model’s
state
𝑆(Total step number: 𝑁 = )
3.

𝑊

●

●

Incremental step:● Input boundary
window● Output boundary

</m>

𝒀𝑛

●

●

Step n = 2

Step n = 1
Encoder𝑿𝑛: last speech frame in the input: </m> token in the output text

W frames

W frames

Alignment learning → Attention transfer

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan14

http://www.naist.jp/

15

Neural Incremental Speech Recognition

Attention-Transfer Incremental Speech Recognition
Attention Transfer
Train ISR (student) to learn the attention-based alignment froma standard seq2seqASR (teacher)1) Extract speech-text alignment from attention matrix generated by
the teacher ASR during teacher-forcing text generation(alignment pair = high attention score):

2) Train the ISR by using 𝒀𝑛 + </m> as the target of 𝑿𝑛

</m>

</m>

𝒀2

𝒀1

Teacher ASR attention matrix


Alignment
Char
Token
ID

Seg. ID(n)

Speech seg.
(𝑿𝒏 )Text seg.
(𝒀𝒏 )1

x1 – xw

y1 – y32

xw+1 – xw*2

y4 – y5(etc.)

𝑿1

𝑿2
n=1

Speech Frame Block ID(1 block = W frames)
無限の可能性、ここが最先端 －Outgrow your limits－

n=2

ISR delay can be managed by changing 𝑿𝑛 and 𝒀𝑛 size during
training
e.g. higher delay : combine several segments into one
Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan15

http://www.naist.jp/

16

Neural Incremental Speech Recognition

AT-ISR Performance

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan16

http://www.naist.jp/

Neural Incremental Speech Recognition

17

AT-ISR Performance

Experiment Dataset●

Wall Street Journal (WSJ) [Paul, 1992]
o 284 speakers, Englisho Training set : SI-284 set (81 hours of speech)o Test set: eval92 set●

TED-LIUM release 1 [Rosseau et al., 2012]o 118 hours of speech (English)o 600 speakers● Speech features: 80 dim. log-Mel spectrogram (50 ms window, 12.5 ms shift)● Text token unito Character : Basic Latin alphabet (WSJ, TED-LIUM)
o Subword : 16,000 subwords (TED-LIUM)

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan14

http://www.naist.jp/

18

Neural Incremental Speech Recognition

AT-ISR Performance
Model Configuration•

AT-ISR/Teacher ASR structure: Seq2seq (identical)
y1

y2

yT

y3

Decoder

LSTM•

AT-ISR with input overlap :o Main frames : Aligns with output text seg.
o Look-ahead frames : Next to the main input(contextual input)

Emb.

time
y0

Hierarchicalsub-sampling
8 speechframes into 1
encoder state

y1

yT-1

y2

Attention

Encoder
BiLSTM

Step 1
input(overlap)

BiLSTM
Step 2
input

BiLSTM[main] [ahead]FNN

AT-ISR basic incremental unit
8 speech frames = 1 block (0.14 sec)

x1

無限の可能性、ここが最先端 －Outgrow your limits－

x2 x3

x4

x5

x6

x7

x8

Nov. 19. 2020

xS-1 xS

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan18

http://www.naist.jp/

Neural Incremental Speech Recognition

19

AT-ISR Performance
Evaluation Setting
ISR performance evaluation was made by comparing various model:● Non-incremental ASR : Toplineo Standard seq2seq ASR (Our Att Enc-Dec; teacher)o Other existing neural ASR●

Incremental ASR:o Baseline neural ISR:
− Seq2seq ISR without attention transfer− Incremental steps were taught by using alignments from forced-alignment by HMM ASRo Proposed ISR: AT-ISR (attention transfer; student)o Other existing neural ISR: Unidirectional LSTM + CTC [Hwang and Sung, 2016]

Evaluation metric:o CER, WER
o Delay (speech input size)
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan19
http://www.naist.jp/

20

Neural Incremental Speech Recognition

AT-ISR Performance
Speech recognition performance ofcharacter-level models trained on WSJ dataset

ResultDelay (sec)Model

CER (%)
Input

Computation●

Avg. utterance length: 7.88 sec●

Machine: Intel® CoreTM i7-9700K CPU @ 3.60GHz

Non-incremental ASR (Topline)
Att Enc-Dec (ours)
BiLSTM-CTC [1]Joint CTC+Att [1](NVIDIA GeForce RTX 2080Ti GPU)

7.88 (avg)0.32 (avg)

6.26
8.97
7.36

Baseline neural ISR
Input/step: 1 m + 1 laInput/step: 1 m + 4 la

0.24
0.54

0.02
0.05

20.15
11.95

0.24
0.54

0.02
0.05

18.37
7.52

Proposed AT-ISR
Input/step: 1 m + 1 laInput/step: 1 m + 4 la

Other existing neural ISR
LSTM-CTC beam search
[2]

-

-

10.96

CER
diff.:

1.3%●

ISR performance limitation: short-segmentbased recognition (incomplete information)●

Contextual input (la) improves performance
AT-ISR performs well with a short delay bylearning non-incremental ASR’s knowledge*Notem= main input blockla= look-ahead block (contextual input)
1 block = 8 frames = 0.14 sec[1] Suyoun Kim, Takaaki Hori, and ShinjiWatanabe. Joint CTC-attention based end-to-end speech recognition using multitask learning. In Proceedings of ICASSP, pages 4835-4839, New Orleans, USA, 2017.
[2] Kyuyeon Hwang and Wonyong Sung. Character-level incremental speech recognition with recurrent neural networks. In Proceedings of ICASSP, pages 5335 - 5339, Shanghai, China, 2016.

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan20

http://www.naist.jp/

21

Neural Incremental Speech Recognition

AT-ISR Performance

ISR Delay

WER (%) of AT-ISR trained on TED-LIUM dataset

How did the ISR delay affected
the ISR performance?
Trade-off: Higher delay, lower WER●

Subword-level ISRoo●Lower WER than character-level ISR
Keep word context longer thancharacters

Character-level ISRooMaintains the teacher’s
performance better than the
subword-level ISR
ISR with delay 2.04 starts to have aclose performance to the teacher
ASR

Subword-level model

50
46.14
45.39

40

WER %

●

Character-level model

33.94
31.06

32.24

30

28.76
28.26

27.79

27.38

27.7
25.27

24

20
0.44

0.64

0.84

2.04

3.94

7.58

(25% S)(50% S)(100% S;
Topline)Delay (sec)*S = average full-utterance length (7.58 sec)21
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

AT-ISR Demo Video – NAIST Lecture

22

Input segment size/step : 0.84 sec. Machine: Intel ®CoreTMi7-5500U CPU @ 2.40GHz x 4

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan20

http://www.naist.jp/

23

Neural Incremental Speech Synthesis

Neural Incremental Speech Synthesis

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan23
http://www.naist.jp/

24

Text-to speech and Incremental Text-to-speech
Text-To-Speech(TTS)The speech is synthesized Sentence-by-sentence.
1. Input is text or phoneme sequence.
2. Acoustic features are predicted by acoustic model
3. Speech waveforms are reconstructed by vocoder.

Incremental Text-To-Speech(iTTS)
Speech is synthesizes a speech in shorter delay.
It can synthesize a speech before finishing text input.
Suitable for real-time task

Real-time speech translation

24
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

25

Incremental Text-to-speech

Incremental Text-To-Speech(iTTS)
Speech is synthesizes a speech in shorter delay (e.g. word).
It can synthesize a speech before finishing text input.

Challenges
How to improve speech quality?
Speech quality of Incremental TTS

How to estimate target prosody from an incomplete sentence?
target prosody is typically calculated from long-window features. (e.g. co-articulation)-> predicts next information(e.g. word) at step of input-to-acoustic-features.
-> wait next word when synthesizing a current word.
25
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

26

Related Works of iTTS(1/2)
Statistical approach (pipeline)
Hidden Markov model TTS[Baumann et al., 2014.],[[Pouget et al., 2015]],[Yanagita, et al., 2018]
No neural End-to-end iTTS approach

Text

Text
analysis

Acoustic model
Linguistic(HMM)
featureEstimation ofduration andacoustic feature

Synthesized
waveform

Vocoder(Digital filter)

previous/current/next phoneme
previous/current/next POS
previous/current/next accentual info.

26
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

27

Related Works of iTTS(2/2)
End-to-end TTS [Wang, et al., 2017.], [Sotelo, et al., 2017], [Shen, et al., 2018.]
Encoder-decoder with an attention mechanism-> Output prediction starts after the input sequence.
The speech is also synthesized Sentence-by-sentence.
-> It can generate High quality speech close to human.

Challenge of the neural iTTS system.
More natural synthesized speech

Neural iTTS[Yanagita et al., 2019]no wait next word for synthesis.
Control output sequence with stop flag

Prefix-to-Prefix Framework [Ma et al., 2020]
wait next word for synthesis.
Control output sequence with attention weight and stop flag
One word look ahead at least for synthesis.
27

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

28

Neural iTTS

Neural iTTS[Yanagita et al., 2019]
Tomoya Yanagita, Sakriani Sakti and Satoshi Nakamura,“Neural iTTS: Toward Synthesizing Speech in Real-time with End-to-end
Neural Text-to-Speech Framework”, 10th Speech Synthesis Workshop(SSW10) , Sep. 2019

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan28

http://www.naist.jp/

29

End-to-End TTS
Encoder

Decoderwith attention

We use Tacotron[Wang, et al., 2017.].
Stop flag prediction to control output seq. is also used.

•
•
•
•
•
•

xN : Input sequence (N length)hN : hidden representatiaon of encode
cs : context vector (S length )ys : mel spectrogram
Ls : Linear spectrogram
Ss : Stop flag29
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

30

Neural iTTS[Yanagita et al., 2019]
End-to-End iTTS
Motivation: We use normal End-to-end TTS as incremental one.
Simple method: Tacotron is synthesized chunk-by-chunk as short sentence.
Ex. “we talk about TTS.”
: sentence start, 
: end of sentence
Tacotron

Tacotron

Tacotron

Tacotron

Tacotron
 Today 

 we 

 talk 

 about 

 TTS. 


無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan30
http://www.naist.jp/

31

Proposed Dataset preprocess
: sentence start
: sentence end<m>: middle sentence start</m>: middle sentence end

Dataset is divided sentence into three parts.
– use location symbol to indicate locations– use all data for training

Inference: Ex. “Today we talk about TTS.”
Encoder

Encoder

Encoder

Encoder
 Today 
<m> we 

 Today </m><m> we </m>

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan31
http://www.naist.jp/

32

Experimental Dataset
JSUT [https://sites.google.com/site/shinnosuketakamichi/publication/jsut]
single female speaker, Japanese, 10 hours
Training set: 5k utterance
Test and dev: 100k utterance
LJ-speech [https://keithito.com/LJ-Speech-Dataset/]
single female speaker, English, 24 hours
Training set: 10k utterance
Test and dev: 100k utterance
Input sequence
Phoneme and accentual information (Japanese)
Word character (English)

Preprocess dataset
Ja. Dataset is divided sentence into three parts in the basis of phrase position.
En. Dataset is divided sentence into three parts in the basis of word position.
Acoustic features: 80 dim. mel-spectram, 1024 dim. Linear-spectrogram32
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

33

Evaluation method

We concatenated all the synthesized waveforms into sentence-based waveforms.
◦ Synthesis various input length (e.g. word-by-word, 2words-by-2words)◦ To compare to normal TTS waveforms
Evaluation methods
MOS test for naturalness◦ Evaluator listens one waveformand score 5 scales(1: very bad, 2:bad, 3:normal, 4: good, 5: very good)33
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

34

Result of English MOS
Still big gap between natural speech and synthesized speech.
Three words unit has the best En. score.

4.98

5

4
3.17

3.12

3.01

Three
words

Half
sentence

Sentence

2.79

3
2.08

2
1
One
word

Two
words

Naturalspeech34
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

35

Result of Japanese MOS
Still big gap between natural speech and synthesized speech.
Half sentence unit ≒ the full sentence units (Ja.).
4.95

5
4
2.81

3

2.88

3.13

3.18

2.22

2
1
One
Two
Three
Half sentenceaccent phrase accent phrases accent phrases

Sentence

Natural35
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

36

Neural Incremental Speech Recognition
Summary – AT-ISR
Neural ISR system (AT-ISR) with a low recognition delay without increasing the complexity of thestandard ASR system
1. AT-ISR with delay < 1 sec. achieved a close performance to standard ASR with delay > 7 sec.

2. AT-ISR as an ISR framework with an efficient development mechanism and reliable
performance via attention transfer that applies an identical architecture as the standard ASR

Recent ISR Trend● Streaming ASR with RNN-Transducer (RNN-T) [Saitnah et al., 2020; Li et al., 2020]● Streaming transformer ASR [Miao et al., 2020; Moritz et al., 2020; Tsunoo et al., 2020]

無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan36
http://www.naist.jp/

37

iTTS Summary
Incremental End-to-end TTS
Previous work: HMM-based iTTS
We challenge neural iTTS system by extending conventional neural TTS-> add location symbols for input-> use initial input for decoder
Future work
The wide gap between natural speech and synthesized speech.
-> wavenet vocoder
Improvement of stop flag prediction for English model-> very short sentence (e.g. “It”)
Calculation of delay
37
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/

38

NAIST AHC Lab.

38
無限の可能性、ここが最先端 －Outgrow your limits－

Nov. 19. 2020

AIST 2020 ©Satoshi Nakamura, AHC Lab, NAIST, Japan

http://www.naist.jp/