Neural Incremental Speech Recognition
Through Attention Transfer
Sashi Novitasari1, Andros Tjandra1, Sakriani Sakti1,2, Satoshi Nakamura1,21
2

Nara Institute of Science and Technology (NAIST), JapanRIKEN Center for Advanced Intelligence Project (RIKEN AIP), Japan{sashi.novitasari.si3, andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Copyright © 2020 NAIST Japan, All rights reservedNeural Incremental Speech Recognition ThroughAttention Transfer

OutlineCopyright © 2020 NAIST Japan, All rights reserved

IBackgroundII

AT-ISRIIIExperiments

IVConclusion2

Neural Incremental Speech Recognition ThroughAttention Transfer

I. Background

Copyright © 2020 NAIST Japan, All rights reserved

IBackground

IIAT-ISRIII

Experiments

IVConclusion

3

Neural Incremental Speech Recognition Through Attention Transfer

Background

Simultaneous Speech Translation
Simultaneous speech translation system●

Human interpreter interprets speech in real-time●Examples○ Meeting○ Lecture talk●

Automatic (machine)o Mimic human interpreter and translate incoming
speech to the target language with a low delay(incremental)
oRequire ASR that can recognize speech immediately
after speech start

Copyright © 2020 NAIST Japan, All rights reserved

4Neural Incremental Speech Recognition Through Attention TransferBackground

Automatic Speech Recognition
Generate transcription of a speech utterance●Non-incremental ASR○

Start recognition after speech ﬁnish○

State-of-the-art ASR: Att Enc-Dec (end-to-end)→ Not suitable for simultaneous translation●

Incremental ASR (ISR)○

Recognize without the need for waiting for the speech toﬁnish [Selfridge et al., 2011]○

Part-by-part recognition 🡪 challenge→ Suitable for simultaneous translation

Copyright © 2020 NAIST Japan, All rights reserved

5Neural Incremental Speech Recognition Through Attention Transfer

Background

Incremental Speech Recognition
Current situation●HMM-based ASR is incremental but not end-to-end [Rabiner, 1989; Gales, 2008]●Seq2seq ISR : train by learning input-output parts alignments (e.g. Neural transducer [Jaitly et al., 2016] )●End-to-end seq2seq ISR → more complex training than standard seq2seq ASR○○

Learn the incremental step?
Ground alignments?
■ Alignment generation during training based on ISR model (multiple times)■ Alignment generation using forced-alignment system (once)

Copyright © 2020 NAIST Japan, All rights reserved]🡪 Expensive

6

Neural Incremental Speech Recognition Through Attention Transfer

Background

GoalAttention Transfer Incremental Speech Recognition (AT-ISR)
AT-ISRISR that learns to mimic attention-based alignment from attention-based ASR●●

Reliable ISR with simple construction mechanism by using attention-based seq2seq non-incremental ASR○

ISR architecture: Att Enc-Dec ASR, identical conﬁguration as a non-incremental ASR○Incremental step

: Learn the attention alignment knowledge from non-incremental ASR🡪 Attention transfer

Attention transfer : Attention knowledge transfer from teacher to student model○ Prev. works : image recognition tasks■ Teach another model with smaller architecture [Zaguruyko and Komodakis, 2017]■ Domain transfer (image to video) [Li et al., 2017]○ Has not been utilized for ISR construction

Copyright © 2020 NAIST Japan, All rights reserved

7

Neural Incremental Speech Recognition ThroughAttention Transfer

II. AT-ISR

Copyright © 2020 NAIST Japan, All rights reserved

IBackgroundIIAT-ISRIIIExperiments

IVConclusion

8Neural Incremental Speech Recognition Through Attention Transfer

AT-ISROverview

Output: Text

Sequence-to-sequence ASR●

Encoder-decoder with attention mechanism●3 main parts
a. Encoder
Encode input speech features X into
encoder hidden state heb. Decoder
Predict output Y by processing previous
output, decoder hidden state hd andencoded informationc. Attention
Calculate alignment score between
encoder states (input) and decoder states(output)

Attention Matrix

Decoder

Attention

Encoder

Input: Speech
Encoder-Decoder architecture with an attention component[Bahdanau et al, 2015], courtesy of [Tjandra et al., 2017]

Copyright © 2020 NAIST Japan, All rights reserved

9Neural Incremental Speech Recognition Through Attention Transfer

AT-ISR
Step 2

Step 1AT-ISR Recognition Method●Recognize speech segment-by-segment sequentially●

Delay = W speech frames (window)●For each recognition step:
1.

Encode W speech frames (block)2.

Decode for the output that aligned to the main input
block, until end-of-block </m> token predicted or max.
length reachedo3.

●</m>

</m>

DecoderAttention


Encoder

Attend the current input

Shift the input window W frames

Alignment learning → Attention transfer
W frames

Copyright © 2020 NAIST Japan, All rights reserved

W frames

10

Neural Incremental Speech Recognition Through Attention Transfer

AT-ISR

Attention Transfer

2) Train ISR by using Yn + </m> as target of Xn

Train ISR (student) to learn the attention-basedalignment from attention-based ASR (teacher)
</m>

</m>

1) Extract segment-level speech-text alignment from non-incremental
ASR attention matrix (alignment pair = high alignment score)

Alignment

TextTokenID


Seg. ID(n)Speech seg.
(Xn)

Text seg.
(Yn)

1x1 – xw

y1 – y3

2xw+1 – xw*2

y4 – y5(etc.)
n=1Speech Frame Block ID(1 block = w frames)
Copyright © 2020 NAIST Japan, All rights reserved

n=2

ISR delay can be managed by changing Xn and Yn size
E.g. higher delay : combine each several segments into one

11

Neural Incremental Speech Recognition ThroughAttention Transfer

III. Experiment

Copyright © 2020 NAIST Japan, All rights reserved

IBackgroundIIAT-ISRIIIExperiment

IVConclusion

12Neural Incremental Speech Recognition Through Attention TransferExperiment

Experiment Setting
DataDataset●●LJ Speech [Ito, 2017]o 24 hours of speech (En)o Single-speaker
Wall Street Journal (WSJ-si284) [Paul and Baker, 1992]
o 80 hours of speech (En)o 280 speakers

Copyright © 2020 NAIST Japan, All rights reserved

Model Conﬁguration
Structure: Encoder-Decoder with Attention●Encoderoo

1 FFN → 3 Bi-LSTM
Downsample 8 speech frames into 1
encoder state (ISR basic delay= 1 block = 8
frames = 0.14 sec)

●Decodero Embedding → 1 LSTM
o Output unit: character●Same structure for non-incremental ASR(teacher) and AT-ISR (student)
13

Neural Incremental Speech Recognition Through Attention Transfer

Experiment

Experiment Result

ISR Input Segment
Speech frame
timestep

Speech Recognition Result
Model

Delay (sec)

Step 1

CER%

Look-ahead input (context)

LJ Speech
Topline ASR (Teacher)

Step 2

6.54 (avg)

2.78

Baseline ISR (input/step: 1 m)

0.14

80.34

AT-ISR (input/step: 1 m)

0.14

23.04

AT-ISR (input/step: 1 m + 4 la)

0.54

4.45

7.88 (avg)

6.80

0.54

9.06

WSJ-si284
Topline ASR (Teacher)
AT-ISR (input/step: 1 m + 4 la)●●●●m : main block (main recognition target)
la : look-ahead context block (frames next to the main block)1 block = 8 frames = 0.14 sec
Baseline: Incremental recognition by using teacher ASRCopyright © 2020 NAIST Japan, All rights reserved

Main input●

AT-ISR has a close performance to the teacher
ASR when the input includes look-ahead blocks●

Look-ahead blocks complete the information inthe main block●

Non-incremental ASR (delay > 6 sec) and AT-ISR(delay = 0.54 sec) CER difference: 2%
AT-ISR performs well with a short delay bylearning non-incremental ASR’s knowledge
14Neural Incremental Speech Recognition Through Attention TransferExperiment

Effect of Speech Recognition Delay●●●LJ Speech dataset
Tradeoff : Higher delay 🡪 higher performance
Insigniﬁcant improvement after certain delay conf. 🡪 shortest delay with best performance

AT-ISR CER(%) and Delay
AT-ISR Input: [W main + 1 look-ahead] blocks/step

A = average frame blocknumber in an utterance in
LJ Speech set(6.54 sec~ 520 blocks)

1 speech block = 8 frames= ~0.14 sec
Delay :W:
Copyright © 2020 NAIST Japan, All rights reserved

1 speech block = ~2 chars

15

Neural Incremental Speech Recognition ThroughAttention Transfer

IV. Conclusion

Copyright © 2020 NAIST Japan, All rights reserved

I

BackgroundII

AT-ISRIII

Experiment

IVConclusion

16

Neural Incremental Speech Recognition ThroughAttention Transfer

Conclusion
We constructed ISR that performs a low-delay speech recognition●AT-ISR learn attention knowledge from non-incremental ASR that has an identical structure●AT-ISR able to perform closely to the teacher by incrementally recognizing short input segments withcontext blocks (low latency and reliable)oLJ Speech CER

🡪 teacher 2.84% (delay 6.54 sec) ; student 4.45% (delay 0.54 sec)oWSJ CER🡪 teacher 6.80% (delay 7.88 sec) ; student 9.06% (delay 0.54 sec)

Copyright © 2020 NAIST Japan, All rights reserved

17

Neural Incremental Speech Recognition Through Attention Transfer

Thank You

Copyright © 2020 NAIST Japan, All rights reserved

18

Neural Incremental Speech Recognition Through Attention Transfer

Appendix

Copyright © 2020 NAIST Japan, All rights reserved19

Neural Incremental Speech Recognition Through Attention Transfer

ASR Output
DelayASR Output

CER%

0.14 sec(1 block)which probably represent the burtabic condition before limbs were a quired*

11.5

0.24 sec(2 block)which probably represent the burtabrit* condition before limbs were acquired,

7.7

1.74 sec(25% utt. length)

which probably represent the burtebrate condition before limbs were acquired,

3.8

3.34 sec(50% utt. length)

which probably represent the vertebrate condition before limbs were acquired*

1.3

6.64 sec(Non-incremental)

which probably represent the vertebrate condition before limbs were acquired,

1.3

Correct text

which probably represent the vertebrate condition before limbs were acquired;(*) : missing character
red character : incorrect character
Copyright © 2020 NAIST Japan, All rights reserved20