Sequence-to-Sequence Learning via
Attention Transfer for
Incremental Speech Recognition
Sashi Novitasari1, Andros Tjandra1, Sakriani Sakti1,2, Satoshi Nakamura1,21 Nara Institute of Science and Technology (NAIST), Japan
2 RIKEN Center for Advanced Intelligence Project (RIKEN AIP), Japan{sashi.novitasari.si3, andros.tjandra.ai6, ssakti, s-nakamura}@is.naist.jp

Copyright © 2019 NAIST Japan, All rights reservedSequence-to-Sequence Learning via Attention
Transfer for Incremental Speech Recognition

OutlineCopyright © 2019 NAIST Japan, All rights reservedI

BackgroundII

AT-ISRIIIExperiments

IVConclusion2

Sequence-to-Sequence Learning via Attention
Transfer for Incremental Speech Recognition

I. Background

Copyright © 2019 NAIST Japan, All rights reserved

IBackground

IIAT-ISR

III

Experiments

IVConclusion3

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Background

Simultaneous Speech Translation●Translate incoming speech to the target language in realtime with low delay (incremental)●

Examples of use○ Meeting○ Lecture talk
○ Live video●Automatic → require ASR that can recognize the speech
immediately after the original timing

Copyright © 2019 NAIST Japan, All rights reserved4Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Background

Automatic Speech Recognition
Generate transcription of a speech utterance●

Non-incremental ASR○

Wait for the speech to finish first○State-of-the-art ASR: Att Enc-Dec (end-to-end)→ Not suitable for simultaneous translation●

Incremental ASR (ISR)○

Recognize without the need for waiting for the speech to finish[Selfridge et al., 2011]○

Part-by-part recognition→ Suitable for simultaneous translation
Copyright © 2019 NAIST Japan, All rights reserved5Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Background

Incremental Speech Recognition●HMM-based ASR is incremental but not end-to-end●Seq2seq ISR : train by learning input-output parts alignments (e.g. Neural transducer [Jaitly et al., 2016] )●End-to-end seq2seq ISR → more complex training than standard seq2seq ASR○○

Learn the incremental step?
Ground alignments?
Expensive■ Generate it during training based on ISR model (multiple times)
à (especially if module not■ Generate it by using external module (once)available)]

How to make reliable ISR with simple method?

Copyright © 2019 NAIST Japan, All rights reserved

6Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Background

GoalAttention Transfer Incremental Speech Recognition (AT-ISR)● Simple training & recognition à Exploit attention-based seq2seq ASRo ISR architecture: Att Enc-Dec ASR (seq2seq)o Incremental step & alignment : Learn the attention knowledge from ASR
à attention transfer● Attention transfer : Attention knowledge transfer from teacher to student○ Prev. works à image recognition tasks■ Teach another model [Zaguruyko and Komodakis, 2017]■ Domain transfer (image to video) [Li et al., 2017]o Has not been utilized for ISR construction
AT-ISRISR that learns to mimic attention-based alignment from attention-based ASR
Copyright © 2019 NAIST Japan, All rights reserved

7Sequence-to-Sequence Learning via Attention
Transfer for Incremental Speech RecognitionII.
AT-ISR

Copyright © 2019 NAIST Japan, All rights reserved

I

BackgroundIIAT-ISRIII

Experiments

IVConclusion8Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

AT-ISR

AT-ISR Recognition Method●Att Enc-Dec●Input segment à W frames, consist of:o M frames à main inputo C frames à contextual input (optional, adjacent to main input)●

Recognize segment-by-segment sequentially○

For each recognition step:

1.
2.
3.

●

Encode W speech frames (block)
Decode for the output that aligned to the main input block, until end-of-block token
predicted or max. length reached
2.1 Attend the current input
Shift the input window M frames

How to learn the end-of-block (</m>)? → Attention transfer

Copyright © 2019 NAIST Japan, All rights reserved

9Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

AT-ISR

Encoder-Decoder with Attention Mechanism
3 main parts:●

Encoder
Encode input features●

Decoder
Decode encoded information into output●Attention
Calculate alignment score between encoder
states (input) and decoder states (output)→ attention matrix

Copyright © 2019 NAIST Japan, All rights reserved

Att Enc-Dec

Attention Matrix[Bahdanau et al, 2015],
courtesy of [Tjandra et al.,
2017]

10

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

AT-ISR

Learning the Alignment

Attention Transfer for ISR
Train ISR (student) to learn the attention-based alignment
from attention-based ASR (teacher)
Attention-based alignment● 1 encoder state represents M input frames→ 1 output aligned to M frames●

M: downsampling rate in encoder

Ground Alignment for ISR training● Token aligned to an encoder state with highest
attention alignment score (monotonic)●</m> placed after each last-aligned token in a segment●Alignment generation by teacher-forcing

Copyright © 2019 NAIST Japan, All rights reserved

char1, char2, char3 aligned to enc_state1 = frame block 1(M = 8 frames)

11

AT-ISR Training
Given
Speech frames X = [x1,x2,...xS]
Transcription Y = [y1,y2,...,yT]
Non-incremental ASR
P(Y | X)
AT-ISRFor each step n:P(Yn | Xn)
where:● Xn = [x((n-1)w)+1, … , xnw]
● Yn = [yn,1, ... , yn,Kn]● yn,Kn = </m> token● 0 <= Kn <= K < T● Yn aligned to Xn(attention alignment)

Copyright © 2019 NAIST Japan, All rights reserved

12Sequence-to-Sequence Learning via Attention
Transfer for Incremental Speech Recognition

III.
Experiments

Copyright © 2019 NAIST Japan, All rights reserved

I

BackgroundII

AT-ISR

III

Experiments

IVConclusion

13

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition:

Experiments

Data and Features●Dataset (English)
Dataset

LJ Speech[Ito et al., 2017]

Wall Street Journal[Paul et al., 1992]

Speaker

Length(hour)

Train

Expr. Sets (utterance)
Dev.

Test12412000

400

40080 (si84)
280 (si284)

16 (si84)
80 (si284)

7000 (si84)
37000 (si284)

500 (dev93)

300 (eval92)●Features○ 80-mel spectogram○ Window length 50 ms, shift 12.5 ms●

Output representation: Character (basic Latin alphabet)

Copyright © 2019 NAIST Japan, All rights reserved

14Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition:

Experiments

Model Configuration
Non-incremental ASR & AT-ISR → Att Enc-Dec (parameters based on [Tjandra et al., 2017]):●Encoder○ 1 FNN layer (256 units) , 3 BiLSTM layers (256 units/LSTM layer)○ Downsampling: 2 states for each BiLSTM layer■ Final encoder states represent 8 frames each
■ ISR input block unit: 1 block = 8 frames = ~0.14 sec●

Decoder : 1 embedding layer (256 units) , 1 LSTM layer (512 units)●Attention : MLP-scoring with multi-scale alignment and contextual history [Tjandra et al., 2018]●No language model

Copyright © 2019 NAIST Japan, All rights reserved15

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech RecognitionExperiments

Experiment Scenario
ToplineBaseline: Non-incremental recognition by teacher ASR: Incremental recognition by teacher ASR (no attention transfer)

Experiments:
1) Mechanism configuration
How to take encoder and decoder input, how to treat model states
2)

Delay
How the AT-ISR delay affects the performance

Copyright © 2019 NAIST Japan, All rights reserved

16Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Experiments

Experiment 1

Mechanism Configuration
Encoder Inputa.

Overlapping inputs●Include context blocks in addition to the mainblock (adjacent):○ Look-back : prev. to the main block○ Look-ahead : next to the main block●Output → tokens that aligned to the main blockb.

No overlapping input

Copyright © 2019 NAIST Japan, All rights reserved

Decoder Initial Inputa.

Last token from the prev. step (before </m>)b.

Beginning-of-block <m>17

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Experiments

Experiment 1Result●AT-ISR model states:○ Reset at the beginning of step
○ Keep the states from previous step●

ISR input segment size in each step:○ 1 main block (~0.14 sec)
○ Overlap: +1 look-ahead block●Best mechanism:○ Encoding : Input overlap○ Decoding : Last character fromprev. step as initial input
o Model states: Keep→ as default mechanism for AT-ISR

Copyright © 2019 NAIST Japan, All rights reserved

Utterance-based CER% on LJ Speech Dataset
Enc-Inp

Topline ASR
Baseline ISR
No overlap
No overlap
Overlap
Overlap
No overlap
No overlap
Overlap
Overlap

Dec. InitialInp

Delay(sec)

6.54 (avg.)
0.14AT-ISR - reset state<m>
0.14prev. char
0.14
<m>
0.24
prev. char
0.24
AT-ISR - keep state<m>
0.14
prev. char
0.14
<m>
0.24
prev. char
0.24

Dev.

Test

2.84
79.63

2.78
80.34

32.51
26.15
23.74
13.40

32.35
26.52
23.40
14.22

24.35
22.69
8.83
8.82

24.44
23.04
8.16
8.39

18

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Experiments

Experiment 2

Delay: Main block●●●LJ Speech dataset
Tradeoff : Higher delay à higher performance
Insignificant improvement after certain delay conf. à shortest delay with best performance

Impact of Main Input Block Size on Utterance-based CER
AT-ISR Input: [N main + 1 ahead] blocks/step

A = average frame blocknumber in an utterance inLJ Speech set(6.54 sec~ 520 blocks)

1 block = 8 frames = ~0.14 sec
DelayN::

Copyright © 2019 NAIST Japan, All rights reserved

Delay = Time diff. of speech’s
start timing and start timing of
1st recognition step

19

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Experiments
Experiment 2

Delay: Context blocks●●●

AT-ISR Input segment

LJ Speech, utterance-based CER
Without context block à CER 22.7% (dev.)
Context blocks help the recognition, especially look-ahead blocks(context)

MainblockN:
Delay :

0.34 sec

0.44 sec(context)

AT-ISR input: [1 main + N ahead] blocks/step

best

CER %
Non-incr ASR dev. 2.84 (6.54 sec)

L.aheadblock

Impact of look-ahead block

Impact of look-back block

AT-ISR input: [N back + 1 main + 1 ahead] blocks/step

CER %L.backblock

Non-incr ASR dev. 2.84 (6.54 sec)

0.64 secN:
Delay :

0.24 sec

0.34 sec

0.54 sec*1 block = 8 frames = ~0.14 sec
Copyright © 2019 NAIST Japan, All rights reserved20

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition:

Experiments

Performance on Multi-speaker Data
Utterance-based CER (%) on eval92 Set
Non-incremental ASR (Topline)

Model

Delay

CTC [Kim et al., 2017]
Att Enc-Dec Content [Kim et al., 2017]
Att Enc-Dec Location [Kim et al., 2017]
Join CTC+Att (MTL) [Kim et al., 2017]
Att Enc-Dec (Teacher)

7.5 sec(avg.)

si84
20.34
20.06
17.01
14.53
17.05

si284
8.97
11.08
8.17
7.36
6.80

si84
30.81
18.05

si284
19.78
9.06

●

WSJ dataset●

Train set:- si84 : 80 speakers- si284 : 280 speakers●

WSJ si284 modelà delay 0.54
sec, CER difference to teacher~2%●

AT-ISR able to perform closely toteacher on multi-speaker data

AT-ISR (1 main input block/step)

Look-back/step
0 block
0 block

Look-ahead/step
1 block
4 block

Delay
0.24 sec
0.54 sec*1 block = 8 frames = ~0.14 sec
Copyright © 2019 NAIST Japan, All rights reserved21Sequence-to-Sequence Learning via Attention
Transfer for Incremental Speech Recognition

IV.
Conclusion

Copyright © 2019 NAIST Japan, All rights reserved

I

BackgroundII

AT-ISR

III

Experiments

IVConclusion

22

Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Conclusion●

Incremental speech recognition with AT-ISR -- ISR that learned the same attention alignment
as the teacher non-incremental ASR●AT-ISR able to perform closely to the teacher by incrementally recognizing short input
segments (low latency and reliable)○○●LJ Speech CER à teacher 2.84% ; student 4.45% (delay 0.54 sec)
WSJ CERà teacher 6.80% ; student 9.06% (delay 0.54 sec)

Optimum AT-ISR performance achieved by, for each step, including few ahead blocks andsetting the last character from the last step as the initial input in decoding

Copyright © 2019 NAIST Japan, All rights reserved25Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition

Thank You

Copyright © 2019 NAIST Japan, All rights reserved

24

Example
Non-incremental ASR

AT-ISR

Text generation:f e w <spc> y e a r s <spc> a g oAttention
MatrixTraining: Encode W frames, decode alignedchars+</m> as the target

Att Enc-Dec

0SAlignment based on attention scores:f (none) e w <spc> y e a r s <spc> a g o
Block
0

Block
60

W

2W

3W

4W

5W

6W

7W

Example of alignment index calculation for the 6th character ‘y’(k → 1 enc state = 1 frame block = W frames (downsample) )Text generation: Encode W frames anddecode until </m>, 
, or reach max. length25TED-LIUM
Performances (%) on TED-LIUM release 1● Unk. word : Rate of words that does not exists
in the train data (eval. set original text = 1.55%)● Character-to-subword:
Sentencepiece:●
Wait for 1 word then convert it into
subwords (1 word = 8 characters (avg.))●
Same CER, WER, and unk. word rate as
the character-level ISR
Seq2seq:●
Incremental: convert 8 characters bylooking 8 characters ahead●Speech-character and charactersubword models trained separately

ISR input-output

CERSWERWER

Unk. word

Full-utterance ASR (avg. delay: 7.58 sec)
sp-ch-sw (sentencepiece)

15.21

20.16

27.37

3.02

sp-ch-sw (seq2seq)

15.81

20.46

28.32

1.03

sp-sw

13.35

18.91

23.98

0.62

ISR (input segment: 1 main + 4 ahead bocks →delay: 0.54 sec)
sp-ch-sw (sentencepiece)

21.00

31.87

41.10

11.7

sp-ch-sw (seq2seq)

22.36

27.53

39.71

1.34

sp-sw

21.28

25.70

36.78

0.66

ISR (input segment: 4 main + 4 ahead bocks → delay: 0.84 sec)
sp-ch-sw (sentencepiece)

16.22

23.11

31.04

5.19

sp-ch-sw (seq2seq)

17.99

22.60

31.80

1.66

sp-sw

15.20

19.88

28.26

1.04

sp: speech features
ch: charactersw: subword
sp-ch-sw: char-level ISR and character-subword model

26