Neural Incremental Speech Recognition
Through Attention Transfer
Sashi Novitasari1 Andros Tjandra1 Sakriani Sakti1,2 Satoshi Nakamura1,2
1
Nara Institute of Science and Technology, Japan
2
RIKEN AIP,Japan{sashi.novitasari.si3, andros.tjandra.ai6, ssakti, s-nakamura}@.is.naist.jp

概要

model and mimic the speech-text attention alignments from it to achieve a reliable ISR with a simple

We constructed incremental ASR (ISR) for a
low-latency recognition by exploiting an attention-construction mechanism. Here we refer our approachas attention-transfer ISR (AT-ISR).

based non-incremental ASR framework. The nonincremental ASR is treated as a teacher to teach

2Seq2Seq ASR Overview

the ISR through attention transfer. Our experimentshows that the ISR CER with a delay 0.54 sec that

The non-incremental character-level seq2seq ASR

trained via attention transfer only 2% behind the[3, 2] consists of encoder, decoder, and attention

non-incremental model with a delay more than 6

modules to directly models the conditional proba-sec.

bility P (Y|X), where X is a speech feature frames
sequence (length S) and Y is a characters sequence

1Introduction(length T ). The encoder transforms X into hidden
representative information he . The decoder predicts

One of the challenges that have to be confronted

the target sequence probability pyt , given the previ-to achieve a simultaneous speech translation systemous output Y<t , the current context information ct ,is the incremental ASR (ISR) development. Hid-and the current decoder hidden state hdt . The con-den Markov model (HMM) ASR [10, 5] performs atext ct is produced by attention module [1] at time

low-delay recognition, but it cannot do an end-to-t based on encoder and decoder hidden states. The

end modeling. The state-of-the-art end-to-end ASR

context ct calculation produces alignment scores be-[8, 4] with an attention-based sequence-to-sequence

tween the encoder and decoder states, which can be(seq2seq) deep learning framework has a long delay

represented as a matrix. The AT-ISR is constructed

due the necessity to attend a complete input. Neu-by utilizing these alignment scores.

ral transducer (NT) framework [7] previously proposed for the seq2seq incremental recognition by

3

Proposed Seq2seq ISR

segment-based recognition. It is done by learning
the segment-level alignments from an external sys-AT-ISR, with the same structure as the non-tem such as forced-alignment by HMM ASR or fromincremental ASR, performs several recognition steps

the alignment approximation based on the training

incrementally to recognize an utterance, as shown instates.

Figure 1. For each recognition step n, the mecha-

The existing ISR framework has a more complicated mechanism than the standard ASR because of
the incremental step learning and its preparation. In
this work, we conduct a study of ISR that applies anidentical structure to the standard non-incremental

nisms in AT-ISR are the following:• Encoder: Encode Xn, a segment from X withthe length W where W < S.

FFN layer and three bidirectional LSTM layers, applies state downsampling of eight input frames into
one state. In our basic delay, an output token isaligned to eight frames or one frame-block (0.14 sec).
The decoder consists of an embedding layer and an
LSTM layer.

4.2

Result

The experiment result is shown in Table 1. In
this experiment, we utilized the basic delay as theISR delay to see the performance in the delay as

図 1: Training method of proposed AT-ISR.

• Decoder: Attend the encoder states based on

short as possible. The topline is the teacher ASR andthe baseline is incremental recognition by using theteacher, which was not trained through attentiontransfer for an incremental task.

Xn and decode for Yn, a segment of Y with thelength Kn where Kn ≤ K < T . Decoding instep n stop if an end-of-segment </m> token ispredicted or the length reaches K tokens. This

表 1: Speech recognition performance.

block; la=look-ahead block; 1 block=8 frames)
ModelDelay (sec)

CERTeacher ASR

6.54 (avg)

2.78

token is learned during the training phase byattaching </m> at the end of Yn.

(m=mainLJ Speech
Baseline ISR

0.14

80.34

The recognition in step n + 1 will be done by keep-

AT-ISR (input/step: 1 m)

0.14

23.04

ing the model states and shifting the input window

AT-ISR (input/step: 1 m + 4 la)

0.54

4.45

W frames. The AT-ISR delay equals to the input

WSJ-si284
7.88 (avg)

6.80

0.54

9.06

segment size.
AT-ISR is trained via attention knowledge transfer

Teacher ASR
AT-ISR (input/step: 1 m+ 4 la)from the non-incremental ASR as the teacher. The
segment-level alignments are generated once throughteacher-forcing text generation by the teacher. Eachoutput token is aligned to an encoder state, whichcorresponds to input frames, with the highest attention score. By using the segment-level alignments,
AT-ISR is taught to encode a speech segment to decode the target output segment.

AT-ISR resulted in a close performance to theteacher, especially when the input includes lookahead blocks. The experiment with LJ Speech showsthat a longer look-ahead block resulted in a better
performance. The look-ahead block, which is theframe-block next to the main input block, provides
contextual information to complete the informationin the main input.

4Experiment

4.1

Setting

AT-ISR cannot perform wellwithout it because of the information limitation in
the main input block. The experiment with WSJ

We utilized LJ Speech [6] and Wall Street Journal(WSJ) si284 [9] datasets for our experiments. Ourfeatures consist of the 80-Mel spectrogram with window size 50 ms and shift length 12.5 ms.
Our non-incremental ASR and ISR have an identical structure. The encoder, which consists of anor multi-speaker dataset also shows a similar result.
For both datasets, the CER diﬀerence between theteacher, with a delay 6 sec, and AT-ISR, with a delay 0.54 sec by including look-ahead input blocks, isaround 2%. This implies that AT-ISR is able to perform well with a short delay by learning the knowledge from a non-incremental ASR.

5Conclusion

Speech and Signal Processing (ICASSP), Calgary, Canada, 2018.

We constructed ISR that performs a low-delay
recognition by transferring the attention knowledge[5] David Huggins-Daines, Mohit Kumar, Arthur

from a non-incremental attention-based ASR thatChan, Alan W Black, Mosur Ravishankar, andhas an identical structure. By delaying the recog-

Alex I. Rudnicky. POCKETSPHINX: A free,nition 0.54 sec, the character-level AT-ISR able toreal-time continuous speech recognition system.

produce the output with the overall quality 2% be-

In Proceedings of IEEE International Confer-hind the non-incremental ASR with delay more than
6 sec.

ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 185–188, Toulouse, France,
2006.

6Acknowledgement[6] Keith

Part of this work was supported by JSPS
KAKENHI

Grant

Numbers

JP17H06101

and

JP17K00237.

Ito.

TheLJ

speechdataset.

https://keithito.com/LJ-Speech-Dataset/,2017.
[7] Navdeep Jaitly, Quoc V. Le, Oriol Vinyals, Ilya
Sutskever, David Sussillo, and Samy Bengio. An
online sequence-to-sequence model using par-参考文献[1] Dzmitry Bahdanau, Kyunghyun Cho, andYoshua Bengio. Neural machine translation bytial conditioning. In Proceedings of Advances in
Neural Information Processing Systems (NIPS),
pp. 5067–5075, Barcelona, Spain, 2016.

jointly learning to align and translate. arXiv
preprint arXiv:1409.0473, 2014.

[8] Suyoun Kim, Takaaki Hori, and Shinji Watanabe. Joint CTC-attention based end-to-end[2] William Chan, Navdeep Jaitly, Quoc V. Le,

speech recognition using multitask learning.

and Oriol Vinyals. Attention-based models for

In Proceedings of IEEE International Confer-speech recognition. In Proceedings of the Ad-

ence on Acoustics, Speech and Signal Processing

vances in neural information processing systems(ICASSP), New Orleans, USA, 2017.

(NIPS), pp. 577–585, Montreal, Canada, 2015.
[9] Douglas B. Paul and Janet M. Baker.
[3] William Chan, Navdeep Jaitly, Quoc V. Le,and Oriol Vinyals.

Listen, attend and spell:The

design for the Wall Street Journal-based CSR
corpus.

In Proceedings of the Workshop onA neural network for large vocabulary conver-Speech and Natural Language (HLT), pp. 357–

sational speech recognition. In Proceedings of362, 1992.

the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
pp. 4960–4964, Shanghai, China, 2016.

[10] Laurence Gillick Robert Roth Paul Bamberg,
Yen-lu Chow and Dean Sturtevant. The Dragoncontinuous speech recognition system: A real-[4] Chung-Cheng Chiu, Tara N. Sainath, Yonghui

time implementation.

In Proceedings of theWu, Rohit Prabhavalkar, Patrick Nguyen,Workshop on Speech and Natural Language,

Zhifeng Chen, Anjuli Kannan, Ron J. Weiss,Pennsylvania, USA, 1990.

Kanishka Rao, Ekaterina Gonina, Navdeep
Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. State-of-the-art speech recognition withsequence-to-sequence models. In Proceedings ofIEEE International Conference on Acoustics,