Mask-CTC-based Encoder Pre-training for
Streaming End-to-End Speech Recognition
Huaibo Zhao⋆ , Yosuke Higuchi⋆ , Yusuke Kida† , Tetsuji Ogawa⋆ , Tetsunori Kobayashi⋆⋆

Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan†
LINE Corporation, Tokyo, Japanzhao@pcl.cs.waseda.ac.jp

enables block-synchronous beam search decoding. Apart fromthe above streaming models based on attention-based encoderdecoder, the Transducer-based model can be naturally applied
to streaming ASR. Transducer trains a model to align the
output of the acoustic encoder with the output of the label
encoder, enabling frame-synchronous decoding and making it
a suitable framework for streaming ASR [8]. TransformerTransducer (Transformer-T) [9] adopts Transformer for the
acoustic encoder, where the chunk-wise attention mask limits
the look-ahead range of the self-attention layer to ensure
streaming properties.
For streaming ASR models in general, performance degradation occurs when the look-ahead range is limited fromglobal to local, suppressing the advantage of the Transformer
architecture (i.e., processing with long-range contexts). Consequently, longer look-ahead ranges are often required to provide
adequate future contexts, leading to the growth of latency
requirements. Therefore, capturing long-term contexts withinshort look-ahead ranges is essential for building successful
streaming ASR systems.
One approach to realizing such a property is to utilize theMask-CTC [14], [15] framework. With conditional masked
language model (CMLM) [16], [17] and CTC multi-task training, Mask-CTC trains an encoder network to extract acoustic
feature representation that contributes to capturing long-term
output dependencies by anticipating future contexts. Its capability of learning context-rich bi-directional representations has
been validated in the spoken language understanding task [18],[19]. In our previous work [20], we conducted supervised pretraining with the Mask-CTC objective on the encoder and
CTC modules of the triggered attention-based model, which
has shown effective results for improving accuracy while
reducing latency. In this work, we aim to further examine theeffectiveness of Mask-CTC-based pre-training for streaming
models with various architectures.
In addition, a detailed perspective on the latency can be analyzed by focusing on the timing of output spikes. In streaming
ASR, the timing of output spikes is generally delayed due tothe lack of long-term context information. We expect that the
proposed pre-training based on Mask-CTC will introduce thelook-ahead capability to the encoder, thereby enabling accurate prediction of the timing of output spikes. This property
contributes to the early determination of recognition results,
which is essential for streaming applications (e.g., a system

Abstract—Achieving high accuracy with low latency has always
been a challenge in streaming end-to-end automatic speech
recognition (ASR) systems. By attending to more future contexts,
a streaming ASR model achieves higher accuracy but results inlarger latency, which hurts the streaming performance. In the
Mask-CTC framework, an encoder network is trained to learn
the feature representation that anticipates long-term contexts,
which is desirable for streaming ASR. Mask-CTC-based encoder
pre-training has been shown beneficial in achieving low latency
and high accuracy for triggered attention-based ASR. However,
the effectiveness of this method has not been demonstrated forvarious model architectures, nor has it been verified that theencoder has the expected look-ahead capability to reduce latency.
This study, therefore, examines the effectiveness of Mask-CTCbased pre-training for models with different architectures, such
as Transformer-Transducer and contextual block streaming ASR.
We also discuss the effect of the proposed pre-training method
on obtaining accurate output spike timings, which contributes tothe latency reduction in streaming ASR.
Index Terms—Streaming automatic speech recognition, latency
reduction, Mask-CTC

I. I NTRODUCTION
In recent years, deep learning has become the core technology of automatic speech recognition (ASR) [1], [2]. Endto-end ASR further integrates the traditional separated components (i.e., acoustic, pronunciation, and language models)
into a single deep neural network, significantly contributing to
the simplicity of ASR developments [2]–[4]. End-to-end ASR
models can be realized in various approaches, including connectionist temporal classification (CTC) [5], Transducer [6],and attention-based encoder-decoder [2], [4]. These end-toend ASR approaches have greatly benefited from the adoptionof Transformer [7]–[11], enabling a model to capture global
contexts using the self-attention mechanism.
Streaming properties (i.e., real-time processing) are of vital
importance in the applications of ASR systems. With thesuperior performance of Transformer, many efforts have been
devoted to making Transformer-based ASR models streaming.
Triggered attention-based ASR [12] obtains alignment information from CTC and realizes frame-synchronous decoding
according to the CTC spike timings. Meanwhile, contextual
block streaming ASR (CBS-ASR) [13] splits the input into
blocks (chunks), and streaming encoder feature extraction isconducted on each block with the contexts inherited fromthe previous blocks. A block boundary detection algorithm
is applied to detect the index boundary in each block, which

ISBN: 978-9-4645-9360-0

56

EUSIPCO 2023

that interacts with a user in real-time).
This study, therefore, attempts to demonstrate the effectiveness of Mask-CTC-based pre-training for streaming models with different architectures, including Transformer-T and
CBS-ASR, and to discuss the contribution of the proposed
pre-training to latency reduction.
The rest of this paper is organized as follows. Section II
overviews the streaming ASR models and the Mask-CTC
model. Section III describes the proposed pre-training approach for constructing low latency and high recognition accuracy streaming ASR models. In Section IV, we demonstrate
the effectiveness of Mask-CTC-based pre-training through experiments and discuss the latency reduction effect with output
spike timing measurements. Finally, Section V concludes thispaper.

B. Contextual block streaming ASR
Contextual block streaming ASR (CBS-ASR) [13] introduces streaming properties to attention-based encoder-decoder
models. For streaming feature extraction in the encoder, CBSASR utilizes block processing with a context inheritancemechanism proposed in [23]. The speech input is segmented
into blocks containing past, central, and future frames with
the numbers of Nl , Nc , and Nr . The input blocks are passed
on to the encoder, where the central frames are utilized
for the output with local contexts provided by the past andfuture frames as well as the global contexts provided by acontext embedding vector inherited from the previous block.
Streaming decoding is achieved by a block boundary detection(BBD) algorithm [13], which takes end-of-sentence prediction
or token repetition as stopping criteria from detecting theindex boundaries on-the-fly and enables the beam search
synchronous to the encoded blocks. The streaming processingin CBS-ASR is calculated as follows:

II. BACKGROUND
In this section, we introduce two types of streaming
ASR approaches that we study in this work: TransformerTransducer [9] and contextual block streaming ASR [13]. We
also describe Mask-CTC [14], which is the key to our proposed
method.

Hb , cb = BlockEncoder(Zb , cb−1 ),α(y0:i , H1:B ) ≈

B
X

Ib
X

log p(yi |y0:j−1 , H1:b ).

(5)
(6)

b=1 j=Ib−1 +1

Eq. (5) represents the streaming encoding of the b-th input
sequence Zb , where |Zb | = Nl + Nc + Nr . The encodedacoustic features Hb is obtained from Zb and the contextual
vector from the previous block cb−1 . Eq. (6) represents thescore of the partial hypothesis y0:i during streaming beam
search decoding, where y0 is the start-of-sequence token. Ib
denotes the index boundary of the b-th input block derived
from the BBD algorithm.

A. Transformer-Transducer
A Transducer-based ASR model contains three components:
acoustic encoder, label encoder, and joint network. Given a
streaming input to a current time index t, the output probabilityof the u-th token is calculated as follows:
hAE
t = AcousticEncoder(x1:t ),

(1)

hLE
u−1 = LabelEncoder(y1:u−1 ),LE
h = Tanh(Linear(hAEt ) + Linear(hu−1 )),(2)C. Mask-CTC(3)

P (yu |y1:u−1 , x1:t ) = SoftMax(h).

(4)

The Mask-CTC framework [14] aims to learn feature
representations suitable for anticipation of future contexts.
Mask-CTC trains an encoder-decoder model with the joint
CMLM [16] and CTC objectives. During training, tokens in
the ground truth are randomly masked, and the masked tokens
are predicted based on contextual information captured by
the encoder and other unmasked output tokens. For the input
X and observed tokens Yobs , the output probabilities of themasked tokens Ymask are computed as follows:Y
Pcmlm (Ymask |Yobs , X) =Pcmlm (y|Yobs , X),(7)First, the acoustic encoder embeds the input sequence x1:t
into vector hAEt (Eq. (1)). Meanwhile, the label encoder generates hLE
u−1 from the previous output token sequence y1:u−1(Eq. (2)). The two outputs are then sent to the joint network,
projected to the same dimension, and added up (Eq. (3)).
Finally, the output probabilities against tokens in a vocabulary
V are calculated based on the previous result (Eq. (4)). The
Transducer framework predicts the current symbol for each
input frame based on the past output tokens, which naturally
introduces streaming fashion into decoding.
Various neural network types can be applied to implement
the acoustic and label encoders [6], [21], [22]. In the workof [9], Transformer [7] is applied to the acoustic encoder toachieve high accuracy and LSTM [21] for the label encoder inconsideration of the model size control. Chunk-wise attention
masks are applied to the self-attention layers of the Transformer acoustic encoder to enable streaming feature extraction.
This architecture is referred to as a Transformer-Transducer(Transformer-T).

y∈Ymask

where Yobs is Y \Ymask . Based on the CMLM mask predictionof the decoder, the encoder network of Mask-CTC is trained
to consider the long-term bidirectional dependencies between
output tokens, which enables it to generate acoustic feature
representations that anticipate future information.
Such properties are desirable in streaming ASR as it allows
the model to capture more future contexts with a limited lookahead range. In such a way, the Mask-CTC framework can bea potential solution for improving streaming ASR, enhancing
a model to achieve high accuracy while keeping low latency.

57

CMLM
DecoderCTC

model to capture long-term contextual information and reduce
the latency dependency.

Transformer-Transducer

Mask-CTC

Joint network

Label
Encoder

EncoderIV. E XPERIMENTS
Speech recognition experiments were conducted to examine
the effectiveness of the Mask-CTC-based pre-training method
using ESPnet2 [24], [25]. We also investigated the essential
effect of the proposed pre-training method by studying theoutput token alignments of the streaming ASR models.

Acoustic
EncoderModule Initialization
Stage 1: Feature representation learning

Stage 2: Streaming ASR training

A. Datasets

Fig. 1. Illustration of Mask-CTC-based pre-training using TransformerTransducer model. In stage 1, encoder is trained with Mask-CTC framework. In stage 2, Transformer-Transducer model is initialized with pretrained encoder and fine-tuned with streaming objective.

The models were trained and evaluated using the Wall Street
Journal (WSJ) [26] dataset, which contains 81h English utterances of read articles from the newspaper and the TED-LIUM2(TED2) [27] dataset, which contains 207h English spontaneous
speech. For the output tokens, we used SentencePiece [28] toconstruct a 80 subword vocabulary for WSJ and a 500 subword
vocabulary for TED2, respectively. For robust model training,
we applied SpecAugment [29] to the input data.

III. M ASK -CTC- BASED PRE - TRAINING METHOD
We present a simple and general Mask-CTC-based pretraining method for achieving high-accuracy and low-latency
streaming ASR. Specifically, this paper aims to demonstrate
the effectiveness of the Mask-CTC pre-training regardless ofmodel architectures and discusses whether such pre-training
can extract features suitable for anticipation as intended,focusing on the alignment of the output tokens.
As different end-to-end streaming ASR models, we focus
on Transformer-T (see Section II-A) and CBS-ASR (seeSection II-B), which cover both Transducer and encoderdecoder model architectures. For both models, the adoptionof Transformer has realized high recognition accuracy in their
non-streaming baselines. However, when applied to streaming
scenarios, the look-ahead ranges of self-attention layers arelimited from global to local. This leads to an inevitableperformance drop by degrading the Transformer’s capability tocapture long-range contexts, which limits applications where
low latency is a top priority for recognition.
To remedy such an effect, we need the feature representation
for the input sequence that considers long-term contextual
dependencies and anticipates future information, which corresponds to the properties of the Mask-CTC encoder network as
described in Section II-C. To introduce the desirable propertiesof the Mask-CTC model into the streaming ASR, we propose
a simple two-step training method as follows, which is also
described in Fig. 1:• Stage 1 (Feature representation learning): The MaskCTC model is pre-trained to obtain an encoder networkthat can consider long-term dependencies and anticipate
future information.
• Stage 2 (Streaming ASR training): The pre-trained
Mask-CTC model is exploited to initialize the streaming
ASR models. For Transformer-T, the acoustic encoderwith the chunk-wise attention is initialized with theMask-CTC encoder. For CBS-ASR, both the Mask-CTC
encoder and CTC networks are used to initialize thecorresponding components.
With the two-step training method above, we expect toinherit the characteristics of Mask-CTC to a streaming ASR

B. Experimental setup
For the Transformer-T model, the acoustic encoder was
implemented with 12 Transformer encoder layers and a single
LSTM layer for the label encoder. For streaming feature
extraction, a chunk-wise attention mask was implemented andapplied to the encoder layers as in [9]. The latency value wascalculated as the product of the maximum look-ahead range(i.e., chunk size − 1) and a frame rate of 40ms.
For WSJ experiments, the CBS-ASR model consisted of
6 Conformer encoder layers [30] and 6 Transformer decoder
layers. The input block settings followed Nl as eight, Nc as
four, and Nr varying from 0 to 6. The latency for CBS-ASR
was calculated as the product of the maximum look-ahead
range in the block (i.e., Nc +Nr −1) and a frame rate of 40ms.
For TED2 experiments, the CBS-ASR model consisted of 12
Conformer encoder layers [30] and 6 Transformer decoder
layers. The Nr was set to 6.
For the pre-trained Mask-CTC model, the encoder wasconstructed with the identical setting as the target streaming
model. The CMLM decoder was built with six Transformer
decoder layers. All the models were trained by 150 epochs,
and the final models were obtained by averaging the snapshotsof the ten epochs of the minimal loss for Transformer-T andthe best accuracy for CBS-ASR. For decoding, a beam search
was conducted with a beam size of ten for all. We used theword error rate (WER) for measuring the ASR performance.
C. Experimental results
For both the Transformer-T and CBS-ASR systems, theperformances of the following models are compared.
• Baseline [9], [13], [31]: Existing streaming ASR models,
including Transformer-T and CBS-ASR. The parameters
for all the components were randomly initialized.
• Enhanced: Streaming ASR models with Mask-CTCbased pre-training. Components of the streaming ASR
were initialized with pre-trained Mask-CTC modules. For

58

TABLE I
W ORD ERROR RATES ON WSJ DATASET.

nothing
1

said

the

businessman

Non-streaming

WER [%] (↓)
Model

Latency [ms]Initialization

eval92dev93
0Baseline

Transformer-T

CBS-ASR

120
160
200

19.5
16.8
15.1

23.3
20.9
18.9

1

Random∞Random

14.7

17.3

0

200
280
360

Random

14.4
13.2
12.9

18.1
16.2
16.1

1

1240

Random

11.2

14.2

0

Enhanced
Transformer-T

120
160
200

Mask-CTC

16.6
15.0
14.8

20.8
19.0
18.5

CBS-ASR

200
280
360

Mask-CTC

13.5
12.9
12.2

17.2
16.0
16.1

Latency [ms]Initialization

WER [%] (↓)

280

Random11.3

1240

Random9.8

280

Mask-CTC

11.1

30

40

50

60

70

80

D. Analysis of output token alignments
The work of [32] argued that the streaming model attends
to shift the token boundaries to the future side to obtain more
contextual information, which results in delay of the posterior
probability spikes for the output tokens compared to nonstreaming models. In contrast, if the encoder network learns
the feature representations that anticipate future information,
the output tokens can be confirmed earlier and the token
boundary shifting issue should be remedied in some instances.
Therefore, we measured the delay of the spike occurrencesin streaming models by comparing them to the alignments
obtained from a non-streaming model. The delay is expected
to be reduced with the Mask-CTC-based pre-training method.
We conducted measurements on the dev93 validation setof WSJ. We used the baseline and enhanced models with
200ms latency settings for Transformer-T and compared theiralignments with a non-streaming Transformer-T model. The
alignments were obtained from the output of the joint network.
For CBS-ASR, the latency was also set to 200ms, and wecompared the output token boundaries between the baseline
and enhanced models. The ASR alignments were obtained
from the CTC predictions of CBS-ASR in the same manneras [32] and the reference alignments were obtained with
the Montreal Forced Aligner [33]. Figure 2 illustrates one
example of output token alignments given by TransformerT. Here, the color in the background represents the referencealignment to the speech input. The non-streaming ASR (top)managed to predict accurate token alignments. However, theEnhanced
CBS-ASR

20models with low latency and high accuracy. For TED2 dataset,the enhanced CBS-ASR model also achieved 0.2 percentage
point of WER reduction compared to the baseline model,
which proves the general effectiveness of the proposed method
regardless of the dataset. The results for systems with different architectures, such as Transducer and Encoder-Decoder,also demonstrated that the Mask-CTC-based pre-training was
effective regardless of the model architecture.

Baseline
CBS-ASR

Streaming

Fig. 2. Output token alignments of non-streaming and streaming
Transformer-Transducer models.

TABLE IIW ORD ERROR RATE ON TED2 DATASET.
Model

Streaming
w/ Mask-CTC

Transformer-T, the acoustic encoder was initialized with
the Mask-CTC encoder. For CBS-ASR, both encoder and
CTC modules were initialized with corresponding MaskCTC modules.
The experimental results of Transformer-T and CBS-ASR
are summarized in Table I and Table II. Non-streaming
Transformer-T and CBS-ASR with 1240ms latency were used
as lower bounds in the experiments.
The results on WSJ show that for both Transformer-T and
CBS-ASR, the enhanced models outperformed the baselinemodels by achieving lower WERs under all latency settings,
suggesting the accuracy enhancements introduced by theMask-CTC-based pre-training method. For WSJ dataset, 40ms
and 80ms latency reductions were reached for Transformer-T
and CBS-ASR, respectively, while achieving better or equal
recognition accuracy than the baseline models. For instance,
the enhanced Transformer-T with 120ms latency achieved
lower WERs (16.6% for eval92 and 20.8% for dev93) thanthe WERs of the baseline with 160ms latency (16.8% foreval92 and 20.9% for dev93). Such results demonstrated that
our method contributed to the construction of streaming ASR

59

baseline streaming ASR (bottom) showed a significant delayin the alignments, indicating token boundary shifting due
to the lack of contexts. Meanwhile, our enhanced streaming
ASR (middle), with a Mask-CTC-based pre-trained encoder
network, largely improved the alignments of the streaming
ASR. We calculated the average output delay reduction acrossthe dev93 validation set for both Transformer-T and CBSASR. For Transformer-T, the spike output delay was reducedby 44ms, and for CBS-ASR, 46ms. Such results help us tounderstand the knowledge learned from the Mask-CTC-based
pre-training method and the reason for the latency reduction
capability.

[12] Niko Moritz, Takaaki Hori, and Jonathan Le, “Streaming automatic
speech recognition with the Transformer model,” in Proc. ICASSP, 2020,
pp. 6074–6078.
[13] Emiru Tsunoo, Yosuke Kashiwagi, and Shinji Watanabe, “Streaming
Transformer ASR with blockwise synchronous beam search,” in Proc.
SLT, 2021, pp. 22–29.
[14] Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, andTetsunori Kobayashi, “Mask CTC: Non-autoregressive end-to-end ASR
with CTC and mask predict,” in Proc. INTERSPEECH, 2020, pp. 3655–
3659.
[15] Yosuke Higuchi, Hirofumi Inaguma, Shinji Watanabe, Tetsuji Ogawa,and Tetsunori Kobayashi, “Improved Mask-CTC for non-autoregressive
end-to-end ASR,” in Proc. ICASSP, 2021, pp. 8363–8367.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,“BERT: Pre-training of deep bidirectional Transformers for language
understanding,” in Proc. NAACL-HLT, 2019.
[17] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer,“Mask-predict: Parallel decoding of conditional masked language models,” in Proc. EMNLP-IJCNLP, 2019, pp. 6114–6123.
[18] Mohan Li and Rama Doddipatla, “Non-autoregressive end-to-end
approaches for joint automatic speech recognition and spoken language
understanding,” Proc. SLT, pp. 390–397, 2023.
[19] Quentin Meeus, Marie-Francine Moens, and Hugo Van hamme, “Bidirectional representations for low resource spoken language understanding,” ArXiv, vol. abs/2211.14320, 2022.
[20] Huaibo Zhao, Yosuke Higuchi, Tetsuji Ogawa, and Tetsunori Kobayashi,“An investigation of enhancing CTC model for triggered attention-based
streaming ASR,” in Proc. APSIPA ASC, 2021, pp. 477–483.
[21] Sepp Hochreiter and Jurgen Schmidhuber, “Long short-term memory,”
Neural Computation, vol. 9, pp. 1735–1780, 1997.
[22] Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, RazielÁlvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu,
Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li,
Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo yiin Chang, Kanishka
Rao, and Alexander Gruenstein, “Streaming end-to-end speech recognition for mobile devices,” in Proc. ICASSP, 2019, pp. 6381–6385.
[23] Emiru Tsunoo, Yosuke Kashiwagi, Toshiyuki Kumakura, and Shinji
Watanabe, “Transformer ASR with contextual block processing,” inProc. ASRU, 2019, pp. 427–433.
[24] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro
Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann,
Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa
Ochiai, “ESPnet: End-to-end speech processing toolkit,” in Proc.
INTERSPEECH, 2018, pp. 2207–2211.
[25] Florian Boyer, Yusuke Shinohara, Takaaki Ishii, Hirofumi Inaguma, andShinji Watanabe, “A study of Transducer based end-to-end ASR with
ESPnet: Architecture, auxiliary loss and decoding strategies,” Proc.
ASRU, pp. 16–23, 2021.
[26] Douglas B Paul and Janet Baker, “The design for the wall street journalbased CSR corpus,” in Speech and Natural Language: Proceedings ofa Workshop Held at Harriman, New York, 1992.
[27] A. Rousseau, P. Deléglise, and Y. Estève, “Enhancing the TED-LIUM
corpus with selected data for language modeling and more TED talks,”
in Proc. LREC, 2014.
[28] Taku Kudo and John Richardson, “Sentencepiece: A simple andlanguage independent subword tokenizer and detokenizer for neural text
processing,” arXiv preprint arXiv:1808.06226, 2018.
[29] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret
Zoph, Ekin Dogus Cubuk, and Quoc V. Le, “Specaugment: A simple
data augmentation method for automatic speech recognition,” ArXiv,
vol. abs/1904.08779, 2019.
[30] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang,
Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, andRuoming Pang, “Conformer: Convolution-augmented Transformer forspeech recognition,” in Proc. INTERSPEECH, 2020, pp. 5036–5040.
[31] Niko Moritz, Takaaki Hori, and Jonathan Le Roux, “Triggered attention
for end-to-end speech recognition,” in Proc. ICASSP, 2019, pp. 5666–
5670.
[32] Hirofumi Inaguma and Tatsuya Kawahara, “Alignment knowledge
distillation for online streaming attention-based speech recognition,”
ArXiv, vol. abs/2103.00422, 2021.
[33] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner,and Morgan Sonderegger, “Montreal forced aligner: Trainable textspeech alignment using kaldi,” in Interspeech, 2017.

V. C ONCLUSION
In this study, an attempt was made to demonstrate theeffectiveness of Mask-CTC-based pre-training for achieving low latency and high accuracy in streaming speech
recognition. Experimental results showed the effectivenessof the method on various model architectures, including
Transformer-Transducer and contextual block streaming ASR.
Furthermore, by studying the output spike timings of thestreaming models, we discovered that more precise alignmentsof the input and output sequences are learnt by the pre-training,which contributes to the latency reduction in streaming ASR.
R EFERENCES[1] Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman
Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick
Nguyen, Tara N Sainath, et al., “Deep neural networks for acoustic
modeling in speech recognition: The shared views of four researchgroups,” IEEE Signal processing magazine, vol. 29, no. 6, pp. 82–97,
2012.
[2] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho,and Yoshua Bengio, “Attention-based models for speech recognition,”in Proc. NeurIPS, 2015.
[3] Alex Graves and Navdeep Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in Proc. ICML, 2014, pp. 1746–
1772.
[4] William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals, “Listen,attend and spell: A neural network for large vocabulary conversational
speech recognition,” in Proc. ICASSP, 2016, pp. 4960–4964.
[5] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber, “Connectionist temporal classification: Labelling unsegmented
sequence data with recurrent neural networks,” in Proc. ICML, 2006,
pp. 369–376.
[6] Alex Graves, “Sequence transduction with recurrent neural networks,”
ArXiv, vol. abs/1211.3711, 2012.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion
Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention
is all you need,” in Proc. NeurIPS, 2017, pp. 5998–6008.
[8] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott,
Stephen Koo, and Shankar Kumar, “Transformer Transducer: A streamable speech recognition model with transformer encoders and RNN-T
loss,” in Proc. ICASSP, 2020, pp. 7829–7833.
[9] Xie Chen, Yuehua Wu, Zhenghao Wang, Shujie Liu, and Jinyu Li,“Developing real-time streaming Transformer Transducer for speech
recognition on large-scale dataset,” in Proc. ICASSP, 2021, pp. 5904–
5908.
[10] Linhao Dong, Shuang Xu, and Bo Xu, “Speech-Transformer: A norecurrence sequence-to-sequence model for speech recognition,” in Proc.
ICASSP, 2018, pp. 5884–5888.
[11] Shigeki Karita, Nelson Yalta, Shinji Watanabe, M. Delcroix, A. Ogawa,and T. Nakatani, “Improving Transformer-based end-to-end speech
recognition with connectionist temporal classification and language
model integration,” in Proc. INTERSPEECH, 2019.

60