人工知能学会第二種研究会資料
AIチャレンジ研究会
SIG-Challenge-06-03

End-to-end integration of online and oﬄine encoders
using auxiliary losses for automatic speech recognition
Muhammad Shakeel1∗
1

Yui Sudo1

Yifan Peng2

Shinji Watanabe2(株) ホンダ・リサーチ・インスティチュート・ジャパン
1
Honda Research Institute Japan Co., Ltd.
2カーネギーメロン大学
2
Carnegie Mellon University

Abstract: End-to-end (E2E) automatic speech recognition (ASR) models have two desirable
properties: online and oﬄine modes. The online ASR mode, which operates under strict latency
constraints, processes speech frames in real-time to provide transcription. Conversely, the oﬄine
ASR mode waits for the complete utterance of speech frames before generating a transcription.
Recently, the integration of online and oﬄine ASR for recurrent neural network transducers (RNNT) can be achieved through the joint training of online and oﬄine encoders with a shared decoder.
However, this integration comes at the cost of performance degradation in the oﬄine ASR mode,as the shared decoder must handle features of varying contexts. Namely, with E2E integration
framework of online and oﬄine encoders, we explore two approaches to enhance the performanceof both the ASR modes. First, we introduce separate RNN-T decoders for each ASR mode while
maintaining shared encoders, thereby eﬀectively managing features of diﬀerent contexts. Second,
we explore multiple auxiliary loss criteria to introduce additional regularization, thereby enhancing
the overall stability and performance of the framework. Overall, evaluation results show 1.8%-2.5%
relative character error rate reductions (CERR) on corpora of spontaneous Japanese (CSJ) for
online ASR, and 4.4%-6.3% relative CERRs for oﬄine ASR within a single model compared toseparate online and oﬄine models.

1

Introductiontures of online encoder could lead to a conﬂict withfull-context oﬄine encoder. This is because the model
might not be able to eﬀectively diﬀerentiate between
the features relevant to each encoder, leading to theinclusion of irrelevant negative samples. To address
this issue, we propose an E2E-ASR framework that
integrates online and oﬄine ASR and combines theunique capabilities of both the ASR models.
In this study, our primary objective is to optimize
both online and oﬄine ASR modes. This framework
employs multiple encoders, with one designated foronline ASR and another for oﬄine ASR, and separate decoders for each. The framework extracts thehidden states of the online encoder and uses themas input for the oﬄine encoder. This method allows
us to integrate the functionalities of both encoders
while maintaining separate decoders for online andoﬄine modes. As a general approach rather thanrelying only on the cascaded integration [4] for performance optimization, our method introduces sepa-

End-to-end automatic speech recognition (E2E-ASR)
systems [1] strive to achieve low latency and high performance for a variety of tasks, including online [2]and oﬄine [3] ASR. However, the creation of distinct
architectures for each task is neither scalable nor ﬂexible. Therefore, a single E2E-ASR framework, capable of handling multiple tasks with high accuracy andadaptability, is preferable. One potential solution is
the joint training of online and oﬄine E2E-ASR tasks
using shared weights. However, most of the existing
methods suﬀer from negative transfer, a phenomenon
where the performance of one task interferes with another and degrades the performance of either of them.
For example, in the case of shared weights between online and oﬄine E2E-ASR, the limited contextual fea∗ 連絡先： (株) ホンダ・リサーチ・インスティチュート・ジャパン〒 351-0188 埼玉県和光市本町 8-1
E-mail: shakeel.muhammad@jp.honda-ri.com

9rate online and oﬄine recurrent neural network transducer (RNN-T) [5] decoders to leverage varying contextual information from both online and oﬄine encoders. Additionally, we employ connectionist temporal classiﬁcation (CTC) [6], attention mechanism [7],and masked language model (MLM) [8] based auxiliary losses to bring more regularization and reﬁnement to the E2E-ASR framework. The CTC loss
helps in aligning the input features with the output
labels, the attention mechanism provides a dynamicalignment between the input and output sequences,and the MLM loss aids in predicting the masked input features.
Through extensive experimentation, we have beenable to demonstrate the eﬀectiveness of each auxiliary
loss in improving the performance of the oﬄine ASR
model. Our results indicate a signiﬁcant improvementin the accuracy and eﬃciency of both online and ofﬂine ASR models, validating the eﬀectiveness of ourintegrated framework and auxiliary loss techniques.

2

図 1: An end-to-end architecture of online and oﬄine
encoders integration with auxiliary losses: In black,the original losses. In red the new auxiliary losses.

decoder may not eﬀectively diﬀerentiate between thelimited and full context features, leading to the inclusion of more negative samples; thus, degrading the
ASR performance. While multitask learning strives
to enhance the performance of multiple tasks concurrently, the selective use of auxiliary losses [13] cansupport the primary task and improve the generalization ability of the framework. Therefore, we propose
an end-to-end integration of online and oﬄine ASR
using separate RNN-T decoders and auxiliary losses,
such as CTC, attention, and MLM loss, each of whichbrings its own strength in a single model. We expectthat our proposed model can achieve better performance than the existing methods.

Related work

One of the current challenges in E2E-ASR is to develop a uniﬁed model [9, 10] that can handle both online and oﬄine scenarios. Online ASR is suitable forapplications that require low latency and real-time
feedback, such as voice assistants and online meetings. However, online ASR can only use limited context information from the past and present frames,
which may limit its accuracy and robustness. On theother hand, oﬄine ASR is suitable for applications
that do not have strict latency constraints, such as
oﬄine transcription and speech analysis. Oﬄine ASR
can exploit full context information from the whole utterance, which may improve its performance and generalization. Therefore, diﬀerent context information
may require diﬀerent acoustic features and networkarchitectures, making it diﬃcult to jointly optimize asingle model for both scenarios.
In recent studies [11], authors have explored theuniﬁcation of online and oﬄine encoders by using thesame decoder for diﬀerent input features, or by using
the output of the online encoder as the input of theoﬄine encoder [12]. However, integrating online andoﬄine encoders often face challenges in oﬄine scenarios, particularly when the online mode is prioritized
during optimization. A common approach involves
using a single shared decoder based on RNN-T to handle features of varying contexts. However, the shared

3

Framework

This section introduces the proposed end-to-end integration of online and oﬄine ASR framework, followed by detailed descriptions of each of our design
modules.

3.1

Online and oﬄine encoder

We propose a joint architecture that combines anonline encoder and an oﬄine encoder, each with its
own RNN-T decoder, to handle both online and oﬄine
application scenarios. The online encoder is basedon block processing [2], which preserves the previous
context information using context embeddings. The
oﬄine encoder is based on conformer [3], which captures the full context information from the whole utterance. The RNN-T decoders are based on recurrent

10neural network transducer, which models the sequential nature of speech and text. Figure 1 shows theoverview of our proposed architecture. The online encoder consists of M encoder layers, each of which has
a context inheritance mechanism. The context inheritance mechanism computes a context embedding foreach block at each sublayer, and passes it to the next
sublayer. The context embedding encodes the pastand present context information of the block. The
block size and hop length are denoted by Lblock andLhop , respectively. The b-th block of the input audio
feature sequence Xb is deﬁned as:

The RNN-T model optimizes its parameters by minimizing the negative log-likelihood, as deﬁned in Eq.
(5):∑
Lrnn-tlog Ponl (y | Honl ).
(5)
onl = −
(X→Honl ,y)This loss ensures that the model is continually optimized to learn the online contextual features during
training.

3.3

The oﬄine encoder-decoder modules receives theprocessed hidden sequences from the online encoderand employs full-context conformer as an encoder andseparate RNN-T as an oﬄine decoder. Here, the ofﬂine RNN-T decoder computes the marginal likelihood of the output y over all possible alignments, as
shown in Eq.(6):X b = (Xt |t = (b − 1)Lhop + 1, ..., (b − 1)Lhop+ Lblock + 1)

(1)

The hidden state for each block, labeled as the bth block, is encoded whereas each block contains aseries of hidden states of Lblock - length, i.e., H b =(hb1 , . . . , hbLblock ). The encoding process is carried out
sequentially, resulting in a series of hidden states witha length of T . These features are then input into theoﬄine encoder, where they are transformed into a subsampled sequence of hidden states, also of length T ,as given in the Eq.(3).
Honl = OnlineEncoder(X).

(2)

Hoﬀ = OﬄineEncoder(Honl ).

(3)

Poﬀ (y | Honl ) ==−Lrnn-t
oﬀ

3.4

∑

[T +S
∏

o∈O(y)i=1]
P (oi | hti , gsi ) ,

[T +S
∏

a∈A(y)i=1]
P (ai | hti , gsi ) ,(6)∑

log Poﬀ (y | Hoﬀ ),(7)

Auxiliary losses

In this study, we enhance the performance of theoﬄine ASR by optimizing the oﬄine encoder-decoder
module. This optimization is achieved through a multitask approach, where a full-context Conformer encoder is shared with the CTC, attention, and MLMbased auxiliary losses. The oﬄine encoder, which relies on the limited context hidden state features processed by the online encoder, often experiences performance degradation due to the restricted context
information. Our approach mitigates this issue andenhances the robustness of the framework by incorporating additional auxiliary losses into the oﬄine RNNT decoder. The inclusion of these losses serves a dual
purpose. Firstly, they contribute to the regularizationof the framework, ensuring stability during the learning process. Secondly, they aid in the optimizationof the model by providing additional signals for error
correction during training.

Ponl (o|X) =o∈O(y)∑(Honl →Hoff ,y)The acoustic features, denoted as X = (x1 , . . . , xT )are initially processed by the online module. This
module employs a contextual block conformer as an
encoder and the RNN-T as an online decoder, as described in [2] . The online RNN-T decoder computes
the marginal likelihood of the output y over all possible alignments, as shown in Eq.(4):∑Poﬀ (a|Honl ) =where the oﬄine RNN-T loss reﬁnes the model parameters by minimizing the negative log-likelihood,as shown below:

Online encoder-decoder loss

Ponl (y | X) =∑a∈A(y)In this study, we have the online contextual conformer
encoder functioning as an independent online encoderdecoder module. This is linked to an oﬄine encoderdecoder module via an output derived from the online
encoder.

3.2

Oﬄine encoder-decoder loss

(4)

11

3.4.1

CTC loss

Finally, the oﬄine loss (Loﬀ ) is computed using theweighted sum of individual loss objectives as deﬁned
in Eqs.(7), (9), (11) and (12):The likelihood of the CTC is given in Eq.(8):Pctc (y | Hoﬀ ) =∑P(z | Hoﬀ ) =z∈Z(y)∑[T∏z∈Z(y)

t=1

rnnt
att
mlm
Loﬀ = λctc Lctc
oﬀ + λrnnt Loﬀ + λatt Loﬀ + λmlm Loﬀ ,(13)
where λctc , λrnnt , λatt and λmlm are tunable hyperparameters and are determined experimentally. However, for this work we used the hyperparameters asreported in [14] and obtained optimal results.
Finally, we deﬁne the total multi-task learning objective for end-to-end integration of online and oﬄine
encoders as:]
P (zt | ht ) ,(8)where it serves as an auxiliary loss to reﬁne the model
parameters by minimizing the negative log-likelihood,as shown in the Eq.(9) below:∑
Lctc = −log Pctc (y | Hoﬀ ),
(9)

3.4.2

(Honl →Hoff ,y)Lmtl = λonl Lonl
rnn-t + λoﬀ Loﬀ

Auxiliary attention loss

where λonl and λoﬀ are the weighting terms and Lonl
is the online loss obtained from Eq.(5) and Loﬀ is theoﬄine loss obtained from Eq.(13).

The likelihood of an attention mechanism is described as in Eq.(10):Patt (y | Honl ) =S∏

P (ys | ys−1 , Hoﬀ ) ,

(10)

4

Experiments

4.1

Dataset(14)s=1

In this study, we primarily focus on a speciﬁc subset, referred to as subset A, of the Corpus of Spontaneous Japanese (CSJ) [15]. This subset consists ofacademic lecture-based ASR tasks and comprises 236
hours of speech data.
For evaluation purposes, we have divided the dataset
into three distinct tasks: eval 1, eval 2, and eval 3.
These tasks contain 1.9 hours, 2.0 hours, and 1.3
hours of speech data respectively, providing a comprehensive and diverse set of data for our analysis.

The auxiliary attention loss provides an alignment
between the input and output sequences and brings
model regularization by optimizing the corresponding
negative log-likelihood, as shown below:∑
Latt = −log Patt (y | Hoﬀ ).
(11)(Honl →Hoff ,y)3.4.3

Auxiliary MLM loss

The MLM auxiliary loss in our framework estimates
the token sequence using the full sequence given by
4.2 Experimental setup
Hoﬀ as shown in Eq.(3), similar to the attention mechanism. However, during the training phase, MLM disIn our study, we utilized the ESPnet2 toolkit [16]tinguishes itself from attention by masking randomly
as a foundation to build our baseline models and theselected tokens, denoted as ymask , with a special toproposed E2E framework. This framework is based onken.
the concept of multi-task learning for ASR task. We
Subsequently, ymask is predicted based on the reintegrated additional modules for encoder, decoder,maining unmasked tokens, yobs , as Pmlm (ymask |yobs , Hoﬀ ).
and auxiliary losses into this framework, capitalizing
Here, the MLM reﬁnes the model parameters by minon existing specialized architectures. Our online enimizing the negative log-likelihood as outlined in thecoder is composed of twelve layers of 256-dimensional
equation below:
contextual block conformer, each layer having 1024∑
feed-forward dimensions and 4 attention heads. We
Lmlm
log Pmlm (ymask |yobs , Hoﬀ ).
oﬀ = −
apply a dropout rate of 0.1 to each layer. For block(Honl →Hoff ,y)processing [2], we conﬁgure the block size to 40 and(12)
maintain a look-ahead and hop size of 16 to optiThis approach allows the MLM to contribute to themize online streaming performance. Our oﬄine enreﬁnement of the model parameters, enhancing thecoder comprises of twelve layers of 256-dimensionaloverall performance of the system.

12表 1: On Corpus of Spontaneous Japanese (CSJ) : Absolute (abs.) character error rate (CER) and relative(rel.) CERR numbers on CSJ data for i) single baseline models (B1, B3, B5, B6, & B7), ii) a baseline cascaded
encoder with shared decoder models (B2 & B4) [12], iii) proposed end-to-end integration of online and oﬄine
encoders with separate RNN-T decoders (P1 & P2) and auxiliary losses (P3, P4 & P5). All the results are
decoded with a beam size of 10.

CSJ (SUBSET A)
Mode

ID

Method

eval1

eval2

eval3

abs.↓ rel.(%)↑

abs.↓ rel.(%) ↑abs.↓ rel.(%)↑

OnlineB1
B2
P1

Context-Transducer (baseline)
Cascaded [12] (baseline)
Online-Transducer (ours)

6.84
6.81 (0.44)
6.67 (2.49)

4.95
5.07 (-2.42)
4.86 (1.81)

11.72
11.89 (-1.45)
11.79 (-0.60)

Oﬄine

B3
B4
P2

Conformer-Transducer (baseline)
Cascaded [12] (baseline)Oﬄine-Transducer (ours)

5.78
5.60 (3.11)
5.48 (5.19)

4.15
4.04
3.89

(2.65)(6.27)

9.94
9.68 (2.61)
9.50 (4.42)

B5
P3

Conformer-CTC (baseline)
Conformer+Lctc
oﬀ (ours)

5.50
5.51 (-0.18)

3.88
3.72

(4.12)

9.76
9.62 (1.43)

B6
P4

Conformer-Transformer (baseline)
Conformer+Lattoﬀ (ours)

5.16
5.29 (-2.46)

3.87
3.71

(4.13)

9.90
9.55 (3.54)

B7
P5

Conformer-MLM (baseline)
Conformer+Lmlm(ours)
oﬀ

5.53
5.58 (-0.90)

4.00
3.81

(4.75)

9.51
9.88 (-3.74)

Oﬄine

Oﬄine

Oﬄine

4.3

full-context conformer [3], each layer having 1024 feedforward dimensions and 4 attention heads. The output from the online contextual block conformer ischanneled into the oﬄine conformer block, resulting
in a cascaded architecture. For the online encoder,we employ a separate RNN-T decoder with a 256dimensional embedding prediction network and a 320dimensional joint network. Conversely, for the oﬄine
encoder, we utilized a distinct RNN-T decoder with a
256-dimensional embedding prediction network and a320-dimensional joint network. To augment the regularization of the oﬄine encoder-decoder, we introduced auxiliary losses based on CTC, attention, andMLM mechanisms. We train this end-to-end architecture for 50 epochs with a learning rate of 0.0015 andwarmup steps of 1500. We employ a training weight of
1 to the online encoder to maximize the performancecapacity of the online ASR mode. However, for the ofﬂine mode and auxiliary losses, we adopt the training
weights for λrnnt , λctc , λatt , λmlm as proposed in [14],i.e., 0.10, 0.15, 0.30, and 0.45 respectively.

Main results

In this work, we evaluate the performance of our
proposed E2E framework with several baseline models. These include the standalone online contextual
block conformer transducer (Context-Transducer), theoﬄine full-context conformer transducer (ConformerTransducer), and a cascaded architecture with a shared
RNN-T decoder, as proposed in a previous study [12].
To ensure a fair comparison, we maintained the same
number of encoder layers (twelve) for both online andoﬄine modes in all models, including the Context-T
and Conformer-T.
For our ﬁrst primary analysis, we perform an ablation study conducted on the Corpus of Spontaneous
Japanese (CSJ) dataset, and is presented in Table 1.
In this table, the standalone online and oﬄine transducer baseline models are represented by ContextTransducer (B1) and Conformer-Transducer (B3), respectively. We also developed a baseline cascaded architecture with shared decoders (B2 & B4) as proposedin the referenced study [12]. These models are compared against our proposed framework (P1 & P2). The

13

参考文献

character error rates (CER) listed in Table 1 are obtained using a beam width of 10 for both online andoﬄine RNN-T modules. Our ﬁndings indicate thatthe proposed framework improved the performance ofthe online ASR path compared to the standalone online model. We observed a relative CER (CERR) improvement ranging from 1.81% to 2.49% across multiple evaluation sets. Moreover, our proposed framework also demonstrates substantial performance improvement for the oﬄine ASR path, with a CERR
between 4.4% and 6.3%. These results highlight theeﬀectiveness of our proposed E2E framework in improving the performance of both online and oﬄine
ASR modules.
att
Next, we compare our auxiliary tasks Lctc
oﬀ (P3), Loﬀ(P4) and Lmlm
(P5) against the separately trainedoﬀ
Conformer-CTC (B5), Conformer-Transformer (B6),
and Conformer-MLM (B7) models. Table 1 summarizes our experiments to understand how each task
improved or degrades the performance on CSJ evaluation sets compared to the standalone models. Overall, most auxiliary tasks show improved performanceon eval2 and eval3 test sets with an exception for test
set eval1 which shows performance degradation. Thisdegradation can be attributed to the fact that training weights in this study are optimized to improve
the overall performance of the online and oﬄine transducer modes. The performance for each auxiliary task
can potentially be improved by assigning more optimal weights to the individual task. It will allow themodel to focus more on optimizing the performance ofeach auxiliary task, rather than prioritizing the overall performance of the transducer modes.

5[1] R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schlüter,and S. Watanabe, “End-to-end speech recognition: A
survey,” 2023.
[2] E. Tsunoo, Y. Kashiwagi, T. Kumakura, andS. Watanabe, “Transformer ASR with contextual
block processing,” in Proc. ASRU, 2019.
[3] A. Gulati, C. Chiu, J. Qin, J. Yu, N. Parmar,
R. Pang, S. Wang, W. Han, Y. Wu, Y. Zhang,
and Z. Zhang, “Conformer: Convolution-augmented
transformer for speech recognition,” in Proc. Interspeech, 2020.
[4] B. L. et al., “A better and faster end-to-end model
for streaming asr,” in Proc. ICASSP, 2021.
[5] A. Graves, “Sequence transduction with recurrent
neural networks,” in Proc. ICML, 2012.
[6] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classiﬁcation: Labelling unsegmented sequence data with recurrent
neural networks,” in Proc. ICML, 2006.
[7] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in Proc.
ICASSP, 2016.
[8] Y. Higuchi, S. Watanabe, N. Chen, T. Ogawa, andT. Kobayashi, “Mask ctc: Non-autoregressive endto-end asr with ctc and mask predict,” in Proc. Interspeech, 2020.
[9] F. Weninger, M. Gaudesi, M. A. Haidar, N. Ferri,
J. Andr’es-Ferrer, and P. Zhan, “Conformer withdual-mode chunked attention for joint online and ofﬂine asr,” in Proc. Interspeech, 2022.
[10] Y. Sudo, M. Shakeel, Y. Peng, and S. Watanabe,“Time-synchronous one-pass beam search for parallel online and oﬄine transducers with dynamic block
training,” in Proc. Interspeech, 2023.
[11] J. Yu, W. Han, A. Gulati, C.-C. Chiu, B. Li, T. N.
Sainath, Y. Wu, and R. Pang, “Dual-mode {asr}:
Unify and improve streaming asr with full-context
modeling,” in Proc. ICLR, 2021.
[12] A. Narayanan, T. N. Sainath, R. Pang, J. Yu, C.-C.
Chiu, R. Prabhavalkar, E. Variani, and T. Strohman,“Cascaded encoders for unifying streaming and nonstreaming asr,” in Proc. ICASSP, 2021.

Conclusion

In this study, we propose a novel approach to integrate online and oﬄine ASR modules in an end-toend manner, utilizing auxiliary losses. This framework is designed to optimize the combination of online and oﬄine RNN-T decoders, leveraging the powerof multi-task learning. The primary objective is toenhance the learning of contextual representations,
thereby oﬀering increased ﬂexibility in the E2E-ASR
framework. Our approach demonstrates a signiﬁcant
improvement in CERR for the CSJ corpus, compared
to traditional cascaded architectures [12]. This improvement is particularly noticeable with the introduction of auxiliary losses, which provide additional
regularization and reﬁnement to the framework.

[13] C. Liu, F. Zhang, D. Le, S. Kim, Y. Saraf, andG. Zweig, “Improving rnn transducer based asr with
auxiliary tasks,” in Proc. SLT, 2021.
[14] Y. Sudo, M. Shakeel, B. Yan, J. Shi, and S. Watanabe, “4D ASR: Joint modeling of CTC, attention,transducer, and mask-predict decoders,” in Proc. Interspeech, 2023.
[15] K. Maekawa, H. Koiso, S. Furui, and H. Isahara,“Spontaneous speech corpus of japanese,” in Proc.
LREC, 2000.
[16] S. W. et al., “Espnet: End-to-end speech processing
toolkit,” in Proc. Interspeech, 2018.

14