In-Situ Text-Only Adaptation of Speech Models with Low-Overhead Speech Imputations
Authors: Ashish Mittal
Sunita Sarawagi
Preethi Jyothi
Venue: ICLR 2023 poster (2023)
Keywords: Text-Only Adaptation
End-to-end Speech Recognition

Abstract: Fast and accurate adaptation of automatic speech recognition (ASR) systems using only text data in the target domain is a problem of long-standing practical relevance. Text-only adaptation was easy in traditional cascaded ASR systems with completely decoupled acoustic and language models. Recently, the RNNTransducer (RNN-T) has emerged as a default ASR model because of its high accuracy, low latency, and capability of supporting streaming input. However text-only adaptation of the RNN-T model is significantly more challenging due to its tight integration of acoustic and language models and end-to-end training. Existing recent approaches for text-only adaptation of RNN-Ts, either entail significant modification to the network or introduce high latency during decoding. We propose a new approach (TOLSTOI) that imputes speech representations internal to a baseline RNN-T, starting from text-only inputs, and performs in-situ adaptation that results in higher adaptation accuracy without any runtime overheads during decoding. Our imputation model is a function of the labeled data and trained parameters of the ASR model, and that we show, is more effective in controlling catastrophic forgetting compared to existing methods. We establish the effectiveness of TOLSTOI using three target domains and two ASR models of varying complexity. We yield up to 35% relative reduction in word error rate with text-only adaptation while forgetting the least compared to existing adaptation approaches. Our method is easy to implement and can be harnessed on existing RNN-T models without requiring ASR model training from scratch.

Published as a conference paper at ICLR 2023
IN-SITU TEXT-ONLY ADAPTATION OF SPEECH MOD- ELS WITH LOW-OVERHEAD SPEECH IMPUTATIONS
Ashish Mittal
Sunita Sarawagi & Preethi Jyothi
IBM Research, IIT Bombay arakeshk@ i bmm com
IIT Bombay
{ (sunita, p iitb.ac.in
ABSTRACT
Fast and accurate adaptation of automatic speech recognition (ASR) systems ing only text data in the target domain is a problem of long-standing practical relevance. Text-only adaptation was easy in traditional cascaded ASR systems with completely decoupled acoustic and language models. Recently, the RNN- Transducer (RNN-T) has emerged as popular ASR model because of its high accuracy, low latency, and capability of supporting streaming input. However text-only adaptation of the RNN-T model is significantly more challenging due to its tight integration of acoustic and language models and end-to-end training. Existing recent approaches for text-only adaptation of RNN-Ts, either entail sig- nificant modification to the network or introduce high latency during decoding. We propose a new approach (TOLSTOI) that using text imputes speech represen- tations internal to the ASR model, and performs in-situ adaptation that results in higher adaptation accuracy without any runtime overheads during decoding. Our imputation model is function of the labeled data and trained parameters of the ASR model, and that we show, is more effective in controlling catastrophic for- getting compared existing methods. We establish the effectiveness of TOLSTOI using three target domains and two ASR models of varying complexity. We yield up to 35% relative reduction in word error rate with text only adaptation, while for- getting the least compared to existing adaptation approaches. Our method is easy to implement and can be harnessed on existing RNN-T models without requiring ASR model training from scratch.
INTRODUCTION
Text-only adaptation of end-to-end (E2E) automatic speech recognition (ASR) systems to new target domains is of much practical interest since in many situations, e.g. mobile phones, it is easier to get target-specific text data than the corresponding audio. Efficient and effective text-only adaptation remains an open problem in large part due to the nature of E2E ASR systems that use single model to jointly learn both a mapping from speech to text and a language model (LM), thus rendering traditional LM adaptation techniques for ASR (Bellegarda 2004) ineffective.
RNN-Transducer (RNN-T) models (Graves 2012) are one of the most popular E2E ASR architec- tures that achieve high accuracy and enable real-time decoding of speech, thus making them the predominant choice for ASR on mobile devices (He et al. 2019). Customizing RNN-T models us- ing text-only data has gathered momentum in recent years. For any ASR applications using RNN-T models, running at real-time is a critical requirement. Thus, we seek simple and accurate text-only adaptation techniques that do not increase the model complexity. an approach TOI.STOI that is simple in its design and works with pretrained RNN-T models while enabling fast and accurate adaptation to the target domain. We will first review existing approaches to the problem of text-only adaptation to help contextualize TOLSTOI better.
this work, we propose such
A popular solution for text-only adaptation of E2E ASR systems is shallow fusion (Hanun et al., 2014) where scores from the E2E model are combined with scores from an external LM trained on the target text during beam search decoding. While simple in its design, this technique significantly increases decoding time due to the reliance on external LM during inference. More recent work on adapting RNN-T models using only text aims at directly updating the parameters of the prediction network (Pylkkonen et al. 2021 Chen et al. 2022a However such techniques do

---

Published as a conference paper at ICLR 2023
not yield very accurate adaptation to the target text and also involve architectural changes to the RNN-T that necessitate training the model from scratch. Text-only adaptation can also be tackled by generating corresponding speech via text-to-speech synthesis (TTS). The main limitations of TTS-based adaptation are significant computational costs and the reliance on high-quality TTS systems that are available only for a small subset of high-resource languages and accents.
From these prior works, the key requirements for practical text-only adaptation of ASR that emerge are i) the model should adapt to target-domain text-only data with high accuracy ii) the adaptation should be applied to existing pretrained models without any retraining iii) the inference should be fast and inexpensive and iv) the adaptation should not lead catastrophic forgetting (Goodfellow iet al.) 2011): Takashima et al. [2022.. We propose TOLSTOI that addresses all four requirements. Starting from text in the target domain, we impute speech representations as would have been produced by the transcription network of a pretrained RNN-T model. Our imputation model is a simple feedforward network (with roughly 200K parameters) that incurs minimal overhead in its training by harnessing forced alignments and representations from the ASR model. Using the trained imputation model, we generate sequences of speech representations for all the text in the target domain which are used for in-situ adaptation of the RNN-T ASR model. TOLSTOI can be used with any existing pretrained RNN-T. We do not introduce any new parameters in the RNN-T and do not rely on any external LMs, thus incurring additional overhead latency at inference time. Along with yielding fast and accurate adaptation to the target domain, TOLSTOT also safeguards against forgetting since the imputation model is trained mimic representations from the source distribution. TOLSTOI yields up to 35% relative word error rate (WER) reduction on a new target domain, while maintaining the same decoding latency as the base RNN-T model and ensuring minimal forgetting of its source information when compared to three other competitive baselines. We also present detailed ablation study to justify the various design choices of TOLSTOI.
2 RELATED WORK
LM adaptation in traditional ASR systems Unlike end-to-end models, traditional ASR sys- tems adopt cascaded structure with the LM being completely decoupled from the acoustic model (Mohri et al. 2002)) This enables easier adaptation of the LM to target domain (Hori et al., 2003 Bellegarda| 2004] Neubig et al.] 2009; Gangireddy et al., (016)) and also allows for ASR lattice rescoring with an external L.M (Park et 2010; Xu et al. 2018).
LM fusion A popular approach for text-only adaptation of end-to-end ASR is "shallow fusion" where an external LM is log-linearly interpolated with the RNN-T output during beam decod- ing Kannan et al. (0018) For RNN-T models, another recent approach is to extract internal LM probabilities and discount with the ratio of external and internal LM probabilities (McDermott et al., 2019 Meng et al. 20 b Udagawa et al. 2022). These techniques incur a significant overhead at inference time due to the external LM and also require careful tuning of the interpolation weight used for the external LM.
Synthesizing audio Another approach to text-only adaptation is synthesize audio using text-to- speech (TTS) synthesis (Zheng et al. 2021 Deng et al. 2021 Joshi & Singh [022] Hayashi et al.] 2018 Hori et al. [2199] Baskar al. 202 Chen et 2022c). However, this slow generation process and relies on access to high-quality TTS Shen et al. (2018) which is absent for most languages. To address these issues, recent work on text-only adaptation has investigated generating simpler pseudo-speech representations called "textograms" by repeating one-hot encodings of the output labels for fixed duration (Thomas et al. (022). The input the RNN-T is augmented to accept textogram as an additional channel. This model requires training the RNN-T from scratch and also negatively impacts the decoding latency.
Fine-tuning RNN-T model parameters Recent approaches exploit the inherent structure of the RNN-T to perform in-situ text-only adaptation. Pylkkonen et al. (202I) adds a separate LM output head to the prediction network in RNN-T (that handles text-only inputs) and both are jointly finetuned using the target text. [Chnn et al. (202a)) first factorize the prediction network into two networks that separately handle "blank" tokens (capturing alignment with the audio) and the output vocabulary tokens, before adapting the latter with the target text. This technique requires retraining the RNN-T model and does not yield accurate adaptation.
2

---

Published as a conference paper at ICLR 2023
Ficartly slovs]
inferince time
M
External LM
My
Head
Papendan Rwiwiring
h1
h
9
Imputation Model (faar)
9a
Textogram
My
Mr
h
Mr
ht
9u-1
Ztt
9w-1
yu-1
yu-1 1
Standard RNN-T model and three prior approaches
STOO
Figure 1: Figure on the left shows a schematic diagram the standard RNN-T architecture (in blue). Three existing approaches for text-only adaptation are shown as appendages the RNN-T model (in red): (1) "Tex- togram" increases the dimensionality of the input to the transcription network M,s thus requiring an expensive model retraining. (2) Shallow fusion uses external LM during inference that leads significant degradation of test-time latency. (3) A separate LM head can be added and jointly trained with the prediction network on the larget text. This, however, does not result in very accurate adaptation. The figure on the right shows TOLSTOI that uses a lightweight imputation model to generate speech representations corresponding to the target text.
Figure ssowss a technique from each of the mbnte-mentioned categories and how integrated within the standard RNN-T architecture. (A recent line of work focuses on learning shared speech and text representations, thus allowing for the use of unpaired text (Bapna al.] 2021 Ao et al. 2021 Tang et al.] 2022 Chen et al.) 2022b). An extension of these ideas to streaming RNN-T models (Sainath et al. 2023) was concurrent to our work and would be interesting to explore further.)
3 OUR APPROACH (TOLSTOI)
We are given an ASR model M trained on labeled dataset D {(x yy)..., (x>,, y M) where x' E X, the space of speech utterances, and y' Y, the space of text transcripts. Each speech utterance x' comprises of variable number of frames ,i,...,,ff 2/, where each æ; is a fixed-length real-valued vector denoting features such as spectrogram of the audio frame. Each text sequence comprises of variable number of tokens y' = (vi...., ..;;) where each Yo E V. the output vocabulary. Popular choices for the output vocabulary are characters and subwords (Sennrich al. 2015). Typically the number of text tokens U, < T,, the number of audio frames. Let P)
denote the distribution of speech and text from which the training data is sampled.
Our goal is to deploy the ASR model M on target domain whose distribution P(X.)) differs from the training distribution. For the target domain, we only have text data D {ū) ...... 8*) where the number of text samples in D in the target is generally much smaller than the size of the training set D. Since we are only given text samples in the target distribution we assume that the training and target distributions differ only on the text marginals P(Y) and the distribution of the speech given the text stays unchanged. That is, P(X = P(Xp))) We seek to use Đ to fine-tune the parameters of M so that the updated model M is accurate on speech corresponding to new text from the target distribution P, without catastrophically deteriorating accuracy on samples from the training distribution P. We propose to perform in-situ adaptation without introducing new layers or external LMS during deployment.
As mentioned earlier, we focus adapting the RNN-Transducer (RNN-T) architecture (Graves) 2012 Graves et al. 2013) as the ASR model since it has recently emerged as a popular choice, par- ticularly in mobile devices, because of its high accuracy, low latency, and capability of supporting streaming input. We first present a brief background of RNN-Ts.
Background of RNN-T The RNN-T network comprises of three modules: (1) A speech module Ms with parameters Og that converts speech frames x 21 ...,..,2 Fe vectors h hi.... a hr typ-
3

---

(hss gs) ten Published as a conference paper at ICLR 2023
h2
ha
hs
ha
(h.-1990)
90
Best Alignment
(hv.go) h2
91
(h2. 91) - hg
y2
(hs. 92) - h4
JIMP
9a
(A4. 92) hos
Y3
93
h+
Figure 2: Training the imputation model of TOLSTOI using RNN-T alignments.
ically using Transformer or recurrent network, (2) A language module Mr with parameters Or. ................................ yu to contextual vectors 91..... IU using a recurrent network
that converts text tokens y = V/.............................. so that Iu summarizes the tokens before Yus and (3) A thin joint network MM with parameters 0,, that combines vectors h, and 9n. and outputs softmax distribution spanning the vocabulary V plus the blank symbol Ø. generate token sequence y. the RNN-T implicitly aligns each token yu to one frame t where it is output. An example of a valid alignmen apppars in Figure
Even though the network modular, all parameters are trained jointly end-to-end by maximizing the likelihood of the output y given speech æf over all (x',y') E D. During training, the log- likelihood of the target sequence y' is computed by marginalizing over all possible alignments of the T, frames of x' and the U tokens of using an efficient forward backkard algorithm; will refer to this objective as the RNNT-loss (Graves 2012).
Σ RNNT-loss(y', {My(199. = MM Mu-1), -Ms(01 æT, ()}) (w (w'))")
min 0s.0s.0,
During inference, beam-search finds the best possible alignment (t, u) and the predicted sequence a concatenation of the non-blank tokens at each aligned (1, u) (Graves 2012 Saon al. 2020).
Motivation of our approach Figure[]rreeent schematic diagram of our imputation based adap- tation. Given only text data D for adaptation, we propose to update the language parameters 0 and joint parameters 0, while keeping the speech parameters 0g fixed. This is challenging since even though the network architecture is modular, the training is not. Treating the M as a language model and updating part of the network using text-only data, as proposed in (Pylkkonen et al. 2021 Chen et al. 2022a), has the potential of deteriorating performance as the output vector of M gets incom- patible with the vector from Ms. We, therefore, propose to first augment the missing speech data in D by imputing using a separate generator. However, training a full-fledged TTS model requires sub- stantial training data and resources, and high-quality TTS models are not available for low-resoures languages. Our key insight is to impute, not the full raw speech æ, but the h vectors from the last layer of the speech module Ms. Generating the last layer vectors h is significantly casier than gen- erating the raw audio signals æ. In fact, only a thin joint layer separates h from the output character distribution. So the h vectors are expected be "closer" to text than speech. We are able to design a very simple and low-overhead model for imputing the h vectors from the text. In Section 3.] we describe the design and training of the imputation model. Once the imputation model is trained, for any new target domain, we attach each text y E δ with imputed values create a proxy parallel dataset for fine-tuning M. and Mr. In Section 3.2 we describe how we perform this fine-tuning.
3.1 IMPUTATION MODEL
Let H denote the space of vectors from the last layer of Mg. The goal of our imputation model is to directly model (HH))) instead of P(XD)) that TTS models attempt to do. The imputation model generates proxy output vectors hi,..., hr of the speech module MM given only a text sequence V1....9c yu so as to mimic the output of the speech module MM(x1, TT) without having access to the real audio frames 21,..., A default option is to train full-fledged sequence to
In valid alignment, at each step u the output character is either yu or Ø so that the concatenation of the tokens matches y after ignoring the z.
4

---

Published as a conference paper at ICLR 2023
hy
h1
lts
hss
Ma
10
9u
M1
Mr
h0
hgp-1
Mr.
90
gi
92
9h
yo
91
9u-1
Vu-1
Figure 3: RNN-T finetuning using imputed speech representations from fIMP with alignments from the FixedGram model.
sequence model that takes as input text tokens y1.... yu and generates h ,,. ht using a standard encoder-decoder network. However, such an approach would require training a heavy-weight model from scratch. Instead, we developed simple low-overhead model that we present next.
Designing the Low-overhead Imputation Model Our approach is to leverage the existing trained RNN-T model M to reduce the overheads of generating the h sequence given token sequence y- First, we reduce the need for encoding discrete tokens VI..... yu from scratch by starting from the output gI..... 9u of the language model Mr, Second, we reduce the need for cross-attention training of a full seq2seq approach by pre- pre-aligning the h sequence to the token sequence y. We denote an alignment as A, a sequence of (t, u) pairs to show that the LM state gu was aligned with audio frame h4 in a valid execution of the RNN-T to generate a sequence. We will discuss how we generate such alignments during training and deployment of the Imputation model. Given an alignment A we factorize the generator of the h sequence as follows:
hrlyI. YU, A) = II (I,
(h
In the above we have further assumed that h, depends only on ht-1 and is independent of other prior h,ss given h--1- We model the above distribution using an imputation model fIMP which consists networks that takes as input the language model encoding Iu and of a simple feedforward produce the next audio encoding h,. Figure 2 presents an overview. the last audio encoding h,
Training the Imputation Model To create the training data for the imputation model fIMP. use the training data of the ASR model. For each utterance æ' in D. we trace the lattice of the highest probability beam to find the best valid alignment A' that consists of sequence of h, and the aligned 9a as shown in Figure 2 We use the alignment-length synchronous decoding algorithm (Saon et al. 2020) for this. From these alignments we extract training instances for the imputation model as ( ), h2) for each (4, u) E A'. When multiple gu, 's align with the same h, in the best alignment output, we select the last gu- We train the parameters of the fIMP model to minimize the reconstruction loss of h; with input (h:-1.gu)) Iw) as follows:
frup arg min
Σ (A.))AA
Loss(hi, f(i
where he = and the space of f are parameters of a standard feed forward network as described above. For the loss function, we considered several candidates and found the L1 distance to provide the best results as we show in our ablation studies.
3.2 RNN-T FINETUNING USING THE IMPUTATION MODEL
In this section we describe how we fine-tune the RNN-T model M on the target text D using the trained fIM model. For each text y' E D we first sample an alignment A' as elaborated in Section 3.2.. Next use A' and fIMP to sample sequence of vectors h' hi,.... ht A: as shown in the alignment example in Figure 2 We first feed the token sequence y to the language module to get a sequence of outputs ...... gu. Next we invoke the imputation model fIMP to generate for each (t, u) A in increasing order of t, h, ( gu). This gives us pseudo labeled training set Dimp = [(h', )')} that we fine-tune parameters 0r and f, of the RNN-T using the same loss
5

---

Published as a conference paper at ICLR 2023
function of maximizing the likelihood of y over all possible alignments. Since the generated h come from the Pr(h/3/ y) distribution, and only the P(y) distribution changes in the target distribution, this fine-tuning step adapts to the target distribution.
min 0r
RNNT-loss(y, {Ma(-1gu = (u Bu Mu-1),h4))
(3)
3.2.1 GENERATING ALIGNMENTS GIVEN A TEXT SEQUENCE
We tried many different wight methods of generating alignments for a given text token sequence y. A simple method is to generate a fixed number of blanks(@) before each to- ken Yu- We call this the FixedGram model. A second option is to train a distribution over the number of blanks for each token v V. Finally, a more evolved method would be to use a state- based model, much like the imputation model that takes input Iu and h to generate if the next token should be blank. We will show that the simple FixedGram model was adequate for adaptation task. Figure [ presents an overview of our steps during fine-tuning under the FixedGram alignment model. Algorithm presents the overall pseudocode of TOLSTOI. Note, the training of the imputation model is performed once and is independent of the adap- lation dataset.
Algorithm 1 Text-only adaptation in TOLSTOI
Require: Trained RNN-T model M, Training data D, Adaptation Text D for (æ "0',y/ y²) E D do
A'. hh' . Apply .M on speech extract alignments and h, g vector sequences end for
fImP = Train imputation model aligned pairs hi, 9: using Equation
1* Fine-tune M for D 8/ for E D do
91 .....gu gu MM)y). ho= GGeneraatAliiuunnnttig
(f, u) E A in increasing order of do h = fiMr(h,-1,.9
end for loss RNN-T loss(softmax Backpropagate(loss, (0., 0r.))
end for Return 0z. 0,,
4 EXPERIMENTS
We present an extensive evaluation of TOLSTOI against three existing approaches with two ASR models and three target domains.
4.1 EXPERIMENTAL SETUP
We perform adaptation experiments on two RNN-T models of very different capacity: (i) SWB 300H trained on the Switchboard 300H dataset (Godfrey et al. 1992) containing over 300 hours of labelled training data and (ii) SWB 2000H trained on 262 hours of Switchboard-300
1698 hours
of Fisher data (Cieri et al. 2004) and 15 hours of CallHome data (Martin & Przybocki 2000).
The transcription network the RNN-T model consists of bidirectional LSTM layers with hidden size of 640 cells and projection layer that projects down the encoder embeddings to 256 dimensions. The prediction network consists single-layer unidirectional LSTM with a hidden size of 768 cells and projection layer reducing the prediction network embeddings 256 dimen- sions. The joint network first combines the transcription network embeddings (256-d) and prediction network embeddings (256-d) via a Hadamard product joint operation (o) (Saon et al.) [202I), fol- lowed by tanh non-linearity and a final output softmax layer that produces probability distribution over 45 non-blank characters (V) and blank character (Ø). The total number of parameters in the RNN-T model is roughly 56 million.
The audio features are mean and variance normalized log-Mel filterbank features computed for every 10ms of the audio. These features are further appended with delta-spectral and doublc-delta spectral features( (Mason & Zhang 1991 Picone 1993)) where every two consecutive frames are stacked resulting in 2400-dimeniional (240-d) input vectors with receptive field of 20ms. Speed and tempo perturbations are applied to each input with the value of 0.9, 1.1 to augment the training data. We also use SpecAugment (Park et al. 2019) for additional augmentation. The RNN-T models were trained for 20 epochs using Adam (Loshchilov & Hutter] 2017) optimizer with the maximum learning rate of 5e-4 and the OneCycleLR policy (Smith & Topin 2019) policy consisting of linear warmup phase from 5e-5 to 5e-4 followed by a linear annealing phase O. A batch size of 64 was used to train the Pytorch models on V100 GPUs.

---

Published as a conference paper at ICLR 2023
Adaptation Datasets For the adaptation experiments, we choose three diverse datasets with differ- ent amounts of adaptation text coming from three different domains. (1) ATIS (Hemphill et al. 1990) consists of roughly 5K sentences from the airline reservation domain for training and 893 (speech, text) utterance pairs for testing. (2) CarperValleyyBan
(HVB) (Wu et al. 2020) consists of roughly 15K sentences from the banking domain for training and 2797 (speech, text) utterance pairs for testing. (3) Librispeech (Panayotov et al. 2015) consists of roughly 29K sentences from audiobooks for training and 2619 (speech, text) utterances for testing. Note, in true spirit of text-only adaptation, we do not assume the availability of parallel validation dataset in the target domain.
Imputation Model The imputation model of TOLSTOI is 2-layer feed-forward network with a tanh non-lineant -oneaiiaar The first layer projects the 512-d imput of (h4-1. to 256-d and the second layer outputs 256-d l4. The imputation model contains roughly 200K parameters, in contrast to the 56 million parameters of the ASR model. The imputation model is trained using the same learning rate, optimizer, and learning rate schedule as the baseline training with a batch size of 2048. The SWB 300H training data was aligned with the corresponding ASR model yielding total of 42 million (h,, 9a) pairs to train the imputation model.
Fine-tuning details For fine-tuning of My using the imputation model, we keep the same optimizer and learning rate scheduler as the starting RNN-T training except that the maximum learning rate used for fine-tuning was 5c-5. We fine-tune for fixed number of 2000 updates since we do not have a validation set for target-specific hyper-parameter selection.
Metrics We evaluate three metrics: (1) RTF; Real Time Factor(RTTF) which measures the average time (in seconds) to decode one second of audio (50 audio frames in our case). is computed by averaging over the entire test set single CPU. (2) WER on Target-Only: Word error rate(WER) on the test set of the target domain. (3) WER on the target domains datasets is small. In any ASR system deployed in the real world, it is important for the adapted model to not be significantly worse than the unadapted model on sentences outside its narrow domain because of catastrophic forgetting during adaptation. Therefore, we also measure the average WER of the target and source test datasets. The test data of the source domain comprises 4458 (speech utterance, text) pairs from Hub5'00 00 and CallHome.
Mixture: The test sizes for each of
We compare TOLSTOI against three existing methods of text-only adaptation of RNN-Ts.
Shallow Fusion (Kannan et al.) 2018) is a standard method which uses an external LM trained on the target domain text to interpolate the RNN-T probabilities during inference. For the external LM, we use the same configuration as our prediction network and set the interpolation parameter (A) to 0.3 for all cases. Since do not assume access to any validation set from the target domain, fine-tuning the byper-parameter is not an option.
NN-LM In this method, the prediction network is viewed as an auto-regressive language model. To get the LM scores, a linear layer is added to project the prediction network representations to the output vocabulary space V and fine-tuned on the target domain text for one epoch.
Textogram (Thomas et al. 2022) In this method, the transcription network is modified to work with two input modalities:
(i) standard acoustic features and (ii) one-hot encoding of the units in the text, with the encoding of each unit being repeated a fixed number of times such that each feature has duration similar to the spectrograms. During the training of the base model, either the acoustic feature or textogram feature is used for training the RNN-T objective. During fine-tuning, only the textogram features for each text from the target domain are used for adaptation. Different from NN-LM, in textogram RNN-T T loss is used for fine-tuning the joint network and prediction network parameters much like in our method.
Other methods We also compared with two other methods: (I) The factorized model proposed in Chen et al. (2022a). However, their factorized RNN-T model had significantly worse accuracy than our unadapted RNN-T model both before and after text-only adaptation.
(2) TTS where we used a proprietary IBM synthesis engine (Knn et al. 2019 Fernandez et al. (022) based on Tacotron (Shen) et al. 2018) to generate audio for the text-only inputs. This gives competitive reduction in WERs of mixed (3.2/6.5 and 8.6/9.8 on the ATIS and HVB test sets, respectively). However, it is worth noting here that all three target domains consist of US- English speech and the TTS system we employ is carefully tuned to produce natural-sounding US accented English samples. Building TTS of similarly high quality for low-resource languages is non-trivial
mrnget-only/sourc-ttrrg
7

---

Published as a conference paper at ICLR 2023
ATIS (WERC))
HVB (VB(WERC)
Librispeech (WER(.)
Baseline Model
Methad
RTF (sec)
Target
Mixed
Target
Mixed
Target
Mixed
Unadapted
0.33
5.8
7.3
17.5
13.1
12.5
10.6
NN-LM
0.33
7.2
16.4
13.1
10.5
SWB 2000H
Textogram
0.72
19.2
12.2
26.0
23.6
Shallow Fusion
0.85
4.3
15.0
12.8
11.9
12.0
10.8
TOLSTOI
0.33
4.0
6.6
11.2
10.4
11.1
10.0
Unadapted
0.33
12.5
12.6
34.4
23.5
20.3
16.5
NN-LM
0.33
12.4
12.6
33.3
23.6
20.3
16.5
SWB 300H
Textogram
0.72
10.8
22.3
24.5
33.6
19.1
19.2
Shallow Fusion
0.85
8.1
13.7
29.0
23.8
19.8
17.2
TOLSTOI
0.33
10.4
12.0
23.8
18.8
19.1
16.1
Table 1: Comparison on decode time (RTF) and WERs target-only and target-source mixed data of different adaptation methods on two different ASR models trained on SWB 2000H (top) and SWB 300H (bottom) adapted to three different domains. WERs on SWB test are 8.8 and 12.7, respectively, using SWB-2000 and SWB-300. When deployed on an equal mixture of source and target data, TOLSTOI provides the highest reduction in WER.
Reference
need to reset would like to reset my password
Shallow Fusion
need to reset would like to reset my password
TOLSTOI
need to reset would like to resct my password
Reference
hi lost my debit card can you send me new one my name is robert davis by the way hi lost my debit card can you send me new one my name robert davis by the way
Shallow Fusion
TOLSTO1
hi lost my debit card can you send mc a new one my name is robert davis by the way
Table 2: Anecdotal examples comparing TOLSTOI with shallow fusion. Deletion errors are under- lined and highlighted in red.
task (Ogayo et al. 2022). In contrast, TOLSTOI relies only on access to the ASR pretraining data to train the imputation model and can potentially scale well to low-resource languages.
4.2 OVERALL COMPARISONS
Table presents the decode time (RTF) and target-only WER and source-target mixed WER of different adaptation methods on two ASR models adapted to three different target domains. If we focus only on target WERs, shallow fusion provides good reductions in WER. However, this comes at the cost of 2.5 times blow-up of decode time (RTF) which could be unacceptable in many applications. Also, the external LM fine-tuned on the target text leads to significant catastrophic forgetting as observed by the significantly increased WER the mixed corpus. TOLSTOI provides the best WER when measured on the mixed test set while being competitive with shallow fusion on the target-only set. The decode time of TOLSTOI is exactly the same as the basic RNN-T since we do not modify the RNN-T architecture. NN-LM does not provide much gains beyond the unadapted model. The Textogram more successful adaptation but also subject to significant deterioration on the mixed corpus. Also, the Textogram modified the network architecture of the RNN-T resulting significantly larger decode time compared to the basic RNN-T.
Table 2 presents two illustrative examples from ATIS comparing predictions using shallow fusion and TOLSTOI. Shallow fusion does not handle disfluencies well if the target text used to train the external LM is devoid of it. For example, the false start "I need to reset" and the filler phrase "by the way" are omitted. More anecdotal examples are shown in Table in Appendix
4.3 ABLATION STUDY
Table presents ablations of various design choices governing TOLSTOI. We investigate three di- mensions of the imputation model: 1) choice of loss function, 2) choice of alignment to sample at test-time, and 3) choice model architecture. On the choice loss function, we replaced the loss in Eqn 2 with L2 loss and contrastive loss. We used the contrastive loss from Chen et al. (0200) for the imputed features in each batch. We also tried a distillation loss where a generated h; is combined with the corresponding 9n via M, to generate probability distribution over the vocabulary and KL-Divergence loss is imposed between this predicted distribution and the proba- bility distribution from the real h,. These losses were all worse than L.I. On choice of the alignment
8

---

Published as a conference paper at ICLR 2023
Method
ATIS
HarperValleyBank
Target
Mixed
Target
Mixed
Unadapted
5,8
7.3
17.5
13.1
TOLSTO1
4.0
6.6
11.2
10.4
Imputation Model Loss Function
Imputation Model with L2 Loss
4.2
6.5
11.3
10.4
Imputation Model with Contrastive Loss
7.8
8.6
17.1
13.1
Imputation Model with Distillation Loss
5.3
7.1
15.4
12.4
Alignment Generation Models Imputation Model with blank Imputation Model with 4 blanks
7.2
8.1
23.4
16.4
4.2
6.6
11.3
10.4
Imputation Model with dynamic length model
4.4
6.7
11.3
10.4
Imputation Model Architecture (gw)
9.1
9.0
9.5
16.0
(h--1, ou- 9u+1)
4.4
6.7
11.8
10.7
(h:-1, 11 Mu++1)
5.6
7.4
15.6
12.5
Transformer Imputation Model
5.7
7.2
17.3
13.0
-oo Switchboard Data
4.4
7.2
11.7
11.6
Table Ablation of various design choices of TOLSTOI. More complicated loss functions, or align- ment models, or imputation models performed worse than the simple design choices of TOLSTOI.
generation model, we tried the FixedGram model with I blank or 4 blanks before each output token instead of the default of We also tested feed-forward regression model for length that takes Iu as its input and outputs the number of speech frames learnt from the alignment data (c.f., Sec- tion 3.T). All these alignment generation models fared worse than using FixedGram model with 3 blanks in TOLSTOI. We next tried different tweaks the architecture of the imputation model. We tried changing the input context to the imputation model by adding Ju+1, adding output char- acters Yu-1, Bu Yu+1 omitting h We also tried Transformer-bases (Vaswani et al., 2017) incoder-decode imputation model to transform the complete g. sequence to a full h; sequence. All these variants performed worse than TOL.TOI. During RNN-T finetuning, we found it useful mix an equal number of utterances from the pretraining corpus with the imputed data (Zhu et al. Omitting this step resulted in small WER degradations shown in the last in Table
2022
Figure shows the WERs on ATIS and HVB as function of decreasing amount of target text. As expected, there is an increase in WERs with reducing the amount of target text. However, the reduction in WERs with using only 50% of the data versus using the complete dataset is not substantial. This suggests that TOLSTOr could work well even in target domains with very limited amounts of text-only data.
15
13
10
11.8
11.2
WER
5.4
4.8
ATIS
HVB
Figure 4: WER vs. Adaptation Text(%)
5 CONCLUSION
In this paper we presented TOLSTOI, low-overhead method of adapting RNN-T models using text- only data in a target domain. Our key insight it to impute speech features from the last layer of the transcription network, which allows accurate fine-tuning of subset the RNN-T parameters. We proposed a very simple design of the imputation model by leveraging existing text-speech repre- sentations and alignments from the trained RNN-T model. Unlike existing methods, TOLSTOI does not modify the base RNN-T architecture and can adapt existing pre-trained ASR models without increasing inference time during decoding. Via experiments on three target domains and two ASR models, show that TOLSTOI provides the best accuracy on mixed source-target test set since is least subject to catastrophic forgetting while reducing target WER by more than 35%. With a detailed ablation over more complicated models, justify the effectiveness of our simple impu- tation model. As part of future work, we would like to experiment our approach on low-resource languages and explore techniques for online adaptation of the ASR model.

---

Published as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
All our experiments are performed on publicly available datasets such as Switchboard, ATIS, HarperValleyBank, and Librispeech. We also use published train/test splits specified for each of these datasets, thus enabling reproducibility. We have provided sufficient implementation details of our baseline models, our imputation model and the RNN-T fine-tuning process to help reproduce main results.
ACKNOWLEDGEMENT
We would like to thank George Saon, Samuel Thomas and Jeff Kuo for insightful discussions. The authors from IIT Bombay gratefully acknowledge support from IBM Research, specifically the IBM AI Horizon Networks-IIT Bombay initiative.
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-moo modall encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:21 10.00205,, 2021.
Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. Slam: A unified encoder for speech and language modeling via speech-text joint pre-training, arXiv preprint arXiv:2110. 10329, 2021.
Murali Karthick Baskar, Lukáš Burget, Shinji Watanabe, Ramon Fernandez Astudillo, et Eat: Enhanced asr-tts for self-supervised speech recognition. In ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6753-6757, IEEE, 2021.
Jerome R Bellegarda. Statistical language model adaptation: review and perspectives. communication, 42(1))93-108, 2004.
Speech
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xie Chen, Zhong Meng, Sarangarajan Parthasarathy, and Jinyu Li. Factorized neural transducer for efficient language model adaptation. In ICASSP 2022 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8132-8136. IEEF, 2022a.
2022-2022
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ranabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen. Maestro: Matched speech text representations through modality matching. arXiv preprint Xiw::2204.0044000 2022b.
Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, and Gary Wang. Tis4pretrain 2.0: Advancing the use text and speech in asr pretraining with consis- tency and contrastive losses. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7677-7681. IEEE, 2022c.
Christopher Cieri, David Miller, and Kevin Walker. The fisher corpus: A resource for the next generations of speech-to-text. In LREC, volume 4, pp. 69-71, 2004.
Yan Deng. Rui Zhao, Zhong Meng. Xie Chen, Bing Liu, Jinyu Li, Yifan Gong, and Lei He. Im- proving RNN-T for Domain Scaling Using Semi-Supervised Training with Neural TTS. In Proc. Interspeech 2021, pp. 751-755, 2021. doi: 10.21457/nterspeechh2201. 2021111-02....211111..
Raul Fernandez, David Haws, Guy Lorberbom, Slava Shechtman, and Alexander Sorin. Transplan- tation of conversational speaking style with interjections in pruuecce-to-squence speech synthe- sis. arXiv preprint arXiv:2207. 12262, 2022.
10

---

Published as a conference paper at ICLR 2023
Siva Reddy Gangireddy, Pawel Swietojanski, Peter Bell, and Steve Renals. Unsupervised adaptation of recurrent neural network language models. In Interspeech, pp. 2333 2237, 2016.
John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, volume 1, pp. 517-520, IEEE Computer Society, 1992.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211,22 2013.
Alex Graves. Sequence transduction with recurrent neural networks.
arXiv preprint
arXiv:1211.3711, 2012.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645 6645.6649. Icce, 2013.
Anmol Gulati, James Qin, Chung- -heng Chiu, Niki Parmar, Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: for speech recognition. arXiv preprint 2rX0::005.008000, 2020.
augmented transformer
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 1412.5567. 2014.
Tomoki Hayashi, Shinji Watanabe, Yu Zhang, Tomoki Toda, Takaaki Hori, Ramon Astudillo, and Kazuya Takeda. Back-translation-ssyyl data augmentation for end-to-end asr. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 426-433. IEEE, 2018.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu. Ruoming Pang, et al. Streaming end-to-end speech recog- nition for mobile devices. In ICASSP 2019 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6381 6381-6385. IEEE, 2019.
Charles T Hemphill, John J Godfrey, and George R Doddington. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990, 1990.
Takaaki Hori, Daniel Willett, and Yasuhiro Minami. Language model adaptation using wfst-based speaking-style translation. In 2003 IEEE International Conference Acoustics, Speech, and Signal Processing, 2003. Proceedings '03), volume 1, pp. I-I. IEEE, 2003.
Takaaki Hori, Ramon Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, and Jonathan Le Roux. Cycle-consistency training for end-to-end speech recognition. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6271- 6275. IEEE, 2019.
Raviraj Joshi and Anupam Singh. A simple baseline for domain adaptation in end to end ast systems using synthetic data. In Proceedings of The Fifth Workshop on e- e-Commerce and NLP (ECNLP 5), pp. 244 249, 2022.
Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N. Sainath, ZhiJeng Chen, and Rohit Prab- havalkar. An analysis of incorporating external language model into sequence-to-sequene model. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5828, 2018. doi: 10.1109/CASSP..0...... 8462682.
Zvi Kons, Slava Shechtman, Alex Sorin, Carmel Rabinovitz, and Ron Hoory. High quality, lightweight and adaptable tts using lpenet. arXiv preprint arXiv: 1rX.vv9955.00590 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
11

---

Published as a conference paper at ICLR 2023
Alvin Martin and Mark Przybocki. The nist 1999 speaker recognition evaluation overview. Digital signal processing, 10(1-3):1-18, 2000.
John S Mason and X Zhang. Velocity and acceleration features in speaker recognition. In Acous- tics, Speech, and Signal Processing, IEEE International Conference on, pp. 3673-3674, IEEE Computer Society, 1991.
Erik McDermott, Hasim Sak, and Ehsan Variani. A density ratio approach to language model fusion in end-to-end automatic speech recognition. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.
IEEE, 2019.
Zhong Meng. Sarangarajan Parthasarathy, Eric Sun, Yashesh Gaur, Naoyuki Kanda, Liang Lu, Xie Chen, Rui Zhao, Jinyu Li, and Yifan Gong. Internal language model estimation for domain- adaptive end-to-end speech recognition. In 2021 IEEE Spoken Language Technology Workshop (SI.7), pp. 243-250. IEEE, 2021a.
Zhong Meng, Yu Wu, Naoyuki Kanda, Liang Lu, Xie Chen, Guoli Ye, Eric Sun, Jinyu Li, and Yifan Gong. Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition. In Proc. Interspeech 2021, pp. 2021-2075.
2021b. doi: 10.21437/Interseecch
Mehryar Mohri, Fernando Pereira, and Michael Riley. Weighted finite-state transducers in speech recognition. Computer Speech & Language, 16(1):69-888 2002.
Graham Neubig, Shinsuke Mori, and Tatsuya Kawahara. A wfst-based log-linear framework for speaking-style transformation. In Tenth Annual Conference of the International Speech Commu- nication Association. Citeseer, 2009.
Perez Ogayo, Graham Neubig, and Alan W Black. Building african voices. arXiv preprint arXiv:2207.00688 006888, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 5206-5210 IEEE, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph. Ekin D Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 1904498799999,.... 2019.
Junho Park, Xunying Liu, Mark JF Gales, and Phil Woodland. Improved neural network based language modelling and adaptation. In Eleventh Annual Conference of the International Speech Communication Association, 2010.
Joseph W Picone. Signal modeling techniques in speech recognition. Proceedings of the IEEE, 81 19):1215-1247, 1993.
Janne Pylkkonen, Antti Ukkonen, Juho Kilpikoski. Samu Tamminen, and Hannes Heikinheimo. Fast text-only domain adaptation of rnn-transducer prediction network. 22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021, pp. 1882-1888 ISCA, 2021. doi: 221443//nnerspeechh221111111 2021-1111.
Tara N Sainath, Rohit Prabhavalkar, Ankur Bapna, Yu Zhang, Zhouyuan Huo, Zhehuai Chen, Bo Li, Weiran Wang, and Trevor Strohman. Joist: A joint speech and text streaming model for asr. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 52-59. IEEE, 2023.
George Saon, Zoltán Tüske, and Kartik Audhkhasi. Alignment-lenge length synchronous decoding for transducer. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7804-7808. IEEF, 2020.
George Saon, Zoltán Tüske, Daniel Bolanos, and Brian Kingsbury. Advancing transducer tech- nology for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp- 5654-5658. IEFF, 2021.
12

---

Published as a conference paper at ICLR 2023
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv: 150.:1099.0999 2015.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitl,, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779-4788 IEEE, 2018.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pp. 369-386. SPIE, 2019.
Yuki Takashima, Shota Horiguchi, Shinji Watanabe, Paola Gareía, and Yohei Kawaguchi. Up- dating only encoders prevents catastrophic forgetting of end-to-end asr models. arXiv preprint arXiv:2207.002/6 2022.
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, al. Unified speech-text pre-training for speech translation and recognition. arXiv preprint arXiv:2204.0544 2022.
Samuel Thomas, Brian Kingsbury, George Saon, and Hong-Kwang J Kuo. Integrating text inputs for training and adapting mnn transducer asr models. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8127-8131. IEEE, 2022.
Takuma Udagawa, Masayuki Suzuki, Gakuto Kurata, Nobuyasu Itoh, and George Saon. Effect and analysis of large-scale language model rescoring on competitive ast systems. arXiv preprint arXiv:2204. 00212, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Mike Wu, Jonathan Nafziger, Anthony Scodary, and Andrew Maas. Harpervalleybank: domain- specific spoken dialog corpus. arXiv preprint arXiv:2010. /3999, 2020.
Hainan Xu, Tongfei Chen, Dongji Gao, Yiming Wang, Ke Li, Nagendra Goel, Yishay Carmiel, Daniel Povey, and Sanjeev Khudanpur. A pruned mnlm lattice-rescoring algorithm for automatic speech recognition. In 2018 IEEE international conference on acoustics, speech and signal pro- cessing (ICASSP), pp. 5929-5933. IEEE, 2018.
Xianrui Zheng, Yulan Liu, Deniz Gunceler, and Daniel Willett. Using synthetic audio to improve the recognition of vocabulary words in end-to-end ast systems. In ICASSP 2021-202/ IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5674-5678. IEEE, 2021.
Han Zhu, Gaofeng Cheng, Jindong Wang, Wenxin Hou, Pengyuan Zhang, and Yonghong Yan. Boosting cross-domain speech recognition with self-supervision. arXiv preprint 27X.:22006.09783, 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EVALUATION TOLSTOI USING CONFORMER MODEL
In this section, we compare the effectiveness of TOLSTOI using Conformer (Gulati al. 2020) encoder the RNN-T. While FixedGram-based TOLSTOI works well on the bidirectional-LSS TTM en- coder shown in Section 4.2] the purpose of this experiment is to evaluate its effectiveness for the Conformer encoder which would distribute acoustic features in the neighborhood. The Conformer- Transducer model uses an encoder network of 10 Conformer blocks (5 11--dimensional rred-forward module, 31 -e-kenll convolution block, and 64-dimensional attention heads). We train the model on the SWB 300H dataset for this experiment. All the other model architecture choices and hyperpa- rameters remain the same.
Method
ATIS
HVB
Target
Mixed
Target
Mixed
Unadapted
11.5
11.9
39.7
26.0
NN-LM
11.3
11.9
38.1
25.6
Shallow Fusion
9.1
14.1
32.2
23.6
TOLSTOI
10.4
12.4
28.8
21.9
Table 4: Comparison on WERs on target-only and target-source mixed data of different adaptation methods on conformer-based RNN-T ASR model trained on SWB 300H adapted ATIS and HVB domains.
We empirically observe that the FixedGram-basee TOLSTOI works well with the Conformer-baid based RNN-T model. The reduction in the WER for the target domains is significant and catastrophic forgetting is also minimal compared to shallow fusion.
A.2 EVALUATION TOLSTOI ON RNN-T MODEL WITH BPE UNITS
In this section, compare the effectiveness of TOLSTOI based BPE units (Sennrich et al. 2015). While the FixedGram based TOLSTOI works well on the Aidirectional-LSThy encoder with character units, the purpose of this experiment to evaluate the effectiveness the model with subword units. To this end, we train an SWB 300H model using the subword units using the same model configuration as for the character model, except that output vocabulary now comprises 1000 BPE units as opposed to the 45 characters. The subword encoder learned on the text obtained from the training set. We show results on TOLSTOI using the FixedGram approach.
RNN-T model trained subword-
Method
ATIS
HVB
Target
Mixed
Target
Mixed
Unadapted
12.7
12.5
41.9
27.1
NN-LM
12.5
12.5
39,8
26.2
Shallow Fusion
11.1
22.6
34.2
35,0
TOLSTOI
10.0
11.5
33.1
22.9
Table Comparison on WERs on target-only and target-source mixed data of different adaptation methods on subword-based RNN-T model trained on SWB 300H adapted to ATIS and HVB do- mains.
We empirically observe that the FixedGram-based TOLSTOI works well even with the subword units. The reduction in the WER for the target domains significant and catastrophic forgetting is also minimal compared to the Shallow Fusion. Additionally, observe that the catastrophic for- getting for shallow fusion increases for the BPE-based models as opposed to the Character-basee model.
14

---

Published as a conference paper at ICLR 2023
A.3 EVALUATION OF TOLSTOI ON RNN-T MODEL TRAINED EXTREMELY LARGE DATASET
In this section, we compare the effectiveness of the TOLSTOI on RNN-T model trained extremely large dataset consisting of proprietary data totaling close to 56K hours of labeled speech.
Method
ATIS
HVB
Target
Mixed
Target
Mixed
Unadapted
5.2
6.8
12.2
10.3
NN-LM
5.1
6.8
12.0
10.3
Shallow Fusion
2.8
7.1
8.6
12.2
TOLSTOI
2.8
5.9
7.9
8.7
Table 6: Comparison on WERs on target-only and target-source mixed data of different adaptation methods RNN-T model trained on proprietary data adapted to ATIS and HVB domains.
We empirically observe that the FixedGram-based TOLSTOT works well even when used with very large amounts of training data. Our results at such scale are consistent with our results on smaller SWB 300H and SWB 2000H benchmarks.
A.4 ANECDOTAL EXAMPLES
Reference
we going to fly listen are going to fly over saint louis we are going to fly listen we are going to fly over saint louis we are going to fly listen we are going to fly over saint louis we are going to fly listen we are going to fly over saint louis
Unadapted
Shallow Fusion
TOLSTOI
Reference
he went moved to texas he went moved to texas he went and moved to texas he went moved to texas
Unadapted
Shallow Fusion
TOLSTOI
Reference
would like to transfer money between my accounts would like to transfer money between my accountants would like to transfer money between my account would like to transfer money between my accounts
Unadapted
Shallow Fusion
TOLSTOI
Reference
yeah we played softball last night we were clobbered
Unadapted
yeah we played softball last night we were clobbered yeah we played softball last night we were clovered
Shallow Fusion
TOLSTOI
yeah played softball last night were clobbered
Table Anecdotal examples on the mixed test set comparing TOLSTOI with shallow fusion. Dele- tion errors are underlined and highlighted in red.
15

---

Published as a conference paper at ICLR 2023
Reference
thanks you too thanks you too thanks you too thank you TODAY they plan to get married in topeka since most of their friends are there they plan to get married in to figure since most of their friends there
Unadapted
Shallow Fusion
TOLSTOI
Reference
Unadapted
Shallow Fusion
they plan to get married into since most of their friends are there they plan to gel married in to peak us since most of their friends are there show me weekday flights from milwaukee to orlando one way show me weak day flights from milwaukec to orlando one way show me weekday flights from milwaukee to orlando one way show me weakday flights from milwaukee orlando one way
TOLSTOI
Reference
Unadapted
Shallow Fusion
TOLSTOI
Table 8: Anecdotal examples showing the mistakes with TOLSTOI and shallow fusion. Errors are underlined and highlighted in red. When the language model context is not clear (e.g, for word topeka), TOLSTOI produces acoustically similar predictions as opposed to the Shallow Fusion and Unadapted models.
16


Reviews:
[{"review_id": "rv4SaMY1_E", "reviewer": "Reviewer_USzV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies RNN-T ASR model adaptation using unpaired text in the target domain. The authors proposed to use a module called “Imputation Model” which predicts speech representation of the speech encoder ($M_s$) given text, such that text encoder $M_L$ and the joint network $M_j$ in RNN-T can be fine-tuned using text and pseudo speech embedding.\n\nThe use of the imputation model is compatible with any existing RNN-T model, does not increase the total number parameters during inference, suffer milder catastrophic forgetting, and yields better performance compared to the included baselines (shallow fusion, $M_L$ as LM, textogram).\n", "summary_of_the_review": "This paper presents a practical solution to RNN-T ASR model adaptation using unpaired text. Experiments and presentations are good. I would encourage the author to improve the comparison with existing works.", "strength_and_weaknesses": "Strengths\n* The design of the imputation model is well motivated (keeping inference time short, drop-in replacement for existing models, good performance on both target and general domain)\n* It is very easy to follow the paper and all details are included.\n* Performance is strong compared to the baselines. Ablation studies are informative, showing a) the impact of loss function, b) simple alignment generation is sufficient, c) reusing $M_L$ embedding simplifies the imputation model.\n\nWeakness\n* The idea of generating internal representations of an end-to-end model to fine-tune a subset of parameters has been studied before [1,2]. I can see that the design of the “imputation models” (or so-called text-to-embedding (TTE) models in prior studies) are different. However, the authors should discuss these studies and present experiments to compare with prior models.\n\n[1] Hayashi, Tomoki, et al. \"Back-translation-style data augmentation for end-to-end ASR.\" 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018.\n\n[2] Hori, Takaaki, et al. \"Cycle-consistency training for end-to-end speech recognition.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oo6qfnfkJY", "reviewer": "Reviewer_hxLw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to do domain adaptation without textual data from target domain for Transducer based ASR. Text-only adaptation for ASR enables a compact model for target domain without relying on external LMs. The idea proposed in this paper is also novel and interesting. Experiments on a small and medium size ASR data shows the improvement over multiple related baselines while maintains the performance on source domain.", "summary_of_the_review": "Given the strength and weakness, the reviewer likes the novelty and still intend to have this paper accepted.", "strength_and_weaknesses": "Strength\n1. Novelty; The idea proposed in this paper indeed meet the requirements mentioned in this paper for text-only domain adaptation, which are high accuracy, no retraining, no impact on inference speed and no deterioration on source domain. As far as I know, previous studies fail to meet all four requirements. \n2. Intensive experiments and ablation studies show the effectiveness of the design choices.\n\nWeakness\n1. The dataset used in this paper is not large enough. There exists gigaspeech with 10k hours training data. With large dataset, it could relieve the overfitting on source domain data and provide better generalization ability, thus making the results more convincing.\n2. Transducer based models are specific to ASR domain, thus it's hard to apply the method in this paper to other fields, such as speech translation/machine translation domain, making the contribution of this paper less accessible. If similar idea could be applied to LAS based ASR, it could be better. However, the current model design can only be used to transducer based ASR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SgT7asmHg4h", "reviewer": "Reviewer_cvPh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a text-only adaptation method for RNN-Ts, allowing to make these models better for a new domain than the one represented in the training data. While existing methods require a change in the RNN-T architecture, a full retraining, or introduce some latency, the proposed approach allows to only fine-tune the language and joint network, similarly to methods relying on TTS. Here, instead of generating audio for domain-specific texts using TTS, the output representations of the speech model are generated from the text directly. \n\nThe model that generates these intermediate representations is trained using the forced alignments of the labeled training dataset. A small recurrent neural network is trained to predict the next hidden speech vector from the current one and current language hidden vector. To adapt to a new domain, alignments are first generated from the text, and the small network is used to predict a sequence of hidden speech vectors. These vectors are in turn used to fine-tune the language and joint networks.\n\nThrough experiments with two base training sets and 3 adaptation domains, the paper shows that the proposed method with a 56M-parameter model outperforms the existing adaptation approaches without a significant increase of latency or degradation of performance on the original domain.", "summary_of_the_review": "The paper is good and the idea worth sharing with the community. The ability to adapt an end-to-end ASR model is important and represents a challenging task, and the presented method allows to do so without changing the RNN-T architecture and without requiring a complete fine-tuning of the model, which not only saves energy but also allows to use that method without requiring a lot of computational power. \n\nFor these reasons, I think the paper is interesting for ICLR.\n\nSome clarifications and additional experiments could nevertheless add more value to this paper:\n\n   - is teacher forcing used for training imputation model?\n   - to measure catastrophic forgetting, WER is computed on a source-target mixture dataset: why not measure on source and target separately?\n   - is the alignment sampling method (and the overall adaptation technique) also good when the model outputs subwords instead of characters?\n   - the \"Real Time Factor(RTF) which measures the average time (in seconds) to decode 50 audio frames\": 50 audio frames is indeed, in the described setup, 1 second of audio: it would be clearer and more consistent with the literature to define RTF as the total processing time divided by the actual audio duration maybe?\n   - more analysis of the representations learned by the imputation model, and maybe a focus on rare words would be very interesting for an ICLR paper", "strength_and_weaknesses": "Strength: the proposed method is very interesting, does not require to retrain the whole network, gives good results, and seems quite simple to implement and to test. Among the different design choices, the simplest ones (L1 loss, fixed gram for alignment generation, imputation network architecture) seem to work best, which is also a nice result. Finally, a pretty small imputation network looks sufficient.\n\nWeakness: The evaluated RNN-Ts predict characters, while is looks more common to use subwords. It would be nice to see if the results tranfer also to subword modeling. Although the focus here is not really rare words or named entities, I would also be curious to see the results for this challenging task, compared to biasing techniques.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2gvL9pIHyA", "reviewer": "Reviewer_BcMJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel text-only adaptation method for RNNT-based ASR systems. Unlike previous approaches (shallow fusion, TTS, fine-tuning), the proposed method called TOLSTOI is lightweight with a cheap training cost. The key idea of TOLSTOI is to train an “imputation model\" that mimics the speech module outputs from the language module outputs. The imputation model is trained first, and then the language and joint models are fine-tuned. Experimental results show that the proposed method achieves the lowest WER when the source and target domains are different.", "summary_of_the_review": "Overall, the paper is novel and tackles an important problem. The strengths are strong, and the weaknesses are minor. Although the scalability to other models needs further validation, I recommend this paper be accepted.", "strength_and_weaknesses": "Strengths:\n\n- The paper tackles important and practical problems with a simple approach. The motivation of the paper is meaningful, and the related works are studied well and clearly demonstrated in a general framework (Figure 1).\n- The success of FixedGram is interesting that a fixed number of blanks are sufficient for pseudo-generating the speech features.\n- Measuring not only WER but also RTF clearly shows the advantage of the method. Also, it is good to consider catastrophic forgetting.\n\nWeaknesses:\n- The model architectures are slightly old-fashioned. How about trying Transformer-based acoustic models instead of LSTM-based ones? Also, I guess that in the case of ContextNet or Conformer, convolution operations would distribute the acoustic features and FixedGram may not work well.\n- As I understand, the imputation model is removed after the adaptation during inference. It would be better if authors investigate the difference between learned IMP output and actual output from speech module (for both source and target domains).\n- Just a question; training takes multiple steps; is “end-to-end” an appropriate explanation? It seems that there are many manual steps, for example, extracting alignments and training different modules with different hyperparameters.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster