HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASR
Authors: Hainan Xu
Travis M. Bartley
Vladimir Bataev
Boris Ginsburg
Venue: ICLR 2025 Poster (2025)
Keywords: speech recognition
sequence modeling
non-autoregressive models
RNN-T
Transducers
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce a novel speech recognition model that supports different levels of autoregression, achieving better speed-accuracy tradeoff than alternative methods.

Abstract: We present Hybrid-Autoregressive INference TrANsducers (HAINAN), a novel architecture for speech recognition that extends the Token-and-Duration Transducer (TDT) model. Trained with randomly masked predictor network outputs, HAINAN supports both autoregressive inference with all network components and non-autoregressive inference without the predictor. Additionally, we propose a novel semi-autoregressive inference method that first generates an initial hypothesis using non-autoregressive inference, followed by refinement steps where each token prediction is regenerated using parallelized autoregression on the initial hypothesis. Experiments on multiple datasets across different languages demonstrate that HAINAN achieves efficiency parity with CTC in non-autoregressive mode and with TDT in autoregressive mode. In terms of accuracy, autoregressive HAINAN achieves parity with TDT and RNN-T, while non-autoregressive HAINAN significantly outperforms CTC. Semi-autoregressive inference further enhances the model's accuracy with minimal computational overhead, and even outperforms TDT results in some cases. These results highlight HAINAN's flexibility in balancing accuracy and speed, positioning it as a strong candidate for real-world speech recognition applications.

Published as a conference paper at ICLR 2025
HAINAN: FAST AND ACCURATE TRANSDUCER FOR SHBRID-AUTOREGRESST ASR
Hainan Xu, Travis M. Bartley, Vladimir Bataev, Boris Ginsburg NVIDIA Corp., USA
(hainanx, tba thart wbataev/enviddia.cmm
ABSTRACT
We present Hlybrid-Autoregressive INference TrANsducers (HAINAN), a novel architecture for speech recognition that extends the Token-and- Trans- ducer (TDT) model. Trained with randomly masked predictor network outputs, HAINAN supports both autoregressive inference with network components and non-auttoregressive inference without the predictor. Additionally, we propose a novel semi-autoregressive inference method that first generates an initial hy- pothesis using non-autoreerresive inference, followed by refinement steps where each token prediction is regenerated using parallelized autoregression on the initial hypothesis. Experiments on multiple datasets across different languages demon- strate that HAINAN achieves efficiency parity with CTC in non-autoregressive mode and with TDT in autoregressive mode. In terms of accuracy, autoregres- sive HAINAN achieves parity with TDT and RNN-T, while non-autoregressive HAINAN significantly outperforms CTC. Semi-autoregrrssivee inference further enhances the model's accuracy with minimal computational overhead, and even outperforms TDT results in some cases. These results highlight HAINAN's flex- ibility in balancing accuracy and speed, positioning it strong candidate for real-world speech recognition applications.
INTRODUCTION
End-to-end neural automatic speech recognition (ASR) has seen significant advancements in re- cent years, namely due to the development of three architecture paradigms: Connectionist Tem- poral Classification (CTC) (Graves et al. 2006), Recurrent Neural Network Transducers (RNN-T) (Graves) 2012). and Attention-based Encoder and Decoder Models (Chorowski et al.] 2015 Chan jet al. 2016). These models have gained widespread adoption, supported by open-source projects such ESPNet (Watanabe et al., 2018), SpeechBrain (Ravanelli et al., 2021), and NeMo (Kuchaiev ctt al., 2019). etc. Of those models, CTC and RNN-T share frame- frame-synchrome synchronous design, enabling streaming processing of speech input. However, they differ in their inference approaches: CTC mod- els support hon-autoregressive (NAR) inference based on a munditionllindependeer
assumption, while RNN-T models require autoregressive (AR) inference due their dependence on the history context of partial hypotheses. This fundamental difference creates a trade-off' between accuracy and efficiency, with CTC models offering greater speed but typically lower accuracy than Transducers.
Our work bridges this gap with Syybrid-Autorggessste INference TrANsducer (HAINAN)) a novel implementation of RNN-T capable inference. Our contributions are as follows:
antorrrrrrrreevvee autoregressive and, semi-autoregressivve
We propose method to train Token-and- -urration Transducer (TDT) model by stochastically masking out the predictor. The approach is easy to implement and only requires a one-line change in the training code of token-and and-duration model implementation.
We present a novel semi cemi-autoregressitt (SAR) inference method that first uses NAR to generate initial hypotheses, and then uses autoregression to refine the hypotheses in parallel iteratively.
Despite similar names, our work is unrelated Hybrid Autoregressive Transducer Variani et al. 2020

---

Published as a conference paper at ICLR 2025
We present means model the output of the NAR HAINAN model containing the token and duration predictions as a directed acyelic graph (DAG), and propose a simple Viterbi- based de- coding method for inference, with further accuracy gains.
Our results demonstrate the following:
AR HAINAN achieves on-par WER accuracy with RNN-T and TDT models, with inference speeds similar to TDT-based decoding, and significantly faster than RNN-T inference.
NAR HAINAN outperforms CTC accuracy while maintaining similar efficiency. Our investiga- tion shows that NAR HAINAN can better capture potential ambiguities in the speech by learning more nuanced representations, a capability that frame-by-frand NAR like CTC inherently lacks,
SAR consistently brings accuracy gains over NAR, approaching the performance of AR inference with minimal computational overhead.
HAINAN's high performance is due to learning of more diverse encoder outputs, a property that allows our Viterbi decoder to achieve the same accuracy of AR inference despite the use of non aott-ettrrrrrsiveee outputs.
We will open-source our implementation and release trained model checkpoints for public use.
2 RELATED WORK
Recent research in speech recognition has focused enhancing WER accuracy while mitigating reductions in inference speed. This has been most notable for self-attention--asse architectures, with improvements in latency for Transformer models (Ych et al. 2019) leading creation of the Conformer (Gulati et al.) 2020) and it variants (c.g. Zipformer (Yao et al. 2023). ) FastConformer (Rekesh et al. [2023). ). While all have notable architectural improvements, they can be generalized as attempts to improve model capacity and/or reducing the latency caused by attention inference over large audio inputs.
For Transducer models specifically, various optimizations have been explored. He et al. (2019) proposed a caching mechanism for the RNN-T predictor reduce redundancy in inference com- putations. Jain et al. (2019) introduced a pruning method for RNN-T beam search to reduce the search space, resulting in faster decoding. Ghodsi al. (2020) investigated predictors with state- less networks for Transducer models. While this approach reduced the computational complexity of the original LSTT predictor, it came the cost of slight accuracy degradation. Kang et al. (2023) proposed monotonic constraints on RNN-T training and inference, reducing the number of steps RNN-T inference stays on the same frame, and reported faster inference with those models. |Bataev et al. (2024) proposed novel label-looping decoding algorithm for Transducers, using parallel GPU calls for the majority of the decoding operation and achieved significant inference speedup. Galvez et al. (2024) optimized the decoding algorithm from hardware perspective and minimized the GPU idle time during decoding. further improving based inference. Multi-blank Transducers (Xu let al. 2022) and loken-anddDDrattinn Transducers (TDT) (Xu al. 2023b) identified large num- bers of blank predictions during RNN-T inference, proposing duration-predictionn mechanisms to skip frames during processing. This improvement greatly reduced the number of decoding steps required for inference. While these methods improved Transducer inference speeds, the resulting models remained autoregressive. As such, Transducer inference speeds still significantly lagged behind non -oto-urrreeersvveee frameworks (e.g. CTC).
Our proposed autreervtreee method falls under the category of "fast/slow" methods, where the model combines faster but less accurate module, and slower but more accurate module to achieve good accuracylefficiency tradeoffs. [Inaguma et al. (202T) use fast CTC model to generate multiple hypotheses, to be re- re-ranked by slower AR model to improve speech translation perfor- mance. Mahadeokar et al. (2022) propose a cascade model architecture, where the fast model uses a relatively shallow encoder for quicker results, while the slow model uses additional encoder layers along with a separate joint network to improve results. Arooa et al. (0024) proposed a model that di- vides speech input into blocks, and adopts AR processing among blocks but NAR processing within a block to balance speed with accuracy. Deliberation methods Hu et al. (2020) and its variants WWag al. (2022) cascaded encoders and separate decoder, and use the shallow encoder generate
2

---

Published as a conference paper at ICLR 2025
initial hypotheses, which to be fed into second-pass Sralien-Atend-Sppe (LAS) model with ad- ditional encoder and decoders to improve accuracy. Compared with those "fast-slow" works, our work is unique in that our "slow" model only adds marginal parameters top of the "fast" model, and the "refinement" process requires minimal computation overheads.
BACKGROUND
In this Section, we provide the technical background for models that influenced HAINAN architec- ture: CTC, RNN-Transducer (RNN-T), and Token-and- Transducer (TDT).
3.1 CONNECTIONIST TEMPORAL CLASSIFICATION (CTC)
CTC (Graves al. 2006) is sequence-to-sen touuenne prediction model designed to address the chal- lenge of aligning input and output sequences of different lengths. Let x = (α), 22, IT) represent the input sequence, y = (y), 1/2,..., -tr) the output sequence, and y the vocabulary, where E V includes all tokens in and a special "blank" symbol. A CTC model predicts P ((v))r) ) for each time crucial for handling the discrepancy between imput and output se-
step The "blank" symbol quence lengths, allowing for many- mon alignments and representing frames that do not contribute additional information to the output. CTC models employ a set of rules to generate CTCCaaugmntedd sequences that are equivalent to the original sequence: (1) When a token is repeated in the original sequence, at least one "blank" token must be inserted between the repeated tokens, and (2) Any token can be repeated an arbitrary number of times.
The training objective CTC models to maximize the probability of y given the input x. This probability is defined the of probabilities for all possible augmented sequences of the ref- erence sentence that match the input sequence length. CTC adopts a conditional independence assumption when computing the probability of augmented sequence given the input. This sumption posits that each token in the augmented sequence is conditioned solely on the encoder output frame at its corresponding time step. Consequently, this allows for fully parallelized compu- tation of P P(u(t) across different time steps, resulting in highly efficient inference.
CTC models with non-autoregressive components like self-attention and convolution widely accepted as non-autoregresssive models (Higuchi et al.] 2022 2020 Xu et al.) 2023a), although during inference, CTC's token deduplication and blank handling would still depend on the last emitted token. We point out that this NAR categorization is based on the neural network computation of the model. CTC, all neural network computation during inference can be carried out non- autoregressively, and the only autoregression is done on basic data types (strings, lists, etc.), which induces negligible computation compared to the costly neural network computations. In this work, we also follow this principle in categorizing models as autoregressive or not.
3.2 RECURRENT NEURAL NETWORK TRANSDUCER (RNN-T)
The ANN-Transducer, proposed by Graves (012)). represents significant advancement in sequence-to-sqquee -e-queeece modeling. RNN-T comprises three key components: an encoder, predic- tor, and a joint network. The encoder extracts high-level features from the audio input, while the predictor processes the text history. The joint network then combines these outputs to generate a probability distribution over the vocabulary.
Like CTC, RNN-T incorporates blank symbol, but with different rules for augmented sequences. (1) RNN-T sequences add zero or more blanks between adjacent tokens in the original sequence, rather than permitting repetition of both blanks and tokens, and (2) Only blank tokens consume input frames; the time step & does not increment with non-blank token predictions of the model.
RNN-T's training aligns with CTC, maximizing the probability of the target sequence by summing over the probabilities of all possible augmented sequences. However, RNN-T diverges from CTC's conditional independence assumption. Instead, it leverages the predictor's 's output, representing text history, in the joint network to compute output distributions. As such, its predictions conditioned by the alignment of input time step and position of current text output, making token estimation P(u|z,, V.. This incorporation of text history enables RNN-T models to outperform CTC.
3

---

Published as a conference paper at ICLR 2025
Despite its superior performance, RNN-T's reliance the predictor and text history introduces autoregressive behavior in inference. Consequently, RNN-T inference resists full parallelization, resulting in lower efficiency compared CTC. This trade-off between performance and efficiency highlights the ongoing challenge of designing optimal sequence-fo-ssqqueeee models for ASR.
3.3 TOKEN-AND-DURATION TRANSDUCERS (TDT)
Token-and- Siker-and-Duration Transducer (TDT) model is introduced by Xu al. (2023b). Unlike tradi- tional Transducers where only blank predictions increment the encoder index by exactly onc, TDT supports advancing by multiple steps for both blank and non-blank predictions. A TDT model gen- erates two probability distributions, one for token, and the other for durations, which determines the number of audio frames to skip. By supporting frame skipping, TDT models require significantly fewer decoding steps compared to traditional Transducers, resulting in faster inference.
HYBRID-AUTOREGRESSIVE INFERENCE TRANSDUCER (HAINAN)
We propose Hyybrid-Autoregresstit Inference Transducers (HAINAN), extension of the TDT model designed to support multiple modes of autoregression in inference. HAINAN incorporates stochastic predictor masking: during training, the predictor's output is randomly masked before being passed to the joint network. We implement this with a masking probability of 0.5, sampled text-index: level. Except for the random masking performed in training, the
at the (batch, rest of the model training of HAINAN (computation in other components, and loss computation, etc) remains identical to that of TDT training. As result of this change, during training, half the time the joint network is learning to predict with the whole model, and the other half the time it is learning to predict with information from the encoder only. Therefore when the model is well-trained, the joint network can generate valid distributions regardless of whether the predictor output is provided.
The HAINAN model supports three inference modes: autoregressive (AR), non-autoregressive (NAR) depending on whether we use the predictor network during inference. In addition, we also propose autorrrrgeeeee (SAR) inference, which combines elements of both NAR and AR in- ference to achieve better speed-accuracy tradeoff, and Viterbi-based inference, which leverages the better representations learned by HAINAN models for better accuracy.
Algorithm Autoregressive Inference
Algorithm Non-AR Inference
I: Input: acoustic input
1: Input: acoustic input
2: enc encoder(2:) # [T. H]
2: enc cncoddr((2) # [T. H]
3: hyp = [];
3: token.probs, dur dur.probs = parallel. joint(ccn, pred=None) # [T, V] and [T, DJ
4: while < len(enc) do
5: pred = predictryyy)) # [II]
4: tokens argmax(token_ probs, dim=-1) # [T]
6: token_probs, dur prubsprobss joint(enc[{], pred)
5: durations argmax (wup:rrrrosddimmi11)) # [T]
[V] and [D]
6: hyp []: t=
7: token= argeax probs)
7: while < len(enc) do
8:
duration = argmax dur probs)
8: token = tokens[t]
9:
if token is not blank then
9: if token not blank then
typeap)end(token) 11: += duration
10:
10:
hyp.append(token)
11: += max(1, duration(t)) avoid infinite loop
12:
Return hyp
12: return hyp
4.1 AUTOREGRESSIVE INFERENCE
Autoregressive inference of HAINAN models is identical to TDT. For ease of reference, we provide Algorithm I which rewrite of the original Algorithm from Xu et al. (2023b) with slightly changed notation/naming conventions to be consistent with this paper. To facilitate understanding of the Algorithms, we include tensor shapes as comments in the Algorithm, and we use T, V, D, U, H to represent audio-length, num-tokens, num-durations, text-index, and hidden-dimensions.
4

---

Published as a conference paper at ICLR 2025
4.2 NON-AUTOREGRRESIVV INFERENCE
Algorithm shows NAR inference procedure. Compared to AR inference, NAR inference requires simple modifications of (1) passing all-zero tensor instead of predictor output to the joint network; (2) moving the joint network computation out of the decoding loop, allowing parallel processing of all encoder outputs: (3) after each step, we increment by at least one avoid infinite loops.
4.3 SEMI-AUTOREGRESSVVE INFERENCE
SAR inference is novel algorithm combining elements of both AR and NAR approaches, as de- tailed Algorithm 3 For ease of description, we assume the NAR inference procedure also returns the corresponding time-stamps of token outputs. The Algorithm consists of two main phases:
1. Initial Hypothesis Generation (lines 2-4) performs NAR inference to generate an initial hy- pothesis. It also uses the time stamps of the initial hypothesis to extract token-emitting frames from the encoder output for later processing.
2. Hypothesis Refinement (lines 5-8) uses the predictor to compute representations of all par- tial history contexts in the current hypothesis. It then combines the predictor's s output with se- lected encoder output frames in the joint network to re-compute probability distributions for those frames, in paralle This procedure can be repeated to further improve the results. Note if re- peated, all but the last round of the refinement should limit the argmax line 9 to non-blank tokens since the token would need to be passed in the predictor for the next round of refinement.
The SAR approach combines the strengths of AR and NAR methods, allowing high degrees of parallelization while leveraging text histories in predictor computation to improve model accuracy.
Algorithm 3 Semi--utoregreessive Inference of HAINAN Models
I: Input: acoustic imput x
2: enc = =encoder(x) [T. H]
3: hyp, time_stamps = NARInferencc((nc) # [U] and [U]
4: esefullffrance enc [time_stampss # [U, H]
5: shifted.hyp [BOS] hyp[:-1] # [U]
pred= prediitoor(shifeeddyyy
[U, H]
7: token probs, duration prohs parallel joint(useful frames, pred) # of shapes: [U, V] and [U, D]
8: hyp= argmax(tokee) probs, dim=-1) 1) [U]
9: Return: hyp
4.4
VITERBI-DECODING
In addition to AR/NAR and SAR inference, we also propose Viterbi-based inference method for HAINAN models. Given an audio sequence length T, we view the output of the NAR HAINAN model as directed acyclic graph (DAG) with T + nodes, while the first T correspond to frames and the last acts as a special end-of-sentence (eos) token. For each frame, we only consider the arg- max token probability, which represents the weight of the first T nodes (the cos node has a weight 1). The duration outputs at each frame correspond to transition weights into neighboring nodes. For example, in our models with max-duration 8, each node has most 8 incoming and 8 outgoing connections connecting to its neighbors.
With this setting, speech recognition the input audio is equivalent to finding the best path of the DAG from the first node to the last, which can be efficiently solved by simplified Viterbi decoding algorithm. Notably. Viterbi and SAR techniques are orthogonal, and after Viterbi decoding generates the tokens and time stamps, we can use SAR techniques to further improve the results. The Algorithm for Viterbi decoding shown in Algorithm For simplicity, this Algorithm returns a "backtrack" structure that encodes the selected frames found by the best-path algorithm. The Algorithm's key between lines 7 and 14, where nested loops run over (time, duration) pairs, and for each node, it finds the probability of the best path from the first node to that node using a dynamic
The computation at line 7 is done fully parallelized, and line 6 may require some sequential computation if the predictor is L.M. If the predictor stateless, this computation also be fully parallelized well.
5

---

Published as a conference paper at ICLR 2025
programming algorithm. The Algorithm can be carried out efficiently with (TT)) time complexity, where T is the input sequence length and D is the number of durations. terms of space complexity, this Viterbi algorithm requires (IT)) (to store best-prob and backtrace pointers for each time index).
Because duration O is not allowed in NAR modes, we assume there's 's no duration 0 in N.
Algorithm Viterbi Decoding of HAINAN Models
I: Input: acoustic input 2, supported durations N
2: enc = encoder(x)
3: token.probs, duration_prodee probs parallel_joint(nne pred=None) # of shapes [T, V] and [T,, DJ max(token_prett probs, dim==1) # of shape [T]
4:
5: best.prob = [0.0 for range(len(x) 1)]
range(len(x)) 1)] we include a terminal node for backtracing 7: for target in range(1, len(x)+ + 1) do
backtrack = [-1
8: for idx, n in enumerate(N) do
9:
source max((-n, 0) # duration states cannot originate before initial node alpha = prob- sroussuucee trans-prob-
10:
11:
wttp_ppe_frraeeeeeeeeeppp (targtt]
12:
if alpha trams prob >> best_proobttttteeeeeeeeee then
13:
best_prooee prob[target] = alpha trans prob
14:
backtrack[target)- source
15:
Return: backtrack
EXPERIMENTS
We evaluate HAINAN ASR in two languages: English, German. All experiments are conducted using the NeMo (Kuchaiev et al., 2019) toolkit, version 1.23..0 0. We first extract 80-dimensional filterbank features on audio, with 25 ms windows at 10 ms strides. All models use FastConformer encoders with the first three layers all performing 2X subsampling, therefore 8X subsampling in total. Both TDT and HAINAN models use durations {0. 1, 2, 7, 8). BPE tokenizer (Sennrich al. 2016: Kudo & Richardson 2018) of size 1024 used for text representation. For all experi- ments, we let models train for sufficient steps until validation performance degrades (no more than 150k training steps), and run model averaging on 5 best checkpoints to generate the final model for evaluation. For all datasets, we report ASR performance measured by WER and the inference speed of RNN-T, CTC, TDT and HAINAN models. For HAINAN models, we include results of AR, NAR and SAR modes, and use the notation "SAR-n indicate rounds of refinement runs). We include Transducer models with stateless predictors with Ittoken context denoted as "stateless", and use "/0" to denote TDT/HAINAN models that use all positive durations 2,, ..., 7, 8}.
5.1 ENGLISH ASR
We train our English models on the combination of Librispeech (Panayotov et al. 2015). Mozilla Common Voice (Ardila et al. 2019). VoxPopuli (Wang al., 2021), Fisher (Cieri al. 2004
People's Speech (Galvez et al. 2021). Wall Street Journal (Paul & Baker 1992). National Speech Corpus (Koh et al. 2019). VCTK (Yamagishi et al. 2019). Multilingual Librispeech (Pratap al.] 2020 Europarl (Koehn 2005) datasets, plus Suno AI datasets. In total, our dataset consists of approximately 60,000 hours of speech. The encoder uses FastConformer-XXL architecture with 42 layers of conformer blocks, each of which uses 8 heads of self-attention layers with model hidden dimension 1024, totaling around 1.1b parameters. The convolutions in the conformers use kernel size = 9. The encoder was initialized with the public checkpoint https:: : co/nvidiiiiiooccccooo a/ pa Darakeet-tar tot- 1. 1b, For standard RNN-T, TDT and HAINAN models, their predictors consist of 2-layer LSTMs, with hidden dimension 640. The joint network is 2-layer feed-forward network with ReLU in between and with hidden dimension 1024.
We evaluate the models using the Huggingface ASR leaderboard (Srivastav et al. 2023 including the following datasets: AMI test (ami), Earnings22 test (e22). Gigaspeech test (giga), Librispeech test-clean and test-other (clean and other), Spgispeech test (spgi), Tedlium test (ted), and VoxPopuli test (vox). The results are shown in Table [[] Since there are multiple datasets, we direct readers' attention to the average WER achieved by different models. From the Table see:

---

Published as a conference paper at ICLR 2025
Table HAINAN English ASR: accuracy (WER%) and wall time on Huggingface ASR leaderboard datasets. AR/NAR/SAR represent www.errresiieeooonutttreegee number after SAR represents the number of refinement runs. Decoding time (seconds) is measured on only librispeech-test-oliee not included since it's not comparable due to the different nature of the search algorithm. All
and cemi-autoregressive. The
using batch=1 and beam=l, running 2 A6000 GPUs. Viterbi time
models have around 1.1 billion parameters differing by at most 600k in actual number.
model
AR
ami
c22
giga
clean
other
spgi
ted
VOx
AVG
time
RNN-T
AR
16,90
13.87
9.76
1.43
2.75
3,40
3.63
5.49
7.15
179
TDT
AR
16.18
14.62
9.61
1.39
2.53
3,47
3.75
5.52
7.13
88
TDT/O
AR
16.00
14.64
9.61
1.38
2.61
3.36
3.67
5.50
7.10
87
stateless TDT
AR
15.96
14.43
9.63
1.36
2.59
3.37
3.59
5.58
7.06
84
CTC
NAR
16.59
14.43
9.93
1.53
2.93
3.95
3.84
5.81
7.38
39
AR
15.92
14.16
9.70
1.46
2.76
3.59
3.59
5.60
7.10
89
NAR
16.13
14.38
9.79
1.49
2.83
3.61
3.64
5.68
7.19
HAINAN
-Viebii
16,23
14.03
9.75
1.48
2.81
3.53
3.60
5.60
7.13
SAR-1
16.01
14.23
9.73
1.47
2.76
3.60
3.64
5.61
7.13
45
16.13
13.93
9.71
1.47
2.77
3.51
3.60
5.56
7.09
SAR-2
15.99
14.17
9.72
1.47
2.75
3.59
3.63
5.61
7.12
48
AR
15.69
13.64
9.68
1.49
2.82
3.47
3.74
5.72
7.03
82
NAR
15.78
13.73
9.77
1.49
2.91
3.49
3.75
5.75
7.08
40
stateless HAINAN
Viterii
15.87
13.45
9.72
1.47
2.87
3.42
3.74
5.70
7.03
SAR-1
15.77
13.72
9.72
1.49
2.84
3,48
3.74
5.71
7.06
42
15,86
13.42
9.69
1.47
2.82
3,41
3.74
5.69
7.01
SAR-2
15.75
13.61
9.70
1.48
2.84
3.47
3.72
5.71
43
AR
15.75
13.37
9.73
1.40
2.83
3.52
3.56
5.76
6.98
92
NAR
15.94
13.67
1.44
2.87
3.55
3.54
5.74
40
HAINANO
- Viterti
15.88
13.38
9.73
1.41
2.86
3.48
3.48
5.65
6.98
SAR-1
15.84
13.45
9.74
1.41
2.85
3.52
3.54
5.69
7.00
44
-Viuubi SAR-2
15.79
13.21
9.68
1.39
2.83
3,46
3.50
5.60
6.93
15.79
13.36
9.73
1.40
2.83
3.52
3.55
5.69
6.98
48
Regardless of the nature of the predictor, AR HAINAN models achieve superior accuracy than RNN-T and TDT counterparts, with similar efficiency.
NAR-HAINAN achieves consistently better accuracy than CTC, with similar inference speed. In particular, NAR stateless HAINAN achieves better accuracy than the AR TDT model, while rumning over 2X faster; it also outperforms RNN-T model, while being over 4X faster.
SAR further improves the accuracy of NAR inference with small computational overhead. With standard HAINAN, each additional SAR refinement run adds around 3-4 seconds processing for the whole dataset, while stateless HAINAN only requires 1-2 seconds per SAR run. This is significantly faster than fully AR TDT runs which takes around 40 seconds more processing time in total. Also in both models, just one round of SAR makes the model catch up with TDT performance while running at least 2X faster.
Viterbi decoding can further improve model accuracy. We present an analysis in subsection 6.2 that can explain this phenomenon.
In particular, the HAINAN model with non-zero durations achieve the best results overall. Later we present our discovery that can explain this phenomenon.
in Section 6.4
5.2 GERMAN ASR
Our German models are trained German Mozilla Common Voice, Multilingual Librispeech, and VoxPopuli datasets, totaling around 2000 hours. We use FastConformerlarge configuration of around 110m parameters P and evaluate the models the German Multilingual Librispeech (MLS) and VoxPopuli (Vox) testsets. For training, all model encoders are initialized with public
Thhe hyper-parameters can be found at https://github..om
yaml

---

Published as a conference paper at ICLR 2025
wo/.viiiaasst.ttcccccc.c.c...... checkpoint httpp/////...o//ovai/////tttttt
fastconformer http:/ommmeerrraaaeeeeeeeeeeeeee large The results of those models are shown in 2
We similar trends compared to our English models, where HAINAN achieves similar AR accu- racy and efficiency with TDT models, and significantly improved NAR accuracy compared to CTC models, with similar inference speed. SAR further closes the accuracy gaps between AR and NAR, while adding small overheads, and still being around 4X faster than AR inference.
Table 2: German ASR: accuracy (WER%) and inference wall time (seconds) on Multilingual Lib- rispeech and VoxPopuli. Decoding time measured with batch=1 and beam=1.
model
AR
MLS
time
VOX
time
RNN-T
AR
4.90
429
9.47
113
TDT
AR
4.77
141
9.41
52
TDT0
AR
4.86
140
9,47
stateless TDT
AR
4.88
131
9.36
49
CTC
NAR
5.92
22
10.25
AR
4.75
141
9.22
NAR
5.11
20
9.63
HAINAN
SAR-1 1
4.87
29
9.47
11
SAR-2
4.83
36
9,44
AR
4.85
127
9.35
46
stateless HAINAN
NAR
5.09
22
9.57
SAR-1 1
4.89
24
9,45
SAR-2
4.87
25
9.44
10
AR
4.92
140
9,59
NAR
5.37
22
10.08
HAINAN/O
SAR-1
5.12
29
9.87
11
SAR-2
5.05
35
9.84
14
ANALYSIS AND DISCUSSIONS
6.1 LIMITATIONS OF SAR METHODS
The key of SAR method is reuse the time stamps generated in the initial NAR inference, and only regenerates the token outputs for those frames. This puts limitations on the types of errors that can be corrected by the method. We provide some analysis here.
From the subword level, SAR can only fix substitution errors by replacing a wrong subword with the correct one, and insertion errors by replacing the inserted wrong subword with blank. It can't fix deletions where the hypothesis shorter than the reference.
From the word level, SAR can correct deletion error.. Note, although SAR can't add more subwords to the original output, it can replace a non-word-beginning subword with a word- beginning subword, and thus increase the number of words in the SAR output. E.g. the SAR can change "forty" to "for tea", , by replacing subwords for ty" with for tea",, where means the begin-of-word symbol.
Table 3: Breakdown of Error types (%) of German HAINAN Models Voxpopuli Dataset.
inference
sub
del
ins
total
NAR
5.19
2,64
1.80
9.63
SAR-1
5.00
2.85
1.62
9.47
SAR-2
4.96
2.87
1.61
9.44
We report detailed breakdown of error types in our German HAINAN model on Voxpopuli compar- ing NAR and SAR in Table We that SAR models primarily reduce substitution and insertion
8

---

Published as a conference paper at ICLR 2025
errors; in fact, the total amount of deletion enrors increased with SAR models. Even though the overall deletion error numbers go up. we do observe in certain utterances, SAR can correct deletion errors from NAR outputs. We provide some examples in Appendix subsection A.3
6.2 NAR HAINAN'S CAPABILITY OF CAPTURING ACOUSTIC AMBIGUITY
In this subsection, we highlight aspect of NAR HAINAN model that grants it greater modeling power compared CTC models: its ability to support more diverse model representation and multiple hypotheses across different time stamps, especially for potentially ambiguous speech. To illustrate this, we present real-world example using an audio recording of the phrase "ice cream".. First, we run our English CTC model on the utterance to extract the arg-mas token per frame, resulting in:
<b>
<b>
<b>
<b>
<b>
<b>
<b>
ce
<D>
<b>
cre
<b>
<b> am
<b>
<b>
<b>
<b>
The CTC model output correctly represents the "ice cream" hypothesis, with each non-blank token appearing once, surrounded by blanks. This aligns with the well-documented peaky behavior of CTC (Zeyer et al. 2021). In contrast, the arg-max tokens emitted from NAR HAINAN model are:
<b> 1
cre
<b>
<b>
Notably, the per-frame arg-max output not only encodes "ice cream" but also captures an alternative hypothesis: "i scream". These hypotheses are dispersed across different time frames. For clarity, we have color-coded the tokens: orange for "ice cream", blue for scream", and green for shared tokens. It's worth noting that these hypotheses are acoustically similar, and their different align- ments to the audio demonstrate that the NAR HAINAN's per-frame max output provides a more accurate representation of the audio input. We believe this is the reason why the Viterbi algorithm can achieve improved accuracy.
6.3 IMPACT OF DURATION CONFIGURATION
In this subsection, we examine how duration configurations affect HAINAN model performance. We vary the max-durations our German models and report their ASR performance Table Our results reveal a clear trend: models with max max-durations of and 2 perform significantly worse, while and yield improved results. This observation aligns with our earlier analysis, suggesting that shorter maximal durations constrain the model's ability to learn complex structures, and the max duration needs be sufficiently large to represent the real distribution. Interestingly, max-duration 4 provides the best overall results, despite our choice of max-duration 8 for most experiments. We selected the latter to accommodate greater linguistic variability across datasets, as we did not exten- sively tune this parameter.
6.4 DURATION DISTRIBUTION ANALYSIS
Building on our previous analysis of maximum duration settings, now examine the actual dura- tion predictions made by our models. This analysis provides insights into how the models utilize the allowed duration range and how the HAINAN training procedure affects duration prediction behav- ior. We conducted experiments using HAINAN and TDT (trained with max max-ddration=8)) running different types of inference on the Librispeech test-other and German Multilingual Librispeech test
Table 4: German ASR results of NAR HAINAN models with different maximum durations.
max-duration
ML.
VOX
1
7.58
12.46
2
5.79
10.27
4
5.06
9.56
8
5.09
9.57

---

Published as a conference paper at ICLR 2025
MRNAINNN
NARHANNAN
TDT
AR HAINAN
NARHAINNN
TDT
40000
100000
30000
75000
count
20000
count
50000
10000
25000
duration
duration
Figure Duration prediction count of HAINAN and TDT models on Librispeech test-other (left)
and German Multilingual Librispeech test (right).
sets. Figure T shows the distribution of duration predictions for HAINAN (in both AR and NAR
modes) compared to standard TDT model. The results reveal several interesting patterns:
Zero-duration mitigation: HAINAN's training procedure dramatically reduces the occurrence of 0 duration predictions, effectively addressing the potential issue of infinite loops in NAR infer- ence. This phenomenon explains the results from Table 1 where excluding duration 0 gives the best results among all models.
Preference for shorter durations: Despite allowing durations up to 8, all models show pref- erence for durations of or less, with the preference stronger with HAINAN models. This aligns with previous finding that max-duration=4 tends to yield better performance.
Consistency across inference modes: HAINAN shows remarkably similar duration distributions in AR and NAR modes, suggesting a robust learning of duration prediction strategies.
These findings demonstrate that while allowing longer maximum durations (e.g., 8) provides flex- ibility, the HAINAN model naturally learns to favor shorter durations. This behavior reconciles with earlier observation that max-duration=4 yields optimal performance: the model has the flexibility to use longer durations when necessary but primarily operates within the 1-4 range.
CONCLUSION AND FUTURE WORK
In this paper, we introduced Autrid-Autoreeressivve INference TrANsducers (HAINAN), novel approach that aims to bridge the gap between autoregressive and non-autoregressive methods in speech recognition. By implementing simple stochastic masking technique during training, we' ve developed model that offers flexibility in its inference strategy. Our experiments across multiple languages demonstrate promising results:
In AR mode, HAINAN achieves on-par performance with TDT and RNN-T models.
In NAR mode, HAINAN achieves significantly improved performance to CTC models.
We propose a autreervtoee (SAR) mode, a new approach to balancing speed and accuracy.
We also propose a simplified Viterbi decoding method to further improve model accuracy.
There are several future research directions, including applying method other speech and language-related tasks including speech translation, spoken language understanding, and speech synthesis, etc. We plan investigate more flexible model architectures further to bridge the gaps between AR and NAR models. Furthermore, we plan to further improve the autrerrotorre algorithm for better speed-accuracy balancing.
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A masiivll--ultilinguaal speech corpus. arXiv: 2019.
10

---

Published as a conference paper at ICLR 2025
Siddhant Arora, George Saon, Shinji Watanabe, and Brian Kingsbury. Semi-autoregressie stream- ing ast with label context. In ICASSP 2024-2024 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 11681-11685. IEEE, 2024.
Vladimir Bataev, Hainan Xu, Daniel Galvez, Vitaly Lavrukhin, and Boris Ginsburg. Label-looping: Highly efficient decoding for transducers. arXiv:2406.06220. 2024.
William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, Attend and Spell. In ICASSP, 2016.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. NeurlPS, 28, 2015.
Christopher Cieri, David Miller, and Kevin Walker. The fisher corpus: resource for the next generations of speech-to-text. In LREC, volume 4, pp. 69-71, 2004.
Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The people's ' speech: A large-scale diverse english speech recognition dataset for commercial usage. arXiv:2111.09344, 2021.
Daniel Galvez, Vladimir Bataev, Hainan Xu, and Tim Kaldewey, Speed of light exact greedy de- coding for mn-t speech recognition models on gpu. arXiv:2406.03791, 2024.
Mohammadreza Ghodsi, Xiaofeng Liu, James Apfel, Rodrigo Cabrera, and Eugene Weinstein. RNN-Transducer with stateless prediction network. In ICASSP, 2020.
Alex Graves. Sequence transduction with recurrent neural networks. In 2C12.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tempo- ral classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
Anmol Gulati, James Qin, Chung-Cheng Chiu, et Conformer: Convolution-augmenntee trans- former for speech recognition. In Interspeech, 2020.
Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, al. Streaming end-to-end speech recognition for mobile devices. In ICASSP, 2019.
Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask ctc: Non-autoreerressive end-to-end asr with etc and mask predict. arXiv preprint arXiv:2005.08700, 2020.
Yosuke Higuchi, Hirofumi Inaguma, Shinji Watanabe, Tetsuji Ogawa, and Tetsunori Kobayashi. Im- proved mask-ctc for aunooregrgrreivve end-to-end asr. In /CASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8363 0367. IEEE, 2021.
Ke Hu, Tara N Sainath, Ruoming Pang, and Rohit Prabhavalkar. Deliberation model based two- pass end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7799-7803. IEEE, 2020.
Hirofumi Inaguma, Yosuke Higuchi, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. Non- autoregressive end-to-end speech translation with parallel autoregressive rescoring. arXiv preprint arXiv:2109.044/1, 2021.
Mahaveer Jain, Kjell Schubert, Jay Mahadeokar, Ching-Feng Ych, Kaustubh Kalgaonkar, Anuroop Sriram, Christian Fuegen, and Michael Seltzer. RNN-T for latency controlled ASR with im- proved beam search. 1rXiv:1911.01622 2019.
Wei Kang, Liyong Guo, Fangjun Kuang, Long Lin, Mingshuang Luo, Zengwei Yao, Xiaoyu Yang, Piotr Zelasko, and Daniel Povey. Fast and parallel decoding for transducer. In ICASSP, 2023.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. Proceedings of machine translation summit papers, 2005.

---

Published as a conference paper at ICLR 2025
Jia Xin Koh, Aqilah Mislan, Kevin Khoo, Brian Ang. Wilson Ang, Charmaine Ng. and YY Tan. Building the Singapore English national speech corpus. Malay, 3, 2019.
Nithin Rao Koluguri, Travis Bartley, Hainan Xu, Oleksii Hrinchuk, Jagadeesh Balam, Boris Gins- burg, and Georg Kucsko. Longer is (not necessarily) stronger: Punctuated -ong-eequence training for enhanced speech recognition and translation. arXiv:2409.0560) 2024.
Oleksii Kuchaiev, Jason Li, Huyen Nguyen, et al. Nemo: toolkit for building ai applications using neural modules. In NeurlPS Workshop on Systems for ML, 2019.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. /
2018.
Jay Mahadeokar, Yangyang Shi, Ke Li, Duc Le, Jiedan Zhu, Vikas Chandra, Ozlem Kalinli, and Michael L Seltzer. Streaming parallel transducer beam search with fast-slow cascaded encoders. arXiv preprint arXiv:2203. /5773, 2022.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In ICASSP, 2015.
Douglas B Paul and Janet Baker. The design for the Wall Street Journa--based CSR corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, 1992.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A large-scale multilingual dataset for speech research. In Interspeech, 2020.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, et al. SpeechBrain: A general- purpose speech toolkit. In Interspeech, 2021.
Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, et al. Fast Conformer with linearly scalable attention for efficient speech recognition. In Automatic Speech Recognition and Understanding Workshop (ASRU), 2023.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of the 54th Annual Meeting of the ACL, 2016.
Vaibhav Srivastav, Somshubra Majumdar, Nithin Koluguri, Adel Moumen, Sanchit Gandhi, et al.
Open automatic speech recognition leaderboard. bettps://huggingffaee ctt.aaacecccceee..ooooooeeeeeeee spcssppcs/// /tppa///cccccccccccccss////
leaderboa rd 2023.
Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley. Hybrid autoregressive transducer. In ICASSP, 2020.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proc. of the 59th Annual Meeting of the ACL and the 1th International Joint Conf. on NLP, 2021.
Weiran Wang. Ke Hu, and Tara N Sainath. Deliberation of streaming mn-transducer by non- autoregressive decoding. ICASSP 2022-2022 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 7452-7456 IEEE, 2022.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, et al. ESPnet: End-to-end speech processing toolkit. In Interspeech, 2018.
Chen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun, Yuhao Zhang, Murun Yang, Qianqian Dong, Tom Ko, Mingxuan Wang, Tong Xiao, al. Cte-based non-autoregressive speech translation. arXiv preprint arXiv:2305.17355 2023a.
Hainan Xu, Fei Jia, Somshubra Majumdar, Shinji Watanabe, and Boris Ginsburg. Multi-blank transducers for speech recognition. arXiv:2211.03541, 2022.
Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg. Effi- cient sequence transduction by jointly predicting tokens and durations. In ICML, 2023h.
12

---

Published as a conference paper at ICLR 2025
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English
Multi-speaker Corpus for CSTR Voice Cloning Toolkit (ver. 0.92), 2019. URL https://aaas
Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition. In ICLR, 2023.
Ching-Feng Yeh, Jay Mahadeokar, Kaustubh Kalgaonkar, et Transformer- end speech recognition with self-attention. 1XXi::1100.1277, 2019.
Transformer-Tanndduerr Transducer: End-to-
Albert Zeyer, Ralf Schlüter, and Hermann Ney. Why does CTC result in peaky behavior?
arXiv:2105, 14849, 2021.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TRAINING ALIGNMENTS
The strength of NAR HAINAN to better model acoustic ambiguity can also observed during model training. Figure Sigure shows the force-alignment probabilities for both "ice cream" and "i scream" hypotheses. Notably, the HAINAN model can align both hypotheses with high probabilities by separating them at different time stamps, capability not possible with CTC models due to their frame-by-frand independence assumption and lack frame-skipping mechanism.
To understand why CTC models cannot assign high probabilities to both hypotheses simultane- ously, consider the following scenario: one frame assigns high probability to "ce" (from "ice"), and another frame assigns high probability to "sc" (from "scream"), the resulting sequence would inevitably be recognized as "cesc" or "scce", matching neither "ice cream" nor scream". This lim- itation is unique to frame-by-frame In contrast, the HAINAN model's s ability to skip frames and represent multiple hypotheses allows to better capture the inherent ambiguity in speech.
models with conditional independence assumptions like CTC.
Alignment for ce cre am"
1.0
0.8
hrobaiiltty
0.6
Alignment for
sc re am"
0.2
10 Time Frame (t)
14
16
18
0.0
Figure 2: Alignments generated by HAINAN model without predictor, on "ice cream" and "i scream".
A.2 COMPARISON WITH HYBRID-TITT-CTT MODELS
To further validate the effectiveness of our approach, we compare HAINAN with Hybrid-TDT-CTC models (Koluguri et al.] 2024), which combines elements of both CTC and TDT models. With the Hybrid-TDT-CTC model, the encoder output processed by both CTC embedding layer and TDT joint/predictors, and the model is trained using linear interpolation of CTC and TDT losses. During inference, the model supports both AR (using TDT components) and NAR (using CTC components) inference modes. Table 5 presents the performance comparison between Hybrid-TDT- CTC and HAINAN models across various datasets from the Huggingface ASR leaderboard.
Key observations from this comparison:
Table 5: Hybrid-TDDe TDT-CTC model accuracy (WER%) on Huggingface ASR leaderboard datasets
model
inference
ami
c22
giga
clean
other
spgi
ted
vox
AVG
TDT (AR)
16.17
14.77
9.56
1.37
2.64
3.65
3.71
5.56
7.18
hybrid
CTC (NAR)
17.13
15.33
9.90
1.51
2.88
3.65
3.95
5.71
7.51
AR NAR
15.92
14.16
9.70
1.46
2.76
3.59
3.59
5.60
7.10
HAINAN
16.13
14.38
9.79
1.49
2.83
3.61
3.64
5.68
7.19
14

---

Published as a conference paper at ICLR 2025
Performance: HAINAN outperforms Hybrid- Hybrid-TDT-CTC in both AR and NAR modes across most datasets, with lower average WER scores.
HAINAN does not require storing separate sets of parameters for AR and NAR inference, leading to more compact model.
Model Efficiency: Unlike
Flexibility: HAINAN offers an additional semi-auttorerressive (SAR) inference mode, providing more flexible speed- -eceracyy trade-off not available in Syyrri-TDI-CTO models.
These results underscore the advantages of HAINAN's unified architecture, which achieves superior performance while maintaining model simplicity and offering greater inference flexibility.
A.3 EXAMPLES OF DELETION ERRORS CORRECTED BY SAR METHODS
Although relatively rare, SAR methods can correct deletion errors of NAR inference results. Here are some examples German and English that we observe our runs. We use green highlight words that were missing in NAR hypotheses but included by SAR.
German:
Table Example German ASR where SAR reduces deletion errors from NAR output
reference
die meisten vorfälle passieren bei der implementierung bedauerlicherweise im krankenhaus häufig auch in der anwendung
NAR
diesfälle passieren bei der implementierung bedauerlicherweise im krankenhaus häufig auch in der anwendung die handfälle passieren bei der implementierung bedauerliicerweeiee im krankenhaus häufig auch in der anwendung
SAR
reference
ich ersuche den kommissar hier sofortmaaBnahmen zu ergreifen ichsuche den kommissar hier sofort maßnahhen zu ergreifen ich versuche den kommissar hier sofortmaßnahmen zu ergreifen
NAR
SAR
English:
Table 7: Example of English ASR where SAR reduces deletion errors from NAR output
reference
then one of them says kind soft and gentle then of them says kinda soft and gentle then one of them says kind of soft and gentle
NAR
SAR
reference
say mester gurr sir which thankful i am to you for speaking so but you don't tt really think as he has come to harm
NAR
say mister gurrsir which thankful am for you for speaking but you don't really think as he has come to harm
SAR
say mister gurr sir which thankful i am for you for speaking so but you don't really think as he has come to harm
We notice that, in all those examples of corrected deletion errors, the SAR method replaces word-beginning subword with a word-beginning subword, e.g. "_kind a" is replaced with " "_kind _of". This aligns with our analysis in subsection 6.1 since this is the only way increase the number of words while keeping the number of subwords fixed.
A.4 REASON OF ZERO-DURATION MITIGATION WITH HAINAN
In subsection 6.4 show empirically that duration 0 is greatly mitigated by HAINAN model. In this subsection, we provide some analysis of why this is the case. our description, we use E, and D., to denote encoder/predictor outputs.
15

---

Published as a conference paper at ICLR 2025
1. By performing stochastic masking of predictor output during training, the model is learning to make the output distribution of joint( E,, Du) similar to joint(E,, D. 0).
2. If duration 0 is predicted from joint( Et, D.),, then the next step of processing would be joint( Ex, Du+1), where i remains the same but gets incremented by 1.
3. From 1, we know that model is learning to make joint E,, Du) similar to joint( Ex, Du 0); also it's learning to make joint( (e, Du+1) similar joint( Et. Du+1 * 0). Consequently, the model would make joint( E,, D..) similar to joint(E;, Du+1), since joint E, D,, * 0) is the same as joint( P,, Du+1 * 0), both of which has predictor masked out.
4. However, we know that joint( Ee, Da) and joint( Ex, Du+1) should predict different tokens since the text histories are different. This causes contradiction.
Therefore, HAINAN models, by adopting stochastic predictor masking, would naturally suppress the prediction of duration 0 in its processing.
16


Reviews:
[{"review_id": "D4CYn9tcdQ", "reviewer": "Reviewer_rxjr", "summary": "This paper proposes a training method to improve the performance of Token-and-Duration Transducer (TDT, Xu et al., 2023), which enables TDT to perform decoding in several different modes that improve efficiency without sacrificing performance much.\n\nThe main contributions are as follows\n1. Introduce “transcript masking” as a regularization method for TDT training, which improves the regular auto-regressive decoding performance\n2. Decode with “empty transcript” (ie feed entirely masked transcript to the predictor module) to greatly reduce the runtime for AR decoding. (This is referred to as non-autoregressive (NAR) decoding in the paper, but I do not agree it should be called NAR. See details in later sections)\n3. A hybrid decoding approach to improve the hypothesis proposed by empty transcript decoding. In particular, it uses that hypothesis as input the predictor module, gets predictor embeddings for each text position “in parallel”, and the take the argmax from each position in parallel.", "questions": "* (Line 254) How important is it to initialize HAI-T with TDT? Are the baseline models’ encoder initialized from a PT model? What’s the performance drop for HAI-T if it is not initialized from TDT?\n* Is the stateless predictor merely an embedding table?\n* Has the authors compared NAR with NAR-Viterbi? How are the duration posterior and token posterior combined (ie is a weight introduce to rebalance token and duration posterior)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Masking target is an effective approach verified in several other models (TDS for CTC, MaskPredict for NAR translation), which can serve as a method of regularization or a way to build non-autoregressive model. Applying to TDT is a clever idea which serves both purposes.\n2. Empirical results on large scale study are convincing in terms of the effectiveness of improving NAR and the more efficient decoding methods (NAR, SAR) proposed in the paper based on TDT\n3. The paper also presents a good set of ablation studies (stateless / no zero duration / duration range / duration distribution analysis / argmax token analysis) to provide readers more insights on why it works and how the model behaves.", "weaknesses": "1. Technically it is not correct to call the method non-autoregressive. Despite that the predictor does not take the predicted token as input, it still depends on the duration predicted from the previous step to determine what enc[t] to compute argmax on (line 7 of Xu et al. (2023)). Hence, they cannot be fully parallel and should be still considered autoregressive. I acknowledge that runtime wise it is almost identical to NAR because argmax can be precomputed for all (t, u), but this is still wrong in terms of the nomenclature.\n2. Missing discussion of limitations. For example, the refinement step cannot fix the error if the NAR hypothesis is shorter or longer than the ground truth. \n3. Missing comparison with similar or related methods. For example, how does it compare with generating top-K hypothesis with HAI-T NAR or CTC, and then rescore by an AR model\n4. The argument of HAI-T being superior than RNN-T and TDT is too strong. On individual datasets it loses in quite a few cases (e22, giga, clean, other, spgi, vox). The authors should just claim on-par.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qv7Hip1bSS", "reviewer": "Reviewer_n6nQ", "summary": "The paper introduces the Hybrid-Autoregressive Inference Transducer (HAI-T), an extension of the Token-and-Duration Transducer (TDT) designed to support autoregressive (AR), non-autoregressive (NAR), and semi-autoregressive (SAR) inference within a single framework. The key innovation is the use of stochastic predictor masking during training, enabling seamless switching between inference modes.\n\nAR inference follows the TDT process, while NAR inference simplifies processing by using a zeroed tensor in place of predictor outputs for single-pass decoding. SAR inference combines the strengths of both approaches, generating an initial hypothesis with NAR and refining it with AR-like processing using shifted representations. The paper also proposes a Viterbi-based decoding method to enhance results. Experimental evaluations across English and German datasets show that HAI-T outperforms CTC in NAR mode and matches or exceeds TDT and RNN-T in AR mode, with further improvements observed using Viterbi decoding.", "questions": "1. Why was the Viterbi-based decoding added as an after-thought rather than in the main experiment? By how much does it increase the inference times?\n\n1. Could you provide more details on the training setup? The link to hyperparameters in footnote 3 gives a 404-not found error.\n\n2. How were the inference times computed, and provide the confidence intervals?\n\n3. Did you try training the HAI-T from scratch? Especially since you have access to a large English dataset, training from scratch should be feasible.\n\n4. What are the potential impacts of initializing the German model with an English encoder checkpoint? \n\n5. When applied to data with high variability or noise, how does the model handle such conditions compared to existing approaches? \n\n6. Table 1 and 3 exceeds the margin, please ensure it fits within the margins.\n\n7. The algorithms could use some refinement to avoid ambiguity in mathematical operations and naming conventions. For example `shifted_hyp` instead of `shifted-hyp`", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper introduces a simple but effective technique: masking the predictor output 50% of the time during training, which allows the joiner to handle non-autoregressive (NAR) inference. This enables the model to support multiple inference strategies—AR, NAR, and SAR. The carefully crafted NAR and SAR inference strategies leverage the TDT architecture well and show improved performance across multiple datasets.\n\n**Quality**: The paper provides thorough experimental evaluations across various datasets in both English and German, and includes comparisons with standard models like CTC, TDT, and RNN-T. The consistent improvements in performance across these benchmarks demonstrate the robustness of the proposed approach.\n\n**Clarity**: The authors do a great job explaining how the AR, NAR, and SAR inference modes work, making it easier for readers to follow how HAI-T switches between different inference strategies. The inclusion of a code snippet to show how the masking is implemented during training is a nice touch that makes the paper accessible to researchers at various levels of ASR expertise.\n\n**Significance**: The introduction of NAR inference with TDT, which outperforms CTC, and the development of SAR inference, which balances the speed of NAR with the accuracy of AR, are important contributions for real-world ASR applications where both efficiency and accuracy are needed. HAI-T effectively addresses some of the limitations of existing transducer models and sets a new standard for flexible and adaptable ASR models. This work is likely to encourage more research in hybrid inference strategies within the ASR field.", "weaknesses": "1. **Incremental Architectural Innovation**: While HAI-T's hybrid approach to inference is creative, it builds on the existing Token-and-Duration Transducer (TDT) model. The primary novelty lies in its training strategy and inference flexibility rather than any fundamental architectural innovations. This could make the contribution seem incremental.\n\n2. **Lack of Detailed Training/Inference Specifications**: The paper omits crucial details about the training setup, such as the type and number of GPUs used, learning rates, schedules, and batch sizes. Additionally, the reported inference times lack confidence intervals and clarity on whether they were computed using a CPU or GPU. These omissions limit reproducibility and make it difficult to gauge the true computational demands of the model.\n\n3. **Dependency on Pretrained Models**: The reliance on pretrained models for initializing the encoder, which are already robust and highly optimized, raises questions about the origin of the reported performance gains. This reliance could make it challenging to separate the contribution of HAI-T's unique elements from the advantages conferred by the pretrained encoder. Furthermore, the use of an English checkpoint for initializing the German model might influence the results, yet the paper does not address this potential impact.\n\n4. **Clarity on Limitations**: The paper would benefit from a clearer discussion of the potential limitations of HAI-T. For instance, it does not explore how the model performs under challenging conditions, such as noisy or highly varied audio inputs, or whether it would require significant adjustments to maintain performance in such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "at6XDcexAX", "reviewer": "Reviewer_jHSk", "summary": "This paper proposes a simple yet effective method for training a Token-and-Duration Transducer (TDT) model by stochastically masking out the predictor. It also introduces a novel semi-autoregressive (SAR) inference mode, which first uses non-autoregressive (NAR) decoding to generate initial hypotheses and then applies autoregression to iteratively refine these hypotheses in parallel. Evaluation on multiple datasets demonstrates that the proposed method, HAI-T, achieves substantial performance gains compared to vanilla CTC in non-autoregressive mode. Furthermore, the proposed semi-autoregressive mode provides greater flexibility in balancing accuracy and speed.", "questions": "* In line 259, the authors should clarify what is meant by [1-8].\n* The proposed semi-autoregressive inference mode generates an initial hypothesis through NAR inference and then refines the hypothesis using the predictor. How is the proposed semi-autoregressive mode different from CTC/RNN-T-based joint decoding? Could the authors expand the discussion in the related work section?\n* Please provide results for the HAI-T model without duration settings in Table 2 for a better understanding across different language settings.\n* Include decoding time results using Viterbi decoding in Table 3 for completeness.\n* Add x and y labels in Figure 2.\n* For deeper understanding, the authors could include results on RTF and emission delays for both the proposed and baseline methods.\n* A realistic comparison would include the proposed semi-autoregressive inference mode against CTC/RNN-T results in joint decoding modes.\n* The results appear promising. However, a detailed breakdown of the Word Error Rate (WER) improvements with semi-autoregressive mode would be beneficial. For instance, are there notable changes in substitutions, deletions, or insertions? Which error type shows the most improvement, or are they all enhanced to a similar degree? This information could provide clearer insights into the impact of semi-autoregressive decoding.\n* It is difficult to understand how the HAI-T training procedure dramatically reduces the occurrence of 0-duration predictions, given that it is trained by stochastically masking out the predictor. Could the authors expand on this phenomenon in the discussion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper introduces a straightforward approach that requires only a single-line change to the existing joint computation code in the Token-and-Duration (TDT) implementation and performs effectively in practice.\n* Evaluations are conducted on multiple ASR corpora, including the AMI test, Earnings22, Gigaspeech test, Librispeech test-clean and test-other, Spgispeech test, Tedlium test, and VoxPopuli test.", "weaknesses": "Although the method is simple, the proposed HAI-T appears to be an incremental improvement over TDT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qr7C08ncLB", "reviewer": "Reviewer_pghP", "summary": "This paper introduces a model called HAI-T, an extension of the Token-and-Duration Transducer (TDT). TDT itself is an extension of RNN-T, where token prediction and duration prediction are decoupled. The authors propose randomly dropping the predictor network of TDT so that the model can operate in both auto-regressive (AR) and non-autoregressive (NAR) modes. They also investigate a semi-AR mode, where the NAR decoding result is refined through AR decoding.", "questions": "Please address the concerns and questions raised in the weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper presents a novel idea of randomly dropping the predictor network to enable the TDT model to function in both AR and NAR modes. Especially, as far as I am aware, this is the first work to evaluate the NAR version of the TDT model.", "weaknesses": "- Insufficient Contributions\n  - The idea of randomly dropping the decoder network is quite simple, making this more of an investigation paper rather than a presentation of a highly novel idea. In this context, rigorous experimental validation is crucial, but the evaluation provided is not sufficient to substantiate their claims (see the next section).\n  - The combination of NAR and AR modes -- or more generally, the combination of a faster, less accurate model with a slower, more accurate model -- has been explored extensively. The method proposed in this paper is relatively straightforward and lacks theoretical novelty. I have listed a few papers as examples, but there are many more relevant works. I suggest explicitly discussing these prior works and highlighting specific differences in methodology or performance.\n    - NAR + AR:\n      - H. Inaguma, et al., “Non-autoregressive end-to-end speech translation with parallel autoregressive rescoring,” 2021.\n      - S. Arora, et al., “Semi-Autoregressive Streaming ASR with Label Context,” ICASSP 2024.\n    - Fast + Slow:\n      - J. Mahadeokar, “Streaming parallel transducer beam search with fast-slow cascaded encoders,” Interspeech, 2022.\n\n- Insufficient Evaluation\n  - The authors claim that the proposed method outperforms RNN-T and TDT in AR mode, but their experimental evidence is not convincing. In the first 7 rows of Table 1, the AR mode results for RNN-T, TDT, and HAI-T (the proposed model) average 7.15%, 7.13%, and 7.10%, respectively. The difference from 7.13% to 7.10% is too minor to definitively demonstrate HAI-T's superiority. Moreover, when examining individual test sets like “ami” and “e22,” the proposed model often underperforms compared to RNN-T and TDT. It is likely that there is no statistically significant difference between TDT and HAI-T in AR mode, and the observed difference may be due to random fluctuation (e.g., the trend could easily reverse with additional testing data). More extensive experiments with statistical significance tests over multiple evaluation sets are suggested.\n  -  Additionally, while the authors show that the HAI-T model performs better with a “stateless” decoder or when excluding 0-duration configurations in the remaining 8 results of Table 1, the same configurations should have been applied to the baseline models for a fair comparison.\n  - The authors claim that their proposed method outperforms CTC in NAR mode, but this is not convincingly demonstrated. In the NAR setting, the HAI-T model slightly outperforms CTC (7.38% vs. 7.19%) but with a slight increase in computational overhead (41 vs. 39 in time). While the increase in time is minor, the improvement in WER is also small, raising concerns that the improvement may simply be due to increased parameters. \n  - Additionally, the paper does not define the term “time,” which presents another issue. Please explicitly define how \"time\" is measured (e.g., wall clock time, CPU time, number of operations) and under what conditions (e.g., hardware specifications, batch size) instead of merely mentioning Huggingface ASR leaderboard.\n  - It would be beneficial to include the “time” metric in Table 3.\n\n- Description Issues\n  - The descriptions of their “code (page 4)” and “algorithm (page 5)” are inadequately defined.\n    - The code on page 4 seems unnecessary. The algorithm is simple enough that additional explanation is not required.\n    - The algorithm on page 5 relies on hidden assumptions known only to the authors. For example, (1) it is unclear what the output of the “joint” function is, and why the second output is omitted, and (2) the application of “dim=-1” in argmax assumes a specific shape of “token-probs,” which remains undefined. Please make the algorithm self-contained by appropriately defining each notion.\n  - The decoding strategy used for NAR-mode is unclear.\n    - From the explanation in Section 4.1, it appears that the authors still applied a decoding algorithm used for TDT even when the model is used in NAR mode. I have several questions:\n      - I assume that the decoding algorithm is still left-to-right. If this is the case, can it still be called \"NAR\"? Please explicitly define the notion of NAR in the paper.\n      - Is there any notion of a 'beam' in the decoding algorithm for HAI-T? This is a clarification question, and I guess the answer is no. However, if beam search decoding was used, the configuration needs to be specified. In addition, if all experiments were conducted without beam search, it is still highly recommended to evaluate each method using the beam search configuration, as it is the most widely used.\n      - What decoding algorithm is used for CTC (best path, prefix search)? Can the author provide the impact of decoding algorithms (not only for CTC, but also for RNN-T with beam size) to understand the impact of it?\n    - In Section 6.2, the authors suddenly introduce Viterbi-based decoding in the experimental section. They should have described the details of the decoding strategies (both the one in the main experiment and Viterbi decoding) in the proposed method section to clearly show the differences between them. It would have been better to explicitly describe Algorithm 2 of Xu et al. (2023) instead of merely referencing it, as the algorithm is not widely known within the community. Including the memory footprint is also highly recommended, as Viterbi decoding may require significantly more memory during inference, especially for long input durations.\n  - The discussion in Section 6.3 is unclear to me. The authors claimed that \"the ability to skip more frames enables the model to learn better representations\" by presenting Table 4 with different max-duration settings, where the shorter max-duration setting provided worse results. However, isn't it just showing that the shorter max-duration is not able to capture the real duration distribution? I think that the only conclusion we could draw from Table 4 is \"the max duration needs to be sufficiently large to represent the real distribution,\" and their claim that \"the ability to skip more frames enables the model to learn better representations\" cannot be drawn.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)