BlockDecoder: Boosting ASR Decoders with Context and Merger Modules
Authors: Darshan Prabhu
Preethi Jyothi
Venue: NeurIPS 2025 poster ()
Keywords: Automatic speech recognition
Attention Encoder Decoder
Efficient Decoder
Primary area: applications
TL;DR: We propose BlockDecoder, a novel ASR decoder architecture that separates textual context building from audio-text integration, achieving a ~2x speed-up over traditional decoders without performance degradation across datasets, languages and tasks.

Abstract: Attention-based encoder decoder models remain a popular choice for state-of-the-art automatic speech recognition (ASR). These models combine a powerful audio encoder that extracts rich acoustic features with a decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe a systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the later layers largely focus on merging acoustic and textual information for the final predictions. Leveraging this key insight, we propose **BlockDecoder**, a novel decoder architecture comprising two distinct components: a text encoder that is purely text-based, and a **Merger** that combines information from the audio encoder and text encoder to generate output tokens. Unlike traditional decoders, the **Merger** autoregressively predicts a sequence of K tokens within a *block* of size K, while relying on the same precomputed contextual information from both text and audio encoders across the block. This design choice allows for the efficient reuse of encoder representations. The separation of the decoder into the text encoder and the **Merger** promotes modularity and more flexible control of parameters via the number of text encoder and **Merger** layers. As a result, **BlockDecoder** yields a significant speedup ($\sim2$x) compared to traditional decoders, across diverse datasets, languages, and speech tasks, without any degradation in performance. The code is available at https://github.com/csalt-research/blockdecoder.

BLOCKDECODER; Boosting ASR Decoders with Context and Merger Modules
Darshan Prabhu Department of CSE ITT Bombay darshanp@cse iitb. ac
Preethi Jyothi Department of CSE IIT Bombay Pjyothi@cse witb. in
Abstract
Attention- based encoder decoder models remain popular choice for state-of-the-arr automatic speech recognition (ASR). These models combine a powerful audio encoder that extracts rich acoustic features with decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the later layers largely focus on merging acoustic and textual information for the final predictions. Leveraging this key insight, we propose BLOCKDECODER, a novel decoder architecture comprising two distinct components: a text encoder that is purely text-based, and a MERGER that combines information from the audio encoder and text encoder to generate output tokens. Unlike traditional decoders, the MERGER autoregressively predicts a sequence of K tokens within block of size K, while relying on the same precomputed contextual information from both text and audio encoders across the block. This design choice allows for the efficient reuse of encoder representations. The separation of the decoder into the text encoder and the MERGER promotes modularity and more flexible control of parameters via the number of text encoder and MERGER layers. As a result, BLOCK BECCKDECOO DECODER yields significant speedup (~~ 2x) compared to traditional decoders, across diverse datasets, languages, and speech tasks, without any degradation in performance. The code is available at/htps:: htt///ggtttbbbt cwmmcsaltttttccccccooccccc.cccce cswltssaattoe..eaac www.rrrrcc/ollcceceeoooeeooooooe
Introduction
Several prominent sate--of-thearr automatic speech recognition (ASR) systems are derived from attention-based encoder-decoder architectures 1 While the audio encoder's role these architectures is to transform input speech into acoustically-rich representations, the decoder autoregressively generates text by combining acoustic information from the encoder with previously predicted text. Numerous variants of the encoder have been explored for ASR in prior work, ranging from self-supervised architectures k to convolutionally-enrichee ASR pipelines 6 In contrast, there have been relatively fewer innovations of the ASR decoder.
The ASR decoder comprises Transformer { layers with both self self-attention and cross- cross-attention mod- ules. The self-attention module attends to previous tokens at given time-step, and the cross-attention module attends to speech representations from the audio encoder. We analyze the behaviour of both these attention modules across all decoder layers and make the following key observations: (1) Self-ar- tention increasingly focuses on local context as we progress deeper into the decoder: As shown in Fig- ure I the attention weights in the initial decoder layers are well-distributed across the entire sequence.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Self Atteettoon
Cross Attention
1xyer
1ayw?
Layer
1ayer
Tayer
Jayer 6
Figure 1: Attention plots illustrating the patterns learned by self-attention and cross-attention blocks across all layers of a standard Transformer decoder a hybrid CTClattention ASR system. The plot consists of two rows, one for self-attention and another for cross-attens attention and six columns, corresponding to the six decoder layers (ordered left to right). Brighter colors indicate higher attention weights. These visualizations 960h dataset; we see similar patterns across most of the examples.
generated for single example from model trained on the Librispeech
However, these weights become strongly diagonal as we approach the penultimate layers. similar phenomenon has been previously observed in the audio encoder [5] but never analyzed in the decoder. (2) Cross-attention blocks appear to be less effective in the initial decoder layers: As illustrated in Fig- ure cross-attention in the early layers fail to learn any alignment between the audio and text sequences suggesting they might be redundant in these layers. Figure 6 in Appendix D additionally shows very similar patterns as illustrated in Figure
aggregate attention plots over 250 utterances;
Based on these systematic attention patterns, the decoder appears to be assigning specific roles to its initial and final layers. Specifically, the initial layers of the decoder are mainly tasked with the generation of text-only context by relying extensively on the token sequence, thus rendering cross-attention less effective. The final layers of the decoder merge textual context from the early layers with acoustic information from the audio encoder for its final predictions. We draw inspiration from these findings and propose BLOCK as alternative to the conventional decoder architecture. BLOCK
partitioned into two sub-modules, text encoder with only self-attention layers to build textual context and a merger with self-attention and two cross cooss-attentiont layers to integrate acoustic representations from the audio encoder with contextualized outputs from the text encoder to generate the final predictions. Unlike traditional decoders, the merger is designed to autoregressively generate K tokens (referred to as block of size K, and hence the name DECODDEEC)E) from every position in the token sequence, while relying the contextual information from both encoders across the block. This block-based design enables different inference strategies, as discussed in section 3.3 and reduces inference latency. Since the merger combines representations that are already contextualized by the audio and text encoders, a small number of merger layers suffice to achieve high performance, thus resulting in additional latency reductions. Together, these complementary design choices result in B1.OCKDECODS containing fewer parameters compared to the traditional decoder, while achieving double the inference speed with no performance degradation on multiple speech tasks.
Related Work
ASR systems have significantly evolved in the last decade from deep neural network-hidden Markov model (DNN-HMM) based architectures (9] 10 to fully end-to-end (E2F) ASR systems [T] 12
These E2E models typically adopt either an encoder-only [3] or an encoder-decoder architecture where the encoder processes the audio while the decoder handles text generation. Prior work has
largely focused on improving these architectures either by refining individual components [4 [5] or joint developing joint training techniques that integrate multiple objeecive to enhance model robustness [6].
One such videly-adopted jointly trained model is the CTC+attention system [17], which consists of an audio encoder, an autoregressive decoder and a CTC module. Improvements to hybrid ASR have largely focused on either making the encoder more expressive
4 [8] enhancing the robustness of CTC [19] or introducing new training objectives (20] 21). While many efforts have aimed towards refining the audio encoder and CTC, comparatively less attention has been given the decoder despite its critical role in text generation. BLOCKDECODER is a step towards addressing this gap.

---

Kargez
B0X
Feod Fanward
Postiormes
NN
Norm
CTC
Merger (K-3)
Merger (K=3)
Marger (K-3)
Text Ba odde
KHGAA2
Layer
Layer Norm
Food Fonward
MHCA-1
Norm
Audio Enoocer
M
MHSA
Convolution Subsampligg
Gf
Layer Norm
LayerNom
Text Emooder (12R)
Fbank generator
Layar
Embooding Layer
(b)
Figure 2: Overview of the Hybrid CTC/Attention framework with our proposed BLOCKDECODER (a) Schematic representation highlighting interactions between the audio encoder, text encoder, CTC. and MERGER modules during training. For the example transcript BLOCK DECODER", we illustrate the MERGER'S capability of predicting K =3 tokens autoregressively from every position of the token sequence, while being conditioned on the same contextual representations provided by the audio encoder and text encoder. Specifically, at each stage, the MERGER receives the representation of progressively longer text prefix (e.g., [<sos>], [<sos>, B] etc.) from text encoder (shown via the blue bars) along with the corresponding audio representation from audio encoder (shown via orange lines), and autoregressively predicts the next three tokens. (b) A detailed expansion of the text encoder and MERGER modules, with residual connections and dropout omitted for brevity. Here, MHSA=Multi- headed Self-Attention and MHCA=Mult--eadded Cross-AAtention.
The autoregressive nature of the decoder in ASR systems contributes to high inference latency. To mitigate this, non-auuoreegressve (NAR) decoders have been explored [22 23 24), allowing parallel token prediction thereby improving inference efficiency. However, like CTC, NAR decoders generally underperform compared to autoregressive decoders. To bridge this gap, hybrid approaches have been proposed, where an NAR decoder first generates an initial prediction, which is then further refined by an autoregressive decoder [25]. Another line of work investigates speculative decoding techniques 26 (7], that predicts multiple future tokens in parallel, thereby further reducing inference costs. While most prior works have focused eliminating autoregressive decoding modifying decoding strategies (28) to the best of our knowledge, there have been no detailed investigations of the internal functioning of the decoder refine its design. In our work, we redesign the decoder based on consistent empirical insights drawn from attention patterns. These design choices are similar in motivation to -NN-Transducer (RNN-T) systems used for streaming ASR (29], which employs separate predictor and joiner modules, each serving distinct function. However, unlike the joiner in RNN-T, which is simple feedforward network, we propose the MERGER, that incorporates multiple attention layers, enabling richer and more flexible context integration. Furthermore, we note that our approach operates on blocks of tokens, which an emerging design choice in recent years (30 31 (24.
Methodology
3.1 Overall Architecture
BLOCKDECODER scaffolds on top of hybrid CTC/attention models, a state-of-the-ar framework for ASR systems (17). This framework consists of: shared audio encoder, an autoregressive decoder and a CTC decoder. Given an input speech sequence = {x ......TTTT ,, E IRd, the audio encoder maps contextualized speech representation H= [ //......h hr Both the CTC and autoregressive decoders are conditioned on H to produce the label sequence y = Ui....wwll. E V where V is a predefined token vocabulary. The CTC branch minimizes the Connectionist Temporal Classification (CTC) loss 32) by marginalizing over all possible alignments of with H. decoder, on the other hand, is
audio encoder typically uses subsampling to reduce the length of the speech sequence to T = [2"/S] where S is the sampling factor.
3

---

an attention based autoregressive module that is trained to maximize the conditional likelihood of producing Y4 given H and previous tokens Vo.-..ui-1. where Yo denotes the <sos>> token. In this work, we focus on the decoder by replacing it with our proposed Br.OKKDFCOME which is more efficient and performs at par with (or better than) the decoder.
3.2 BLOCKDECODER
We replace the standard decoder with specialized modules: text encoder and MERGER, each designed for a specific purpose, as shown in Figure The text encoder generates rich textual context for the token sequence y, that the MERGER further combines along with H, to autoregressively generate K tokens within block. These two modules are collectively referred to as BroCKDFCODDE shows a schematic overview of our ASR system incorporating these modules. If the original decoder comprises N layers, then we redistribute these layers (with minor modifications) across the text encoder (in (32.1) and MERGER (in (.2.2) BLOCKDECODE to be commensurate size with the decoder.
Fig. (a
3.2.1 Text Encoder
The text encoder is stack of Nr decoder layers that attends over the token sequence y to produce contextualized representations. These layers consist only of self-attention and feed-forward sub-layers (and cross-attention),, since text encoder operates only on the text and does not attend to the audio at all. To prevent information leakage, each token y; is allowed to attend only to itself and the preceding tokens (yo-V) -V2..... using appropriate masking. The architecture of the text encoder is shown in Figure 2b It consists of a stack of Nr decoder layers that are designed to generate rich textual
context for the token sequence while preventing information leakage. Specifically, the output C of the jth text encoder layer is calculated follows:
MHSA(C C c ,askk
C + LayerNorm...
Č Ć' (Linear
C LayerNormmaa
where č, c' E R xd and mask [alue) Wxw Here, MHSA(*) denotes multi-heade self-attention R. During attention computation, mask serves as a Boolean matrix, where mask ba indicates whether the th token is allowed to attend to the 1th token in y. For the text encoder, we set mask ma (b S a) to prevent information leakage. Finally, the input to the first layer is defined as C PE + (Embedding [yo. ./wll,, where PE represents sinusoidal position embeddings. observe that having Layer at the end of every layer is crucial for stable training of the BLOCKDECODER. We use E ig xd to denote the final output from the text encoder.
We also employ dropout [33] in every text encoder layer. Additionally,
3.2.2 MERGER
Given audio context H from the audio encoder and text-only context C from the text encoder, we introduce MERGER that jointly attends to contexts from both modalities to autoregressively generate block of K tokens. MERGER achieves this integration via multiple attention blocks, as shown in Figure 2b Specifically, MERGER first employs self-attention block, that focuses on local context by causally attending attendigg to tokens in the current block. Next, MERGER uses two cross-attention modules for C and H, respectively. The first cross-attention module attends to the contextualized representations in C prior to the current block; the second cross-attention block integrates acoustic features from H, thereby ensuring that the tokens generated by MERGER remain faithful to the audio. At the 7th time step. MERGER trained to maximize the probability of generating the token sequence (y4.....4KK conditioned on H, C and the previous token 1/-1i
K-1 II
P([y, 1 V--1, H, Cos-1)
MERGER ((UGE ... 11+ H, Co:s-1) k=
important to note that at time-step i, while generating each of tokens block, MERGER conditions the same first entries of C written as Co:-1- Since MERGER predict block of

---

K future tokens at each time step, we note that the probability for any token in the block can now be computed using multiple combinations of entries from C and the set of preceding tokens. For example. with a block size of 3. the probability of ys can be computed using the following combinations: (u44.Co:al (Ly3, y/ Co:3) and (112, Ma. ya) Co:2). Section 3.3 elaborates on multiple inference strategies supported by BLOOCDDEODDER.
To facilitate efficient attention and loss computations, we create a modified token sequence x ( - K +).. This
- K sequence contains (W-K+D +K+ blocks, each comprising K tokens, that are color-coded for improved readability. (Since the length of y grows linearly with K, we typically employ small values for K.) This modified sequence now enables us to restrict self- and cross-attention by appropriately configuring the attention masks. Specifically, the output of the jth layer M from MERGER (consisting of NM layers overall) is computed as:
y' [1 !/K -1 1,3/133/2,... "...,,.......................... sw", K+ :.W: of length y
M' MHSA (M .M .M--- mask
M- M LayerNorm
M MHCA CA (M. M,C.C.C. C maakoonnex)
M M + LayerNorm contec) (M*
M' M Layer andio austo (MHCA (M,,.H)
M M La (M')
where M" .MM ER M*SA(*) and M*CA(+) denote the standard multi-headed self- and cross-
attention 8 and masksell mask.cuntext E {0.1) WXW are Boolean matrices where mask, (a. b] = 1 indicates that the ah item in one sequence is allowed to attend to the 6th item in another sequence. To constrain self-attention to causally attend over tokens within the block, we set mask [a,b] = (b- < a) & ( - L To prevent information leakage from C. we set mask [0.bb] 1 Lastly, we use embeddings for tokens within each block of the sequence y with appropriate positional embed- dings as input to MERGER. Finally, the output probabilities from MERGER are generated using sim- ple feed-forwel layer to project to the token vocabulary size V, followed by a softmax transformation.
To ensure that C is contextually rich, we allocate the majority of our layer budget to the text encoder. With H and C being substantially rich in information, even a small number of MERGER layers are adequate to effectively combine both modalities. This careful division of responsibilities between a coss-attentionn-ree text encoder (that generates contextualized textual representations) and a lightweight MERGER (that allows late integration of speech and text information) enables BL.CKK to function accurately and significantly faster than the traditional decoder.
3.3 Inference Strategies
During inference, we we employ label-synchrooous autoregressive beam search proposed by 17 to find the best hypothesis. That is, at the 7th decoding step. the score for the partially decoded hypothesis ø is computed as a weighted combination of the log probabilities from both CTC and BLOCKDECODER, Specifically:
(1) where 8 E [0.1] is hyperparameter that determines the relative importance of the two scores Sexe and San from the CTC and BLOCKDECODES modules, respectively. Snn can be computed using various combinations of inputs to the text encoder and MERGER. We explore three inference strategies to compute the score for the partially decoded hypothesis Ug,, as illustrated in Figure 3
S(gg))= Sen. (1g:)++++1111)1)) San (ūgs)
3.3.1 Strategy 1: Naive Block Decoding
The simplest approach involves making a forward pass through both text encoder and MERGER at every decoding step, by utilizing all positions within the block. Specifically, the attention score for Uz is computed as:
(ūs:)) MERGER (U) j=1
5

---

Toxt Erd
Merger
Tost End
H
Merger
Cacas
razes
Feod End
Text End
Marrger
Feot Enc
Merger
Merger
DCX
DOX
BCX
+
CCK
+
boX
Feed End
Maegge
roor
Iest Enc
H
Merger
Text Enc
H
Merger
Text Eno
Merger
casan
caass
caur
Tho Enc
Merger
Text Ern
Merger
Text Enc
Merger
OCK
DCX
Figure 3: Overview of the three inference strategies across the first four beam search steps. "Text Enc" refers the text encoder. From left to right: (1) Naive Block Decoding: Score obtained from only the last position within the block, requiring one forward pass through both the text encoder and MERGER each step. (2) Iterative Block Decoding: Score computed iteratively within the block, requiring one forward pass through MERGER at each step, and one pass through text encoder every K steps. (3) Full Block Averaging: Score averaged across all valid input combinations to text encoder and MERGER, requiring one forward pass through text encoder and multiple passes through MER GER at each step.
where C' = text encoder( (30..1....jjKK' 1). This approach always utilizes the last position within the block while generating the score. This approach sub-optimal involves forward passes through both text encoder and MERGER at every decoding step.
3.3.2 Strategy 2: Iterative Block Decoding
To truly exploit the MERGER's blockwise ability predict K tokens, we adopt iterative block decoding approach, where the MERGER continuously predicts K tokens before performing a forward pass through the text encoder update the context vector C'. Here, the score for Úz computed as:
San (in(
II 3=1
where C' text encoder( (Vo. }) and s= ا layers and we make a forward pass through text encoder once every K steps, this strategy is more efficient compared to the previous strategy.
x K. Since the MERGER contains only a few
3.3.3 Strategy 3: Full Block Averaging
This last approach aims to boost performance by utilizing the scores from all the positions within the block. The final score for Ú computed as the average of the scores from all the combinations of C' and the preceding tokens as shown below:
San (ū = log
11 M
where C' = text encoder( (rrcoddr((1o.))....//p 1). While this strategy improves performance, it incurs higher inference latency due K invocations of MERGER as opposed to single invocation in the previous strategy.
3.4 Complexity of BL.OCDDRCODER
We analyze the computational complexity of BLOCKDECODER in comparison to the stan- dard decoder. Let T be the length of the acoustic features H, denote the length the token sequence y, K be the block size for the MERGER and denote the dimensionality for attention computations. (For ASR, typically & T.) BLOCKDECODER consists of Ny and NM layers in text encoder and MERGER, respectively. (Recall that N Nr + Nas.)

---

Table 1: Performance comparisons (CER WER %) between BLOCK and the baseline Transformer decoder across three datasets and two encoder architectures. indicates no statistically significant WER difference compared to the baseline, determined by the MAPSSWE test [4). denotes significant improvement in RTF, while indicates RTF performance comparable to the baseline.
Librispoech (WER)
Tedlium2 (WER)
AISHELI
Method
Test Clon
Oeber
ParamecM))
Text
Yert
Conformer [
w/Trrantoomer Decoder
34.2
17.74
1.38
30.8
7.69
2.16
33.6
4.58
0.44
w/ BLOCKDECODER
Naive Black Decoding
33.7
6,63
17.63
-.73(-1
30.2
7.81
33.1
4.76
0 0.28
-Teeativve Decoding
0.66
33.1
(-25
Full Block Averaging
33.7
6.58
17.62
1.68
30.2
7.79
2.39
33.1
4.63
0.51
E- Branchhfomeer G
T'Trnnsfommrr Decoder
38.5
1.52
35.0
37,9
w/ BLOCKDECODDR
Naive Block Decoding
37.9
6.15
16.52
(.77
34.5
7.61
37.4
Decoding
37.9
0.67
34.5
0.91
37.4
0.36
0.300
Full Block Averaging
37.9
6.14
16.85
1.68
34,5
7.61
2.40
37.4
4,53
0,54
We focus primarily on comparing the total number of attention calculations required for single example during training and inference.
Table 2: WERs of BLOCK and other well-known ASR systems (from prior work) on Librispeech 960h.
Without LM
Method
Params (M)
Tixi Cliea
Test Other
RTF
During training, the text encoder takes y its input while the MERGER uses the modified sequence y as its input, where ly' (W K + 1)K WK. Thus, the overall attention computation for BLOCKDECODER has a complexity of O(Nr(W2d) + Nas (WK2 W2K WKT)d) in comparison to decoder that takes (NN(22 + aT time. Upon fur- ther simplification (detailed in Ap- pendix A. we show that choosing K 2NM allows DECDDR
Transducer
Transformer
139
2.4
5.6
ContextNet 14
112.7
2.1
4.6
Conformer (M)
30.7
2.3
5.0
Conformer (L)
118.8
2.1
4.3
CTC
QuartzNet (L)
35
19
3.9
11.3
Hybrid CTC+Attention
Transformer 36
270
2.9
7.0
Conformer 37
116.2
2.9
7.0
Conformer (L)
6
147.8
2.2
4.7
Branchformer
116.2
2.4
5.5
Branchformer (L) 6
146.7
2.2
E-Branchformer
148.9
2.1
Our baselines (Hybrid)
to operate with approximately the same number of attention operations as decoder.
E-Branchformer
w/ Transformer Decoder
2.1
7.734
Our work (Hybrid) w/ BLOCKDDDDDDER
146.8
2.1
(-266)
During inference with beam-search (of B beam width), the standard decoder takes O(W B(NW + d) time for its attention calculations. In contrast, BLOCKDECODER with naive block decoding takes D(WB(NW NN +T)) time. Since K T and Nar & N, BLOCK- DECODER requires significantly fewer attention operations than decoder. The time complexity of iterative block decoding is similar to that of the naive strategy, but benefits from making forward pass through text encoder only once every K steps thus leading to slightly faster inference. The complexity of full block averaging and other details are in Appendix A
Table 3: Comparison (accuracy % and F1) of our system
against Transformer decoder baseline on the SLU task.
Params (M)
Intent Test Acc.
Entity SLU-FIT
Method
RTF
Conformer 4
w/ Transf, Decoder
109.5
85.9
0.76
2,46
BLoCKDEOODD
107.4
88.9
0.77
1.17(2211)
E-Branchformer 6
Transf. Decoder
110.2
87.2
0.78
3.11
w/ BLocKDECODER
108.1
87.1
0.78
1.31 1.3
4 Experiments
4.1 Experimental Setup
Datasets. We show experiments two tasks: ASR and Spoken Language Understanding (SLU). For ASR, we use: (1) Librispeech (38) consisting of 1000 hours of English read audiobooks with 100-hour and 960-hour training splits, (2) Tedlium2 (39] consisting of 200 hours of TED talk recordings,

---

Table CERs/WERs of BLOCK and the baseline decoder across five languages from the MCV corpus. All experiments use the E- Branchformer encoder. indicates the lan- guages for which CER is reported.
Table WERs of BLOCK with varying block sizes on Librispeech 100h using two training
strategies: (1) full: MERGER trained on every block (2) sampled: MERGER trained on random
subset of blocks.
Chinese
Language
Decoder used
Transformer
Params
51.1M
Dev
14,0
Test
13.6
RTF
0.41
Block size
Training Strateggy
Training (hes)
Clean"
Test
Test Other
RTF I
Ours
50.5M
14.2
13.8
0.22
0.23
Rastline
15.4
6,4
17.0
1:52
Czech
Ours
46.9M
12.7
124
13.4
13.8
0.25
0.25
K ==3
K
full
full
15.6
6.2
17.5
16.9
(273
0.67.-330
Italian
Tranffomer
47.4M
9.5
10.1
K-5
full
162
6.3
17.1
0.66--300
Ours
46.9M
9.7
10.5
o-200
sampled
15.4
6.4
17.7
066
Japanese
Transformer
49.9M
1.3
12.1
0143
K ==7
full
17.0
6.2
16.9
0.65--300
Ours
49AM
5.3
12.3
sampled
0:6
Tamil
Transformer
47.4M
17.4
20.0
0.57
K K=9
full
17.5
6.5
17.4
0.68/2290
46.9M
17.5
20.1
sampled
29.6
0-68
(3) Aishell [40] containing 170 hours of Mandarin Chinese speech data, and (4) Mozilla Common- Voice [1], multilingual dataset with durations ranging from 10 to 2500 hours per language. We select five languages with training data spanning 100 to 400 hours. For SLU, we use the SLURP corpus [42), a 60-hour multi-domain English dataset evaluated for intent classification and entity recognition.
Implementation Details. All our experiments are conducted using the ESPnet toolkit 4 on NVIDIA A100 and A6000 GPUs Across all experiments, we apply 3-way speed perturbation with ratios {0.9, 1.1}, along with SpecAugment [44 Our experimental setup follows the recommended configurations in ESPnet recipes. Across all experiments, we employ standard efficient inference techniques such KV-caching and Automatic Mixed Precision (AMP). For SLU experiments, we first train the model with an ASR objective, where the output label sequences are sentences with intent and entity-related tags. Then, during inference, we first decode the sequence as in ASR, and then compute SLU metrics by parsing the decoded output. A detailed summary of the hyperparameters used for each dataset is available in Appendix F Real-time factor (RTF) P values are reported using CPU inference for all experiments. Finally, unless explicitly stated, all BLOCKDECODER related experiments use a block size of K 3=3, Ny 4 text encoder layers, and NM =2 MERGER layers.
4.2 Main Results
Table [ compares the word error rates (WERs) of BLOCKDDECDDER with standard Trans- former decoder (referred to as baseline) across three widely used ASR tasks: Librispeech 100h and Tedlium2 for English, and Aishell for Mandarin. For each dataset, we experiment with two state-of-the-art encoder architectures: Conformer 4 and E-Branchformer [ and report results on all three inference strategies outlined in Section We find that BLOCKDECODER consistently matches or outperforms the baselines despite having fewer parameters and achieves significant latency gains. The first two inference strategies are fast, achieving a near 2x speedup in RTF by effectively utilizing the third strategy does not improve RTF but yields slight performance boost. Iterative block decoding yields the best RTF gains and will be used in all subsequent experiments (unless specified otherwise).
the block structure while
In Table 2, we further compare the WERs between BLOCKDECODER and decoder on the full Lib- rispeech 960-hour dataset, along with other prominent baselines from prior work. Even on a large- scale dataset, BLOCKDECODER remains comparable in performance with the strongest baseline, E- B-Branchformer, while achieving nearly a 2.5x speedup in RTF on CPUs. Additionally, since inference on Librispeech 960h often employs large beam size. Appendix [ we also compare the performance of BLoCkDEcoDD against the decoder with inference using GPUs. Even in this setting, BLOCKDE- CODER achieves better RTF and FLOPs (Floating Point Operations) compared the standard decoder.
4.3 More Task and Language Experiments
Non-English ASR. In Table we show additional experiments on five languages from diverse language families of the MCV corpus (Chinese, Czech, Italian, Japanese, Tamil). BLOCKDECODERR
We ensure that all experiments for a particular dataset are conducted on the same GPU and environment. RTF is the ratio between the time taken by the model to process the input and the actual input duration.
8

---

consistently yields WERs comparable to the baseline across all languages while maintaining significant latency gains.
SLU task. In T we show results on an SLU task, SLURP 42
and evaluate using accuracy and Fl-score on intent classification and entity recognition, respectively. As with ASR, BLOCKDECODER for SLU also results in comparable performance to the bascline and significantly improves latency.
4.4 Ablations and Analysis
Attention plots. Figure shows the attention patterns learned by the text encoder and MERGER trained on Librispeech 960h. Figure 4a shows self-attention plots of the text encoder with well-distributed attention distributions across the sequence for all layers, indicating that the text encoder focuses tion plots for the two cross-attention blocks of the MERGER. The first cross-attention block that attends to text encoder's output is predominantly diagonal, reinforcing the idea that the global context produced by the text encoder is sufficiently informative for the MERGER. The second tross-attention block, that attends to the audio encoder's output, demonstrates accurate alignment with the audio features, thus highlighting the importance of these blocks in inte- grating contexts.
global, text-only context. Figures 4b and 4c show atten-
Impact of the number of MERGER layers.
(a)
the effect of varying the number of MERGER layers (N)) in the BLOCKDECODER. We see that single MERGER layer (i.e. NM = 1) is sufficient to achieve rea- sonable WERs while yielding signif- icant latency gain with a nearly 3.5x speedup. Adding a second layer (i.e NM = matches the baseline in WER, while still maintaining 2x latency gain. As the number of MERGER lay- ers increases, performance continues to improve, while consistently achiev- ing better RTF than the baseline.
(b)
Figure 4: Attention plots demonstrating the patterns learned by self- and cross-attentions from BLOCKDE- CODER. (a) text encoder's self-attention (b) MERGER'S cross-attention text encoder (c) MERGER'S cross-attention w/ audio encoder.
Effect number merger layers on BlockDecoder
Impact of block size K. Table presents comparison of BLOCKDE- CODER'S ' performance on Librispeech 100h for varying block sizes K 7,9.5 As larger K increases training time, we also explore an al- ternative training strategy in which the MERGER trained on only a ran-
PatchDecoder varying layers
18.0
Decoder
Word Enror Rate (WER)
17.5
Ny
16.5
dom subset ly K blocks such that the total number of tokens roughly equivalent in length to the original input y (marked sampled). All non-sampled variants achieve perfor- mance comparable to the baseline while demonstrating substantial RTF improvements. RTF gains initially increase with K since iterative block decoding requires fewer forward passes through text encoder. The gains subsequently plateau with larger K values due to the increased computational overhead of the MERGER that offsets latency benefits gained from skipping text encoder. Additionally, as K increases, we see slight degradation in performance, due to the increased complexity of the MERGER'S 's learning objective. All sampling-basee
0.55
1.15
1.55
Fector (RTF)
Comparison of the performance of BLOCKDE- CODER on changing number of MERGER layers. WERs are from the test-other split of Librispeech 100h.
Figure 5:

---

experiments underperform relative to their non- sampled counterparts, likely due to insufficient training, but match the baseline terms of training time.
BLOCKDECODER versus CTC only models. CTC-only models are a popular alternative to encoder- decoder architectures due to their fast inference enabled by greedy decoding. However, they are known to significantly underperform compared to encoder-decodee models. Hybrid CTC encoder-decoder architectures a popular choice to achieve state-of-the-arr performance; hence, we primarily focus on this framework. Further detailed performance comparison between BLoCKDEccDDER and multiple CTC-only models (that focus efficiency) are presented in Appendix C
Decoder without cross-attention layers. We also investigate a simplified variant of BLoCK CODER, which retains the decoder architecture but removes vross-attnntion blocks from the initial layers. This modification results in worse performance compared to our proposed BLOCKDEcODERR Further details are in Appendix
Adaptation of BLOCKDECODER to other settings. Our proposed architecture is particularly effective in settings where the encoder output substantially longer than the target sequence (e.g.. ASR). Similar input-output dynamics exist in tasks such as document summarization, which often involves long input sequences and shorter outputs, making them promising candidates for adaptations DECCKDDDDD Moreover, since BLOOK separates the decoder into text-only encoder and merger module, our framework naturally supports replacing the text encoder with a pretrained LLM, thereby enabling seamless ASR-LLM integration. Finally, the architectural modularity introduce could inspire efficient pruning or compression strategies for large-scale models such as Whisper (1), thereby accelerating inference, especially for real-time applications.
5 Conclusion
In this work, propose BLOOCDDECODER, novel decoder architecture for ASR inspired by attention patterns observed in traditional Transformer decoders, specifically that cross-attention is under- under-utilized in initial layers and self-attention becomes more localized in later layers. BLOCKDECODER consists of a text encoder, tasked with generating rich text-only context and a MERGER, responsible for efficiently integrating audio and text contexts autoregressively produce K tokens within block. Together, these modules ensure that BLOCK DECODER is on-par with (or better than) the standard decoder, while achieving significant RTF gains across a wide range of languages and tasks. The block structure within the MERGER opens up avenues for further investigations into inference methods, optimizations, and additional performance enhancements.
Acknowledgments
The authors would like to thank all the anonymous reviewers and chairs for their valuable feedback and constructive suggestions, resulting in significant improvements the quality of this submission.
References
[1] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision, 2022.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2244-112460 Curran Associates, Inc., 2020.
[3] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 1.21.
10

---

[4] Anmol Gulati, James Qin. Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution -uggeonnde transformer for speech recognition. In Interspeech 2020, pages 5036-5040, 2020.
[5] Yifan Peng. Siddharth Dalmia, lan Lane, and Shinji Watanabe. Branchformer: Parallel MLP- attention architectures to capture local and global context for speech recognition and understand- ing. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 17627-17643, PMLR, 17-23 Jul 2022.
[6] Kwangyoun Kim, Felix Wu, Yifan Peng, Jing Pan, Prashant Sridhar, Kyu J. Han, and Shinji Watanabe. E-branchformer: Branchformer with enhanced merging for speech recognition. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 84 91, 2023.
[7] Darshan Prabhu, Yifan Peng, Preethi Jyothi, and Shinji Watanabe. folrreconfformrr: Extending conformer with multiple convolution kernels. In Interspeech 2024, pages 232-236, 2024.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
[9] Frank Seide, Gang Li, and Dong Yu. Conversational speech transcription using context-dependent deep neural networks. In Interspeech 20 pages 437 437-440, 201
[10] Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):22-77, 2012.
[11] Jinyu Li. Recent advances in end-to-end automatic speech recognition. APSIPA Transactions on Signal and Information Processing, ((1))=22023
[12] Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, and Shinji Watanabe. End-to- end speech recognition: A survey, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:325-355 2,,,--35,,0222
[13] Yifan Peng. Yui Sudo, Muhammad Shakeel, and Shinji Watanabe. OWSM-CT An open encoder- only speech foundation model for speech recognition, translation, and language identification. In Lei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10192-10209, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[14] Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang. and Yonghui Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. In Interspeech 2020, pages 3610-3614, 2624,0020...... 2020.
[15] Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. SpeechUT: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1663-1676, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[16] Yui Sudo, Shakeel Muhammad, Brian Yan, Jiatong Shi, and Shinji Watanabe. 4d asr: Joint modeling of ctc, attention, transducer, and mask mansk-predict decoders. In Interspeech 2023, pages 3312-3316, 2023.
[17) Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey, and Tomoki Hayashi. Hybrid ctclattention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 113): 2017.
[18] Zengwei Yao, Liyong Guo, Xiaoyu Yang, Wei Kang, Fangjun Kuang, Yifan Yang, Zengrui Jin, Long Lin, and Daniel Povey. Zipformer: A faster and better encoder for automatic speech recognition. In The Twelfth International Conference on Learning Representattinns 2024.
11

---

[19] Hairong Liu, Zhenyao Zhu, Xiangang Li, and Sanjeev Satheesh. Gram-CTT Automatic unit selection and target decomposition for sequence labelling. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2188 2197.. PMLR, 06-11 Aug 2017.
[20] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming and-to-end speech recognition with mnn-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 193-199, 2017.
[21] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833, 2020.
[22] Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi. Mask ctc: Non- aon-autoregressvee end-to-end asr with etc and mask predict. In Interspeech 2020, pages 3655-3659, 2020.
[23] Ethan A. Chi, Julian Salazar, and Katrin Kirchhoff. Align-refine: Non-autoregressive speech recognition via iterative realignment. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for C Computational Linguistics: Human Language Technologies, pages 1920-1927, Online, June 2021. Association for Computational Linguistics.
[24] Tianzi Wang, Xurong Xie, Zhaoqing Li, Shoukang Hu, Zengrui Jin, Jiajun Deng, Mingyu Cui, Shujie Hu, Mengzhe Geng, Guinan Li, Helen Meng. and Xunying Liu. Towards effective and efficient von-auvoreeressive decoding using block-based attention mask. In Interspeech 2024, pages 262-266. 2024.
[25] Zhengkun Tian, Jiangyan Yi, Jianhua Tao, Shuai Zhang, and Zhengqi Wen. Hybrid autoregressive and non-autoregressive transformer models for speech recognition. IEEE Signal Processing Letters, 29:762-766, 2022.
[26] Yael medusa's i ear: Multi-head efficient decoding for sunsformer-basee asr, 2024.
Aviv Shamsian, Aviv Navon, Gill Hetz, and Joseph Keshet. Whisper in
[27] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple 11m inference acceleration framework with multiple decoding heads. In Proceedings of the s International Conference on Machine Learning, ICML'24. JMLR.org. 2025.
(28) Niko Moritz, Takaaki Hori, and Jonathan Le Roux. Triggered attention for end-to-end speech recognition. ICASSP 2019 2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 5666-56770 2019.
[29] Yanzhang Hc, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby, Shuo-yiin Chang, Kanishka Rao, and Alexander Gruenstein. Streaming end-to-end speech recognition for mobile devices. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6381 6381-6334 13855, 2019.
[30] LILI YU, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting million-byte sequences with multiscale transformers. In Thirty-sevent Conference on Neural Informatinn Processing Systems, 2023.
[31] Namgyu Ho, Sangmin Bae, Taehyeon Kim, hyunjik.jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Young Yun. Block transformer: Global-to-local language modeling for fast inference. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

[32]
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, page 369-376, New York, NY, USA, 2006. Association for Computing Machinery.
[33] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, (5(56):1999-11
2014.
[34] L. Gillick and S.J. Cox. Some statistical issues in the comparison of speech recognition algorithms. In International Conference on Acoustics, Speech, and Signal Processing, pages 532- 535 vol.1,
1989.
[35] Samuel Kriman, Stanislav Beliaev, Boris Ginsburg, Jocelyn Huang, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, and Yang Zhang. Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6124 6128, 2020.
[36] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end ASR: from supervised to semi-supervised learning with modern architectures. In /CML 2020 Workshop on Self-supervision in Audio and Speech, 2020.
[37] Pengeheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li. Daniel Garcia-Romers Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuckai Zhang. Recent developments on espnet toolkit boosted by conformer. In ICASSP 202/ 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5874-5878, 2021.
[38] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210 2015.
[39] Anthony Rousseau, Paul Deléglise, and Yannick Estève. Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck. Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 2014. European I Language Resources Association (ELRA).
Reykjavik, Iceland, May
[40] Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An spen-source mandarin speech corpus and speech recognition baseline. In 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O- (O-COCOSDA), pages 1 1-5, 2017.
[41] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers. and Gregor Weber. Common voice: A massively- muliiillaamul speech corpus. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Mae- gaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Lnguage Resources and Evaluation Conference, pages 2218-4222, Marseille, France, May 2020. European Language Resources Association
[42] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural I Language Processing Baacessing(EMNIIPP) (EMNIP), pages 7252-7262, Online, November 2020. Association for Computational Linguistics.
[43] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduch- intala, and Tsubasa Ochiai. ESPPet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207-2211,2011
13

---

[44] Daniel S. Park, William Chan, Yu Zhang, Chung- Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoe Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019, pages 2613-2617, 2019.
[45] Jaesong Lee and Shinji Watanabe. Intermediate loss regularization for cic-based speech recogni- tion. In ICASSP 2021 -221 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6224-6228, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we propose more efficient decoder architecture for ASR. The motivation and design detailed in Section and we support our claims through range of experiments across diverse tasks and setups, presented in detail in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation as long it clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper diseuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Although we not discuss the limitations depth, Section 4.4 highlights few key shortcomings of our approach along with potential directions address them.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Liimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: In Section we present comprehensive time complexity comparison between our approach and baseline architectures, with detailed proofs provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All our experiments are conducted on publicly available datasets using an
open-source toolkit. Further details are provided in Section 4.1 and the corresponding
hyperparameters are also listed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-soures source models, it may be that access to the model is limited in some way (e-g., to registered users), but it should possible for other researchers have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We conduct all our experiments on publicly available datasets using an open- source toolkit, detailed in Section 4.1 We plan release code, configurations, and models upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
publicc/guides/CoeeSttuuue sos.sooaaaasoooooooooooooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /twpsscccppbllc/////// guwdes/
httppp//b////....ooooooo///aaaaa Cotp//oooeeSessseeeeeeeee stw.uummiiisooooooooocoooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
These details can be found in Section 4.1 and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct MAPSSWE tests on all our experiments show that the results comparable with the baseline.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should he given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer:
[Yes]
Justification:
These details can be found in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special considera- tion due to laws regulations in their jurisdiction).
10. Broader impacts
societal impacts and negative societal Question: Does the paper diseuss both potential positive societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have appropriately cited all the datasets and open-source code used in experiments, More details are available in Section 4.1
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pawpeett.pparwwiiii.. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coodemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this rescarch does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
http://erripssccccooffeeeeeeeeee cc/Comnferencees/222//LLMM
21

---

A Detailed Complexity Analysis of BLOCKDECODER
Let I' denote the length of the acoustic features H. W be the length of the token sequence y. K be the block size for the MERGER and let d be the dimensionality for the attention computations. Additionally, N is the total number of layers in the original decoder, and Ni and NM are the number of layers in text encoder and MERGER respectively. As discussed in Section we choose Ny and NM such that N N + Nas. The goal here, outlined in Section [34] is to mainly compare the total number of attention computations between our BEOODDE and the traditional decoder. To begin with, we know that, during training, decoder takes ((NW=&+NWW) In comparison, BLOCKDECODEE has time complexity of:
time for its attention calculations.
O NT dw + +Na
W Self-Attention within the block
+
W2
WKTd ))
text encoder
Cross- Attention Cross- over H
over C
MERGER
Simplifying this expression, we get:
(Since N N- N
=0(NW"+++N.WW.dd22)))
(Since K ET and W <T)
Doing an elementwise comparison with the time complexity of the decoder, we find:
N
2= Km
2NM
Similarly, during inference, weemploy beam search to generate the B-best hypothesis, where B denotes the beam width. In the case of the traditional decoder, the time complexity for performing inference on y is OW BN BN(W+T)d) That is, in total, we perform W beam steps (since y I=W), with each step processing B partial hypothesis, which are all passed through N decoder layers, where each layer first performs self-attention over the previously generated tokens, followed by cross-attention over the acoustic features H. These operations result in W +7 attention computations, for a single forward pass of one entry in the beam. In contrast, we show that our BECODD significantly reduces the supports
number of attention computations required for beam-search. Given that BLOCKDECOM multiple inference strategies, we now analyze the time complexity for each of those approaches.
NAIVE BLOCK DECODING & ITERATIVE BLOCK DECODING: Although ITERATIVE BLOCK DECODING makes a forward pass through text encoder once every steps, asymptotically, the total number of attention computations are still equivalent to NAIVE BL.CK DECODING. Thus, to calculate the overall time complexity, we assume that a forward pass through both the text encoder and MERGER at each beam-step is performed. Then, the time complexity becomes:
OWN (K+
Simplifying this expression, we get:
WWW (Since N NM)
=O(WB(NW+NNr7) (Since K TT)
22

---

Finally, since NM N, the overall time complexity is significantly smaller than that of the standard decoder. Although ITERATIVE BLOCK DECODING of the same time complexity as NAIVE BLOCK DECODING, since modern hardwares are optimized for parallel computations, it further reduces inference time.
FULL BLOCK AVERAGING:
This strategy involves making multiple forward passes through the MERGER. As result, it has an overall time complexity of:
O(WB(N, W KK
Simplifying this expression, we get:
-OWWB(N-NW/WWWNKKKKKK------- (Since N Nr NM)
N
N
<NWWB(NW+NM, 2NM
(K+W+ (Since K
2N
NO(WB(NW + (K+++T))c)
NO(WB(NW+ + (W (W)))) (Since K <T)
2
3NW NT 2
R BOWB
)dd)
2
~ (WWWBN(++T)d)
Thus, with K ~0 decoder.
N 2NM
this strategy requires approximately similar number of operations as the standard
B BLOCKDECODER vs decoder with cross-attentions only in the later layers
A simpler variant of our architecture can be designed by removing cross-attention blocks from few initial layers of the standard decoder. Since cross-attention the decoder typically causes the per- formance bottleneck, reducing the number of such blocks should improve the decoder efficiency. a performance comparison between the baseline, this simplified decoder variant and We observe that removing cross-altention blocks does provide significant la- tency gains, but at the cost of performance degradation. However, our proposed BLOCK DECODER still achieves the best RTF gain and exhibits no performance degradation, owing its ability to effectively utilize the block structure.
BLOCK
Table 6: Performance comparison (CER or WER %) between Transformer decoder, our BI.OCK- DECODER and a variant of decoder containing cross-attentions only at the later layers (referred to as decoderaticiem) on Librispeech 100h dataset. All experiments use the E-Branchformer encoder.
Architecture
Params
Test Clean +
Test Other 1
RTF
Transformer decoder
38.5M
6.39
17.03
1.52
decoder.dicien::
37.5M
6,51
17.26
0.73
BLOCKDECODER
37.9M
6.19
16.94
0.66(-223x)
C BLOCKDECODER vs CTC-only architectures
In this section, we evaluate DECDDR against vanilla CTC-only models and a widely used CTC- only variant, InterCTC 45), which introduces auxiliary CTC objectives at intermediate encoder layers.
23

---

Since CnCy relelsrely solely on the encoder, we increase their number of encoder layers to match the total number layers (encoder decoder) BLOCKDECODER, Table a performance comparison between the standard Transformer Decoder, the vanilla CTC-only model, InterCTC, and our DECODR, As shown, CTC-only models yield significantly worse performance compared to both the full Transformer decoder and BLoOCKDECODER, Although CTC-only models offer faster inference due to their greedy decoding, this comes at notable cost recognition accuracy. These results support our claim that BLOCKDECODER achieves more favorable trade-off between latency and performance.
Table 7: Performance comparison (CER or WER %) between Transformer decoder, vanilla CTC-only, InterCTC, and our BLOCKDECODER on Librispeech 100h dataset. E-Branchformer an encoder.
All experiments use the
Architecture
Params
Test Clean +
Test Other +
RTF +
Transformer decoder
38.5M
6.39
17.03
1.52
BLOCKDECODER
37.9M
6.19
16.94
(-66kx)
Vanilla CTC-only
37.9M
10.05
23.87
0.36(
InterCTC
37.9M
8.80
21.29
0.37 (~4.1x)
D Aggregate Attention Plots
Figure presents preseet aggregated self-attention and cross-altention plots computed over 250 utterances using the decoder from a model trained on the full Librispeech dataset. The observed patterns are consistent with those in Figure To generate these plots, we first select utterances with minimum audio length of min and a minimum label sequence length of min and then compute the average attention scores over the first min audio frames and W min label positions.
SafAttinton
Cross Cross-4tenn 4thertn
LOyer
Layer
Layer
Layer
Figure 6: Attention plots illustrating the patterns learned by self-attention and cross-attention blocks across all layers of a standard Transformer decoder in hybrid CTClattention ASR system. The plot consists of two rows, one for self-attention and another for cross-attention and six columns, corresponding the six decoder layers (ordered left to right). Brighter colors indicate higher attention weights. These visualizations are generated by aggregating attention scores over 250 utterances using the decoder of model trained on the Librispeech 960h dataset.
E Detailed Results on LibriSpeech 960h
Table 8 presents the complete results on Librispeech 960h dataset, with inference conducted on both CPU and GPU. condensed version of these results is provided Table As outlined in Section with CPU based inference, BLoCKDEcODER achieves nearly 2.5x speedup over the strongest baselines. However, the case GPU-based inference, the RTF values for BEOODDE and baseline are nearly identical. Through detailed profiling, we found that the majority of GPU inference time spent on bookkeeping operations performed by the toolkit to support beam search, while the actual decoder forward passes contribute only a small fraction to the total inference time. Since

---

modifying the entire inference pipeline of the toolkit is both cumbersome and beyond the scope of this work, we additionally report the average total FLOPs (Floating Point Operations) and the mean decoder forward time incurred solely by the decoder during beam search. This offers more focused assessment of the decoder's s computational efficiency. As shown in Table our proposed B1.OCK DECODER reduces the FLOPs and the decoder forward time by approximately 60% compared to the baseline across both CPU and GPU-based inference.
Table 8: Comparison of the performance (WER %) of our system against other architectures on the full Librispeech dataset. Here, DFT stands for "Decoder Forward Time".
Common Metries
Inference
GPU Inference
Method
Params
Text Clean
Test Other
Average FLOPPs in TFLOPS)
KIV
Average DFT
(in (imaa))
Averagy IFT
(inwec)
Transformer 21
139
2,4
5.6
ContexNNet
112.7
4,6
Conformer (M)
5.0
Conformer (L)
118.8
2.1
4.3
QuartzNNet (L.) 35
11.3
Hybrid CTC+Attentin
Transfomeer
Conformer 37
116.2
2.9
7.0
Conformer LL)
147.8
4.7
Branchformer
116.2
5.5
Branchformer (L) 6
146.7
2.2
4.8
Our baselines (Hybrid)
E- Branchfommer
Our work (lyyriid)
w/ BLOCKDECODER
146.8
2.1
4.4
(.3
2.98,
16.02
0.306
0.151,..1.0
F Implementation Details
The hyper-parameters used in all our experiments are reported in Table
25

---

Table 9: A detailed summary of the configurations used for every dataset mentioned in experiments.
Librispeech
Hyper-parameters
Tedlium2
Aishell
Commonvoice
SLURP
100h
960h
Frontend
window length
400
512
400
512
400
512
hop length
160
160
160
128
128
SpecAug
time warp window
5
5
num of freq masks
2
2
2
2
2
freq mask width
(0, (7)
(0,
(0,
(0,27)
(0,30)
(0, 30)
num of time masks
5
10
5
10
2
time mask width
(0, 0.000)57)
(0,0.057)
(0, (.0.05T)
(0, 0.0SS)
(0, 40)
(0,
Architecture
feature size (d)
256
512
$12
attention heads (h)
nurn of encuder layers (Nss
12
17
12
12
12
encoder hidden size (daasss))
1024
1024
1024
1024
2048
1024
depth-wise cony kernel
31
31
31
31
31
31
num of decoder layers (N)
6
6
6
decoder hidden size (dite
2048
2048
2048
2048
2048
2048
Training
epochs
60
learning rate
2e-3
2e-3
2e-3
le-3
1.0
le:3
warmup stops
15k
40k
15k
35k
25k
35k
weight decay
le-6
le-6
le-6
le-6
1e-6
1e-6
Gradient Accum
4
4
2
dropout rate
0.1
0.1
0.1
0.1
0.1
0.1
weight (5)
0.3
0.3
0.3
0.3
0.3
0.3
lahel smoothing weight
0.1
0.1
0.1
26


Reviews:
[{"review_id": "xezMNjO3YD", "reviewer": "Reviewer_zAsc", "summary": "This is a very well-written, detailed proposal of a new architecture for end-to-end ASR that (1) separates text-only layers and audio+text layers in the decoder, and (2) predicts blocks of K tokens at a time, for faster inference. The architecture is motivated by insights from observing actual self- and cross- attention patterns in existing decoders, and by the desire for low-latency inference.", "questions": "Some comments/questions, intended as clarifying or constructive feedback.\n\nL33: \"This phenomenon has been previously observed in the AUDIOENCODER [5] but never analyzed in the DECODER\". \nThough not quite the same context, similar analysis of layer-wise transformer attention scores was done e.g. in \n\n@inproceedings{Irie_2019, series={interspeech_2019},\n   title={Language Modeling with Deep Transformers},\n   url={http://dx.doi.org/10.21437/Interspeech.2019-2225},\n   DOI={10.21437/interspeech.2019-2225},\n   booktitle={Interspeech 2019},\n   publisher={ISCA},\n   author={Irie, Kazuki and Zeyer, Albert and Schlüter, Ralf and Ney, Hermann},\n   year={2019},\n   month=sep, collection={interspeech_2019} }\n\nFig. 1: what is the y-axis of each sub-plot? Tokens, I assume?\n\nL37, \"Based on these systematic attention patterns\": given that so far this is analyzing the situation for just a single sample, it seems like a stretch to say these are \"systematic\" patterns.\n\nL43, proposal of \"BLOCK DECODER\": I am wondering at this point if the \"block\" is conceptually the same as, or distinct from, the fact that a _block_ of tokens will be predicted. I.e., I could have a similar unit, call it a block, combining the two modules mentioned, that did not predict a block of tokens. I could also have a standard decoder, that does not distinguish text only from text+audio, that did predict a block of tokens. So upon first reading, I'm wondering whether the \"block\" in the name \"block decoder\" is actually helping the prediction of a \"block\" of tokens. Upon second reading, I see that the block decoder is indeed helping the prediction of a block of tokens -- could the authors make the presentation more easily accessible?\n\nL47, \"lesser\" --> \"fewer\"\n\nL59, \"or introducing new training objectives [20, 21]\". Great that the work is also considers neural transducers here and in the evaluation section. Transducers are more than a new training objective, though -- in fact, the transducer decoder, being purely text based, is quite strongly related to the proposed architecture in this submission. In a way a neural transducer is like the proposed architecture, but with a very simple Merger (the transducer \"joiner\"). This link could be discussed in more detail in a way that highlighted the merit of the proposed architecture.\n\nFig 2(b): these two blocks just seem like vanilla transformer stacks, one with self-attention, the other with cross-attention? If so, do we need to illustrate them? I note that the fonts in the figure, e.g. for \"Audio Encoder\" and \"Text Encoder\", are not consistent. Furthermore, the example, \"_B L OCK\" etc., propagates from Merger to merge by only apparently adding one new token, so this doesn't illustrate the key feature of the proposal, the prediction of multiple tokens?\n\nL111 \"dropouts\" --> \"dropout\"\n\n\"Section 3.2.2 Merger\": finally, getting to the equation (which should be numbered) at L124, and the text at L133, I get a sense for the details of the multi-token prediction. Since this is a key part of the proposal, ideally the basic approach here would be described more specifically, and much earlier in the paper, and as I suggested earlier, it should be made clearer how the multi-token prediction benefits from the BlockDecoder architecture.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The core idea of leveraging separate text and text+audio blocks specifically to improve multi-token inference latency is interesting and apparently effective. A real strength of the work is its good writing and precise mathematical exposition, with further analysis of computational cost in the Appendix. Furthermore, a comprehensive set of WER evaluation is presented on some well-known standard test sets, with decent results -- matching, sometimes bettering, existing baselines, but more notably, producing improvements in inference latency.\n\nWhile well-written and precise, the work is nonetheless often unclear in the bigger picture. Precise details can obscure the larger points, making the work slightly inaccessible. The first several pages feel like a rather long-winded exposition of the key aspects of the architecture. The reader really has to go over the proposal in detail to grasp how the proposed architecture is different from existing architectures. At first glance, it does not seem that original, e.g. reminds me of this study:\n\n@misc{deng2023adaptableendtoendasrmodels,\n      title={Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax}, \n      author={Keqi Deng and Philip C. Woodland},\n      year={2023},\n      eprint={2302.08579},\n      archivePrefix={arXiv},\n      primaryClass={eess.AS},\n      url={https://arxiv.org/abs/2302.08579}, \n}\n\nHowever, what I finally realized is that the proposal leverages this decoupling specifically for multi-token inference. That is a very interesting property -- but one that is not explained very clearly, and in fact not even mentioned in the Abstract.\n\nIt seems that overall the work could, ideally, be presented more effectively and with greater impact by significantly compressing the description of the aspects of the architecture that closely follow existing standard architectures, and highlighting the novel aspects, e.g. the multi-token prediction mechanism.", "was_revised": true, "final_justification": "In their rebuttal, the authors' showed good understanding of my central concerns regarding the key concepts and the manner of presentation, and indicated willingness to address those concerns -- particularly relating to expanded discussion of the connection with neural transducers, better presentation of the core and combined contributions of modular + block structure, improved notation and improved figures.", "rebuttal": "Dear Reviewer,\n\nThank you for your very thoughtful and constructive assessment of our work! Below, we address each of your comments in detail:\n\n> Fig. 1: what is the y-axis of each sub-plot? Tokens, I assume?\n\nYes, the $y$-axis represents the token (or label) sequence. \n\n> L37, \"Based on these systematic attention patterns\": given that so far this is analyzing the situation for just a single sample, it seems like a stretch to say these are \"systematic\" patterns.\n\nThanks for pointing this out!  We computed the average attention weights for all the attention blocks of the decoder across 1000 examples, and observed the same patterns shown in Figure 1. We will include this figure in the revised draft to be more precise when claiming systematic patterns.\n\n> L43, Clarifying the Role of \"Block\" in the Block Decoder and Its Comparison to Related Variants \n\nThank you for this insightful comment!  The name BlockDecoder is indeed primarily meant to reflect its ability to predict a block of tokens (auto-regressively). However, we would like to emphasize that our overall gains come from both predicting a block of tokens and decoupling the decoder into text-only (with self-attention) and text+audio (with cross-attention) modules. To further clarify: if the merger predicted only one token at a time, then the architecture, despite being modular, would incur similar computational costs to a standard decoder with early cross-attention layers removed (please refer to Appendix E for more details). Conversely, if a full decoder predicted tokens in blocks but retained all cross-attention layers, the speed-up would be limited, since the dominant cost arises from attending to long encoder outputs at every layer. Our approach combines modularity (removing early cross-attention) with block-level prediction, where only every $k^{\\text{th}}$ token attends to the text-encoder, reducing computation further by a factor of $k$. These two components are complementary: block prediction alone or modularity alone would yield partial gains, but together they enable the full efficiency benefits we report. We will revise the draft to clarify this dual role of “block” in the naming and motivation behind BlockDecoder.\n\n> L59,  On the Architectural Similarities and Differences with Neural Transducers\n\nWe agree that our proposed architecture shares conceptual similarities with neural transducers, as both decouple the encoder and decoder streams and rely on a composition mechanism to integrate them. As the reviewer noted, the Joiner in the Transducer is a shallow feedforward network. In contrast, our Merger includes multiple attention layers, enabling richer and more flexible context integration. We will revise the draft to acknowledge this connection more clearly and highlight the architectural contributions of our approach.\n\n> Fig 2(b): these two blocks just seem like vanilla transformer stacks, one with self-attention, the other with cross-attention? If so, do we need to illustrate them? I note that the fonts in the figure, e.g. for \"Audio Encoder\" and \"Text Encoder\", are not consistent. Furthermore, the example, \"_B L OCK\" etc., propagates from Merger to merge by only apparently adding one new token, so this doesn't illustrate the key feature of the proposal, the prediction of multiple tokens?\n\n1. The Merger is indeed designed to autoregressively predict a block of multiple tokens. At each stage, the Merger receives representation of a progressively longer text prefix (e.g., [\\<sos\\>], [\\<sos\\>, _B] etc.) from Text-Encoder (shown via the blue bars) along with the corresponding audio representation, and predicts a block of three tokens auto-regressively. We will revise both Figure 2 and the caption to make it clearer that multiple tokens are predicted. \n2. We included the stacks in Fig 2b just to highlight that the Text Encoder contains only self-attention layers, while the Merger contains both self-attention and cross-attention layers. We will aim to visually highlight this better.\n3. The font differences were intended to distinguish newly introduced components from pre-existing ones, but we will make the fonts consistent to avoid any confusion.\n\n> L33, prior work on attention (Irie et al.); Minor edits (L47 “lesser” → “fewer”, L111 “dropouts” → “dropout”; and section 3.2.2, the suggestion to present the core multi-token idea earlier and more clearly.\n\nThank you for these detailed comments! We will revise the draft to incorporate these changes. \n\nThanks again for your thoughtful review, and please let us know if any further clarifications would be helpful."}, {"review_id": "L3rHSQ1vxk", "reviewer": "Reviewer_1eQS", "summary": "A novel decoder architecture, BlockDecoder, is proposed for\nattention-based encoder-decoder ASR. It is motivated after observing\nthat attention layers in conventional Transformed-based decoders first\nfocus on building textual context, and then on merging acoustic and\ntextual information. Based on this observation, BlockDecoder is\nproposed as an alternative to conventional decoders. It consists of\ntwo separate sub-modules, a text encoder with only self-attention\nlayers to build textual context, and a lightweight merger with\ncross-attention layers fusing acoustic information from the encoder\nand contextualized outputs from the text encoder. Empirical results\nshow that the proposed decoder roughly doubles in speed conventional\ndecoders without degradation in performance.", "questions": "* Could you please provide more evidence supporting the observation inspiring\n  BlockDecoder? Please consider showing attention weights averaged over a set\n  of examples instead of a single example.\n* The conventional Transformer Decoder does not include efficient inference\n  techniques. Could you please run additional experiments using these\n  techniques? It would be good to see RTF results from these experiments in\n  all relevant Tables.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* Good work; its technical side and sound empirical assessment are of\n  good quality. Improving the efficiency of cutting-edge ASR systems\n  without performance degradation is challenging; and the authors\n  manage to do so.\n* The paper is largely clear.\n* To my knowledge, the idea behind the proposed decoder is\n  original. Speeding up transformer architectures is a hot research\n  topic, and thus this work fits well within current research trends.\n\n**Weaknesses**\n* The observation inspiring BlockDecoder is based on limited evidence.\n* The paper does not consider related work on efficient transformer inference\n  such as KV Cache and FlashAttention. This renders significance to the main\n  contribution.", "was_revised": true, "final_justification": "I have updated my rating to 5 (accept) since my main doubts have been cleared up.", "rebuttal": "Dear Reviewer,\n\nThank you for your valuable feedback. Below, we address your comments in detail:\n\n> Could you please provide more evidence supporting the observation inspiring BlockDecoder? Please consider showing attention weights averaged over a set of examples instead of a single example.\n\nThanks for pointing this out!  We computed the average attention weights for all the attention blocks of the decoder across 1000 examples, and observed the same patterns shown in Figure 1. We will include this figure in the revised draft.\n\n> The conventional Transformer Decoder does not include efficient inference techniques. Could you please run additional experiments using these techniques? It would be good to see RTF results from these experiments in all relevant Tables.\n\nThank you for raising this important point. We would like to clarify that all the results reported in our paper already incorporate widely used efficient inference techniques, including key-value (KV) caching and Automatic Mixed Precision (AMP). We will make sure to explicitly state this in the revised draft. \n\nThanks again for your feedback, and please let us know if any further clarifications would be helpful."}, {"review_id": "DaBZA8WUPV", "reviewer": "Reviewer_w6fN", "summary": "This paper introduces a novel decoder architecture for attention-based encoder-decoder ASR systems. The core idea is to disentangle the decoder into two specialized components: a text encoder block and an audio attention block, based on empirical observations. These are subsequently merged using a dedicated merge function. The architecture is designed in a block-wise manner, offering greater efficiency during inference compared to conventional decoder designs that rely on integrated self-attention and cross-attention mechanisms. Experiments on several standard ASR and SLU benchmarks demonstrate state-of-the-art performance, along with improved inference efficiency. Ablation studies further support the authors’ hypothesis that separating the text encoding and audio attention functions leads to both performance and efficiency gains.", "questions": "- It would be valuable to discuss how the proposed findings and architectural modifications could be extended to other models, such as Whisper—which uses a decoder with prompt conditioning—or decoder-only models. Such a discussion would enhance the generalizability and impact of the proposed method across different architectures.\n- Similarly, is there relevant literature or comparable architectural separation in neural machine translation (NMT)? Discussions with NMT could broaden the relevance of the work.\n- Please include a discussion of training time and scalability. Given the growing focus on model and data scalability in recent AI research, it would be helpful to understand how the proposed approach scales with respect to training time, data size, and model complexity.\n- Could the text encoder block be replaced or initialized with a pre-trained large language model (LLM)? Leveraging LLMs may enhance the performance and attractiveness of the proposed architecture.\n- In Section 3.1, line 82, the phrase “(CTC) loss [31] non-autoregressively” is misleading. While CTC inference is non-autoregressive, the CTC loss itself is computed using the forward-backward algorithm, and thus is not non-autoregressive in nature. Please clarify this distinction.\n- In Section 3.2.1 (footnote), the expression $T=T'/S$ should specify that the result is rounded, since time frame division generally results in non-integer values.\n- The equations describing the merge function are difficult to follow. They are currently unnumbered, and the associated explanations are hard to trace. It would help to number these equations and provide a clearer, step-by-step interpretation.\n- The SLU task setup needs further clarification. For example, typical intent classification is a non-sequential task and does not require a decoder, whereas slot filling is sequential and does. If your SLU task combines ASR with intent classification, it would become a sequence-to-label problem. Please clarify the SLU formulation in the paper.\n- In Section 4.4, the attention visualizations are informative. However, could you also include numerical metrics to quantify attention patterns? For instance, a diagonality score has been used in some studies to evaluate attention sharpness and alignment quality.", "limitations": "The authors acknowledge training time trade-offs, which appear to stem primarily from technical limitations. However, the paper does not address potential negative societal impacts. For instance, large-scale encoder-decoder ASR systems such as Whisper are known to exhibit hallucination issues due to the complexity and size of their decoder networks. Since the proposed method also relies on a decoder architecture, similar risks may be inherited. That said, the block-wise processing approach introduced here may help mitigate some of these issues. A discussion of such societal implications, especially in terms of reliability and misuse, would strengthen the broader impact assessment of the work.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper presents new insights into decoder behavior by highlighting the functional separation between text encoding and audio attention components.\n- A novel decoder architecture is proposed based on this finding, utilizing a modular design with clear functional roles.\n- The approach achieves efficient inference and demonstrates strong performance across multiple standard ASR and SLU benchmarks, supported by comprehensive ablation studies.\n\nWeaknesses\n- The motivation for adopting block-wise processing is not clearly justified. While it contributes to inference efficiency, it is unclear whether block processing is essential to the proposed architecture, as the core idea could potentially be applied to a standard decoder.\n- The merge function is difficult to understand. The equation presented on page 4 is abstract, and the accompanying explanations are not fully grounded in the mathematical formulation.\n- The experimental section lacks some practical implementation details, such as training time, hardware specifications (e.g., CPU type used for real-time factor measurements), and overall computational cost.\n- The paper could benefit from a broader discussion of related work on decoder architectures. For instance, recent work exploring structured state space models and non-autoregressive decoding (e.g., mask prediction) are relevant and should be considered:\n  - Miyazaki, Koichi, Masato Murata, and Tomoki Koriyama. \"Structured state space decoder for speech recognition and synthesis.\" ICASSP 2023.\n  - Higuchi, Y., Watanabe, S., Chen, N., Ogawa, T., & Kobayashi, T. \"Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict.\" Interspeech 2020.", "was_revised": true, "final_justification": "The authors’ responses address some of my concerns regarding clarifications, which will certainly improve the paper. However, they do not fully resolve the weaknesses I previously identified. In addition, these changes would require major revisions, making it difficult to properly assess the final paper at this stage. Therefore, I would like to maintain my score as Borderline Accept.", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive and detailed feedback! Below, we address each of your comments in detail:\n\n> on extending the proposed architecture to Whisper, Decoder-Only Models, NMT, and Integration with Pretrained LLMs\n\nThanks for these valuable suggestions! We agree that broadening the discussion to other areas/models/applications could further strengthen the impact of our work. While our proposed architecture is particularly effective in settings where the encoder output is substantially longer than the target sequence (e.g., ASR), similar dynamics occur in tasks such as document summarization. These often involve long input sequences and shorter outputs, making them promising candidates for adaptations of BlockDecoder. Additionally, since BlockDecoder separates the decoder into a text-only encoder and merger module, our framework naturally supports replacement of the text encoder with a pre-trained LLM, a direction we plan to explore in future work. Finally, the architectural modularity we introduce could inspire pruning strategies in large models such as Whisper to accelerate inference, especially in real-time use cases.\nWe will add a discussion addressing these points in the revised draft.\n\n> Please include a discussion of training time and scalability. Given the growing focus on model and data scalability in recent AI research, it would be helpful to understand how the proposed approach scales with respect to training time, data size, and model complexity.\n\nThank you for the suggestion. Sections 3.4 and 4.4 briefly discuss the impact of our architectural modifications on both training and inference time. To summarize: our experiments span datasets ranging from 100 to 1,000 hours of audio. Across this range, training time remains comparable to the baseline, while inference time is consistently reduced. We will revise the draft to better highlight these findings and add a more explicit summary of training/inference efficiency.\n\n> In Section 3.1, line 82, the phrase “(CTC) loss [31] non-autoregressively” is misleading. While CTC inference is non-autoregressive, the CTC loss itself is computed using the forward-backward algorithm, and thus is not non-autoregressive in nature. Please clarify this distinction.\n\nWe respectfully clarify that CTC is indeed non-autoregressive. The CTC loss is computed using the forward-backward algorithm (that involves dynamic programming over time) to marginalize over all possible alignments but it does not introduce autoregressive dependencies (i.e., conditioning the output at time t on previous output tokens).  \n\n> In Section 3.2.1 (footnote), the expression $T = T’ / S$ should specify that the result is rounded, since time frame division generally results in non-integer values.\n> The equations describing the merge function are difficult to follow. They are currently unnumbered, and the associated explanations are hard to trace. It would help to number these equations and provide a clearer, step-by-step interpretation.\n\nThank you for these edits. We will revise the draft to incorporate these changes. \n\n> The SLU task setup needs further clarification. For example, typical intent classification is a non-sequential task and does not require a decoder, whereas slot filling is sequential and does. If your SLU task combines ASR with intent classification would become a sequence-to-label problem. Please clarify the SLU formulation in the paper.\n\nApologies for the ambiguity. In our SLU setup, the model is indeed first trained with an ASR objective, where the output label sequence is augmented to include intent and entity related tags. At inference time, we decode the sequence as in ASR, and compute SLU metrics by parsing the decoded output. We will revise the SLU section to make this setup explicit.\n\n> In Section 4.4, the attention visualizations are informative. However, could you also include numerical metrics to quantify attention patterns? For instance, a diagonality score has been used in some studies to evaluate attention sharpness and alignment quality.\n\nThank you for this suggestion! We will include diagonality scores in a revised version for a more quantitative measure of the attention patterns.\n\nThanks again for your thoughtful review, and please let us know if any further clarifications would be helpful."}, {"review_id": "ISleliaMA6", "reviewer": "Reviewer_pHX8", "summary": "This paper introduces an approach to disentangle the self and cross attention layer in a transformer decoder for AED based ASR. This is based on an interesting observation by the authors that the initial layers of cross attention and the later layers of self-attention in a AED based ASR models are quite redundant. Based on this observation, the authors propose context and merger modules that disentangle the two aforementioned attention mechanisms. Furthermore, the merger module is equipped with the ability to perform block decoding which contributes to fast decoding capabilities of the proposed model as evidenced by the results.", "questions": "Here are some clarification questions I had while reading the paper:\n1. In the equations between lines 105-106, there are residual connections after $LayerNorm_{att}$ and $LayerNorm_{ff}$ whereas no such residual connections are shown in figure 2(b). This can be a source of confusion for the reader.\n2. Figure 2(a) was very confusing for me. It is not clear how to interpret it. Concretely, in Figure 2(a), it seems like $y_i ... y_{i+K-1}$ is conditioned on all previous tokens, but the equation in 3.2.2 suggest otherwise (the dependency seems to be only on $y_{i-1}$).\n3. It is also unclear what the role/effect is of the CTC module in figure 2(a).\n4. Figure 2(a) might benefit from a more detailed/thorough caption.\n5. The authors' use of the modified token sequence suggests an increase in the training data to cover for all subsequences. Does this result in an increased training time? If so, how much? Furthermore, if there is an increase in training time, it raises the question of whether the proposed technique might be an overkill just to attain speed improvements and not much WER improvements?\n6. The results suggest that the only improvements gained from the proposed model is in RTF/speedup. The authors should compare their model with state-of-the-art CTC models like self-conditioned,  hierarchically conditioned, BERT-based CTC models and self-conditioned folded encoders[1-4]. What is the speed up compared to these models and what is the corresponding trade-off with WER performance.\n7. In appendix E, the authors show that their model is better than just disentangling self and cross attention in a standard transformer decoder. However, it is not clear why that is the case? Does the improvement come from the block decoding strategy or is there something else also?\n8. I would like the authors to comment on whether it is possible to use a pre-trained language model or even an LLM as a substitute for the text encoder. Have the authors explored that direction?\n\n\n[1] Nozaki et al. Relaxing the conditional independence assumption of CTC-based ASR by conditioning on intermediate predictions. \n\n[2] Higuchi et al. Hierarchical conditional end-to-end ASR with CTC and multi-granular subword units.\n\n[3] Higuchi et al. BERT meets CTC: New formulation of end-to-end speech recognition with pre-trained masked language model.\n\n[4] Komatsu et al. Non-autoregressive asr with self-conditioned folded encoders.", "limitations": "The authors have addressed limitations and my concerns to a large extent hence I would suggest acceptance of the paper which was my original recommendation at a score of 5.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. This paper builds on a good observation made on the working of transformer decoders which reflects good research insights.\n2. The research contribution is significant in that the paper proposes a technique to further faster inference in ASR models.\n3. The research contribution is original to the best of my knowledge.\n4. Diverse set of experiments and ablation studies are conducted. \n\nWeaknesses:\n1. The paper has some clarity issues in properly explaining the technical details of their contribution (please see \"Questions\" section)\n2. There is no significant WER improvements using the proposed model. Although the speed of inference improves, it is hard for me to see whether similar performance gains cannot be achieved by carefully training existing non-autoregressive models like CTC, self-conditioned CTC or hierarchically conditioned CTC models which are fully non-autoregressive and can do a very fast decoding.", "was_revised": true, "final_justification": "This paper is built on a good research insight and furthers the advancements of inference speed in autoregressive models. Hence I suggest acceptance.", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful and constructive feedback! Below, we address each of your comments in detail:\n\n> In the equations between lines 105-106, there are residual connections after $LayerNorm_{att}$ and $LayerNorm_{ff}$ whereas no such residual connections are shown in figure 2(b). This can be a source of confusion for the reader.\n\nWe deliberately omitted the residual connections in the figure to avoid visual clutter. This omission is mentioned in the Figure’s caption. We will revise the caption to clarify this further.\n\n> Figure 2(a) was very confusing for me. It is not clear how to interpret it. Concretely, in Figure 2(a), it seems like $y_i, \\ldots y_{i+K-1}$ is conditioned on all previous tokens, but the equation in 3.2.2 suggest otherwise (the dependency seems to be only on $y_{i-1}$). Figure 2(a) might benefit from a more detailed/thorough caption.\n\nThank you for this comment, and we apologize for the confusion. At each stage, the Merger receives representation of a progressively longer text prefix (e.g., [\\<sos\\>], [\\<sos\\>, _B] etc.) from Text-Encoder (shown via the blue bars) along with the corresponding audio representation, and predicts a block of three tokens auto-regressively. For example, in the first step, Merger is trained to predict three tokens conditioned on [\\<sos\\>], and in the second step, it is trained to predict the next three tokens conditioned on [\\<sos\\>, \\_B], and so on. Since each block prediction is itself auto-regressive, we need a starting token. While works like Megabyte [1] use an underscore (\\_)  as the first token in the block, we use the last token of the prefix (e.g., _B in the second step), which helps maintain continuity. \n\nWe will revise both Figure 2 and the caption to make these points much clearer. \n\n[1] Yu, Lili, et al. \"Megabyte: Predicting million-byte sequences with multiscale transformers.\" Advances in Neural Information Processing Systems 36 (2023): 78808-78823.\n\n> It is also unclear what the role/effect is of the CTC module in figure 2(a).\n\nWe follow the hybrid CTC/Attention Encoder-Decoder framework [1], which uses a multi-task training objective. The CTC loss on the encoder encourages faster convergence and, during inference, helps guide the decoder to adhere to its monotonic alignments.  \n\n[1] S. Watanabe, T. Hori, S. Kim, J. R. Hershey and T. Hayashi, \"Hybrid CTC/Attention Architecture for End-to-End Speech Recognition,\" in IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, Dec. 2017, doi: 10.1109/JSTSP.2017.2763455.\n\n> The authors' use of the modified token sequence suggests an increase in the training data to cover for all subsequences. Does this result in an increased training time? If so, how much? Furthermore, if there is an increase in training time, it raises the question of whether the proposed technique might be an overkill just to attain speed improvements and not much WER improvements?\n\nThank you for the suggestion. Sections 3.4 and 4.4 briefly discuss the impact of our architectural modifications on both training and inference time. To summarize: our experiments span datasets ranging from 100 to 1,000 hours of audio. Across this range, training time remains comparable to the baseline, while inference time is consistently reduced. We will revise the draft to better highlight these findings and add a more explicit summary of training/inference efficiency.\n\n> The results suggest that the only improvements gained from the proposed model is in RTF/speedup. The authors should compare their model with state-of-the-art CTC models like self-conditioned, hierarchically conditioned, BERT-based CTC models and self-conditioned folded encoders[1-4]. What is the speed-up compared to these models and what is the corresponding trade-off with WER performance.\n\nThank you for this suggestion! While CTC-only models are generally faster due to greedy decoding, they often underperform compared to encoder-decoder models in terms of accuracy. Since our work adopts the encoder-decoder paradigm, we primarily compare against strong sequence-to-sequence baselines. Incorporating comparisons with advanced CTC-based methods is a promising direction, and we will aim to  include such comparisons in future work.\n\n> In appendix E, the authors show that their model is better than just disentangling self and cross attention in a standard transformer decoder. However, it is not clear why that is the case? Does the improvement come from the block decoding strategy or is there something else also?\n\nYes, the improvement stems from the block-level design of the Merger. In the model from Appendix E, a single self-attention layer attempts to model both local (within block) and global (outside block) contexts simultaneously. Our design explicitly separates these requirements: self-attention in the block captures local structure, while cross-attention accesses broader text context. This modularity allows the Merger to independently learn and integrate these representations, improving expressivity and performance.\n\n> I would like the authors to comment on whether it is possible to use a pre-trained language model or even an LLM as a substitute for the text encoder. Have the authors explored that direction?\n\nIndeed, the modular structure of BlockDecoder naturally supports replacing the Text Encoder with a pre-trained language model, including LLMs. This is an exciting direction we intend to explore. Moreover, our architectural separation offers a pathway to apply pruning or distillation techniques to large models like Whisper to reduce inference cost, especially in streaming or real-time scenarios. \n\nThanks again for your feedback, and please let us know if any further clarifications would be helpful."}]

Author rebuttal:

Meta-review: This paper proposes a new architecture for ASR which uses text-only layers and audio+text layers and then predicts multiple outputs concurrently to speed up decoding. The reviewers praised the central idea, the good experimnetal results, and the efficient inference. Some of the reviewers concerns around limited discussion of other related work was addressed during the discussion period as well as the better presentation of the central ideas. Not all questions could be resolved during the discussion period, but I believe this is a good paper and I recommend acceptance.
Decision: Accept (poster)