Title: BlockDecoder: Boosting ASR Decoders with Context and Merger Modules
Venue: NeurIPS 2025
Accepted: yes
Keywords: Automatic speech recognition

Attention Encoder Decoder

Efficient Decoder
Abstract: Attention-based encoder decoder models remain a popular choice for state-of-the-art automatic speech recognition (ASR). These models combine a powerful audio encoder that extracts rich acoustic features with a decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe a systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the later layers largely focus on merging acoustic and textual information for the final predictions. Leveraging this key insight, we propose **BlockDecoder**, a novel decoder architecture comprising two distinct components: a text encoder that is purely text-based, and a **Merger** that combines information from the audio encoder and text encoder to generate output tokens. Unlike traditional decoders, the **Merger** autoregressively predicts a sequence of K tokens within a *block* of size K, while relying on the same precomputed contextual information from both text and audio encoders across the block. This design choice allows for the efficient reuse of encoder representations. The separation of the decoder into the text encoder and the **Merger** promotes modularity and more flexible control of parameters via the number of text encoder and **Merger** layers. As a result, **BlockDecoder** yields a significant speedup ($\sim2$x) compared to traditional decoders, across diverse datasets, languages, and speech tasks, without any degradation in performance. The code is available at https://github.com/csalt-research/blockdecoder.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach to disentangle the self and cross attention layer in a transformer decoder for AED based ASR. This is based on an interesting observation by the authors that the initial layers of cross attention and the later layers of self-attention in a AED based ASR models are quite redundant. Based on this observation, the authors propose context and merger modules that disentangle the two aforementioned attention mechanisms. Furthermore, the merger module is equipped with the ability to perform block decoding which contributes to fast decoding capabilities of the proposed model as evidenced by the results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper builds on a good observation made on the working of transformer decoders which reflects good research insights.\n2. The research contribution is significant in that the paper proposes a technique to further faster inference in ASR models.\n3. The research contribution is original to the best of my knowledge.\n4. Diverse set of experiments and ablation studies are conducted. \n\nWeaknesses:\n1. The paper has some clarity issues in properly explaining the technical details of their contribution (please see \"Questions\" section)\n2. There is no significant WER improvements using the proposed model. Although the speed of inference improves, it is hard for me to see whether similar performance gains cannot be achieved by carefully training existing non-autoregressive models like CTC, self-conditioned CTC or hierarchically conditioned CTC models which are fully non-autoregressive and can do a very fast decoding.\n\nQUESTIONS:\nHere are some clarification questions I had while reading the paper:\n1. In the equations between lines 105-106, there are residual connections after $LayerNorm_{att}$ and $LayerNorm_{ff}$ whereas no such residual connections are shown in figure 2(b). This can be a source of confusion for the reader.\n2. Figure 2(a) was very confusing for me. It is not clear how to interpret it. Concretely, in Figure 2(a), it seems like $y_i ... y_{i+K-1}$ is conditioned on all previous tokens, but the equation in 3.2.2 suggest otherwise (the dependency seems to be only on $y_{i-1}$).\n3. It is also unclear what the role/effect is of the CTC module in figure 2(a).\n4. Figure 2(a) might benefit from a more detailed/thorough caption.\n5. The authors' use of the modified token sequence suggests an increase in the training data to cover for all subsequences. Does this result in an increased training time? If so, how much? Furthermore, if there is an increase in training time, it raises the question of whether the proposed technique might be an overkill just to attain speed improvements and not much WER improvements?\n6. The results suggest that the only improvements gained from the proposed model is in RTF/speedup. The authors should compare their model with state-of-the-art CTC models like self-conditioned,  hierarchically conditioned, BERT-based CTC models and self-conditioned folded encoders[1-4]. What is the speed up compared to these models and what is the corresponding trade-off with WER performance.\n7. In appendix E, the authors show that their model is better than just disentangling self and cross attention in a standard transformer decoder. However, it is not clear why that is the case? Does the improvement come from the block decoding strategy or is there something else also?\n8. I would like the authors to comment on whether it is possible to use a pre-trained language model or even an LLM as a substitute for the text encoder. Have the authors explored that direction?\n\n\n[1] Nozaki et al. Relaxing the conditional independence assumption of CTC-based ASR by conditioning on intermediate predictions. \n\n[2] Higuchi et al. Hierarchical conditional end-to-end ASR with CTC and multi-granular subword units.\n\n[3] Higuchi et al. BERT meets CTC: New formulation of end-to-end speech recognition with pre-trained masked language model.\n\n[4] Komatsu et al. Non-autoregressive asr with self-conditioned folded encoders.\n\nLIMITATIONS:\nThe authors have addressed limitations and my concerns to a large extent hence I would suggest acceptance of the paper which was my original recommendation at a score of 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel decoder architecture for attention-based encoder-decoder ASR systems. The core idea is to disentangle the decoder into two specialized components: a text encoder block and an audio attention block, based on empirical observations. These are subsequently merged using a dedicated merge function. The architecture is designed in a block-wise manner, offering greater efficiency during inference compared to conventional decoder designs that rely on integrated self-attention and cross-attention mechanisms. Experiments on several standard ASR and SLU benchmarks demonstrate state-of-the-art performance, along with improved inference efficiency. Ablation studies further support the authors’ hypothesis that separating the text encoding and audio attention functions leads to both performance and efficiency gains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper presents new insights into decoder behavior by highlighting the functional separation between text encoding and audio attention components.\n- A novel decoder architecture is proposed based on this finding, utilizing a modular design with clear functional roles.\n- The approach achieves efficient inference and demonstrates strong performance across multiple standard ASR and SLU benchmarks, supported by comprehensive ablation studies.\n\nWeaknesses\n- The motivation for adopting block-wise processing is not clearly justified. While it contributes to inference efficiency, it is unclear whether block processing is essential to the proposed architecture, as the core idea could potentially be applied to a standard decoder.\n- The merge function is difficult to understand. The equation presented on page 4 is abstract, and the accompanying explanations are not fully grounded in the mathematical formulation.\n- The experimental section lacks some practical implementation details, such as training time, hardware specifications (e.g., CPU type used for real-time factor measurements), and overall computational cost.\n- The paper could benefit from a broader discussion of related work on decoder architectures. For instance, recent work exploring structured state space models and non-autoregressive decoding (e.g., mask prediction) are relevant and should be considered:\n  - Miyazaki, Koichi, Masato Murata, and Tomoki Koriyama. \"Structured state space decoder for speech recognition and synthesis.\" ICASSP 2023.\n  - Higuchi, Y., Watanabe, S., Chen, N., Ogawa, T., & Kobayashi, T. \"Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict.\" Interspeech 2020.\n\nQUESTIONS:\n- It would be valuable to discuss how the proposed findings and architectural modifications could be extended to other models, such as Whisper—which uses a decoder with prompt conditioning—or decoder-only models. Such a discussion would enhance the generalizability and impact of the proposed method across different architectures.\n- Similarly, is there relevant literature or comparable architectural separation in neural machine translation (NMT)? Discussions with NMT could broaden the relevance of the work.\n- Please include a discussion of training time and scalability. Given the growing focus on model and data scalability in recent AI research, it would be helpful to understand how the proposed approach scales with respect to training time, data size, and model complexity.\n- Could the text encoder block be replaced or initialized with a pre-trained large language model (LLM)? Leveraging LLMs may enhance the performance and attractiveness of the proposed architecture.\n- In Section 3.1, line 82, the phrase “(CTC) loss [31] non-autoregressively” is misleading. While CTC inference is non-autoregressive, the CTC loss itself is computed using the forward-backward algorithm, and thus is not non-autoregressive in nature. Please clarify this distinction.\n- In Section 3.2.1 (footnote), the expression $T=T'/S$ should specify that the result is rounded, since time frame division generally results in non-integer values.\n- The equations describing the merge function are difficult to follow. They are currently unnumbered, and the associated explanations are hard to trace. It would help to number these equations and provide a clearer, step-by-step interpretation.\n- The SLU task setup needs further clarification. For example, typical intent classification is a non-sequential task and does not require a decoder, whereas slot filling is sequential and does. If your SLU task combines ASR with intent classification, it would become a sequence-to-label problem. Please clarify the SLU formulation in the paper.\n- In Section 4.4, the attention visualizations are informative. However, could you also include numerical metrics to quantify attention patterns? For instance, a diagonality score has been used in some studies to evaluate attention sharpness and alignment quality.\n\nLIMITATIONS:\nThe authors acknowledge training time trade-offs, which appear to stem primarily from technical limitations. However, the paper does not address potential negative societal impacts. For instance, large-scale encoder-decoder ASR systems such as Whisper are known to exhibit hallucination issues due to the complexity and size of their decoder networks. Since the proposed method also relies on a decoder architecture, similar risks may be inherited. That said, the block-wise processing approach introduced here may help mitigate some of these issues. A discussion of such societal implications, especially in terms of reliability and misuse, would strengthen the broader impact assessment of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA novel decoder architecture, BlockDecoder, is proposed for\nattention-based encoder-decoder ASR. It is motivated after observing\nthat attention layers in conventional Transformed-based decoders first\nfocus on building textual context, and then on merging acoustic and\ntextual information. Based on this observation, BlockDecoder is\nproposed as an alternative to conventional decoders. It consists of\ntwo separate sub-modules, a text encoder with only self-attention\nlayers to build textual context, and a lightweight merger with\ncross-attention layers fusing acoustic information from the encoder\nand contextualized outputs from the text encoder. Empirical results\nshow that the proposed decoder roughly doubles in speed conventional\ndecoders without degradation in performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Good work; its technical side and sound empirical assessment are of\n  good quality. Improving the efficiency of cutting-edge ASR systems\n  without performance degradation is challenging; and the authors\n  manage to do so.\n* The paper is largely clear.\n* To my knowledge, the idea behind the proposed decoder is\n  original. Speeding up transformer architectures is a hot research\n  topic, and thus this work fits well within current research trends.\n\n**Weaknesses**\n* The observation inspiring BlockDecoder is based on limited evidence.\n* The paper does not consider related work on efficient transformer inference\n  such as KV Cache and FlashAttention. This renders significance to the main\n  contribution.\n\nQUESTIONS:\n* Could you please provide more evidence supporting the observation inspiring\n  BlockDecoder? Please consider showing attention weights averaged over a set\n  of examples instead of a single example.\n* The conventional Transformer Decoder does not include efficient inference\n  techniques. Could you please run additional experiments using these\n  techniques? It would be good to see RTF results from these experiments in\n  all relevant Tables.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a very well-written, detailed proposal of a new architecture for end-to-end ASR that (1) separates text-only layers and audio+text layers in the decoder, and (2) predicts blocks of K tokens at a time, for faster inference. The architecture is motivated by insights from observing actual self- and cross- attention patterns in existing decoders, and by the desire for low-latency inference.\n\nSTRENGTHS AND WEAKNESSES:\nThe core idea of leveraging separate text and text+audio blocks specifically to improve multi-token inference latency is interesting and apparently effective. A real strength of the work is its good writing and precise mathematical exposition, with further analysis of computational cost in the Appendix. Furthermore, a comprehensive set of WER evaluation is presented on some well-known standard test sets, with decent results -- matching, sometimes bettering, existing baselines, but more notably, producing improvements in inference latency.\n\nWhile well-written and precise, the work is nonetheless often unclear in the bigger picture. Precise details can obscure the larger points, making the work slightly inaccessible. The first several pages feel like a rather long-winded exposition of the key aspects of the architecture. The reader really has to go over the proposal in detail to grasp how the proposed architecture is different from existing architectures. At first glance, it does not seem that original, e.g. reminds me of this study:\n\n@misc{deng2023adaptableendtoendasrmodels,\n      title={Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmax}, \n      author={Keqi Deng and Philip C. Woodland},\n      year={2023},\n      eprint={2302.08579},\n      archivePrefix={arXiv},\n      primaryClass={eess.AS},\n      url={https://arxiv.org/abs/2302.08579}, \n}\n\nHowever, what I finally realized is that the proposal leverages this decoupling specifically for multi-token inference. That is a very interesting property -- but one that is not explained very clearly, and in fact not even mentioned in the Abstract.\n\nIt seems that overall the work could, ideally, be presented more effectively and with greater impact by significantly compressing the description of the aspects of the architecture that closely follow existing standard architectures, and highlighting the novel aspects, e.g. the multi-token prediction mechanism.\n\nQUESTIONS:\nSome comments/questions, intended as clarifying or constructive feedback.\n\nL33: \"This phenomenon has been previously observed in the AUDIOENCODER [5] but never analyzed in the DECODER\". \nThough not quite the same context, similar analysis of layer-wise transformer attention scores was done e.g. in \n\n@inproceedings{Irie_2019, series={interspeech_2019},\n   title={Language Modeling with Deep Transformers},\n   url={http://dx.doi.org/10.21437/Interspeech.2019-2225},\n   DOI={10.21437/interspeech.2019-2225},\n   booktitle={Interspeech 2019},\n   publisher={ISCA},\n   author={Irie, Kazuki and Zeyer, Albert and Schlüter, Ralf and Ney, Hermann},\n   year={2019},\n   month=sep, collection={interspeech_2019} }\n\nFig. 1: what is the y-axis of each sub-plot? Tokens, I assume?\n\nL37, \"Based on these systematic attention patterns\": given that so far this is analyzing the situation for just a single sample, it seems like a stretch to say these are \"systematic\" patterns.\n\nL43, proposal of \"BLOCK DECODER\": I am wondering at this point if the \"block\" is conceptually the same as, or distinct from, the fact that a _block_ of tokens will be predicted. I.e., I could have a similar unit, call it a block, combining the two modules mentioned, that did not predict a block of tokens. I could also have a standard decoder, that does not distinguish text only from text+audio, that did predict a block of tokens. So upon first reading, I'm wondering whether the \"block\" in the name \"block decoder\" is actually helping the prediction of a \"block\" of tokens. Upon second reading, I see that the block decoder is indeed helping the prediction of a block of tokens -- could the authors make the presentation more easily accessible?\n\nL47, \"lesser\" --> \"fewer\"\n\nL59, \"or introducing new training objectives [20, 21]\". Great that the work is also considers neural transducers here and in the evaluation section. Transducers are more than a new training objective, though -- in fact, the transducer decoder, being purely text based, is quite strongly related to the proposed architecture in this submission. In a way a neural transducer is like the proposed architecture, but with a very simple Merger (the transducer \"joiner\"). This link could be discussed in more detail in a way that highlighted the merit of the proposed architecture.\n\nFig 2(b): these two blocks just seem like vanilla transformer stacks, one with self-attention, the other with cross-attention? If so, do we need to illustrate them? I note that the fonts in the figure, e.g. for \"Audio Encoder\" and \"Text Encoder\", are not consistent. Furthermore, the example, \"_B L OCK\" etc., propagates from Merger to merge by only apparently adding one new token, so this doesn't illustrate the key feature of the proposal, the prediction of multiple tokens?\n\nL111 \"dropouts\" --> \"dropout\"\n\n\"Section 3.2.2 Merger\": finally, getting to the equation (which should be numbered) at L124, and the text at L133, I get a sense for the details of the multi-token prediction. Since this is a key part of the proposal, ideally the basic approach here would be described more specifically, and much earlier in the paper, and as I suggested earlier, it should be made clearer how the multi-token prediction benefits from the BlockDecoder architecture.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}