Title: Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
Venue: NeurIPS 2025
Accepted: no
Keywords: automatic speech recognition

text to speech

speech

language models
Abstract: We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is typically cast in an offline manner: the model consumes the complete input sequence before generating the first output timestep. DSM instead models time-aligned streams with a decoder-only language model. By furthermore introducing delays between streams, and selectively feeding or sampling them, DSM provides streaming inference of arbitrary output sequences, from any input combination, making it applicable to many sequence-to-sequence problems. In particular, given a text and audio stream, automatic speech recognition (ASR) corresponds to the text stream being delayed, while the opposite gives a text-to-speech (TTS) model. We perform extensive experiments for these two major sequence-to-sequence tasks, showing that DSM provides state-of-the-art performance and latency while supporting arbitrary long sequences, being even competitive with offline baselines. We demonstrate DSM applications on https://delayed-stream-modeling.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DSM, a delayed streaming modeling approach applied to decoder-only architectures. Building upon the proposed multistream architecture (Moshi) by Défossez et al. (2024), the paper introduces a controllable delay for the streaming processing. The DSM  approach is evaluated on automatic speech recognition (ASR) and text-to-speech (TTS) tasks. The model is trained on English and French speech (automatically transcribed with Whisper) and tested on short- and long-form benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper applies the recent multistream architecture (Moshi) to streaming tasks, introducing a controllable delay.\n- The model achieves good results in terms of quality and is trained on large-scale settings \n\nWeaknesses:\n- Translation and, in general, crosslingual tasks are at the core of the sequence-to-sequence modeling, and historically represented more challenging tasks for shifting from attention-based encoder-decoder architectures to alternative architectures such as transducers [1,2] for the streaming processing. However, such tasks are not addressed in the paper, which addresses only monolingual and monotonic tasks such as ASR and ST, strongly limiting its impact and a sound validation of the approach.\n- The paper is not well written in general, and often the final message (or the purpose) of the paragraphs is not clear. For instance, the introduction seems very fragmented, starting with describing the streaming sequence-to-sequence task, then moving to offline architectures, then decoder-only architectures, then neural compression algorithms without a clear flow. Instead, many relevant works in streaming processing in ASR [3], MT [4], and ST [5] are not mentioned. \n- The paper points out that current models operate by regularly sampling the audio or video modality, preventing their applications to continuous transcription or translation (lines 40-42), but then introduces a method operating at constant framerates (lines 48-49), which it is not clear how it solves problem. Moreover, it is stated that \"Yet, a major limitation of these decoder-only is their incompatibility with streaming. First, their prefix-based formulation requires access to the full input sequence before generation, which prevents real-time inference\" while previous work [6] showed, instead, its feasibility for streaming applications.\n- The multistream architecture adopted in the paper is not a novel component of the paper, as it has been proposed by Defossez et al. (2024). Therefore, the novelty part is in introducing the delay component, which is pretty limited. Indeed, the \"delay conditioning\" is not a novel concept, as most streaming and simultaneous models allow for partial or complete control over the delay of the model (e.g., the simplistic wait-k policy [7] that waits for *k* words before starting the output emission, and *k* is controllable by the user). Moreover, as already mentioned, the feasibility of adopting decoder-only architectures has already been addressed in previous work [6].\n- The Related Works, and, in particular, the Streaming Sequence-to-Sequence Learning paragraph, contains mostly incorrect information. First, the SeamlessM4T family is not a family of streaming models (but, instead, offline models), but they were cited (multiple times) in the paragraph. SeamlessStreaming, which is a subsequent work focusing on streaming processing, would have been more appropriate. Moreover, other works [8] successfully used Seamless with a simultaneous policy, and there is no reason for claiming that they are \"incompatible with batching\", which, moreover, does not make sense as we are dealing with streaming applications, which is the task of processing a continuos stream of information, without any batching. Lastly, there exist streaming models that are not modality specific (speech-to-text only or text-to-speech only), conversely to what has been affirmed by the authors (lines 66-67), such as SeamlessStreaming and Streamspeech, already cited by the authors. \n- The performance of the proposed architecture is only compared against one system for ASR (WhisperStreaming), which is rather insufficient, considering the works mentioned in the related works. Moreover, there is no detail about WhisperStreaming, which is the only relevant streaming ASR baseline considered in the paper, and the arbitrary choice of using 2.5s as a hyperparameter is not motivated. Likewise, the choice of the baseline for TTS is not motivated, and the baselines are not streaming, as latency is only reported for the proposed method (Table 5). Therefore, the comparisons are not sound for a streaming processing paper, for which the results discussion parts are also particularly short.\n\n[1] C. Yeh, J. Mahadeokar, K. Kalgaonkar, Y. Wang, D. Le, M. Jain, K. Schubert, C. Fuegen, and M. L Seltzer, “Transformer-transducer: End-to-end speech recognition with self-attention,” 2019.\n\n[2] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and RNN-T loss,” in Proc. ICASSP, 2020, pp. 7829–7833.\n\n[3] Dominik Macháček, Raj Dabre, and Ondřej Bojar. 2023. Turning Whisper into Real-Time Transcription System. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, pages 17–24, Bali, Indonesia. Association for Computational Linguistics.\n\n[4] Javier Iranzo-Sánchez, Jorge Civera, and Alfons Juan. 2022. From simultaneous to streaming machine translation by leveraging streaming history. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long\nPapers), pages 6972–6985, Dublin, Ireland. Association for Computational Linguistics.\n\n[5] Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3692–3707, Bangkok, Thailand. Association for Computational Linguistics.\n\n[6] Xue, Jian & Wang, Peidong & Li, Jinyu & Sun, Eric. (2023). A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability. 1-7. 10.1109/ASRU57964.2023.10389799.\n\n[7] Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019. STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3025–3036, Florence, Italy. Association for Computational Linguistics.\n\n[8] Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation. In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), pages 72–79, Bangkok, Thailand (in-person and online). Association for Computational Linguistics.\n\nQUESTIONS:\n- Why do the authors cite Vaswani et al. (2017) for decoder-only Transformer architectures (Introduction)? \n- Why do the authors only use automatically transcribed speech as training data while gold-labeled data exists (at least, for high-resource languages as English)?\n- What is the purpose of reporting the results on both short- and long-form ASR if no latency measure is presented?\n- Why is batching a desired feature for streaming processing?\n- Why have you chosen monotonic tasks such as ASR and TTS for validating this framework?\n\nLIMITATIONS:\nThe limitations mentioned in the paper are rather short and do not address any technical limitations of the work, such as the languages analyzed, the choice of certain baselines, and, most importantly, the decision to restrict the paper's scope to ASR and TTS.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an innovative Delayed Streams Modeling (DSM) method and demonstrates its application in the realms of Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks. This work offers a valuable streaming solution for a variety of sequence-to-sequence problems by effectively leveraging the temporal delays between input and output streams.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. The paper is well-structured and clearly written, supported by comprehensive experiments and comparisons. The authors' intention to open-source the code is commendable and will facilitate reproducibility and further research.\n2. The proposed idea is innovative and represents a novel approach to the problem.\n3. The ASR performance is strong, achieving results that are even comparable to non-streaming TTS models.\n\n\nWeakness\n1. The details of the DSM-TTS implementation, particularly regarding the lookahead and action streams (Sec 3.3), could benefit from further clarification. \n      a. For instance, Figure 3 lacks clear indications of which streams correspond to which functions. Additionally, the speaker conditioning approach appears somewhat confusing. \n      b. The reason using a fine-tuned mini-codec for each speaker is not well-explained. If this approach is indeed used, it may imply that the model is not designed to handle unseen speakers. \n      c. The implementation of dialogue generation and the provision of multi-speaker embeddings are not clearly explained in this section.\n2. The Real-Time Factor (RTF) of 2.7 is relatively high, especially for a TTS model intended for streaming applications. This may limit its suitability for real-time practical applications. The evaluation of F5TTS in the appendix reports an RTF of 14.7, which seems inconsistent with typical values observed in the literature (usually less than 0.5). It is possible that there may be some misunderstanding or misinterpretation of the RTF calculation.\n3. Some references, particularly in the dialogue generation section, are missing.\n\n[1] Zhang, L., Qian, Y., Zhou, L., Liu, S., Wang, D., Wang, X., ... & Zeng, M. (2024). CoVoMix: Advancing zero-shot speech generation for human-like multi-talker conversations. Advances in Neural Information Processing Systems, 37, 100291-100317.\n\n[2] Lu, H., Cheng, G., Luo, L., Zhang, L., Qian, Y., & Zhang, P. (2025). SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation. arXiv preprint arXiv:2501.00805.\n\n[3] Ju, Z., Yang, D., Yu, J., Shen, K., Leng, Y., Wang, Z., ... & Li, X. (2025). MoonCast: High-quality zero-shot podcast generation. arXiv preprint arXiv:2503.14345.\n\nQUESTIONS:\nIn addition to the points raised in the weakness section, I still have several further questions:\n\n1. The pre-training data employ \"whisper-transcribed\" data, yet the ASR performance exceeds that of Whisper-large v3. What factors contribute to this improvement, and why does the model outperform the Whisper-large v3 despite using its transcriptions for pre-training?\n\n2. The DSM-TTS is reported to have a delay of 2 seconds (as mentioned in the paper on line 185), which may be considered lengthy for real-time human-interactive applications. However, the latency reported in Appendix Table 5 is 185 milliseconds. Could you clarify the distinction between the latency mentioned here and the actual latency experienced in real-world applications?\n\n3. The throughput of the model is notably high. What specific factors or optimizations contribute to this high throughput??\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper introduces Delayed Streams Modeling (DSM), a framework for streaming speech-text sequence-to-sequence learning. DSM builds on an autoregressive backbone to process streaming input and generate output sequences, both operating at a constant frame rate to enable efficient batching. The output sequence is delayed relative to the input, allowing access to additional context. On ASR, DSM achieves a word error rate (WER) close to the best offline model on the Open ASR Leaderboard with just a 2.5-second delay. On TTS, it outperforms baseline non-commercial models in synthesis quality.

STRENGTHS AND WEAKNESSES:
# Strengths
1. DSM achieves a WER within 0.2% of the best offline model on the Open ASR Leaderboard, while maintaining a 2.5-second delay in a streaming ASR setup.
2. The architecture naturally supports efficient batching during streaming inference, as demonstrated in Table 4 and Figure 5.

# Weaknesses
1. Table 2 omits models like Parakeet-TDT-v2, which also support long-form ASR over several hours, limiting the completeness of comparison.
2. The paper lacks a clear differentiation from prior architectures such as Moshi and Hibiki, which also use streaming input and delayed output. A more thorough discussion of DSM’s unique contributions is needed.
3. The paper does not analyze how computational latency scales with batch size in the ASR setting.
4. DSM has not been evaluated on streaming speech translation, a more challenging task involving word reordering and more complex reasoning.

QUESTIONS:
1. Have you considered initializing the model with a pretrained LLM backbone instead of training from scratch? 
2. In Equation (6), the second equation should use $\tilde{Y}_t$ instead of $\tilde{Y}_1$.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Delayed Streams Modeling (DSM), a flexible framework for streaming multimodal sequence-to-sequence learning. DSM employs a decoder-only language model to process time-aligned token streams across modalities, with controlled delays between streams to balance quality and latency. It unifies tasks like automatic speech recognition (ASR) and text-to-speech (TTS) under a single architecture, enabling bidirectional generation. \n\nExtensive experiments show DSM achieves state-of-the-art performance on both ASR and TTS, competing with offline baselines while supporting streaming inference, long-form sequences, and real-time latency control via delay conditioning. The authors provide detailed implementation details, experimental results across diverse datasets, and plans to release code, models, and evaluation data.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Novel Streaming Formulation: DSM addresses a critical gap in streaming seq2seq learning by introducing 可控 delays between aligned streams, enabling real-time inference without sacrificing performance. This contrasts with existing streaming models that often trade off batching, symmetry, or quality.\n2. Unified Architecture: By leveraging discrete token representations for audio (via neural codecs) and text, DSM unifies ASR and TTS into a single decoder-only framework, supporting bidirectional generation—a significant advance over modality-specific architectures (e.g., Tacotron for TTS vs. LAS for ASR).\n3. Strong Empirical Performance: Across short and long-form ASR tasks, DSM outperforms streaming baselines (e.g., Whisper-Streaming) and matches top offline models (e.g., Parakeet-TDT-V2) with an average WER of 6.3%. For TTS, it achieves the lowest WER among open-source models, with competitive speaker similarity to commercial systems (e.g., ElevenLabs).\n4. Rigorous Experimentation: The authors evaluate across diverse datasets (e.g., LibriSpeech, TED-LIUM, custom long-form dialogs) and provide comprehensive metrics (WER, latency, subjective quality), enhancing result credibility.\n\n## Weaknesses\n1. Similar methods: For Delayed Streaming Modeling (DSM),  it seems that there have been similar explorations in sequence modeling before [1][2], which to some extent limits the originality of this article. \n2. Model Size: DSM-ASR uses a 3B-parameter backbone, and DSM-TTS is 3.7B parameters, which may restrict deployment on resource-constrained devices—an important consideration for streaming applications.\n\n\n[1] Gao, Heting, et al. \"LUCY: Linguistic Understanding and Control Yielding Early Stage of Her.\" arXiv preprint arXiv:2501.16327 (2025).\n\n[2] Ding, Ding, et al. \"Kimi-audio technical report.\" arXiv preprint arXiv:2504.18425 (2025).\n\nQUESTIONS:\n## Questions\n1. The TTS architecture uses an action stream and a lookahead text stream. Were there ablation studies comparing these components to simpler alternatives (e.g., fixed pauses)? How do they impact generation fluency?\n2. The author should conduct a more comprehensive investigation of existing work to demonstrate the differences and advantages of the method proposed in this article compared to existing methods.\n3. The paper mentions using Mimi codec for audio tokenization. How sensitive is DSM’s performance to the choice of codec (e.g., compared to SoundStream)?\n4. Have authors conducted some ablation experiments on the model size to verify its usability under the condition of a small number of parameters?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}