Title: SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Simultaneous Speech Translation

Streaming Text-To-Speech
Abstract: Simultaneous Speech Translation (SimulST) enables real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Existing systems struggle to balance translation quality, latency, and semantic coherence, particularly in multilingual many-to-many scenarios where divergent read/write policies hinder unified strategy learning. In this paper, we present SimulMEGA(Simultaneous Generation by Mixture-of-Experts GAting), an unsupervised policy learning framework that combines prefix-based training with a Mixture-of-Experts refiner to learn effective read/write decisions in an implicit manner, without adding inference-time overhead. Our design requires only minimal modifications to standard transformer architectures and generalizes across both speech-to-text and text-to-speech streaming tasks. Through comprehensive evaluation on six language pairs, our 500 M-parameter speech-to-text model outperforms the Seamless baseline, achieving under 7% BLEU degradation at 1.5 s average lag and under 3% at 3 s. We further demonstrate SimulMEGA’s versatility by extending it to streaming TTS via a unidirectional backbone, yielding superior latency–quality trade-offs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a learned policy for simultaneous speech translation (SimulST) called SimulMEGA. SimulMEGA combines a prefix-based SimulST policy, which is trained in an unsupervised way, with Mixture-of-Experts (MoE) to improve the system's performance. The framework is applied to both speech-to-text (S2T) and text-to-speech (TTS). The SimulST model is trained in two steps: an offline pretraining and a simultaneous training with a standard prefix loss and a novel refiner loss for the MoE. The performance is validated on six languages and popular benchmarks and ablation studies are presented.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- It is the first time that the MoE approach is applied to SimulST, to the best of my knowledge.\n- The architectural modifications required to apply this method are minor, facilitating its adoption.\n- The results of the proposed architecture are promising, outperforming the tested baselines.\n- The ablation study clarifies the importance of some experimental choices.\n\nWeaknesses:\n- No results reported using LAAL, which is the most used latency metric for SimulST (e.g., see the last IWSLT Evaluation Campaign editions from 2024 and 2025)\n- Insufficient baselines and lack of clarity in their presentation: The paper compares with \"Seamless\", but it is not clearly stated that the original SeamlessM4T models (both Medium and Large-v2) are different from the simultaneous version of Seamless, StreamingSeamless [1], which I suppose is the model adopted for the comparisons in Figure 3. Moreover, the wait-k policy (which is one of the two terms of comparisons for SimulST) has been outperformed for many years now (see also the related works), and is not a strong term of comparison. The choice of the remaining baseline, Dig-SST, has not been justified, and it is unclear why recent state-of-the-art SimulS2TT works (already cited by the authors), such as models tailored for SimulST tasks and publicly available (such as StreamSpeech [2] and NAST-S2X [3]) and policies applicable to models without any retraining (such as EDAtt [4] and AlignAtt [5]), have not been used for this comparisons. It is good to report additional baselines, such as wait-k, but the set of baselines should be carefully selected and clearly explained to ensure the validity of the paper's claims. Likewise, results for the SimulS2ST task lack comparison with the aforementioned works (StreamSpeech and NAST-S2X), which also cover the speech-in-output modality. \n- It is not clear if the system actually processes long-form speech (akin to real-world streaming processing) or if the speech has been previously segmented into shorter segments. It seems that the authors used CoVoST and FLEURS, which are human-segmented, but no details about this are provided. Following recent work [6], this should be clarified to position the paper's contribution appropriately.\n- While it is true that the approach requires a minimal modification in the architecture, it requires an ad-hoc simultaneous training stage to be applied. Moreover, the pretraining stage, despite being offline, already entails some choices for making the model more efficient during inference (chunk-AR and NAR), and even if desirable in general, it is not the standard way offline S2TT and S2ST models are trained, meaning that the same findings might be not applicable when starting from common, publicly-already available models.\n\n[1] Barrault, L., Chung, Y.A., Meglioli, M.C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.A., Ellis, B., Elsahar, H., Haaheim, J. and Hoffman, J., 2023. Seamless: Multilingual Expressive and Streaming Speech Translation. arXiv preprint arXiv:2312.05187.\n\n[2] S. Zhang, Q. Fang, S. Guo, Z. Ma, M. Zhang, and Y. Feng, “StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning,” 2024.\n\n[3] Zhengrui Ma, Qingkai Fang, Shaolei Zhang, Shoutao Guo, Yang Feng, and Min Zhang. 2024. A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1557–1575, Bangkok, Thailand. Association for Computational Linguistics.\n\n[4] S. Papi, M. Negri, and M. Turchi, “Attention as a Guide for Simultaneous Speech Translation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 13 340–13 356.\n\n[5] S. Papi, M. Turchi, and M. Negri, “AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation,” in INTERSPEECH 2023, 2023, pp. 3974–3978.\n\n[6] Papi, S., Polak, P., Macháček, D. and Bojar, O., 2025. How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System?. Transactions of the Association for Computational Linguistics, 13, pp.281-313.\n\nQUESTIONS:\n- I suggest adding LAAL as a metric, since it is now the standard for SimulST evaluation and avoids rewarding over-generating systems.\n- Why did the author choose the wait-k and the DigSST policies for comparison while excluding state-of-the-art works such as StreamSpeech, NAST-S2X, EDAtt, and AlignAtt? \n- What is the framework used for evaluation? I suggest adding to the inference details (and, maybe, in the limitations) that the model has been tested on short-form and human-segmented test sets such as CoVoST and FLEURS, thereby simulating only part of a real-world SimulST model.\n- How much does the choice of the mixed chunk-AR and NAR encoder affect the final results?\n\nLIMITATIONS:\nLimitations are really strict to the point. I suggest elaborating a little on the drawbacks mentioned in lines 299-301.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SimulMEGA (Simultaneous Generation by Mixture-of-Experts Gating), a novel framework for Simultaneous Speech Translation (SimulST), which enables real-time cross-lingual communication by integrating speech recognition and machine translation under stringent latency constraints. Unlike traditional offline systems, SimulMEGA processes streaming audio input to generate translations incrementally, mimicking human interpreters. The framework addresses challenges in balancing translation quality, latency, and semantic coherence, particularly in multilingual many-to-many translation scenarios. SimulMEGA combines prefix-based training with a Mixture-of-Experts (MoE) refiner module to learn implicit read/write policies without introducing inference-time overhead. It employs a hybrid streaming speech encoder (combining chunk-wise autoregressive and non-autoregressive blocks) and a standard transformer-based text decoder, augmented by a global routing gate and an MoE refiner active only during training. The framework supports both speech-to-text and text-to-speech (TTS) streaming tasks, achieving state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Simultaneous Speech Translation is a very valuable and promising research topic.\n- The paper presents a high-quality technical contribution, supported by rigorous evaluation across six language pairs.\n- The paper is well-structured, with clear descriptions of the SimulMEGA framework.\n\nWeaknesses:\n- While the evaluation on six language pairs is comprehensive, the paper does not provide detailed analysis of performance across specific language pairs or linguistic phenomena (e.g., low-resource languages or complex syntactic structures). This limits insight into the framework’s robustness across diverse linguistic contexts. Additionally, the TTS evaluation is less detailed than the speech-to-text results.\n- The performance of the proposed method in Simultaneous Speech-to-Speech Translation should be further evaluated on the CVSS benchmark which is commonly used in previous work.\n\nQUESTIONS:\n- The paper describes the MoE refiner as comprising prefix and global experts, with merging weights determining the read/write policy. Could the authors provide more details on how these weights are learned during training?\n- The paper references prior policy-based frameworks but does not explicitly compare SimulMEGA’s policy learning approach to these methods. How does the unsupervised MoE-based policy learning differ from, say, the CTC-based policy or the attention-based policy.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SimulMEGA, an unsupervised policy learning framework for Simultaneous Speech Translation. It introduces a MoE Refiner and a routing gate that jointly enable the model to implicitly learn read/write policies without adding inference-time overhead. The proposed system supports both speech-to-text and text-to-speech streaming tasks and demonstrates good latency-quality trade-offs across six languages. The authors provide extensive experiments, including comparisons with SeamlessM4T, streaming TTS baselines, and simultaneous speech-to-speech translation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The proposed method achieves SOTA performance on the SimulST task, supported by extensive experiments and strong comparisons against baselines.\n- The paper further extends the evaluation to streaming TTS and simultaneous speech-to-speech translation tasks, demonstrating the versatility of the proposed framework.\n- The method is relatively simple and efficient, requiring minimal architectural modifications to existing systems while still providing significant performance improvements.\n- The supplementary materials include comprehensive demos and audio samples.\n\n### Weaknesses\n- The clarity of the paper needs improvement. Many important concepts (such as $h_{\\text{prefix}}$ and $L_{\\text{offline}}$) are not clearly explained, which makes the paper harder to follow.\n- Since the core contribution of the paper is the Mixture of Experts (MoE) refiner, the authors should provide more motivation for introducing it. Why is MoE necessary? How does it compare to native solutions without MoE? The use of gating mechanisms is also not particularly novel and would benefit from a more in-depth justification.\n- The implementation details of the streaming TTS component need to be described more thoroughly.\n\nQUESTIONS:\n- What specific problems does the introduction of the MoE refiner solve, and how does it contribute to the final performance improvement? It would be helpful if the authors could provide additional experiments to explicitly demonstrate the effectiveness of the MoE module.\n\n- The authors are encouraged to provide a more detailed explanation of the various loss functions introduced in the model, including their exact formulations. Additionally, key concepts such as $h_{\\text{prefix}}$ and other latent representations should be clearly defined.\n\n- More detailed implementation descriptions for the Streaming TTS component would be beneficial, particularly regarding system design, latency control, and how streaming is achieved in practice.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SimulMEGA, a policy learning framework for simultaneous speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). It introduces a mixture-of-experts (MoE) refiner on top of a standard Transformer architecture to enable adaptive read/write decision-making. The framework also incorporates a joint loss function designed to balance translation quality with latency-aware decision accuracy. Experiments on CoVoST2 and FLEURS across six languages demonstrate that SimulMEGA outperforms Seamless in multilingual settings.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. SimulMEGA outperforms Seamless in both offline translation quality and the quality-latency trade-off for simultaneous translation, despite being a smaller  multilingual model.\n2. The framework naturally extends to simultaneous text-to-speech synthesis, achieving near-offline synthesis quality with a latency of approximately 1 second.\n\n# Weakness\n1. The use of Average Lagging (AL) is problematic, as it has been shown to be not robust to over-generation [1]. A more reliable alternative is Length-Adaptive Average Lagging (LAAL), which has been adopted as the standard latency metric in recent IWSLT evaluations.\n2. Missing strong baseline: The paper omits AlignAtt [2], one of the most effective streaming policies that can be directly applied to offline ST models. Including it would provide a more comprehensive and fair comparison.\n3. Lack of experimental details: The paper does not specify the threshold configurations used during training and inference for read/write decision.\n\n[1] Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation (Papi et al., AutoSimTrans 2022)\n\n[2] AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation (Papi et al., InterSpeech 2023)\n\nQUESTIONS:\n1. What is the rationale behind keeping the non-autoregressive (NAR) layers from Whisper? Is this a trade-off between translation quality and computational efficiency?\n2. The score normalization in Equation (7) is somewhat unclear. Could you elaborate on the rationale behind this formulation?\n3. How is inference performed for the SimulMEGA-TTS model?\n4. Which translation API is used to obtain the reference translations?\n5. Lines 280–281 mention router scores in Figure 5(b), but I couldn’t find them in the figure. Could you clarify?\n6. What exactly does $L^{\\text{trunc}}$ refer to in line 285?\n\nLIMITATIONS:\nThe model is not tested on unbounded speech, where input streams last minutes or even hours, as in real-world applications. Evaluating on such long-form inputs is essential for assessing robustness and practical usability in realistic streaming settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}