SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
Authors: Chenyang Le
Bing Han
Jinshun Li
Chen Songyong
Yanmin Qian
Venue: NeurIPS 2025 poster ()
Keywords: Simultaneous Speech Translation
Streaming Text-To-Speech
Primary area: applications

Abstract: Simultaneous Speech Translation (SimulST) enables real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Existing systems struggle to balance translation quality, latency, and semantic coherence, particularly in multilingual many-to-many scenarios where divergent read/write policies hinder unified strategy learning. In this paper, we present SimulMEGA(Simultaneous Generation by Mixture-of-Experts GAting), an unsupervised policy learning framework that combines prefix-based training with a Mixture-of-Experts refiner to learn effective read/write decisions in an implicit manner, without adding inference-time overhead. Our design requires only minimal modifications to standard transformer architectures and generalizes across both speech-to-text and text-to-speech streaming tasks. Through comprehensive evaluation on six language pairs, our 500 M-parameter speech-to-text model outperforms the Seamless baseline, achieving under 7% BLEU degradation at 1.5 s average lag and under 3% at 3 s. We further demonstrate SimulMEGA’s versatility by extending it to streaming TTS via a unidirectional backbone, yielding superior latency–quality trade-offs.

SimulMEGA: MoE Routers are Advanced Policy Makers for Simultaneous Speech Translation
Chenyang Le nethermanprods jtu edu. cn
Bing Han hanbing970sj edu. cn
Jinshun Li
Songyong Chen chensy30@sjtu. edw.eijj.uudddc.oo. edu. en
Yanmin Qian aanmin juu. edu. on
muzi seci-jingshundajjte edu, cn
Auditory Cognition and Computational Acoustics Lab MoE Key Lab of Artificial Intelligence, AI Institute School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Abstract
Simultancous Speech Translation (SimulST) enables real-time cross-lingual com- munication jointly optimizing speech recognition and machine translation under strict latency constraints. Existing systems struggle to balance translation quality, latency, and semantic coherence, particularly in multilingual many-to-man -to-many narios where divergent read/write policies hinder unified strategy learning. In this paper, present
Generation by Kexture-of-Experrt GAting), an unsupervised policy learning framework that combines prefix-based training with Mfxpurt-of-Expperts refiner to learn effective read/write decisions in an implicit manner, without adding inference-time overhead. Our design requires only minimal modifications to standard transformer architectures and generalizes across both speech-to-texl and text-to-speech streaming tasks. Through comprehen- sive evaluation on six language pairs, our 500 paraameter speech-to- to-text model outperforms the Seamless baseline, achieving under 7 % BLEU degradation at 1.5 average lag and under 3% at s. We further demonstrate SimulMEGA's versatility by extending it to streaming TTS with unidirectional backbone, yielding superior latency-qualie trade-offs.
Introduction
Simultaneous Speech Translation (SimulST) addresses the critical need for real-time cross-lingual communication by jointly optimizing speech recognition and machine translation under strict latency constraints. Unlike conventional offline systems that process complete utterances, SimulST operates on streaming audio input, incrementally generating translations while simultaneously decoding ongoing speech- capability mirroring human interpreters' cognitive processing. This technology enables transformative applications in international diplomacy, live media localization, and low- latency dialogue systems. However, it faces fundamental challenges in reconciling three competing objectives: translation quality, computational latency, and semantic coherence, exacerbated by fragmented acoustic patterns and partial contextual dependencies.
Recent progress combines adaptive segmentation strategies 1. 2, 3] with streaming architectures like RNN-T transducers[4, 5, 6] and Monotonic Attention mechanisms(7, 8], alongside policy-based based
"Corresponding Author
"The code can be found in httpp::/ggtcuu..cco//eeeeeeeeeee http://////tttaoooo/////ottttttp
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Lofflie
Hahai
Pooling
MoE
rreinerr
protter
Hellien
0000000
Router
Text
p pruuter
True Write
Decoder
SSE
O
Crrefix
SSE
Text
Decader
Norte 0000
Othec
Text
SSE
SSE
Decoder
*esojets ne +0i00
Streaming Input
False Read
(a) Offlin Training
(b) Prefix-based Simultaneous Training
(c) Simultaneous Inference
Figure Overview of the training and inference paradigm of SimulMEGA. SimulMEGA is composed of a Streaming Speech Encoder, text decoder, an MoE routing Gate(Router), and an MoE refiner. In the first stage, the model pre-trained on Cofliine In the second stage, the model is trained on combination of Loftline: Lrrefix and Crefiner, SSE denotes Streaming Speech Encoder.
decision frameworks[99, 10, 11, 12, 13, 14]. Despite these advances, current systems struggle to achieve human-like efficiency-accuracy trade-offs across diverse linguistic contexts and acoustic con- ditions. A particularly underexplored challenge lies in multilingual many-to-many translation, where divergent read/write policies across language pairs complicate the learning unified operational strategies.
This paper introduces SimulMEGA, an unsupervised policy learning framework that synergizes prefix-based training with Mixture-off-xppeee (MoE) mechanisms to address these limitations. Our approach enables high-performance multilingual SimulST through three key innovations: (1) auxiliary MoE refiner module enabling implicit policy learning without inference-time overhead, (2) Minimal architectural modifications to standard transformers for broad applicability, and (3) A unified framework supporting both speech-to-text and text-to-speech streaming tasks. Experimental results demonstrate state-of-the-arr performance across six languages, with our 500M-parameter speech-to-ted model outperforming ScamlessM4T[15] by achieving <7% BLEU degradation at 1.5s average lagging (AL) and <3% at 3s AL. What's more, we extended our methods to streaming text-to-speech (TTS) task via Cosy Voice2's unidirectional backbone 16], the system achieves superior latency control compared to conventional interleaved streaming approaches.
Our principal contributions are as follows.
Introduction of SimulMEGA: An unsupervised policy learning framework that combines prefix- based training with a Mixture-off of--xperts (MoE) refiner module, enabling efficient and effective simultaneous speech translation without adding inference-time overhead.
Multilingual Advancement: SimulMEGA achieves SOTA performance for quality-latency trade- offs in many-to-many translation, demonstrating cross-lingual robustness through comprehensive 6-language evaluation.
Universal Streaming Framework: SimulMEGA supports both speech-to-text and text-to-speech streaming tasks within single framework, making it broadly applicable and easily integrable into existing models for multilingual SimulST tasks.
General Conversational System: The integrated solution combining low-latency speech-to-text and text-to-speech conversion, validated through real-world dialog interpretation scenarios.
2 Related Works
SimulST Early works in SimulST focus on finding the input segmentation boundary and then applying variant of the wait-k policy[1, 17, 2. 3]. Later work seeks to find a more sophisticated dynamic policy. [18] designs learnable matrix between source and target
an integrate-and-
fire(IF) policy, and [19, 20] use attention score for a similar approach. [11] proposes policy based on divergence between the prefix and full input. [12] employs an extra CTC[21] head for
policy control. Another approach includes using inherently streaming architecture like monotonic

---

attention[7, 22, 8, 23], transducer[4, 5, 6] and Hidden Markov[24). [25] explores non- con-autoreeresive generation for SimulST.
Streaming TTS Many works TTS claim to be streamable(26, 27], yet they not optimized for fine-graine fie-grained(chunt or token level) streaming and suffer from degraded quality due to a lack of future information. [28] adapts streaming model from an offline model via restricted attention. [29] uses a fixed number text tokens for look-ahead. [30] processes each text chunk in a separate AR loop with look-ahead. [16] interleaves text and speech tokens fixed ratio, which potentially incurs larger latency.
3 SimulMEGA: Simultaneous Translation via Mixture-of-Experts Routing
In this section, we introduce SimulMEGA (Simultaneous Generation by Mixture-of- Efpertssss GAting), an innovative framework that effectively converts an offline autoregressive model into simultaneous system with minimal computational overhead and negligible performance degradation.
The core principle of simultaneous training lies in enabling the model to autonomously determine whether the currently available input suffices for generating the next output token. Previous ap- proaches typically rely either artificial policy signals, which constrain the model's self-learning capacity, or architectural modifications that compromise generation quality. In contrast, our frame- work leverages Mixture-of Experts (MoE) architecture achieve unsupervised policy learning while maintaining an identical structure to the offline model, thereby preserving translation per- formance. Our approach employs prefix-based training strategy that closely mimics real-world streaming inference scenarios.
3.1 Architecture Overview
Figure illustrates the overall architecture of our SimulMEGA framework, which consists of four key components: (1) a streaming speech encoder, (2) a text decoder, (3) global routing gate, and (4) Mixture-of- (MoE) refiner module.
The streaming speech encoder and text decoder adopt the Transformer architecture, maintaining com- patibility with conventional speech-to-text translation (S2TT) systems. As illustrated in Figure 2(a), the encoder combines chunk-wise autoregressive (Chunk-AR) blocks with con-autoregressive (NAR) blocks in a hybrid design. The Chunk-AR blocks optimize inference efficiency through cached key-value mechanism, while the NAR blocks preserve translation quality by capturing global context. To handle streaming inputs lacking an explicit end-of-sequence (EOS) token, we prepend learnable end-of-stream (EoSt) flag (a binary embedding) to the NAR blocks' input, signaling whether the current chunk terminates the audio stream. The text decoder follows standard autoregressive Transformer architecture.
The core innovation lies in the routing gate and MoE refiner, which enable unsupervised learning of simultaneous translation policies without architectural modifications. The MoE refiner follows transformer-like architecture and shares language model head with the text decoder, which combines prefix information and global information and predicts the target translation sequence. only activated during training, introducing zero additional computational overhead during inference.
3.2 Unsupervised Policy Learning.
This section introduces the Mixture Experts (MoE) refiner module, whose architecture is shown in Figure 2(b). The module comprises Niefiner blocks, each containing two specialized experts: prefix expert (Ep) and a global expert (Eg). The merging weights of these experts implicitly define a policy that determines whether the current input prefix contains sufficient information for generating the target token. Besides the dual-experts module, each block also contains a previous-outge attention module (similar to self-attention) and MLP module, which is analogous to standard transformer decoder block.
Global Routing Gate For decision consistency, each refiner layer employs a shared global gate implemented as two-layer MLP with a Sigmoid head. The gate projects the text decoder's final hidden state (Odee) into scalar value p E [0, 1). which determines the expert weights: PRouter =
and PRouter 1 p.

---

pref (yr Fes, hpaefix Hglobbll
LM Head
NAR Blocks
MLP
prouter
Cached
prefix
Chunk AR Blocks 20
Computted
POA
Fooh Flag
X retiner
Previous
Input
dec
Odee
Halobal
Speech
Chonk
(a) Streaming Speech Encoder
(b) MoE Refiner
Figure 2: (a) Structure and inference example of the streaming speech encoder of SimulMEGA, It
comprises 20 chunkwise autoregressive (Chunk-AR) blocks and aan-autorrgresiveNNAAR blocks. After each read, the Chunk-AR blocks only compute the new chunk while the NAR blocks recompute the whole sequence. An End-of-Streat (FoSt) flag is given before NAR blocks. (B) The structure of
the MoE Refiner, in which the gate decides over the mixture proportion of the Prefix Expert and the Global Expert. This proportion reflects the model's confidence in the prefix sequence, leading to natural read/write policy. The self-attention module is replaced by previous output attention(POA) module to prevent global information leakage.
Dual-Expert Architecture As shown in Figure 1(b). during training, the encoder processes both the complete offline speech and randomly truncated prefix. producing corresponding hidden states h ooffine and hprefix Then the gate and the dual-expers architecture will decide which hidden states to use. While the router typically favors the more informative J offfine we introduce an information bottleneck to balance this preference. Specifically, we apply temporal mean-pooling to hoffline yielding a condensed global embedding Hislobll for F.g. Fg is a two-layer MLP module whose calculation is as follows:
X
LayerNorm Xx
in
X;
Fx
W Eg
ReLU
(Wfr
x Halobal)
where 2 denotes the i-th position's 's input, I; represents vector concatenation, WE, projection weights (biases omitted). The prefix expert E, employs standard cross-attention
in/out are learnable
W MHA (W x W prefix
WF prefix)
Where MHA denotes multi-head attention. The final output combines both experts' contributions through a gated residual connection:
out = in Router i,Fo
pluutee F,
Ep
Global Information Leakage In standard transformer architectures, the self-attention mechanism inherently leaks global information across the sequence. This poses a critical issue for our design: when the gate assigns PRouter 0 for certain position, the hidden state at that position may still access global context via self-attentio,, thereby undermining the intended expert specialization. To enforce strict isolation of global information, we replace self-attention with a previous-ouue output attention mechanism. Instead of attending hidden states within the same layer, each position only attends to the decoder's prior outputs(o dee effectively preventing unintended information flow while preserving sequential dependencies.
Training Protocol SimulMEGA leverages pre-trained offline S2TT model as its foundation. The training process consists of two stages:
1. Offline Pretraining: We first train the model with the standard offline S2TT objective cotline (incorprraing streaming chunk masks the encoder) until convergence, as illustrated in Figure 1(a).

---

2. Simultaneous Training: We then introduce two additional loss functions to enable simultaneous capabilities as illustrated in Figure 1(b):
Crehner
Σ
log pr (wilvvs, hrrefx Hslobbl)
(3)
Crrefix
logpter(s yess prrofix)
(4)
where pde and pdee are the output distribution of the MoE. refiner and the text decoder. À is pre-defined hyperparameter that restricts the losses to the confident position. Crefier Creftinn.rrefner learns the read/write policy while [prefix strengthens the prefix-based translation capability, this stage, the total loss is the weighted sum of the above three losses:
C Loffiiue
Crefiuer Wpp Cprefi
In our experiment, we set Up 0.2 to prioritize the offline training.
Inference Policy During inference, SimulMEGA employs a straightforward threshold-based policy:
Write
Pt, < À
Action
(6)
Read
otherwise
where Pt,i is the gating score at input time / and target position i.
3.3 Score Distribution Control
The raw output scores from the routing gate lack inherent interpretabilityy as they emerge solely from neural network optimization through gradient descent. These scores exhibit inconsistent statistical properties, both mean and variance, between different tasks, language pairs, and training data. This variability poses significant challenges during inference, necessitating task-specific language- specific threshold tuning that fundamentally compromises the generalizability of our approach. To overcome this limitation, we introduce the following techniques:
Score Normalization We introduce a heuristic that aligns the average gating score with the relative information content between prefix and global contexts. Since sequence length serves practical proxy for information quantity, we formulate a normalization loss based on the prefix length (lp) and full sequence length (l,))
min(1p, 4,) 0.5 (lg - 4p)
Lporm = SmoothLLLoss p.
(7)
Lg
where p represents the sequence's mean gate output, and 4, a buffer hyperparameter (set to 1.5 seconds in our experiments) that prevents abrupt score changes near prefix boundaries. The 0.5 weighting factor accounts for the partial information available at prefix edges.
Pre-Sigmoid Gaussian Noise Following prior work [7]. incorporate Gaussian noise before Sigmoid activation to promote discretization of gate output. While this discretization does not directly improve model performance, it significantly enhances robustness by reducing sensitivity threshold selection across diverse tasks. In our implementation, we apply zero-mean unit-variance Gaussian noise (a 1) to the pre- mi-Sigooid logits.
Combined with our normalization technique, this approach enables precise control over both the mean and variance of gating scores. The complete training objective is:
Crotal
Csimul Wn moor
(8)
where we set small normalization weight wn 0.01.
3.4 Streaming TTS
common practice speech-to-spest translation (S2ST) systems involves integrating a speech text translation (S2TT) module with text-to-speech (TTS) component. For simultaneous translation scenarios, however, streaming TTS system essential to preserve the low-latency requirement of the overall pipeline. While specialized streaming TTS systems exist, demonstrate that SimulMEGA
5

---

offers a universal and straightforward method to convert a standard transformer- autoregressive TTS system into a streaming-
variant.
For our experiments, we adopt Cosy Voice 2 as the base model, which consists of a decoder-only language model and a flow-based acoustic model. During adaptation, we retain the flow model and fine-tune the language model using the techniques and loss functions outlined in Sections 3.2 and 3.3. Randomly initialized routing gate and the MoE refiner are incorporated during training. Since the backbone lacks an encoder, we derive the prefix and offline hidden states (/preii and hottliee) from the final hidden representations of the text tokens. A detailed architectural diagram is provided in the appendix. We compare our approach against CosyVoice2" Voice2's ' default dual-stream mode.
Through this work, not only develop more comprehensive S2ST system capable of real-tine time speech-to-speech translation but also validate the compatibility of SimulMEGA with diverse tasks and model architectures. We leave an end-to-end S2ST system with SimulMEGA for future work.
4 Experiments and Results
4.1 Experiment Settings
Task & Data We collect data and train on many-to-many speech-to-text translation across six popular languages: EN, ZH, DE, ES, FR, and IT. We collect various open-source speech recognition datasets covering the six languages, including LibriSpeech [31], Multilingual Librispeech(MLS) [32], VoxPopuli [33], Common Voice[34]. WenetSpeech(335), KeSpeech[306 36] and Emilia [37]. Then we create pseudo translation labels for these data by translating the transcription into different languages through a cloud text-to-ted text translation API. The dataset consists of approximately 100K hours of training data. For TTS experiment, we use a subset of ST training set that only contains Chinese and English data due to the language compatibility of the Cosy Voise 2.
Model Our offline base model for ST is derived from the whisper medium model. For inference efficiency, we transformed the encoder into streaming encoder and pruned half of the layers in the decoder. This results in an encoder with 20 chunk-AR blocks and 4 NAR blocks, and decoder with 12 layers. Offline base model for TTS is CosyVoice TTS experiment, the MoE refiner consists of Nrefine 6 layers, with the hidden size matching that of the base model.
Voice 2 as mentioned in section 3.4. In both the ST and
Training Each experiment is trained in FP16 with Nvidia H800 GPUs In stage 1 offline training of the ST experiment, to retain the capability of the whisper encoder, we employ Low-Rank Adaptation(LoRA)]E
(a = 64) at the chunk-AR blocks of the encoder. We train the offline model for 1M steps, which takes around 1 week. In stage 2 training. the chunk-AR blocks are frozen. The router and MoE Refiner module are randomly initialized. The stage 2 training takes about days. We use A optimizer[39] and linear learning rate scheduler with 5000 steps of warmup. The maximum learning rate is le-4 in stage training and le-5 in stage 2 training.
Evaluation In S2TT evaluation, we employ case- sess-sensitive BLEU[40] with punctuation' as our primary quality metric and average lagging (AL)* [43] for latency assessment. Our evaluation spans two benchmark datasets: CoVTS2 [44] and Fleurs [45]. For CoVoST2, we report averaged scores across 5 -ingElllshh to English (X-EN) translation pairs and 2 English to non-English (EN-X) pairs. The Fleurs dataset enables comprehensive many-to-many evaluation through its parallel data across six languages, yielding results for all 30 possible language pair combinations. We present separate averages for 5 X-EN pairs, EN-X pairs, and 30 X-X pairs.
For TTS evaluation, we combine LibriSpeech-PCC test-clean [31, 46) and Seed-TTS test-zh [47] with CoVoST2 data. Our analysis includes recognition word error rate (WER), speaker similarity (SIM), and st--umtttt--secc--uete average lagging(AL). In S2ST evaluation on VoST,, we measure AL. For English metrics, we utilize Large- V3, while Mandarin evaluations employ Paraformer [48]. And for SIM, we employ a speaker verification model based on WavI-Mlarrge [49] to extract speaker embeddings. These embeddings are then used to compute the cosine similarity between synthesized speech and ground truth speech.
ASR- BER-BLEU against
https://gihhh.ommmjppssssssrrrr com//njpost/sccreeeef11111
'Code is borrowed from https///tttbb.ooomeeeeeoosss/sss

---

Table 1: The offline BLEU Score(%) of different models Co and Fleurs testset, where all results are based on greedy search. Only parameters involved in the S2TT inference are calculated. The value in the bracket denotes the performance degradation compared to the offline base model.
CoVoST2
Fleurs
Models
Param
X-EN
EN-X
X-EN
EN-X
X-X
Offline
Models
Sea Medium
821M
34.4
35,9
25.6
27.1
16.1
SeamlessM4T Large-v2
1.5B
38.3
40.8
29.8
31.5
19.6
S2T Base (Ours)
561M
37.0
38.9
26.4
32.4
25.1
Simultaneous Models
Seamless-S27
2.0B
35.3
37.667.8%)
28.0g).0%)
28.9(8.38)
18. 18.16775)
SimulMEGA-S22 (Ours)
561M
38.5G..0%) (6.9
26.3(a.4%)
31.4.a1%)
24.7(17%)
Baselines In simultaneous S2TT, our method compared against the Seamless model family [15] and four custom baselines: Wait-K [43], Dig-SST [11], EDATT [19] and AlignATT [20], all implemented using our offline base model architecture. For streaming TTS comparisons, we benchmark against the native streaming implementation in Voice. In simultaneous S2ST evaluation. Seamless serves as our primary comparator. Both the Seamless framework and our proposed system incorporate voice preservation capabilities. We also compare against two publicly available models, StreamSpeech[12] and NAST-S2S[25], whose parameter size and data size are smaller.
4.2 Main Results
Offline Multilingual S2TT Performance The results summarized in Table 1. Leveraging high-quality pseudo-labeled data, our offine base model achieves performance comparable to Seam- lessM4T large-v2 across in-domain six languages, despite the latter having three times the parameter count. The Seamless models, constrained by an English-centie data distribution, slightly underperform ours in the FLEURS many-to-many evaluation. For simultaneous models, we report BLEU score degradation (in brackets) relative to their respective base models. Our analysis reveals that SimulMEGA exhibits minimal degradation, ranging from 0.3% 3%, while Seamless suffers a more significant 7 % drop. Consequently, SimulMEGA outperforms Seamless in 4 out of the evaluated scenarios.
SimulMEGA S2T SimulMEGA-S2T-ofllee
Snamless-S S2T Seemlles-S21-offlmne
S2T Base
EDATT
DIG-SST
SeamlesM4
AlignATT
Walk-K
38
36
22
32
34
20
30
32
30
1.0
1.5
2.0
2.5
3.0
3.5
1.0
1.5
2.0
2.5
3.0
3.5
1.0
1.5
2.0
2.5
3.0
3.5
LAAL (Seconds)
LAAL (Seconds)
LAAL (Seconds)
CoVoST2 X-EN(5)
CoVoST2 EN-X(2)
FLEURS X-X(30)
Figure 3: The multilingual simultaneous speech-to-text translation quality (BLEU) against the latency metrics (LAAL) on different testsets. The reported BLEU and LAAL are the average of all language splits in the testsets(5 in Co X-EN, 2 in CoVoST2 EN-X and 30 in Fleurs X-X)
Simultancous Multilingual S2TT Performance evaluate generalizability across language pairs, we apply the same threshold configuration to all methods under identical settings. As illustrated in Figure 3, SinulMEGA demonstrates robust performance across diverse language pairs and outperforms bascline methods in all three evaluation scenarios. Compared to the offline base

---

Table TTS evaluation results. ZS denotes zero-shot mode (prompt added in AR stage), and CL denotes cross longull mode (prompt not added in AR stage), ALs are calculated by the number of input text tokens. We use the generated text as the WER label for CoVoST2 evaluation.
LibirSpeech
SeedTTS zh
CoVoST2 zh-en
CoVoST2 en-zh
WER
WER
SIM
WER
SIM
Cosy Voice2-ZS
2,44
0.658
22.3
1.62
0.757
23.1
Cosy Voice2-S-ZS"
5.31
0.651
18.5
7.98
0.760
20.9
Cosy DosyVoicc2-S-CL.
3.13
0.535
8.4
4.33
0.707
8.4
19.94
0.388
7.74
0.352
Seamless
20.05
0.292
18.86
0,301
SimulMEGA-TI 2.54
0.661
1.2
1.90
0.755
0.5
12.26
0.391
4.51
0.405
model, SimulMEGA exhibits minimal degradation 3% 3%, and 5 5% degradation at an AL of seconds seconds-while Seamless suffers significantly higher degradation (12% 17%, and 9%). Notably, SimulMEGA achieves low-latency performance with <7 % degradation at 1.5 seconds AL and nears offline model quality (<3% degradation) at 3 seconds AL.
In contrast, Seamless, despite its self-learning design, struggles with substantial performance gap due to limitations in multi-head monotonic (MM)))) The static wait-K strategy incurs high latency and degradation across all settings. Mcanwhile, DiG-SST, ED-ATT and AlignATT exhibits instability, failing specific language pairs. In experiment ED-ATT is most stable among baselines but still shows more than 0.5s additional LAAL compare to SimulMEGA under same quality.
Streaming TTS & Simultaneous S2ST Our experiments on LibriSpeech and SeedTTS zh eval- uate SimulMEGA-TI under extreme streaming conditions, processing one text unit time. For the Co cross-lingual setting, the system receives incremental text chunks generated by SimulMEGA-SSS We compare against Cosy Voice2-S operating its default rext-to-speech ratio. As shown in Table 2, SimuIMEGA-TTS demonstrates comparable speaker similarity to both Cosy Voice2 variants, attributable to their shared flow model. While showing marginally lower alignment latency (AL) than Cosy Voice systems, SimulMEGA-TTSS maintains equivalent speech intelligibility (WER) to the original Cosy Voice2. In cross-lingual evaluation, SimulMEGA-TTS surpasses Seamless by 10 SIM points while achieving more than 40% lower WER. Figure 4 presents our simultancous speech-to-speect translation (S2ST) results. SimulMEGA-S22 maintains tight latency control, adding less than 200 ms AL compared to speech-to-text translation (S2TT). The quality degradation from S2TT to S2ST shows 7% (ZH-EN) and 6% (EN-ZH) BLEU reduction for our system, lower than Seamless's ' 10% and 20% reductions, respectively. Furthermore, SimulMEGA MEMM--AA-255 S2S outperforms synthesizing by Cosy Voice2's s streaming mode both output quality and latency metrics.
SimulMEGA-S2S SimulMBCA-S2T
Seamless-S2S Seamless-S2T
SimulMEGA-S2T Cosyvveee- SimulMEGA-S2S-oflinn
NAST-S2S
-.
StreamSpeech
44
40
20363E C
Aai
BLEU ASRRBBEEU
40
35
38
Antrrn
22
30
34
32
20
2.5
3.0
3.5
4.0
2.0
2.5
3.0
3.5
1.5
2.0
2.5
3.0
3.5
4.0
LAAL / AL (Seconds) (a) CVSS ZH-EN
LAAL / AL (Seconds)
LAAL AL (Seconds)
(b) CVSS EN-ZH
(c) CVSS FR-EN
Figure 4: Bi-directional Simultancous S2TT and S2ST Result between Mandarin and English in CoVoST2 test set. The translation quality metric is BLEU for S2TT and ASR-BLEU for S2ST. We use the same threshold group between S2ST and S2TT. Results of NAST-S2S and StreamSpeech are taken from original paper.

---

4.3
Ablation Study
We conduct ablation experiments on three key designs SimulMEGA: 1) pre-Sigmoid noise scale, 2) router score normalization, and 3) training losses. For these ablations, employ the same base model and conduct stage-2 simultaneous training as the main experiment. Training is performed on subset of the main dataset containing only the Common Voice English data, with evaluation the CoVoST2 EN -H test set.
Pre-Sigmoid Noise We examine three noise settings: No noise, 2)
and 3
Figure 5(a) indicates that noise magnitude has minimal effect on average performance. However, increasing noise leads to more deterministic router scores, thus limiting the flexibility of the latency- performance curve. Conversely, the absence of noise results in overly wide score range and unstable low-latency performance. A moderate noise level (a = 1) achieves an optimal balance between manageable dynamic range and stable translation performance.
Score Normalization Figure 5(b) shows that without normalization, router scores are disproportion- ately concentrated between thresholds 0.5 and 0.8. complicating threshold selection across different tasks or language pairs. Incorporating normalization yields smoother latency- sernec--erformanee curve, reflecting more balanced score distribution within the [0.2, 0.8] range, thereby facilitating easier and more consistent threshold tuning.
Training Loss We investigate the effects of removing either Loffline or Preffx during simultane- training. Results presented in Figure 5(c) demonstrate that removing Coffine reduces overall performance by approximately 1 BLEU point, highlighting the necessity of offline loss to maintain simultaneous translation quality. Conversely, omitting _ leads negligible performance degra- dation, suggesting its optional nature. We hypothesize that the generalization to prefix translation tasks provided by Loffline and Crefiner mitigates the need for truncation-specific training.
46.0
46.0
45.8
45.5
45.6
BLEU
45.0
45.4
weo Lovel
w/o noise
45.2
44.5
weo
0=1
w/ L
44.5
d=3
45.0
W/O
Offline All Losses
Otfline w/o
2.0
2.5
3.0
3.5
4.0
2.5
3.0
3.5
4.0
2.5
3.0
3.5
4.0
LAAL (Seconds)
LAAL (Seconds)
LAAL (Seconds)
(a) Ablation on Noise Scale
(b) Ablation on Score Norm
(c) Ablation on Losses
Figure 5: Ablation Studies. The threshold for all ablation experiment evaluation is 0.8, 0.7, 0.2 from left to right.
5 Conclusion and Discussion
SimulMEGA addresses multilingual SimulST challenges through unsupervised policy learning framework integrating prefix-based training and Mixture-of-Expee refinement. By enabling mini- mal architectural modifications to standard transformers, achieves state-of-the-ard quality-latencyy tradeoffs across six languages while supporting unified speech/text streaming. The integrated system advances real-time cross-lingual communication with miniature BLEU degradation at low latency and robust conversational performance, establishing versatile foundation for low-latency multimodal translation systems.
Limitation
Though SimulMEGA S2S is strong simultaneous S2ST system, it suffers from the drawback shared by cascaded systems: inconsistent text tokens(between Whisper tokens in S2TT and
Using the officially released code and checkpoint (Commit fbab274), CosyVoice 2 exhibits hallucination issues in streaming mode, leading increased WER and deviations from reported results.

---

Qwen2 tokens in TTS), extra latency(around 1.1 seconds) and so on. What's more, SimulMEGA- only supports two languages by now, which hinders its utility. In the future, we plan to add more language support, as well as integrate into more unified end-to-end system. Additionally, currently SimulMEGA only support 30 seconds of maximum input duration. Therefore, still rely on an VAI) model to cut input stream into less than 30s. In the future we will explore continuously generation without segmentation by sliding window or history selection.
This work proposes new streaming techniques and is dedicated providing faithful translation of the original speech. However, with the voice cloning capability of SimulMEGA- TTS, the system may be subject some misuse.
Broader Impacts
6 Acknowledgment
This work was supported by China STI 2030-Major Projects under Grant No. 202IZD0201100
References
[I] X. Ma, Pino, and P. Koehn, "SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to- to- End Simultaneous Speech Translation," P Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, K.-F. Wong. K. Knight, and H. Wu, Eds.
Suzhou, China: Association for
Computational Linguisties, 2020, pp. 582-587.
[2] Q. Dong, Y. Zhu, M. Wang, and L. Li, "Learning When to Translate for Streaming Speech," 2022.
[3] S. Zhang and Y. Feng, "End-to-End Simultaneous Speech Translation with Differentiable Segmentation." 2023.
[4] J. Xue, P. Wang, J. Li, M. Post, and Y. Gaur, "Large-Scale Streaming End-to-Fnd Speech Translation with Neural Transducers," 2022.
[5] K. Deng and P. C. Woodland, "Label-SSnnchonous Neural Transducer for E2E Simultaneous Speech Translation," 2024.
[6] J. Zhao, N. Moritz, E. Lakomkin, R. Xie, Z. Xiu, K. Zmolikova, Z. Ahmed, Y. Gaur, D. Le, and C. Fuegen, "Textless Streaming Speech-to Speech Translation using Semantic Speech Tokens." in ICASSP 2025 2025 IEEE International Conference Acoustics. Speech and Signal Processing (ICASSP), 2025, pp. 1-5.
[7] C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck, "Online and lincar-time attention by enforcing monotonic alignments," in International conference on machine learning. PMLR, 2017, pp. 2837-2844
[8] X. Ma. J. Pino, J. Cross, Puzon, and Gu, "Monotonic multihead attention." arXiv preprint arxiv: 1..Xv:1909.1222 1999.124006, 2019.
[9] S. Guo, S. Zhang, and Y. Feng. "L.arning Optimal Policy for Simultancous Machine Translation via Binary Search," " in Proceedings of the 6t Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics. 2023, pp. 2318-2333.
[10] A. Yin, T. Zhong, H. Li. S. Tang, and Z. Zhao, "Language Model is Branch Predictor for Simultaneous Machine Translation," "" 2023.
[11] X. Chen, K. Fan, W. Luo, Zhang, L. Zhao, X. Liu, and Z. Huang, "Divergence-Guid Simultaneous Speech Translation," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, pp. 17 799-97 807, 2024.
[12] S. Zhang, Q Fang, S. Guo, Z. Ma, M. Zhang, and Y. Feng, "StreamSpeech: Simultancous Speech-to-Speece Translation with Multi-task Learning," 2024.
[13] L. Zhao, J. Li, and 7. Zeng, "PsFuture: A Seuudo-Fuuur-bassed Zero-Shot Adaptive Policy for Simultaneous Machine Translation," 2024.
[14] S. Guo, S. Zhang, Z. Ma, and Y. Feng, "Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation," " 2025.
[15] S. Communication, L. Barrault, Y.-.. Chung, M. C. Meglioli, D. Dale, N. Dong, M. Duppenthaler, P.-.. Duquenne, B. Ellis, H. Elsahar, J. Haaheim, J. Hoffman, M.-. Hwang. H. Inaguma, C. Klaiber, Ramakrishnan, Kulikov, P. Li, Licht, J. Maillard, R. Mavlyutov, A. Rakotoarison, K. R. Sadagopan,
T. Tran, G. Wenzek, Y. Yang. E. Ye, I. Evtimov, P. Fernandez, C, Gao, P. Hansanti, E. Kalbassi, A. Kallet, A. Kozhevnikov, G. M. Gonzalez, R. S. Roman, C. Touret, C. Wong, C. Wood, B. Yu, P. Andrews, C. Balioglu, P.-. Chen, M. R. Costa-jussà, M. Elbayad, H. Gong, F. Guzmán, K. Heffernan, S. Jain, J. Kao.
10

---

A. Lee. X. Ma, A. Mourachko, B. Peloquin, J. Pino, S. Popuri. C. Ropers, S. Saleem, H. Schwenk, A. Sun, P. Tomasello, C. Wang. J. Wang. S. Wang, and M. Williamson, "Seamless: Multilingual Expressive and Streaming Speech Translation," " 2023.
[16] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, E. Yu, H. Liu,
Z. Sheng. Y. Gu, C. Deng. W. Wang. S. Zhang. Z. Yan, and J. Zhou, "Cosy Voice 2: Scalable Streaming
Speech Synthesis with Large Language Models," 2024.
[17] x. Zeng, L. Li, and Q. Liu, "RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer." 2021.
[18] S. Zhang and Y. Feng, "Information- caaariiionnraaapt---bbbeeee Policy for Simultaneous Translation," in Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, Y. Goldherg, 7. Kozareva, and Y. Zhang, Eds. Abu Dhabi, United Arah Emirates: Association for Computational Linguistics, 2022, pp. 992-1013.
[19] S. Papi, M. Negri, and M. Turchi, "Attention as a Guide for Simultancous Speech Translation." in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), A. Rogers, Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 13 13.340-3333 356.
[20] S. Papi, M. Turchi, and M. Negri, "AlignAtt: Guide for Simultancous Speech Translation," in INTERSPEECH 2023, 2023, pp. 3974-3978.
Using inrention-based Audio-Translation Alignments as
[21] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks," 2006.
[22] N. Arivazhagan, C. Cherry. W. Macherey, C.-C. Chiu, S. Yavuz, R. Pang. W. Li, and C. Raffel, "Monotonic infinite lookback attention for simultaneous machine translation." " arXiv preprint (xXixv1906.02/// 2019.
[23] X. Ma, A. Sun, S. Ouyang, H. Inaguma, and P. Tomasello, "Efficient Monotonic Multihead Attention." 2023.
[24] S. Zhang and Y. Feng, "Hidden Markov Transformer for Simultancous Machine Translation," in The Eleventh International Conference on Learning Representations, 2022.
[25] Z. Ma, Q. Fang. S. Zhang, S. Guo, Y. Feng, and M. Zhang, "A Non-autorcgressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volune I: Long Papers). Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 1557-1575.
[26] M. Lajszzzak, G. Cámbara, Y. Li, F. Beyhan, A. Korlaar, F. Yang, A. Joly, Á.
A. Abbas. A. Michalski, A. Moinet, S. Karlapati, E. Muszyúska, H. Guo, B. Putryez, L. Gambino, K. Yoo, F.. Sokolova, and Drugman, "BASE TTS: Lessons from building a billion- parlmmter Text-to- Speech model on 100K hours of data," 2024.
[27] T. Dang. D. Aponte, D. Tran, and K. Koishida. (2024-06-10) LiveSpeceh: Low-Latency Zero- shot Text-to- Text-to-Speech via Autoregressive Modeling of Audio Discrete Codes. [Online]. Available:
http://axiiv.or//ab///0666
[28] A. Dekel, S. Shechtman. R. Fernandez. D. Haws, Z. Kons, and R. Hoory, "Speak While You Think: Streaming Speech Synthesis During Text Generation," 2023.
[29] Z. Sheng, Z. Du, Zhang, Z. Yan, Y. Yang. and Z. Ling, "SyneSpeech: Low-Latency and Efficient mall-Stream Text-to- Soeetth-Seech based on Temporal Masked Transformer," 2025.
[30] T. Dang, D. Aponte, D. Tran, T. Chen, and K. Koishida, "Zero-Shot Text-to-Speech from Continuous Text Streams," " 2024.
[31] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: an ast corpus based on public domain audio books," in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206-5210.
[32] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, "Mis: A large-scale multilingual dataset for speech research," " arXiv preprint wxXiv2002..3411 2020.
[33] C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, "Voxpopuli: A large-scale multilingual speech corpus for representation leaming. semi-supervised learning and interpretation." arXiv preprint arXiv:2101.000 00090,, 2021.
[34] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, "Common voice: A massively-mulilingu multilingual speech corpus." arXiv preprini arXiv:1912.06670, 2019.
[35] B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu. X. Chen, C. Zeng et al., "Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition." in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6182-6186.
11

---

[36] Z. Tang, D. Wang. Y. Xu, J. Sun, X. Lei. S. Zhao, C. Wen, X. Tan, C. Xie. S. Zhou et al., "Kespeech: An open source speech dataset of mandarin and its eight subdialects. " in Thirty-fifth Conference on Neural Information Procexsing Systems Datasets and Benchmarks Truck (Round 2). 2021.
[37] H. He, Z. Shang, C. Wang, X. Li, Y. Gu, H. Hua, L. Liu, C. Yang, J. Li, P. Shi, Y. Wang. K. Chen, P. Zhang, and Z. Wu, "Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation." " in Proc. of SLT, 2024.
[38] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint (etd:::2006.09666 2021.
[39] Loshchilov, "Decoupled weight decay regularization," arXiv preprint Xix:: 2017.
[40] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: A Method for Automatic Evaluation of Machine Translation," in Proceedings of the 40th Annual Meeting the Association for Computational Linguistics, P. Isahelle, F. Chamiak, and Lin, Eds. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, 2002, pp. 311-318.
[41] M. Post, "A call for clarity in reporting BLEU scores," in Proceedings the Third Conference on Machine Translation: Research Pupers. Belgium. Brussels: Association for Computational Linguistics, Oct. 2018,
pp. 186-191. [Online]. Available:
18-6319
[42] X. Ma, M. J. Dousti, C. Wang, Gu, and Pino. "Simuleval: An evaluation toolkit for simultaneous translation." " in Proceedings of the EMNLP, 2020.
[43] M. Ma. L. Huang. H. Xiong, R. Zheng, K. Liu, B. Zheng, C. Zhang. Z. He. H. Liu, X. Li, H. Wu. and H. Wang. "STACL: Simultancous Translation with Implicit Anticipation and Controllable Latency using Prefix-to Pr-fffxx Framework," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Marquez, Eds. Florence, Italy; Association for Computational Linguistics, 2019, pp. 3025-3036.
[44] C. Wang, A. WU, Gu, and J. Pino, "Covost and massively multilingual speech translation." in Interspeech, 2021, pp. 2247-2251.
[45] A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axclrod, S. Dalmia, J. Riesa, C. Rivera, and A. Bapna, "Fleurs: Few-shot learning evaluation of universal representations of speech," in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 798-805.
[46] A. Meister, M. Novikav, N. Karpov, E. Bakhturina, V. Lavrukhin, and B. Ginsburg, "Librispeech-pe Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models," in 2023 IEEE automatic speech recognition and understanding workshop (ASRU). IEEE, 2023, pp. 1-7.
[47] P. Anastassiou, J. Chen. J. Chen, Y. Chen, Z. Chen, Z. Chen. J. Cong, L. Deng, C. Ding. L. Gao. M. Gong, P. Huang, Q. Huang, Z. Huang, Y. Huo, D. Jia. C. Li, F. Li. H. Li. J. Li, X. Li. X. Li. L. Liu, S. Liu, Liu, X. Liu, Y. Liu, Z. Liu, L. Lu, J. Pan, X. Wang, Y. Wang, Y. Wang, Z. Wei, J. Wu, C. Yao, Y. Yang, Y. Yi, J. Zhang, Q. Zhang, S. Zhang, W. Zhang, Y. Zhang, Z. Zhao, D. Zhong, and
X. Zhuang, "Seed-tts: A family of high-quality versatile speech generation models," 2024. [Online]. Available: http://aari..orrrab/2260....2
[48] Z. Gao, S. Zhang. I. McLoughlin, and Yan, "Paraformer: Fast and accurate parallel transformer for non-autoregresssive end-to-end speech recognition," arXiv preprint arXiv:2206.083/ 2022.
[49] S. Chen, C. Wang. Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li. N. Kanda, T. Yoshioka. X. Xiao al., "Wavlm: Large-scale supervuue pre-training for full stack speech processing." IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505-1518, 2022.
12

---

A SimulMEGA-TTS Implementation Details
Figure 6 shows how to implement SimulMEGA on top of a uni-directional decodcr-only backbone. We take Voice 2 as the base model and fine-tune it into SimuIMEGA-TTS. We take the last layer hidden states output of text tokens as the h A and the hidden state of the EOS token Helobal
In S2TT, normally, when the input reaches the end of the stream, no longer instead continue the generation until the EOS token appears. However, the TTS task, the end speech relatively ambiguous. Therefore, if we employ the same strategy as S2TT, the hallucination problem may occur the end of the generation. Therefore, we employ mixed strategy to tell the end of the generation:
the read/write strategy and
True if Fr E Trext and (Fig E T Tepeech p., Xead)
End of Generation = False
(9)
otherwise
Where Pays denotes text/speech EOS token. Pi,s is the router output at the current position after all input streams read. Aend 0.9.
/speach Teee//ppach denotes text or speech tokens in the current sequence.
Text Token
End of Text
Tun of Sprech
H Hglobal
(00, 0.1. 0.9 1.0, 1.4)
Speech
Fnd of Speech
Ignore Token
MoE. Refiner
Router
h
Cosy Voice
Language Model
Cosy Voice 2 Language Model
3
Figure 6: Illustration of SimuIMEGA--TSS. The left uses offline text input while the right uses prefix text input for policy learning.
B Experiment Details
B.1 S2TT Baselines
Meta Seamless series: Meta Seamless series contains a models including SeamlessM4T-large- v2 for offline translation, ScamlessStreaming for simultancous translation and SeamlessExpressive for voice cloning. They also combines the capability of SeamlessStreaming with ScamlessExpressiv into single model named Seamless. In experiment use SeamlessM4TT-argee> for offline evaluation, SeamlessStreaming for simultaneous S2TT evaluation and Seamless for simultaneous S2ST evaluation.
2. Wait-K: For cach input test sample, We use VAD to determine the starting time. Then we wait for two chunks (1.28s) and start generation fixed ratio, ranging from 0.5 to 2.5 tokens per chunk.
3. DiG-SST: Following the original paper, we freeze the base model and add three learnable policy layers on top of the text decoder to predict the divergence between prefix and global input. It was trained for 100k steps on the same training of SimulMEGA.
4. EDATT: We follow the paper and official code for implementation. We re-tuned the thresholds and hyperparameter and set attention layer to be the 8th layer and set attention frame to be 4.
5. AlignATT: We follow the updated official code". frame-wist normalization applied to attention score to mitigate maximum attention fixation.
our experiment observed that both DiG-SST and AlignATT behave poorly on all X2ZH pairs, which hinders the overall performance. We hypothesis that this might caused by the UTF-8 byte-level BPE tokenizer. It encodes each CJK character into multiple tokens, potentially leading to unexpected output distribution and attention pattern.
13

---

B.2 Quality-latency variable
Table shows the choice of the variable that controls the quality-latency tradeoff in the main experiment. From left to right the quality and latency gradually increase.
Table 3: Quality-latency variables in the main experiment.
Method
variable
S2TT
S2ST
SimulMEGA
A
[0.9, 0.7, 0.5, 0.3, 0.1]
[0.8, 0.7.... 0.2] [0.3, 0.5, 0.7, 0.9]
Seamless
tEMMA
[0.3, 0.5, 0.7, 0.9, 1] [0.1, 0.07, 0.05, 0.03, 0.01]
DiG-SST
X
EDATT
[0.05, 0.03, 0.02, 0.015, 0.01]
AlignATT
[32. 48, 64, 80, 96] [2.5, 2.0, 1.5, 1.0, 0.5]
Wait-K
ratio
C S2TT computation overhead.
For evaluating the computation overhead, we plot the conputation-aware(LL) quality-latency trade-off curve in the Co FR-EN testset. Computation-aware latency considers the actual inference time of the model. The evaluation is performed on fully idle machine with a single Nvidia-H 1100 GPU. As shown in Figure 7, for SimulMEGA, the extra AL due to computation around 50 ms, whereas the number for Seamless is around 200
ms.
D. SimulMEGA-S2T SimulMEGA S2T CL
b Seamless-S27
Offline Base(ours)
Seamless Deamloss-S2T-CL. S2T-CL
SoamlessM4T-Largee
41
BLEU
38
1.2
1.4
1.6
1.8
2.0
2.2
LAAL (Seconds)
CoVoST2 FR-EN
Figure 7: Computation aware quality-latency curve on CoVoST2 FR-EN. CL denotes Computation Aware.
D S2ST System Deployment & Latency.
We deploy our S2ST system on an Ubuntu server equipped with NVIDIA H100 GPUs. For the frontend and backend, we utilize FastRTC' and FastAP1", respectively. To enhance inference efficiency, vLLM is used to accelerate the language model in TTS, and TensorRT is used to accelerate the flow model.
We evaluate the system latency with streaming inputs, using four language pairs (ZH-EN, EN-ZH, FR-EN, FR-ZH). For each direction, we two samples, each 20-30 seconds in length, and record event timestamps samp
for latency calculation. Virtual Audio Cable" software employed to eliminate ccho interference and ensure
ll
14

---

consistent audio imput across lests. The average latency results presented in Table 4. Here, the S2T/S2S Start Offset indicates the time required to generate the finst text speech chunk, while the S2T/S2S End Offiset measures the delay the final text chunk speech frame relative to end the source speech.
should noted that these results provide only rough estimation system latency, actual values may vary due to server or network conditions, output content length, speech rate, or other factors.
Table 4: Simultaneous S2ST system latency statistics (in seconds). denotes using non-stream TTS and S denotes stream TTS
Offline
SimulMEGA
Seamless
N
N
S2TT
Start Offset 13.56
3.12
2.95
3.55
End Offset
0.78
0.44
0.58
0.61
S2ST
Start Offset 15.98
7.43
3.87
4.33
End Offset
14.16
7.65
7.54
6.42
E
Numerical results
Numerical results of simulMEGA S2TT CoVoST 2 testset is shown in Table 5.
Table 5: Numerical results of simulMEGA S2TT on CoVoST testset.
Threshold
0.9
0.7
0.5
0.3
0.1
0
BLEU
21.06
25.37
26.01
26.40
26.63
26.89
zh-en
LAAL
1.395
2.249
2.539
2.807
3.180
BLEU
32.90
36.08
36.94
37.15
37.41
37.44
de-en
LAAL
1.217
1.862
2.149
2.420
2.856
BLEU
37.21
40.27
40.94
41.28
41.46
41.51
es-en
LAAL
1.047
1.521
1.760
2.001
2.405
BLEU
36.50
38.76
39.33
39.60
39.76
40.02
fr-en
LAAL
0.952
1.391
1.590
1.812
2.195
BLEU
34.64
37.42
38.12 18.12
38.57
38.89
38.90
it-en
LAAL
1.101
1.621
1.873
2.125
2.512
BLEU
39.93
42.77
43.11
43.17
43.42
43.57
en-zh
LAAL
1.539
2.305
2.680
2.976
3.381
BLEU
29.65
32.89
33.16
33.61
33.66
33.75
en-de
LAAL
1.266
1.813
2.121
2,404
2.808
S2ST Case Study
We visualize two S2ST example of SimulMEGA in FR-EN and ZH-EN, respectively. The second row is the isochronic text label of source speech and the forth row indicates the timestamps when each target text chunk are generated.
15

---

sabolage bourgeds
un bourgecis apeurè
the paralt ridicule
transfomer
en sabteur
parter uta
bourgeols sabotage
an afraid bourgeois
seéms ridcaloos
transfomed
into dabateur
Speaking of
me.
10
12
Figure 8: (a) FR-EN
according the trademank nature Shaxian snacks and its industry regulations
Ma Wn' th
be opera undet the authenattion of Shaxian locals.
However
must
0
2
10 12
14
Figure 8: (b) ZH-EN
16

---

17

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are clearly stated.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: have included the discussion on limitation in the last section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate" "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well sell---ecifiiation. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might he violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, c.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
authors should reflect on the factors that influence the performance the approach. example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
18

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have disclosed full details to reproduce the result in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution the contribution is specific model empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require relcasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g.. large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to deseribe the particular way they provide closcd-sourc source models, it may be that access to the model is limited some way (c.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
reprodueibilityy In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer:
[Yes]
Justification: All ASR data we open-sourced, and there various ways to pseudo label it(Perhaps with LLMs). code will soon opensourced on Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines httpa/////niss.ccppppppee
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nnis/ cc/public/ ou..dess/Cooe@uuttieessoooooo for more details.
19

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparamcters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
We have included most of the details. The remaining can be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix. supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Reporting statistical significance is common practice for setting due the heavy training and evaluation overhead. However, we have tested our method tasks, two backbones, and more than 40 language pairs, which to some extent statistically proves the effectiveness our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should he explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We reports the device and execution time in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies https://neutssi
Answer: [Yes]
Justification:
We confirm we follow every respect of the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have reported that voice cloning part may subject misuse.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from foodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The risk of misuse is not high. Normally, people will turn to other dedicated TTS models for faking voice.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
21

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification:
We credit all assets and respect all licenses of these assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ dataastss has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Currently, we haven't published any new assets. And in the future, we will document it. Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The work does involve crowdsourcing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
22

---

Answer:
[NA]
Justification:
The work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy (htticettttt/eeuueeeeee httpaa/ww.......ooeeeeeee/////// for what should or should be described.
23


Reviews:
[{"review_id": "0Nf44P9gNm", "reviewer": "Reviewer_U9os", "summary": "This paper presents SimulMEGA, a policy learning framework for simultaneous speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). It introduces a mixture-of-experts (MoE) refiner on top of a standard Transformer architecture to enable adaptive read/write decision-making. The framework also incorporates a joint loss function designed to balance translation quality with latency-aware decision accuracy. Experiments on CoVoST2 and FLEURS across six languages demonstrate that SimulMEGA outperforms Seamless in multilingual settings.", "questions": "1. What is the rationale behind keeping the non-autoregressive (NAR) layers from Whisper? Is this a trade-off between translation quality and computational efficiency?\n2. The score normalization in Equation (7) is somewhat unclear. Could you elaborate on the rationale behind this formulation?\n3. How is inference performed for the SimulMEGA-TTS model?\n4. Which translation API is used to obtain the reference translations?\n5. Lines 280–281 mention router scores in Figure 5(b), but I couldn’t find them in the figure. Could you clarify?\n6. What exactly does $L^{\\text{trunc}}$ refer to in line 285?", "limitations": "The model is not tested on unbounded speech, where input streams last minutes or even hours, as in real-world applications. Evaluating on such long-form inputs is essential for assessing robustness and practical usability in realistic streaming settings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. SimulMEGA outperforms Seamless in both offline translation quality and the quality-latency trade-off for simultaneous translation, despite being a smaller  multilingual model.\n2. The framework naturally extends to simultaneous text-to-speech synthesis, achieving near-offline synthesis quality with a latency of approximately 1 second.\n\n# Weakness\n1. The use of Average Lagging (AL) is problematic, as it has been shown to be not robust to over-generation [1]. A more reliable alternative is Length-Adaptive Average Lagging (LAAL), which has been adopted as the standard latency metric in recent IWSLT evaluations.\n2. Missing strong baseline: The paper omits AlignAtt [2], one of the most effective streaming policies that can be directly applied to offline ST models. Including it would provide a more comprehensive and fair comparison.\n3. Lack of experimental details: The paper does not specify the threshold configurations used during training and inference for read/write decision.\n\n[1] Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation (Papi et al., AutoSimTrans 2022)\n\n[2] AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation (Papi et al., InterSpeech 2023)", "was_revised": true, "final_justification": "My main concerns were:\n1. The use of AL, which has been shown to be a flawed metric.\n2. The absence of strong baselines such as AlignAtt.\n3. Some lack of clarity in the writing.\n\nThe authors’ response to my and other reviewers’ comments shows that:\n1. SimulMEGA remains strong under the LAAL metric.\n2. Results for AlignAtt have been added and are not as strong as SimulMEGA.\n3. The method also performs well in long-form inference.\n\nOverall, I will maintain my positive score.", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful comments and constructive feedback on our work. We have carefully addressed each of your concerns as follows:\n\n### Addressing Weaknesses\n\n1. **Latency Evaluation**: We have incorporated numerical evaluation results on the CoVoST2 testset using LAAL (Average Lagging) as the latency metric, presented in Table 1. The results confirm our original conclusion that SimulMEGA continues to marginally outperform baseline methods. In the final version, all quality-latency curves will be systematically re-plotted using the LAAL metric for comprehensive comparison.\n\n2. **Baseline Comparison**: While AlignATT did not function effectively with our base model (due to persistent attention fixation on particular frames leading to over-generation), we successfully integrated EDATT (Papi et al., 2023), which shares conceptual similarities. As shown in Table 1, EDATT outperforms both DIG-SST and wait-K baselines, further validating our approach's competitive advantage.\n\n3. **Threshold Documentation**: The final version will include comprehensive tables in the appendix detailing numerical results alongside thorough explanations of threshold selection methodology. For your immediate reference, Table 1 presents our threshold selection rationale, with additional implementation details available in the supplementary materials.\n\n4. **Notation Correction**: We sincerely appreciate your careful reading—the reference to $L^{\\text{trunc}}$ in line 285 should indeed be $L^{\\text{prefix}}$. This was an oversight in notation consistency, and we will correct it in the final version.\n\n5. **Real-world streaming processing:** Our supplementary materials include a deployed web demonstration that processes streaming input using a VAD (Voice Activity Detection) system to segment audio into optimal 10-20 second chunks for model processing. While standardized long-form evaluation benchmarks remain scarce in the field, we constructed a specialized test set following Whisper's methodology, comprising 17 TED-LIUM talks with an average duration of nine minutes. We translated the reference transcripts to Chinese using ByteDance's Doubao translation service for evaluation. The S2TT performance metrics are presented in Table 2. \n\n### Clarifying Conceptual Questions\n\n1. **Architecture Design**: Our decision to apply chunk masking selectively rather than across all layers stems from preliminary experiments showing that universal application degraded translation performance. While we acknowledge the value of ablation studies, training additional base models on this data scale would be computationally prohibitive. We note that architectural choices serve as implementation details rather than core contributions of this work.\n\n2. **Routing Score Normalization**: The normalization mechanism is designed to align routing scores proportionally with available input information. Since information content is intractable, we approximate it using input duration. For instance, when processing half a sentence, the average routing score should approach 0.5 (with ideal scores transitioning from 0 in the first half to 1 in the second half). To account for boundary uncertainty, we slightly adjust the normalization target beyond the strict duration proportion.\n\n3. **Generation Process**: Upon receiving each text chunk, we initiate a new autoregressive generation process incorporating all previous text chunks and generated speech tokens as context. Generation terminates when the routing score reaches the predetermined threshold. This approach maintains full compatibility with vLLM's CosyVoice implementation, achieving real-time factors (RTF) of 0.1 in deployment.\n\n4. **Pseudo-Labeling**: We employed Google Translate for pseudo-label generation, collecting translation pairs between November 2024 and March 2025 to ensure temporal consistency in the translation service's behavior.\n\n5. **Router Score Interpretation**: The notation $p^{router}$ represents the router probability score. For visual clarification, Figure 6 in the appendix (supplementary materials) illustrates the dynamic behavior of these scores.\n\nWe would like to highlight that our supplementary materials contain complete implementation code, comprehensive numerical results, extended appendices, and a video demonstration of our deployed system, which may provide additional context for evaluation.\n\nWe sincerely appreciate your thorough review and valuable suggestions. These improvements have significantly strengthened our manuscript, and we welcome any further feedback you might have regarding our responses.\n\n###### Table 1: CoVoST2 Evaluation\n\n|                        | X-EN     |           |          |           |          |      | EN-X     |           |          |           |          |\n| ---------------------- | -------- | --------- | -------- | --------- | -------- | ---- | -------- | --------- | -------- | --------- | -------- |\n| **Seamless Streaming** | **0.3**  | **0.5**   | **0.7**  | **0.9**   | **1**    |      | **0.3**  | **0.5**   | **0.7**  | **0.9**   | **1**    |\n| BLEU                   | 33.50    | 33.75     | 34.05    | 34.37     | 35.05    |      | 33.35    | 33.70     | 33.83    | 34.11     | 35.41    |\n| LAAL                   | 2.280    | 2.410     | 2.522    | 2.665     | 3.149    |      | 1.966    | 2.065     | 2.144    | 2.268     | 2.660    |\n| **Wait-K**             | **2.5**  | **1.9**   | **1.5**  | **1.1**   | **0.5**  |      | **2.5**  | **1.9**   | **1.5**  | **1.1**   | **0.5**  |\n| BLEU                   | 29.27    | 31.82     | 32.64    | 33.45     | 34.93    |      | 34.25    | 35.83     | 36.12    | 36.49     | 37.41    |\n| LAAL                   | 1.731    | 2.151     | 2.296    | 2.565     | 3.309    |      | 1.975    | 2.294     | 2.376    | 2.541     | 3.127    |\n| **DigSST**             | **0.1**  | **0.07**  | **0.05** | **0.03**  | **0.01** |      | **0.1**  | **0.07**  | **0.05** | **0.03**  | **0.01** |\n| BLEU                   | 31.43    | 33.53     | 35.05    | 35.99     | 36.52    |      | 17.12    | 17.51     | 18.10    | 19.44     | 23.38    |\n| LAAL                   | 2.035    | 2.237     | 2.442    | 2.752     | 3.340    |      | 0.734    | 0.770     | 0.808    | 0.902     | 1.252    |\n| **EDATT**              | **0.03** | **0.025** | **0.02** | **0.015** | **0.01** |      | **0.03** | **0.025** | **0.02** | **0.015** | **0.01** |\n| BLEU                   | 33.70    | 34.81     | 35.60    | 36.09     | 36.56    |      | 35.00    | 35.82     | 36.77    | 37.63     | 38.20    |\n| LAAL                   | 1.880    | 2.062     | 2.301    | 2.610     | 3.056    |      | 1.963    | 2.131     | 2.384    | 2.688     | 3.129    |\n| **SimulMEGA**          | **0.7**  | **0.6**   | **0.5**  | **0.4**   | **0.3**  |      | **0.7**  | **0.6**   | **0.5**  | **0.4**   | **0.3**  |\n| BLEU                   | 35.58    | 35.97     | 36.27    | 36.46     | 36.60    |      | 37.83    | 38.11     | 38.14    | 38.28     | 38.39    |\n| LAAL                   | 1.729    | 1.860     | 1.982    | 2.104     | 2.233    |      | 2.059    | 2.246     | 2.400    | 2.542     | 2.690    |\n\n###### Table 2: Long-form S2TT result on TED-LIUM\n\n|           | BLEU  | AL       | LAAL     |\n| --------- | ----- | -------- | -------- |\n| SimulMEGA | 55.13 | 16.15655 | 24.62662 |\n| Seamless  | 38.18 | 7.016107 | 15.42395 |"}, {"review_id": "CLqBPQnQV5", "reviewer": "Reviewer_Eauk", "summary": "This paper proposes SimulMEGA, an unsupervised policy learning framework for Simultaneous Speech Translation. It introduces a MoE Refiner and a routing gate that jointly enable the model to implicitly learn read/write policies without adding inference-time overhead. The proposed system supports both speech-to-text and text-to-speech streaming tasks and demonstrates good latency-quality trade-offs across six languages. The authors provide extensive experiments, including comparisons with SeamlessM4T, streaming TTS baselines, and simultaneous speech-to-speech translation tasks.", "questions": "- What specific problems does the introduction of the MoE refiner solve, and how does it contribute to the final performance improvement? It would be helpful if the authors could provide additional experiments to explicitly demonstrate the effectiveness of the MoE module.\n\n- The authors are encouraged to provide a more detailed explanation of the various loss functions introduced in the model, including their exact formulations. Additionally, key concepts such as $h_{\\text{prefix}}$ and other latent representations should be clearly defined.\n\n- More detailed implementation descriptions for the Streaming TTS component would be beneficial, particularly regarding system design, latency control, and how streaming is achieved in practice.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed method achieves SOTA performance on the SimulST task, supported by extensive experiments and strong comparisons against baselines.\n- The paper further extends the evaluation to streaming TTS and simultaneous speech-to-speech translation tasks, demonstrating the versatility of the proposed framework.\n- The method is relatively simple and efficient, requiring minimal architectural modifications to existing systems while still providing significant performance improvements.\n- The supplementary materials include comprehensive demos and audio samples.\n\n### Weaknesses\n- The clarity of the paper needs improvement. Many important concepts (such as $h_{\\text{prefix}}$ and $L_{\\text{offline}}$) are not clearly explained, which makes the paper harder to follow.\n- Since the core contribution of the paper is the Mixture of Experts (MoE) refiner, the authors should provide more motivation for introducing it. Why is MoE necessary? How does it compare to native solutions without MoE? The use of gating mechanisms is also not particularly novel and would benefit from a more in-depth justification.\n- The implementation details of the streaming TTS component need to be described more thoroughly.", "was_revised": true, "final_justification": "My final score is 4 (borderline accept). I believe this is a paper with innovative contributions. It proposes a MoE Refiner and a routing gate that jointly enable the model to implicitly learn read/write policies without introducing inference-time overhead. The method also achieves very strong results. However, I am not very familiar with this domain (hence my confidence score is 2), and there are some parts I did not fully understand. Therefore, I will keep my current score, which I believe is an acceptable evaluation.", "rebuttal": "Dear Reviewer,  \n\nThank you for your thoughtful feedback regarding the accessibility of our work for readers unfamiliar with simultaneous translation. We acknowledge that space constraints limited our ability to include extensive background in the main text, and we sincerely appreciate your engagement with this challenge. Below, we clarify the core concepts to address your concerns.  \n\nThe fundamental challenge in streaming (simultaneous) translation is the **read/write decision problem**: as the system processes speech in continuous chunks, it must dynamically decide *whether* to generate output tokens or wait for future input. Suboptimal decisions lead to excessive latency (waiting too long) or over-generation (outputting incorrect tokens due to incomplete input).  \n\nWhile a robust read/write policy is essential, **supervised training data for such policies is scarce**, and existing data may not reflect optimal decisions. We thus propose an *unsupervised* approach where the model learns the policy intrinsically. Our solution leverages a **Mixture-of-Experts (MoE) gating mechanism** to enable this learning:  \n\n1.  **Training Setup**: For a given source sentence, the model *only accesses the first half* during training.  \n2.  **Expert Roles**:  \n    - A *prefix expert* analyzes the visible partial input ($h^{prefix}$).  \n    - A *global expert* infers high-level context from the *full* sentence ($H^{global}$)\n3.  **Policy Learning**: The MoE router learns to select between experts when predicting each target token (e.g., preferring the prefix expert for early tokens and the global expert for later tokens). Crucially, **no explicit policy labels are used**—the router *implicitly* learns when input is sufficient for generation.  \n\n**At inference** (streaming mode), the global expert is *unavailable* (since future input is unknown). If the router \"attempts\" to use the global expert (i.e., assigns it high weight), this signals **insufficient input**—trigerring a *wait* decision. Thus, the MoE mechanism **naturally emerges as a read/write policy** without supervision.  \n\nRegarding your query about \"native\" baselines: **all comparative methods inherently implement *some* read/write policy** (e.g., fixed latency rules or learned triggers). There is no \"policy-free\" baseline for simultaneous translation, as the task *requires* such decisions. We have added a new baseline per other reviewers’ suggestions to further strengthen comparisons.  \n\nFor streaming TTS, the underlying mechanism is identical to simultaneous translation (as noted in Appendix X), differing only in modality-specific implementations. We kept this section concise to prioritize core contributions but are happy to expand it if needed.  \n\nWe believe all notations (e.g., $h^{prefix}$, $H^{global}$, $L^{prefix}$) are clearly defined in Figures 1–2 and Equations 4. Should any term remain unclear, we will gladly refine explanations in revision. Thank you for your time and valuable critique—we welcome the opportunity to improve the manuscript further."}, {"review_id": "5J2V9WRpSH", "reviewer": "Reviewer_FapF", "summary": "The paper introduces SimulMEGA (Simultaneous Generation by Mixture-of-Experts Gating), a novel framework for Simultaneous Speech Translation (SimulST), which enables real-time cross-lingual communication by integrating speech recognition and machine translation under stringent latency constraints. Unlike traditional offline systems, SimulMEGA processes streaming audio input to generate translations incrementally, mimicking human interpreters. The framework addresses challenges in balancing translation quality, latency, and semantic coherence, particularly in multilingual many-to-many translation scenarios. SimulMEGA combines prefix-based training with a Mixture-of-Experts (MoE) refiner module to learn implicit read/write policies without introducing inference-time overhead. It employs a hybrid streaming speech encoder (combining chunk-wise autoregressive and non-autoregressive blocks) and a standard transformer-based text decoder, augmented by a global routing gate and an MoE refiner active only during training. The framework supports both speech-to-text and text-to-speech (TTS) streaming tasks, achieving state-of-the-art performance.", "questions": "- The paper describes the MoE refiner as comprising prefix and global experts, with merging weights determining the read/write policy. Could the authors provide more details on how these weights are learned during training?\n- The paper references prior policy-based frameworks but does not explicitly compare SimulMEGA’s policy learning approach to these methods. How does the unsupervised MoE-based policy learning differ from, say, the CTC-based policy or the attention-based policy.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Simultaneous Speech Translation is a very valuable and promising research topic.\n- The paper presents a high-quality technical contribution, supported by rigorous evaluation across six language pairs.\n- The paper is well-structured, with clear descriptions of the SimulMEGA framework.\n\nWeaknesses:\n- While the evaluation on six language pairs is comprehensive, the paper does not provide detailed analysis of performance across specific language pairs or linguistic phenomena (e.g., low-resource languages or complex syntactic structures). This limits insight into the framework’s robustness across diverse linguistic contexts. Additionally, the TTS evaluation is less detailed than the speech-to-text results.\n- The performance of the proposed method in Simultaneous Speech-to-Speech Translation should be further evaluated on the CVSS benchmark which is commonly used in previous work.", "was_revised": true, "final_justification": "Thank you for your response. After carefully considering both the content of the paper and the rebuttal, I have decided to keep my score unchanged.", "rebuttal": "Dear Reviewer\n\nThank you for your valuable feedback. We have carefully addressed each of your concerns as follows:\n\n### Addressing Weaknesses\n\n1. **Evaluation on specific language pair**: Comprehensive numerical results for each language pair in the CoVoST2 testset are currently available in our supplementary materials. In the final version, we will expand these into a complete table presented in the appendix for easier reference. Additionally, the supplementary materials include detailed translation examples that illustrate linguistic phenomena affecting performance. For instance, we observe higher translation latency between English and Mandarin due to their inverted grammatical structures—a finding that aligns with established linguistic research on language directionality in translation systems.\n2. **TTS evaluation**: Our current TTS implementation supports two languages, which explains the more focused evaluation scope. Nevertheless, our results demonstrate that SimulMEGA-TTS achieves lower streaming latency while maintaining the high-quality output standards of CosyVoice 2. We invite you to experience the practical performance through our audio samples and video demonstration in the supplementary materials, which showcase real-time translation capabilities across various speech patterns and speaking rates.\n3. **Benchmark Clarification**: While CVSS [1] is derived from CoVoST2, it specifically focuses on speech-to-speech translation pairs. In the final version, we will revise our benchmark description to accurately reflect this distinction and better align with the S2ST task framework.\n\n### Responding to Conceptual Questions\n\n1. **MoE router weights learning**: As the MoE mechanism, the router will automatically select the best expert for a given input hidden state.  The router weights are learning through gradient decent -- weights for the optimal expert will increase during training as the gradient suggests. Under this mechanism, we turn the MoE router into a read/write decision maker:\n   - **Training Setup**: For a given source sentence, the model *only accesses the first half* during training.  \n   - **Expert Roles**:  \n     - A *prefix expert* analyzes the visible partial input ($h^{prefix}$).  \n     - A *global expert* infers high-level context from the *full* sentence ($H^{global}$)\n   - **Policy Learning**: The MoE router learns to select between experts when predicting each target token (e.g., preferring the prefix expert for early tokens and the global expert for later tokens). Crucially, **no explicit policy labels are used**—the router *implicitly* learns when input is sufficient for generation.  \n   - **At inference** (streaming mode), the global expert is *unavailable* (since future input is unknown). If the router \"attempts\" to use the global expert (i.e., assigns it high weight), this signals **insufficient input**—trigerring a *wait* decision. Thus, the MoE mechanism **naturally emerges as a read/write policy** without supervision.  \n2.  **Difference to CTC-based policy or attention-based policy:** \n   - **CTC based policy** [2]  These require learning CTC force alignments between source speech and target translations. In multilingual settings, this becomes problematic as the system must establish alignments across multiple target languages—a challenge with no demonstrated successful implementations in multilingual SimulST to date as far as we know.\n   - **Attention-based policy** ( [3]) These leverage attention patterns from offline models for streaming decisions. While innovative, they typically require using the offline model without streaming-specific fine-tuning, potentially compromising performance in real-time scenarios.\n\n[1] Jia, Yeting et al. “CVSS Corpus and Massively Multilingual Speech-to-Speech Translation.” *International Conference on Language Resources and Evaluation* 2022.\n\n[2] S. Zhang, Q. Fang, S. Guo, Z. Ma, M. Zhang, and Y. Feng, “StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning,” 2024.\n\n[3] S. Papi, M. Negri, and M. Turchi, “Attention as a Guide for Simultaneous Speech Translation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 13 340–13 356."}, {"review_id": "7Xm5wUVyrO", "reviewer": "Reviewer_2Fcn", "summary": "The paper presents a learned policy for simultaneous speech translation (SimulST) called SimulMEGA. SimulMEGA combines a prefix-based SimulST policy, which is trained in an unsupervised way, with Mixture-of-Experts (MoE) to improve the system's performance. The framework is applied to both speech-to-text (S2T) and text-to-speech (TTS). The SimulST model is trained in two steps: an offline pretraining and a simultaneous training with a standard prefix loss and a novel refiner loss for the MoE. The performance is validated on six languages and popular benchmarks and ablation studies are presented.", "questions": "- I suggest adding LAAL as a metric, since it is now the standard for SimulST evaluation and avoids rewarding over-generating systems.\n- Why did the author choose the wait-k and the DigSST policies for comparison while excluding state-of-the-art works such as StreamSpeech, NAST-S2X, EDAtt, and AlignAtt? \n- What is the framework used for evaluation? I suggest adding to the inference details (and, maybe, in the limitations) that the model has been tested on short-form and human-segmented test sets such as CoVoST and FLEURS, thereby simulating only part of a real-world SimulST model.\n- How much does the choice of the mixed chunk-AR and NAR encoder affect the final results?", "limitations": "Limitations are really strict to the point. I suggest elaborating a little on the drawbacks mentioned in lines 299-301.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- It is the first time that the MoE approach is applied to SimulST, to the best of my knowledge.\n- The architectural modifications required to apply this method are minor, facilitating its adoption.\n- The results of the proposed architecture are promising, outperforming the tested baselines.\n- The ablation study clarifies the importance of some experimental choices.\n\nWeaknesses:\n- No results reported using LAAL, which is the most used latency metric for SimulST (e.g., see the last IWSLT Evaluation Campaign editions from 2024 and 2025)\n- Insufficient baselines and lack of clarity in their presentation: The paper compares with \"Seamless\", but it is not clearly stated that the original SeamlessM4T models (both Medium and Large-v2) are different from the simultaneous version of Seamless, StreamingSeamless [1], which I suppose is the model adopted for the comparisons in Figure 3. Moreover, the wait-k policy (which is one of the two terms of comparisons for SimulST) has been outperformed for many years now (see also the related works), and is not a strong term of comparison. The choice of the remaining baseline, Dig-SST, has not been justified, and it is unclear why recent state-of-the-art SimulS2TT works (already cited by the authors), such as models tailored for SimulST tasks and publicly available (such as StreamSpeech [2] and NAST-S2X [3]) and policies applicable to models without any retraining (such as EDAtt [4] and AlignAtt [5]), have not been used for this comparisons. It is good to report additional baselines, such as wait-k, but the set of baselines should be carefully selected and clearly explained to ensure the validity of the paper's claims. Likewise, results for the SimulS2ST task lack comparison with the aforementioned works (StreamSpeech and NAST-S2X), which also cover the speech-in-output modality. \n- It is not clear if the system actually processes long-form speech (akin to real-world streaming processing) or if the speech has been previously segmented into shorter segments. It seems that the authors used CoVoST and FLEURS, which are human-segmented, but no details about this are provided. Following recent work [6], this should be clarified to position the paper's contribution appropriately.\n- While it is true that the approach requires a minimal modification in the architecture, it requires an ad-hoc simultaneous training stage to be applied. Moreover, the pretraining stage, despite being offline, already entails some choices for making the model more efficient during inference (chunk-AR and NAR), and even if desirable in general, it is not the standard way offline S2TT and S2ST models are trained, meaning that the same findings might be not applicable when starting from common, publicly-already available models.\n\n[1] Barrault, L., Chung, Y.A., Meglioli, M.C., Dale, D., Dong, N., Duppenthaler, M., Duquenne, P.A., Ellis, B., Elsahar, H., Haaheim, J. and Hoffman, J., 2023. Seamless: Multilingual Expressive and Streaming Speech Translation. arXiv preprint arXiv:2312.05187.\n\n[2] S. Zhang, Q. Fang, S. Guo, Z. Ma, M. Zhang, and Y. Feng, “StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning,” 2024.\n\n[3] Zhengrui Ma, Qingkai Fang, Shaolei Zhang, Shoutao Guo, Yang Feng, and Min Zhang. 2024. A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1557–1575, Bangkok, Thailand. Association for Computational Linguistics.\n\n[4] S. Papi, M. Negri, and M. Turchi, “Attention as a Guide for Simultaneous Speech Translation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 13 340–13 356.\n\n[5] S. Papi, M. Turchi, and M. Negri, “AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation,” in INTERSPEECH 2023, 2023, pp. 3974–3978.\n\n[6] Papi, S., Polak, P., Macháček, D. and Bojar, O., 2025. How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System?. Transactions of the Association for Computational Linguistics, 13, pp.281-313.", "was_revised": true, "final_justification": "I decided to keep my score, as I think it already represented the correct assessment of this work. The weaknesses are clear from my comments, as I think that, despite being a performing method (and the reason for my positive rating), the overall paper is strongly based on architectural requirements that are not standard at all in practice (i.e., inference-efficient approaches such as chunk-AR and NAR) and therefore the claim that it only requires a lightweight finetuning is not true. The additional experiments provided in the rebuttal regarding the change in architecture are not compared with other state-of-the-art methods (AlignAtt, EDAtt) applied to the mentioned architectures, but are only compared with SeamlessStreaming. Lastly, the evaluation framework is not very clear, as it does not follow standard frameworks and would be difficult to reproduce.", "rebuttal": "Dear Reviewer,\n\nThank you for your insightful comments and constructive feedback on our work. We have carefully addressed each of your concerns as follows:\n\n### Addressing Weaknesses\n\n1. **Latency Evaluation**: We have incorporated numerical evaluation results on the CoVoST2 testset using LAAL, as the latency metric, presented in Table 1. The results confirm our original conclusion that SimulMEGA continues to marginally outperform baseline methods. In the final version, all quality-latency curves will be systematically re-plotted using the LAAL metric for comprehensive comparison.\n2. **Baselines and presentation clarification**: \n   - While AlignATT did not function effectively with our base model (due to persistent attention fixation on particular frames leading to over-generation), we successfully integrated EDATT (Papi et al., 2023), which shares conceptual similarities. As shown in Table 1, EDATT outperforms both DIG-SST and wait-K baselines, further validating our approach's competitive advantage.\n   - Publicly available implementations of StreamSpeech and NAST-S2X differ significantly from SimulMEGA in three critical dimensions: model architecture scale, training data volume, and multilingual capability scope. To provide meaningful comparison while acknowledging these differences, we present results on the CVSS FR→EN benchmark in Table 2, where SimulMEGA achieves more than 10 BLEU points higher than these public implementations. We maintain AL (Average Lagging) as the latency metric to ensure comparability with the original publications. These comparative results will be included in the appendix of the final version.\n   - Regarding the \"Seamless\" baseline: this refers to the integrated system described in Section 6 of the Seamless documentation, which combines Seamless Expressive with Seamless Streaming components. We will expand our baseline descriptions in the appendix to prevent any potential misinterpretation.\n3.   **Real-world streaming processing:** Our supplementary materials include a deployed web demonstration that processes streaming input using a VAD (Voice Activity Detection) system to segment audio into optimal 10-20 second chunks for model processing. While standardized long-form evaluation benchmarks remain scarce in the field, we constructed a specialized test set following Whisper's methodology, comprising 17 TED-LIUM talks with an average duration of nine minutes. We translated the reference transcripts to Chinese using ByteDance's Doubao translation service for evaluation. The S2TT performance metrics are presented in Table 3.\n4. **Reproducibility on standard pretrained ST model:** Unlike attention-based policies (EDATT and AlignATT) or divergence-based approaches (DIG-SST), SimulMEGA introduces two features: (1) it fine-tunes the base model without freezing parameters, and (2) it learns the read/write policy without relying on empirical patterns. Our training process demonstrates stable convergence in both loss values and routing score distributions. Theoretical analysis suggests minimal impact from the chunk streaming mask design, which is empirically validated by consistent performance across different mask configurations. Furthermore, SimulMEGA's successful adaptation to streaming TTS demonstrates the method's robustness across both architectural variants and task domains.\n\n### Responding to Questions\n\n1. **Evaluation framework**: While we did not utilize SimulEval as our primary evaluation framework, we implemented a rigorously controlled evaluation pipeline where all models undergo identical processing conditions. Our latency metrics directly implement the same computational definitions as SimulEval to ensure measurement consistency with established SimulST literature.\n2. **Effect of streaming mask**: Although comprehensive ablation studies on architectural variations were not our primary focus, we conducted targeted experiments evaluating different mask configurations. Our findings indicate that the 640ms chunk mask configuration yields performance equivalent to processing without chunk masking. However, we observed measurable performance degradation with the 320ms configuration, suggesting an optimal balance between temporal resolution and contextual completeness at the 640ms threshold.\n\nWe appreciate your thorough review and welcome any additional questions you may have regarding these clarifications. All revisions outlined above will be incorporated into the final manuscript to enhance both methodological transparency and experimental rigor.\n\n###### Table 1: CoVoST2 Evaluation\n\n|                        | X-EN     |           |          |           |          |      | EN-X     |           |          |           |          |\n| ---------------------- | -------- | --------- | -------- | --------- | -------- | ---- | -------- | --------- | -------- | --------- | -------- |\n| **Seamless Streaming** | **0.3**  | **0.5**   | **0.7**  | **0.9**   | **1**    |      | **0.3**  | **0.5**   | **0.7**  | **0.9**   | **1**    |\n| BLEU                   | 33.50    | 33.75     | 34.05    | 34.37     | 35.05    |      | 33.35    | 33.70     | 33.83    | 34.11     | 35.41    |\n| LAAL                   | 2.280    | 2.410     | 2.522    | 2.665     | 3.149    |      | 1.966    | 2.065     | 2.144    | 2.268     | 2.660    |\n| **Wait-K**             | **2.5**  | **1.9**   | **1.5**  | **1.1**   | **0.5**  |      | **2.5**  | **1.9**   | **1.5**  | **1.1**   | **0.5**  |\n| BLEU                   | 29.27    | 31.82     | 32.64    | 33.45     | 34.93    |      | 34.25    | 35.83     | 36.12    | 36.49     | 37.41    |\n| LAAL                   | 1.731    | 2.151     | 2.296    | 2.565     | 3.309    |      | 1.975    | 2.294     | 2.376    | 2.541     | 3.127    |\n| **DigSST**             | **0.1**  | **0.07**  | **0.05** | **0.03**  | **0.01** |      | **0.1**  | **0.07**  | **0.05** | **0.03**  | **0.01** |\n| BLEU                   | 31.43    | 33.53     | 35.05    | 35.99     | 36.52    |      | 17.12    | 17.51     | 18.10    | 19.44     | 23.38    |\n| LAAL                   | 2.035    | 2.237     | 2.442    | 2.752     | 3.340    |      | 0.734    | 0.770     | 0.808    | 0.902     | 1.252    |\n| **EDATT**              | **0.03** | **0.025** | **0.02** | **0.015** | **0.01** |      | **0.03** | **0.025** | **0.02** | **0.015** | **0.01** |\n| BLEU                   | 33.70    | 34.81     | 35.60    | 36.09     | 36.56    |      | 35.00    | 35.82     | 36.77    | 37.63     | 38.20    |\n| LAAL                   | 1.880    | 2.062     | 2.301    | 2.610     | 3.056    |      | 1.963    | 2.131     | 2.384    | 2.688     | 3.129    |\n| **SimulMEGA**          | **0.7**  | **0.6**   | **0.5**  | **0.4**   | **0.3**  |      | **0.7**  | **0.6**   | **0.5**  | **0.4**   | **0.3**  |\n| BLEU                   | 35.58    | 35.97     | 36.27    | 36.46     | 36.60    |      | 37.83    | 38.11     | 38.14    | 38.28     | 38.39    |\n| LAAL                   | 1.729    | 1.860     | 1.982    | 2.104     | 2.233    |      | 2.059    | 2.246     | 2.400    | 2.542     | 2.690    |\n\n\n\n###### Table 2: Evaluation on CVSS FR-EN\n\n| **SimulMEGA**    | **Threshold** | **0.8**     | **0.7**     | **0.6**        | **0.5**       | **0.4**       | **0.3**       | **0.2**       | **0.0**  |\n| ---------------- | ------------- | ----------- | ----------- | -------------- | ------------- | ------------- | ------------- | ------------- | -------- |\n|                  | ASR-BLEU      | 33.42       | 34.23       | 33.79          | 34.08         | 34.46         | 34.72         | 35.08         | 35.74    |\n|                  | AL            | 1.391       | 1.511       | 1.605          | 1.767         | 1.810         | 1.929         | 2.058         |          |\n| **NAST-S2S**     | **t_s + t_a** | **320+320** | **640+640** | **1280 +1280** | **1920+1280** | **1920+1920** | **2560+1280** | **2560+2560** |          |\n|                  | ASR-BLEU      | 19.67       | 19.15       | 20.20          | 21.77         | 22.70         | 23.58         | 24.88         |          |\n|                  | AL            | -0.393      | 1.533       | 3.330          | 3.975         | 4.335         | 4.408         | 4.976         |          |\n| **StreamSpeech** | **C×40ms**    | **320**     | **640**     | **960**        | **1280**      | **1920**      | **2560**      | **3200**      | **4800** |\n|                  | ASR-BLEU      | 22.89       | 24.41       | 25.00          | 25.20         | 25.50         | 25.68         | 25.75         | 26.14    |\n|                  | AL            | 1.270       | 2.326       | 2.803          | 3.146         | 3.450         | 3.812         | 4.157         | 4.873    |\n| **Seamless**     | **Threshold** | **0.3**     | **0.5**     | **0.7**        | **0.9**       | **1.0**       |               |               |          |\n|                  | ASR-BLEU      | 31.63       | 31.60       | 31.60          | 31.62         | 31.99         |               |               |          |\n|                  | AL            | 1.819       | 1.947       | 2.058          | 2.199         | 2.826         |               |               |          |\n\n###### Table 3: Long-form S2TT result on TED-LIUM\n\n|           | BLEU  | AL       | LAAL     |\n| --------- | ----- | -------- | -------- |\n| SimulMEGA | 55.13 | 16.15655 | 24.62662 |\n| Seamless  | 38.18 | 7.016107 | 15.42395 |"}]

Author rebuttal:

Meta-review: (a) Summary:

This paper introduces SimulMEGA, a Mixture-of-Experts (MoE) based framework for simultaneous speech translation (SimulST) that learns read/write policies in an unsupervised manner. The method extends to both speech-to-text (S2TT) and streaming text-to-speech (TTS), with minimal architectural modifications. The authors demonstrate improved trade-offs between latency and quality compared to Seamless and other baselines across six languages. They also provide ablation studies, long-form speech experiments, and supplementary demonstrations.

(b) Strengths:

1. Clear technical innovation through the introduction of an MoE refiner that yields implicit read/write policies without inference-time overhead
2. Strong empirical performance, outperforming Seamless on CoVoST2 and FLEURS benchmarks with smaller model size
3. Extension beyond S2TT to simultaneous TTS and speech-to-speech translation, highlighting versatility
4. Thorough experimental analysis, including ablations, long-form settings, and supplementary demonstrations
5. The method is relatively lightweight and adaptable to multiple architectures, as shown by experiments on Whisper-derived and Qwen-based systems


(c) Weaknesses:

1. Evaluation originally lacked LAAL (Length-Adaptive Average Lagging), now the standard in SimulST. Authors added results during rebuttal
2. The selection of baselines ignored  state-of-the-art works such as StreamSpeech or NAST-S2X.
3. The long-form evaluation relies on VAD-based segmentation rather than direct unbounded-streaming methods, which reviewers noted as a limitation.
4. Some reliance on non-standard architectural modifications (chunk-AR, NAR encoders) limit the general applicability to standard S2TT/S2ST architectures.


(d) Reasons for decision:

The paper presents meaningful contributions to SimulST, with demonstrated empirical improvements over strong baselines. It proposes a novel use of MoE for implicit policy learning and shows adaptability across tasks and architectures. However, limitations in evaluation protocol, baseline coverage, and clarity reduce its case for spotlight or oral. Thus poster acceptance is appropriate: the work is innovative and technically solid, but does not yet demonstrate the broad maturity and impact required for a higher recommendation.

(e) Discussion and rebuttal:

* Reviewer U9os raised concerns about the use of AL as a latency metric, absence of strong baselines (e.g., AlignAtt), and lack of threshold details. The authors addressed these by providing LAAL results, adding EDAtt and AlignAtt comparisons, and clarifying thresholds. This satisfied the reviewer, who maintained a positive score.

* Reviewer Eauk emphasized lack of clarity around MoE motivation and TTS implementation. The rebuttal clarified the roles of prefix/global experts and provided more implementation details, which addressed most concerns.

* Reviewer FapF requested deeper analysis across language pairs and more detailed S2ST benchmarks. The authors provided supplementary tables and clarified benchmark scope. Reviewer acknowledged the responses positively.

* Reviewer 2Fcn strongly emphasized missing baselines and clarity on evaluation frameworks. Authors added AlignAtt results, low-latency regime experiments, and comparisons to NAST-S2S and StreamSpeech. Nonetheless, the reviewer remained critical about the reliance on non-standard architectures and the absence of truly low-latency (<1.5s) evaluation. The final score stayed borderline positive.

Overall, all reviewers acknowledged improvements after rebuttal. While some concerns (especially about architecture assumptions and low-latency evaluation) were only partially addressed, the reviewers did not oppose acceptance. The consensus points to a technically solid, novel contribution that deserves inclusion as a poster.
Decision: Accept (poster)