MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Authors: Sahil Kumar
Namrataben Patel
Honggang Wang
Youshan Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: text-to-speech (TTS)
speech synthesis
voice cloning
Mamba
state space models (SSM)
diffusion TTS
prosody modeling
streaming/low-latency
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: SSM-only TTS conditioning at inference (no attention/RNN); a gated Bi-Mamba improves long-form stability/streaming and gives ~1.6× encoder speed with modest, statistically significant quality gains.

Abstract: MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference—removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody—while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time $\mathcal{O}(T)$ conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba--TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel--diffusion--vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba--attention hybrids in MOS/CMOS, F$_0$ RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by $1.6\times$. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability. 
Code available at: \url{https://github.com/sahilkumar15/MVC}.

Published as a conference paper at ICLR 2026
MAMBAVOICECLONIIN EFFICIENT AND EXPRESSIVE SEXT-TO-SPEECE VIA STATE-SPACE MODELING AND
DIFFUSION CONTROL
Sahil Kumar
Namrataben Patel
PhD Program in Mathematics Yeshiva University
PhD Program in Mathematics Yeshiva University
New York, NY 10033, USA skumar @ma 1..11.11111.. yu. edu
New York, NY 10033, USA
npate 13mm i1, yu......... edd
Honggang Wang
Youshan Zhang'
Department of Computer Science Engineering Yeshiva University
School of Artificial Intelligence
Chuzhou University Anhui, 239000, China
New York, NY 10033, USA honggang wang@yu.eds
youshan,zhann@@hhauu edu. cn
ABSTRACT
Mamba VoiceCloning (MVC) asks whether the conditioning path of diffusion- based TTS can made fully SSM-only at inference removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody-wol whhle preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TIB systems that remain hybrid at inference, MVC removes attention- based duration and style modules under a fixed StyleTTS2 mel-diiffusion-vocodee backbone. Trained (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but sta- tistically reliable gains over StyleTTS2. VITS, and Mamba-attentinn hybrids MOS/CMOS, Fo RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and de- ployability. Code available at: pectps://gtthue com/ sahi luuarr157
JJSpeech/LibriTTT and evaluated VCTK, CSS10
MVC
INTRODUCTION
Text-to-Speech (TTS) systems continue improve in naturalness and expressive control Li et al. (202bb): Kim et al. (2021); Ning et al.] (2019): Tan et al. (2021), yet most conditioning stacks rely on transformer attention Vaswani al. (2023) Wang al. (2017) recurrent modules. Atten- tion introduces quadratic computational and memory complexity and global context mixing, while recurrent architectures exhibit long-range drift and unstable memory dynamics. Linear attention variants Choromanski et al. (202l) reduce asymptotic cost but preserve global interactions that com- plicate streaming. Meanwhile, diffusion decoders Popov al. (202T; [Huang et al. (2022): Liu al. 2 Kong et al.) (2020); ZZhang et al. (203) dominate inference runtime, making encoder effi ciency central deployment.
Why Mamba vs. Transformer/RNN. State-space models (SSMs), particularly Mamba Gu & Dao 2024 provide bounded activations, linear-time sequence scans, and state-persistent streaming.
Corresponding author. This research was funded by the Research Project of Chuzhou University (Grant
No. 2025gd36).

---

Published as a conference paper at ICLR 2026
These properties reduce memory pressure relative to attention- models and mitigate drift in recurrent architectures, supporting stable conditioning over multi senteis inputs. However, ex- isting Mamba- -TS systems Diang et al. (2024); Zhang et al. (2024) remain hybrid at inference, retaining attention-based duration or style modules that limit streaming robustness.
This work investigates whether diffusion- -based TTS can adopt fully SSM-only conditioning stack at inference for text, rhythm, and prosody under a strictly matched mel-diffusion -vocooer pipeline. The StyleTTS2 decoder and vocoder remain fixed; only the conditioning path is redesigned. MVC introduces three selective SSM modules: a gated bidirectional Mamba text encoder; a Temporal Bi-Mamba aligned using lightweight monotonic teacher during training only; and an Expressive Mamba with AdaLN modulation. A gated forward-backwart backward fusion mechanism replaces concat- only bi-Mamba fusion used in prior work.
Why NaturalSpeech 3, Cosy 3, and HiggsAudio- V2 are not direct baselines. Industrial- scale TTS systems such as NaturalSpeech 3 Du et al.] (2024), CosyVoice DDu al. (2025), and HiggsAudio- V2 Boson AT (2025) rely mulli-hundred-thousaare to million-hour proprietary mul- tilingual corpora, LLM scale semantic encoders, and multi-stage pipelines. Their performance is driven primarily by scale rather than aanditioninn-architettrr design, making them unsuitable as decoder-matched baselines for a controlled architectural study. For fairness, comparisons are re- stricted to open-data systems trained under identical preprocessing. mel front-end, vocoder, and optimization schedules; a detailed contextual comparison is provided in Appendix [F
Scope of evaluation. The evaluation covers in-distribution speech (LISpeech) Ito & Johnson
2017
LibriTTS Zen et al.] (2019)). zero-shot speakers (VCTK) /Veaux et al. (2017)) cross-lingual CSSIO
(ES/DE/FR) Park & Mule (2019), and 2-6 minute Gutenberg passages for long-form testing. MVC yields consistent improvements over StyleTTS2 Li et al. (2023b), VITS Kim et al. (2021), and capacity-matched Mamba hybrids Jiang et al. (2024); Zhang et al. (2024) in MOS, CMOS, Fo RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by a factor of 1.6. Streaming with a finite look-ahead of 0.5-2.0 seconds preserves non-streaming qual-
ity, consistent with prior monotonic and streaming sequence modeling approaches, and the diffusion decoder remains the primary latency source |Popov et al.] (202T): [Jeong et al. (202T). Additional runtime, memory, and SSM-sensitivity analyses appear in Appendix|A. A.
Contributions. (1) A diffusion- based TTS system with a fully SSM-only inference-time condi- tioning path spanning text, rhythm, and prosody under a fixed decoder. (2) A gated bidirectional Mamba fusion with AdaLN that improves long-range prosody stability and reduces drift on multi- sentence and nut-of-distributiin text. (3) Protocol- and matched basclines that isolate the architectural impact of removing inference-time attention. (4) A deployment-orientee analysis cov- ering memory usage, throughput, SSM hyperparameter sensitivity, long-form behavior, and finite look-ahead streaming, demonstrating predictable linear-time characteristics.
2 RELATED WORK
TTS conditioning spans attention-based encoders, diffusion decoders, zero-shot systems, and recent state-space models. MVC examines how these paradigms affect efficiency, memory usage, and long-form stability.
Attention-based TTS. Transformer-basedd pipelines such as Tacotron, Tacotron2, JETS, StyleTTS, and StyleTTS2 Wang et al. (2017); Shen et al. (2018): [im al. (2022) ; Li al. (003b); Vaswani et al. (202)) provide strong alignment and style modeling but rely on quadratic attention complexity. Even linear attention variants Wang et al. 2020); Choromanski et al. (20TT) maintain global interac-
tions that couple text, duration, and prosody, making streaming synthesis sensitive to memory usage. These limitations motivate conditioning stacks with linear-time behavior and bounded activations.
Zero-shot and large-scale systems. NaturalSpeech Ju et al. (2024), Voice D et al. (2025) and HiggsAudio-Ve V2 Boson Al (2025) leverage
-hhuuddddd--ouuand---- to million-hour propri- etary corpora, multilingual pipelines, and LLM-scale encoders integrating text understanding and expressive control. Their performance is primarily driven by scale rather than conditioning design and cannot be reproduced under academic budgets open-data constraints. MVC instead evaluates conditioning architecture under a fixed diffusion- pipeline and controlled data scale. A contextual comparison appears in Appendix F
2

---

Published as a conference paper at ICLR 2026
Ted Itsir (Tokeo)
RiAle R-Almmee
Croader
Text Algner
Tomppral Profictr
Analysl (tee-ppcdiggarr)
Expeasiie Nomba Enizder
Phia Cinadar
Coprasaive Prediddrr
Audio gra) (Spectogremt)
Tempoool Maniaa Frizader
Decade Siice
proddiana (P
Advererall
traning
Derrinnttes MFFO uRam
Vnll
Devoiee
Canae nead Audo
Figure I: Overview of Mamba VoiceCloning (MVC). The framework uses Bi-Mamba Text Encoders for phoneme modeling. Temporal Bi-Mamba for rhythmic aligmment, and an Expressive Mamba for prosodic control. A lightweight aligner (dotted box) provides phoneme-frame supervision only during training, ensuring an SSM-only encoder at inference. Conditioning features drive diffusion decoder and vocoder for waveform synthesis.
SSMs and Mamba hybrids. Mamba introduces input-gated selective scans for linear-time model- ing with bounded activations Gu Dao (2024). SSMs have been explored in speech enhancement, ASR, and hybrid TTS encoders Miyazaki et al. (2024); Jiang et al. (2024); Kumar al. (2024); Zhang et al. (202)). However, existing Mamba- TTS systems remain hybrid at inference, rein- troducing attention recurrence for duration and style modeling and limiting streaming stability. Runtime and finite look-ahead behavior under diffusion decoders are rarely analyzed.
Positioning of MVC. MVC eliminates attention and explicit RNN-style recurrence layers across the entire inference-ting time conditioning stack for text, rhythm, and prosody, retaining a lightweight aligner only during training. It replaces concat-only bi-Mamba fusion Diang et al. (2024); /Zang fusion and AdaI modulation. Protocol- and capacity- matched baselines isolate the architectural effects of removing inference-time attention and intro- ducing gated AdaLN fusion (Table 12] Appendix A.2).
let al. (2024) with gated forward-backwar
3 METHODOLOOGY
MVC replaces all inference-ting attention and recurrence with selective state-space models (SSMs) for text, rhythm/duration, and prosody. lightweight attention- based aligner provides phoneme- frame supervision during training and is discarded at inference. This yields an SSM-only condition- ing stack with linear-time scans and bounded activations. Unlike prior bi-Mamba encoders Diang et al. 22024); Zhang et al. (2024), MVC employs a gated bidirectional Mamba text encoder, a Tem- poral Bi-Mamba, and an Expressive Mamba with AdaLN conditioning. All Mamba blocks use a state dimension of 96, depthwise convolution kernel size 5, and gating temperature = 1.0 unless otherwise specified.
High-level overview. Figure summarizes MVC. From phonemized text and reference audio, MVC produces three conditioning streams: a gated Bi-Mamba text encoder, a Temporal Bi-Mamba for rhythm/duration, and an Expressive Mamba operating on mel spectrograms with AdaLN. These are fused in speech-dynamics stage and passed to the fixed StyleTTS2 decoder and vocoder. Be- cause decoder and vocoder components are identical across MVC and all baselines (StyleTTS2, VITS, JETS, Hybrid-Mamba, Bi-Mamba), differences in MOS/CMOS, WER, pitch stability, and runtime directly reflect conditioning stack design. During training, the aligner provides soft phoneme- franmme-frame weights; at inference it is discarded, and all encoder modules run in ((T) without attention maps. For streaming, the bidirectional text encoder is replaced by a causal Uni-Mann Mamba with look-ahead I. (.3c enabling explicit latency-context trade-offs.
seaitioninggstack
3

---

Published as a conference paper at ICLR 2026
Notation. Let T. and Tm denote the number of text tokens and mel frames, respectively; d the text embedding dimension; d, the SSM hidden dimension; and de the style-embedding dimension. We write E R7, for token embeddings. M I FxT, for log-mel spectrograms, and R for the global style vector. A compact symbol table in Appendix B.1 consolidates notation and abbreviations for readability.
3.1 INPUT PROCESSING
Given waveform Swav E Rd= at 2Hz we compute an 80-bin log-mel spectrogram M E RxII using Hann-window STFT (FFT 1024, hop 256), mel filterbank projection, and log compression with c=10-;; the full formulation is in Appendix B Text is normalized and phonemized us- Bernar CSSI0 ES/DE/FR); phoneme-graphent
ing phonemizer" Bernard & Titeux (2021). yielding tokens (Wi..... WT, (with language tags for and phoneme-level conditioning have been shown to im- prove prosody and robustness in neural TTS Jia al. [2021 Li et al. (2023a). Token embeddings and the global style embedding are computed as
Tm
x = ( Wr,
RT.x
e
Jo( ERd.
(1)
T
where fo is a shallow conv/GRU module shared across encoders. This embedding captures coarse timbre and expressiveness and provides a shared conditioning signal, important for long-form sta- bility, zero-shot speakers, and cross-lingual tests (Sec.)
3.2 ENCODER STACK
The encoder stack contains three SSM modules: (i) gated Bi-Mamba text encoder (Sec. (Sec. 3.2.1 : (ii)
Expressive Mamba encoder (Sec. (.2.22): and (iii) Temporal Bi-Mamba encoder (Sec. (.2.3] Appendix E.3 and Table 19 shows that moderate hyperparameter variations produce only small changes in MOS and RTF, confirming that performance gains arise from architecture rather than tuning.
3.2.1 BI-MAMBA TEXT ENCODER
We replace self-attention with bidirectional Mamba blocks to obtain linear-time text encoder with bounded activations. Given x E xd we project to d, and apply forward and backward Uni- Mamba scans,
hy (ambay(x), h, Mambas (x),
(2)
where each block follows the selective state-space update Gu & Dao (024)). providing O(T,) complexity and numerically stable recurrent dynamics (Appendix B.3). The lincar-time scan- ning and bounded activation updates ensure that the encoder remains stable long phoneme quences, avoiding attention-fragman fragiiontrrammeetattiino and activation drift that occur in attention-based dura- tion and prosody predictors. These properties ward/backward scans preserve consistent state magnitudes across multi-sentence and multi-minute segments, providing predictable accumulation of prosodic cues without degradation over time.
essential for MVC's long-form behavior: for-
Prior bi-Mamba TTS encoders combine directions via simple concatenation; MVC instead employs gated fusion mechanism:
hr = (o(W, (hy: ha]) O hyl)
(3)
with W g E 2d2 x2d, and W, E 2dd xds The gating module modulates forward/backward con- texts based local syntactic cues, improving long-range prosody, reducing drift, and maintaining temporal coherence in extended passages (Sec. 5 Tables 3 Appendix I.riiixxE.reeeeeeeee reports gate statistics on 2-6 minute Gutenberg passages, demonstrating that the gating pattern remains stable and does not collapse, thereby confirming the model's 's robustness under long-form and streaming conditions.
To incorporate speaker/style information, we apply AdaLN using embedding hr = JalaIN(hhr,ee e),
(4)
where AdaLN(z,e) 7(e) @ LN(z) 3(e). This gated bi-Mamba AdaLN architecture is not present in prior Mamba-TTe TTS systems; Table shows that removing either mechanism significantly degrades long-form MOS and pitch RMSE.
4

---

Published as a conference paper at ICLR 2026
3.2.2 EXPRESSIVE MAMBA ENCODER
The Expressive Mamba encoder injects speaker-specific prosody into the acoustic representation in linear time. Given mel features M and style embedding e. we apply a gated transformation with AdaLN conditioning (Appendix B.4), followed by a Mamba block:
b Mamba(hM..) E R xTmxxd..
(5)
where hM.s is the style-conditioned input. This module is fully SSM-based (no attention) and captures slow prosodic dynamics over long inputs; removing it produces the largest CMOS drop among encoder components on OOD data (Table
3.2.3 TEMPORAL BI-MAMBA ENCODER
The Temporal Bi-Mamba encoder models rhythmic structure and phoneme- durattion alignment. The style embedding is broadcast over frames and modulated via shallow gated transform, producing hg E RT xds_ Forward and backward Mamba blocks plus a local Conv ID then capture context- dependent timing patterns, and their outputs are fused linearly:
hg = by: ho] W
(6)
We keep this fusion linear (no second gating) because prosody disentanglement is handled upstream by the text and expressive encoders; Appendix F.3 shows that adding gating here increases activation memory without consistent MOS gains, clarifying why MVC does not use gating in this module.
3.3 ALIGNMENT AND PITCH MODELING
Training-time aligner. The aligner is a 2-layer transformer with 4 attention heads and hidden di- mension 256, trained with monotonic alignment loss. It maps token encodings hy,, frame-level weights RTxx'', During training only, lightweight attention-based aligner maps token-level encodings hr,s to frame-synchronous representations. Given M and hy,,, the aligner computes attention weights a E RT=xTT and an aligned encoding
h = aa
(7)
The aligner is a 2-layer, 4-head transformer (hidden size 256) used only training-time teacher and completely removed at inference. Appendix p.r/prt its attention maps and shows that MVC tolerates moderate alignment noise (WER increase 0.4 points, MOS drop < 0.05), indicating that MVC does not rely on perfectly specified aligner and preserving the SSM-only deployment claim.
Pitch modeling. Pitch modeling uses both expressive and temporal encodings. We fuse hg and hg via a gated block to obtain hp RR xdl, and predict the final Fo contour via
Fo hp W F bp.
(8)
This design avoids an additional attention based pitch predictor; the prosody path remains SSM-only at inference, which is important for counded-memory streaming.
3.4 SPEECH DYNAMICS AND DECODER CONDITIONING
The speech-dynamics stage refines phonetic and prosodic representations into decoder-ready fea- tures. Starting from 1 and hp, a temporal predictor (Conv ID + SSM) produces a rhythm-aware representation, which is fused with hp via a gated block and projected to iittammental-requeene trajectory Fo and residual noise vector n. The final conditioning sequence
hp [o: n] e R.T (m+d))
and is passed to the diffusion decoder. All dynamics and fusion operations here use SSMs and point- wise gates, so the conditioning path remains linear-time and attention- free at inference. Additional architectural details appear in Appendix B.4
5

---

Published as a conference paper at ICLR 2026
3.5 DECODER STAGE AND LOSSES
The decoder uses the StyleTTS2 diffusion model Li et al. (202b)); Popov et al. (022]) : Kong al. (2020) with matched vocoder; MVC modifies only the conditioning path. Given hn,
the decoder predicts M DiffusionDeccder(bb) {a+]), which the vocoder converts to wave- form Vocoder(M). We reuse the StyleTTS2 multi-period and multi-resolution discriminators (MPD+MRSD) and mel reconstruction loss. The total loss combines mel, adversarial, and alignment
terms:
Lisual meet + Cadv Calign
(10)
Reusing the StyleTTS2 diffusion and vocoder stack ensures protocol-matchen comparisons: Ta- bles and 12 show that MVC improves quality, long-form robustness, and encoder efficiency under an identical decoder/vocoder configuration. Full loss definitions appear in Appendix B.5
(..5
Training procedure and baselines. MVC is trained on triples (x, M, Swav) using Liotal with
AdamW, cosine decay, gradient clipping, EMA, and automatic mixed precision. All hyperparam- eters (batch size, training steps, SSM configuration) are held fixed across MVC and Mamba- baselines to ensure protocol-level parity. Full training steps (Algorithm D and implementation details for Hybrid-Mamba and Bi-Mamba (Concat-only) are provided in Appendix [B.8 and Ap- pendix B.6 where Table 12 shows that MVC's gains persist under strict reproduction controls.
EXPERIMENTS
4.1 DATASETS AND PREPROCESSING
We train on LJSpeech to & Johnson 2017
h2 1 spk.) and LibriTTS Zen et al.
(2019)
h, 1,151 spk.), and evaluate on VCTK Veaux et al. (2017) (109 spk.; zero-shot) and CSSTO
ES/DE/FR Park & Mulc (2019). Audio is resampled to kHz and converted to 80-bin log-mels; text normalized and phonemized using phonemi zer Bernard & Titeux (202) with language- specific espeak-ng. Speaker conditioning uses MVC's mel-derived embedding (Sec. 3). We eval- uate cross-lingual generalization on CSSIO (ES/DE/FR) to assess how well MVC handles phoneme inventories and stress patterns across languages. Detailed results, including the failure modes ob- served in specific languages such as German and French, are available in Appendix [D.5] For long- form evaluation, we construct 2-6 min Gutenberg passages with lexical de-duplication against the training corpora (Appendix C.I). This strict separation prevents text leakage and ensures that long- form and cross-lingual performance reflects genuine generalization. Additional preprocessing de- tails appear in Appendix
Baselines. We compare against StyleTTS2 (2023b). VITS et al. (202I), and JETS] Lim et al. (2022) (built on non-autoregressive duration modeling ideas such as FastSpeech 2 Ren et al. (2022) : Elias et al. (202T) under a fully matched pipeline with identical text normalization, log- mel settings, corpus- matched vocoders (iSTFTNet for LJSpeech; HiFi-GAN for LibriTTS), fixed 5-step diffusion schedule and the shared optimization and training schedule Appendix C.2 All baselines are re-trained in our codebase with the same data splits, optimization schedule, and early- stopping criteria. To isolate Mamba-specific effects, we additionally include Hybrid-Mamin (Con- cat) and Bi-Mamba (Concat-only) as capacity matched controls. Architectural and conditioning- path details for all models are provided in Appendix C.4 (Table 14). This unified setup ensures that performance differences arise solely from the conditioning-stack design, not from preprocessing. training, or vocoder discrepancies.
capacity-matched
Scope of evaluation. All models share the same data, mel front-end. diffusion decoder, and vocoder. Industrial systems (c.g., NaturalSpeech 3, Cosy Voice 3. HiggsAudio-VD V2) rely on proprietary hundred-thousand- million-hour corpora and large semantic modules, so they are not directly comparable under our open-data, decoder-matched setting (see Appendix F Table 22
4.2 IMPLEMENTATION AND METRICS
Model and optimization. The deployed MVC encoder stack contains 21M parameters. The text, temporal, and expressive Mamba encoders pre-trained for stability and then jointly fine-tuned

---

Published as a conference paper at ICLR 2026
with a StyleTTS2-basees diffusion decoder; the lightweight aligner is used only during training. We use AdamW with cosine decay, EMA, gradient clipping, and mixed precision, and we keep batch size. training steps, SSM configuration, and vocoder settings strictly identical across MVC, VITS, JETS, Hybrid-Mamba, and Bi-Mamba. Inference uses a fixed 5-step diffusion schedule shared across all models, ensuring that performance differences reflect only conditioning architec- ture choices. Full schedules and batch sizes are given in Appendix C
Evaluation protocol. Objective metries (Fo RMSE, MCD, WER, PESQ, RTF) are averaged over three seeds; WER uses an ESPnet LibriSpeech Transformer+LM Subjective evaluation uses Ama- zon Mechanical Turk with 5-10 raters per utterance; MOS/CMOS include 95% confidence intervals and paired t-tests with Holm- -onffrrrrrro correction. We follow StyleTTS2's sampling protocol (80 LibriTTS unseen-speaker clips, IJSpeech ID/OOD clips, 20 CSS10 clips per language). All SSM hyperparameters (state dimension, convolution kernel, gating temperature) remain fixed in main experiments, with sensitivity reported in Appendix F.3] ensuring that MVC's gains do not rely on narrow hyperparameter tuning.
5 RESULTS
Across 500 LJSpeech utterances, the diffusion decoder dominates latency (54.2%), followed by the Mamba encoder stack (31.4%) and the vocoder (14.4%) (Table 15); individual encoder modules con- tribute roughly 13-15 ms each (Appendix D.4]. End-to-end RTF gains are therefore moderate, but the SSMM-nnl conditioning path reduces peak memory and improves encoder throughput, enabling longer sequences and larger batch sizes under fixed diffusion configuration. Table T2 further shows that removing inference-time attention (Bi-Mamba Concat-only vs. Hybrid-Mambaj improves RTF and slightly reduces Fo/MCD/WER, with MVC's gated fusion and AdaLN offering consistent ad- ditional gains. Overall, improvements arise from encoder-side efficiency- lower memory, higher conditioning throughput, and more stable long-form behavior while the diffusion decoder remains the primary latency bottleneck.
Comparison to Mamba-Based and Transformer Baselines. Recent work applies Mamba to speech Miyazaki et al. (2024) Zhang et al. (202); Diang et al. (2024), typically in hybrid ar- chitectures that retain attention or recurrence in duration or style modules and provide limited component-level analysis. MVC instead uses modular bidirectional Mamba encoders for text, tim- ing, and prosody within a unified diffusion pipeline, supported by sapacity-matched baselines and component-wist ablations isolate each module's contribution (Tables 6 12).
All baselines (VITS, StyleTTS2, JETS, and Mamba variants) are trained or reproduced under the same mel front-end, diffusion decoder, vocoder, optimization schedule, and data splits, ensuring that performance differences reflect conditioning cewiitiininnn-rccitttttuee choices rather than training discrepancies. This positions MVC as an encoder-side redesign of diffusion based TTS under controlled, open-data conditions rather than black-box box system dependent on proprietary corpora. Industrial- scall sys- tems such as NaturalSpeech Juuu et al. (2024) and Cosy Voice 3 et al. (2025) achieve higher MOS on hundred-thousann tet-thhuuunnhhuur multilingual datasets using large semantic models and closed pipelines; because they differ fundamentally data scale, task scope, and training infrastructure, we treat them contextual references (Sec. 2 rather than numeric baselines, focusing here on fair comparisons against transformer- and
models trained on the same public corpora.
5.1 SUBJECTIVE AND OBJECTIVE QUALITY
MVC achieves 4.22 MOS-N and 4.07 MOS-S on unseen LibriTTS speakers (Table D. slightly surpassing StyleTTS2 (paired I-test, p 0.01). The gains are modest but statistically robust, indi- cating that the SSM-only conditioning stack im- proves naturalness and speaker similarity without altering the diffusion or vocoder. On LJSpeech (Table 4 MVC attains the best MCD (4.91), highest PESQ (3.85), and lowest RTF (0.0169), with comparable Fo RMSE and WER. Absolute
Table Subjective evaluation on unseen Lib- riTTS speakers.
Model
MOS-N 1
MOS-S f
Ground Truth
4.60
4.35
VITS StyleTTS2 MVC (ours)
3.69
3.54
4.15
4.03
4.22
4.07

---

Published as a conference paper at ICLR 2026
differences (e.g.. MOS = +0.07, RTF N -0.0005-0.001 remain small but consistent across seeds and are statistically significant under coln-Booferrrnii correction, supporting our framing of MVC encoder-side refinement rather than a paradigm shift. The 21M-parameter encoder also reduces activation memory and improves conditioning throughput, enabling longer contexts and larger hatch sizes on the same hardware.
Cross-speaker and cross-lingual. MVC matches or exceeds StyleTTS2 on VCTK and CSS10 ES/DE/FR (Appendix D.5). We follow StyleTTS2's protocol: zero-shot speakers on VCTK and cross-lingual naturalness on CSS10 using screened crowd workers and 95% confidence intervals, en- suring comparable subjective scores. These results show that an mnglish-trained SSM-only encoder generalizes across speaker and language shifts when paired with consistent phonemization and style conditioning, rather than overfitting to LibriTTS. MVC is particularly strong on ES and FR, with slight naturalness gains over StyleTTS2: remaining issues (e.g., stress placement in long German compounds) are analyzed in Appendix [D.]] We attribute this robustness MVC's gated bidirec- tional Mamba fusion, which produces stable prosody transfer under phoneme inventory shifts.
5.2 GENERALIZATION TO OOD TEXTS AND LONG-FORM INPUTS
On an 80-utterance Gutenberg OOD set with complex syntax and punctuation, MVC main- tains MOS (3.87+3.88:
Table 2: MOS on in-distribution (ID) and OOD texts.
> 0.1), while VITS and JETS degrade and StyleTTS2 shows only small gain (Table 2). The near-identical ID/OOD scores indicate that the bidirectional Mamba coders generalize to unseen syntactic structures rather than memorizing training text. For long- form evaluation, we synthesize 2-6 minute pas- sages and report MOS/RTF for short (5 and long (>60 s) segments (Table 3p. MVC maintains
Model
MOS-ID
MOS-OOD
GT
3.81
3.70
StyleTTS2
3.83
3.87
VITS
3.44
3.21
JETS
3.57
3.21
MVC
3.87
3.88
naturalness and latency on extended passages (4.16 vs. 3.91 MOS-long for StyleTTS2: RTF 0.0170
vs. 0.0200), showing that the fully SSM-based conditioning stack remains stable across multi- sentence and multi-minute inputs.
Despite this robustness, MVC exhibits a few mild long-form failure modes. Occasional cross-chunk smoothing appears with short reference embeddings, though perceptually minor (Appendix [E.T). Boundary artifacts sometimes occur when punctuation aligns with chunk edges, usually disappear- ing for L 2 0.5 Small pause-placement deviations also arise for morphologically complex words (e.g.. long German compounds), consistent with cross-lingual observations in Appendix D.5 Ap- pendix [E. further shows that gating dynamics remain stable across multi-minute passages, pre- venting drift accumulation. Appendix [.6] also shows that MVC robust short reference audio (2-4s) s), with minimal drops in speaker similarity and naturalness.
Table 3: Short- vs. long-form performance on LJSpeech.
Model
MOS-short
MOS-long
RTF-short
RTF-long
StyleTTS2
4.15
3.91
0.0185 0.0177
0.0200
MVC
4.22
4.16
0.0170
Table 4: Objective metrics on LJSpeech. Arrows indicate the desired direction of improvement (higher better for PESQ, lower is better for others). Values are averaged over three seeds.
Model
Fo RMSE 1
MCD
WER 4
PESQ 1
RTF
VITS
0.667 + 0.011
4.97
0.09
7.23%
3.64 + 0.08
0.0211
StyleTTS2
0.651 + 0.013
4.93 - 0.06
6.50%
0.07
0.0174
MVC (ours)
0.653 + 0.014
4.91 = 0.07
6.52%
3.85 - 0.06
0.0169
5.3 STREAMING WITH FINITE LOOK-AHEAD
8

---

Published as a conference paper at ICLR 2026
Table 5: Streaming perfor-
For streaming, the bidirectional text encoder is replaced with causal Uni-Mamba. At each chunk boundary, the SSM state is car- ried forward without reset, allowing the model to maintain linguistic and prosodic continuity across segments. Look-ahead I. provides the next I. seconds of mel frames, which condition the SSM update
mance with look-ahead L on 2-6 min Gutenberg passages.
I (8)
WER
MOS
0.25
11.2%
3.74
and prevent premature prosodic decisions when punctuation occurs
0.50
9.4%
3.81
near the boundary. Chunk boundaries remain perceptually smooth
1.00
7.8%
3.89
for L 2 0,5s, with only L = 0.25 s showing occasional discontinu- ities or shortened pauses. These behaviors align with the boundary- sensitivity analysis in Appendix I.I where reduced look-ahead produces less stable gating patterns on rare syntactic structures. Overall, the SSM-only conditioning stack degrades gracefully as decreases, while preserving state continuity across chunks.
2.00
7.3%
3.91
5.4 ABLATION STUDIES
We conduct ablations to isolate the contributions of each encoder module and the Bi-Mamba fusion design, ensuring that MVC's gains are not artifacts of capacity differences or training choices. All variants are retrained from scratch under the same optimization schedule, diffusion configuration, and vocoder, with each removed component replaced by a lightweight, shape-preserving alterna- tive so pipeline interfaces remain identical. To verify that improvements do not arise from protocol mismatch relative to prior Mamba-based TTS systems, we additionally evaluate protocol-matcher Wybrid-Mamba and Bi-Mamba (Concat-only) baselines; full details appear in Appendix [B.6] with their results reproduced in Table 12 Removing the Bi-Mamba text encoder STM with layer normalization and a linear projection to d, (parameters within +5%); removing the Expressive Mamba substitutes 2-layer Conv1D+Red block with matching receptive field and dimensions; removing the Temporal Bi-Mamba applies shallow Conv ID duration predictor using the same alignment features. In all cases, the diffusion decoder and vocoder are fixed, so differences in CMOS, pitch metrics, or RTF directly reflect encoder-side design.
4-layer BiL.-
Component removal. On OOD inputs (Ta- ble 6). removing the Expressive Mamba pro- duces the largest CMÓS-N drop (-0.41), showing that the prosody path is central to maintaining naturalness on challenging text.
Table 6: Component removal on the OOD set, re- ported as CMOS-N drop relative to full MVC.
Removed component
CMOS-N drop
Removing the Bi-Mamba text encoder (-0.38)
Bi-Mamba text encoder
-0.38
or the Temporal Bi-Mamba encoder (-0.36)
Expressive Mamba predictor
-0.41
primarily disrupts rhythm and alignment, yield- ing more monotone or locally unstable prosody.
Temporal Bi-Mamba encoder -0.36
Pitch RMSE increases by 0.12-0.15 18 Hz and duration error by 0.6-0.8 frames for all variants. Taken together, these results show that each SSM-based encoder contributes non-redundant information and that MVC's OOD robustness not due to single dominant module or trivial capacity increase.
Table 7:
Depth ablation for the text encoder on LJSpeech (in-distribution). Includes a BiLSTM baseline; results are averaged over three seeds. Lower RTF is better.
Encoder
MOS ID
RTF
Pitch RMSE (Hz) 1
BiLSTM (no Mamba)
3.61 = 0.13
0.0268
29.2
2 Mamba layers
3.65 = 0.12
0.0215
27.5
3 Mamba layers
3.72 = 0.11
0.0203
25.4
4 Mamba layers
3.78 + 0.10
0.0198
24.1
5 Mamba layers
3.85 + 0.10
0.0195
23.7
6 Mamba layers
3.87 - 0.07
0.0189
23.2
7 Mamba layers
3.90 = 0.11
0.0192
23.3
8 Mamba layers
3.88 = 0.12
0.0196
23.5
Depth scaling. Table 7 varies the text-encoder depth (2-8 layers) and includes BiLSTM with com- parable hidden size as a non-SSM baseline. The BiLSTM yields the lowest MOS and highest RTF, confirming that selective scans more efficient than recurrent stacks of similar capacity. While

---

Published as a conference paper at ICLR 2026
the 7-layer model attains higher MOS, the 6-layer encoder provides the best quality-efficieet trade-off, achieving lower RTF with statistically comparable MOS and serving as the default. Shal- lower stacks (2-4 layers) underfit long-range linguistic context and degrade MOS and pitch tracking, whereas deeper stacks (7-8 layers) offer slight MOS gains with increased latency. This pattern in- dicates that the chosen depth is near an empirical optimum rather than pver-parameterized, and that MVC's improvements do not depend on excessively deep encoders.
Table 8: Fusion and conditioning ablation on LJSpeech long-form utterances. Removing gated fusion or AdaLN reduces MOS and increases pitch RMSE. Values are averaged over three seeds.
Variant
MOS long f
Pitch RMSE (Hz) 4
RTF. +
MVC (gated + AdaLN)
4.16 0.07
1.92 + 0.05
0.0177
Gated only (no AdaLN)
4.02 0.08
2.04 = 0.06
0.0186
AdaLN only (no gating)
3.95 = 0.04
2.22 = 0.05
0.0198
Concat (no gating, no AdaLNN
3.64 0.09
2.89 + 0.07
0.0216
Fusion and conditioning. To isolate the effect of Bi-Mamba fusion, we evaluate four variants on long-form L.Speech: (i) the full MVC text encoder with gated bidirectional fusion and AdaLN,, (ii) gated fusion without AdaLN, (iii) AdaLN without gating (concat before modulation), and (iv) concat-only fusion with neither gating nor AdaLN, Ablating either component reduces MOS and increases pitch RMSE, with the concat-only variant degrading the most (Table (8) RTF rises slightly across ablations; we attribute this to reduced conditioning coherence, which can yield marginally higher per-step overhead even with fixed diffusion schedule. The full MVC configuration (gated fusion plus AdaLN) achieves the best naturalness aaturalness-pitc balance with the lowest RTF, indicating that both components are essential for long-form stability rather than superficial additions. The large gap between the full model and the concat-only variant further shows that simply replacing attention with a bidirectional SSM is insufficient; gating and style modulation are required to recover-and modestly surpass transorrrreel quality under the matched diffusion protocol.
6 DISCUSSION AND CONCLUSION
MVC examines whether the entire conditioning path of diffusion TTS system can be made fully SSM-only at inference, removing attention and recurrence across text, rhythm, and prosody while preserving the same front-end, diffusion decoder, and vocoder. By using lightweight atten- tion aligner only during training, MVC deploys a linear-time conditioning pipeline with bounded activations that improves encoder throughput and peak memory without altering the decoder. Under strictly matched protocols-addresse ters-MVC achieves modest but statistically reliable improvements over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, MCD, and PESQ, with parity in WER and RTF, Abla- tions show that the advantages arise from the combination of gated Bi-Mamba fusion and AdaLN modulation rather than model size. Streaming experiments further demonstrate that 1-2s look- ahead preserves non-streaming quality, satisfying requests to characterize finite-latency behavior. Finally, evaluations on VCTK, CSS10 (ES/DE/FR), and Gutenberg text indicate that an English- trained, SSM-only encoder generalizes well to speaker, language, and syntactic shifts, clarifying cross-lingual and long-form robustness.
concerns about fairness and hidden hyperparame-
Limitations. MVC focuses on conditioning efficiency rather than fine-grained emotion control; AdaLN provides global, not expressive, style cues. The model is trained only on English datasets, and the diffusion decoder remains the dominant latency bottleneck. Because MVC enables high- fidelity voice cloning, we assess compatibility with watermarking and forensic detectors and ob- serve no meaningful degradation. Responsible deployment requires explicit speaker consent; our released code includes watermarking and disclosure utilities to support ethical use. MVC demon- strates that a fully SSM-only conditioning stack can match or slightly surpass attention-based and hybrid Mamba baselines while offering practical benefits in memory use, throughput, long-fow form stability, and streaming. Rather than positioning itself as a large-scall competitor to systems such as NaturalSpeech 3 or Cosy Voice 3, MVC provides a controlled encoder-side side redesign that can serve as drop-in conditioning module for future multilingual or industrial pipelines.
10

---

Published as a conference paper at ICLR 2026
USE OF LARGE LANGUAGE MODELS
We used a large language model solely for language polishing (grammar and clarity) on drafts writ- ten by the authors. The LLM did not generate technical content, equations, code, analyses, figures, or results, and it was not used for ideation, literature search, data labeling, or experiments. All scientific claims and evaluations were produced and validated by the authors.
REFERENCES
Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to phones transcription for multiple lan- guages python. Journal of Open Source Software, 6:3958, 12 2021. doi: 10..1 10.21000/oos..0355..
Boson AI. Higgs Audio V2: Redefining Expressiveness in Audio Generation. https:
gi github. at wttp://wwwwwc. www.:wwwboo.co http:.bbsoo...a/log///oo...oo
whgggnggaaiiiiiaaaaaaaaaaggasa hwwphhhigggauuuiaauoeooo 2025. GitHub repository. Release blog available htttps::/www.bosttt
audio-v2
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, tukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In International Confer- ence on Learning Representations, 2021. URL Ua Ua6zukOWRH
net wwtp://ee/fommm..e..em forum?id
Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang, and Jieping Ye. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up up and post-training, 2025. URL https:
7/ //arxiv. orq abs/ 2505. 17589
Isaac Elias, Heiga Zen, Jonathan Shen, et al. Parallel tacotron 2: A non-autoregressive neural its model with differentiable duration modeling. In Interspeech, 2021.
Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces, 2024.
URL htt.:p::/aaxxv.orrg..oo /23312/ 2312. 00.020752
Rongjie Huang, Zhou Zhao, Huadai Liu, et al. Prodiff: Progressive fast diffusion model for high- quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2595-2605, 2022.
Keith Ito and Linda Johnson.
The lj speech dataset.
https://keititto.com
LJ Speecheeehhhc Batacchh-atsttt/// 2017.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff- Its: A denoising diffusion model for tepeeeh- In Proc. Interspeech, 2021. https:
// /www. i isca wwchive......... org/
html
Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. Png bert: Augmented bert phonemes and graphemes for neural tts. arXiv preprint arXiv:2103. 15060, 2021.
Xilin Jiang, Yinghao Aaron Li, Adrian Nicolas Florea, Cong Han, and Nima Mesgarani. Speech slytherin: Examining the performance and efficiency of mamba for speech separation, recogni- tion, and synthesis, 2024. URL https: /ttbbs::///rxv.... wrw.i.iiiiv..//.i.a/aaaaaaaaaaia org/ abs / 2207244477099 09732
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song. Siliang Tang, Zhizheng Wu, Tao Qin, Xiang Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models, 2024.
031 00
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech, 2021. URL / wttpa/.wwww...///////.00///
061 0.3
11

---

Published as a conference paper at ICLR 2026
Zhifeng Kong, Wei Ping, Jiaji Huang, et al. Diffwave: A versatile diffusion model for audio synthe- sis. arXiv preprint arXiv:2009.0976; 2020.
Sahil Kumar, Jialu Li, and Youshan Zhang. Vision transformer segmentation for visual bird sound denoising, 2024, URL ptattts:://ariiii oww. wtwp////////eooooeeooooo/eaeaee/ 2406 221.6/9606666077777
Yinghao Aaron Li. Cong Han, Xilin Jiang, and Nima Mesgarani. Phoneme-level bert for enhanced prosody of rext-to-speech with grapheme predictions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023a.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2: Towards human-level text-to-speech through style diffiusion and adversarial training with large speech language models, 2023b. URL https::/t////c. http://://aaxxv..og//a//////////
07691
Dan Lim, Sunghee Jung, and Resung Kim. Jets: Jointly training fastspeech2 and hifi-gan for end to end text to speech, 2022. URL httt::::/aarstt..org/aa/////////
16852
Songxiang Liu, Dan Su, and Dong Yu. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans. arXiv preprint 1197 2022.
Koichi Miyazaki, Yoshiki Masuyama, and Masato Murata. Exploring the capability of mamba in speech applications, 06 2024.
Yishuang Ning, Sheng He, Zhiyong Wu, Chunxiao Xing, and Liang-Jic Zhang. A review of deep learning based speech synthesis. Applied Sciences, 919):4050, 2019.
Kyubyong Park and Thomas Mulc. Css10: A collection of single speaker speech datasets for 10 languages. In Proc. Interspeech, 2019.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, et al. Gradtts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp. 8599-8608. PMLR, 2021.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech, 2022. URL
04558
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang. Yuxuan Wang, RJ Skerry-Ryan Rif A. Saurous, Yannis Agiomyr- giannakis, and Yonghui Wu. Natural its synthesis by conditioning wavenet on mel spectrogram predictions, 2018. URL httlt::///eeet //axiiv org/ abs 172111111100 05884]
Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. survey on neural speech synthesis. arXiv preprint arXiv:2106. 15561, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need, 2023. httppttts:://axiv
org/abs/1706. 1w0kb 03762
Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit, 2017. URL /ttps::/doi.orrs wtt.:/o....orroo. 7488/
ds/1994
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0447 04768, 2020. URL URLpttss::/arzxivoo 0200/2006.07763
Yuxuan Wang, RJ Skerry- Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis. Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis, 2017. URL.
https://arra
org. abs 1703.11.00 10135
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss. Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech, 2019. URL https : /arxiiv.
org/ 9904 02882
12

---

Published as a conference paper at ICLR 2026
Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, al. A survey audio diffusion models: Text to speech synthesis and enhancement in generative ai. arXiv preprint arXiv:2303.13336 2023.
Xiangyu Zhang, Qiquan Zhang, Hexin Liu, Tianyi Xiao, Xinyuan Qian, Beena Ahmed, Eliathamby Ambikairajah, Haizhou Li, and Julien Epps. Mamba in speech: Towards an alternative self- attention, 05 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 RUNTIME AND MEMORY ANALYSIS
This section complements the main-text runtime results by isolating encoder-side costs un- der a shared implementation. We report encoder parameter counts, relative throughput (to- kens/s), and peak encoder memory for StyleTTS2, representative Mamba-attention hybrid, and MVC, all implemented in Py PyTorch on a single A100-40GB GPU with identical batch size and mel-diiffusion-ooooderr configuration. End-to-end real-time factors (RTF) remain similar across models because the diffusion decoder dominates compute, but the encoder footprint varies sub- stantially and directly impacts deployability for longer utterances and larger batches.
Table 9: Encoder-only throughput and peak memory, normalized to StyleTTS2. All models share the same mel-diefusion-vocodee stack. MVC's SSM-only conditioning achieves the best encoder efficiency and memory usage, enabling larger-batch and longer-context synthesis while leaving the diffusion latency profile unchanged.
Model
Encoder Params
Encoder speedup (x) f
Peak Memory +
StyleTTS2
42M
1.00
100%
Mamba-hybrid
32M
1.15
86%
MVC (ours)
21M
1.60
72 %
A.2 CONTRAST WITH PRIOR TTS SYSTEMS
Table 10 summarizes inference-time architectural differences between MVC, StyleTTS2, and rep- resentative Hamba-based TTS systems, focusing on (i) whether attention is used inference, (ii) how rhythm/duration and prosody/style are modeled, (iii) the fusion or modulation mechanism, and (iv) whether the conditioning stack SSM-only.
Table 10:
Inference-time comparison with attention-centric and Mamba-based TTS systems. "Hy- brid" denotes that attention or recurrence is retained in least one conditioning module (duration, rhythm, or prosody); only MVC deploys an SSM-only conditioning stack across all of them.
System
Inference attention?
Rhythm/
Prosody/ style mech.
Fusion/
SSM-only?
duration
modulation
StyleTTS2
Variance/ duration pred.
Ref./style encoder (attn)
Attention
No
(Transformer)
concat
Miyazaki'24
Hybrid
Mixed (keeps attn)
Mixed (keeps attn)
Concat
No
(SSM+att)
Jiang'25 (Speech
Hybrid
Attn/var. pred.
Ref./style enc. (attn)
Concat
No
Slytherin)
(SSM+attn)
Zhang'24 24
Hybrid
Mixed (keeps attn)
Mixed (keeps attn)
Concat
No
(Mamba
(SSM+attn)
Speech)
MVC (ours)
No (SSM-only)
Temporal Bi-Mamba
Mamba + AdaLN
Gated bi-dir. fusion AdaLN
Yes
Key observations are: (i) MVC is the only system in this comparison that is SSM-only at inference across text, rhythm, and prosody; (ii) MVC replaces concat-only SSM fusions with gated bidi- rectional fusion and AdaLN, which Section 5.4 shows is important for long-form stability and Fo tracking; and (ii) all contrasts are drawn under a shared mel diffusionnvocooee backbone, isolating the impact of conditioning design rather than decoder differences.
14

---

Published as a conference paper at ICLR 2026
B ADDITIONAL METHODOLOGY DETAILS
B.1 NOTATION SUMMARY
Table II consolidates all symbols used in Sec. providing a unified reference for variables, encoder
states, and decoder-side quantities.
Table 11: Notation summary covering all variables used in the Methodology section.
Symbol
Description
Input & Dimensions
Tr
Number of text tokens
T
Number of mel frames
F
Number of mel bins (80) Text embedding dimension SSM hidden dimension Style embedding dimension Input waveform
da
de
Swav
Core Inputs
Token embeddings, RT xd Log-mel spectrogram, RExT= Global style embedding, Rd
M
Encoder States
hy. hs
Forward backward Uni-Mamba scans
hr
Gated Bi-Mamba text features
hr,,
Text features after AdaLN conditioning
h
Style-conditioned mel features (Expressive path)
h
Expressive Mamba features
hs
Style-modulated temporal input
hg
Temporal Bi- Mamba rhythm/duration features
hA
Aligned text features (training only)
hr
Pitch-aware fused features
hp
Final decoder conditioning sequence
Alignment Pitch a
Token-frame attention weights (training only)
Fo
Predicted fundamental frequency trajectory
n
Residual noise vector for diffusion conditioning
Model Parameters
W,, g, W.
Gated fusion matrices (text encoder) W Temporal fusion matrix
WF. bF
Linear layer for Fo prediction
Diffusion Loss
{ar)
Diffusion noise schedule
Lmel
Mel reconstruction loss
Ladv
Adversarial loss
Calign
Alignment regularization loss
Liotal
Total training loss
B.2 MEL-SPECTROGRAM FRONT-END
We follow a standard STFT-mel pipeline compatible with StyleTTS2 and VITS. Given waveform Swav at 24 kHz, compute STFT with Hann window, FFT size 1024, and hop size 256, apply an 80-bin mel filterbank, and take log magnitude with e-10-5 This matches common TTS settings and avoids front-end confounds when comparing MVC to transformer and Mamba baselines.
15

---

Published as a conference paper at ICLR 2026
B.3 BI-MAMBA AND SSM IMPLEMENTATION
Each Mamba block is implemented as a selective state-space model with depthwise convolutional pre-activation, following Gu & Dao 2024). For input sequence R xd,, the block applies: (i) ConviD residual connection, (ii) snput-dependent state updates, and (iii) projection back to d₁- Uni-Mamba scans either forward or backward; the Bi-Mamba text encoder applies both directions and fuses them via Eq. 3 in the main text. We use the same Mamba configuration (state dimension, kernel size, activation) across text, expressive, and temporal encoders keep the design simple and comparable.
B.4 SPEECH DYNAMICS AND DECODER CONDITIONING
Starting from A and hp, Conv ID+SSM temporal predictor yields hy... which is fused with hp by gated block to produce Fo and residual noise n. The final conditioning sequence is
he [Fo: n] E R x(1dda),
which is passed to the diffusion decoder. All operations in this stage use SSMs and pointwise gates only, so the conditioning path remains linear-time and attention-free at inference.
B.5 DECODER STAGE AND LOSSES
We reuse the StyleTTS2 diffusion decoder and HiFi-GANNiSTFTNet vocoder without modification. Given hp the decoder outputs a mel-spectrogram M, which the vocoder maps to waveform S. Training uses: (i) an L1 mel reconstruction loss Lmel = || MIl1. (ii) least-squares GAN losses with multi-period and multi-resolution discriminators (MPD+MRSD), and (iii) an alignment loss Lalign that regularizes the training-time aligner with monotonicity prior. The total loss in Eq. TO of the main text matches StyleTTTS2 up to the alignment term, ensuring that decoder-side training remains protocol-matched across all models.
B.6 PROTOCOL-MATCHER MAMBA-TV BASELINES
To address baseline fairness, we re-implement Mamba-based TTS baselines under the same phone- mization, mel front-end, diffusion schedule, vocoder, optimizer, and training schedule as MVC. Hybrid-Mamba retains inference-time attention in duration/style modules, whereas Bi-Mamba (Concat-only) is SSM-only but uses simple concatenation instead MVC's gated fusion with AdaLN. All models are matched for encoder parameter count within +55. These results are cited in the main tables to show that MVC's gains persist under strict protocol parity with prior Mamba- based TTS designs.
Table 12: Protocol-matched Mamba-TTS baselines under mel/diffisioo//ocodee pipeline LJSpeech. Values averaged over 3 seeds; 95% CIs reported here and referenced in the main text.
Model
FO RMSE
MCD +
WER +
PESQ 1
RTF +
Syybrid-Mamin (Concat)
0.659 + 0.013
4.95 + 0.07
6.68%
3.79 + 0.06
0.0189
Bi-Mamba (Concat-only)
0.656 + 0.014
4.93 - 0.06
6.58%
3.82
0.06
0.0181
MVC (gated + AdaLN)
0.653 + 0.014
4.91 = 0.07
6.52%
3.85 = 0.06
0.0177
B.7 ALIGNMENT TEACHER ROBUSTNESS
The training- time aligner is 2-layer transformer with 4 heads and hidden size 256, trained jointly with the temporal encoder and regularized by a monotonicity loss. To test robustness, we inject Gaussian noise into attention logits before softmax and renormalize. On LJSpeech, perturbations of up to 100% in attention weights increase WER by < 0.4 percentage points and reduce MOS by 0.05, with overlapping 95% confidence intervals. This supports the claim that MVC does not depend on perfectly specified aligner and that the SSM-only inference path is robust to moderate alignment noise.
16

---

Published as a conference paper at ICLR 2026
B.8 TRAINING ALGORITHM
Algorithm D summarizes the training loop for MVC, including style extraction, encoder passes, alignment, pitch modeling, speech dynamics, diffusion decoding. vocoding, and loss updates.
Algorithm MVC Training Algorithm
Input: Dataset D {(x,
epochs E, batch size B, loss weights Amet, Audv Aaligns
diffusion schedule {ar)-
Output: Trained encoder/decoder parameters discriminator parameters Φ.
Initialize 0, Φ and optimizers (AdamW, EMA, cosine decay). for epoch
E do
for batch b = {(x', M', wav do
Forward pass:
Compute style embedding from mel using Eq.
Encode text with Bi-Mamba AdaLN(e) (Sec.
3.2.1
Encode mel with the Expressive Mamba (Sec.
3.2.2).
Encode rhythm with the Temporal Bi-Mamba (Sec. 3.2.3)
Use the training-ting time aligner to obtain frame-synche simme-synchronous features 1 (Sec. 3.3
Build pitch-aware features and predict Fo via Eq.
Construct decoder conditioning ho via Eq. Generate mel via the diffusion decoder and waveform via the vocoder (Sec. 3.5
Loss computation:
Compute Lioaal using Eq. 10
Backward pass:
Update & using VoLioal update ó using V Cadav-
end for
Evaluate the validation set; keep the best checkpoint by mel-71 and Fo RMSE. end for
Return: Trained parameters 0,
We use the same optimizer, schedule, and early-stopping criteria for MVC and all protocol- baselines, providing a complete recipe for reproducing our results.
C ADDITIONAL EXPERIMENTAL DETAILS
This section provides details on the long-form evaluation set, optimization setup, and diffusion-step ablations that were omitted from the main text for space. These additions clarify the experimental protocol for long-form robustness and inference-ting efficiency, directly addressing concerns about reproducibility and the validity of our long-form and runtime claims.
C.1 LONG-FORM SET CONSTRUCTION
For the Gutenberg set, we sample 2-6 minute passages from public-domain audiobooks and filter them to avoid lexical overlap with LJSpeech and LibriTTS. We apply exact-token filtering on nor- malized text and MinHash-based de-duplication, retaining only passages with Jaccard similarity < 0.2 to any training utterance. We then synthesize 40 passages for each model and report WER function of duration and pitch drift per minute, alongside MOS for long-form naturalness. This construction ensures that the long-form and streaming evaluations probe genuine dii-riiuittiiittion generalization rather than memorization of training text, addressing requests for clearer OOD long-form protocol. Qualitative failure wrodes-such as rare punctuation patterns or abrupt topic shifts-an are are analyzed in Appendix [II] where we show that they correlate more strongly with diffu- sion decoding eтrоrs than with gating collapse.
C.2 OPTIMIZATION AND TRAINING SCHEDULE
We use AdamW with learning rate 1 1x10- weight decay 1x10- cosine decay with 10k warmup steps, gradient clipping at 1.0, EMA (0.999), and automatic mixed precision, and we apply this iden-
17

---

Published as a conference paper at ICLR 2026
tical schedule to MVC, StyleTTS2, VITS, JETS, Hybrid-Mamba, and Bi-Mamba (Concat-only). Batch sizes are 16 (LISpeech) and 32 (LibriTTS) on 4 A100 40GB GPUs, with LJSpeech models trained for 200 epochs and LibriTTS models for 300k steps, ensuring protocol-matched optimiza- tion across all baselines. Checkpoints are selected using the same criteria (mel-7, and Fo RMSE), and all baseline models are re-trained under this unified data pipeline rather than using their original scripts, removing discrepancies due to implementation-levd differences. Inference uses fixed 5- step diffusion schedule and identical vocoders (iSTFTNet for LJSpeech, HiFi-GAN for LibriTTS) for every model, isolating the effect of the SSM-only conditioning stack from vocoder or decoder confounds. This fully unified optimization and inference protocol resolves prior concerns about unfair baseline comparisons and ensures reproducibility by allowing any encoder to be swapped in without changing the surrounding pipeline.
C.3 DIFFUSION STEP ABLATION STUDY
We conduct an ablation study determine the optimal number of diffusion steps during inference in MVC. Following prior work Popoy et al. (2021). we evaluate the trade-off between perceptual quality and runtime efficiency the LJSpeech validation set, varying the number of steps from 3 to 9. We report Mean Opinion Score for Naturalness (MOS-N) and Real-Time Factor (RTF), each averaged over 20 utterances with 5 random seeds. Error bars reflect 95% bootstrap confidence intervals. All samples use the same ground-truti durations and pitch to isolate the effect of denoising steps.
Table 13: Diffusion step ablation on the LJSpeech validation set. Increasing steps improves quality but degrades synthesis speed. Five steps yield the best quality-efficennc trade-off and are used in the main experiments.
Steps
MOS-N f
RTF +
3.62 0.12
0.0151
4
3.74 - 0.09
0.0164
5 (used)
3.87 + 0.07
0.0177
3.88 + 0.08
0.0190
7
3.89 + 0.08
0.0205
3.89 + 0.08
0.0221
As shown in T3] naturalness improves steadily with more steps but plateaus beyond five steps. Steps 6-9 offer only marginal MOS-N gains (< 0.03) while increasing RTF by more than 20%. Steps below five suffer from unstable prosody and noisy pitch contours, particularly for long ex- pressive utterances. These findings mirror prior diffusion-TTSS observations Popov et al.: (202 few steps lead to over-smoothed benefits at substantial runtime cost. We therefore select five steps as the default for all experiments, clarifying that our runtime improvements not obtained by using unusually few diffusion steps but by improving the efficiency of the conditioning stack itself.
T): too under-articulated speech, while too many steps yield negligible
C.4 BASELINE CONFIGURATION SUMMARY
Table 14 summarizes the encoder architectures and conditioning paths for baselines under the unified preprocessing, mel front-end, and diffusion/vocooer pipeline described in Sec. 4.1 and Ap- pendix C.2 All models use the same audio preprocessing (24 80-bin log-mels, FFT=1024, hop=256), corpus-matched vocoders (iSTFTNet for LJSpeech, HiFi-GAN for LibriTTS).
D ADDITIONAL RESULTS
This section provides qualitative and quantitative results that complement the main evaluation: waveform and spectrogram comparisons, training convergence, runtime breakdown, and cross- speaker cross-lingual MOS. Together, these analyses substantiate claims about MVC's training efficiency, perceptual quality, and generalization beyond the English training setting, and clarify where the gains modest but reliable.
18

---

Published as a conference paper at ICLR 2026
Table 14: Protocol-matched baseline configurations under the shared mcl/diffusion/vocodeer pipeline. The table highlights only encoder and conditioning- path differences; training schedule and vocoders are identical across all models (Sec. 4.1, Appendix
Model
Encoder Conditioning Path
StyleTTS2
Transformer acoustic text encoder with reference style encoder; attention-based duration and prosody predic- tors; diffusion decoder conditioned via attention and style embeddings.
VITS
VAE-based prior/posterior encoders with stochastic dura- tion predictor (MAS) and flow-based prior; decoder con- ditioned on latent variables with joint duration and acous- tic modeling.
JETS
FastSpeech2-styl acoustic model with duration predictor and reference encoder; non-autoregressive conditioning on predicted durations and style embeddings under joint training scheme.
Hybrid-Mamba (Con- cat)
Mamba text encoder paired with atlention-bused duration and style modules; diffusion decoder receives concate- nated SSM features and attention-derived style signals. SSM-only encoder path with bidirectional Uni-Mamba scans for text, temporal, and expressive streams; condi- tioning formed by simple concatenation of SSM features without gating AdaLN.
Bi-Mamba
(Concat-
only)
MVC (SSM-only)
Gated Bi-Mamba text encoder, Temporal Bi-Mamba for thythm/duration, and Expressive Mamba with AdaLN conditioning: fully SSM-only conditioning stack at infer- ence, with no attention or recurrence.
Ground Truth
MVC
StyleTTS2
Allgers heard soundedd like the popping of firerrakier.
MOSID (CI)
0.100
Walter Watts was inventer
Mist Hi
MOSID
4.45 co 0.10)
4.45 0.10)
4.25 0.100
According the vice president K0eers, Willsem Wealdmen
MOSID ICI)
4.50 0.10)
4.48 Ca 0.50)
4.11 0.10)
Figure 2: Waveform comparison of synthesized speech from different TTS models on LJSpeech, evaluated using MOS (95% CI). MVC closely aligns with the ground truth, capturing finer prosodic variations and outperforming StyleTTS2 and JETS in expressiveness and naturalness.
D.1 WAVEFORM AND SPECTROGRAM ANALYSIS
Figure compares synthesized waveforms from MVC, StyleTTS2, and JETS against ground truth on I.ISpeech. MVC-generated waveforms exhibit closer alignment to ground truth temporal structure, prosodic variation, and amplitude consistency, and obtain the highest MOS (with 95% confidence intervals) across the evaluated utterances. StyleTTS2 produces high-quality speech with MOS close to MVC but shows minor rhythm and expressiveness deviations. JETS displays more pronounced distortions and energy inconsistencies, leading to lower MOS and reduced naturalness. These qualitative trends visually corroborate the MOS and CMOS gains reported in the main tables and provide intuitive, signal-level evidence that the SSM-only conditioning stack improves long- form prosody and local timing.
19

---

Published as a conference paper at ICLR 2026
D.2
TRAINING CONVERGENCE ANALYSIS
Validation MOS Epochs
FD RMSE vs Epochs
NOS 3.8 StyeTTS2 MVC
StyleTTS2
3.9
NVC
3.8
0.72
0.70
AMSE
3.0
0.68
3.4
3.3
0.60
10
12
14
16
30
12
14
16
Epoch
Epoch
Figure 3: Validation MOS and Fo RMSE curves over training epochs for MVC and StyleTTS2 on LISpeech. MVC reaches strong validation quality and stable pitch error in fewer epochs under matched optimization schedule.
To substantiate the claim of improved training efficiency, we track validation MOS and Fo RMSE over training epochs for MVC and StyleTTS2 on LJSpeech. Figure 3 shows that MVC reaches validation MOS of approximately 3.8 within about 10 epochs, whereas StyleTTS2 requires roughly 16 epochs to reach similar level. Likewise, Fu RMSE stabilizes about 20% faster for MVC. This indicates that MVC not only more efficient at inference, but also converges faster during training under matched optimizer, learning rate schedule, and data pipeline, suggesting that the modular SSM conditioning stack easier to optimize. These convergence curves address concerns that encoder-side gains might be offset by slower or less stable training dynamics.
D.3 SPECTROGRAM ANALYSIS
Figure presents spectrograms synthesized speech from MVC, StyleTTS2. and JETS versus ground truth for three representative utterances. Highlighted rectangular regions emphasize har- monic continuity and spectral energy distribution; square regions focus on formant transitions and high-frequency harmonics.
Ground-truth recordings show well-defined harmonic bands and clean formant trajectories. MVC closely preserves these structures, maintaining smooth phonetic articulation and stable energy dis- tribution. StyleTTS2 retains strong overall fidelity but shows mild harmonic distortions and slightly blurred formant transitions. JETS exhibits spectral discontinuities, attenuation, and smearing, which manifest as degraded articulation and reduced naturalness. These qualitative observations align with the MOS, PESQ, and MCD differences reported in Tables 4 and 2 indicating that MVC's improve- ments extend beyond narrow metric choice and are reflected in fong-form harmonic continuity.
D.4 MODULE-WISE RUNTIME BREAKDOWN
To better understand MVC's inference efficiency, we break down the average runtime contribu- tions by module. Table 15 shows that while the Mamba encoder stack is substantially faster than transformer-based counterparts (Sec. [.4]. the diffusion decoder remains the dominant la- tency contributor. This decomposition underpins the main-text claim that MVC's practical benefits are encoder-side peak memory and conditioning throughput-and and that overall RTF is ultimately bounded by the diffusion decoder until it is replaced by a lighter generative backbone. These mea- surements confirm that our reported RTF improvements are attributable to the SSM-only condition- ing path rather than hidden changes in the diffusion vocoder components.
20

---

Published as a conference paper at ICLR 2026
2043
2000
2048
1024
10C1
1024
512
$12
512
231
300
40
Ground uuth (GT)
MVC
Styli
IETS
4110
8397
4090
4C58
2049
2048
2048
20
Ko Netznding prottee attine
Ground
StyleTT52
312
236
235
200
120
64
64
083411244 4644
10
Stat
Figure 4: Spectrogram comparison of synthesized speech from ground truth, MVC, StyleTTS2, and JETS on IJSpeech for three representative utterances. Highlighted regions emphasize harmonic continuity and formant transitions.
Table 15: Average inference time per utterance (milliseconds) and proportion of total runtime, mea- sured on 500 LISpeech utterances on single A100 with FP16 inference.
Module
Avg. time (ms)
Proportion (%)
Bi-Mamba encoder stack
42.5
31.4
Diffusion decoder
73.4
54.2
Vocoder (HiFi-GAN/ (iSTFTNet)
19.5
14.4
Total
135.4
100.0
D.5 CROSS-SPEAKER AND CROSS-LINGUAL, GENERALIZATION
Datasets and protocol. We assess zero-shot speaker generalization VCTK (20 unseen speak- ers; 5 sentences per speaker) and cross-lingual robustness on CSS10 ES/DE/FR (30 prompts per language). Ratings are collected Amazon Mechanical Turk with 5-10 native listeners per clip, using MOS for naturalness (MOS-N) and similarity (MOS-S), mirroring the evaluation setup of StyleTTS2 and applying standard rater screening and confidence interval estimation.
D.5.1
VCTK: ZERO-SHOT SPEAKER GENERALIZZTION
Table 16: VCTK zero-shot speaker generalization (MOS with 95% confidence intervals). Higher is better.
Model
MOS-N
MOS-S
VITS
3.66 + 0.12
3.53 0.13
StyleTTS2
4.12
+.10
MVC (ours)
4.18 + 0.10
0.11
21

---

Published as a conference paper at ICLR 2026
MVC matches slightly exceeds StyleTTS2 on both MOS-N and MOS-S (paired two-sided tests vs. StyleTTS2 with -onnferorrro correction: p < 0.05 for MOS-S; trend-level for MOS-N, S 0.1). These results indicate that the SSM-only conditioning does not compromise, and may slightly improve, zero-shot speaker transfer relative to transformer-based baselines. We observe that especially expressive speakers benefit from the Expressive Mamba path, which better preserves pitch variance and speaking style.
D.5.2 CSS10: CROSS- NATURALNESS (ES/DE/FR)
Table 17: CSS10 cross-lingual naturalness (MOS-N with 95% confidence intervals). Higher is better.
Model
ES
DE f
FR 1
VITS
3.48 + 0.12
3.39
0.13
3.46 0.12
StyleTTS2
3.84 + 0.12
3.76
0.11
3.85 + 0.11
MVC (ours)
3.91 + 0.11
3.82 = 0.10
3,93 0.10
Despite being trained only on English corpora, MVC maintains quality on ES/DE/FR and modestly exceeds StyleTTS2 in ES and FR (Holm-Bonferroni < 0.05), while matching it in DE. This suggests that the modular Mamba encoder stack, combined with language-tagged phonemization, generalizes beyond English phoneme inventories without explicit multilingual training. Remaining errors often involve stress misplacement and vowel length in long German compound nouns or infrequent liaison patterns in French; these failure modes are consistent with the encoder's lack of explicit prosodic labels rather than instability of the SSM itself.
D.6 REFERENCE LENGTH SENSITIVITY
We evaluate MVC's robustness to different durations of reference audio used to compute the global style embedding, Following the StyleTTS2 protocol, the main experiments use fixed 6-second reference. Table 18 reports MOS-S and MOS-N for reference lengths of 2, 4, 6, and 8 seconds on VCTK zero-shot set.
Table 18: Effect of reference length on zero-shot VCTK speaker similarity (MOS-S) and naturalness (MOS-N).
Reference length
MOS-S 1
MOS-N 1
seconds
3.87 : 0.10
4.03 0.11
seconds
3.94 + 0.09
4.11 = 0.10
seconds (main)
4.02 = 0.09
4.18 0.10
seconds
4.03 + 0.09
4.19 0.10
Reducing the reference to 4 seconds results in only small MOS-S and MOS-N drop, and 2-second references incur slightly larger but still moderate degradation. These results confirm that MVC's mel-based style embedding remains stable for short reference durations, with similarity and natu- ralness improving monotonically with available context and saturating around 6-8 seconds, making the method practical in scenarios where long reference clips are not available.
E ADDITIONAL ABLATION AND SENSITIVITY STUDIES
This appendix complements the main ablations in Sec, 5.4 by analyzing gating behavior, the robust- ness of the alignment teacher, and the sensitivity of MVC to key SSM hyperparameters. The goal is to verify that MVC's improvements are stable under implementation-lee depend fragile gating dynamics aggressively tuned Mamba configurations.
perturbations and do not
22

---

Published as a conference paper at ICLR 2026
E.1 GATING STABILITY AND FAILURE MODES
For the bidirectional Mamba text encoder, we examine the learned gate values in the fusion module that combines forward and backward states. We track the mean and variance the gating weights across timesteps for LJSpeech long-form utterances and Gutenberg passages. Empirically, the gate histograms remain well-balanced with no collapse to a single direction: the average gate allocation is approximately 0.53 to the forward branch and 0.47 the backward branch, with moderate per- utterance variance. On OOD Gutenberg passages, the distribution shifts slightly toward the forward branch (approximately 0.56 vs. 0.44), but we do not observe degenerate behavior where one direc- tion is effectively ignored. These diagnostics indicate that the gating mechanism remains stable on long sequences and under domain shift, rather than collapsing to a purely uni uni-directional encoder.
Qualitative failure cases primarily involve rare punctuation patterns or abrupt topic shifts, where both MVC and StyleTTS2 may misplace minor pauses. In these cases, the gating distribution remains non-degenerate, and observed errors appear to arise from diffusion decoding rather than encoder collapse. This analysis supports the view that Bi-Mamba gating in MVC is stable design choice for long-form inputs, rather than source of fragility.
E.2 ALIGNMENT TEACHER ARCHITECTURE AND ROBUSTNESS
The lightweight attention aitention-bbsed aligner used during training is two-layer transformer with 4 heads, hidden dimension 256, and temporal Mamha encoder but discarded at inference. To probe robustness, we inject noise into the aligner attention maps at training time by randomly perturbing attention weights by 100% and renormalizing before they are used to construct frame-synchronous features. Under this per- turbation, WER on LJSpeech increases by less than 0.4 percentage points and MOS LibriTTS decreases by less than 0.05, with overlapping 95% confidence intervals.
peattooiccty-cosstaiintte attention loss. It is trained jointly with the
These results suggest that the temporal Bi-Mamba encoder does not rely on perfectly specified attention maps and can tolerate moderate alignment noise without catastrophic degradation. Con- sequently, the use of an attention- teacher is compatible with the claim that MVC deploys an SSM-only path at inference, and the overall system is robust to reasonable vaining-time misalign- ment.
E.3 SSM HYPERPARAMETES SENSITIVITY
We examine the sensitivity of MVC to key Mamba SSM hyperparameters: (i) state dimension d-a-=
(ii) convolution kernel size keoov, and (iii) gating temperature Tgate A sweep of these hyperparam- eters conducted using the same training protocol as outlined in Section 4 with evaluations performed on the LJSpeech in-distribution test set and the Gutenberg out-of-distribution (OOD) set. The results, presented in Tables [1 show that the largest MOS change between neighbor- ing configurations is less than 0.05, and RTF changes by less than 10%. Given these results, we fix deam = 96, Kconv = and Tgate 1.0 in all main experiments, attributing the observed improvements in MVC's performance primarily to its architectural design, rather than narrow hyperparameter tun- ing.
E.3.1 STATE DIMENSION desM
Table 19 varies the state dimension dsm E {64, 96, 128, 160} while keeping the number of layers fixed Six per encoder) and all other hyperparameters unchanged. We report MOS on in-distribution text (MOS in-dist.), MOS on the Gutenberg OOD set (MOS OOD), and real-time factor (RTF).
MVC is relatively insensitive to moderate changes in dsmi increasing the state dimension from 96 to 160 yields less than 0.03 MOS improvement while increasing RTF by approximately 9%. We therefore select dum as a favorable extreme, indicating that MVC's gains do not hinge on an unusually large state size.
trade-off rather than a heavily tuned
23

---

Published as a conference paper at ICLR 2026
Table 19: Sensitivity state dimension dsa on LJSpeech. MOS values are averaged over three seeds with 95% confidence intervals; lower RTF is better. The configuration used in the main paper in bold.
State dimension
MOS (in-dist.)
MOS (OOD) 1
RTF (
64
3.96 + 0.09
3.88 + 0.09
0.0164
96
4.02 = 0.08
3,92 1: 0.09
0.0169
128
4.03 = 0.08
3.93 = 0.08
0.0176
160
4.04 = 0.09
3.94 + 0.09
0.0184
E.3.2 CONVOLUTION KERNEL SIZE kcONY
We next vary the depthwise convolution kernel Kconv E 5, in the selective scan. Table 20 reports MOS and pitch RMSE on long-form LJSpeech utterances (duration >> 100 seconds). Larger kernels slightly reduce pitch RMSE but incur higher latency, and the qualitative difference between kernel sizes and 7 is small. We therefore adopt keom in the main experiments as a balanced choice, and do not rely extreme kernel sizes to obtain the reported MOS robustness figures.
Table 20: Sensitivity to convolution kernel size feooy the Mamba block. Pitch RMSE is computed on long-form LJSpeech utterances; lower is better.
Kernel size
MOS (Iong) 1
Pitch RMSE (Hz)
RTF 4
3
4.08 + 0.08
2.06 + 0.06
0.0172
5
4.16 0.07
1.92 = 0.05
0.0177
7
4.17 + 0.07
1.90 = 0.05
0.0184
E.3.3 GATING TEMPERATURE TOATE
Finally, we study the softmax temperature Tgate in the Mamba gating mechanism, which controls how sharply each state attends to its local history. We sweep Tyate E {0.7, 1.0, 1.3) and evaluate OOD text robustness on the Gutenberg set. Sharper gating (0.7e=0.7) slightly harms MOS and stable but do not yield clear gains beyond =ga0==1.0. We therefore fix =.0 for all main results, and the small deltas across temperatures indicate that MVC's behavior is robust to reasonable changes in gating sharpness.
WER, suggesting over-confident local decisions, whereas higher temperatures
Table 21: Sensitivity to gating temperature Tgate on the Gutenberg OOD set. CMOS-N is measured relative to the default configuration with Tgate Tgat==1.0..
Temperature
MOS (OOD) t
CMOS-N t
WER +
0.7
3.83 0.09
-0.06
7.12%
1.0
3.88 + 0.09
0.00
6.89%
1.3
3.86 0.09
-0.02
6.97%
Overall, the small performance variations across state dimensions, kernel sizes, and gating tem- peratures support the view that MVC's improvements arise from its three-way SSM conditioning architecture and gated fusion design, rather than from fine-tuning a narrow hyperparameter regime.
F INDUSTRIAL-SCALE SYSTEMS: CONTEXT AND COMPARISON
Industrial TTS systems such as NaturalSpeech 3, Cosy Voice 3, and HiggsAudio- V2 differ funda- mentally from MVC across every axis: data scale 100 hours 24+245 hours), multilingual vs. English-only training, LLM-scale semantic/tokemizer modules, and multi-task objectives (zero- shot, dialogue, editing, music/SFX generation). They also evaluate private domain-specific
24

---

Published as a conference paper at ICLR 2026
Table 22: Qualitative positioning of MVC relative recent industrial-scale systems. Natural- Speech 3, CosyVoice 3, and HiggsAudio-Vi V2 operate at much larger data and model scales, with different objectives and proprietary evaluation pipelines. MVC is open-data encoder study under unified meeldifflsion-wocodeer setup, and thus numeric comparisons would be misleading.
System
Training
Langunges
architecture setting
Representative metrics
reported
NaturalSpesch 2-00k multi- speaker, multi- style speech
Multilingual
Factorized diffusion TTS
On LihriSpeech lest-clean:
Industry (Mulri- hundred thou- sand proprictary)
with FACodec; separate gossolylomeen/ccosstitim modules; zero sbot und long-
Sim-O 0.67, Sim-R 0.76,
SMOS
(public private)
form gencration.
CosyV 3kh 170k Multilingual (zh/en CV3 (v3lBval)
MinMo-based acoustic tok- TIS LM condi- lional flow matching (CFM) decoder; task TTS/S2S.
On SEED-TTS EVAL: CER 1.27
Industry (Large multilingual corpus. propri- etury)
moltilingual pora
(en). WER 6.96 (hard) with strong
HigusAudio-
10Mh Audio-
Muliliinual multi-
Audio language model (5.88 params) with codehookk RVQ tokenizer and Du- alFFN adapter: unified spech//usic/SFX.
Bead WER 2.44%, 67.7% speaker similarity; dia- logue WER 18.83%
Industry (Large scale, muldi-dominn, proprictary)
V2
Verse (speech, music, SFX)
domain
MVC (nurs)
24h LJSpeech 245h LibrTTS (public)
English
Fully SSM based condition- ing stack (hi-Mamha text, Temporal Bi-Mamba, Expres- sive Mamba + AdaLN) un- diflusionn/veeeer pipeline.
Improves MOSCMOS and WER over StyleTTS2, VITS, and hybrid Manba-atteation baselines under identical pre- processing vacuders.
Acudemic (24h 245h, public datascts)
benchmarks that cannot be reproduced under our controlled mel-diifuuioon-wocderr setting. For this reason, numeric side-by-side MOS/WER comparison would be misleading. Instead, Table 22 summarizes the qualitative distinctions.
25


Reviews:
[{"review_id": "D43WL43Mmd", "reviewer": "Reviewer_Dek4", "summary": "MambaVoiceCloning proposes a TTS system whose entire deployed conditioning path (text, rhythm/duration, prosody) is state–space only at inference, paired with a diffusion decoder for quality, yielding linear-time encoding, bounded activation memory, long‑form stability, and feasible streaming with finite look‑ahead. It uses three SSM modules: (i) a gated bidirectional Mamba text encoder, (ii) a Temporal Bi‑Mamba trained with a lightweight aligner used only during training, and (iii) an Expressive Mamba with AdaLN conditioning.\n\nThe proposed method shows statistically reliable quality and efficiency gains over StyleTTS2 and Mamba-attention hybrids under a matched mel–diffusion–vocoder setup, while confirming the diffusion decoder remains the main latency bottleneck. \n\nAll encoder modules run in O(T) with bounded activations and no global attention maps, reducing peak memory, improving encoder throughput, and enabling longer inputs/batches; the diffusion decoder still dominates latency.", "questions": "1. How are Mamba selective-scan hyperparameters (state size, convolution kernel, gating temperature) chosen for each encoder, and how sensitive are results to these choices across datasets and lengths?\n\n2. The Bi‑Mamba text fusion uses a learned gate plus AdaLN; is the gating stable on very long sequences or under domain shift, and are there failure cases where one direction collapses?\n\n3. Why is fusion in the Temporal Bi‑Mamba kept linear (no gating) while gating is used elsewhere; are there quantitative trade-offs or counterexamples where temporal gating would help long-form speech?\n\n4. The lightweight attention aligner is used only during training; what architecture and capacity does it have, and how robust are learned alignments if the aligner is partially mis-specified or noisy?\n\n5. How long must the reference be for stable similarity in zero-shot cloning?\n\n6. The paper claims finite look‑ahead streaming with Uni‑Mamba; what look‑ahead L is used in experiments, and how does WER/MOS vary with L for 2–6 minute inputs?\n\n7. MVC is trained on English but tested on CSS10 ES/DE/FR with shared vocab and <lang> tags; what failure modes occur in cross-lingual phoneme inventories and stress patterns, and does alignment degrade on languages with different syllable timing?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method provides linear‑time scans, bounded activations, lower peak memory, and higher encoder throughput (≈1.6×).\n\n2. The proposed method offers modest improvements over StyleTTS2 and Mamba‑attention hybrids in MOS/CMOS and objective metrics", "weaknesses": "1. Since the decoder model dominates the latency, the linear-time encoder does not translate into large overall RTF gains.\n\n2. On LJSpeech and LibriTTS, differences in metrics as compared to the baselines are small; the paper frames MVC as a practical encoder-side refinement, not a paradigm shift.\n\n3. The system lacks explicit emotion or fine-grained style controls beyond AdaLN conditioning, limiting targeted prosody manipulation compared to methods with richer style token or reference-attention mechanisms.\n\n4. Claims of long-form stability are positive, but focus on MOS/WER and pitch drift; cross-chunk smoothing and boundary artifacts under streaming conditions are not deeply analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCkpWHDPDC", "reviewer": "Reviewer_PzfM", "summary": "The paper proposed to use MAMBA style SSM blocks in all conditioning encoders of a diffusion-based TTS model.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper replaces all encoders with MAMBA layers and show some improvements in latency, RTF and peak memory usage.\n2. The paper open sourced the code for inspection, which helps with reproducibility.", "weaknesses": "1. The method is only compared against a few older TTS models like VITS and StyleTTS. There are many newer models (NaturalSpeech/CozyVoice/Higgs Audio etc) that are not compared against. While the authors didn't claim SOTA, I think the paper is weaker without comparison with SOTA methods.\n2. The ablation study section is short and lacking details. By \"removing expressive MAMBA\" what does the author mean exactly? Is it replacing the expressive MAMBA encoder with some attention/RNN based encoder or remove the encoder all together? Also the authors said one of the contribution is the gated fusion for the bi-MAMBA layer, but it is not properly ablated in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8YdnYEePZ", "reviewer": "Reviewer_sKr7", "summary": "The author proposed a Text to speech(TTS) model, MembaVoiceCloning(MVC). MVC is the first state-space model throughout the entire encode/conditioning stack at inference for TTS, by removing attention and RNN modules. The model is consists of 3 components. It applies noval Bidirectional Mamba encoder with AdaLN(adaptive gating and layer normalization). It also applies Expressive Mamba Encoder that uses gated fusion and AdaLN-conditioned modulation within an SSM to enable expressive control in linear time while disentangling style from phonetic content. \n\nThe model has 21M parameters and was finetuned standalone and then joint-trained with diffusion decoder. Compared to baseline models, MVC shows the best MOS and Out-of-Distributed(OOD) robustness. Detailed ablation study shows the effectiveness of all three components.\n\nOverall the paper is well structured, easy to follow and shows good vision, novelty and results.", "questions": "The baselines are focused on transformer based SOTA. Are there any comparisons to other Mamba-based TTS models?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The model has good novelty on model architecture by replacing attention and RNN blocks by SSM-only blocks at inference time. The components are innovative on their ideas and implementations. For example, it introduces learnable gated fusion into previous bi-mamba encoders and it enhances prosodic disentanglement and long-form stability. Expressive mamba encoder also uses gated fusion and AdaLN-conditioned modulation.\n\n2. Benefits from the SSM-only blocks, the model operates in linear time and bounded memory, which is optimized for efficiency. This is good for scalability. During inference, the proposed encoder model is not dominating the runtime. \n\n3. Experiments show that the model has SOTA performance on both subjective and objective results compared to baselines. Ablation study is comprehensive and detailed.", "weaknesses": "1. The LJSpeech and LibriTTS are standard TTS datasets in english, but it might be good to also extend the experiments on other languages -  this has been mentioned in the conclusion already", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hPJWsRs80s", "reviewer": "Reviewer_9i8Z", "summary": "This paper presents MambaVoiceCloning (MVC), a text-to-speech framework that removes all attention and recurrent components from the encoder and conditioning path at inference, relying solely on SSMs. The proposed system includes three Mamba structured modules. \nThe key claimed is that MVC is the first fully SSM-only conditioning stack for TTS, distinguishing it from prior Mamba–attention hybrids that retain recurrence or attention in duration/prosody modeling. \nThe MVC is trained based on LibriTTS, LJSpeech while tested based on VCTK and CSS10 dataset. The results in this paper show consistent improvements over StyleTTS2 and Mamba–attention baselines in terms of MOS, CMOS, F0 RMSE, MCD, and WER, while achieving a smaller parameter count and higher encoder throughput. \nOverall, the paper offers a technically report by demonstrating the practical benefits of fully SSM-based conditioning in TTS. The writing of this article is obscure and not well-structured, failing to effectively demonstrate its innovativeness.", "questions": "1. Due to the improvement in the synthesis effect of long sentence input brought by the use of mamba block, have tests been conducted on longer inputs? The current definition seen is 10 seconds, but 10 seconds is not considered a long sentence for the audiobook scenario mentioned in the paper. Nowadays, there are many zero-shot large language model based speech synthesis solutions that can support longer text input for synthesis, such as cosyvoice series . What are the advantages of the Mamba-based model solution?\n2. In Section 4.1 \"DATASETS AND PREPROCESSING\", “LJSpeech Ito & Johnson (2017) (13k, ∼24 h, 1 spk.), LibriTTS Zen et al. (2019) (∼245 h, 1,151 spk.), VCTK Veaux et al. (2017) (∼44 h, 109 spk.; zero-shot), and CSS10(ES/DE/FR) Park & Mulc (2019) (1 spk./lang.).” What do the contents in the brackets represent? And why aren't they unified?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This article applies the mamba block to the innovation of the TTS model structure, replacing the transformers based structure. By leveraging the advantages of the mamba block, it designs and implements the transfer of voice characteristics and style effects in speech synthesis.", "weaknesses": "1. The writing logic of the article needs to improve. Starting from the introduction section, there is no clear explanation of the specific reasons for using the Mamba block to replace the transformer structure.\n2. Regarding the calculation symbols and abbreviations in the method module, there is no unified explanation. They are scattered throughout the article and seem disorganized.\n3. In the experimental module, the baseline for comparison does not provide a detailed explanation of their training configuration. Especially for the VITS and JETS comparison schemes, no detailed description is given of the adopted structure and training situation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)