UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice
Authors: Sitong Cheng
Bianweizhen
Xinsheng Wang
Ruibin Yuan
Jianyi Chen
Shunshun Yin
Yike Guo
Wei Xue
Venue: ICLR 2026 Poster (2026)
Keywords: Speech-to-Speech Translation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: UniSS achieves state-of-the-art speech-to-speech translation performance while preserving voice, emotion, and duration consistency by transfer pre-trained LLMs' text translation abilities to speech in a single-stage architecture.

Abstract: The ultimate goal of expressive speech-to-speech translation (S2ST) is to accurately translate spoken content while preserving the speaker identity and emotional style. However, progress in this field is largely hindered by three key challenges: the scarcity of paired speech data that retains expressive styles, the complexity of multi-stage processing pipelines, and the limited transfer of translation capabilities from large language models (LLMs). In this work, we address these challenges by introducing UniSS, a novel single-stage framework for expressive S2ST. Our approach features carefully designed speech semantic and style modeling, enabling seamless integration with existing text-based LLM frameworks to develop a unified text-speech language model. To transfer translation capabilities from text to speech, we propose a cross-modal chain-of-thought prompting process that progressively aligns audio semantics with text and ensures style preservation in the decoded results. Furthermore, we construct and release a large-scale, high-quality expressive S2ST dataset, UniST, comprising 44.8k hours of data. Experimental results show that UniSS significantly outperforms previous methods in translation fidelity and speech quality while preserving voice, emotion, and duration consistency. Our work establishes a simpler and more effective paradigm for building the next generation of expressive S2ST systems. Audio samples are available at https://cmots.github.io/uniss-demo/.

Published as a conference paper at ICLR 2026
UNISS: UNIFIED EXPRESSIVE SPEECH-TO-SPEECH TRANSLATION WITH YOUR VOICE
Sitong Cheng', Weizhen Bian', Xinsheng Wang?, Ruibin Yuan', ,uan',Jianyi Chen',
Shunshun Yin", Yike Guo"; Wei Xue
Hong Kong University of Science and Technology
Soul Lab
Website: https://cmoos.gihubbbooouuisssis
ABSTRACT
The ultimate goal of expressive speech-to-speecl translation (S2ST) is accu- rately translate spoken content while preserving the speaker identity and emotional style. However, progress in this field largely hindered by three key challenges: the scarcity of paired speech data that retains expressive styles, the complexity of multi-stage processing pipelines, and the limited transfer of translation capabilities from large language models (LLMs). In this work, we address these challenges by introducing UniSS, novel single-stage framework for expressive S2ST. Our ap- proach features carefully designed speech semantic and style modeling, enabling seamless integration with existing text-based LLM frameworks to develop a uni- fied text-speech language model. To transfer translation capabilities from text to speech, we propose a cross-modal mhain-of-thoughh prompting process that pro- gressively aligns audio semanties with text and ensures style preservation in the decoded results. Furthermore, we construct and release a large-scale, high-quality expressive S2ST dataset, UniST, comprising 44.8k hours of data. Experimental results show that UniSS significantly outperforms previous methods in trans latioation fidelity and speech quality while preserving voice, emotion, and duration consis- tency. Our work establishes simpler and more effective paradigm for building the next generation of expressive S2ST systems. Audio samples are available /ttps::/ cmwt.s..t..o.mm.mmooooooo. gi thub. ioaeb..... 10/ un sessssdee /emo//
INTRODUCTION
Speech-to-speede translation (S2ST) enables conversion from a spoken utterance in one language to a spoken utterance in another, facilitating critical applications like real-time interpretation and cross-lingual video dubbing. Recent ad advancements have significantly improved translation fidelity from semantic perspective (Jia et al. 2019 Lee et al., 2022] Inaguma et al., 2023). However, achieving high-quality translation that also faithfully preserves the expressive aspects of speech, such as speaker identity and emotional style, still remains an open challenge.
A conventional cascaded S2ST system typically consists of three sequential components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthe- sis (Casacuberta et al.] 2004 Nakamura et al. 2006). However, this cascaded architecture often suffers from error accumulation across stages and struggles to retain paralinguistic features of the original speech. To address these limitations, subsequent rescarch has shifted towards end-to-end approaches that aim to translate speech to another language while preserving expressive character- istics (Barrault al. 2023 Wang et al. 2024 Le et al. 2024).
Most recently. the application of large language models (LLMs) to generative speech tasks has further accelerated the development of S2ST systems. Current approaches typically fall into two categories: (1) single-stage methods, which directly predict multi-stream acoustic tokens autore- gressively via multi-head outputs (Labiausse al.) 2025); and (2) two-stage pipelines, which first generate semantic tokens autoregressively. followed by another autoregressive (AR) model to pre- dict acoustic tokens (Dong al. 2024) Rubenstein al. (023). [ong & Veluri (2024) and Peng et al. (2024) use single AR language model to jointly model semantic and partial acoustic to- kens, along with a non- non-autoregressvve (NAR) model for complete acoustic information. While these
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Voice SIM
UniSS
Cross-Modal Prompting
Translation Siprech-BLEE))
Pre-tralned
Quality
<Listeno 113 (L
Mode
Transiatee Hello wordd! <Speate
+
LLM
Performance
Mode
word <Speaks
Emotio 3IM
Transfer text translation
Previous Methods
capabiliies to speech
Mutti-stream
Simple Architecture
Duration Consistenyy (SL))
Hoto world!
(UTMDS)
Architectural complexity
Uri5s Stage
LLM
Stago
mills
Untapped LLM potential
Searriess
Figure 1: UniSS integrates pre-trained text LLMs and transfers their translation capabilities to the speech domain. UniSS outperforms both end-to-end S2ST systems and cascaded systems trans- lation fidelity, voice preservation, duration consistency, and speech quality.
approaches have shown promising results, they also introduce significantly more architectural com- plexity than textual LLMs.. Additionally, they treat the LLM as a rqquencefo--squennc converter, failing to leverage the pre-trained knowledge for textual translation embedded within the LLM.
In pursuit of high-fidede figh-fidelity. expressive S2ST, and to overcome the limitations outlined above, our vision is defined by three core principles: (1) a single-stage architecture to eliminate complexity; (2) unified model that aligns speech and text modalities; and (3) mechanism to explicitly leverage the proven text translation capabilities of LLMs. To our knowledge, no existing approach satisfies all three principles simultaneously.
To address the challenge of architectural complexity, we present UniSS, a unified single-stage S2ST framework that preserves timbre, emotion, and duration consistency. Figure demonstrates ap- proach and performance results. UniSS builds upon pre-trained Qwen2.5-14 1.5.---.55-Innttrr (Yang et al. 2024) the backbone, modeling content and speaker information in a single AR language model without architectural modifications. We transfer the LLM's pre-trained text translation capabili- ties to the speech domain through cross-modality chain-of-thour 2022]. During inference, our Quality Mode guides the model to decompose the complex S2ST task into sequential listen, translate, and speak steps within single inference pass. We also introduce Performance Mode that trades translation fidelity for faster inference, enabling deployment across diverse scenarios.
(CoT) prompting (Wei al.,
Realizing the vision outlined above demands large-scale, high-quality training data that preserves both translation accuracy and speaker expressiveness. However, existing S2ST datasets are either small-scale to train powerful unified models (Jia et al. (022b) suffer from quality control issues when scraped from the web (Barrault al. 2023). To address this data challenge, we design a scal- able synthesis pipeline and contribute UniST, T, a 44.8k-hour Chinese-English S2ST dataset offering high translation fidelity and rich speaker preservation.
Our main contributions are follows:
We propose UniSS, a single-stage AR model for expressive S2ST that directly leverages pre-trained textual LLM, eliminating the architectural complexity of prior work.
Unified Stagle-Stage Architecture for S2ST:
Transfer of Text Translation Capabilities to Speech: We demonstrate how to effectively lever- age pre-trained LLMs' superior text translation abilities for speech translation, bridging the modality gap through cross-modal CoT prompting and enabling flexible control quality- efficiency trade-offs.
A Large-Scale, High-Quality S2ST Dataset: We design reproducible pipeline create pressive S2ST datasets and construct UniST, a large-scale Chinese-English S2ST dataset with speaker voice and emotion preservation.
RELATED WORK
2.1
SPEECH-TO-SPEE TRANSLATION
srcech-to-speeche translation must preserve semantic accuracy during cross cross-lingual conversion. Tra- ditional cascaded systems (Casacuberta al. 2004 Nakamura al. 2006) chain ASR, MT, and
2

---

Published as a conference paper at ICLR 2026
TTS components sequentially, suffering from enror accumulation and information loss through text bottlenecks. Early direct methods (Jia et al. 2019) encountered significant challenges with trans- lation quality and synthesis artifacts. The breakthrough came with discrete unit-based methods, where speech-to-unit translation (S2UT) approaches (Lee et al. 2022 Zhang et al. 2021: Inaguma et at 2023) employ discrete speech units (subwords, phonemes, semantic tokens) intermediate representations, which enables effective disentanglement of linguistic content from acoustic prop- erties. SeamlessM4T (Barrault et al., 2023) achieved robust multilingual performance via unified multitask optimization of translation and synthesis. Recent systems focus enhancing expressive- ness while maintaining translation fidelity (Jia et al.) 2022a Song et al. 2023; Wang et al. 2024). Seamlessexxressivee (Barrault et al.) 2023) designs PRETSSEL vocoder (Hwang et al. 2024) to enhance expressiveness preservation. TransVIP (Le et al. 2024) employs feature disentanglement to separately model semantic, acoustic, and temporal information for voice and isochrony control.
2.2 S2ST WITH AUTOREGRESSIVE LANGUAGE MODELS
The success of LLMs in text processing has inspired their adaptation to S2ST through speech tok- enization. Wang et al. 2023) and Hu et al. (2025) focus primarily on semantic translation without preserving voice characteristics. To address voice preservation, some works model acoustic features using multiple AR models (Rubenstein et al. 2023 Dong al.. 2024). Recent unified approaches attempt to model both semantic and acoustic features within a single AR model (Peng et al.) [202]] Gong & Veluri 2024). However, they still require an additional NAR model for complete acoustic generation. Hibiki (Labiausse et al. 2025) proposes a multi-stream architecture processing source and target speech simultaneously, but necessitates substantial architectural modifications, including nested Transformers and training from scratch. A fundamental limitation across these approaches their reliance on complex architectures to complete acoustic representations. Moreover, they fail to explicitly leverage the L.M's pre-trained text translation capabilities, treating them merely as generic sequence converters. UniSS addresses both limitations through a single-stage architecture while effectively transferring LLM's text translation capabilities to speech.
METHOD
As illustrated in Figure 2 UniSS is unified AR language model for expressive speech-to-specd translation, built upon a single-stage architecture with cross-modal chain-of-thought prompting framework, and optimized via a progressive training strategy. This section details the three core components of UniSS: model architecture, cross moosl CoT prompting, and training methodology.
3.1 UNISS ARCHITECTURE
The goal of UniSS is to perform expressive S2ST by modeling the conditional distribution P(Yige|Xarch where X and Yigt denote the source and target speech waveforms, respectively. To achieve faithful content translation while preserving speaker identity and expressiveness, UniSS introduces three types of speech tokens: speaker tokens Sepk with fixed sequence length to capture global style attributes, e.g., timbre, prosody, and emotion; linguistic tokens Şling to encode the tent of the source utterance; and semantic tokens Seem to represent the expressive target utterance and can be directly decoded into waveform when combined with speaker tokens.
The overall pipeline follows a single-stage autoregressive generation process:
X
Tokenize
(Sspk glingy arc: src
Spesch Translate
spk Seem srC Sig
(S
Detokenize
Y Yigt-
This unified design allows UniSS to handle expressive S2ST without intermediate acoustic repre- sentations cascaded systems, maintaining fidelity in both content and voice.
3.1.1 UNIFIED TEXT-SPEECH LANGUAGE MODEL
We build our generation model upon the pre-trained Qwen2.5- 1-552.5-1.BB-nnttrree (Yang et al. 2024), a strong LLM backbone. By expanding the model vocabulary to include the discrete speech tokens, UniSS treats speech and text uniformly as token sequences, allowing both modalities to be processed within the same transformer architecture.
The model takes a concatenated prompt of speaker tokens Sapk and linguistic tokens Sling as input, and autoregressively generates the semantic token sequence Seem. The model is trained using
3

---

Published as a conference paper at ICLR 2026
Large Language Model
Progressive Training
Refine
ASA
Drecct 525T 5 ythg
5 gine
5
Speaker Tokenizer
Linguistic Tokenizer
f
Hello ANW
Codec Decoder
"Ste""
Tasks
Croas-modal Prompting
Ted
Qually Stel
Transiaton
s
5tm
Tokena
Speaker Token
Linguetcc Token
Performanza 5 g
IargenCheress
T g
Speode 0
Bogin of Transiate
Text Token
Semenio Token
Figure 2: Overview of UniSS, cross-modal modal CoT prompting, and 3-phase progressive training. Con- trol Tokens instruct the task (S2ST Quality Mode in the figure), target language (Chinese in the figure), and output speed ratio compared with input (usually 1.0 means duration matching).
standard next-token prediction objective:
TToue
LAR =
Σ
log Po (Tout, [P, Tou,
where Po(-I-) is the conditional probability of the next token given the context, as modeled by the LLM with parameters 0. P is the input prompt containing (Stpk Slin)) and Tout,<t denotes the sequence of all previously generated tokens. Tout represents the target sequence containing text tokens, semantic tokens, or both, depending the specific task mode.
3.1.2 SPEECH TOKENIZER
To represent both the content and expressive characteristics of speech, UniSS adopts triple- tokenizer strategy, transforming the waveform W into single-stream discrete token sequences:
15*nk,gling S Tokenizer(W).
Here, Sepk and S are produced by the speaker tokenizer and semantic tokenizer, respectively, both introduced in BiCodee (Wang al. 2025). Specifically, the speaker tokenizer uses BiCodec's global encoder to extract speaker-related attributes, such as timbre, emotion, and prosody, into fixed-length sequence of 32 tokens Sapk The semantic tokenizer captures other semantically rele- vant information from the waveform and encodes it into Ssem at 50 tokens per second.
Meanwhile, Sling is generated by the linguistic tokenizer, which adopts the GL.M-4 speech tok- enizer (Zeng et al. 2024). It leverages quantized Whisper encoder (Radford et al., 2023) to con- vert speech into variable-length sequence of linguistic tokens at a rate of 12.5 tokens per second, enabling robust content understanding.
In the S2ST task, the source waveform Xarc represented by (Sapk Sling), while the target wave- form Ytgt is represented by Spem. This design is motivated by preliminary experiments, which revealed that although BiCodec's s semantic tokens are highly effective for waveform reconstruction, their self- supervieed nature makes them suboptimal for content understanding. By separately repre- senting speaker identity, linguistic content, and generation-oriented semantics, UniSS enables more accurate and controllable S2ST modeling.
3.3. SPEECH DETOKENIZER
We employ the BiCodec decoder to reconstruct the waveform Yigt from the semantic tokens Spem generated by the LLM. The decoder is conditioned on the source speaker tokens Sapk to ensure voice and emotional style consistency. The decoding process is based the simple concatenation of speaker and semantic tokens, represented as:
Y Egt Decoder( S Sege")
(3)
This single-stage process directly reconstructs high-fidelity audio at a 16kHz sampling rate.
3.2 CROSS-MODAL CHAIN-FF-THOUGHT PROMPTING
Direct speech-to-speede translation is an inherently complex task. Inspired by the success of chain- of-thought (CoT) prompting in LLM, introduce cross-modal CoT prompting framework to decompose this task into a more manageable sequence of listen, translate, and speak steps. This approach is designed to effectively transfer LLM'' powerful, pre-trained text translation capabilities
4

---

Published as a conference paper at ICLR 2026
to the speech domain. Our framework provides two controllable prompts to trade off translation fidelity and efficiency.
UniSS is guided by imput prompt P. which is structured sequence of control and source tokens:
tgt P [Ctask: "aang Capeed:
Sepk arc Stiee)
where Ctasks Canng and Capeed are special tokens specifying the task mode, target language, and duration ratio between source and target speech, respectively. A special bogin-of-traasllttion token, BOT, signals the model to begin generation, producing output sequence Tout that is terminated by an -edooddd token, EOD.
Lgt
This mode follows the full CoT prompting path to maximize translation fidelity. The model first listens by generating the source transcription Tre. then translates it into the text in the target language Trtt, and finally speaks by generating target semantic tokens Stgm. Prompted with Ctask Quality Mode, this explicit chain allows the model to leverage its robust text translation abilities, formally represented
Quality Mode.
Tout = [T ares T Tegt Egt
Performance Mode. This mode compresses the CoT path for faster inference. It skips the tran- scription step and directly generates the target text translation Teg followed by the corresponding semantic tokens Sigt Prompted with Ctask Performance Mode, the output becomes:
Toat
T igt
S Stg
3.3 PROGRESSIVE TRAINING STRATEGY
We adopt a three-phase progressive training strategy optimize the model, aiming to gradually build cross-modal capabilities while mitigating catastrophic forgetting of the LLM's foundational text translation abilities.
Phase 1: Speech- Text Text Alignment. We begin by adapting the pre- pre-trained LLM to the speech modal- ity through a multi-task learning stage involving four foundational tasks: ASR, TTS, Speech-to- Text Translation (S2TT), and MT. The ASR, TTS, and S2TT tasks align speech and text, while MT pre- serves the model's foundational translation capabilities. Each task is defined by specific prompt structure and target output sequence:
ASR: P
Caers lang"
src Sopk sling], Tout = Tsre. arc
S2TT: P
[ceuu Cang
tgt
Sapk Sling], Tout Teyt.
TTS: P
[Cets> are
Gang: :
Sapk: T Tsre), Tout S.em
zgt MT: P = [Cma, lang" Tare], Tout Tigt.
Gang denotes the source language, and Stem denotes semantic tokens from source speech. This phase endows the model with robust speech understanding and generation while prioritizing the preservation of text translation capabilities.
Here cere
Phase 2: S2ST with CO In the second phase, we introduce the core S2ST task. The model is trained to generate outputs using the CoT prompting formats described in Section 3.2 well simplified direct generation mode that bypasses intermediate text outputs:
Agt P [cu2str Caang" Cspeed
Sapk Sling] Tout Seem
This phase utilizes the alignment from Phase 1 and transfers the LLM''s text translation capabilities to the speech domain.
Phase 3: Refinement. the final phase, we fine-tune the model on the full S2ST task using both CoT prompting modes. This phase uses an annealed learning rate to stabilize the learned CoT patterns and optimize the final translation performance.
5

---

Published as a conference paper at ICLR 2026
TTS Corpus
UniST High-qualt quality
UnS/ Gerend
U451 lewe
Ratio Filber
Sorpo
ASR Filtrr
ABR Ratio
Text Translale
Filter
Sciipl
24h
48 Trarsiation
Stript)
Expressive TTS
Translaton
(R) The pipeline of UniTT catasot.
(b) Distribution duruaaton Ratio in UniST
(c) Distribution of speech duration in UniST
Figure 3: Overview of the UniST data construction pipeline and analysis of duration ratio and speech duration distributions.
THE UNIST DATASET
Training end-to-end expressive S2ST models faces significant bottleneck due to the scarcity of large-scale, parallel data that preserves speaker characteristics. To address this limitation, we design a scalable data synthesis pipeline. The pipeline generates UniST, our large-scale Chinese-Fnglish expressive S2ST dataset. Figure Baa illustrates the data construction process.
Data Sources and Cleaning. Our pipeline begins with large-scale TTS corpora containing paired speech and transcription (Xree: combined from several Chinese and English public datasets (details in Appendix E. Following VoxBox (Wang et al.) 2025). first clean the corpus using Paraformer (Gao et al. (022) to re-recognize speech and compute Word Error Rate (WER) between re-recognized and original transcriptions, discarding samples with WER 0.05.
Translation and Target Speech Synthesis. After source cleaning. Tree is translated by Qwen2.5- 72B Instrucct (Yang et al. 2024) produce target text Tigt in another language. Prompts used in translation are shown in Appendix E.22 We then apply an expressive TTS model, SparkTTS (Wang al. 2025), to synthesize the target speech egt from Tegt, conditioned by X X preserve the source speaker's voice. To enable fine-grained speed control, we calculate the duration ratio between source and target speech and discretize it into speed tokens Capeed with 0.1 intervals. This creates complete parallel samples (Xare, Tree Tege, Yig Cspeed).
Quality Filtering and Dataset Variants. The synthesized data undergoes final quality filtering. We apply ASR to Yigs and discard samples with WER greater than 0.01 against T igtt- We also apply duration ratio filter to keep synthesized speech with duration within [0.5, 2.0] times of the source speech. The filtered data forms our UniST General dataset (44.8k 88 hours). Our refined UniST High- Quality dataset (19.8k hours) is created by applying additional Voice Activity Detection (Team) 2024) to remove silence at the beginning and end of speech segments, and stricter duration ratio filter of [0.7.1.5].
As shown in Figure 3 bbc), UniST General provides greater data diversity and generalization poten- tial due to its larger range of duration ratios and speech lengths. UniST High-Quality focuses on temporal consistency, making it ideal for the final refinement phase of our progressive training. In Appendix E..] we show that the UniST dataset also preserves emotional style in synthesis.
EXPERIMENTS AND RESULTS
5.1
TRAINING SETUP
Phase I establishes text- UniSS is trained in three progressive phases, introduced in Section 3.3
speech alignment using 77.1k hours of speech data and MT text data from WMT17 (Bojar et al.] 2017). Phase 2 introduces CoT prompting with UniST General dataset, mixed with Phase data at a 2:1 ratio. Phase fine-tunes the model exclusively on the UniST High-Quality dataset.
Training employs the AdamW optimizer [Lohccilovv & Hutter 2019) with 2.3M-token batch size, weight decay of 0. and momentum parameters (0.9, 0.95). All audio is resampled to 16 kHz, and the LLM vocabulary is expanded to 180,407 to include speech and control tokens. Learning rates progress from 8e-4 in Phase 1 to 2e-4 in Phase 2, and finally anneal from 5e-5 to 5e-6 in Phase 3. The model is trained on 16 NVIDIA H800 S0G GPUs using the Megatron-LM Framework (Shoeybi et al. 2019) for efficient large-model training. Complete training details and hyperparameters are provided in Appendix B.I

---

Published as a conference paper at ICLR 2026
5.2 EVALUATION SETUP
5.2.1 DATASETS
For our primary S2ST evaluation, we use the Chinese (ZH) and English (EN) test sets from CVSS- T (Ji et al., 2022b), which contain 4,897 utterance pairs (8.2 hours for Chinese and 6.3 hours for English). For emotion preservation evaluation, we randomly sample 300 utterances from each emotional speech dataset (ESD (Zhou et al., 2021) and CREMA-D (Cao al. 2014)), covering selected set of 4 emotions (happy, sad, angry. neutral). We evaluate both English-to-Chinee (EN- ZH) and Siiinese-to-English (ZH-EN) translation directions.
5.2.2 METRICS
We employ comprehensive objective and subjective metrics to evaluate S2ST performance. For translation fidelity, we measure BLEU scores (Papineni et al., (002)) on both ASR-transcribed speech outputs (Speech-BLEU) and intermediate text (lext-BLEUX) Prosody preservation is as- sessed through prosody similarity (A.PCP) (Andrews et al., 2022). Duration consistency is eval- uated using Speech Length Compliance (SL.C) within 0.2 and 0.4 ratio tolerance (Wu et al. 2023). Speech quality is measured using UTMOS (Saeki et al. 2022). Additionally, we conduct a sub- jective Mean Opinion Score (MOS) evaluation where bilingual speakers rate emotion similarity, speaker similarity, and naturalness on a 5-point scale. Detailed descriptions of all evaluation metrics and implementation procedures are provided in Appendix
5.2.3 BASELINES
We compare UniSS against strong and diverse set of state-of-the-art (SOTA) systems. Details of baseline implementation are introduced in Appendix B.2
Cascaded Systems. To establish an upper bound for modular approaches, we construct
three cascaded baselines using top-perforning open-source models: a 3-Stage-MTT pipeline of Whisper-large-22 (ASR), 2 (TTS) (Du et al. 2024); a 3-Stage-LLM pipeline of Whisper-large--3 (ASR), Qwen2.5-1.. 1.5bb Instruct (MT), and Cosy Voice (TTS); and 2-Stage pipeline of SeamlessM4TT-v2-L. (S2TT) and Cosy Voice 2 (TTS).
HLLB-200-distilles (MT) (Koishekenov al. 2023). and CosyVoice
Multimodal Large Language Models (MLLMs). To benchmark against the latest generation of general-purpose models, we include two leading MLLMs: GPT-4o (Hurst et al., 2024). an enterprise-level model from OpenAl with strong speech-to-speech capability; and Qwen2.5- Omni Xu et al. 2025). powerful open-source MILM building on large-scale audio-language pretraining, enabling both speech understanding and synthesis.
End-to-End S2ST Systems. We compare against dedicated S2ST models that represent the current open-source SOTA: SeT (Medium and Large V2, denoted Seamless-M and Seamless-L), and its expressive variant SeamsessExpressivee (Seamless-Ei, For subjective met- rics, we also evaluate Seed LiveInterpret 2.0 (Cheng et al., [2025). denoted Seed Live, an enterprise-level S2ST system with duplex speech understanding and generation abilities.
5.2.4 INFERENCE CONHIGURATION
For UniSS experiments, we deploy vLLM (Kwon et al. (003)) to support inference. We set a decod- ing temperature of 0.7, top-k of -1, top-p of 0.8, and repetition penalty of 1.1. We report results for both Performance mode, denoted as UniSS (P), and Quality mode, denoted as UniSS (Q).
5.3 OBJECTIVE PERFORMANCE
Table presentss the main comparison results on the CVSS-T dataset. The results clearly demonstrate that UniSS establishes new SOTA in translation fidelity while maintaining voice characteristics, prosody, duration consistency, and speech quality, validating our core design principles. Additional results on other datasets and tasks are provided in Appendix
Translation Fidelity. UniSS achieves state-of-the-art translation fidelity on both EN-ZH and ZH- EN directions. The UniSS (Q) variant achieves a Speech-BLEU of 32.20 EN-ZH and 24.28 on ZH-EN, substantially outperforming all prior end-to-end and cascaded baselines. The efficient UniSS (P) also delivers strong results, surpassing most existing systems. Notably, in terms of in- termediate text metrics, UniSS models perform on par with or better than larger multimodal LLM- based approaches.
7

---

Published as a conference paper at ICLR 2026
Table I: Main comparison results the CVSS-T dataset. Results are presented as EN-ZH ZH-EN. Higher scores indicate better performance. denotes unavailable results. Best scores are in bold and second-best scores are underlined.
Category
Model
#Size
Speech-BLEU
Text-BLEU
A.PCP
SLC
SLC 0.4
UTMOS
GT
CVSS-T
2.37
0.32
0.72
2.27
3.56
3-Stage-MT
2.6
25.02
16.62
25.80
17.08
2.80
2.85
0.56
0.54
0.82
0.88
3.76
3.50
Cascaded
3-Stage-IIIM
3.5B
26.74
17.69
28.36
18.51
2.84
2.82
0.60
0.48
0.87
0.84
3.76
3.48
2-Stage
2.8B
26.94
20.86
27.38
22.20
2.87
2.64
0.67
0.52
0.93
0.70
3.79
3.48
GPT-4o
31.64
19.27
2.66
2.58
0.47
0.37
0.71
0.61
3.46
4.18
MLLM
Qwen2.5-O
7B
7.10
22.66
34.85
24.39
1.90
1.92
0.31
0.35
0.57
0.61
3.23
4.30
Seamless-M M
1.2B
14.53
14.36
24.80
18.44
2.34
2.29
0.54
0.22
0.82
0.45
2.73
3.559
Seamless-L L
2.3B
25.05
17.67
27.61
21.95
2.41
2.15
0.67
0.36
0,95
0.62
2.69
4.04
Seamless-Fi Ex
1.7B
24.45
15.84
26.59
16.74
2.83
2.87
0.68
0.52
0,94
0.77
2.46
2.90
S2ST
UniSS (P)
1.5B
30.28
23.61
30.93
24.45
2.73
2.75
0.98
0.84
0.99
0.97
3.77
3.86
UniSS (Q)
1,5B
32.20
24.28
32.95
26.28
2.71
2.74
0.98
0.87
0.99
0.97
3.76
3.86
Prosody Preservation. In terms of prosody, UniSS achieves competitive performance. UniSS (P)
variant achieves the second-highest A.PCP score (2.73 and 2.75), closely following Seamless-Ess which incorporates dedicated prosody encoder. The performance gap is marginal at only 0.10 (EN- ZH) and 0.12 (ZH-EN), highlighting the efficacy of UniSS in preserving prosodic patterns without specialized modules.
Duration Consistency. UniSS demonstrates superior duration consistency. UniSS (Q) achieves near-optimal SLC 0.2 scores on EN-ZH and the best performance on ZH-EN, improving over the previous best end-to-end system (Seamless-Exx by 44% and 67%. On the more relaxed SLC 0.4 metric, while competing systems achieve scores above 0.90, both UniSS variants deliver near-perfect performance with scores of 0.99 (EN-ZH) and 0.97 (ZH-EN).
Speech Quality. UniSS achieves sfate-of-the-art speech quality among S2ST models. In the EN-ZH direction, UniSS (P) achieves a competitive UTMOS score of 3.77, matching cascaded models (3.76 and 3.79) while surpassing all end-to-end models. In the ZH-EN direction, UniSS surpasses 3-stage systems with a score of 3.86 compared to 3.50.
5.4 SUBJECTIVE RESULTS
In addition to the objective metrics, we conduct comprehensive subjective assessments on emotion preservation, voice preservation, and speech naturalness. The results shown in Table 2
Emotion Preservation. UniSS (Q) demonstrates strong capability in preserving emotions, achiev- ing a MOS of 1. This represents substantial 27% improvement over the expressive S2ST base- line Seamless-Ex (3.56) and surpasses the 3-Stage-M'T cascaded system (4.48). Notably, UniSS (Q) also approaches Seed Live (4.56), indicating its ability in capturing emotional nuance.
Speaker Preservation. UniSS effectively retains speaker voice characteristics without requiring an additional NAR stage-2 model. For speaker similarity. UniSS (Q) achieves a score of 4.42, outperforming all other models. This represents 0.07 improvement over the 2-stage system (4.35), which deploys a carefully designed TTS model. These results underscore the ability of UniSS to maintain voice characteristics in an end-to-end fashion.
Speech Naturalness. The naturalness of speech from UniSS also favorably rated. UniSS (Q) achieves a naturalness MOS of 4.45, surpassing Seamless-Ex (3.10) and the 3-stage cascaded base- line (4.31) that incorporates a specialized TTS model. This demonstrates UniSS's s ability to generate high-quality, natural-sounding speech without dedicated text-to-speech components.
5.5 INFERENCE SPEED AND QUALITY TRADE-OFF
Our framework provides flexible control over the quality-efficiene trade-off through its different CoT prompting modes. We evaluate inference speed on the AR language model component using the Transformers library (Wolf et al. 2020), using 400 utterances (200 per direction) from CVSS-T on a single H800 GPU without batching.
Table sshows the Performance mode achieves 1.07 speedup over Quality mode with only 1.84 point reduction in Sreech-BLEUS demonstrating a favorable speed-quality trade-off. Furthermore, trained UniSS-Small based on Qween2.5-0.5-IInsmmucc achieving significant computational sav-
8

---

Published as a conference paper at ICLR 2026
Table Subjective MOS evaluation on the expressive emotion dataset. *Seed Live is closed source. Best scores are in bold and second-best scores underlined.
Table 3: Inference speed of AR language model and Speech-BLEU comparison. Speech-BLEU is the av- erage of EN-ZH and ZH-EN. Time is the total infer- ence time on 400 utterances without batching infer- ence. Best results are in bold.
Model
Emo Sim.f
Spk Sim."
Naturalness
3 3-Stage-MT
4.48
4.33
4.31
Model
#Size
Speech-BLEUf Thme(s).
Speedupt
2-Stage
4.27
4.35
4.27
UniSS (Q)
1.5B
33,66
1521.52
1.00x
Seed Live"
4.56
4.19
4.69
UniSS (P)
1.5B
31.82
1426.54
1.07x
Qwen2.5-O
2.84
N/A
4.47
UnSS Small (Q)
0.58
28.17
1339..2
1.14x
Seamloss-Ex
3.56
2.94
3.10
UniSS-Smal (P)
0.5B
25.68
1212.65
1.25x
(Q)
4.51
4.45
ings. UniSS-Smi Small (P) delivers 1.25x speedup with competitive translation fidelity (25.68 Speech- BLEU), making it suitable for resource-constrained deployment scenarios while maintaining the advantages of our unified architecture.
5.6 ABLATION STUDIES
We conduct ablation studies to validate the effect of our design. Table 4 presents the detailed results comparing variants against a base model trained through Phases and 2. The results in type Base and Train are evaluated in the Performance mode.
5.6.1 IMPACT OF PROGRESSIVE TRAINING
Table 4: Ablation study on UniSS components evaluated on CVSS-T test set. w/ and w/o denote 'with' and 'without". respectively. Best scores are in bold.
Our three-phase progressive training signifi- cantly impacts performance. Phase refine- ment (w/ Phase 3) contributes improvements of +0.90 and +2.06 Speech-BLEU points, vali- dating the importance of high-quality data fine- tuning for final optimization. Removing the initial Phase alignment (UniST only) causes severe performance degradation of -7.18 and -10.15 points. This drop demonstrates that Phase 1 texe-speech alignment is essential for subsequent S2ST learning.
Speech-BLEUS
Type
Variant
EN-ZH
ZH-EN EN-ZH ZH-EN
Base
Phase 1+2
29.38
21.55
w/ Phase
30.28
23.61
+0.90
+2.06
Train
UniST only
22.20
11.40
-7.18
-10.15
w/o GLM
14.37
12.82
-15.01
-8.73
Infer
Direct S2ST
14.44
7.15
-14.94
-14.40
5.6.2 IMPACT OF LINGUISTIC TOKENIZER
Replacing our content-focused GLM-4 linguistic tokenizer with BiCodec's self-supervised semantic tokens (w/o GLM) leads significant performance degradation of -15.01 and -8.73 Speech-BI BLEU points. This drop reveals that while BiCodec's semantic tokens excel at speech generation tasks, their self-super nature limits their effectiveness for content understanding the S2ST task.
5.6.3 IMPACT OF CHAIN-OF-THOUGHT FRAMEWORK
Removing the intermediate text generation and performing direct speech-to-speed translation (Direct S2ST) results in a severe performance degradation of -14.94 and -14.40 Speech-BLED points in the inference. This demonstrates that our cross-modal CoT prompting enables the transfer of textual translation expertise to the speech domain and improves translation fidelity.
CONCLUSION, LIMITATIONS AND FUTURE WORKS
This paper introduces UniSS, a unified single-stage expressive speech-to-speech translation frame- work that preserves voice and emotional style. Our approach eliminates architectural complexity through well-designed speech semantic and style modeling, enabling seamless integration with pre- trained LLMs. By transferring LLMs' text translation capabilities to speech via cross-modal CoT prompting, UniSS surpasses previous S2ST systems across multiple dimensions: translation fidelity, speech naturalness, and preservation of voice, emotion, and duration consistency. Beyond the model architecture, design a reproducible pipeline to create expressive S2ST datasets and construct UniST, large-scale Chinese- English expressive S2ST dataset. Our work demonstrates a simple and effective approach for building the next generation of expressive S2ST systems.

---

Published as a conference paper at ICLR 2026
Limitations and Future Works. Current limitations point to several research directions. (1) Language Support: We trained UniSS only on Chinese and English due to resource limitations. The data construction pipeline and training framework can be easily extended to multilingual scenarios, which represents our immediate next step. (2) Speech Tokenizer: While the linguistic, speaker, and semantic tokenizers perform well in UniSS, they originate from different audio tokenizers, causing vocabulary size expansion. Future work will focus on training a unitied tokenizer to merge these components and reduce vocabulary size.
ACKNOWLEDGEMENTS
The research was supported by Early Career Scheme (ECS-HIKSTT2201322), Theme-based Re- search Scheme (45-2005/1-N) from Hong Kong RGC, and Generative Al Research and Develop- ment Centre from InnoHK.
ETHICS STATEMENT
We acknowledge that work speech-to-spest translation, which preserves a speaker's 's vocal identity, has the potential for dual use. Our primary goal is to foster more natural and personal cross- lingual communication, benefiting applications like international conferencing and film dubbing. However, the technology could be misused by malicious actors. Potential risks include the creation of audio deepfakes for disinformation campaigns, impersonation for fraud, targeted harassment.
REPRODUCIBILITY STATEMENT
We ensure that both the methods and the dataset pipeline are reproducible. Training details are provided in Appendix B.I All raw data originates from open-source datasets, with details provided in Appendix
REFERENCES
Pierre Andrews, Guillaume Wenzek, Kevin Heffernan, Onur Çelebi, Anna Y. Sun, Ammar Kam- ran, Yingzhe Guo, Alexandre Mourachko, Holger Schwenk, and Angela Fan. stopes modular machine translation pipelines. In Proc. Empirical Methods Nat. Lang, Process. (EMNLP), pp- 258-265, 2022. (Demos).
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A massively-mutllingual speech corpus. In Proc. Int. Conf. Lang, Resources and Eval. (LREC), pp- 4218-4222, 2020.
Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi -peakr english tts dataset. In Proc. Int. Conf. Speech Commun. (InterSpeech), pp. 2776-27888 2021.
Loic Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong. Mark Duppenthaler, et al. Seamless: Multilingual expressive and streaming speech translation, 2023.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, et al. Findings of the 2017 conference on machine translation (wmt17).
In Proc. Workshop on Machine Translation (WMT), pp. 2017..
Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Trans. Affect. Com- put., 544)3377,,,0 2014.
Francisco Casacuberta, Hermann Ney, Franz Josef Och, Enrique Vidal, al. Some approaches to statistical and finite-state speech-to-speach translation. Comput. Speech Lang., 18(1)
2004.
Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai- jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan.
10

---

Published as a conference paper at ICLR 2026
Gigaspeech: An evolving, multi-domain corpus with 10,000 hours of transcribed audio. In Proc. Int. Conf. Speech Commun. (InterSpeech), pp. 6670-3674, 2021.
Shanbo Cheng, Yu Bao, Zhichao Huang, Yu Lu, Ningxin Peng, Lu Xu, Runsheng Yu, et al. Seed liveinterpret 2.0: End-to-end simultaneous speech-to-spe to-eeeech translation with your voice, 2025.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv 2024.
preprint arXiv:2407.1011
Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech. In Proc. Inf. Conf. Spoken Lang. Process. (SLT), pp. 798-805, 2022.
Qianqian Dong, Zhiying Huang, Qi Tian, Chen Xu, Tom Ko, Yunlong Zhao, al. Polyvoice: Language models for speech to speech translation. In Proc. Int. Conf. Learn. Representations (ICLR), 2024.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang. et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models, 2024.
Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregreessive end-to-end speech recognition. In Proc. Int. Conf. Speech Commun. (InterSpeechi, pp. 10063-2061 2022.
Hongyu Gong and Bandhav Veluri. Seamlessexpressivelm:: Speech language model for expressive speech-to-speede translation with chain-of-thoughh 2024.
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, Yuancheng Wang, Kai Chen, Pengyuan Zhang, and Zhizheng Wu. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. In Proc. Int. Conf. Spoken Lang. Process. (SLT), pp. 885-890, 2024.
Yuxuan Hu, Haibin Wu, Ruchao Fan, Xiaofei Wang, Heng Lu, Yao Qian, and Jinyu Li. Phi-omni-st: A multimodal language model for direct speech-to-speech translation, 2025.
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, et al. Gpt-4o system card, 2024.
Min-Jae Hwang, Ilia Kulikov, Benjamin N. Peloquin, Hongyu Gong, Peng-Jen Chen, and Ann Lee. Textless acoustic model with self-supervised distillation for noise-robust expressive speech-to- speech translation. Findings of Proc. Assoc. Comput. Linguistics (ACL), pp. 15524, 2024.
Hirofumi Inaguma, Sravya Popuri, Ili Kulikoy, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. Unity: Two-pass direct speech-to-speech translation with discrete units. In Proc. Assoc. Comput. Linguistics (ACL), pp. 15655-15680, 2023.
Ye Jia, J.n Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with a sequence-to-sqquene model. In Proc. Int. Conf. Speech Commun. (InterSpeech), pp. 1123-1121 127, 2019.
Ye Jia, Michelle Tadmor Ramanovich, Tal Remez, and Roi Pomerantz. Translatotron 2: High- quality direct speech-to-speech translation with voice preservation. In Proc. Int. Conf. Mach. Learn. (ICML), pp. 10120-10134, 2022a.
Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. Cvss corpus and massively multilingual speech-to-speeed translation. In Proc. Int. Conf. Lang. Resources and Eval. (LREC), pp. 6691-6703, 2022b.
Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory-efficient nl0b-200: Language-specific expert pruning of a massively multilingual machine translation model. Proc. Assoc. Comput. Linguistics (ACI.), pp. 3567-3585, 2023.

---

Published as a conference paper at ICLR 2026
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. Libritts-r: A restored multi-speaker text-to- speech corpus. In Proc. Int. Conf. Speech Commun. (InterSpeech), pp. 5496-5500, 2023.
Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination:: Accelerating large language models without impacting per- formance, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proc. ACM Symp. Operating Syst. Principles (SOSP), pp. 611- 626. 2023.
Tom Labiausse. Laurent Mazaré, Edouard Grave, Patrick Pérez, Alexandre Défossez, and Neil Zeghidour. High-fidelity simultancous speech-to-speech translation, 2025.
Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang. Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, and Michael Zeng. Transvip: Speech to speech translation system with voice and isochrony preservation. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2024.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma. Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, and Wei-Ning Hsu. Direct speech-to-speech translation with discrete units. In Proc. Assoc. Comput. Linguistics (ACL). pp. 3327-3339, 2022.
Keon Lee, Kyumin Park, and Daeyoung Kim. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. In Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 1-5, 2023.
Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. Generative expressive conversational speech synthesis. In Proc. ACM lnt. Conf. Multimedia (ACM MM), pp. 4187-4196, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. Int. Conf. Learn. Representations (ICLR), 2019. (Poster).
Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. Wenetspeech4tss: A 12, 2000hhuuur mandarin its corpus for large speech generation model benchmark. In Proc. Int. Conf. Speech Commun. (InterSpeech), 2024.
Ltd. Magic Data Technology Co. Open source data: Mandarin chinese speech cor-
pus. http: /ww.. wwwagiddaattecccc com /eaam/indx...//ee Lndex.php/ http://mmee/tttppesssuueeee
data info/id/101 2019. Accessed: 2025-08-04.
Ltd. Magic Data Technology Co. Iscslp 2024 magicdata dataset. magicdatatech. www/iscampppppppoocooooccc.ccppe 2024 2024. Accessed: 2025-08-04.
https: //ww..
Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Gen ichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, Jinsong Zhang, et al. The atr multilingual speech-to-speech translation sys- tem. IEEE Trans. Speech Audio Process., 14(2):365-77,, 2006.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proc. Assoc. Comput. Linguistics (ACL), pp. 311-318, 2002.
Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu, Changhan Wang, and Hongyu Gong. Mslm-s2st: A multitask speech language model for textless speech-to-speech translation with speaker style preservation, 2024.
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. Associ- ation for Computational I Linguistics.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. Int. Conf. Mach. Learn. (ICML). pp. 28492-28518, 2023.
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, et al. Audiopalm: A large language model that can speak and listen, 2023.
Takaaki Sacki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. In Proc. Int. Conf. Speech Commun. (InterSpeech), pp. 4521-4522 2022.
Michael Schoeffler, Rabeen-Robert Stöter, Sebastian Westphal, Henning Bayerlein, Bernd Edler, and Jürgen Herre. webmushra - a comprehensive framework for web-based listening tests. Journal of Open Research Software, 6(1):8, 2018.
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3: A multi-speaker mandarin tts corpus and the baselines, 2021.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism, 2019.
Kun Song. Yi Ren, Yi Lei, Chunfeng Wang, Kun Wei, Lei Xie, Xiang Yin, and Zejun Ma. Styles2st: sero-shot style transfer for direct speech-to-speech translation. In Proc. Int. Conf. Speech Com- mun. (InterSpeech), pp. 42-46, 2023.
Silero Team. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number de- tector and language classifier. https://github.com wtt.://aaeee////sleerooooooo 2024. Accessed: 2025-08-01.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, al. Cstr vetk corpus: English multi- speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
Changhan Wang, Anne Wu, and Juan Pino. Covost and massively multilingual speech-to-text translation, 2020.
Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation, 2023.
Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Lingin Li, Zheng Liang, et al. Spark-tts: An efficient llmebased text-to-speech model with single-stream decoupled speech tokens, 2025.
Yongqi Wang, Jionghao Bai, Rongjie Huang, Ruiqi Li, Zhiqing Hong, and Zhou Zhao. Speech- style transfer. In Proc. Assoc. Comput. Linguistics
to-speech translation with (Volume 4: Student Research Workshop) (ACL), pp. 42 49, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, et al. Transformers: State-of of-ie--f-the-aae art natural language processing. In Proc. Empir- ical Methods Nut. Lang. Process. (Demos), pp. 38-45, 2020.
Yihan Wu, Junliang Guo, Xu Tan, Chen Zhang, Bohan Li, Ruihua Song, Lei He, Sheng Zhao, Arul Menezes, and Jiang Bian. Videodubber: Machine translation with speech-aware length control for video dubbing. In Proc. AAAI Conf. Artif. Intell. (AAAI), pp. 13772-13779, 2023.
13

---

Published as a conference paper at ICLR 2026
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report, 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like and-to-end spoken chatbot, 2024.
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tic- Yan Liu. Uwspeech: Speech to speech translation for unwritten languages. In Proc. AAAI Conf. Artif. Intell. (AAAI), pp. 14319-1432) 2021.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with new emotional speech dataset. In Proc. Int. Conf. Acoust. Speech Signal Process. (ICASSP), pp. 920-924, 2021.
14

---

Published as a conference paper at ICLR 2026
A EXAMPLES
We present real examples from different models here, with speech converted to text using ASR for display purposes. As shown in Figure UniSS achieves competitive BLEU scores on challenging sentences where GPT-4o struggles with translation quality. Note Seamless-L and GPT-4o cannot preserve source voice and emotional style.
logal comasponding concept between China and Germany.
UniSS
Seamless-Ex
22.8
Seamless-L.
22.8
UniSS
workplace these identiyy alma mater.
Seamlese Ex
7.5
R H2
Seamless-1
SPP ATA.BAAIIAAAHH00000000 22 C mogge
18 Me 11 1el..
couples may acoption baby.
UNSS
These couples develop adoption for thiir childies
43.0
Seamless-Ex
These couples are likeyy to drow adoption plans for their children.
10.8
couples may edoction plans
childoon.
GPT-4e
couplea may adection
chidren.
UNSS
It also found strong carbon dioxide spectral lines the intrared spectrum of Venus.
42.9
Seamios-Ex
It has also dalacted strong Codwal linas the carafully mapped X-raya.
5.8
spactrum
Seamless-L
found strong carbon light source light poot. cartin dioside
GPT-4o
intrarad
Figure AI: Comparison of speech translation outputs from different models. Examples show source speech (converted text via ASR), ground truth translations, and model outputs with corresponding BLEU scores. UniSS demonstrates superior performance on challenging sentences where other models struggle. Best scores are in bold and second-best scores are underlined.
B IMPLEMENTATION DETAILS
B. UNISS
Table B1: UniSS training configuration showing datasets, hyperparameters, and training settings for each phase.
Phuse
Datasct
Training Tokens
Epochs
Learning Rate
Warm-up
Batch Size
77.lk hours speech data
Phuse
2.3B MT tokens
~>32B per epoch
3
8e-4 (constant)
epoch
2.3M
UniST General dataset Mixed with Phase data (2:1)
Phuse
~55B total
2e-4 (constant)
5% epoch
Phase 3
UniST qualiquality
~ 1008 total
0.9/1
5e-5 + 5e-6 (cosine)
None
2,3M
B.1.1 TRAINING PROTOCOL.
As shown in Table B1 the UniSS trained in three phases progressively. All audio is resampled to 16 kHz. The LLM vocabulary is expanded 180,407 include speech and control tokens. We use the AdamW W optimizer with a weight decay of 0. and momentum parameters (0.9, 0.95). The batch size is fixed 2.3M tokens for all phases.
Phase 1: Text-Speech Alignment. The model is trained mixture of ASR, TTS, S2TT, and MT tasks. This phase utilizes l hours of speech data and 2.3B translation tokens from WMT17, totaling approximately 32B tokens per epoch. The model is trained for epochs with a constant learning rate of 8e-4 and l-epoch warm-up.
Phase Chain-of- Shhough Prompting. We introduce CoT prompting using our UniST General dataset. The model leamss three generation paths: Performance mode, Quality mode, and direct
15

---

Published as a conference paper at ICLR 2026
S2ST data without intermediate text. New data is mixed with Phase data at 2:1 ratio, totaling approximately 55B tokens. We train for one epoch with a constant learning rate of 2e-4 and 5% warm-upp
Phase 3: Annealing. We fine-tune the model exclusively on the UniST High-quality dataset. The learning rate follows cosine schedule, gradually annealing from 5e-5 to 5e-6 over epoch without warm-uu. This phase consists of approximately 10B training tokens. During this stage, the training loss reaches its minimum around 0.9 epochs, and we select the checkpoint at this point as our final model weights.
B.1.2 IMPLEMENTATION DETAILS
Our language model is trained on 16 NVIDIA H800 80G GPUs. We utilize the Megatron-LM, Framework for efficient large-model training. Because audio duration distribution is uneven, padding would waste significant computational resources. We use the sequence packing (Krell et al.) 2022) technique to concatenate multiple samples into a single 18k token long sequence. We use global batch size of 128. Completing all three phases of training takes approximately days.
B.2 BASELINES
Table B2: Overview baseline models used in experiments.
Pipeline
Stage
Model Name
Model Source
Size
ASR
Whisper-large-w3
openai/whisper-lagee
Large (~1.5B)
Three-stage
NLLB-2200 Ristille-600OM
facehook/allb-222 200p-l---20---ilileddeee 6000
600M
TTS
CosyVoice Voice 2
FunAudioLLLMCooyyyccc22 Voice200.BB
0.5B
Speech Translation
SeamlesMM4T-22LLrg
Large (~2.3BB
Two-stage
TTS
CosyVoice 2
MunuudioLLM/Coayyaaccc2000.
0.5B
Speech-to- Tpeech-to-Teex
Qwen2-audio
Qwen/
7B
Multinodal LLMs
Speech-to to-Speech
Qwen2.5 Omni
Doven/Qwen2.5-00mm
7H
Speech-to-Speee Speechhh
GPT-4o-Audio-Previn
OpenAl Official API
Not disclosed
S28T
SeamlessMAT Medium
Bacebook/xeamles m41 medium
Medium (- (=-1,2
End-to-Ead S2ST
S2ST
SeamlessM4T Large-v2
facebook/seamless westllesss----ttee lart-arg---2
Large (~2.3B) ~1.7B
S2ST
ScamlessExpressive
faceboo//eemleessxxpeessss
this section, we provide detailed descriptions of all baseline models evaluated against our pro- posed UniSS framework. Table B2 summarizes the baselines, model versions, and parameter sizes used in our experiments. All baseline models are evaluated using their default inference parameters without any additional tuning modifications. Evaluation is conducted directly on the CVSS-T and FLEURS test sets without extra data preprocessing, prompts, or output post-processing. We present the implementation specifics of each baseline in the following subsections.
B.2.1 CASCADED SYSTEMS
We implement two cascaded baselines: conventional three-stage baseline (ASR - MT - TTS) and two-stage baseline (S2TT TTS). We select state-of-the-art, widely adopted models each stage to represent strong and practically relevant baselines.
Three-Stage Baseline (Whisper + NLLB + Voice...)). This baseline consists of three sequen- tial modules:
Automatic Speech Recognition (ASR): We select Whisper-large-3,, transformer-based encoder-decoder model trained on large-scale multilingual speech corpora, for its strong multilingual transcription performance and wide adoption in the community.
Machine Translation (MT): We use Meta's
model, multilin- gual transformer trained on parallel corpora covering 200 languages. We select this model for its balance between translation quality and computational efficiency.
16

---

Published as a conference paper at ICLR 2026
Text-to-Speech (TTS):
We employ CosyVoice 2 (0.5B), wansformer-based expressive TTS model designed for high-quality multilingual speech synthesis. Its ability to generate natural and expressive audio makes it suitable for evaluation.
Two-Stage Baseline (SeamlessM4TTv2-LLam Cosy Voice). The two-stage pipeline simplifies the cascaded approach by combining speech recognition and translation into one direct Speech-to- Text Translation (S2TT) module:
Speech-to-Text Text Translation (S2TT): We select Seamless.M4T-v2-Larm multilingual transformer-based model that jointly performs ASR and MT. Its end-to-end design reduces error propagation across stages and offers strong performance in speech translation.
Text-to-Speech (TTS): We generate synthesized speech using the same Cosy Voice 2 model described the three-stage pipeline.
In experiments, observe several limitations associated with cascaded baselines. First, the inference process noticeably slower due to the sequential execution of multiple large models, making the pipeline less suitable for real-time applications. Second, we find that errors introduced in earlier stages, particularly in ASR, tend to propagate downstream, often leading to degraded trans- lation quality unnatural speech synthesis. These limitations highlight the challenges of building efficient and robust cascaded speech translation systems.
B.2.2 MULTIMODAL LARGE LANGUAGE MODELS (MLLM)
We employ three advanced multimodal large language models as baselines: Qwen2-audio, Qwen2.5- and GPT-4o-audio. These models are capable of handling speech input and pro- ducing either textual or speech output, depending on the task.
Qwen2.5-Omní (Speech-to-Speech)) Qwen2.5-Omm is selected for its ability to perform fully end-to-end speech-to-specch translation without requiring intermediate text generation, making promising multimodal LLM baseline for direct S2ST tasks. In our experiments, we find that the model's performance highly sensitive the prompt format. When using a simple default instruction, the model sometimes appends assistant-like phrases (e.g., "Do you need anything else?") to the translated speech, likely due to conversational fine-tuning.
conversation
"role"; "user", "content":
"type": "text",
"text"; "Transiate the following Chinese speech into English."
(
"type": "sudio", "audio"; input_path
}
}
To mitigate this, also experimented with explicit and constrained prompts, such as:
17

---

Published as a conference paper at ICLR 2026
conversation
"role": "user", "content":
"type": "text",
"text"; "You are an expert translator specializing
Chincse-English Interprotation, Your task is to listen to the following Chinese speech and translate it accurately and fluently into spoken Rnglish, Your response ahould:
Preserve the original meaning and tone of the apeaker. Use natural and professional English phrasing. Output only the translated speech without adding any commentary or assistant-like phrases. Maintain appropriate speech rhythm and avoid stretching pausing unnaturally. Please begin
the translation after receiving the audio input."
"type": "audio",
"audio": saput_path
However, we find that such detailed prompts often lead to degraded audio output quality. Specifi- cally, in our experiments, the model occasionally produces abnormal acoustic artifacts, such elon- gating a single word unnaturally producing disfluent prosody. This suggests that Swee22.5-mmm speech generation may be overly sensitive instruction format, and that more verbose prompts can negatively impact generation fluency despite offering better control over semantic behavior.
Qwen2-audio (Speech-to-Text). Qwen2-audio (Chu et al., 2024 is used for direct speech-to- translation. Considering its strong performance on multilingual speech comprehension tasks and its ability to follow structured prompts, we adopt for S2TT evaluation. Prompts are designed to explicitly guide the model to translate input speech into larget-language text. For example:
Translate Chinese into English.
All other parameters kept at their default values. However, using Qwen2-audio requires addi- tional input-output processing. Specifically, all input audio is resampled to 16 kHz and converted to mono channel to meet the model's input requirements. For output handling, filter the returned results to remove prompt-related metadata and retain only the translated text for evaluation purposes.
GPT-4o. For S2ST evaluation, employ 1r1-4--audio-prvieee which supports multimodal speech translation tasks with direct speech-to- to-seeech generation capabilities. Given the instability of Qwen2.5-Omni under complex prompt instructions observed in preliminary experiments, select GPT-4o-audio-previen as more robust alternative. Our experiments confirm that GPT-4o-audio- preview demonstrates superior stability and consistently produces higher-quality outputs compared to previous models.
We guide the model using simple structured prompts such as:
Please translate this English audio into Chinese. Just state the translation itself without any additional explanation polite words.
Although occasionally encounter conversational artifacts, such appended phrases like "Do you need help with anything else?" or generic fallback responses like "I'm sorry, I can't process this audio", the overall translation fluency and speech quality significantly improved compared to
18

---

Published as a conference paper at ICLR 2026
earlier baselines. However, GPT-4o is not open-sourced, and using its API for large-scale translation incurs considerable cost, which poses a practical constraint in real-world deployment scenarios.
B.2.3 END-TO-END SPEECH- TO- SOEEEC MODELS
For direct end-to-end baselines, we evaluate several state-of-the-ar S2ST systems that translate input audio directly into target-language speech, bypassing intermediate textual representations. Specifi- cally, we include SeT Medium, SeamlessM4T Large-v2, and Eearsesse in our evaluation.
SeT Medium and Large-v2. SeamlessM4T Medium and Large-v2 multilingual speech translation models that support speech-to-speede translation without explicit intermediate transcription synthesis stages. These models adopt a unified encoder-decoder architecture and designed to support both low-resource and high-resource language pairs. In our experiments, we use the official inference pipelines provided by Meta with default decoding parameters. No additional prompt tuning, input preprocessing, or output filtering is applied.
GramlessExpressivee SeamlessExpressive extends SeamlessM4T by introducing expressiveness- aware modeling, enabling more natural and emotionally rich speech synthesis the target language. The model conditions generation prosodic and expressive cues from the source speech and is particularly well-suited for conversational and affective scenarios. We evaluate SeamlessExpressive using its official inference scripts and default hyperparameters.
Seed Livelnterpret 2.0.
Seed Livelnterpret 2.0 an enterprise-level simultancous interpretation model that performs real-time speech-to-speech translation with voice cloning capabilities. The model employs duplex framework that processes input audio and generates target-languant speech directly without intermediate text representations. In our evaluation, we used the official inference API with default parameters.
Despite the capability to perform direct speech-to-spect translation, current end-to-end models still face notable limitations in audio quality. our experiments, the generated speech often contained audible artifacts or background noise, especially for longer noisier input segments. While Seam- lessExpressive improves upon previous models by attempting to clone speaker identity, prosody, and expressiveness, its expressive fidelity remains limited. The cloned voice frequently lacks the richness and nuance natural human speech, and variations in emotion or emphasis are not always faithfully reproduced.
C EVALUATION METRICS
C.I OBJECTIVE EVALUATION
To assess the performance of speech-to-speech translation (S2ST), we report a range of objective metries covering translation accuracy, speaker identity preservation, prosodic alignment, temporal consistency, and speech quality. For all metrics, higher values indicate better performance.
Text-BLEU evaluates fidelity of generated translations by computing corpus-level BLEU scores using the SacreBLEU library (Posi) 2018). Before scoring, we apply language-specifie preprocessing: English text is lowercased and stripped of punctuation (excluding upostrophes), while Chinese text is normalized to simplified characters, punc- tuation is removed, and characters are separated by spaces. This ensures consistency with standard BLEU evaluation practices. We use the corpus BLEU score across the whole dataset. Chinese samples are scored in 'zh' mode.
function calculate the
Speech-BLEU, ASR-BLEU, evaluates the translation fidelity of speech-to-speech trans- lation systems. We transcribe the generated speech using ASR models, Whisper Whisper-large- large-v3 v3 for English and Paraformer-zh for Chinese, and compute the BLEU score between the tran- scribed output and the ground-un reference. Preprocessing follows the same pipeline as Text-BL.EU.
19

---

Published as a conference paper at ICLR 2026
AutoPCP is automatic implementation of the Prosodic Consistency Protocol, designed to evaluate how well the prosodic characteristics of the source speech-suue such as pitch con- tours, energy dynamics, and temporal structure- are preserved in the translated output. It extracts these features from both the input (source) and output (translated) audio and computes their similarity. By comparing the prosody between these signals, AutoPCP provides guantitative measure of rhythm and speaking style consistency.
SLC-0.2 and SLC-0.4 (Speech Length Compliance) assess the temporal consistency be- tween the source and translated audio by measuring whether the output duration falls within +20% or +40% of the original speech length. These metrics are important in real-world scenarios such as simultaneous interpretation and audiovisual dubbing, where maintaining similar speech pacing and timing is essential for naturalness, synchronization, and listener comprehension.
UTMOS is neural network-based estimator that predicts the Mean Opinion Score (MOS) for synthesized speech, serving as proxy for human judgment of audio quality. It produces scalar value typically ranging from 1 to 5, reflecting aspects such as naturalness, fluency, and intelligibility. This objective metric helps mitigate the subjectivity and variability of human MOS ratings, enabling reproducible and scalable quality assessment.
C.2 SUBJECTIVE EVALUATION
To comprehensively evaluate expressive qualities, we conducted a Mean Opinion Score (MOS) lis- tening study. Six bilingual speakers rated synthesized speech on a 5-point scale across three di- mensions: emotion similarity (Emo Sim.), speaker similarity (Spk Sim.), and speech naturalness (Naturalness). Similarity is compared against the original audio instead of the synthesized audio.
We implemented the MOS evaluation using webMUSHRA (Schoeifler et al., 2018), a MUSHRA- compliant web-based audio evaluation framework that facilitates controlled listening experiments. The platform provided key features for audio assessment, including seamless audio switching and Likert scale questionnaires. Experimental configurations were defined through YAML files, with results automatically exported as CSV files. The user interface is shown in Figure C2
In emotion similarity and speaker similarity tasks, the scale options are: "1. Definitely Different", "2. Probably Different". "3. Not be sure", "4. Probably the same", "5. Definitely the same". In speech naturalness task, the scale options are: "1. Definitely unnatural", "2. Probably unnatural", "3. Not be sure", "4. Probably natural", "5. Definitely natural".
Planeree Emateer Unalurity Guster deriarity weatenateeeeeee "Megr" compast ands
FAU
Figure C2: User interface of the webMUSHRA-based MOS evaluation system showing the three assessment dimensions: emotion similarity, speaker similarity, and speech naturalness.
20

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTAL RESULTS
We also evaluate the Chinese and English test subsets of FLEURS (Conneau et al. 2022), massively multilingual speech dataset spanning 102 languages with n-way parallel speech and text data. UniSS demonstrates robust performance on this challenging multilingual benchmark across multiple evaluation tasks. Additionally, UniSS achieves sute-of-the--e performance speech-to- text translation (S2TT) and competitive results on automatic speech recognition (ASR) and text-to- speech synthesis (TTS) tasks.
D.1 S2ST PERFORMANCE ON FLEURS
To verify the robustness of our framework across datasets, we further evaluate performance on the FLEURS set. FLEURS is a multilingual benchmark derived from the FLoRes corpus that pro- vides high-quality parallel speech and text pairs across diverse languages, making it a valuable complement CVSS-T for assessing speech translation systems under standardized multilingual conditions.
Table D3: Main comparison results on the Fleurs dataset. Results presented as EN-ZH ZH-EN. Higher scores indicate better performance. '-' denotes unavailable results. Best scores are in bold and second-best scores underlined.
Calegory
Model
SSize
Speech- BLEU
Text-BLEU
A.PCP
SLC
SLC 0.4
UTMOS
3-Stage-MII
2.68
22.00
21.49
24.56
22.09
2.86
2.73
0.27
10.37
01.8
0.60
2.14
3.82
Cascaded
3-Stage-LLM
3.5B
30.57
18.45
33.78
19.14
2.98
2.73
0.50
0.41
0.78
0.64
2.13
3.82
2-Stage
2.88
31.42
23.01
34.72
23.67
2.94
0.46
0.61
2.13
3.83
GPT-4o
38.58
23.21
2.72
2.39
0.45
1042
0.71
3.36
4.25
MLLM
Qwen2.5-O
7B
3.37
24.20
37.76
25.15
1.93
2.21
0.07
0.33
0.27
0..66
3.05
4.33
Seamless-M
1,2B
14.10
13.30
27.92
17.71
2.19
2.10
0.24
0.15
0.47
2.64
3.52
Seamless-L
2.3B
28.86
23.84
32.99
24.06
2.18
2.06
0.42
0.14
0.66
0.32
2.42
3.90
Seamless- Ex
1.78
18.80
18.88
34.53
18.89
2.64
2.64
0.69
0.32
0.88
0.58
2.06
3.19
S2ST
UmIS (P)
1.5
22.42
36.12
22.71
2.72
2.65
0.95
0075
0.99
3.40
4.15
UniSS (Q)
1.58
36.16
23.06
36.96
23.37
2.72
2.64
0.95
0.76
0.99
0.93
3.41
4.16
Translation Fidelity. As shown in Table D3 UniSS (Q) achieves strong translation performance with Speech-BLEU of 36.16 16 and Text 36.96 on EN-ZH, outperforming all S2ST baselines and cascaded systems. UniSS (P) also demonstrates competitive performance across both metrics. In the ZH-EN direction, UniSS surpasses all systems with comparable model size, demonstrating the parameter efficiency of our design.
Prosody Preservation. UniSS exhibits excellent prosody transfer capabilities on the FLEURS benchmark. UniSS (Q) achieves A.PCP scores of 2.72 (EN-ZH) and 2.64 (ZH-EN), outperform- ing all end-to-end baselines and matching GPT-4o's performance (2.72 and 2.39). Notably, UniSS achieves these results without incorporating dedicated prosody modeling modules, highlighting the effectiveness of our unified framework.
Duration Consistency. Our system demonstrates exceptional speech duration consistency on the FLEURS test set. Both UniSS variants achieve near-perfect scores on the SI.C-0.2 and SI.C-0.4 metrics, indicating that the generated speech closely mirrors the temporal length of the input au- dio. This precise duration alignment is critical for real-time applications and audiovisual translation scenarios where timing mismatches can disrupt user experience.
Speech Quality. UniSS achieves state-of-the-art speech quality among S2ST models. In the EN- ZH direction, UniSS (Q) attains UTMOS score of 3.41, outperforming all open-source baselines and surpassing GPT-4o (3.36). UniSS (Q) also leads all S2ST models in the ZH-EN direction with a score of 4.16 16. Despite its significantly smaller size. UniSS consistently produces high-qualin speech across both translation directions, confirming its effectiveness as a practical open spurcsource solution for speech-to-speech translation.
21

---

Published as a conference paper at ICLR 2026
Table D4: Performance comparison across different speech tasks. For S2TT, results reported as EN-ZH ZH-EN BLEU scores. Higher Text-BLEU and lower WER indicate better performance.
Task
Dataset
Model
Metric
Result
Qwen2-audio
Text-BLEU
21.86
18.69
CVSS-T
Whisper
Text-BLEU f
12.34
UniSS
Text-BLEU f
33.17
27.33
S2TT
Qwen2-audio
Bext-BLEU
6.81
17.79
FLEURS
Whisper
15.65
UniSS
Text-BLEU
34.91
22.18
Whisper
WER
1.8
LibriSpeech test-clean
UniSS
WER
2.4
ASR
Whisper
WER
3.6
LibriSpeech test-others
UniSS
WER
6.6
SparkTTS
WER 4 SIM 1
1.98
0.584
TTS
LibriSpeech test-clean
UniSS
WER 4
SIM f
4.75
0.568
D.2 COMPARISON WITH S2TT SYSTEMS
Beyond speech-to-speed translation, UniSS also supports speech-to-text translation (S2TT). As shown Table[D] UniSS achieves leading performance on this task as well. Whisper only supports X-EN translation and demonstrates poor performance on multilingual scenarios. UniSS's Text- BLEU scores significantly outperform the 7B multimodal LLM Qwen2-audio across both datasets, demonstrating superior cross-lingual speech understanding capabilities.
D.3 ASR PERFORMANCE
Although UniSS was not specifically designed for automatic speech recognition, demonstrates competitive ASR capabilities shown in Table D4 We evaluate the ASR performance on the LibriSpeech dataset [Panayotov et al., 2015). On the LibriSpeech test-clean subset, UniSS achieves WER of 2.4, which comparable to the open-source state-of-the-arl ASR model, Whisper (1.8 WER). However, on the more challenging LibriSpeech test-others subset, UniSS shows degraded performance with WER of 6.6. It is important to note that UniSS is primarily designed speech-to-speech translation model, and these ASR results demonstrate its versatility beyond its core translation capabilities.
D.4 TTS PERFORMANCE
Although UniSS was not specifically designed for text-to-speech synthesis, we evaluate its TTS capabilities the LibriSpeech test-clean dataset shown in Table UniSS achieves a WER of 4.75 and speaker similarity of 0,568, compared to SparkTTS, which obtains 1.98 WER and 0.584 similarity. The performance gap is expected given that SparkTTS is specifically optimized for TTS, whereas UniSS is primarily designed for speech-to-spece translation. These results demonstrate that UniSS can generate intelligible speech output.
E UNIST AND CONSTRUCTION PIPELINE
E. SOURCE DATASET
We select diverse collection of publicly available Chinese and English speech datasets the foundation for UniSS training data. Our data sources encompass multiple domains and speaking styles to ensure robust cross-lingual speech translation capabilities.
22

---

Published as a conference paper at ICLR 2026
Below is the list of source corpora used:
AISHELL-3 Shi et al. 2021: A multi-speaker Mandarin speech corpus for text-to-speech
synthesis.
CoVoST2 (Wang et al. from Common Voice.
2020): A multilingual speech-to-text translation corpus derived
Common Voice (EN & ZH) (Ardila ct al., 2020): A large-scale multilingual speech corpus with speaker metadata. We use Common Voice 4.0.
CVSS-T (Jia et al. 2022b): A benchmark dataset for speech-to-speech translation in
English-Chinese language pairs.
Dailytalk (Leee et al. tional style recordings.
2023): A multi-speaker English speech corpus featuring conversa-
Emilia (He al. 2024): A multi-speaakr multilingual speech corpus covering six lan- guages.
FLEURS (Conneau al.
2022): multilingual benchmark derived from FLoRes,
taining parallel speech and text data.
Gigaspeech (Chen al. 2021): large-scale multi-speaker English reading corpus.
Hi-Fi TTS (Bakhturina et al.
2021): A multi-speaker English corpus with high-fidelity
recordings.
HIQ-oovvessations (Magic Data Technology Co. versational dataset.
2024): A multi-speaker Mandarin
LibriSpeech (Panayotov et al. audiobooks for TTS.
2015): A multi-speaker English speech corpus with reading
LibriTTS-R (Koizumi al. text-to-speech synthesis.
2023): A high-quality version of the LibriTTS corpus for
MAGICDATA (Magic Data Technology Co.
2019): A conversational Mandarin Chinese
spoken corpus.
NCSSD-C and NCSSD-R (Liu al. synthesis.
2024):
Multi-speaker bilingual datasets for speech
VCTK (Veaux et al. 2017): A multi-speaker English corpus featuring diverse regional accents.
WenetSpeech4TTS Ma et al. 2024 A large-scale Mandarin corpus for text-to-speech
synthesis.
E.2 TEXT TRANSLATION STEP
After cleaning the source dataset, we employ Doven-2.5-7BB-nnsttee as our translation engine to translate source transcriptions into the target language. For source language Chinese audio, we use the following prompt:
P Please transiate the tollowing text into Engiish (Note that, aside from the translation, other responses or explanations should be provided.):
For source language English audio, use the corresponding Chinese version of this prompt to ensure consistent translation quality across language directions.
Following the translation process, we clean the translated text based on the reply format characteris-
through several steps: (1) removing common prefixes such as "Sure, here is the translation:" and its corresponding Chinese version that are automatically generated by the model; (2) eliminating explanatory notes and comments (e.g.. "Note:" and corresponding Chinese version) that often accompany the translations; (3) normalizing text formatting by remov- ing line breaks and excessive whitespace characters; and (4) filtering out mixed-languaal content to ensure linguistic purity, where translations containing both Chinese and English characters are discarded invalid outputs.
tics of Qwwen-255722B-Innse
23

---

Published as a conference paper at ICLR 2026
E.3 ANALYSIS OF UNIST
We conduct subjective assessment on randomly sampled subset of UniST to evaluate the quality of our dataset. Four raters listened to 150 examples and provided MOS scores across three dimen- sions: Emotion Similarity, Speaker Similarity, and Speech Naturalness. As shown in Table [S UniST synthesized by SparkTTS achieves high speech naturalness and effectively preserves both the source speaker's voice characteristics and emotional expressiveness.
Table E5: Subjective MOS evaluation on the UniST General dataset.
Dataset
Emo Sim.t
Spk Sim.t Naturalnesst
UniST General
4.69
4.31
4.61
F USE OF LLMS
Large language models are only used to polish the writing of this article without making any modi- fications the content, data, or other elements. After applying LLM polishing, the authors consider the LLM outputs rather than directly copying them.
G FREQUENTLY ASKED QUESTIONS
G.1 WHY USE SPARK-TTS TO SYNTHESIZE THE DATASET?
Although Spark-TTS does not achieve state-of-the-art performance in objective evaluations of speaker similarity, we observe that its performance in subjective evaluations is comparable. Fur- thermore, assume that its voice decoupling capability reduces the modeling complexity for large language models, which is why we deploy BiCodec for the generation speech tokenizer in the LLM. Additionally, Spark-TTS generates BiCodec tokens directly, making training more efficient by elim- inating an extra encoding step.
G.2 DOES THE PROGRESSIVE TRAINING STRATEGY INCREASE MODEL. COMPLEXITY?
The progressive training strategy pertains only to the training process and does not increase the model's 's structural complexity. The alignment in phase existing, easily accessible open-source data. In the ablation study section, we demonstrate the impact different stages on model perfor- mance, proving that progressive training is effective.
G.3 WHY DEPLOY THREE DIFFERENT SPEECH TOKENIZERS?
The speaker tokenizer is used to represent global speaker information. We hypothesize that this voice decoupling reduces the difficulty of speech translation by allowing the model to focus more on the linguistic content. We employ different linguistic and semantic tokenizers for understanding and generation to optimize each task. To generate more natural and expressive audio, the gener- ation process uses semantic tokens, which retain information beyond the core content. However, this additional information makes content understanding more challenging. Therefore, the linguistic tokenizer, which focuses solely on content information, is better suited for the speech understand- ing phase. As simple speech representation method, we believe this design does not introduce significant complexity in either training or inference.
H REVISION LOG
Specifically, we have included the revision log here with the date format of dd/mm//yyy (Anywhere on Earth). We welcome any suggestions that may contribute to the improvement of this paper.
24

---

Published as a conference paper at ICLR 2026
H.1 UNISS-L.1: 22/1
1. Section 5.2.3 Add a new baseline 3-Stage-LLM in cascaded systems. The original 3-stage method renamed as 3-Stage-MT.
2. Table Table D3 Add the results of new baseline 3- S-tage---LMM which performs better than 3-Stage-MT. However, it does not affect the comparison results between UniSS and cascaded systems.
3. Table 2 Add MOS results for Qwen2.5-O to demonstrate the current performance of MILMS on the S2ST task.
25


Reviews:
[{"review_id": "NYEuDCNjso", "reviewer": "Reviewer_yrvR", "summary": "The paper introduces UniSS, a single-stage autoregressive framework for expressive speech-to-speech translation (S2ST) that preserves speaker voice, emotion, and duration while achieving high translation fidelity. Built on the pre-trained Qwen2.5-1.5B-Instruct LLM, it uses a triple-tokenizer strategy (speaker, linguistic, semantic tokens) and cross-modal CoT prompting to transfer text translation capabilities to speech. Two modes (Quality and Performance) balance fidelity and efficiency. The authors release UniST, a 44.8k-hour Chinese-English expressive S2ST dataset. Experiments on CVSS-T and FLEURS show superior performance over baselines like SeamlessM4T, GPT-4o, and cascaded systems in metrics such as Speech-BLEU, prosody consistency, duration compliance, and speech quality. Subjective MOS evaluations confirm high emotion/speaker similarity and naturalness.", "questions": "1. To make a fair comparison, it will be better if you test another cascaded baseline that uses the same Qwen-2.5-1.5B-Instruct model for MT and the same SparkTTS model for TTS. Besides, consider training the baselines on the UniST dataset. This would show if the UniSS framework is truly better, or if it just benefited from better components and training data.\n\n2. How does the cross-modal CoT prompting specifically bridge text and speech modalities? Provide some studies of intermediate outputs (e.g., T_src, T_tgt) to explain whether UniSS addresses error propagation problems.\n\n3. The motivation emphasizes unifying expressiveness, but much relies on existing tools (BiCodec, SparkTTS). What novel insights does UniSS offer for LLM-based S2ST, and how does it differ from prior CoT adaptations in multimodal tasks?\n\n4. For training strategy, what is the individual contribution of each task to overall performance? Could you add ablations removing or weighting specific tasks to quantify their effects?\n\n5. Since UniST is fully a synthetic dataset generated by SparkTTS, is it unable to effectively capture word-level semantic prosody mapping or relevance in speech translation? For instance, the transfer of semantic nuances such as emphasized stress. It is recommended to provide corresponding analysis and discuss the potential limitations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important problem in speech-to-speech translation: preserving the speaker's voice, emotional style, and prosody across languages. This is crucial for applications like multilingual communication or media dubbing. The motivation is clear, and the cross-modal CoT approach effectively aligns text-based LLM capabilities with speech in a single model. This method is innovative and necessary, as ablations show that removing CoT leads to major drops in performance.\n\n2. The authors release UniST, a 44.8k-hour dataset for expressive S2ST, which tackles the key issue of limited paired data with style preservation. The creation pipeline is well-documented, making it valuable for the research community.\n\n3. The experiments cover a broad set of metrics, including semantic and acoustic part. Baselines are varied, spanning cascaded systems and end-to-end models, ensuring robust comparisons.\n\n4. UniSS outperforms baselines on most metrics, such as emotion similarity and duration matching. The demopage also illustrates the model's success in maintaining expressive qualities.\n\n5. The paper is well-structured and readable, with clear descriptions of the architecture, training strategy, and ablations that validate the approach.", "weaknesses": "1. The novelty is limited. The model's strength in maintaining speaker voice, emotion, and prosody largely relies on pre-existing components, such as the BiCodec tokenizer for speaker representations and SparkTTS for synthesizing expressive data. UniSS itself mainly integrates these by organizing a multiple token sequence then inputing it into a LLM, reducing its core innovation and novelty.\n\n2. The \"Single-Stage\" claim is misleading.  While positioned as a streamlined single-stage model to avoid cascaded errors and latency, the Quality mode generates intermediate source text (T_src) and target text (T_tgt) before speech tokens, mirroring the sequential steps of a traditional 3-stage pipeline (ASR → MT → TTS). This undermines the claimed advantages and could lead to similar error accumulations and delays.\n\n3. Training and evaluations are centered on English-Chinese pairs. Besides, the use of GLM tokenizer and SparkTTS also constrains the method for applying on other languages. This raises questions about scalability to diverse languages, accents, or noisy real-world scenarios, where data scarcity for expressiveness could amplify issues.\n\n4. A significant issue is the unfair setting in evaluations, particularly for translation accuracy (BLEU). The UniST training data uses high-quality translations from the large Qwen2.5-72B model, while the 3-stage cascaded baseline relies on the much smaller pretrained NLLB-600M for machine translation. Why not use Qwen-2.5-1.5B-Instruct LLM in the cascaded system?\n\n5. The framework's heavy reliance on text-based elements. For example, CoT prompting with intermediate transcripts makes it unsuitable for unwritten or low-resource languages without scripts. This contradicts goals in textless S2ST research and should be highlighted as a key limitation, especially given the focus on expressive translation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NenGM9iOb6", "reviewer": "Reviewer_41XM", "summary": "The paper presents **UniSS**, a unified single-stage speech-to-speech translation (S2ST) system designed to translate spoken content while preserving both speaker identity and expressive emotional style. UniSS eliminates multi-stage pipelines by leveraging a pretrained large language model (LLM) backbone, incorporating speech and text with a shared token vocabulary, and introducing cross-modal chain-of-thought (CoT) prompting that transfers text translation capabilities to the speech domain. Besides model proposals, the authors construct UniST, a large-scale expressive S2ST dataset (44.8k hours). Experimental results on several benchmarks demonstrate substantial improvements in translation fidelity, expressiveness, and runtime efficiency versus established baselines.", "questions": "1. In Phase 1, is the TTS task really useful? After all, your ultimate goal is translating source speech to target speech.\n\n2. How reliable is the metric A.PCP?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. UniSS is a conceptually clean, single-model approach that collapses ASR, MT, and TTS into one autoregressive LLM with expanded vocabulary for both text and discrete speech tokens.\n2. The construction of UniST (44.8k hours) as a high-quality, diverse Chinese-English corpus with explicit control over speaker, emotion, and timing, is a notable resource for the field.", "weaknesses": "1. The subjective evaluation does not include MLLMs such as GPT-4o, and the model’s performance is also inferior to Seed Live or Seamless-Ex on certain metrics.\n\n2. I don’t think the so-called cross-modal prompting can be regarded as a form of Chain of Thought (CoT).\n\n\n3. Despite its claim of being “unified,” UniSS relies on three distinct speech tokenizers (speaker, linguistic, and semantic), each originating from separate models. This design increases the vocabulary size (over 180k tokens) and may limit scalability, efficiency, or ease of deployment when supporting additional languages or larger vocabularies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j8PZwgosc7", "reviewer": "Reviewer_9Wh8", "summary": "This work introduces UniSS, a novel single-stage framework for expressive speech-to-speech translation (S2ST) that addresses key challenges such as the scarcity of expressive paired speech data, the complexity of multi-stage pipelines, and the limited transfer of translation capabilities from large language models. UniSS integrates speech semantics and style modeling through cross-modal chain-of-thought prompting, transitioning from text-only models to text-to-speech generation. Opensourced 44.8k hours of high-quality expressive S2ST data, UniSS achieves substantial improvements in translation fidelity and speech quality, while effectively preserving voice, emotion, and duration consistency compared to prior approaches.", "questions": "- Is the prompt used during multi-task training also required at inference for different tasks? If so, how would it impact the available context length?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Introduces a single-stage autoregressive model that directly builds on a pre-trained textual LLM, simplifying prior multi-stage architectures for expressive speech-to-speech translation.\n- Provides an open-source, reproducible pipeline for constructing expressive S2ST datasets, including UniST, a large-scale Chinese-English corpus preserving speaker identity and emotional style.\n- Delivers a thorough evaluation combining objective metrics (Speech/Text-BLEU, A.PCP, SLC, UTMOS) with subjective MOS assessments of emotion, speaker similarity, and naturalness.", "weaknesses": "- Limited language coverage: Supports only Chinese–English translation.\n- Ununified tokenizer: Using separate tokenizers for linguistic and semantic tokens raises concerns about vocabulary size and token consistency.\n- Language model considerations: Employing a large LLM (Qwen2.5-1.5B-Instruct) for translation may be excessive compared to traditional MT models; more efficient parameter training could better leverage other model capacities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)