FlexiVoice: Enabling Flexible Style Control in Zero-Shot TTS with Natural Language Instructions
Authors: Dekun Chen
Xueyao Zhang
Yuancheng Wang
Kenan Dai
Li Ma
Zhizheng Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Instruction TTS
Controllable Speech Synthesis
Decoupling
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This study proposes FlexiVoice, a text-to-speech (TTS) synthesis system capable of flexible style control with zero-shot voice cloning. The speaking style is controlled by a natural-language instruction and the voice timbre is provided by a speech reference in zero-shot manner. FlexiVoice is built with an LLM core, which takes text as input, and also takes an optional natural language instruction and an optional speech reference to control style and timbre, respectively. FlexiVoice is equipped with a novel Progressive Post-Training (PPT) scheme that progressively unlocks accurate and flexible controllability. In particular, it first employs Direct Preference Optimization (DPO) to enable FlexiVoice to accurately follow both natural language instruction and speech reference simultaneously. It then uses a multi-objective Group Relative Policy Optimization (GRPO) to disentangle style instruction, reference timbre, and textual content. Finally, it adapts instruction GRPO for more advanced instruction following. Experimental results show that FlexiVoice surpasses competing baselines and demonstrates strong capability in decoupling control factors. Human evaluations further confirm its naturalness, controllability, and robustness.
Audio samples are available at https://flexi-voice.github.io/.

Published as a conference paper at ICLR 2026
FLEXIVOICE: ENABLING FLEXIBLE STYLE CONTROL IN ZERO-SHOT TTS WITH NATURAL LANGUAGE IN- STRUCTIONS
Dekun Chen', Xueyao Zhang', Yuancheng Wang', Kenan Dai Li Ma², Zhizheng Wu1343 'he Chinese University of Hong Kong, Shenzhen Huawei Technologies Co., Ltd 3 Shenzhen Loop Area Institute *City University of Macau AAmphion Technology Co., Ltd.
ABSTRACT
This study proposes FlexiVoice, text-to-speech (TTS) synthesis system capa- ble of flexible style control with zero-shot voice cloning. The speaking style is controlled by a natural-language instruction and the voice timbre is provided by speech reference in zero-shot manner. FlexiVoice is built with an LLM core, which takes text as input, and also takes an optional natural language instruc- tion and optional speech reference control style and timbre, respectively. FlexiVoice is equipped with a novel Progressive Post-Tran Training (PPT) scheme that progressively unlocks accurate and flexible controllability. In particular, it first employs Direct Preference Optimization (DPO) to enable Flexi Voice to ac- curately follow both natural language instruction and speech reference simultane- ously. It then uses a multi-objective Group Relative Policy Optimization (GRPO) disentangle style instruction, reference timbre, and textual content. Finally, it adapts instruction GRPO for more advanced instruction following. Experimen- tal results show that FlexiVoice surpasses competing baselines and demonstrates strong capability in decoupling control factors. Human evaluations further con- firm its naturalness, controllability, and robustness. Audio samples are available at https://feeiivviieegihbbbbb
INTRODUCTION
DC Instruction
Milpy Reference
Text
FlexiVoice
www. quee))
Pre-Training
Progressive Post Deaii-Taaning
ST: Mulli-Modally DPO
S2: Decoupling GRPO
S2: Instruction ORPO
Irsrucionn de Spocch Paired training data
Phoditiona
Compatoree
Year
(2)
T
L
LL M-Mesaisted
Emottin Rewsrd
Per
Annotation
Lows
Speaker Reward
Melsmroo
Plaard medk's
Zyos
Zeep
[Z)
C3]
Consietency
Prere pwwfencce dale
Reseurt pconea
Rinard nechls
An overview of FlexiVoice that supports diverse style generation with arbitrary voice timbres. It takes an optional natural language instruction for style and an optional reference speech for timbre. It consists of Pre- Pre-Training and Progressive Post- trainTrainnn (PPT) stages. The PPT process includes three processes, S1: Multi-modality DPO, S2: Decoupling GRPO, S3: Instruction GRPO.
Figure 1:

---

Published as a conference paper at ICLR 2026
Recent advances in text-to-speech (TTS) have been largely driven by the emergence of Large Lan- guage Models (LLMs) and refined post-training techniques. A notable breakthrough is zero-shot TTS (Du et al. 2024 Zhou et al.] 2025), which enables voice cloning with only short reference speech, effectively capturing and reproducing a speaker's timbre. Beyond timbre, controlling speak- ing style has become an important challenge. One direction, exemplified by Vevo (Zhang et al.) 2025c) and IndexTTST (Zhou et al., (025), employs two separate speech references to control tim- bre and style separately. Another line of work, instruction-based TTS (Vyas al. 2023) Zhou al., 2024b, Ji et al. 2024b), leverages natural language instructions to specify the target style. How- ever, existing instruction-drivedt models often struggle either to faithfully follow the instructions or to maintain stable timbre consistency.
Achieving flexible style control zero-shot TTS presents unique challenge: the 'Style-Timbre- Content Conflict'. In standard supervised training, models tend to over-rely on the strong acoustic priors from the reference speech (timbre leakage) or infer prosody from the text (content leakage), often ignoring the explicit style instruction. Merely applying instruction conditioning is insufficient to resolve these entangled modalities. Therefore, a robust framework is required not just to con- dition the model, but to actively decouple these factors and enforce instruction adherence against conflicting acoustic cues.
In this work, propose FlexiVoice, TTS system that can flexibly control the speaking style with a zero-shot voice. In particular, can take a natural language instruction and a reference speech of them for flexible controllability. The natural language instruction aims control speak- ing styles (e.g. emotion, speaking speed) and the speech reference is to control timbre for speaker identity. FlexiVoice is built on top of a pre-trained large language model (LLM). The LLM core equips FlexiVoice with a robust and comprehensive instruction-followii ble controllability, we first construct a large-scal scale and diverse speech dataset with natural language instructions. named FlexiVoice-Instruct The Flexi Voice- Instrect dataset is annotated with the help of L.M.. We then pre-train Vlexe with Emilia (He et al., 2024) and Flexi Voice-Instruce In post-training, we propose a novel Progressive Post- post-Training (PPT) framework. Unlike general LLM aligmment, PPT is specifically designed to resolve the modality conflicts TTS through a systematic curriculum. It consists of three stages: (I) Multi-modality DPO establishes the initial alignment for instruction adherence; (2) Decoupling GRPO introduces a multi-o multi-objectivv optimiza- tion with conflicting data scenarios to mathematically enforce the separation of style from timbre and content; and (3) Instruction GRPO leverages an audio-language model (ALM) reward to gen- eralize this capability complex, open-ended instructions. This progressive formulation transforms the instruction-followiin task from simple conditioning into a rigorous disentanglement process.
ability. To achieve flexi-
We evaluate FlexiVoice from flexible controllability and instruction-following ability aspects, using emotion datasets and the InstructTTSEval (Huang et al. (2025) benchmark. The experimental results show that Flexi can decouple speaking style (e.g. emotion) and speaker identity. In comparison with baselines, it achieves large gains in instruction adherence and robustness on the multi-modality control evaluation, and demonstrates strong performance on complex instruction tasks. Human evaluation results further confirm the naturalness and robustness of the generated speech.
The contributions are summarized as follows:
We propose FlexiVoice, a TTS system that enables flexible and precise style control for zero- shot voice synthesis. The speaking style is controlled by an optional natural language instruc- tion and the timbre is guided by an optional reference speech. It enables any combination of style and timbre for flexible control.
We develop a large-scale and diverse speech dataset with natural language instructions, FlexiVoice-Instruct The annotations are created using an LLM (i.e. Deepseek-V3 (Liu al.) 2024a)) based on the speech properties. It has a broad coverage of human-like instructions, including expressive scenarios.
We propose a Progressive Post-Training (PPT) framework, a novel curriculum learning ap- proach tailored for controllable TTS. By strategically sequencing preference alignment and multi-objective optimization, PPT explicitly solves the Style-Timbre-Cae conflict, achiev- ing robust disentanglement that standard training paradigms fail to handle.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Controllable Speech Synthesis Research on controllable speech synthesis has mainly followed Iwo directions. (i) Zero-shot TTS clones timbre from short reference speech (Chen et al,, 2024
Wang al. 2024 Zhou et al. 2025 Zhang al., 2025c), while partially controlling speak- ing style. (ii) Instruction based TTS uses natural llnguuag prompts to specify style; for instance, PromptTTS (Guo al.) 2023) and PromptStyle (Liu al. 2023) enable text-guided style control but within limited spaces, and Parler-TTTS (Lyth & King (024) scales conditioning with weak la- bels without supporting timbre control. More recent systems, such as VoxInstruct (Zhou et al. 2024b), AudioBox (Vyas et al. 2023), ControlSpeech (Ji et al. 2024b). and Cosy Voice2 (Du et al, 2024). combine instruction with reference speech, but still lack robust disentanglement and broad style diversity. These limitations highlight the need for unified framework that processes multi- modal inputs, generates speech following cnerucctoo-definee style, preserves timbre, and explicitly addresses disentanglement.
Instruction-Speech Dataset Instruction- TTS has driven datasets that couple text descrip- tions with speech. TextrolSpeech (Ji et al. 2024a) introduced large prompt-speech pairs with five style factors; Audiobox (Vyas et al. 2023) broadened the paradigm to multi-modal audio genera- tion, though its lack of public release limited accessibility. Parler-TTS (Lyth & King 2024) scaled weak labels for speaker/ssyle/connitions to tens of thousands of hours. SpeechCrali (Jin et al. 2024) and ParaSpeechCaps (Diwan al. 2025) enriched granularity with automatic captioning and di- verse paralinguistic attributes. Nonetheless, most corpora come from homogeneous sources and emphasize templated descriptions, leaving insufficient coverage of natural, diverse instructions. Our dataset targets higher-qualite more natural annotations to strengthen instruction-following
Reinforcement Learning in Speech Synthesis Reinforcement learning has recently been plored to improve controllability in speech synthesis. INTP (Zhang et al. 2025a) applies DPO to challenging zero-shot cases for better intelligibility. Emo-DPO (Gao al. 2025) extends preference alignment to emotional control. Vevo2 iZhang et al., 2025b) employs obbii-oeeective post-training to jointly enhance intelligibility and prosody across both speech and singing. These works highlight the promise of RL-based alignment for targeted aspects of TTS. In contrast, our approach adopts a progressive curriculum that leverages reinforcement learning to explicitly address modality disen- tanglement and complex instruction following, thereby enabling broader controllability in zero-shot multi-modality TTS.
3 OVERVIEW OF FLEXIVOICE
Vlixi is built on top of an LLM similar other recent TTS systems (Du et al. 2024 Zhou al., 2025). In Voice, a speech tokenizer converts speech into discrete tokens as inputs and outputs of an LLM. The LLM core processes the input text, Latural- instruction, and reference tokens to generate discrete speech tokens. The generated tokens are transformed into Mel-spectrogram features via flow matching (Lipman al., 2023). and finally converted into waveform audio with vocoder. A detailed description is provided in Appendix [A.] Voice is first pre-trained on large-scale dataset as FlexiVoice-Baas and then post-trained using a progressive strategy.
3.1 PRE-TRAINING
FlexiVoice-BBe is pre-trained model of FlexiVoice. Is pre-trained using Emilia (He et al. 2024 and diverse set of instruction speech dataset as listed in Appendix A.2
Emilia (He et al.) 2024) is large-scale, multilingual, and diverse speech dataset that covers a broad spectrum of speaking styles and scenarios, and primarily supports the fundamental ability of speech generation. To further enable cnerructionguuider TTS, we construct a high-quality and diverse dataset, TlexiVoice-Instruee consisting of natural-language instructions across various sce- narios. The processing pipeline is detailed in Section 4 In addition, we enrich the pre-training phase with existing instruction mnstrcctionsspechh corpora (Diwan et al. 2025) and specialized resources featuring at- tributes such emotion, age, and debate, with processing details provided in Appendix A.2 We also incorporate NVSpeech (Liao et al., (025). which provides paralinguistic tags and expressive coverage. Together, these resources form the pre-training corpus for our model.
3

---

Published as a conference paper at ICLR 2026
Following the traditional pre-training strategy for TTS systems (Du et al. 2024 Zhou al. 2025 we only train the LLM core and keep other modules frozen, without incorporating the reference speech during pre-training. As shown in Figure the text and instruction are first formatted by the LLM input template. The paired ground-truth speech preprocessed by a frozen speech tokenizer into discrete tokens, which used compute loss with generated tokens during pretraining. To ensure consistency in input format, we apply Speak the following text as the default instruction when encountering data from Emilia (He et 2024 and NVSpeech (Liao al. 2025) that lacks an explicit instruction.
3.2 POST-TRAINING
We propose Progressive Post-nraining (PPT) scheme to disentangle speaking style and timbre and to enable FlexiVoice to have complex instruction-followee ability. It is inspired by Curriculum Learning (Bengio et al. 2009), which starts from simpler objectives and gradually advances to harder ones for stable optimization and better generalization. After pre-trianing FlexiVoice zero-shot TTS capability but still struggles with multi-modality inputs and complex instructions. PPT is to advance Flexi FlexiVoice-ben Voice-base for generalization and better performance. This progressive curriculum ultimately yields robust multi-modality instruction TTS model, FlexiVoice.
has solid
PPT has three stages:
S1 aligns multi-modality controllability of instruction and reference speech in controlled emotion-centric tasks with explicit labels.
S2 disentangles the timbre and style in reference speech, and the content and style in target text, within the same scenario as SI,
S3 extends to complex real-world instructions that are more ambiguous and harder align.
3.2.1 S1: MULTI-MODALITY CONTROLLABILITY
S1 mainly employs DPO to empower Flexi processing style instruction and timbre reference at the same time. The first two stages target emotional instructions to reduce task complexity and explicitly address the core issue. We restrict instructions templates like Use {label) emotion to read it, with labels chosen from Neutral, Happy, Angry, Sad, and Surprised.
For emotion-related tasks, paired preference data be directly obtained from Speech Emotion Recognition (SER) datasets. We use the Emotional Speech Dataset (ESD) (Zhou et al. (021) where the same speaker reads identical sentences with different emotions. Following Gao al. (2025)
for each data point, we assign a target emotion label (c.g.. Happy) via an instruction template (all templates are listed in Appendix A.5), select sentence with the target emotion as the preferred sample, the identical sentence with a different emotion (e.g., Angry) as the dis dis- preferred one, and use neutral sample from the same speaker as the reference speech (an example pair is provided in Appendix A.5).
DPO directly aligns the model's emotional output with the instruction and reference speech without requiring an explicit reward model (Rafailov et al., 2023). The preference dataset D consists of tuples (z. Ywr, vt), where x includes instruction, text, and reference; yoe the "winner" response that matches the instruction, and is the "loser." The DPO loss is defined as:
(V) T xx)
To (U)lz)) )
LoPo(Te: (To; )ett -B log (log
$ log
T (2)
where Ne is the policy model and The is the reference model, both initialized from FlexiVoice-Base
3.2.2 S2: DECOUPLING OF REFERENCE SPEECH TARGET TEXT
S2 focuses on FlexiVoice's decoupling capability in scenarios where speech reference and target text contain styles conflicting with instructions. After DPO training, the model follows emotional instructions well under neutral references, but interference remains when references or texts them- selves are emotion-laden, conflicted with the target emotion defined in the instruction. To explicitly suppress these effects, we adopt a multi-objective GRPO formulation by constructing conflicting training scenarios (e.g., Happy instruction vs. Sad reference). The reward rser (defined below) acts
4

---

Published as a conference paper at ICLR 2026
style constraint, penalizing the model if leaks style from the reference text. Conversely, rev acts as a timbre constraint, ensuring speaker identity is preserved. By optimizing the joint advan- tage, the model is forced to decouple these factors to maximize the total reward. We use the same instruction templates, combine neutral and emotional clips from SER datasets as reference speech, and sample texts from NCSSD (Liu et al. 2024b) (details in Appendix A.44.
Rewards are defined as follows: (1) Tser E (0, 1), the probability score for the instructed emotion given by the emotion recognition result from Emotion2vec-Larr (Ma al. 2024); (2) Tavv E {0, 1]. the speaker verification result from CAM++ (Wang et al. 2023) to ensure timbre consistency. Fol-
lowing Zhang (2025b), the advantage with multi-objective rewards is:
mean(r den
mean(r
A Almo
ser
std(r ser)
std (2))
where i indexes the i-th completion among K candidates for the same input I.
3.2.3 S3: ENHANCEMENT ON COMPLEX INSTRUCTION
The final stage enhances instruction following on complex, real-world directives beyond emotion tasks. Since paired preference data are infeasible at this scale, we directly employ GRPO. Unlike the first stage, where rewards are available from SER models, assessing the consistency between speech and open-ended instructions is more difficult. Huang et al. (2025) shows that Gemini-2.55pm provides reliable judgments aligned with human preferences, but its use GRPO is cost-prohine pott-prohiiitive We instead adopt the open-sourced Aumii-Adioo-B-Instteee (Ding et al. 2025) as the reward model due to its strong speech comprehension. It is prompted to output binary yes/no decision on whether the generated speech matches the instruction, which is mapped into a reward Film (1,0).
For this stage, may conflict with open-ended constraints (c.g., gender) and destabilize training. Details of the data construction process given in Appendix A.4
only instruction and text inputs, discarding references, since references
To mitigate catastrophic forgetting, we mix a small portion of S2-GRPO data during this stage. Thus, the final GRPO training becomes multi task, multi-objective optimization. The single-objective advantage in this stage (left) and the final advantages (right) are:
1/ rit mean(rinm))
A A
for inputs S2
1
(td
A
for inputs in S3.
FLEXIVOICE-INSTRUC DATASET
4.1 OVERVIEW
To equip the model with fundamental instruction-followinn capability in the pre-training phase, is essential build large-scale instruction-speect dataset covering diverse styles and scenarios. We therefore construct a high-quality and diverse dataset totaling 4,3 366 hours, where speech-related tex- tual metadata processed with an based annotator. This approach enables efficient generation of natural, high-level instructions that better reflect real human usage patterns.
4.2
MAIN PROCESS
We employ two data sources, Emilia and game voice acting, using a unified processing strategy:
Emilia (He et al. 2024) is a large-scale, diverse, in-the-wild speech dataset. Because the audio orig- inates from video platforms and podeasts, some entries include source metadata such as video titles and tag lists. Combined with transcriptions, these cues allow us to infer speaking style and scene context, enabling the generation of open-ended instructions even without direct speech analysis. We employ Deepseek-V3 V3 (Liu al. 2024a) for instruction annotation. To filter noisy samples (e.g., URLs or conflicting metadata transcription pairs), the LLM prompted to evaluate the informa- tional value of metadata for style and scenario inference, ensuring higher data quality.
eeetadata-transeripioo
To enhance expressiveness and stylistic richness, game voice acting data from two popular games is additionally incorporated. A distinctive feature of this data is the strong link between speaking
5

---

Published as a conference paper at ICLR 2026
Emilia [EN, 7
Instruction
Prompt
Serene and ifformative documentary narration about botanical gardens, delivered with a calm and educational tome.
that
the
Sec Talle
Daconmentary rim- Lardon Sacrute
Contest based
metadata, and deduce
Src
Gardezing, Sodiey
style description the time of utterancce bused on transcription, ultimutely summarining its Informational value
Game Voice Acting (EN, ZH. KO)
Instruction
lona es the core costent intersile,,
With elegantly playful tone, styly hinting somome carefully orchestrated prank.
my leart ba ditaiis
the
Your Milzo
Figure 2: Processing flow and examples for VeriioiicInssuuce For both sources, the speech transcription and related meta information to prompt LLM to generate natural and human-like de- scriptions, the instruction in our pre- tre-trainngg stage.
style and character personality. LLMs, trained on large-scale web corpora, often recognize these characters and capture their iconic styles. We thus use Deepseek-V V3 (Liu et al. 2024a to generate instructions conditioned on both transcription and character name. Concretely, we first supply the full character list and let the LLM identify those it can reliably recognize. For known characters, we prompt the LLM infer personality and speaking style from the name, then refine the description using the transcription. As with Emilia, we incorporate an informational value check to filter noisy annotations. Prompts are provided Appendix A.3 while the overall processing flow and examples are shown in Figure
5
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Benchmarks To evaluate the model's multi-modality controllability as well as capacity fol- low complex real-world instructions, we employ one self-constructed dataset and one open-source benchmark. For multi-modal control and disentanglement, we define two task types with two dif- ficulty levels each, focusing instruction- emotional TTS, summarized in Table [ For complex instruction following, we adopt the InstructTTSEval (Huang et al. (025) benchmark.
Table Tasks for multi-modality control and disentanglement evaluation. There are two tasks with two difficulties, with example for each task.
Task Type
Difficulty
Text
Reference
Example
Instruction: Speak happy emmaion.
Single Modality: Text-Only (TO)
Easy
Neutral
Today Monday.
Reference Speech:
Instruction: Speak using happy emotion.
Hard
Emotional
Text: I'm so sad that it's mining cusside.
Reference Specch:
Instraction: Speak using hoppy emotion.
Multi-Modalit::
Easy
Neutral
Neutral
Monday
Reference Speech: [A nestral voice]
Text and Reference (TR)
Instruction: Speak using happy
Hard
Neutral
Emotional
Text: Today is Monday.
Reference Specch: [A sad vaice]
The first evaluation set is built from MEAD (Wang et al. 2020 (English) and CSEMOTIONS (Tian et al. 2025) (Chinese). In TR tasks, reference speech is randomly selected as either neutral con- flicting emotional clips from the same speaker as the ground-truth For TO-hard, the text is replaced ground-trui -ruth audio exists for this
with sentences carrying emotions different from the target, case. We retain five emotion categories from both datasets and randomly sample 500 examples each for English and Chinese, ensuring balanced coverage of target emotions.

---

Published as a conference paper at ICLR 2026
Table 2: Multi-modality controllability and disentanglement evaluation result in different tasks and
two languages.
Text-Only input (TO)
Text and Reference Speech imput (TR)
Model
Easy ACC-I7
Hard
Easy
Hard
E-SMM-
ACC-r
ACC-T
ACC-I
E-SIM
SV1
ACC-I
ACC-Rİ
ESIM
EN
Groud-trth
93.4
1.00
93.4
1.00
93.4
0.6
1.00
Parler-TTS
44.6
0.72
12.2
42.0
PromptStyple
43.8
0.70
14.0
33.6
Promp(TTS
57.8
0.79
15.0
41.0
Cosy
14.4
VoxInstruct
70.6
17.8
41.2
58.5
0.81
89.0
49.7
23.9
VlexiVoice-Base
72.4
0.83
39.4
30.6
58.8
0.81
99.2
48.8
32.2
0.78
99.4
FlexiVolce
97.4
0.89
89.4
6.6
89.4
0.90
91.0
78.2
10.6
0.87
95.8
ZH
Groud-truth
61.6
1.00
61.6
1,00
61.6
4.4
1,00
Cosy Voice2
44.4
0.84
47.8
15.3
100.0
VoxInstruct
48.6
0.76
29.0
21.2
19.4
0.75
46.8
18.7
23.2
0.73
59.8
FlexiVoice-Base
78.4
0.76
66.8
14.2
25.2
0.78
99.6
22.4
38.0
0.74
99.2
FlexiVoice
99.8
0.72
98.4
0.8
81.8
0.85
98.8
75.8
13.2
0.80
98.4
The second benchmark, InstructTTSEval (Huang et al., 2025). serves as a comprehensive test set for complex instruction following beyond simple emotions. comprises 1,000 English and 1 1,000 Chinese samples covering 12 distinct speech attributes: gender, pitch, texture, clarity, fluency, speed, accent, age, volume, emotion, tone, and personality. These attributes encapsulated in instruc- tions across three task devels-Acoustic ParssicParrrmtter Specification (APS), Descriptive-Style Directive (DSD), and Role-Play (RP)-allowing diverse and fine-grann fine-grained style controls.
rigorously evaluate the model's versatility in handling
Metrics For the first evaluation set, emotion embedding extraction. We report instruction-followiin accuracy with respect to the target emotion (ACC-I, 1), and the hard settings also report aceuracy against conflicting emotions from text or reference (ACC-T, ACC-R, 4), where lower is better. Cosine similarity between emotion embeddings of ground-trut and generated speech (E-SIM, t) further reflects adherence. To ver- ify timbre preservation in TR tasks, we use CAM++ (Wang et al.) 2023) for speaker verification accuracy (SV, 1). We also evaluate intelligibility and perceptual quality: word error rate for En- glish (WER, J. ASR by Whisper-Large- V3) and character error rate for Chinese (CER, J. ASR by Paraformer zh), speech quality MOS (Q-MOS, 1-5, 1), and Comparative MOS (CMOS, -2 to 2, 1). Details of the subjective setup provided in Appendix A.9
Fmootion2ec--Large (Ma al. 2024 for SER and
For complex instruction following, we follow InstrucITTSEval (Huang et al. 2025). which uses Gemini as the judge model to assess consistency, and report macro- averrge accuracy.
Baselines We compare Flexi against representative open-source instruction TTS systems, including Parler-TTS (Lyth & King 2024), reproducible variants (Ji al., 2024b) of Prompt- Style (Liu et al. 2023 and Prompt (Guo et al. 2023), VoxInstruct (Zhou et al. 2024b). and Voice2 (Du et al. 2024). which jointly processes natural- (Huang et al. 2025), which cover both the above open-source systems and closed-source commercial models 2eemini-2.5. ---.....-afhsh-eeeieee gemini-2.55 2:iimi-2.55pr---- pro-preview-tts gpt-4o-mini-tt -ts, and Hume), and we further include MiMo-Audio-7B: Instruct, recent large audio-la model with strong cnstucctionfoolloiim capability.
speech. In addition, we report results from InstrucTTTSEN language instructions and reference
5.2 DECOUPLING ABILITY OF FLEXIVOICE
observe that FlexiVoice demonstrates strong capabilities in handling multi- modality inputs and disentangling content, timbre, and style across both English and Chinese tasks. TTe discussion is organized into four perspectives.
From Table 2

---

Published as a conference paper at ICLR 2026
Table 3: WER scores and subjective evaluation results in the first evaluation set.
Text-Only as imput (TO)
Text and Reference Speech input (TR)
Model
Easy
Easy
Hard Q-MOST
WER,
Q-MOS
CMOS
WERJ
Q-MOS-
CMOS
WER.
CMOST
EN
Groud-truth
4.50
3,16 #0.07
0.00
4,50
3.50 #9.18
0.00
4.50
4.26 #0.22
0.00
CosyVo
3,71
3.50
40.30
-0.75
40.32
3.60
3.68 #0.48
-0.88
+0.21
Voxinstruct
7.29
3.02 10:34
-0.75 10.44
14.44
2.10
10.35
-1.50 10.19
12.61
2.66 10.63
-0.86 10.36
FlexiVoice-Base
5.01
3,72 to.14
-0.12 +0.m0
5.31
3,90
t0.20
-1.25
+0.29
6.55
3.82 +0.29
-0.56 19.31
FlexiVoice
5.99
4.08 +0.20
0.91 +0.20
5,23
3.62
#0.20
0.89
#0.30
6.99
4.06 #0.30
0.78 #0.40
ZH
Groud-truth
4.55
3,78 10.14
0.00
4.55
3.38 10.14
0.00
4.55
3.92 1:007
0.00
CosyVoice2
0.78
3.54 10:37
-1.14
10.24
0.89
3.58 2:0.3
0.36 20.29
VoxInstruct
3.37
3,18 10.22
-1.60 kou.0
10.04
2.62 10.28
-1.88 10.0
9.40
3.04 1:017
-1.62 10.19
FlexiVoice-aas
5.01
4.10 #0.26
0.75 +0.40
3.08
3.74 #0.28
-1.50 +0.28
3.66 #0.00
-0.88 +0.46
FlexiVoice
10.33
0.60
10.4
4.34
3.79 10.36
-0.36 1.0.4
3.76 1.039
0.57 19.14
Multi-modality Control Comparing the easy sets of these two tasks, under the guidance of the instruction, TO-easy only includes the text condition (where the model uses a random timbre), while TR-casy incorporates both text and reference speech as multi-modality condition inputs (where the model uses the reference's 's timbre). Most baselines cannot support both simultancously, and even those that do, together with FlexiVoice Base, show a large gap, particularly for Chinese. After progressive post-training, Flexi Voice achieves substantial gains, reaching 97.4% ACC-I in English and 99.8% in Chinese on TO-easy, surpassing ground-truth in some cases. On TR-easy, it maintains strong accuracy (89.4% EN, 81.8% ZH) while preserving speaker consistency, highlighting robust multi-modality control capacity.
Target Text Disentanglement For the two difficulty levels of TO, the easy level uses neutral text while the hard level employs text with emotions differing from the instruction. The purpose is to test whether the model can ignore conflicting emotional cues in text. Baselines and FoexiVoice- Base generally fail here, showing low ACC-I and high ACC-T, indicating they are biased toward the text's implied style. In contrast, FlexiVoice substantially improves disentanglement, achieving 89.4% ACC-I with only 6.6% ACC-T in English, and 98.4% vs. 0.8% in Chinese. Notably, the Chinese gap between easy and hard settings reduced to just 1.4%, showing Voice' strong ability to ensure style is controlled by instruction alone.
Reference Speech Disentanglement For TR, the easy task uses neutral speech as the timbre refer- ence, while the hard uses emotionally charged reference speech differing from the instruction. Most baselines exhibit large drops in accuracy and high ACC-R, showing disruption by reference's 's style. FlexiVoice alleviates this issue, achieving 78.2% ACC-I (EN) and 75.8% (ZH) with correspondingly low ACC-R (10.6% EN, 13.2% ZH). This demonstrates that Flexi Voice effectively disentangles ref- erence timbre from style, preserving timbre while adhering to isstuuction-define emotion.
Trade-off between Style Control and Speaker Verification We notice that in certain scenar- ios (c.g.. TR tasks in Table 2). the pre-trained FlcciVoice-BB exhibits slightly higher SV scores compared to the final Flexi Voice. This is expected phenomenon inherent to the task of style synthesis. The Flexi Base model, lacking explicit disentanglement capabilities, tends to clone the prosody of the reference speech directly. While this yields high speaker similarity, it fails to follow the style instructions (as evidenced by its low ACC-I scores). In contrast, FlexiVoice must significantly modify acoustic features (such as pitch and energy) to satisfy the target style instruction (e.g., converting sad reference speech into a happy voice). These necessary prosodic alterations inevitably result in minor numerical reduction in the cosine similarity of speaker embeddings. Cru- cially, PlexiVoice maintains high SV scores while achieving superior instruction adherence, demon- strating that successfully prioritizes style control without compromising the fundamental speaker identity.
Intelligibility and Subjective Evaluation Table 3 presents intelligibility and perceptual quality. FlexiVoice exhibits marginal increase in WERICER compared to VoiceVoiceBBaal This phe-
8

---

Published as a conference paper at ICLR 2026
Table 4: Complex instruction-folloving evaluation results InstructTTSEEval, for closed-source and open-source models on both languages.
InstrucTTTSEvi
Instruc(TTSEvat-ZI
Model
APS
DSD
RP
Avg.
APS
DSD
RP
Avg.
Groud-truth
96.2
89.4
67.2
84.3
90.9
86.7
69.8
82.5
Closed-suured
Gemini-flash
92.3
93.8
80.1
88.7
88.2
90.9
77.3
85.4
Gemini-pro
87.6
86.0
67.2
80.3
89.0
90.1
75.5
84,8
GPT-4o-mini-TTS
76.4
74.3
54.8
68.5
54.9
52.3
46.0
51.1
Hume
83.0
75.3
54.3
71.1
Open-sourced
ParlerT'S
PromptStyle
VoxInstruct
54.9
57.0
39.3
50.4
47.5
52.3
42.6
47.5
PrompeTTS
64.3
47.6
MiMo- A Miliio-B--nsstucc
80.6
72.6
75.7
61.5
70.5
FlexiVoice-Ban
63.6
56.7
59.5
FlexiVoice
81.2
nomenon is consistent with prior findings (Li al. 2023). which indicate that standard ASR models often degrade on highly expressive or emotional speech due to significant prosodic variations. How- ever, this does not imply loss of intelligibility for human listeners. As evidenced by the consistently higher Q-MOS scores (e.g., 4.08 vs. 3.72 in EN-TO-Fasy), FlexiVoice maintains superior percep- tual clarity while delivering richer stylistic expression. In addition, the positive CMOS (up +0.9) indicates better overall expressiveness and instruction adherence, ensuring that the quality of gener- ated speech remains unaffected the same time. In contrast, baselines often obtain negative CMOS, showing lower preference in human judgments compared with the groud-truth. These results con- firm that PlexiVoice not only achieves superior disentanglement but also maintains naturalness and quality close to ground-truth.
5.3 COMPLEX INSTRUCTION-FOLLOWIG ABILITY
InstrucrTTSEval (Huang et al. 2025) defines three levels of complex instruction following. These tasks range from low-level acoustic control open-ended style generation and character imitation, thereby testing different aspects of instruction adherence. As shown in Table the pre-trained model FlexiVoice-Bart already performs competitively, especially on DSD (75.0 EN) and RP (60.6 EN), due to alignment between these tasks and the natural instructions present in our constructed dataset. After applying the Progressive Post-Traiee Dost-Training strategy, FlexiVoice Voice achieves consistent gains across all task types. On English tasks, it improves by over 12 points on APS (81.2 vs. 63.6) and nearly 11 points on RP (71.4 vs. 60.6), reaching an overall average of 79.3, close Gemini-pro (80.3). On Chinese tasks, FlexiVoice attains 70.8 average accuracy, surpassing MiMo-Audio-7BD -B-nttr--I (70.5) and reducing the gap to Gemini models. Overall, FlexiVoice consistently outperforms all open- source baselines and narrows the gap with closed-source commercial systems, demonstrating robust instruction- -oollwtion-folloiingg ability in complex, real-world scenarios. Furthermore, empirical results in Appendix Aorpendix...eeeeo A.8 confirm that Flexi Voice also excels in fine- ineeggained control of non-emotional attributes like speaking speed and pitch, significantly outperforming baselines in correlation analysis.
5.4 EFFECTIVENESS OF PROGRESSIVE POST- BOST-TRANIIGG
To rigorously validate the necessity of the proposed Progressive Post Training (PPT) scheme, conduct comprehensive ablation studies comparing different training orders and optimization strate- gies. The results are summarized in Table
We first investigate whether the order of stages matters. As shown in the first block of T starting directly with complex instruction alignment (+S3) yields poor performance (InstrucTTTSEval Avg. 72.3 Ours 79.3). This suggests that without the fundamental alignment provided by SI, the model struggles to optimize for the sparse and high-level rewards in S3. Furthermore, the alternative order +S3 S1 $2 results in significantly lower performance
Importance of Training Order

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study of FlexiVoice different training orders and strategies. We compare the Pre-traine Fre-trained model (Base), the effect of starting with Instruction GRPO (S3 first), Joint Training, and our proposed Progressive Post-Training (PPT) strategy (SI S2 + S3).
Decoupling Set (EN)
InstrucTTTSEva (EN)
Training Strategy
TO
TO
TR
TR
Avg.
APS
DSD
RP
Avg.
Easy
Hard
Easy
Hard
Flexi Vaice-VoceBaaa
72.4
39.4
58.8
48.8
54,9
63.6
75.0
60.6
66.4
Alternative Training Order
S3
72.8
59.0
48.6
38.2
54.7
73.3
78.8
64.7
72.3
+ S3
96.0
79.6
80.8
72.6
82.3
74.3
81.2
67.5
+ S3 + -2
94.8
80.2
86.8
75.8
84.4
74.9
81.4
68.2
74.8
Joint Training
SI $2 S3 (Joint)
94.8
80.8
86.2
74.6
84.1
78.3
80.5
67.8
75.5
Progressive Post- Training (PPT)
+ S1
96.0
81.4
83.3
72.2
83.2
63.4
79.3
64.2
69.0
+ S1
+ S2
96.4
88.0
89.4
80.2
88.5
66.8
80.3
67.9
71.7
+ +1
+ 82 S3 (Ours)
97.7
89.4
89.4
78.2
88.7
81.2
85.2
71.4
79.3
complex instructions (74.8) compared to our approach. This indicates that applying the fundamen- tal DPO (S1) after complex instruction training causes catastrophic forgetting of the fine-grained instruction-following capabilities learned in S3. These findings confirm that SI acts necessary "cold start" foundation, establishing robust multi-modal responsiveness before the model tackles more advanced disentanglement and generalization tasks.
Progressive vs. Joint Training Another key question is whether S2 and S3 could be optimized simultaneously. since they both apply GRPO algorithm. We compare our progressive approach against joint training strategy (+S1- where the decoupling and instruction objectives are combined. The results show that joint training underperforms the progressive strategy on both benchmarks (Decoupling Avg. 84.1 vs. 88.7; InstructTTTSEva Avg. 75.5 vs. 79.3). We attribute this to the conflicting nature of the gradients: S2 enforces strict constraints to suppress style leakage us- ing fixed classifiers, while S3 encourages open-ended style generalization via ALM-based rewards. Optimizing them jointly leads to interference, preventing the model from mastering either task at the same time.
Cumulative Gains of Our PPT Strategy The final block of Table demonstrates the step-by- step efficacy of our proposed path. The Multi-modality DPO (+S1) first yields large gains on basic controllability (+28.3 Decoupling Avg.). Subsequently, the Decoupling GRPO (+S2) significantly boosts the model's ability separate style from content and timbre, raising the Decoupling Avg. to 88.5. Finally, the Instruction GRPO (+S3) further enhances the model's capability on complex, real-world instructions (+7.6 on Instruc(TTSEval Avg.) while maintaining the high disentanglement capability achieved in the previous stage. The full PPT curriculum achieves the best balance across all metrics, representing substantial improvements over the base model and validating the progres- sive design stable and effective optimization path.
CONCLUSION
In this work we introduce FlexiVoice, a TTS system for multi-modality control that uses natural- language instructions for style and reference speech for timbre, supported by a high-quality in- struction-speech dataset and Progressive Post-Training (PPT) paradigm. PPT first strengthens the multi-modality controllability and disentanglement of instruction, text, and reference via emotion- centric DPO/GRPO, then scales to complex instruction following with an ALM-based reward, yield- ing stable optimization and broad generalization. Experiments show consistent gains on disentan- glement (e.g., large ACC-I improvements with low interference from text/reference) and strong performance on InstrucrTTSEval, where FlexiVoice surpasses competed baselines and narrows the gap to closed-source systems, while maintaining naturalness and robustness in human evaluations.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work investigates controllable text-to-speech with multi-modality inputs. The instruc- tion-speech dataset was created from publicly available licensed resources, filtered remove offensive, biased, or harmful content. No personal or sensitive user data were collected or pro- cessed. While advances in speech synthesis can potentially be misused (e.g.. generating deceptive or harmful audio), our study focuses on improving controllability, robustness, and transparency for research purposes, and we encourage responsible use of the proposed methods. All authors have read and adhered to the ICLR Code of Ethics, and confirm that this research complies with standards of fairness, privacy, and research integrity.
REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure the reproducibility of our work. Detailed descriptions of the model architecture, training objectives, and evaluation protocols are provided in the main text and appendix. We will release the instruction -peetiioonsseechhheeeeeeeee dataset, model checkpoints, and all training and in- ference code to facilitate replication and further research. Hyperparameter settings, data processing procedures, and evaluation scripts will also be included in the release materials.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilae, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang. Shuohang Wang. Xin Wang. Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang, Yifan Yang, Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: www.iiivvvv...........aaaa.avv.. org/ abs 2404. 14219
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning, pp. 41-48, 2009,
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-33 3593353559-00 2008
Houwei Cao, David G Cooper, Michael Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Waylm: Large-scal self self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics Signal Processing, 15(6):1505- 1518, 2022.
11

---

Published as a conference paper at ICLR 2026
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F3-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint NrXiv:24100068855 2024.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.18425, 2025.
Anuj Diwan, Zhisheng Zheng, David Harwath, and Eunsol Choi. Scaling rich to-speech datasets, 2025. URthttps::/ //tp://a///vvv/// org/ aww.///...... 2205330313133
text-
Zhihao Du Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412. 101 2024.
Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. Emo-dpo: Control- lable emotional speech synthesis through direct preference optimization. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2023.
Haorui He, Zenggiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, al. Emilia: An extensive, multilingual, and diverse speech dataset for sarge-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IEFE, 2024.
Kexin Huang, Qian Tu, Liwei Fan, Chenchen Yang, Dong Zhang, Shimin Li, Zhaoye Fei, Qinyuan Cheng, and Xipeng Qiu. Instructttseval: Benchmarking complex natural- Satuuall-angugge instruction following in text-to-speech systems. arXiv preprint arXiv:2506. 16381, 2025.
Yiqiao Huang, Yuancheng Wang, Jiaqi Li, Haotian Guo, Haorui He. Shunsi Zhang, and Zhizheng Wu. Debatts: Zero-shot debating text-to-speech synthesis. arXiv preprint 2rXiv:241.006544 2024.
Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. Textrolspeech: A text style control speech corpus with codec language text-to- speech models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10301-10305. IEEE, 2024a.
Shengpeng Ji, Jialong Zuo, Wen Wang. Minghui Fang, Siqi Zheng, Qian Chen, Ziyue Jiang, Hai Huang, Zehan Wang, Xize Cheng, al. Controlspeech: Towards simultaneous zero-shot speaker cloning and zero-shot language style control with decoupled codec. arXiv preprint 25ti:2200601200. 2024b.
Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. Speechcraft: fine-grained expressive speech dataset with natural language description. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1255-1264, 2024.
Jiaqi Li, Xiaolong Lin, Zhekai Li, Shixi Huang, Yuancheng Wang, Chaoren Wang. Zhenpeng Zhan, and Zhizheng Wu. Dualcodec: A low-frame-rate, semantically-enhaneed neural audio codec for speech generation. arXiv preprint arXiv:2505. 13000, 2025.
Yuanchao Li, Zeyu Zhao, Ondrej Klejch, Peter Bell, and Catherine Lai. Asr and emotional speech: A word-level investigation of the mutual impact of speech and emotion recognition. In Interspeech 2023, pp. 1449-1453. International Speech Communication Association, 2023.
Huan Liao, Qinke Ni, Yuancheng Wang, Yiheng Lu, Haoyue Zhan, Pengyuan Xie, Qiang Zhang, and Zhizheng Wu. Nyspeech: An integrated and scalable pipeline for human-like speech modeling with paralinguistic vocalizations. arXiv preprint arXiv:2508.04 2025.
12

---

Published as a conference paper at ICLR 2026
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In 1 1tth International Conference on Learning Representations, ICLR 2023, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Guanghou Liu, Yongmao Zhang, Yi Lei, Yunlin Chen, Rui Wang, Lei Xie, and Zhifei Li. Prompt- style: Controllable style transfer for text-to-speech with natural language descriptions. In Proc. Interspeech 2023, pp. 4888-4892, 2023.
Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. Generative expressive conversational speech synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 4187- 4196, 2024b.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 1355)xe016639, 2018.
R. Lotfian and C. Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4))471-483
Deober- 2019. doi: 10. 10.1109/AAFFC::2012276665
Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations, 2024.
Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shil.ang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15747-15760, 2024.
Kari Ali Noriy, Xiaosong Yang, and Jian Jun Zhang. Emns/inz//crpus: An emotive single- speaker dataset for narrative storytelling in games, television and graphic novels. arXiv preprint 1rXiv:2305.13137, 2023.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 527-536, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Mamning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 14:53728-53741, 2023.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier- bassee neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.0083 008144 2023.
Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao Cheng Wen, Xingjun Tan, Chuandong Xie, Shuran Zhou, Rui Yan, Chenjia Lv, Yang Han, Wei Zou, and Xiangang Li. Kespeech: An open source speech dataset of mandarin and its eight subdialects. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Fengping Tian, Chenyang Lyu, Xuanfan Ni, Haoqin Sun, Qingjuan Li, Zhiqiang Qian, Haijun Li, Longyue Wang, Zhao Xu, Weihua Luo, et al. Marco-voice technical report. arXiv preprint arXiv:2508.000333 2025.
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312 15821, 2023.
Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. Cam++: A fast and efficient network for speaker verification using context-aware masking. In INTERSPEECH, 2023.
13

---

Published as a conference paper at ICLR 2026
Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking- face generation. In ECCV, 2020.
Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng. Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024.
Xueyao Zhang, Yuancheng Wang, Chaoren Wang, Ziniu Li, Zhuo Chen, and Zhizheng Wu. Ad- vancing zero-shot text-to-speech intelligibility across diverse domains via preference alignment. arXiv preprint 250.:::5500000113, 2025a.
Xueyao Zhang, Junan Zhang, Yuancheng Wang, Chaoren Wang, Yuanzhe Chen, Dongya Jia, Zhuo Chen, and Zhizheng Wu. Vevo2: Bridging controllable speech and singing voice generation via unified prosody learning. arXiv preprint arXiv:2508. 16332, 2025b.
Xueyao Zhang, Xiaohui Zhang. Kainan Peng, Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, et al. Vevo: Controllable zero-shot voice imi- tation with self-supervised disentanglement, arXiv preprint arXiv:2502.07223, 2025c.
Guanlong Zhao, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Houuhaare-HHuilaineee John Levis, and Ricardo Gutierrez-Osuna L2-arctic: A non-native english speech corpus. In Proc. Interspeech, pp. 2783-27888 2018. doi: 10.214377lntesspeechh.o 201. 2018- 1110. URL http::/dx.
doi Boi.org/10... 212371444112 211../11ee3//nteeeeeeee. 201 81100
Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. M3ed: Multi-modal multi-scene multi-label emotional dialogue database. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 5699-5710, 2022.
Jiaming Zhou, Shiyao Wang, Shiwan Zhao, Jiabei He, Haoqin Sun, Hui Wang, Cheng Liu, Aobo Kong, Yujie Guo, and Yong Qin. Childmandarin: A comprehensive mandarin speech dataset for young children aged 3-5. arXiv preprint arXiv:2409. 18584, 2024a.
Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. voice conversion with new emotional speech dataset. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
Seen and unseen emotional style transfer for
IEEE, 2021.
Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, and Jingchen Shu. Indextts2:: A breakthrough in emotionally expressive and duration-controlled auto-regressive zero-shot text- to-speech. arXiv preprint arXiv:2506.22 21619, 2025.
Yixuan Zhou, Xiaoyu Qin, Zeyu Jin, Shuoyi Zhou, Shun Lei, Songtao Zhou, Zhiyong Wu, and Jia Jia. Voxinstruct: Expressive human iestruction-tosspeee generation with unified multilin- gual codec language modelling. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 554-563, 2024b.
A APPENDIX
A.1 MODEL. STRUCTURF
As illustrated in Section 3 our model mainly contains two stages: auto-regressive LLM and flow matching. In the first stage, the model receives the inputs of text, instruction, and reference speech. The text and instructions are formatted according to the LLM's input template, with the reference speech transcription concatenated before the text. We use the semantic code extracted from Dual- Codec (Li et al.) 2025) to represent the speech in discrete form within our system. Therefore, the reference speech will first be converted into discrete tokens via DualCodec, and the tokens resulting from processing the formatted text and instructions through the LLM encoder will be concatenated to the front. Together, they will serve as input for the rego-segeessive LLM. Here we employ Phi- 3.5-mini-instruct (Abdin al. 2024) due to its suitability for multi-modal tasks. We first expand
14

---

Published as a conference paper at ICLR 2026
Text
Instruction
<jaystomms>
speesh generation. need Instruction, please spasic follownng instruction instrucioon-jeedd
speak the proviced
<juseri>
instruction: Say this kind and slightly cheerful tone. test freference text) Nice meet you. I'm FlexiVoice.
Vocoder
<endd--
<jassistant
LLM
*ballpj Reference
Speech Tokenizer
Mell Spectrogram
Flow Matching
Extractor
Figure 3: The complete structure of FlexiVoice.
its vocabulary (equivalent to DualCodec's vocabulary size, 16384), then use the parameters of a text LLM the initial state for pee-training and post-training, illustrated the main part in our work.
Mainstream TTS works (Du et al.) 2024; Zhou et al. 2025 Zhang et al. 2025a) chose flow match- ing (Lipman et al. 2023) the decoder because of its high-quality reconstruction of speech details. For the second stage in this work, we employ a flow matching module trained on Emilia (He et al.)
2024) to convert generated code into mel-spectrum, using reference speech code the condition. Finally, the mel-spectrum is converted into the target audio via a vocoder (using Vocos (Siuzdak)
2023 in this case). The model structure is shown in Figure
A.2
DATA PROCESS FOR PRE-TRAINING
Table 6: Instruction-speech datasets used in pre-training stage.
Source
Dur (h)
Description
ParaSpeechCaps Diwan al.
2025
2847
Open-sourced style-prompted speech data
ChildMandarin (hoou et al. 2024a
41
Specch clips of child's voice Speech clips with debating style
Debatts (Huang et al.
2024
67
Emotion set
117
Speech clips with different emotions
KeSpeech Tang et al.
2021
1541
Speech clips with different dialects, ages, and genders
12-ARRTIC (Zhao
2018
Speech corpus of non-native English
NVSpeech Liao al.
2025
775
Paralinguistic tagged speech data
Flnstueice-Instruttt
4316
Expressive speech with high-level instructions
Including CREMA-D [Cao
2014],
EMNS
(Nrry
2023
ESD (Zhou 2021
IEMOCAP Bussa
2008
M3ED Zhn al.
202 MELD Poria et all 2019) MSP Podcast LLtifan Busso 2019 RAVDESS
Livingstone & Russo
2018
For ParaSpeechCaps (Diwan et al. 2025), , due to its bottom-up data annotation process, some speech entries possess detailed acoustic feature dictionary alongside summarized deseription. For these data points, we concatenate the feature dictionaries together and randomly sample descriptions. For other data, utilize their descriptions instructions.
For ChildMandarin (Zhou et al.) 2024a), Detatts (Huang et al. 2024), and the emotion set, since they all single-label datasets (boy, dehate scene, emotion label), pre-generated several instruction
15

---

video], "tags": [Tag list of Published as a conference paper at ICLR 2026
(e.g.. "Speak in the voice of {label)," "Imagine you are in a debate scene," " "Express the emotion of (label]"). Then, for each speech data point, we randomly selected template and filled in the corresponding label.
templates using Deepseek- V3 (Liu al. 2024a)
KeSpeech (Tang et al. 202T) and L2-ARCTIC (Zhao et al. 2018) are both multi-label datasets. For example, each speech KeSpeech has three attributes: age, gender, and accent. For each attribute triplet across the entire dataset, we use Deepseek- V3 (Liu et al.) 2024a) to generate three Chinese instructions and three English instructions, which are then randomly applied the entire dataset.
For NVSpeech, which is a speech corpus with paralinguistic tags, we do not apply any instruction but their tags enlarge the vocabulary size of our core LLM module guarantee expressive paralinguistic generation. All of the instruction-speech paired datasets in the pre-training stage are listed in Table 6
A.3 PROMPTS IN INSTRUCTION DATA CONSTRUCTION
For data source of Emilia
Role and Tisks
analysis expert tasked with generating descript requized for users' Instructien TCS tasks. Your coze responsibility is: Generating voice descriptions styles, amotions, etc.)
are multilingual
(including
limited scenarios,
Detailed Description
The metainfa includes the video title and Hat of video taga from which the audio originales. title.
title is isrelevant le.g., URLs, gibberish, eto.), igaore the
scene speaking style corflict, priorizize the text content.
tags title,
case
Output a concise description of the speeth in nstural humen language instrections. Use colloquial, vivid phraaing. Vary sentenca structures to avaid template-like patterns.
Determine whether valid or speech descriptions text and mata information and classify the information value (high/sediwn/lox).
generated based on
Input/Output Specifications Input Structure:
sspeech Lransoript"; "The speech text to analyse",
"mezainfo"; ("title": [title
source video]}
Output Requirement:
deacription" "Natural language description speech style",
value": "information value hiigh/nedium/oow/
Example Example Input:
"speech transcript";
currently
Tour kille, nearly
6, 000 in gold.", "metainfo": ("title*: "Honer Kings Autumn rinals", "tags": "game, nsports, Honor of Kings, tan, KFL, DYG*)
Example Cutput:
"Professional passionste esports commentary, delivered excitement and enthusissm. "value": "high"
'descriptien":
Core Comstraint
when metainfo conflicts with text, prioritize text content.
Frchibit unsounded speculation; prohibit all uncertain deseil descriptions.
The example oucputs are for reference only; do rot rigidly adhere to their phrasing,
16

---

Published as a conference paper at ICLR 2026
For data source of game voice acting
Role and Tasks
You. are a mult multilingual text enslysis expect tasked with generating voice descriptions required for users' instruction TiS tasks. Your respensibility is: Generating voice descriptions based on text and matainfo (including but not limited to scenarios, styles, emotions, etc.)
Description
Melainfo refera to specific character in [GameName), whose dialogue is the speech transcript.
Infer the character's spesking based their name and output description of the speech.
the speaking style conflicts the texe expressed style, prioritize taxt.
NOT include specific character nanes the deseription.
Use colloquial, vivid phrasing. Vazy sentence structures to aveid template like patterns.
Determins whether wolid rich speech descriptions be ganerated based the text and meta informacion and claasity Lty the information value whigh/
Input/Outpet Specifications Input Struoture:
sspeech transcripe"; "Speech tronscript (dinlogue from a [GameName) charscter) " "matainfo": "Socrea information (character name in [GamaName]] )
Output Requi rement:
"descriptien" "Natural language description of the speech style", vvalue "information value (high/ ttigh/medium/low/
Example
Example Input:
"speech transcript": *As long as the content is interesting, these experts can help spice up.",
"matainfo"; "Yae Miko*
Example Output:
"deecription: "With hist of playful elegance in her tone, if she had long seen through the other's thoughts yet chose to hint at them sub:ly.", *volue" "high"
Core Constraints
metainfo conflicts with sext, prioritize text content.
#rchibit unsounded speculation; prohibit all uncertain deseil descriptions.
The example oucputs are for reference only; do not rigidly adhere to their phrasing,
A.4 DATA CONSTRUCTION FOR GRPO TRAINING
Decoupling Task The training set is constructed from the recording subset NCSSD (Liu et al. 2024b), comprising about 20,000 Chinese and 10,000 English dialogue speech samples. We selected this dataset primarily because its transcriptions align closely with everyday conversations and im- plicitly carry emotional tendencies in most cases. We randomly assign emotion labels to each data point, thereby generating two types of data: those where the instruction and transcription emotions align, and those where they conflict. This approach helps guide the model learn the ability to decouple content and style within text. Simultaneously. we randomly assign neutral and emotional speech from the pre-training data's emotion set (6) each data point with a 90% and 10% prob- ability, respectively, as reference speech. This enables the model to learn to decouple timbre and style from the references. We utilize half of the Chinese data and all of the English data, forming approximately 20,000 balanced data points as the GRPO training set for the decoupling phase.
Complex Instruction To enhance the model's performance in complex instruction followiin tasks, we constructed a rich and diverse GRPO training dataset. Specifically. for each language (Chinese and English), first randomly sampled 1,000 existing instruction-text inputs from the
17

---

Published as a conference paper at ICLR 2026
pre-training data to ensure optimization stability. Next, prompted Deepseek-V3 V3 Liu et al, 2024a) to randomly generate 6,000 instructions across three distinct configurations: (1) Generate detailed and comprehensive acoustic feature descriptions in dictionary format, (2) Generate natural language descriptions incorporating 3/4/5/6 explicit acoustic features, (3) Freely generate instruc- tions for arbitrary scenarios consistent with human usage patterns. Collectively, we generated 14,000
instruction-text paired inputs as GRPO training data for the second stage.
A.5
DETAILS IN EMOTIONAL CONTROL DPO
Instruction Templates
Say the sentence with the emotion of {label}.
H [
Say this with {label} tone.
|
Speak this sentence in {label} manner.
# # [aabbl))
Deliver this text with {label} emotion.
+ (label)
Use a {label} voice.
Express this sentence with (label) feeling.
Read it with {label} inflection.
# # [label |1
Recite this with }label} sentiment.
Hi [
Voice this in a {label} style.
E (abbe))
[label} (Use only one emotion label)
{label}
One Paired Sample (emo=angry, speaker_idee id=0013)
"prompt text": "Chapter eleven on the doorstep.".
"ppropptwwv__peee "Emotion_Speede hatpeh/Daaaat//011////a//////010 "target.text": "What are you waiting for? man.",
"instruction": "Say this with angry tone.".
"chosen_war -pthh": 3ttpoooonSSSeeec..aaaat////1/// "rejected wav waa_"" "Emotion 'Emotion_Speechh www.D/aas///0013///g//01.......0
A.6 REWARD SELECTION IN DECOUPLING GRPO
In decoupling GRPO, we first use SER signal as one of the rewards to control the correct emotion synthesis. With the reference speech input, we also need to guarantee that the generated result is from the same speaker with the reference. There are two choices: speaker verification signal (0 or 1) and speaker similarity signal (values between and 1), using CAM++ (Wang et al. 2023) and WayLM-largee laneeu-ungee (Chen et al. 2022) respectively. Given the DPO optimized model, we train the multi-objectivv GRPO using SER reward with two separate speaker-related rewards for four epochs, then test them on two difficulty levels of TR in the decoupling evaluation set, shown in Figure
Results show that models using speaker verification as the reward signal achieve significant improve- ments, while those using speaker similarity yield largely unchanged or even reduced performance. This occurs because models computing speaker embeddings (WavLM in this case) do not rely solely on speech timbre but incorporate additional acoustic features beyond timbre, such as pitch and emo- tion. When humans speak with varying emotions, acoustic attributes like pitch, speech rate, and volume inevitably differ. Consequently, expressive emotion delivery yields high SER reward scores but low speaker similarity signals. This phenomenon thus be interpreted conflict between
18

---

Published as a conference paper at ICLR 2026
82
Speaker verification
Speaker verification
90-
Speaker similarity
Speaker Similarity
DPO result
80
DPO result
78
76
ACC.
84
epoch
epoch
Figure 4: Comparison of results for two reward signals in decoupling GRPO, on the decoupling evaluation set's TR-easy (left) and TR-hard (right) tasks.
SER reward and speaker similarity reward, hindering the normal optimization process of the multi- objective GRPO task.
Speaker verification signals, as a more loose form of speaker similarity, can accommodate models that generate more expressive emotional speech while ensuring speaker identity. Therefore, select speaker verification as one of the final reward signals in the S1 GRPO stage.
A.7 VALIDATION OF REWARD MODELS
For the complex instruction following in Stage 3 (S3), require robust reward model to evaluate open-ended generation. While Gemini-2.5-PP5 serves as the Gold Standard judge in the InstructiTT- SEval benchmark and aligns well with human preference (Huang et al. 2025), employing it directly as reward model for GRPO is impractical due to its prohibitive inference costs and high latency dur- ing online sampling. Consequently, we propose utilizing -i----uddo----stttuee as cost-effective surrogate.
To validate the reliability of this substitution, we assess the alignment between Kimi-Audio-71 and Gemini-2.55PPp agreement of their judgments using the Macro-F1 score under the same prompt used in our training. As shown in Table [ Kimi-Audio-7B achieves solid agreement (Macro-F1 0.60) with Gemini across all task types in both English and Chinese. These results demonstrate that Kimi-Audio-77 effectively captures the preference patterns of the larger model, providing valid and efficient signals for optimizing complex instruction adherence.
We conduct the evaluation on the InstrucTTTSEval dataset, measuring the
Table 7: Agreement evaluation (Macro-F1) between our surrogate reward model (Kimi-Audio-)) and the gold-standard judge (Gemini-2.5-Pro) on InstructTTSEval. The high consistency validates the effectiveness of using Kimi-Audio as reward model.
English (EN)
Chinese (ZH)
Metric
APS
DSD
RP
APS
DSD
RP
Macro-Fi
0.62
0.62
0.65
0.60
0.64
0.63
A.8 EVALUATION ON
STYLE CONTROL
While main experiments focus emotional expression and complex instruction following, the claim of controlling flexible style necessitates validation on fine- finegrained prosodic attributes beyond emotion. To address this, conducted specifie evaluations Speaking Speed and Pitch control.
19

---

Published as a conference paper at ICLR 2026
Table 8: Spearman Correlation Coefficient evaluation Speaking Speed and Pitch control.
Speed Correlation
Pitch Correlation
Model
EN
Z.H
Z.H
ParlerTTS
0.61
0.22
0.68
0.79
PromptTTS
0.24
0.12
PromptStyle
0.05
0.24
VoxInstruct
0.75
0.49
0.31
0.15
FlexiVoice-Base
0.62
0.47
0.66
0.52
FlexiVoice
0.86
0.78
0.91
0.90
We constructed set comprising 100 samples for English and 100 for Chinese. For each sample, synthesized speech conditioned on instructions corresponding to three distinct levels:
Speed: Fast, Normal, Slow (e.g., instruction: Read this sentence quickly.)
Pitch: High, Normal, Low (c.g., instruction: Speak with a high-pitched voice.)
To quantify the controllability, we calculated the Spearman Correlation Coefficient between the instruction levels (mapped 1, 2, 3) and the extracted acoustic features of the generated audio. Specifically, we measured phoneme duration (phonemes per second) for speed and median FO for pitch. A higher correlation indicates that model more accurately follows the gradient of the instruction.
As shown in Table S Flexi achieves the highest correlation scores across both languages and attributes. Notably, it significantly outperforms the baseline models and the pre-trained FlexiVoice- Base, demonstrating that our Progressive Post-Training (PPT) strategy effectively enhances fine- grained physical parameter control alongside abstract emotional styles, do not apply any targeted optimization on speed or pitch.
A.9 SUBJECTIVE EVALUATION CONFIGURATION
Subjective evaluations are carried out by group of compensated participants, all of whom had strong speech and audio expertise. We selected two baseline models with relatively comprehensive task support, our pre-trained model, and FlexiVoice for subjective evaluation. For each task in each language, we randomly select five paired samples from each model and the ground-trutt truth. To ensure reliability, every audio sample was independently rated by at least three different individuals for both subjective evaluation.
Participants are asked to evaluate the overall quality each generated speech sample on 5-point scale, considering aspects such clarity, naturalness, and absence of distortion/artifactss. ignoring the style instruction and timbre reference. The meaning of each score is defined as follows:
Q-MOS
Speech is highly natural, clear, and pleasant to listen to. No noticeable artifacts distor- tions. Comparable to professionally recorded human speech.
4 Speech is generally natural and intelligible, with only minor imperfections occasional artifacts that do not interfere with understanding or listening comfort.
- Speech is intelligible but has moderate issues such as slight distortion, unnatural prosody, mild background noise. Quality is acceptable but clearly below high-standard human record- ings.
2 3-Speech is somewhat difficult to understand due to significant artifacts, distortions, or un- natural delivery. Quality issues noticeably affect the listening experience.
Speech is largely unintelligible or highly unnatural, with severe artifacts or distortions that make evaluation difficult.
20

---

Published as a conference paper at ICLR 2026
In this task, participants asked to compare two audio samples (one ground-trufe and one generated by a model) under the condition of given target emotion. The primary focus is on the richness and naturalness of emotional expression, which reflects the instruction-follooing and decoupling ability of the models.
CMOS
The additional evaluation rules are: (1) When no reference speech is provided, ignore timbre consis- tency. (2) When reference speech provided, timbre similarity is a secondary criterion (acceptable long both samples sound like the same speaker). (3) Slight mispronunciations or noise should be disregarded; the main comparison is the accuracy and expressivenes: of emotion. And the scoring scale is defined as follows: (Note that the order of paired audio demonstrations' is random.)
+2 Audio much better than Audio A in conveying the target emotion.
+1 Audio B is slightly better than Audio A in emotional expression.
0 Both samples comparable in terms of emotional richness and naturalness.
1 Audio A is slightly worse than Audio B in emotional expression.
-2 Audio A is much worse than Audio B in emotional expression.
A.10 REPRODUCIBILITY DETAILS
To facilitate the replication of our results and future research, we provide detailed specifications regarding the computational resources and training configurations for the Progressive (PPT) scheme. Flexi Voice built upon the Phi-3.5-mini-instrud backbone, which comprises ap- proximately 3.8 billion parameters. All training stages are conducted on 8x NVIDIA A800 (80GB) GPUs. The total training time for the post-training pipeline approximately 3.5 days.
Specifically, the SI (Multi-modalicy DPO) stage serves pleting in approximately 2 hours. We train the model for epochs with a learning rate of 1 x 10-5 and KL penalty coefficient 3 0.1. The S2 (Decoupling GRPO) stage, which involves online sampling to enforce disentanglement, requires approximately 36 hours. For this stage, we employ a group size of G 8, training for 2 epochs with learning rate of 1 10 and 3 = 0.1. Finally, the S3 (Instruction GRPO) stage utilizes a group size of G = to accommodate complex instruction following and runs for 2 epochs. Due to the complexity of the ALM-based reward calculation, this stage takes approximately 42 hours.
efficient alignment foundation, com-
A.11 LLM USAGE
Large Language Models (LLMs) were used solely as an assistive tool grammar correction and language polishing of the manuscript. They did not contribute research ideation, methodology, experiments, analyses, or the generation of scientific content, All conceptual and technical contri- butions entirely those of the authors.
21


Reviews:
[{"review_id": "kSOgcISFVB", "reviewer": "Reviewer_AyBD", "summary": "The paper proposes a text-to-speech (TTS) system that can generate speech in any speaking style with any voice using natural language instructions for style control and reference speech for timbre control in a zero-shot manner. The model is built upon a  large language model (LLM) and trained using a novel Progressive Post-Training (PPT) scheme with three stages. The authors also introduce a new dataset, FlexiVoice-Instruct, annotated with natural language instructions via LLMs. Experiments on emotion datasets and the InstructTTSEval benchmark show FlexiVoice surpasses baselines (e.g., VoxInstruct, CosyVoice2) in style-timbre disentanglement, controllability, and naturalness.", "questions": "Some issues and statements need to be clarified:\n\n1. How does FlexiVoice perform on unseen speaking scenarios (e.g., narrative, dialogue) or in low-resource languages? How sensitive is the system to the balance between style fidelity and timbre preservation?\n    \n2. In Section 3.1, the authors mention using \"Speak the following text\" as a default instruction for data without explicit instructions. For the multi-modality tasks (TR) during inference, what is the precise input format when \"only\" a reference speech is provided without a style instruction? Does the model default to a neutral style, or does it attempt to infer a style from the reference? \n\n3. In Section 3.2.3, you mention that for S3, you \"discard references\" to avoid conflict with open-ended instructions. However, in Appendix A.4, it is mentioned sampling \"existing instruction-text inputs from the pre-training data.\" Please clarify if these pre-training data points included reference speech, and if so, how was this handled? Were the references simply omitted during S3 training?\n\n4. The paper mentions that Kimi-Audio-7B was chosen over Gemini-2.5-Pro due to cost. Any validation is performed to ensure that the ranking or preference judgments from Kimi-Audio are sufficiently aligned with human preferences or with the more powerful Gemini model for your specific task?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work bridges instruction-based and zero-shot TTS research, demonstrating how LLM-based reinforcement learning can generalize across diverse speaking scenarios. The progressive reinforcement learning curriculum (PPT) is novel in the context of TTS controllability. The unified design for style–timbre disentanglement with both natural language and reference inputs is a meaningful advance beyond previous instruction-based or zero-shot models.\n\n2. This work addresses a critical and challenging problem in controllable TTS: achieving independent, flexible control over multiple attributes. The demonstrated ability to decouple style from timbre and content is a step forward. \n\n3. The experimental demonstrates clear and substantial improvements over a wide array of strong baselines. The construction of the large-scale FlexiVoice-Instruct dataset is a major undertaking that adds considerable value to the community.", "weaknesses": "1. The decoupling evaluation is heavily focused on emotional control. While emotions are a key aspect of style, the paper would be strengthened by a more direct analysis of other stylistic aspects (e.g., speaking rate, pitch range, informal tone) to fully validate the \"any style\" claim. A qualitative analysis or case studies on non-emotional instructions would be valuable. \n\n2. The progressive training scheme involving multiple stages of DPO and GRPO is computationally intensive. The paper does not report training time, GPU requirements, or model size, which are critical for replication.\n\n3. Although Table 5 shows training-stage effectiveness, a finer-grained analysis of reward function contributions (DPO vs. GRPO weighting) would help interpret why specific stages contribute differently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9nH1QdulHB", "reviewer": "Reviewer_Uu5e", "summary": "This paper introduces FlexiVoice, a text-to-speech (TTS) model which can control from both natural language instruction and speech prompt for voice timbre.  This model is built on a pretrained LLM (phi-3-mini-instruct) and pretrained on Emilia and a diverse set of speech dataset. \nThe major challenge for this instruct control TTS is the multi-modality controllability and disentanglement of instruction, text and speech reference (e.g. conflict between style and text content or voice timbre). \n\nThe major contributions for the paper are \n\n1). develop a large-scale and diverse speech dataset with instructions (4316 hours) as FlxiVocie-Instruct. The annotations are generate from Deepseek-v3 and speech are in-the-wild speech dataset with diverse coverage and expressive richness such as game, podcast, etc.\n\n2).  propose a progressive post-training (PPT) including 3 stage training as DPO for multi-modality controllability, decoupling GRPO for disengaging speaking style and voice timbre, instruction GRPO for instruction following ability enhancement. \n\nExperiments show FlexiVoice perform strong capability in decoupling factors and maintain the TTS quality measured by human judgements on naturalness, controllability and robustness.", "questions": "1. It seems the introduction of dataset for pretraining on FlexiVoice-Base. Paper mentions the FlexiVoice Instruct as 4316 hours which is used in pre-training stage. Does it mean it use the same corpus with the alignment stage? \n2. Is there any quality comparison like between the FlexiVoice-Base and FlexiVoice? How many data pair used in the S1 DPO stage? And, if the pre-training stage add these instructions, how dose it compare with DPO alignment?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The novel proposal of curriculum learning framework with DPO + 2 stage GRPO achieve the target to control (DPO initial alignment) and GRPO disentanglement and generalization. The PPT schema methodologies are successfully applied to achieve the goal which demonstrate the alignment strategies progressively to realize the controllability and disentangle style instruction, reference timbre, and text contextual content.  \n\n2. The instruct dataset construction and evaluation set design as text-only (TO), text and text and reference (TR) and their easy and hard set. The data pipeline construct based on a in-the-wide diverse and expressive dataset and scalable automatic annotation. The evaluation test set design are also thoughtful and convincing to identify the performance.", "weaknesses": "1.  The paper claims \"can speak in any style with any voice\", however, the instruction motioned as only 5 emotion as neutral, happy, angry, sad and surprised in training and benchmark (TO and TR). There evaluation seems don't clearly demonstrate any other emotion or other natural language style instruction or conflict voice timbre in the following subjective test which make the title is a little overclaimed. \n\n2. For the evaluation, only first evaluation set involve human judge as CMOS test, and it outperforms cosyvoice2 when the hard test, but not on the (ZH) easy test. There isn't subject evaluation on complex instruction-following or open instruction evaluation. As there are two reward model (Kimi-Audio-7B) and Emotion2Vec introduced in the GRPO, however, there are not subjective evaluation in the experiment which prove they are highly related to human perception.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hO0AXbEwzh", "reviewer": "Reviewer_Ct16", "summary": "This paper proposes SUPM-TTS, a framework for style-universal text-to-speech generation that aims to synthesize expressive and natural speech in any speaking style, even unseen ones. The model integrates multiple components—adaptive prompt modulation, disentangled representation learning, and multi-corpus training—into a unified TTS system. The authors also construct new datasets and conduct extensive experiments. Overall, this is an ambitious attempt to generalize TTS systems beyond fixed style categories.", "questions": "Please refer to the Weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an extensive and well-engineered system that integrates multiple components and datasets into a cohesive framework, which is commendable.\n\n- It targets the problem of style-universal speech synthesis, addressing the need for flexible, controllable, and expressive TTS generation.\n\n- The proposed prompt-based approach allows conditioning on various prosodic or emotional cues, contributing to improved speech diversity and controllability.\n\n- The work offers thorough evaluations, including MOS, WER, and SV metrics, across several datasets.", "weaknesses": "### **Problem & Motivation**\n\n- The motivation could be strengthened by explaining how this approach differs from recent prompt-based or diffusion-based TTS systems. The overall goal overlaps with several existing works such as FlexiVoice, PromptTTS, and StyleSpeech, making it difficult to isolate what is novel.\n\n- The paper mentions disentangling timbre and style, but it is unclear how this disentanglement is achieved or guaranteed. Without a clear mathematical constraint or empirical validation, the claim of disentanglement remains qualitative.\n\n### **Method**\n- In general, the method novelty is limited. The proposed framework combines known elements—prompt conditioning, disentanglement, and adaptation—without introducing fundamentally new mechanisms. The design could be more appropriately positioned as an integration of recent techniques like DPO or GRPO for controllable TTS rather than as a new paradigm.\n\n- The claim that the system generalizes to unseen styles is only partially supported; the method seems to rely heavily on the diversity of training prompts rather than a specific generalization mechanism.\n\n### **Experiments**\n\n- The experiments are extensive, but the SV metric results are relatively weak. In several cases, FlexiVoice-Base outperforms FlexiVoice, contradicting the expected hierarchy. The paper should provide an explanation for this discrepancy—perhaps due to model capacity, over-regularization, or mismatch in style conditioning.\n\n- The subjective demo results do not convincingly show that either model performs consistently well. The samples suggest variability across styles, which should be discussed rather than glossed over.\n\n- The paper should clarify whether Chinese experiments use CER (Character Error Rate) and how this metric is computed. Similarly, the reported WER differences may stem from ASR models underperforming on expressive or emotional speech, which should be explicitly acknowledged.\n\n### **Writing**\n- The current experimental sections (5-1, 5-2, 5-3) are fragmented. It would be better to merge them into a single “Experimental Setup” subsection, organizing content at the paragraph level for better flow and readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hof2llkNv8", "reviewer": "Reviewer_Jm4v", "summary": "The paper presented a TTS system that follows natural language instructions. The authors proposed to increase the synthesis naturalness and the instruction-following capability through a three stage Progressive Post-Training process.", "questions": "See weaknesses. The authors should address the points to convince me that the paper is of acceptance quality.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method is straightforward and easy to follow. The presentation is clear. The system performs reasonably well against several well-know baselines.", "weaknesses": "The main weakness of the paper is the experiment design does not validate the main contribution claimed by the paper.\nThe paper claims that the main novelty lies in the three stage Progressive Post-Training, which makes sense as the other parts of the system follows the commonly-used instruction-following TTS.\nHowever, in the experiments (Table 5, Section 5.6), the paper only showed the efficacy of having multiple alignment objectives (S1, S2, S3). It does not justify why the \"Progressive\" training of the three objectives is needed, other than simply post-training with the three objectives all-together. Also, is the order (S1->S2->S3) important? \n\nThe three objectives (S1, S2, S3) to post-train TTS, as already mentioned in Section 3 by the authors, are adapted from existing works (e.g., line 191, 217). This makes me think that the novelty of the paper is in progressively applying them, as also alluded to several times in Section 1. However, there is no evidence showing that the progressive part is crucial to the performance in the experiments. The authors should either explain the main distinctions between the previous approaches (are there anything novel on the post-training objective?), or show the efficacy of the progressive training.\n\nAs for now, I question if the scientific contribution is enough for this paper to be accepted to ICLR, based on the reasons mentioned above.\n\n- Typo: S\"2\" -> \"3\": Instruction GRPO in Figure 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)