BatonVoice: An Operationalist Framework for Enhancing Controllable
  Speech Synthesis with Linguistic Intelligence from LLMs
Authors: name: Yue Wang; name: Ruotian Ma; name: Xingyu Chen; name: Zhengliang Shi; name: Wanshun Chen; name: Huang Liu; name: Jiadi Yao; name: Qu Yang; name: Qingxuan Jiang; name: Fanghua Ye; name: Juntao Li; name: Min Zhang; name: Zhaopeng Tu; name: Xiaolong Li; name: Linus
Published: 2025-09-30 16:52:14

Abstract: The rise of Large Language Models (LLMs) is reshaping multimodel models, with
speech synthesis being a prominent application. However, existing approaches
often underutilize the linguistic intelligence of these models, typically
failing to leverage their powerful instruction-following capabilities. This
limitation hinders the model's ability to follow text instructions for
controllable Text-to-Speech~(TTS). To address this, we propose a new paradigm
inspired by ``operationalism'' that decouples instruction understanding from
speech generation. We introduce BatonVoice, a framework where an LLM acts as a
``conductor'', understanding user instructions and generating a textual
``plan'' -- explicit vocal features (e.g., pitch, energy). A separate TTS
model, the ``orchestra'', then generates the speech from these features. To
realize this component, we develop BatonTTS, a TTS model trained specifically
for this task. Our experiments demonstrate that BatonVoice achieves strong
performance in controllable and emotional speech synthesis, outperforming
strong open- and closed-source baselines. Notably, our approach enables
remarkable zero-shot cross-lingual generalization, accurately applying feature
control abilities to languages unseen during post-training. This demonstrates
that objectifying speech into textual vocal features can more effectively
unlock the linguistic intelligence of LLMs.