BatonVoice: An Operationalist Framework for Enhancing Controllable
  Speech Synthesis with Linguistic Intelligence from LLMs
Authors: name: Yue Wang, name: Ruotian Ma, name: Xingyu Chen, name: Zhengliang Shi, name: Wanshun Chen, name: Huang Liu, name: Jiadi Yao, name: Qu Yang, name: Qingxuan Jiang, name: Fanghua Ye, name: Juntao Li, name: Min Zhang, name: Zhaopeng Tu, name: Xiaolong Li, name: Linus
Organization: Tencent

The rise of Large Language Models (LLMs) is reshaping multimodel models, with
speech synthesis being a prominent application. However, existing approaches
often underutilize the linguistic intelligence of these models, typically
failing to leverage their powerful instruction-following capabilities. This
limitation hinders the model's ability to follow text instructions for
controllable Text-to-Speech~(TTS). To address this, we propose a new paradigm
inspired by ``operationalism'' that decouples instruction understanding from
speech generation. We introduce BatonVoice, a framework where an LLM acts as a
``conductor'', understanding user instructions and generating a textual
``plan'' -- explicit vocal features (e.g., pitch, energy). A separate TTS
model, the ``orchestra'', then generates the speech from these features. To
realize this component, we develop BatonTTS, a TTS model trained specifically
for this task. Our experiments demonstrate that BatonVoice achieves strong
performance in controllable and emotional speech synthesis, outperforming
strong open- and closed-source baselines. Notably, our approach enables
remarkable zero-shot cross-lingual generalization, accurately applying feature
control abilities to languages unseen during post-training. This demonstrates
that objectifying speech into textual vocal features can more effectively
unlock the linguistic intelligence of LLMs.