CapSpeech: Enabling Downstream Applications in Style-Captioned
  Text-to-Speech
Authors: name: Helin Wang, name: Jiarui Hai, name: Dading Chong, name: Karan Thakkar, name: Tiantian Feng, name: Dongchao Yang, name: Junhyeok Lee, name: Laureano Moro Velazquez, name: Jesus Villalba, name: Zengyi Qin, name: Shrikanth Narayanan, name: Mounya Elhiali, name: Najim Dehak

Recent advancements in generative artificial intelligence have significantly
transformed the field of style-captioned text-to-speech synthesis (CapTTS).
However, adapting CapTTS to real-world applications remains challenging due to
the lack of standardized, comprehensive datasets and limited research on
downstream tasks built upon CapTTS. To address these gaps, we introduce
CapSpeech, a new benchmark designed for a series of CapTTS-related tasks,
including style-captioned text-to-speech synthesis with sound events
(CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS
(EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech
comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36
million human-annotated audio-caption pairs. In addition, we introduce two new
datasets collected and recorded by a professional voice actor and experienced
audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside
the datasets, we conduct comprehensive experiments using both autoregressive
and non-autoregressive models on CapSpeech. Our results demonstrate
high-fidelity and highly intelligible speech synthesis across a diverse range
of speaking styles. To the best of our knowledge, CapSpeech is the largest
available dataset offering comprehensive annotations for CapTTS-related tasks.
The experiments and findings further provide valuable insights into the
challenges of developing CapTTS systems.