CLAPSpeech: Learning Prosody from Text Context with Contrastive
  Language-Audio Pre-training
Authors: name: Zhenhui Ye; name: Rongjie Huang; name: Yi Ren; name: Ziyue Jiang; name: Jinglin Liu; name: Jinzheng He; name: Xiang Yin; name: Zhou Zhao
Published: 2023-05-18 07:07:04

Abstract: Improving text representation has attracted much attention to achieve
expressive text-to-speech (TTS). However, existing works only implicitly learn
the prosody with masked token reconstruction tasks, which leads to low training
efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
cross-modal contrastive pre-training framework that explicitly learns the
prosody variance of the same text token under different contexts. Specifically,
1) We encourage the model to connect the text context with its corresponding
prosody pattern in the joint multi-modal space with the elaborate design of the
encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
pipeline to capture prosody patterns in multiple levels. We show how to
incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
on three datasets not only show that CLAPSpeech could improve the prosody
prediction for existing TTS methods, but also demonstrate its generalization
ability to adapt to multiple languages and multi-speaker TTS. We also deeply
analyze the principle behind the performance of CLAPSpeech. Ablation studies
demonstrate the necessity of each component in our method. Source code and
audio samples are available at https://clapspeech.github.io.