CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training
Year: 2023

Abstract:
Improving text representation has attracted much attention to achieve expressive text-tospeech (TTS). However, existing works only implicitly learn the prosody with masked token reconstruction tasks, which leads to low training efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that explicitly learns the prosody variance of the same text token under different contexts. Specifically, 1) We encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space with the elaborate design of the encoder inputs and contrastive loss; 2) We introduce a multi-scale pretraining pipeline to capture prosody patterns in multiple levels. We show how to incorporate CLAPSpeech into existing TTS models for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker TTS. We also deeply analyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in our method. Source code and audio samples are available at https://clapspeech.github.io.

heading: 1 Introduction
text: With the development of deep learning, the audio quality of modern TTS systems has been improved, yet prosody modeling is still a challenging problem. Previous works on expressive TTS have utilized external variation predictors (prediction-based, PB)
∗Equal contribution. †Corresponding author.
(Ren et al., 2021a) and variational generative models (variation-based, VB) (Kim et al., 2020; Liu et al., 2022) to inject prosody variance into the TTS model. Another popular direction is to learn better text representation for prosody prediction (Tan et al., 2021). However, the existing text representation learning methods for TTS are either based on the masked language model task (Devlin et al., 2019; Jia et al., 2021; Chen et al., 2021) (i.e., learn a BERT-like large language model on a text corpus) or masked acoustic model task (Chen et al., 2020; Bai et al., 2022) (i.e., reconstruct the masked mel-spectrogram based on the input text), which result in two disadvantages. Firstly, they only implicitly learn prosody with reconstruction losses, which distracts the model from improving the prosody modeling. Secondly, they do not decouple the pronunciation space and prosody space, which leads to low training efficiency and a waste of model capacity. We perform a case study in Section 4.3.1, in which we can see that previous text representation used in TTS cannot capture the prosody variance under different text contexts.
Technically, prosody can be regarded as the pitch and duration variance of the same token under different conditions (such as text contexts and speakers) (Tan et al., 2021). This paper mainly studies the prosody correlated to the text context. For instance, for the same word "higher", saying "higher up" or "slightly higher" can lead to different prosodies. Inspired by recent cross-modal contrastive learning works in the text-to-image task (Radford et al., 2021; Elizalde et al., 2022), we propose a contrastive learning method that connects the text context and the high-level prosody pattern in the text-speech joint multi-modal space, namely Contrastive Language-Audio Pre-Training
9317
for Text-to-Speech (CLAPSpeech). Specifically, we learn a text encoder to predict the prosody from the text context and a prosody encoder to extract the ground-truth (GT) prosody from the speech segment of the selected token. During training, we select N text-speech pairs that contain the same pronounceable token (e.g., the word "higher" or phoneme "AE0"). By aligning the text token with its corresponding prosody (extracted from GT speech) and pushing away the prosody representation from other text contexts, the text encoder is encouraged to extract prosody from the text context. An intuitive example of pre-training CLAPSpeech can be found in Figure 1. We also observe that the prosody pattern can be expressed at multiple levels. Therefore, we propose a multi-scale pre-training framework that learns two CLAPSpeech models to capture the prosody information at the phoneme and word levels, respectively. After the pre-training stage, our CLAPSpeech can be regarded as a plugin text encoder applicable to all TTS models to provide fine-grained prosody representation.
To prove the effectiveness and generalizability of our approach, we use two large-scale automatic speech recognition (ASR) datasets (LibriSpeech (Panayotov et al., 2015) for English and WenetSpeech (Zhang et al., 2022) for Chinese) to pretrain the CLAPSpeech model. The pre-trained text encoder of CLAPSpeech is then plugged into prediction/variation-based TTS baselines to demonstrate the improvement of CLAPSpeech to the exist-
ing expressive TTS systems. We then evaluate the performance on three TTS datasets, including one single-speaker English dataset, one single-speaker Chinese corpus, and one multi-speaker English dataset. Experiments on all datasets show that CLAPSpeech improves the prosody of the TTS models and outperforms previous representation learning methods.
To summarize, CLAPSpeech has three prominent advantages: 1) It can provide better prosody representation than previous representation learning methods with a much smaller model scale, thanks to its contrastive objective that explicitly learns the prosody. 2) The text representation of CLAPSpeech can be conveniently used in existing TTS systems, only with a minor modification of the front-end network architecture. 3) We also show its potential applications such as fine-grained prosody transfer in Section 4.3.2.

heading: 2 Related Work

heading: 2.1 Expressive TTS
text: In the past few years, modern neural TTS has made significant progress in high practicality and audio quality (Ren et al., 2019; Kim et al., 2020; Huang et al.; Elias et al., 2021; He et al., 2022; Miao et al., 2021; Kim et al., 2021; Donahue et al., 2021; Jiang et al., 2022; Huang et al., 2022c; He et al., 2023; Jiang et al., 2021; Huang et al., 2022b,a). However, modeling expressive prosody given the plain input text is still challenging. To achieve expressive
TTS, one common practice is to use a reference encoder and style tokens (Wang et al., 2018; Jia et al., 2018). But it is difficult to select appropriate reference audios during inference (Tan et al., 2021). Other works seek to improve prosody modeling with advanced network designs, which can be categorized into two classes: (1) the prediction-based (PB) TTS systems (Ren et al., 2021a) learn several external predictors to predict the prosody attributes such as pitch contour, duration, and energy; (2) the variation-based (VB) TTS systems leverage variational auto-encoder (VAE) (Ren et al., 2021b) or normalizing flow (Kim et al., 2020) to model the prosody in the latent space.
There are also some works that explore providing better text presentation with rich prior knowledge to help the prosody prediction. For instance, Liu et al. (2021) and Ye et al. (2022) incorporate syntax information through dedicated modeling methods such as graph networks. Representation learning methods for text pre-training and speech pre-training also show improvements in the prosody of TTS. We will discuss the representation learning works for TTS in the next section.

heading: 2.2 Representation Learning for TTS
text: Self-supervised pre-training methods have been leveraged in TTS to enhance text processing or speech generation capabilities (Chung et al., 2019; Zhang et al., 2019). Some early works (Wang et al., 2015) use pre-trained word embeddings to improve the robustness of TTS systems. Recently, some works explore incorporating pre-trained large masked language models (MLMs) (Devlin et al., 2019; Chen et al., 2021; Jia et al., 2021) to enjoy the rich semantic information learned from the webscale text corpus. However, the above-mentioned works only focus on the text space, it is challenging for them to model expressive prosody considering the models are unaware of the high variable prosody patterns in the speech space. There are several inspiring speech representation learning methods in ASR. Baevski et al. (2020) and Hsu et al. (2021) utilize masked continuous speech features to predict predetermined cluster assignments. As for TTS, ProsoSpeech (Ren et al., 2022) designs a word-level vector quantization bottleneck to extract discrete prosody representation from speech. Masked acoustic model (MAM) (Chen et al., 2020) proposes to learn a speech encoder that generates continuous speech (prosody) representations.
Specifically, during training they replace a span of speech spectrogram with mask tokens and learn to recover the masked spectrogram without text conditions. A3T (Bai et al., 2022) additionally learns a text encoder as auxiliary information for MAM to reconstruct the masked mel-spectrogram.
The difference between CLAPSpeech and previous representation works in TTS is obvious: While previous works implicitly learn the prosody information with the masked token reconstruction task, CLAPSpeech is the first work that utilizes the cross-modal contrastive learning to explicitly learn the context-correlated prosody, which leads to better prosody prediction and more efficient usage of model capacity.

heading: 3 CLAPSpeech
text: We propose CLAPSpeech, a cross-modal contrastive learning approach to provide better text representation for prosody prediction in TTS. As shown in Figure 1, CLAPSpeech comprises a text encoder and a prosody encoder, whose training objective is to connect the text token and the speech segment in the joint prosody space. In this section, we first design the network structure and input features of these two encoders. These elaborate designs enable the text encoder to effectively process the text context and ensure that the prosody encoder focuses on extracting the high-level prosody pattern from the speech segment while eliminating other variables, such as timbre. Then we introduce the multi-scale contrastive pre-training framework, which enables CLAPSpeech to capture prosody in both phoneme and word levels. Finally, we show how the pre-trained text encoder of CLAPSpeech can be conveniently plugged into modern TTS systems to improve prosody prediction. We describe these designs in detail in the following subsections and provide more technical details in Appendix A.

heading: 3.1 Text Encoder and Prosody Encoder
text: The prosody of the same pronounceable token1 varies in different text contexts. CLAPSpeech aims to model the correlation between the text context and the high-level prosody pattern. To this end, we design a text encoder and a prosody encoder to construct a text-speech multi-modal prosody embedding space.
As shown in Figure 2(a), the text encoder uses phoneme and byte pair encoding (BPE) (Shibata
1such as the phoneme "AE0" or the word "higher".
Phoneme Embedder
Conv1D + LN + ReLU x 3
Conv1D + LN + ReLU x 3 x 4
Attentive Pooling 1D [N, T, C]
[N, C] speech encoding
phoneme BPE
BPE Embedder
FFT FFT
FFT
WordPool + Word2Ph
text encoding
[N, T, C]
token encoding [N, C] token indexing
phoneme-level
WP if word-level
CLIP Text Encoder
higher
speech segment
(a) text encoder
Phoneme Embedder
Conv1D + LN + ReLU x 3
Conv1D + LN + ReLU x 3 x 4
Attentive Pooling 1D [N, T, C]
[N, C] speech encoding
phoneme BPE
BPE Embedder
FFT FFT
FFT
WordPool + Word2Ph
text encoding
[N, T, C]
token encoding [N, C] token indexing
phoneme-level
WP if word-level
CLIP Text Encoder
higher
speech segment
(b) prosody encoder
Figure 2: The text / prosody encoder of CLAPSpeech. In subfigure (a), "WP" and "Word2Ph" denotes word pooling and Word2Ph expanding operation, which are illustrated in Figure 3.
et al., 1999) of the input text as the input. The phoneme and BPE sequence help the model extract the prosody pattern related to phonological habits (such as the linking phenomenon in English) and semantic information (which may imply different emotional overtones), respectively. The network structure of the text encoder is composed of several Feed Forward Transformers (FFT) (Vaswani et al., 2017), which have proven the robustness in processing long text sequences in TTS models. Specifically, we learn two independent FFT blocks to process the phoneme and BPE sequences, respectively. This way, the phoneme FFT block could model the phonological habits in phonetic space, and the BPE FFT block could extract the semantic information. One difficulty is fusing the phoneme and BPE sequence of mismatched length. Instead of concatenating these two sequences in the time axis, we use word-level pooling (WP) from Ren et al. (2021b) to process the BPE encoding to the word level, then expand it to the phoneme level (namely the word2ph operation). To be specific, as shown in Figure 3(a), the WP operation averages the phoneme hidden states inside each word according to the word boundary, and the word2ph operation repeats the word hidden states for each phoneme insides the word boundary as illustrated in Figure 3(b).
Once the phoneme sequence and BPE seqneuce is fused, we then use an additional FFT block to fuse the aligned phoneme and BPE encoding to get the final phoneme-level text encoding. During the
HH AE1 Z | N EH1 V ER0
Phoneme/BPE Encoder
WP
HH AE1 Z | N EH1 V ER0
Word2Ph
has never
has never (Phoneme/BPE-level Sequence)
(Phoneme-level Sequence)
(Word-level Sequence)
(Word-level Sequence)
(a) Word Pooling
HH AE1 Z | N EH1 V ER0
Phoneme/BPE Encoder
WP
HH AE1 Z | N EH1 V ER0
Word2Ph
has never
has never (Phoneme/BPE-level Sequence)
(Phoneme-level Sequenc )
(Word-level Sequence)
(Word-level Sequenc )
(b) Word2Ph Expanding
Figure 3: The word pooling and word2ph expanding operation.
pre-training phase, since only one selected token is analyzed, we index from the phoneme-level text encoding to obtain the encoding of the selected token (namely the token encoding in Figure 2(a)) and then linearly project it into the multi-modal embedding space. During the TTS phase, the phoneme-level output of the text encoder can be conveniently utilized as auxiliary features for TTS systems, which we will discuss in Section 3.3.
The prosody encoder aims to extract prosody patterns from the GT speech segment of the selected token. Therefore, we clip the mel-spectrogram with the word boundary2 as the input speech feature. Then the prosody encoder processes the input mel-spectrogram into a global encoding to be connected with the token encoding. Note that the clipped speech segment only contains the local prosody information for the selected token without leaking any contextual information. Thanks to the contrastive learning setting, the extracted global prosody encoding is disentangled from phonetic and speaker space: 1) since the positive sample and negative samples belong to the same pronounceable token, the phonetic information is eliminated; 2) as the speaker information is not provided to the text encoder3, the prosody encoder will filter out speaker information to maximize the prosody information in the output features during training. This way, by connecting the context-aware text encoding with the context-unaware mel encoding, on the one hand, the prosody encoder learns to extract the high-level prosody information from the
2We extract word boundary with a forced alignment tool. 3We assume that text and speaker are independent of each
other (no correlation between them) in our dataset.
speech segment; on the other hand, the text encoder is encouraged to utilize the text context to predict the prosody extracted by the prosody encoder. As shown in Figure 2(b), we use ResNet-50 (He et al., 2016) as the backbone of the prosody encoder due to its robustness. We make several modifications to the original version: 1) to better process the melspectrogram, we use 1D convolution with layer normalization to build the fundamental residual block; 2) to handle the speech segment of dynamic lengths, we use an attentive pooling layer from Radford et al. (2021) to aggregate the output feature map of the ResNet.

heading: 3.2 Multi-scale Contrastive Pre-training
text: The key idea of CLAPSpeech is to model the prosody variance of the same text token under different contexts. Therefore, to construct a minibatch for contrastive pre-training, we randomly select a text token, then sample a batch of N text-speech pairs that contain the selected token (one intuitive sample is shown in Figure 1, where we sample the text-speech pairs that contain the word "higher"). To better extract prosody variance at the phoneme and word level, we introduce a multi-scale contrastive training framework. To be specific, we learn two CLAPSpeech models for phoneme-level and word-level text tokens, respectively.
For clarity, we first illustrate the training process of phoneme-level CLAPSpeech. Let the text context that contains the selected phoneme token (e.g., "AE0") be represented by Xtext. Let the processed speech segment of the phoneme token be Xspeech s.t. Xspeech ∈ RF×T , where F is the number of Mel bins and T is the number of time bins. For simplicity, we use Xtext and Xspeech to represent a batch of N text-speech pairs.
The text and speech are passed through the text encoder ftext(·) and prosody encoder fspeech(·), respectively. As can be seen in Figure 2(a), the output of the text encoder ftext(Xtext) is the phonemelevel encoding of the input text, hence we index from it to obtain the encoding of the phoneme token ftext(Xtext)iph , where iph denotes the index of the phoneme token in the phoneme-level text sequence. As can be seen in Figure 2(b), the output speech encoding fspeech(Xspeech) is a global representation of the input speech segment. The output representations are normalized and then linearly
projected into the multi-modal embedding space:
Tph = Ltext(LN(ftext(Xtext)iph))
S = Lspeech(LN(fspeech(Xspeech))), (1)
where Tph ∈ RN×C is the phoneme token representation and S ∈ RN×C is the speech representation of channel size C. LN means layer normalization, Ltext and Lspeech are linear projections.
Now that the text and speech embeddings are comparable, CLAPSpeech is trained to predict which of the N × N possible text-speech pairings across a batch actually occurred. Specifically, the text encoder and prosody encoder are encouraged to maximize the cosine similarity of the text and speech encoding of the N real pairs in the batch while minimizing the cosine similarity of the embeddings of the N2 −N incorrect pairings. Following Radford et al. (2021), we optimize a symmetric cross-entropy loss over these similarity scores:
Lph = 0.5×(ltext(τ ·Cph)+ lspeech(τ ·Cph)) (2)
where Cph ∈ RN×N is the cosine similarity matrix between the phoneme token encoding Tph and the speech encoding S, measured by Cph = Tph · ST ; τ is a learnable temperature parameter to scale the range of logits; and lk = 1 N Σ N i=0 log diag(softmax(C)) is the cross entropy function along the text and speech axis in C. The word-level CLAPSpeech can be trained similarly. As shown in Figure 2(a), for the word-level CLAPSpeech, we use word pooling to process the phoneme-level text encoding into word level, then index from it to obtain the word token encoding Tword. Similar to Equation 2, the training loss for word-level CLAPSpeech is formulated as:
Lword = 0.5×(ltext(τ ·Cword)+lspeech(τ ·Cword)) (3) where Cword is the cosine similarity matrix between the word token encoding Tword and the speech encoding S.

heading: 3.3 CLAPSpeech Plugged in TTS Systems
text: The text encoder of CLAPSpeech could provide text representation with rich prosody information for the TTS task. Since the generated text representation is at the phoneme level, which is in line with the majority of current TTS models that also utilize phoneme sequence as the text input, CLAPSpeech can be a convenient plugin unit for TTS systems to improve prosody prediction. Specifically, we
take a state-of-the-art variation-based TTS system, PortaSpeech, as an example. As shown in Figure 4, the pre-trained text encoders of CLAPSpeech (marked with a red dashed rectangle) perform as an auxiliary encoder to the original phonetic encoder of PortaSpeech. The phoneme-level outputs of the phonetic encoder and CLAPSpeech text encoder are fused and processed by the following encoder. Note that we fix the parameters of CLAPSpeech text encoders during the training of the TTS system to avoid overfitting. CLAPSpeech can be easily plugged into other TTS systems in a similar way. To demonstrate the universality, we illustrate how to combine CLAPSpeech with a widely-used prediction-based TTS system, FastSpeech 2, in Appendix A.1. We additionally adopt multi-length adversarial training in TTS models to improve audio quality. More details about the the adversarial training can be found in Appendix A.2.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Datasets and Baselines We pre-train CLAPSpeech on two ASR datasets: 1) LibriSpeech (Panayotov et al., 2015), an English database that contains 982 hours of speech from 2484 speakers; 2) WenetSpeech (Zhang et al., 2022), a Chinese speech corpus consisting of 10,000 hours of
speech4. Then we evaluate the pre-trained CLAPSpeech on three TTS datasets: 1) LJSpeech (Ito and Johnson, 2017), a single-speaker database that contains 13,100 English audio clips with a total of nearly 24 hours of speech; 2) Biaobei5, a Chinese speech corpus consisting of 10,000 sentences (about 12 hours) from a Chinese speaker; 3) LibriTTS (Zen et al., 2019), an English dataset with 149,736 audio clips (about 245 hours) from 1,151 speakers (We only use train clean360 and train clean100). The raw text is transformed into phoneme and BPE sequences using open-sourced tools. The GT mel-spectrograms are generated from the raw waveform with a frame size of 1024 and the hop size of 256. We compare CLAPSpeech against two pre-training baselines (BERT (Devlin et al., 2019) and A3T (Bai et al., 2022)) in a prediction-based (PB) TTS model, FastSpeech 2, and a variation-based (VB) TTS model, PortaSpeech.
Model Configuration CLAPSpeech consists of a text encoder and a prosody encoder, whose structures are shown in Figure 2 and discussed in Section 3.2. As for the PB and VB TTS models, we use the same structure in the original papers with an additional multi-length discriminator to improve audio quality. The multi-length discriminator consists of multiple stacked convolutional layers with batch normalization and treats the input spectrogram as images. We put more detailed model configurations in Appendix B.1.
Training and Evaluation Our approach is implemented with Pytorch. We pre-train CLAPSpeech on 4 Nvidia 3090Ti GPUs with a batch size of 1,024 text-speech pairs (256 pairs per GPU). We use the Adam optimizer with an initial learning rate of 0.0005. We train the CLAPSpeech model for 640,000 iterations (which takes about 1 week) and follow the cosine learning rate schedule in CLIP. Then we train the TTS models on 1 Nvidia 2080Ti GPU with a batch size of 64 sentences, following the learning rate schedule in Vaswani et al. (2017). We use HiFi-GAN (Kong et al., 2020) as the vocoder. We conduct the mean opinion score (MOS) and comparative mean opinion score (CMOS) evaluation to measure the prosody and audio quality. Details about the subjective evaluation
4We filter samples with a correctness confidence level above 0.95, finally get a subset of 1000 hours.
5https://www.data-baker.com/opensource.html
can be found in Appendix B.2. As for the objective evaluation, following Ren et al. (2021b), we evaluate the prosody from the aspects of pitch and duration: 1) we compute the average dynamic time warping (DTW) (Muller, 2007) distances between the pitch contours of GT speech and synthesized speech to measure the pitch accuracy; 2) we calculate the average absolute duration error (DE) in micro-seconds6 to measure the duration accuracy.

heading: 4.2 Performance
text: We compare the performance of our CLAPSpeech against BERT and A3T in PB/VB TTS models. GT (the ground-truth audio) and GT (voc.) (the audio waveform generated by the vocoder using the GT mel-spectrogram) are also included in the experiment. We perform the TTS experiments on three datasets as mentioned in Section 4.1. The results are shown in Table 1. We can see that CLAPSpeech outperforms other representation learning methods in both PB and VB TTS baselines in terms of MOS, pitch accuracy, and duration accuracy, which proves that CLAPSpeech could effectively improve the prosody prediction in current expressive TTS models (no matter prediction-based or variationbased). Besides, we observe that CLAPSpeech achieves better performance than BERT and A3T with much fewer model parameters. We suspect it is due to the fact that the MLM-based method (i.e., BERT) require a large model capacity to store the semantic information and MAM-based method (i.e., A3T) have to jointly learn the phonetic information to reconstruct the masked mel-spectrogram. By contrast, our CLAPSpeech eliminates the phonetic space and only focus on the prosody space during pre-training, which is parameter-efficient.
We then visualize the mel-spectrograms generated by different methods in Figure 5. We can see that CLAPSpeech can generate results with more realistic pitch contours, which result in expressive prosody. In conclusion, our experiments demonstrate that CLAPSpeech could help TTS systems synthesize more expressive and prosodic audio.

heading: 4.3 Deeper Analysis

heading: 4.3.1 Token Representation Self-similarity
text: To better understand the performance superiority of CLAPSPeech over existing representation learning methods for TTS, we analyze the token represen-
6In our PB/VB TTS baseline, the duration is predicted in phoneme/word level, respectively.
tation learned by CLAPSpeech and other methods. Following Su et al. (2021), we define the averaged similarity on the selected token under different contexts T = [T1, ..., TN ] as,
s(T ) = 1 N(N − 1) N∑
i=1
N∑
j=1,j ̸=i cosine(Ti, Tj)
(4) where Ti and Tj are the selected token’s encoding extracted by the model from different text contexts. Intuitively, a lower s(T ) indicates that the selected token itself plays a smaller role in generating its representation, which means that the model captures more context-related information from the input text sequence, and thus predicts better prosody.
Quantitative Evaluation We sample 10,000 batches (each batch consists of 256 sentences that contain the same selected token) from the ASR validation datasets and compute the averaged selfsimilarity. The result is shown in Table 2. We observe that our CLAPSpeech learned with the contrastive objective (in Equation 2) achieves the lowest similarity in the off-diagonal entries of the similarity matrix, which denotes that the model has made use of the text context to capture the prosody variance of the same token, thus achieve the best prosody performance in Table 1. Besides, we can see that BERT also achieves a relatively low offdiagonal similarity, which is due to its MLM task during pre-training, in which the model needs to extract semantic information from context to predict the masked token. By contrast, the vanilla TTS text encoder and A3T fail to achieve a low offdiagonal similarity, which means that both models cannot extract discriminative information from different contexts. We suspect the failure of A3T is due to the fact that its MAM objective encourages the model to predict the masked mel-spectrogram patch based on the input unmasked text sequence, which increases the model’s demand for phonetic information of the selected token.
Qualitative Evaluation We sample 8 sentences7 that contain the word "higher" from LibriSpeech and visualize the self-similarity matrix M (where Mi,j = cosine(Ti, Tj)) produced by CLAPSpeech and vanilla TTS text encoder. The results are shown in Figure 6, where a darker color denotes a higher self-similarity score. We also provide the selfsimilarity matrix of BERT and A3T in Figure 9
7We list these sentences in Table 5 of Appendix C.
of Appendix C. We can see that the self-similarities of CLAPSpeech are much lower in the off-diagonal entries.

heading: 4.3.2 Fine-grained Prosody Transfer
text: We perform an intuitive case study about prosody transfer to further validate that our CLAPSpeech’s text-speech joint multi-modal space represents high-level prosody patterns (i.e., the pitch contours and duration information). We take s7/8 in Table 5 as the reference/source audio and expect to transfer the word "higher"’s prosody pattern from s7 to s8. Specifically, we use the text encoder of CLAPSpeech to extract the text prosody encoding of s7 and s8, then replace the text token encoding of "higher" in s8 with that in s7. As shown in Figure 7, the prosody pattern of "higher" in s88 in Figure 7(a) has been successfully transferred into s7 in
8the pitch contours in reference remain flat in the early stage and then rise in the late stage
Figure 7(c). We also provide audio samples of this case study on our demo page. The manipulation of the local prosody proves that our CLAPSpeech extract prosody representation effectively influences the prosody prediction of the TTS system.

heading: 4.4 Ablation Studies
text: Use BPE as Auxiliary Features We first analyze the effectiveness of the BPE as an auxiliary feature to help extract prosody information from the text context. During the pre-training phase of CLAPSpeech, we found removing BPE from the
text encoder significantly degrades the validation CLIP loss from 0.3692 to 0.6764. Then in the TTS phase, as can be seen in line 3 in Table 3, the ablated model using the pre-trained text encoder without BPE leads to a performance drop in terms of CMOS, DTW, and DE. This is possibly due to the fact that BPE could better represent the semantic information than the low-level phoneme sequence.
Multi-scale Pre-training To demonstrate the effectiveness of multi-scale pre-training, as can be seen in line 4/5 in Table 3, we tried to remove phoneme-level or word-level CLAPSpeech from the model, which leads to a worse prosody performance. We also tried to use the untrained CLAPSpeech to prove the necessity of the pre-training process, and we found this ablated model (line 6) achieves a slightly worse performance than the TTS baseline (line 3).

heading: 5 Conclusion
text: In this paper, we propose CLAPSpeech, a crossmodal contrastive pre-training framework that provides better text representation with rich prosody information for TTS. With the design of a text encoder and a prosody encoder, CLAPSpeech learns to connect the text context with its corresponding prosody pattern in the speech. We also introduced multi-scale pre-training to extract prosody patterns at multiple levels. We have demonstrated the performance and generalization ability of CLAPSpeech
on three TTS datasets (English, Chinese, and multispeaker, respectively). We have also deeply analyzed the principle behind the improvement of CLAPSpeech and performed ablation studies to prove the necessity of each component.

heading: 6 Limitations
text: There are majorly two limitations: Firstly, in this work, we only consider the current-sentence text context-related prosody. In future work, we will focus on improving the inter-sentence prosody to achieve coherent, expressive TTS for long-form text. Secondly, other variables are not considered during the contrastive pre-training. One can explore similar approaches that connect prosody to other conditions such as speaker, emotion, etc.

heading: 7 Ethics Statement
text: CLAPSpeech improves the prosody of the synthesized speech, which may cause unemployment for people with related occupations. Besides, the production of fake speeches may cause voice security issues. Further efforts in automatic speaker verification should be made to improve voice security.

heading: 8 Acknowledgment
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000,National Natural Science Foundation of China under Grant No. 62222211 and Grant No.61836002 and Grant No.62072397, and Yiwise.

heading: A Details of Models
text: A.1 CLAPSpeech plugged in FastSpeech 2
We show how to integrate CLAPSpeech into a popular prediction-based TTS system, FastSpeech 2. As shown in Figure 8, the pre-trained text encoders of CLAPSpeech (marked with a red dashed rectangle) perform as an auxiliary encoder to the original phonetic encoder of FastSpeech 2. The phonemelevel outputs of the phonetic encoder and CLAPSpeech text encoder are fused and processed by the following encoder. Note that we fix the parameters of CLAPSpeech text encoders during the training of the TTS system to avoid overfitting.
A.2 Multi-length Adversarial Training
For the tested TTS baselines, we adopt an additional multi-length discriminator to provide a least squared GAN loss to improve the audio quality. The multi-length discriminator is an ensemble of multiple CNN-based discriminators which evaluates the mel-spectrogram based on random windows of different lengths. One could refer to Ye et al. (2022) for more details.

heading: B Detailed Experimental Settings
text: B.1 Model Configurations
We list the hyper-parameters of CLAPSpeech and the tested TTS baselines in Table 4.
B.2 Subjective Evaluation
For each tested dataset, we randomly select 10 texts from the test set and use the TTS systems to generate the audio samples. Each audio has been listened to by at least 20 native listeners, who are
recruited on a crowdsourcing platform, Zhengshu Technology. We tell listeners to "focus on examing the naturalness of prosody (e.g., pitch, energy, and duration) and audio quality (noise, timbre, sound clarity, and high-frequency details)". For MOS, each tester is asked to evaluate the subjective naturalness of a sentence on a 1-5 Likert scale. For CMOS, listeners are asked to compare pairs of audio generated by systems A and B and indicate which of the two audio they prefer and choose one of the following scores: 0 indicating no difference, 1 indicating small difference, 2 indicating a large difference, and 3 indicating a very large difference.

heading: C More Details in Analysis
text: C.1 Example Sentences We list the 8 example sentences in Table 5. These sentences are used as examples in Section 4.3.
C.2 Self-similarity of Other Baselines The self-similarity visualization of A3T and BERT can be found in Figure 9. We discuss the results in Section 4.3.1.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 6.Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: 7.Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 1.Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 4.Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4.1 Experimental Setup & Appendix B.1 Model Configurations
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4.1 Experimental Setup

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.2 & 4.3 & 4.4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.1 Experimental Setup
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.2

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix B.2

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix B.2

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix B.2

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Appendix B.2

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix B.2

heading: Future Work
text: In future work, we will focus on improving the inter-sentence prosody to achieve coherent, expressive TTS for long-form text. Secondly, other variables are not considered during the contrastive pre-training. One can explore similar approaches that connect prosody to other conditions such as speaker, emotion, etc