Contrastive Context-Speech Pretraining for Expressive Text-to-Speech Synthesis

Authors: yujia  xiao
xu  tan
xinfa  zhu
sheng  zhao
tan  lee
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:12:03

Abstract: The latest Text-to-Speech (TTS) systems can produce speech with voice quality and naturalness comparable to human speech. Yet the demand for large amount of high-quality data from target speakers remains a significant challenge. Particularly for long-form expressive reading, target speaker's training speech that covers rich contextual information are needed. In this paper a novel design of context-aware speech pre-trained model is developed for expressive TTS based on contrastive learning. The model can be trained with abundant speech data without explicitly labelled speaker identities. It captures the intricate relationship between the speech expression of a spoken sentence and the contextual text information. By incorporating cross-modal text and speech features into the TTS model, it enables the generation of coherent and expressive speech, which is especially beneficial when there is a scarcity of target speaker data. The pre-trained model is evaluated first in the task of Context-Speech retrieval and then as the integral part of a zero-shot TTS system. Experimental results demonstrate that the pretraining framework effectively learns Context-Speech representations and significantly enhances the expressiveness of synthesized speech. Audio demos are available at: https://ccsp2024.github.io/demo/.

Main paper: 
 introduction: 
 Text-to-Speech (TTS) has experienced significant advancements in recent years, driven by breakthroughs in artificial intelligence, particularly in the field of deep learning. The autoregressive TTS framework, exemplified by the combination of the Tacotron series [25, 31] and WaveNet [17] , has significantly improved the naturalness of synthesized speech. Subsequently, non-autoregressive models [10, 12, 21, 23, 24] drastically accelerating the speech synthesis process while retaining high quality. Recently, zero-shot TTS system have emerged [2, 26, 30] , pushing the envelope further by synthesizing voices they have never been trained on directly.Most of these models are capable of generating speech quality close to the human level, but they mainly focus on single-sentence speech modeling. There are many scenarios requiring speech synthesized in a context-aware manner, such as audiobook, conversational speech, and long-form news reading. A straightforward solution is to leverage external information from contextual data into TTS modeling [7, 33, 35] . However, this approach demands a significant amount of high-quality, long-form voice data, which is extremely scarce and hard to obtain for most target speakers. Additionally, given contextual text information, constructing the correlation between it and the current sentence's speech expression is a non-trivial task. As prosody features are commonly used to represent speech expression, many works focus on predicting prosody information from the input text [19, 23, 28] to improve TTS performance. Nonetheless, efforts to model the prosody considering context beyond the current sentence [34] remain limited due to the requirement for long-form data from the target speaker.In this paper, we propose a Contrastive Context-Speech Pretraining (CCSP) framework to learn cross-modal representations invloving information from both contextual text and current speech expression. As shown in Figure 1 , the CCSP model is trained by a large amount of contextual voice data. With no requirement on explicit speaker identification, the CCSP model can leverage as more as possible contextual voice data from different speakers. For example, Librivox [9] provides huge number of public domain audiobook data read by volunteers from around the world. By leveraging extensive contextual voice data into cross-modal representation learning, the CCSP can align the features of context and speech within a shared space, resulting in enriched feature representations. For instance, the text modality features derived from the surrounding context encapsulate speech representation, and conversely, the speech modality features of the spoken sentence are imbued with contextual information.After obtaining the pretrained CCSP model, we can inject contextual information into downstream TTS model for any target speaker without considering if there is enough contextual voice data. Figure 1 illustrates the strategic utilization of CCSP model in different stages of TTS model. In the TTS training phase, where paired speech-text data is available, CCSP model is used to generate context-aware speech modality features for the current sentence. These enriched speech modality features enable the TTS system to learn how to produce speech that not only aligns with the content but also reflects the surrounding context. During the inference stage, where the TTS model must generate speech solely from text, the CCSP model comes into play by providing text modality features that capture the speech expression related information. These text modality features effectively replace the speech modality features used in the training phase, resulting in context-aware speech synthesis with a heightened level of expressiveness.The key contributions of our work can be summarized as:• We propose a contrastive context-speech pretraining (CCSP) framework to learn cross-modal representations containing both contextual text information and the speech expression of the spoken sentence. • By integrating cross-modal information from the CCSP model into the downstream TTS system, we can create contextaware voice generation model for any speaker, circumventing the limitation of obtaining sufficient contextual voice data from the target speaker. • We conduct context-speech retrieval and downstream TTS tasks to evaluate the proposed CCSP model. Experiments demonstrate that the CCSP model is able to learn effective cross-modal representations, and that integrating these learned features enhance the expressiveness of the downstream TTS model, particularly in long-form reading. 
related work 2.1 cross-modal feature learning: 
 Cross-modal feature learning is a powerful approach for bridging the semantic gap between various data types, like text and images, or text and audio. By learning representations that capture the underlying relationships across modalities, these models can perform a wide range of tasks that require an understanding of multiple types of input. A representative work of cross-modal feature learning is CLIP [22] (Contrastive Language-Image Pretraining). It leverages a dual-encoder architecture comprising: an image encoder and a text encoder, which are trained to project images and text into a shared embedding space. Using a contrastive loss function [18] , CLIP is trained on a diverse and large-scale dataset of image-text pairs to maximize the similarity between corresponding image-text pairs while minimizing it for non-corresponding ones. Similar cross-modal feature learning works between language and speech are wav2vec 2.0 [1] , HuBERT [8] , and w2v-BERT [4] . These work have underscored the ability to utilize extensive speech data to enhance speech processing tasks, particularly in the domain of ASR (Automatic Speech Recognition). These cross-modal features tend to prioritize the extraction of semantic features from speech while potentially underrepresenting supra-segmental features beyond semantics like prosody. For instance, given the same sentence spoken by different speakers or in different expression manner, the goal of an ASR system is to obtain the same text output. Conversely, as a one-to-many task, TTS systems need to produce correct pronunciation and replicate the spoken way of those words.CLAP (Contrastive Language-Audio Pretraining) [5, 32] is a crossmodal feature learning framework designed for text and audio, using specialized pre-trained encoders. Compared to the previously mentioned speech encoding approach, the audio encoder [3] utilized in [32] captures a wider range of shifts and patterns by interpreting audio spectrograms as visual images. Additionally, the text input in such language-audio pretraining tasks is typically a description rather than a transcription of the speech, which extends the modeling goal from local semantic understanding to global pattern description to some extent.The development of context-aware TTS systems has been driven by the quest to produce speech that mirrors the nuanced expressiveness of human speech across different contexts. This advancement is crucial for applications that demand high expressiveness, such as audiobook narration, where the system must convey the mood of the story, or in long-form content reading, where maintaining listener engagement through varied intonation is essential. Traditional TTS systems, while effective in generating intelligible speech, often fall short in capturing the emotional nuances and situational contexts that are intrinsic to natural speech. To bridge this gap, researchers have been exploring various methods to endow synthetic speech with a more dynamic and context-sensitive prosody. This includes techniques in local and global context modeling.Local context modeling in TTS focuses on information from the textual environment within the sentence or the phonetic details surrounding the speech segment. This includes modeling the nuances of pronunciation, intonation, and timing based on nearby words or segments. For instance, Char2Wav [27] represents an early approach to generate speech from characters by a sequence-tosequence model that accounts for local phonetic contexts. Recently, Conformer [6] combine convolutional layers for detailed local context modeling and enhances the expressiveness of generated voice for TTS system [16] . CLAPSpeech [36] employs the CLAP framework to learn prosody from text context, which is also focus on modeling local context at the phoneme/word level. These models primarily concentrate on local context modeling without considering broader context information beyond the current sentence.Global context modeling aims to capture information from an extensive scope beyond the current sentence, potentially encompassing preceding and succeeding contexts or the entire paragraph. This approach enables the TTS system to generate speech that is contextually appropriate and expressive. For instance, crosssentence information can be utilized by a Bert feature-based contextual encoder [7] . ParaTTS [35] and ContextSpeech [33] consider contextual information from both paragraph text and speech. Additionally, graph-based conversational TTS systems [13, 14] model cross-sentence information by treating them as connected graph nodes. These models involve global contextual information but require parallel contextual voice data from the target speaker. 
contrastive context-speech pre-training: 
 In this section, we will introduce the Contrastive Context-Speech Pretraining (CCSP) framework, which is illustrated in Figure 2 . It comprises three components: the context branch, the speech branch, and the contrastive loss mechanism. The speech branch focuses on processing and understanding the acoustic signals, while the context branch deals with the surrounding textual information. The contrastive loss functions as the bridge between the two, aligning the vector spaces of both modalities to facilitate cross-modal representation learning. The main objective is to learn the associative information between the context and the current speech. In the following subsections, we will provide a detailed exploration of each component within our system.To avoid concentrating on modeling the transcription-related semantic information during the feature alignment, we use the surrounding context (preceding and following) as the text input and the current sentence's text is omitted. This design is driven by:• In a TTS system, the input is text and the output is speech.During the TTS modeling process, the current sentence text is provided, carrying with it inherent semantic and prosodic information directly. There exist various explicit modeling methods designed to predict prosody from the text input.Our focus shift to learn the prosody relationship between surrounding context and the current speech. • Retaining the current sentence text would cause the crossmodal representation learning to be predominantly influenced by transcription-related semantic information. Our objective is to employ the context as a prompt to generate natural and expressive speech. Therefore, excluding the current sentence text force the modelling to focus on context & speech expression correlation.As shown in Figure 2 , the context branch takes the contexts (preceding 𝑊 words and following 𝑊 words) as input and then go through a pretrained text encoder to produce three context embeddings, denoted as 𝐸 𝐶 𝑃 , 𝐸 𝐶 𝐹 , and 𝐸 𝐶 𝐴 . Here, 𝐸 𝐶 indicates the embeddings produced by context branch. Based on that, 𝐸 𝐶 𝑃 represents the average text representation derived from the preceding context, 𝐸 𝐶 𝐹 corresponds to that from the following context, and 𝐸 𝐶 𝐴 is the combined average text representation obtained from both the preceding and following contexts. The pretrained text encoder we used in our experiments is RoBERTa-base [15] .In the speech modeling branch, the input speech is initially processed as audio data by the methodology described in [32] . At first, mel-spectrograms are extracted from the variable-length audio samples. These spectrograms are then passed through a feature fusion module to be integrated to a fix length (10s). To be more specific, if the input audio < 10𝑠, we first repeat the input audio and then pad it with zeros. If the input audio > 10𝑠, we downsample the audio to a global input of 10s. Additionally, we extract three separate 10s clips from the beginning, middle, and end 1 3 of the input to serve as local inputs. These four 10s audio segments are combined into a single feature by the feature fusion method described in [32] . The employed pre-trained audio encoder, HT-SAT model [3] , is an audio transformer architecture incorporating a hierarchical structure with a token-semantic module to obtain an effective audio representation, denoted as 𝐸 𝐴 . 3.2.2 Prosody Representation. To enrich the cross-modal representation with more supra-segmental information, we incorporate prosody features, pitch and duration, into the speech modeling branch. Initially, we extract frame-level pitch sequence and duration information. The prosody encoder is then utilized to generate positional prosody features as Equation 1. For each frame 𝑖, 𝑃 𝑣 𝑖 represents the pitch value. The pitch value is then processed through 𝑃𝐸, a convolution layer-based pitch encoding module, transforming the scalar pitch value into an 𝑚-dimensional vector. Additionally, 𝑃 𝑑 𝑖 indicates the phoneme index corresponding to frame 𝑖, and 𝑃𝑆 (𝑃 𝑑 𝑖 ) retrieves the positional embedding according to the method established by [29] . By summing these components, we acquire the positional prosody feature, 𝐸 𝑃 𝑖 . Finally, a contrastive loss function will be used to align the features from different modalities.EQUATIONEQUATIONEQUATIONAfter obtaining the frame-level positional prosody features, we employ a GRU layer to derive a global prosody feature, denoted as 𝐸 𝑃x , for speech segment x in Equation 2. {𝑥 1 , 𝑥 2 , ..., 𝑥 𝑇 } are frames belonging to x. Subsequently, the Audio & Prosody Feature for speech segment x, 𝐸 𝑆x , is acquired by concatenating the prosody feature (𝐸 𝑃x ) with the Audio feature (𝐸 𝐴 ), followed by a projection as in Equation 3. As depicted in Figure 2 , three Audio & Prosody Features are generated: 𝐸 𝑆 𝐵 , 𝐸 𝑆 𝐴 and 𝐸 𝑆 𝐸 , corresponding to speech segment "Beginning", "All" and "End". An assumption here is that the preceding context predominantly influences the beginning part of the current sentence, whereas the following context mainly impacts the end part of the current sentence. To be more specific, speech segments 𝐵, 𝐴 and 𝐸 refer to the first 𝑠 seconds, the entire sentence, and the last 𝑠 seconds, respectively.To effectively learn representations that integrate information from dual modalities, it is essential to align the feature spaces of the respective modal representations. Building on the approaches of [22, 32] , we simultaneously train the speech and context branches with the objective of maximizing the cosine similarity between the modality representations. Within a batch size of 𝑁 , we generate 𝑁 2 context-speech pairings. Each pair is processed through its corresponding branch, resulting in three textual features (𝐸 𝐶 𝑃 , 𝐸 𝐶 𝐴 , and 𝐸 𝐶 𝐹 ) and three speech features (𝐸 𝑆 𝐵 , 𝐸 𝑆 𝐴 , and 𝐸 𝑆 𝐸 ). This process forms 𝑁 2 pairings each for (𝐸 𝐶 𝑃 , 𝐸 𝑆 𝐵 ), (𝐸 𝐶 𝐴 , 𝐸 𝑆 𝐴 ), and (𝐸 𝐶 𝐹 , 𝐸 𝑆 𝐸 ). The goal for these pairings is to optimize the similarity scores for the 𝑁 intra-sample pairs while reducing similarity for the 𝑁 (𝑁 -1) inter-sample pairs. The loss function, 𝐿, based on symmetric cross entropy (𝑓 𝑆𝐶𝐸 ), is detailed in Equation 4, encapsulating this alignment objective. EQUATION4 Downstream Task 
context-speech retrieval: 
 As the CCSP model is designed to learn cross-modal representations between context and speech, cross-modal retrieval task is a straightforward way to verify the effectiveness of the learned representation. Context-to-Speech Retrieval task is selecting the most relevant context instance from a set {𝐸 𝐶 x,1 , 𝐸 𝐶 x,2 , ...} for a given speech representation, 𝐸 𝑆x,𝑖 . Conversely, Speech-to-Context Retrieval task is finding the most corresponding speech statement from a collection {𝐸 𝑆x,1 , 𝐸 𝑆 x,2 , ...}, given a context instance 𝐸 𝐶 x,𝑖 . Compared with evaluation on TTS system, these Context-Speech Retrieval tasks leverage the cross-modal representations produced by the CCSP model directly, obviating the need for an additional modeling process. Therefore, these two retrieval tasks can serve as a rapid verification method to help us validate and adjust the parameters of the CCSP model. They also offer an objective benchmark for further development and refinement.Given the pre-trained CCSP model, we utilize the NaturalSpeech 2 (NS2) [26] framework as the backbone TTS system to evaluate whether the pre-trained model enhances the expressiveness of the TTS model by delivering effective contextual information. Figure 3 illustrates the NS2 model's architecture and the way to inject crossmodal representations from CCSP model. The components of a baseline NS2 model can be described as: 1) Phoneme Encoder. This component encodes the input text into a phoneme sequence, which serves as a linguistic prior for the model. 2) Prosody (pitch/duration) Predictors. Alongside the phoneme encoder, prosody predictors estimate the duration of each phoneme and the pitch contour for the speech synthesis process. These components are crucial for achieving natural prosody and intonation in the synthesized speech. 3) Audio Codec. The audio codec is composed by a encoder, a residual vector-quantizer (RVQ) and a decoder. Codec encoder downsample the input audio and the RVQ converts the reuslt to a sequence of latent vectors. While the codec decoder reconstructs the speech waveform from these latent vectors. 4) Diffusion Model. The diffusion model generates the sequence of latent vectors conditioned on the phoneme sequence and the outputs from the duration and pitch predictors. 5) Speech Prompt Encoder. To facilitate the zero-shot capabilities, the model employs in-context learning, which allows the model to adapt to new speaker characteristics by being conditioned on a speech prompt. The speech prompt is randomly segmented from the ground truth speech during training. For inference, we utilize a speech segment less than 10 seconds from an unseen speaker as reference to generate synthesised samples for testing.The speaker embedding generated by the speech prompt encoder will influence both prosody feature prediction and sound reconstruction. Since prosody information is an essential part of the expressiveness of the generated voice, we incorporate crossmodal representations from CCSP into the speaker embedding to jointly affect the prosody feature predictor. More specifically, 𝐸 𝑆 𝐴 from the Speech Branch of CCSP serves as the injected embedding during training, while 𝐸 𝐶 𝐴 from the Context Branch is used during inference. This approach is taken because ground-truth speech is always available during TTS training, whereas context information is provided or can be generated by a GPT-service [20] during inference. Based on this injection strategy, we can conduct effective contextual modeling for any target speaker even without contextual voice data. 
evaluation 5.1 dataset: 
 LibriVox LibriVox [9] is a unique online project that harnesses the power of volunteers to create free, public domain audiobooks from texts that are also in the public domain. As audiobook data, Librivox contains a wealth of contextual voice data, which aligns with our requirements for training the CCSP model. We downloaded and processed approximately 15,000 hours of data, which LibriTTS-R We use LibriTTS-R [11] as the target data for our TTS task as it is a multi-speaker TTS corpus without ground-truth contextual information. LibriTTS-R is an enhanced version of the original LibriTTS [37] corpus. LibriTTS is a text-to-speech corpus crafted from LibriVox audiobooks. It is optimized for TTS research with features like 24kHz audio for better spectral content, precise text-to-audio alignment, and a diverse range of speakers and accents, primarily in English. LibriTTS-R improves the audio quality by applying speech restoration techniques to the original LibriTTS recordings, which contains 585 hours of data from 2,456 speakers.Objective Test We evaluate the performance of both Context-to-Speech Retrieval and Speech-to-Context Retrieval tasks using the objective metric mAP@10 (mean Average Precision at rank 10). The computation of mAP@10 is presented in Equation 6, where 𝑞 is the index of a query sample, 𝑄 is the total number of test samples, and 𝑝 𝑞 indicates the rank position of the correctly matched sample for the 𝑞 𝑡ℎ query after sorting by their corresponding logit values. For our experiments, we employed a test set reserved from the LibriVox-B dataset, comprising 𝑄 = 1024 samples.EQUATIONEQUATIONTo assess an expressive TTS system, we perform an objective test by computing the Normalized Pitch Dynamic Score. This metric captures the mean pitch variation at the syllable level and normalizes it to a 0-1 range using a sigmoid function. A greater normalized pitch dynamic score indicates a richer variation in pitch sequence, suggesting a higher level of expressiveness in the synthesized speech.Subjective Test For a perceptual assessment of the impact brought by CCSP features to the TTS model, we conduct subjective tests, including the Preference Test and Comparative Mean Opinion Score (CMOS). The Preference test is a blind listening test that assesses whether listeners can detect a difference between two audio samples and express a preference for one over the other. The outcome of this test is expressed in percentages, showing the proportion of test sample pairs where listeners preferred either system A or system B, along with the percentage of cases where there was 'No preference'. For the CMOS test, listeners compare the expressiveness of speech generated by two TTS systems. Unlike the preference test, CMOS test requires listeners to not only choose which sample they consider better but also to rate the extent of its superiority or preference on a scale from -3 to +3, with the reference samples assigned a score of 0. In our experimental setup, we engaged 9 native speakers to do the evaluation in a quiet environment.We create two test sets, a sentence-based test set with 50 cases and a paragraph-based test set with 30 cases. Each paragraph case is composed by 2 to 5 sentences. As paragraph tests involve longer test samples, which add to the complexity for reviewers, we offer higher rewards to participants of paragraph-based evaluations. The scripts for these tests were generated by the GPT-service [20] , ensuring a unbiased selection of content for evaluation. There could be several reasons for these observations. 1) While providing more context has the potential to furnish additional information for the model, the impact of this text tends to decrease as the distance from the target sentence grows, especially when we attempt to model the relationship between the context and the speech expression. 2) The actual context length of each sentence in the dataset might be limited. In Figure 6 , we present the distribution of the actual context length on the LibriVox-B dataset. The distribution reveals that a significant majority of sentences (> 40%) possess a context length less than 20. This observation may offer a plausible explanation for the experimental results. Context Length during Inference. Additionally, we investigated the impact of context length employed during the inference stage. Figure 5 demonstrates the results for two models, trained with context lengths of 20 and 80, respectively. It is apparent that both models perform optimally when the context length during inference matches that used at the training phase. Overall, The combination of a 20-20 (training-inference) context length outperformed other configurations. Considering these findings, we decided to proceed with the 20-20 context length configuration in our subsequent experiments. This is based not only on the performance but also on the advantage of saving memory. Prosody Length. To enclasp more speech expression related information, we incorporate prosody features in speech branch to enhance the cross-modal representation learning. As introduced in Section 3.2.1, the speech data with variant length are processed in a fixed duration of 10 seconds through a feature fusion function. The optimal duration of prosody features remains an additional parameter for analysis. As detailed in Section 3.2.2, three types of prosody features are extracted from the current sentence: "Beginning", "All", and "End". These are then fused with the audio feature. The "Beginning" prosody feature aligns with the preceding context and the "End" prosody feature aligns with the following context. Figure 7 contrasts the performance of CCSP models utilizing different prosody feature lengths, specifically 5s and 10s. The 10-second duration matches the length of the processed audio feature, whereas the 5-second duration is half as long. As shown in Figure 7 , the 5-second configuration significantly outperforms the 10-second setting in both Context-to-Speech and Speech-to-Context retrieval tasks. This suggests that: 1) longer prosody representations might present increased complexity and difficulty for effective modeling, 2) the influence of context on prosody is primarily concentrated around the immediate context-meaning the beginning part of the speech is most influenced by the preceding context, while the end part of the speech is most affected by the following context. Based on these findings, we will adopt a 5-second length for prosody feature in our subsequent experiments.Experimental Setting. As we mentioned in Section 4.2, we use NS2 as our backbone TTS model, which is a state-of-the-art zero-shot TTS system. The configuration of the NS2 model is consistent to that presented in the original publication. The additional contextual embeddings are processed through a 512x512 linear projection layer before being added to speaker embedding. We employ two baseline NS2 model with different training strategies:• NS2-A model, trained from scratch by the LibriTTS-R dataset.• NS2-B model, initially trained on the Librivox-B dataset and subsequently fine-tuned on the LibriTTS-R dataset. It serves as an enhanced baseline, employing an equal amount of data as the pre-trained CCSP model, to validate the proposed framework's effectiveness while isolating the variable of additional data input.The NS2+CCSP model, the NS2 model with injection of CCSP features, is trained on the LibriTTS-R dataset in the same manner as the NS2-A model. Within this framework, the CCSP model serves Notably, the NS2+CCSP model demonstrates a substantial increase in score. This suggests that while data augmentation through adaptation provides a modest enhancement, the incorporation of the CCSP model, which leverages cross-modal contextual information from extensive datasets, significantly enriches the expressiveness of the text-to-speech (TTS) synthesis.Subjective Test. Building on the objective analysis, subjective evaluations were conducted to further compare the performance of the NS2-B and NS2+CCSP models. These evaluations included a preference test and a Comparison Mean Opinion Score (CMOS) test. Figure 8 presents the preference scores for the subjective evaluation of the NS2+CCSP and NS2-B models. In the sentence-based test set, the NS2+CCSP model was preferred 69% of the time, with a 'No Preference' response at 9%, and the NS2-B model preferred In this section, we delve into the efficacy of the individual components of the CCSP model through a series of ablation study. 3 shows that the cross-modal contextual embeddings, derived from the CCSP model trained on a larger dataset, significantly enhance the expressiveness of the TTS system. 
conclusion: 
 In this paper, we present the Contrastive Context-Speech Pretraining (CCSP) model, designed to enhance expressive TTS systems.Our pretraining framework focuses on learning effective crossmodal representations that capture the relationship between global contextual information and the current speech expression. Through the utilization of abundant contextual speech data and explicit prosody modeling, the generated features demonstrate their efficacy in both the Context-to-Speech and Speech-to-Context retrieval tasks. By strategically integrating these cross-modal representations into a downstream TTS system, we observe a significant improvement in the expressiveness of the generated speech, as evidenced by both subjective test and objective test. Overall, our proposed CCSP framework proves beneficial for expressive TTS systems, particularly in scenarios where there is a dearth of sufficient high-quality, long-form speech data for the target speaker. Future Work and Impact Statements. • While the CCSP framework has shown promising results in learning cross-modal representations and enhancing the expressiveness of synthesized speech, this work can be further refined by 1) improved fine-grained control over expressions and 2) broadening its application to various downstream tasks such as speech-to-speech generation and speech style annotation tasks. • This work enhances TTS expressiveness with limited target speaker data, which promises improvements in user experience across digital media and accessibility. However, it also raises ethical concerns like creating challenges in authenticity verification and potentially facilitating misinformation. 
 
 Appendix:  


Decision: This paper introduces a contrastive learning approach for expressive speech synthesis representation learning. The goal is to learn cross-modal information between contextual representation and speech representation, thereby generating higher-quality expressive speech. It is particularly valuable in situations characterized by limited target speaker data and the need for lengthy expressive readings. Experimental results also demonstrate the effectiveness of this pre-training framework across two downstream tasks: Context-Speech retrieval and zero-shot TTS system. Reviewers are somewhat split over the paper, but attribute some novelty (contrastive learning for representation learning in a synthesis context). Some open questions remain even after rebuttal, but authors are working to add further answers (e.g. evaluation results on longer utterances) to the final manuscript, which would address most concerns. Still, since results are not yet available, I cannot give a strong recommendation.

Reasons to accept:
- This paper proposes a simple but effective method to enhance current SOTA zero-shot TTS systems.
- The paper is well-structured and the motivation is solid. The references and discussion of related work is appropriate, the paper is over all well written. 
- The authors conduct extensive experiments with ablations and analysis to show the contributions.

Reasons to reject:
- Reviewer muvW especially (also hPvR) raises several questions that have no been addressed in the rebuttal. Authors indicate they will have more results to share in the camera ready copy (and improve the writing, share code), but as of now we need to treat these questions as unresolved.

Reviews: review: paper_summary: paper_summary: This paper introduces a CCSP framework, which adopts a contrastive learning approach to learn cross-modal information between contextual representation and speech representation, thereby generating higher-quality expressive speech. It is particularly valuable in situations characterized by limited target speaker data and the need for lengthy expressive readings. Experimental results also demonstrate the effectiveness of this pre-training framework across two downstream tasks: Context-Speech retrieval and zero-shot TTS system.
strength_weakness: strength_weakness: - The paper is well-structured and the motivation is solid.
- The reference and surveys are adequate.
- The authors conduct extensive experiments with ablations and analysis to show the strong foundation.
limitations: limitations: For context-aware text-to-speech, synthesizing long texts presents a significant challenge. Therefore, I'm particularly interested in the long-form expressive TTS capabilities of the CCSP model. I do notice that this paper covers paragraph tests composed of 2-5 sentences. However, it would be more convincing and supportive if it could be longer in most cases.
paper_summary: paper_summary: This paper proposes a novel context-aware speech pre-trained model for expressive TTS based on contrastive learning. It is designed to learn cross-modal representations between context and speech and leverages these representations to enhance the expressiveness of the TTS model. When applying it to NaturalSpeech2 framework, it improves the expressiveness of speech even with limited data. The experimental results show that the proposed model significantly enhances the expressiveness of synthesized speech.
strength_weakness: strength_weakness: Novelty: This paper proposes a simple but effective method to enhance current SOTA  zero-shot TTS system. It is novel that it combines the cross-modal retrieval task and TTS.
Theoretical approach: The proposed model is designed to learn effective cross-modal representations that capture the relationship between global contextual information and the current speech expression and makes a significant improvement in the expressiveness of the generated speech. It's of a certain level of technical depth.
Clarity: The article is clearly stated and the sections are closely connected logically
Adequate evaluation: It conducts sufficient objective and subjective tests to evaluate the performance of the proposed model in downstream. Also conduct detailed analysis on context-length and prosody-length which which are directly related to the proposed model.
Application: It shows a potention to breaking the bottleneck of data amount in Zero-shot TTS. And it can be applied to other downstream work related to long text or speech in future.
limitations: limitations: 1. NS2 is an abbreviation. When it first appeared, the full name was not described and no citation was added.
2. The font size in Figure 2 is too large and inconsistent with the article.
3. About the selection of the base models. In order to prove that this pre-trained model is  flexibility in the TTS area, the selection of backbone TTS models should be more diverse. Then the corresponding baseline models will be various and the improvement will be more evident.
paper_summary: paper_summary: This work leverages contrastive learning to extract corresponding prosodic information from contextual text, aiming to improve the performance of long-text TTS models.
strength_weakness: strength_weakness: 1.Although contrastive learning has been employed for prosody prediction in several works, such as ClapSpeech[1], there has been no prior attempt to predict prosody from long contextual text. Thus, the method itself is not highly innovative, but it is applied in a new tts scenario.


2.The authors provide a detailed analysis of the CCSP model in the ablation study section.


[1] Ye Z, Huang R, Ren Y, et al. Clapspeech: Learning prosody from text context with contrastive language-audio pre-training[J]. arXiv preprint arXiv:2305.10763, 2023.
limitations: limitations: 1. I am concerned about the actual utility of the CCSP module. Methodologically, does the audio side of the CCSP module—specifically 
${E_{B}}^{S}$ , ${E_{A}}^{S}$ and ${E_{E}}^{S}$ contain semantic information? If so, I am unsure if contrastive learning will focus on prosody alignment rather than semantic alignment. Experimentally, the ablation study from lines 840 to 853 suggests that the absence of prosodic representation has a minor impact, while the absence of pretrained audio encoder representations has a significant effect. Additionally, the MAP values in the retrieval task are quite low, casting doubt on the efficacy of the CCSP module.


2. Upon listening to the author's demo, I noticed no significant differences, and it appears that no ground truth (GT) was provided. Furthermore, the demo includes only two sets of audio samples. I recommend the authors provide more audio samples (e.g., 30 sets) during the rebuttal period for a more thorough comparison.


3. Evaluating the CCSP module, a plug-and-play component, solely within the NaturalSpeech2 model is insufficient. I am curious why the authors chose a zero-shot TTS baseline rather than adding the CCSP module to a standard TTS model. I suggest the authors include more baselines, such as VALLE, FastSpeech2, and VITS, to demonstrate the effectiveness of the CCSP module.

4. The writing quality needs improvement. Some simple concepts are explained in a convoluted manner. For instance, the CCSP module, which is essentially a straightforward contrastive learning approach for contextual text, is overcomplicated. Additionally, in line 453, the model is referred to as NS2, which might confuse reviewers unfamiliar with zero-shot TTS. Similarly, in line 327, the PE module is not explained, even though it can be inferred as an encoder.


5. In the retrieval domain, it is common to calculate both recall and MAP. I recommend the authors include recall-related experiments for further validation.


6. For fairer comparisons, I suggest the authors conduct tests on public datasets.

7. I am curious whether the authors plan to open-source their code.


8. In Table 3, when the CCSP module is added to TTS, I may have misunderstood the specific content of 
${E_{B}}^{S}$ and ${E_{B}}^{S}$ . Could the authors provide examples about ${E_{B}}^{S}$ and ${E_{B}}^{S}$ during the rebuttal period?


9. The introduction mentions that the CCSP module is designed to guide prosody in long texts. However, the Sentence/Paragraph Test Set experiment (Table 2) shows that the effect is weaker for paragraphs than for sentences. This may be due to unstable prosody in paragraphs, potentially indicating that the CCSP module's effectiveness in guiding prosody for long texts is limited.
paper_summary: paper_summary: This paper introduces a context-aware voice pretraining model leveraging contrastive learning for expressive Text-to-Speech (TTS) synthesis. The model leverages contextual voice data without explicit speaker labels to capture the intricate relationship between context and speech expression. This enables the generation of expressive speech by integrating cross-modal features of context and speech into the TTS model, particularly beneficial when target speaker data is scarce.
strength_weakness: strength_weakness: In both validation tasks where the proposed pretraining model is 1) applied to Context-Speech retrieval tasks for parameter optimization and 2) integrated with a zero-shot TTS system to assess the assistance of learned cross-modal features in expressive TTS modeling, improvements over baseline models were obtained. Both objective and subjective evaluations were carried out in the assessment of the TTS expressiveness. In addition, the paper is well organized and is clearly written.
limitations: limitations: The paper uses NS2 as a baseline system and mentions that NS2 is a state-of-the-art zero-shot TTS system. However, there is no reference to NS2 in the paper. Also, a comparison with other TTS system which can synthesize expressive speech could be carried out. The current comparison with NS2 is limited.