## 1 Introduction  
In recent years, deep learning technology has made remarkable progress in the field of speech synthesis. Traditional speech synthesis systems typically rely on separate modules, such as acoustic models and vocoders, which are designed based on manually crafted features and rules. However, these systems often have difficulty capturing subtle variations and voice characteristics in speech, limiting the naturalness and fluency of the generated speech. Against this backdrop, the introduction of end-to-end speech synthesis methods has brought new hopes to the field of speech synthesis. Compared to traditional systems, end-to-end speech synthesis methods learn the entire synthesis process as a unified model, greatly simplifying system design and enhancing the performance of speech synthesis. However, end-to-end speech synthesis still encounters several challenges that urgently demand resolution, including prolonged training times and unnatural prosodic expressions. As we shift our focus towards similar sequence-to-sequence tasks encountered in the realm of Natural Language Processing (NLP), we observe that with the emergence of transformer [1], attention mechanism [1], and subsequent models such as BERT [2] and its derivatives like RoBERTa [3], substantial advancements have been made in the field of natural language processing. These models, trained on large-scale corpora, exhibit the capability to generate high-quality language representations and have demonstrated remarkable performance across various downstream tasks. Their advantages encompass contextual awareness, transfer learning capability, unsupervised learning proficiency, and profound understanding of language. In the realm of speech synthesis, we can draw upon the principles and methodologies of models like BERT. By devising pre-training models, engaging in transfer learning, and incorporating contextual information, we aim to further augment the performance and efficacy of speech synthesis models. These endeavors hold promise for making significant contributions to the advancement of speech synthesis technology and the progression of intelligent speech interaction systems. Therefore, our focus has been directed towards models such as HuBERT [4] and wav2 vec2.0 [5], which have demonstrated significant improvements in speech recognition performance they were when pre-trained on the LibriSpeech [6] dataset. The objective of extracting hidden representations from speech preprocessing tasks aligns with that of speech recognition in end-to-end speech synthesis. Consequently, we propose that integrating pre-trained models into existing speech synthesis frameworks would substantially enhance the naturalness and prosody of the synthesized speech. For instance, in the VITS [7] model, XPhoneBERT [8] can be employed as a replacement for the text encoder segment. XPhoneBERT shares a similar structure to BERT, albeit differing in input; while BERT's input is a sequence of textual tokens, XPhoneBERT's input comprises a sequence of phonemes. By substituting VITS's basic text encoder with a pre-trained phoneme encoder, the model's understanding of textual input can be markedly enhanced, thereby further amplifying the expressive capabilities of speech synthesis. Therefore, this paper aims to further explore how to leverage pre-trained models to enhance the quality and effectiveness of speech synthesis within the existing end-to-end speech synthesis framework VITS. We selected the pre-trained model HuBERT as our research subject and introduced it into the posterior encoder component of the VITS model. By combining the robust speech recognition capabilities of HuBERT with the end-to-end structure of VITS, we anticipate augmenting the model's proficiency in rhythm and prosody performance, thereby generating more natural and fluent speech.