## 4 Experiments  
The training data consist of the LJSpeech [34] dataset, which is a publicly available dataset for speech synthesis research. It contains English speech samples from a single speaker. This dataset is commonly used for training and evaluating text-to-speech models. The speech samples in the dataset are of high quality, often characterized by good clarity and fluency. They cover a wide range of sentence types, including simple everyday phrases and complex literary works. The speech samples are stored in WAV format and are accompanied by text documents containing the corresponding textual content for each speech sample. We also utilized the Database [35] dataset, provided by Beijing Databaker Technology Co., Ltd., Beijing, China. This Chinese speech dataset is primarily used for research and development in fields such as speech recognition and synthesis. The dataset contains Chinese speech samples from the same speaker, covering various everyday phrases and scenarios, including news reports, movie clips, etc. These speech samples are typically stored in WAV format and come with corresponding text annotations. The Databaker dataset also boasts high-quality speech with excellent clarity and fluency, making it well suited for training and evaluating speech-related models. We expanded the VITS model by modifying its WaveNet-based posterior encoder and concurrently introduced other pre-trained models for comparison. During the experiments, we trained the original VITS model using the best hyperparameters as specified in the original VITS paper. This includes using the AdamW [36] optimizer with parameters \(\beta_{1}=0.8\), \(\beta_{2}=0.99\), weight decay \(\lambda=0.01\), and initial learning rate of \(2\times 10^{-4}\). We run the model for 10K training steps with a batch size of 64 (equivalent to approximately 500 training epochs for both English and Chinese data). During training the new model, after a comparison, we found that freezing the parameters of HuBERT and introducing a separate optimizer to update only the posterior encoder part can prevent adverse consequences like gradient vanishing. However, this approach results in slower training compared to the original posterior encoder. The original VITS model trained on 2 RTX 6000 Ada GPUs (NVIDIA, Santa Clara, CA, USA) with a batch size of 64 achieves a training speed of approximately 1 min per epoch, but the new BERTIVITS trained on the same 2 RTX 6000 Ada GPUs with a reduced batch size of 16 achieves a training speed of approximately 3 min per epoch. Due to memory constraints, BERTIVITS needs to reduce the batch size for training without changing the devices. During the experimental process, we found that the performance of the original text encoder in VITS was not satisfactory. However, improvements made by related works such as XPhoneBERT and pseudo-phonemes significantly enhance the expressiveness of the synthesized speech in VITS. Therefore, based on these related works, we introduced a new posterior encoder and observed positive experimental results. Due to the introduction of the pre-trained model, the loss becomes very large during the initial training stages. Moreover, following the original optimization strategy, using a single optimizer for updating parameters of the entire generator quickly leads to gradient vanishing. Therefore, we set up two optimizers. The first optimizer retains the original hyperparameters, namely the AdamW optimizer, which was mentioned earlier. The second optimizer is a new one we introduced, which is the SGD [37] optimizer. We set its learning rate to \(1\times 10^{-4}\), momentum to 0.9, and weight decay to \(2\times 10^{-5}\). The first optimizer is responsible for optimizing parameters in the generator (_net_g_) except for the posterior encoder. The second optimizer is specifically tasked with optimizing the parameters of the modified posterior encoder. Additionally, we need to freeze the parameters of HuBERT; otherwise, gradient explosion occurs around the 40 th epoch, unaffected by hyperparameters.