## 3 Methods
Then, for each layer, the input \(x\) is convolved through input layers (_in_layers_) and then processed through an activation function. After the activation function processing, residual connections and skip connections are added to the input and Dropout is applied. Finally, the outputs of all layers are summed and multiplied by the input mask (\(x\_mask\)). Such posterior encoder structures tend to be relatively simple, lacking parallel information and failing to effectively extract features from various frequency bands of speech. Figure 1: The training procedure of our new model architecture. Figure 2: The inferencing procedure of our new model architecture. The original posterior encoder is capable of handling linear spectrograms and synthesizing relatively natural speech. However, when compared to the original data, it consistently falls short in capturing rhythm and intonation. Therefore, building upon the original posterior encoder, we introduced the pre-trained model HuBERT, which is a self-supervised speech representation learning method. It leverages an offline clustering step to provide aligned target labels for BERT-like prediction losses. We found that introducing pre-trained models can enhance the posterior encoder's learning effectiveness of spectrogram features. As shown in Figure 3, by using the pre-trained model to extract corresponding hidden states, which are then projected onto the same dimensionality as the output of the original WaveNet (stat), followed by self-attention pooling and fusion of hidden states and stat, we can significantly improve the model's ability to reconstruct mid- and high-frequency components. The corresponding optimization objective has also changed, as depicted in Equation (1), as follows:  
\[ \begin{split}& A=\text{arg max}\underset{\widetilde{A}}{\text{ max}}\log p(x|c_{phoneme},\widetilde{\widetilde{A}})\\ &=\text{arg max}\log N\bigg{(}f(x);\mu(c_{phoneme},\widetilde{ \widetilde{A}}),\sigma(c_{phoneme},\widetilde{\widetilde{A}})\bigg{)},\end{split}  \tag{1}\]  
Figure 3: Our new posterior encoder. and the posterior distribution of the latent variable has also changed, as outlined in Equation (2):  
\[ \begin{array}{l}p_{\theta}\Big{(}z_{hidden}\Big{|}c_{phoneme}\Big{)}\\ =N\Big{(}f_{\theta}(z_{hidden});\mu_{\theta}\Big{(}c_{phoneme}\Big{)},\sigma_{ \theta}\Big{(}c_{phoneme}\Big{)}\Big{)}\Big{|}det\frac{\partial f_{\theta}(z_ {hidden})}{\partial z_{hidden}}\Big{|},\end{array}  \tag{2}\]  
The \(c_{phoneme}\) is the combination of the phoneme of the text and the alignment matrix of the corresponding spectrum, as depicted in Equation (3):  
\[c_{phoneme}=\Big{[}c_{phoneme},A\Big{]} \tag{3}\]  
The framework we proposed is based on the VITS architecture, which utilizes a conditional variational autoencoder (CVAE) [32] structure. VITS consists of a posterior encoder, a prior encoder, a waveform decoder, and a duration predictor. Building upon VITS, we enhanced the expressiveness of the model by modifying the original posterior encoder. The task of the original posterior encoder is to extract the posterior distribution \(q_{\phi}\) (\(z|x_{lin}\)) of latent variables \(z\) given the linear spectrogram \(x_{lin}\), where the decoder generates output audio from sampled \(z\). The new posterior encoder we proposed aims to obtain the distribution of the last hidden state, \(q_{\phi}(z_{hidden}|x_{lin})\), given the linear spectrogram \(x_{lin}\) and the original waveform \(y\), and then sample new latent \(z_{hidden}\) from the distribution to generate the output audio. The prior encoder consists of a normalizing flow and a text encoder, matching \(z_{hidden}\) to the conditional prior distribution \(p_{\theta}\Big{(}z_{hidden}\Big{|}c_{phoneme},A_{phoneme}\Big{)}\) given phonemes \(c_{phoneme}\)--obtained from preprocessing the given text and alignment matrix \(A_{phoneme}\). By using the Monotonic Alignment Search (MAS) algorithm [7], the alignment matrix \(A_{phoneme}\) is computed. The duration predictor utilizes \(A_{phoneme}\) to learn to predict the duration of each phoneme.