# MixDiff-TTS: Mixture Alignment and Diffusion Model for Text-to-Speech
## 2 Materials and Methods
### MixDiff-TTS
#### 2.2.2 Linguistic Encoder  
Figure 2b depicts the detailed architecture of the linguistic encoder, where \"LR\" refers to the length regulator for adjusting input sequence length, and \"WP\" signifies the transformation of input phoneme representations into word representations through word-level pooling. The pre-net structure performs multi-layer nonlinear transformations on the input phoneme sequence. By using activation functions and a dropout mechanism, the model's capacity for feature representation is significantly enhanced. Specifically, its architecture includes multiple fully connected layers and ReLU activations. This structure captures the potential nonlinear relationships between phonemes by performing nonlinear mapping on the original phoneme embeddings. With this design, the model becomes more effective in modeling the phoneme-to-acoustic feature mapping, thereby further enhancing its learning capacity. The phoneme encoder and word encoder share identical components, as depicted in Figure 3a. Specifically, both are constructed by stacking multiple Feed-Forward Transformer (FFT) layers. By stacking FFT layers, they effectively capture sequence features  
Figure 2: Architecture diagram of MixDiff-TTS and its modules. (**a**) Overall architecture of MixDiff-TTS. \(M_{t}\) is the mel-spectrogram at the \(t\)-th step in the diffusion process, \(E_{t}\) is the step embedding, and \(E_{m}\) is the sequence output by the linguistic encoder. (**b**) Detailed structure of the Linguistic Encoder. (**c**) Detailed structure of the Auxiliary Decoder.  
from both local and global contexts, and separately model feature representations at the phoneme and word levels. As shown in Figure 3b, the prosody predictor consists of a pitch predictor and an energy predictor. We employ architectures for both that are identical to the corresponding modules in FastSpeech2 [12], as depicted in Figure 3c. These two predictors, respectively, predict the pitch and energy features in speech. Additionally, prosody prediction is also influenced by duration as pitch and duration are closely related. During prosody prediction, stressed segments tend to have longer durations and may also exhibit higher pitch. The phoneme sequence and word-level hidden states output by the linguistic encoder serve as the common foundation for both pitch and duration prediction.  
The W2P-RelPosAttention module is a Word-to-Phoneme Attention module with a relative position bias mechanism. In this module, to enable the query \(Q\) to find the phonemes associated with the corresponding word, an additional mapping mask is introduced into the attention weights. Building on this, we introduce a relative position bias mechanism to enhance the model's capability for modeling long text sequences, with its computation defined as follows:  
\[output=Softmax(\frac{QK^{T}}{\sqrt{d_{k}}}+P(i,j))V+Q \tag{9}\]  
where \(i\) and \(j\) are the indices of elements in the query \(Q\) and the key \(K\), respectively. \(d_{k}\) denotes the dimension of \(K\). \(P(i,j)\) is the relative position bias, representing the relative position information between \(Q\) and \(K\). \(\frac{QK^{T}}{\sqrt{d_{k}}}\) is used to calculate the similarity between \(Q\) and \(K\). The purpose of the expression \(\frac{QK^{T}}{\sqrt{d_{k}}}+P(i,j)\) is to enable the model to capture the relative positional relationship between \(Q\) and \(K\), rather than merely focusing on their content-based matching similarity. Through encoding the relative positional relationship between \(Q\) and \(K\), this mechanism adjusts the attention scores to incorporate positional dependencies. As a result, the model can make adaptive adjustments across different parts of the sequence. When dealing with long text sequences, the model can understand the positional relationships based on the specific context instead of simply relying on the fixed sequence positions. This enables the model to more effectively capture dependencies in long text sequences.