# End-to-End Mispronunciation Detection and Diagnosis Using Transfer Learning
## 3 Methodology
### Enhanced MDD by Textual Information
#### 3.2.2 Textual Modulation Gate  
The MDD model shown in the Figure 2 takes the reference text (i.e., the canonical transcription) as the input. However, some phonemes in the canonical transcription of the reference text will be \"polluted\" when the speakers mispronounce them. In this case, the \"polluted\" reference text is thus not paired with associated audio features. Therefore, we propose a _textual modulation gate_ based on attention fusion. On the textual side, we run an information monitor to filter out texts whose prior knowledge is strong enough to deteriorate the performance. To this end, we design and compare four different textual modulation gates. The first is shown in the bottom-left corner of Figure 3. For \(\mathbf{H}^{\text{text}}\) and \(\mathbf{H}^{\text{audio}}\), we have:  
\[\boldsymbol{\alpha}_{t,n}=\text{sigmoid}(\text{score}(\mathbf{h }_{n}^{\text{text}},\mathbf{h}_{t}^{\text{audio}})) \tag{4}\] \[\text{score}(\mathbf{h}_{n}^{\text{text}},\mathbf{h}_{t}^{\text {audio}})=\mathbf{h}_{n}^{\text{audio}}(\mathbf{h}_{t}^{\text{text}})^{ \text{T}}\] (5) \[\mathbf{c}_{t}=\sum_{t=1}^{N}\boldsymbol{\alpha}_{t,n}\mathbf{h}_ {n}^{\text{text}}\] (6) \[\mathbf{g}=\text{sigmoid}(W\cdot\mathbf{h}_{t}^{\text{audio}}+ U\cdot\mathbf{c}_{t}+b)\] (7) \[\mathbf{y}_{t}=\mathbf{h}_{t}^{\text{audio}}+\mathbf{g}\odot \mathbf{c}_{t} \tag{8}\]  
where \(\odot\) is an element-wise product. We compute attention weights, between textual embedding \(\mathbf{h}_{n}^{\text{text}}\in\mathbf{H}^{\text{text}}\) and acoustic embedding \(\mathbf{h}_{t}^{\text{audio}}\in\mathbf{H}^{\text{audio}}\), which is used for re-weighting the textual representation. Then we choose the implementation of linear projection, summation, and sigmoid activation sequentially to generate the textual gate before feeding them into the transformer layer for CTC prediction. We refer to the formula above as _TextGate_. Furthermore, we further explore three variants of gate modulating. Two of them only differ in the activation function from _TextGate_: one with the softmax function and another with the tanh function (see the _TextGate_ and _TextGate_ subplots in Figure 3). The third variant uses one gate to control on the textual branch and another to control on the acoustic branch, which is referred to as _DoubleGate_. It should be noted that, except for the BaselineConcatenate model that uses the \"concatenation\" operation, the rest models use the \"add\" operation. This design is for the convenience of easily extending the model by stacking more textual modulation gating layers.  
Figure 2: An overview of baselines: (**left**) The BaselineConventional model from [20]. It takes phoneme sequences and the fbank feature as inputs and improves the aligned representation learning by performing attention for feature aggregation. (**right**) Our BaselineConcatenate model where we replace the audio encoder and text encoder of the BaselineConventional model with the more powerful wav2 vec2.0 model and transformer while keeping the same fusion strategy. It should be noted that only the audio branch is pretrained.