## 2 BERT Pre-Training Model
### MCNMT Model Based on Soft Target Templates and Contextual Knowledge  
The concept of the MCNMT model based on soft target templates and contextual knowledge is inspired by the guided translation model, that is, the source language data and the template with the target language grammar information are fused by the decoder, and the template is used to guide the model to improve the utilization rate of grammar knowledge. The contextual knowledge of the target language extracted from the BERT pre-training model is integrated into the model to improve the translation performance. An example of the template-guided model translation is shown in Figure 6:  
Figure 5: Knowledge Extraction Paradigms at the decoder end. On the basis of the baseline Transformer model, an additional Transformer encoder structure named soft target template is introduced on the encoder end. It has the same composition structure as the Transformer encoder of the source language and is used to encode soft target template data and convert it into hidden layer vectors. At the same time, the dynamic fusion mechanism and knowledge extraction paradigm are integrated into the model in a fusion manner to improve the utilization of contextual knowledge. Figure 7 clearly illustrates the use of integrating various contextual knowledge from the BERT pre-training model into the NMT model. From Figure 7, the dynamic fusion mechanism uses the layer perception attention mechanism and context gating mechanism to respectively extract attentions between different layers and the same layer in the pre-training model and applies them to the encoder end; the knowledge extraction paradigm employs word-level extraction and sentence-level extraction to obtain the output distribution and contextual knowledge of the pre-training model, respectively, and applies them to the decoder end. The overall structure of the model is presented in Figure 8:  
Figure 8: Model Architecture of a Mongolian-Chinese Neural Machine Translation Model Fused with Soft Target Templates. Figure 6: Example of Template-Guided Model Translation. The Mongolian language in the picture means “I like playing basketball”. Through the guidance of the template, the target Chinese language is finally obtained. S represents the subject and VP represents the verb phrase. Figure 7: Obtaining Contextual Knowledge Using the BERT Pre-training Model. The formula for generating Chinese text from Mongolian text with soft templates is as follows:  
\[\mathrm{P}(\mathrm{Y}|\mathrm{X})=\mathrm{P}_{\Theta_{X->\mathrm{T}}}(\mathrm{T}| \mathrm{X})\mathrm{P}_{\Theta_{(X\mathrm{T}->\mathrm{Y})}}(\mathrm{Y}|\mathrm{X },\mathrm{T}) \tag{14}\]  
where, \(\Theta_{X->\mathrm{T}}\) is the parameter of the prediction soft target template Transformer model, and \(\Theta_{(X,\mathrm{T}->\mathrm{Y})}\) is the parameter of the fusion soft target template model. The generation of Chinese translation by the Transformer decoder is based on the hidden state of the source language encoder and the soft target template encoder. Two different sets of attention parameters are used in the decoder's Encoder-Decoder multi-head attention to process two different encoders, respectively, and the hidden layer vectors containing the source language information and these having the soft target template information are fused through the gating unit. The formulas are as follows:  
\[\beta=\sigma\Big{(}\mathrm{W}_{\mathrm{Y}}\mathrm{Z}^{\mathrm{X},\mathrm{Y}}+ \mathrm{U}_{\mathrm{T}}\mathrm{Z}^{\mathrm{X},\mathrm{T}}\Big{)} \tag{15}\]  
\[\mathrm{Z}=\beta\mathrm{Z}^{\mathrm{X},\mathrm{Y}}+(1-\beta)\mathrm{Z}^{ \mathrm{T},\mathrm{Y}} \tag{16}\]  
where, \(\mathrm{W}_{\mathrm{Y}}\) and \(\mathrm{U}_{\mathrm{T}}\) are the parameter matrices, \(\sigma\) is the Sigmoid activation function, \(Z\) is the hidden state of the decoder, and \(\beta\) is the control parameter of the fusion degree of Mongolian text and templates. In this paper, the model parameters are updated using the maximum likelihood estimation algorithm.