## 2 BERT Pre-Training Model
### Knowledge Extraction Paradigm  
In our proposed translation model, the contextual knowledge extracted during BERT pre-training is integrated through the dynamic fusion mechanism on the encoder end. In addition, the knowledge extraction paradigm is also added on the decoder end to facilitate the translation model in learning the specific task representation in the pre-training process.  
In the decoding stage, the model generates the j-th word by maximizing the conditional probability using the following formulas:  
\[\begin{gathered} C_{M}^{D}=\text{Att}\left(Q_{M}^{D},K_{N}^{E},V_{ N}^{E}\right)\\ R_{M}^{D}=\text{LN}\left(FFN\left(S_{M}^{D}+C_{M}^{D}\right)\right)\\ P\left(y_{j}\big{|}Y_{<j},x\right)=\text{softmax}\left(FFN\left(r_{M,j}^{D}\right)\right)\end{gathered} \tag{9}\]  
where, D represents the decoder, M is the number of layers of the decoder, \(Q_{M}^{D}\) is from the output \(S_{M}^{D}\) of the previous layer of the decoder, \(S_{M}^{D}\) is calculated by Equation (2), with K, Q, and V being from the \(R_{M-1}^{D}\) of the previous layer of the decoder, and \(K_{N}^{E}\) and \(V_{N}^{E}\) are from the output \(R_{N}^{E}\) of the encoder.  
The final optimization function of the translation model is:  
\[\mathcal{L}_{T}=\frac{1}{J}\sum_{i=1}^{J}\log P\left(y_{j}\big{|}y_{<j},x; \theta_{T}\right) \tag{10}\]  
where \(\theta_{T}\) is a parameter of NMT.  
Figure 4: Dynamic Fusion Mechanism at encoder end.  
In terms of word-level granularity, the knowledge extraction paradigm mainly learns the output distribution of the model from the pre-training model, and the training function can be represented by:  
\[\mathcal{L}_{W}=\frac{1}{\mathrm{J}}\sum_{\mathrm{j=1}}^{\mathrm{J}}\sum_{ \mathrm{k=1}}^{\mathrm{V}}\mathrm{P}\Big{(}\mathrm{y_{j}}=\mathrm{k}\Big{|} \mathrm{y;\theta_{P}}\Big{)}\cdot\log\Big{(}\mathrm{P}\Big{(}\mathrm{y_{j}}= \mathrm{k}\Big{|}\mathrm{x,y_{<j};\theta_{T}}\Big{)}\Big{)} \tag{11}\]  
where J is the length of the given target sentence \(y\), and V is the vocabulary. \(\mathrm{P}\Big{(}\mathrm{y_{j}}=\mathrm{k}\Big{|}\mathrm{x,y_{<j};\theta_{T}} \Big{)}\) is calculated by Equation (9).  
In terms of sentence-level granularity, the contextual knowledge of the sentence is learned by direct fitting, and the training function is written as:  
\[\mathcal{L}_{\mathrm{S}}=\frac{1}{\mathrm{J}}\Big{|}\Big{|}\mathrm{R}_{\mathrm{ M}}^{\mathrm{D}}-\mathrm{R}_{\mathrm{L}}^{\mathrm{P}}\Big{|}\Big{|}_{2}^{2}= \frac{1}{\mathrm{J}}\sum_{\mathrm{j=1}}^{\mathrm{J}}\Big{|}\Big{|}\Big{|} \mathrm{I}_{\mathrm{M}\mathrm{j}}^{\mathrm{D}}-\mathrm{r}_{\mathrm{L}\mathrm{ j}}^{\mathrm{P}}\Big{|}\Big{|}_{2}^{2} \tag{12}\]  
where M represents the output layer of the decoder, and \(\mathrm{r}_{\mathrm{M}\mathrm{j}}^{\mathrm{D}}\) and \(\mathrm{r}_{\mathrm{L}\mathrm{j}}^{\mathrm{P}}\) are from the decoder and the pre-training model, respectively.  
Finally, the translation model is optimized for training by fitting three loss functions through Formula (13):  
\[\mathcal{L}=\mathcal{L}_{\mathrm{T}}+\eta\mathcal{L}_{\mathrm{S}}+\beta \mathcal{L}_{\mathrm{W}} \tag{13}\]  
where \(\eta\) and \(\beta\) are hyperparameters for balancing the fusion granularity of the word-level and sentence-level knowledge extraction paradigms, and they both are set to 0.5 in this work.  
The conceptual diagram of the knowledge extraction paradigm adopted by the Transformer model on the decoder end is presented in Figure 5: