# Boosting the Transformer with the BERT Supervision in Low-Resource Machine Translation
## 5 Results and Discussion  
### Comparison with Baselines  
Table 2 shows the translation performances of the baselines and the Transformer (base) trained with our approach on the six low-resource translation tasks. It is clear that the Transformer (base) trained with our algorithm achieves the highest BLEU values on these  
tasks. Compared with Transformer(base) trained end-to-end, the improvements achieve 4.2%, 2.5%, 6.4%, 4.6%, 3.3%, and 4.5% on the six tasks, respectively. Among the baselines, the Transformer (base), the Transformer (big), and the Transformer-based transfer learning produce similar performances. RelPos and DeepRepre obtain better performances than the Transformer (base), the Transformer (big), and the Transformer-based transfer learning since they extract better representation using a deep attention mechanism. The BERT-fused model improves the translation quality by incorporating the BERT semantic information in the encoding and the decoding stages. Compared to the Transformer (base) trained end-to-end, our approach improves the performance of the Transformer (base) about 1\(\sim\)2 BLEU point. Our approach is obviously better than the RelPos and DeepRepre. It suggests that adding BERT supervision alleviates the data hungry problem of the Transformer and improves its performance in low-resource machine translation. From another viewpoint, adding the BERT supervision can transfer the BERT knowledge into the Transformer and makes this model converge very well. Table 3 shows the number of parameters and the average inference time of each model in the test stage. We use the average inference time of Transformer (base) as the standard. The inference time of any other model is a multiple of the standard. The BERT-fused model has the most parameters and the slowest inference speed. This is because it incorporates the BERT in the inference process as additional knowledge of the encoding and the decoding modules. The Transformer (big) and DeepRepre also have more parameters than the Transformer (base). Although the BERT-fused model has a good translation quality, its inference speed is very slow. The parameter number of our approach is the same as that of the Transformer (base) and far less than that of the BERT-fused model. Since our approach does not change the structure of the Transformer (base), its inference speed is faster than the BERT-fused model. Considering the comprehensive performance of the Transformer trained with our algorithm in terms of translation quality, model parameters, and inference speed, we draw \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multicolumn{2}{c}{**Approaches**} & \multicolumn{1}{c}{**De\(\rightarrow\)En**} & \multicolumn{1}{c}{**Ro\(\rightarrow\)En**} & \multicolumn{1}{c}{**Vi\(\rightarrow\)En**} & \multicolumn{1}{c}{**De\(\rightarrow\)Zh**} & \multicolumn{1}{c}{**Ko\(\rightarrow\)Zh**} & \multicolumn{1}{c}{**Ru\(\rightarrow\)Zh**} \\ \hline \multirow{6}{*}{Transformer-based Methods} & Transformer (base) & 34.98 & 32.06 & 27.87 & 27.88 & 30.24 & 25.43 \\ \cline{2-8}  & Transformer (big) & 34.85 & 31.98 & 27.82 & 27.79 & 30.28 & 25.35 \\ \cline{2-8}  & Transformer-based transfer learning & 35.01 & 32.10 & 27.75 & 27.72 & 30.13 & 25.15 \\ \cline{2-8}  & RelPos & 35.12 & 32.15 & 28.57 & 28.12 & 30.51 & 25.69 \\ \cline{2-8}  & DeepRepre & 35.41 & 32.27 & 28.48 & 28.35 & 30.59 & 26.14 \\ \hline \multirow{2}{*}{Incorporating Bert Methods} & Bert-fused NMT & 36.34 & 32.35 & 29.04 & 28.82 & 30.79 & 26.29 \\ \cline{2-8}  & Ours & **36.45** & **32.86** & **29.65** & **29.17** & **31.24** & **26.57** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The translation performance of the baselines and our approach.