# Boosting the Transformer with the BERT Supervision in Low-Resource Machine Translation
## 5 Results and Discussion
### Effectiveness of Joint Optimization  
Unlike the end-to-end training methods, our algorithm trains the Transformer in a three-stage fashion, which includes (1) encoder training, (2) decoder training, and (3) joint optimization. We design joint optimization to further fine-tune the parameter in the Transformer, so that the encoder and the decoder can perfectly match in the latent space. The first two stages are necessary, and the third stage is optional in theory. Therefore, we only analyze the results with and without joint optimization in the ablation experiment.  
To verify the effectiveness of this training operation, we conduct ablation studies on the above translation tasks as shown in Table 5. We compared the performances of different optimization epochs. For simplicity, we use EncT, DecT, JO to represent encoder  
\begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt}} \hline \hline
**De\(\rightarrow\)En** & \\ \hline \hline Source & Und warm? Weil sie Dreiecke verstehen und sich-selbst-ver starkende geometrische Muster sind der Schlüssel um stabile Strukturen zu bauen. \\ \hline Target & And why? Because they understand triangles and self-reinforcing geometric patterns are the key to building stable structures. \\ \hline Transformer(base) & And why? Because they understand triangles and self-reinforcing geometric patterns, they are crucial to building stable structures. \\ \hline Ours & And why? Because they understand triangles and self-reinforcing geometric patterns are key to building stable structures. \\ \hline
**Ro\(\rightarrow\)En** & \\ \hline Source & [PERSON] si-a exprimat regretul ca divizarile in consiliu si intre poporul sirian si puterile regionale “au facut acaestá situatie de merezolvát”. \\ \hline Target & He expressed regret that divisions in the council and among the Syrian people and regional powers “made this situation unsolvable”. \\ \hline Transformer(base) & [PERSON] expressed regret that the divisions in the council and between the Syrian people and the regional powers “have made this situation unresolved”. \\ \hline Ours & [PERSON] expressed regret that divisions in the council and between the Syrian people and regional powers “have made this situation intractable”. \\ \hline
**Vi\(\rightarrow\)En** & \\ \hline Source & Tung dong, têné dêu duco cán nhâc vía tên hoc them têing Anh vía tên dûrôt êng ra bât ké víe khoán não phi tr  
training, decoder training, and joint optimization, respectively. The third line \"EncT + DecT\" represents the model only going through the encoder training and the decoder training. We also list the performance of the Transformer (base) trained end-to-end.  
As shown in Table 5, when the Transformer only goes through the encoder training and the decoder training independently, its performance is lower than that of the Transformer (base) trained end-to-end. This is because the encoder and the decoder do not not fully match in the latent space. The performance of the Transformer goes up significantly as joint optimization epochs increase from 40 to 80. The model obtains the highest BLEU 36.38 in De\(\rightarrow\)En task after the full fine-tuning, which shows that joint optimization can boost the Transformer after independently training the encoder and the decoder. With the increase of the epoch of joint optimization, the performance of the Transformer increases. The results on the other tasks also confirm this point.