# Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2 seq Models for Machine Translation
## 4 Experiment
### Ablation Study
#### 4.5.3 Importance of Each Module  
We perform some analysis to show the importance of each module of the pre-trained model in fine-tuning. We compare four models: a random baseline (RND2 RND), a model with the pre-trained encoder and random decoder (ENC2 RND), a model with random encoder and pre-trained decoder (RND2 DEC), and our cross-connected model (ENC2 DEC). We do not add the intermediate layer to the ENC2 RND or the RND2 DEC model because there is no need for mediation, and we neither use it to the cross-connected ENC2 DEC model for fair comparison.  
The results are reported in Table 5. As expected, when reusing a part of a pre-trained model, the encoder is much more important than the decoder. Both the pre-trained encoder and pre-trained decoder improves the model performance, but the ENC2 RND model obtains a larger gain. This seems natural because the decoder depends on the source information from the encoder side, and the performance of a sequence-to-sequence model is heavily affected by the sentence representation that the encoder generates [51]. This result suggests that if only a part of a pre-trained model can be reused, it is more efficient to reuse the encoder part than the decoder part. However, the scores of the ENC2 DEC models are the highest except for It-Ro, which indicates that cross-connection is more effective than reusing only the encoder.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **En-De** & **De-En** & **En-Ro** & **Ro-En** & **En-It** & **It-En** \\ \hline RND2 RND (baseline) & 26.11 & 33.83 & 27.49 & 37.47 & 29.10 & 31.44 \\ ENC2 DEC & 27.13 & 34.88 & 28.01 & 37.86 & 29.81 & 32.10 \\ ENC2 DEC & **29.06** & **37.75** & **30.04** & **39.73** & **32.17** & **35.05** \\ \hline
**Model** & **It-Ro** & **Ro-It** & **It-De** & **De-It** & **De-Ro** & **Ro-De** \\ \hline RND2 RND (baseline) & 21.73 & 22.27 & 18.91 & 19.70 & 18.37 & 19.50 \\ ENC2 RND & 22.69 & 23.14 & 20.64 & 22.58 & 20.43 & 20.85 \\ RND2 DEC & 20.59 & 22.27 & 19.40 & 19.96 & 18.94 & 20.24 \\ ENC2 DEC & 22.36 & **23.91** & **21.53** & **22.65** & **20.98** & **21.20** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Fine-tuning models with different decoder initialization. The Baseline is a randomly initialized Transformer and IL is the intermediate layer. We indicate the best scores with **bold** font.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **En-De** & **De-En** & **En-Ro** & **Ro-En** & **En-It** & **It-En** \\ \hline RND2 RND (baseline) & 26.11 & 33.83 & 27.49 & 37.47 & 29.10 & 31.44 \\ ENC2 RND & 28.35 & 36.63 & 29.63 & 39.22 & 30.66 & 33.80 \\ RND2 DEC & 27.13 & 34.88 & 28.01 & 37.86 & 29.81 & 32.10 \\ ENC2 DEC & **29.06** & **37.75** & **30.04** & **39.73** & **32.17** & **35.05** \\ \hline
**Model** & **It-Ro** & **Ro-It** & **It-De** & **De-It** & **De-Ro** & **Ro-De** \\ \hline RND2 RND (baseline) & 21.73 & 22.27 & 18.91 & 19.70 & 18.37 & 19.50 \\ ENC2 RND & **22.69** & 23.14 & 20.64 & 22.58 & 20.43 & 20.85 \\ RND2 DEC & 20.59 & 22.27 & 19.40 & 19.96 & 18.94 & 20.24 \\ ENC2 DEC & 22.36 & **23.91** & **21.53** & **22.65** & **20.98** & **21.20** \\ \hline \hline \end{tabular}
\end{table}
Table 5: BLEU scores of the fine-tuning models with partial module reusing. We indicate the best scores with **bold** font.