# Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2 seq Models for Machine Translation
## 4 Experiment
### Ablation Study
#### 4.5.4 Low-Resource and Mid-Resource  
The IWSLT14 and IWSLT17 datasets are low-resource datasets. We perform additional experiments to see if the cross-connection method is also effective for a larger dataset. We use WMT16 En-Ro and IWSLT14 En-Ro to exclude the linguistic characteristics and only compare the size of the fine-tuning datasets. For fine-tuning on the WMT16 En-Ro dataset, we use the same pre-trained weights and different training schedule: initial learning rate = 2.0 and warmup steps = 16,000.  
The results are reported in Table 6. We observe that the improvement gap becomes smaller in WMT16 En-Ro as shown in the top section. We conjecture that this result is mainly because of the relatively small amount of our pre-training. However, it is also consistent with the results of other works such as [2; 28] that the effect of pre-training decreases or is lost as the size of the parallel data grows. In the middle section, WMT16 ENC2 RND obtains the best score and RND2 DEC reports a lower score than the random baseline, contrary to IWSLT14. To investigate this further, we illustrate the test scores along with the training steps in Figure 4. WMT16 RND2 DEC is outperformed by the RND2 RND baseline after training 30K steps. We assume that the effect of cross-connection in a richer dataset is limited by this forgetting of decoder pre-training. It seems to be the reason the ENC2 RND obtains a better score than ENC2 DEC in WMT16 unlike in IWSLT14.  
This is in line with the results of [30] where their multilingual BERT and English GPT model (BERT2 GPT) could not reach the highest score in De-En translation. We surmise that their pre-trained GPT forgot its pre-training because WMT14 En-De is a high-resource dataset. Furthermore, their RND2 GPT model gets a score even below the random baseline just like our RND2 DEC, and BERT2 BERT is similar to the BERT2 RND and RND2 BERT is better than RND2 GPT. These are in agreement with our results reported in the bottom section of Table 6. In WMT16 En-Ro, ENC2 ENC obtains the second-best score and RND2 ENC is better than RND2 DEC, whereas RND2 DEC is better than RND2 ENC  
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Models** & **IWSLT14** & **WMT16** \\ \hline RND2 RND (baseline) & 27.49 & 23.89 \\ ENC2 DEC & **30.04** & 24.51 \\ \hline ENC2 RND & 29.63 & **24.77** \\ RND2 DEC & 28.01 & 23.73 \\ \hline RND2 ENC & 27.41 & 24.03 \\ ENC2 ENC & 29.31 & 24.68 \\ \hline \hline \end{tabular}
\end{table}
Table 6: BLEU scores of the IWSLT14 En-Ro dataset and WMT16 En-Ro dataset with partial pre-trained module reusing. We indicate the best scores with **bold** font.  
Figure 4: BLEU scores of the IWSLT14 En-Ro dataset and WMT16 En-Ro dataset along with the training steps.  
in IWSLT14 En-Ro. When the amount of bitext grows, the performance of the pre-trained decoder is soon outperformed by the random baseline even though it was higher at first. Therefore, we conclude that cross-connection of pre-trained encoder and decoder is effective for low-resource language translation, and when the resource is rich enough, it is rather effective to use the pre-trained encoders to initialize both the encoder and decoder.