# Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2 seq Models for Machine Translation
## 4 Experiment
### Ablation Study
#### 4.5.2 Initialization of Decoder Cross-Attention  
As we mentioned earlier, we can use the pre-trained encoder weights to initialize the decoder. Furthermore, since our method pre-trains each model with only monolingual data, the decoder cross-attention could not be suitable for the cross-lingual task. Therefore, we conduct two additional experiments to investigate the initialization of the decoder. First, we use the pre-trained decoder weights but initialize the cross-attention randomly (Random cross-attention). Second, we use the pre-trained encoder weights to initialize both the encoder and decoder as in Section 4.4 and add the intermediate layer (ENC2 ENC). We compare these settings with our cross-connected models (ENC2 DEC) and random baselines (RND2 RND).  
As shown in Table 4, the cross-connection models result in the best scores. The scores decrease when we initialize the cross-attention randomly, and the ENC2 ENC with IL models report similar or lower scores than the random cross-attention models. Therefore, it can be said that the monolingual pre-trained decoder can be more effectively reused than the pre-trained encoder for machine translation. Although its cross-attention never  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & **En-De** & **De-En** & **En-Ro** & **Ro-En** & **En-It** & **It-En** \\ \hline No intermediate layer & 29.06 & 37.75 & 30.04 & 39.73 & 32.17 & 35.05 \\
2 FFN (filter size = 2048) & 29.43 & 38.08 & 30.31 & 40.01 & 31.80 & 35.14 \\ Encoder Sublayer (filter size = 2048) & 29.55 & 38.00 & **30.64** & 40.02 & 32.06 & 35.20 \\
2 FFN (filter size = 1024) & 29.20 & 38.01 & 30.29 & **40.39** & 32.09 & 35.22 \\
1 FFN (filter size = 2048) & 29.37 & **38.10** & 30.23 & 40.21 & **32.29** & 35.45 \\
1 FFN (filter size = 1024) & 29.57 & 37.91 & 30.37 & 40.20 & 32.08 & 35.23 \\ Self-Attention & 29.33 & 38.03 & 30.45 & 40.14 & 32.01 & 35.17 \\
1 FFN (filter size = 768) & **29.63** & 38.01 & 30.27 & 40.20 & 32.02 & 35.44 \\
1 FFN (filter size = 512) & 29.52 & 37.94 & 30.28 & 40.27 & 32.09 & **35.47** \\ Dense (size = 512) & 29.18 & 37.77 & 30.01 & 40.20 & 32.08 & 35.06 \\ \hline
**Model** & **It-Ro** & **Ro-It** & **It-De** & **De-It** & **De-Ro** & **Ro-De** \\ \hline No intermediate layer & 22.36 & 23.91 & 21.53 & 22.65 & 20.98 & 21.20 \\
2 FFN (filter size = 2048) & 22.61 & 23.94 & 21.62 & 23.19 & 21.18 & **21.71** \\ Encoder Sublayer (filter size = 2048) & 22.72 & 23.92 & 21.46 & **23.32** & 20.69 & 21.44 \\
2 FFN (filter size = 1024) & 22.95 & 24.26 & 21.29 & 23.05 & 21.03 & 21.32 \\
1 FFN (filter size = 2048) & 22.96 & 23.50 & 21.41 & 23.17 & 20.58 & 21.19 \\
1 FFN (filter size = 1024) & 22.42 & 24.16 & 21.69 & 23.16 & 21.05 & 20.93 \\ Self-Attention & 22.39 & 24.07 & 21.64 & 22.96 & 20.68 & 21.03 \\
1 FFN (filter size = 768) & **23.03** & 23.74 & 21.90 & 23.01 & 21.00 & 21.03 \\
1 FFN (filter size = 512) & 22.08 & **24.11** & **22.04** & 23.28 & **21.26** & 21.14 \\ Dense (size = 512) & 22.24 & **24.33** & 21.80 & 22.89 & 21.02 & 21.42 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparisons of the fine-tuning models with various intermediate layers for each translation pair. FFN is the feed-forward network with ReLU activation. Dense is a fully connected layer with layer normalization and dropout. Results below ’No intermediate layer’ are sorted by the number of parameters in decreasing order. We indicate the best scores with **bold** font.  
learned cross-lingual tasks, it is better than the simple random initialization. Furthermore, the score gap between the ENC2 ENC+IL and ENC2 DEC+IL is bigger than the score gap of ENC2 ENC and ENC2 DEC without IL in Section 4.4. This indicates that the intermediate layer is more effective for the cross-connected model than for ENC2 ENC.