# Impact of Sentence Representation Matching in Neural Machine Translation
## 4 Experiment Setting
### Setting for Robustness Analysis  
In experiments to investigate the robustness of the model structure and embedding distribution, we reproduced a NMT open-source program ([[https://github.com/OpenNMT/OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py)]([https://github.com/OpenNMT/OpenNMT-py](https://github.com/OpenNMT/OpenNMT-py)), accessed on 18 January 2022) [30] and evaluated the performance of LSTM and the transformer with random, word2 vec, and Bidirectional En \begin{table}
\begin{tabular}{c c c c} \hline \hline LSTM stacks & 4 & parameter \\ Cells per stacks & 1000 & encoder & 3.05 M \\ Dim. of word & 50 & decoder & 3.10 M \\ Dim. of attention & 250 & output & 11 M \\ Batch size & 128 & interface & 0.19 M \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Model settings for impact analysis (M: million, dim. : dimension). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Type** & **Set** & **En** & **Fr** & **En** & **Es** & **En** & **DE** & **Unit** \\ \hline sentence & train & 4.1 & & 3.0 & & 3.6 & \(10^{6}\) \\  & test1 & 2623 & & 2653 & & 2741 & \(10^{0}\) \\  & test2 & 117,861 & & 138,408 & & 151,139 & \(10^{0}\) \\ \hline token & train & 83.8 & 92.2 & 62.9 & 65.6 & 76.4 & 72.6 & \(10^{6}\) \\  & test1 & 48.0 & 53.8 & 48.0 & 51.7 & 52.6 & 51.4 & \(10^{3}\) \\  & test2 & 2.5 & 2.9 & 3.0 & 3.3 & 3.3 & 3.4 & \(10^{6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Data statistics for impact analysis (train: training set; test1: newstest-2013; test2: newsmomentary-v8; validation set is a randomly selected 10% of the training set). -coder Representations from Transformers (BERT) [31] embeddings for the French to English translation task. We used the re-sized Europarl corpus for training ([[https://www.statmt.org/europarl/v7/fr-en.tgz](https://www.statmt.org/europarl/v7/fr-en.tgz)]([https://www.statmt.org/europarl/v7/fr-en.tgz](https://www.statmt.org/europarl/v7/fr-en.tgz)), accessed on 18 January 2022), common-test for validation ([[http://www.statmt.org/europarl/v1/common-test2.tgz](http://www.statmt.org/europarl/v1/common-test2.tgz)]([http://www.statmt.org/europarl/v1/common-test2.tgz](http://www.statmt.org/europarl/v1/common-test2.tgz)), accessed on 18 January 2022), newstest ([[http://data.statmt.org/wmt17/translation-task/test.tgz](http://data.statmt.org/wmt17/translation-task/test.tgz)]([http://data.statmt.org/wmt17/translation-task/test.tgz](http://data.statmt.org/wmt17/translation-task/test.tgz)), accessed on 18 January 2022) and news-commentary ([[http://matrix.statmt.org/test_sets/nc-test2007.tgz](http://matrix.statmt.org/test_sets/nc-test2007.tgz)]([http://matrix.statmt.org/test_sets/nc-test2007.tgz](http://matrix.statmt.org/test_sets/nc-test2007.tgz)), accessed on 18 January 2022) for tests. Applied preprocessing methods are equal. The number of tokens are 50,000. Special symbols, including the start, end, unknown, and blank symbols, remained in the corpus. Data statistics for this robustness analysis are shown in Table 5.  
The tested embedding methods included random generation (RE), importing embedded vectors trained by word2 vec ([[https://github.com/[PERSON]/word2](https://github.com/Andras/word2) vec-pytorch]([https://github.com/Andras/word2](https://github.com/[PERSON]/word2) vec-pytorch), accessed on 18 January 2022), implemented as open-source (WV), and importing embedded vectors of pre-trained multilingual BERT (BE). RE selects an element of an embedding vector from a uniform distribution in [\(-\)0.1, 0.1], as shown in Table 6. WV is generated as in other experiments through learning a language model from the parallel training corpus with the same dictionary.