# Impact of Sentence Representation Matching in Neural Machine Translation
## 5 Results and Discussions
### Impact Analysis  
#### 5.1.1 Structure  
In Figure 3, the token-level precision results of each structure are shown. One-layer structures have a point in the graph because of the fixed dimension for the distance calculation. Two-layer structures are evaluated over varying hidden layer dimensions. The use of 1 or 2 layers showed no significant difference in source- and target-side structures. The both-sides structure showed less precision than the source-side structure.  
\begin{table}
\begin{tabular}{l r r r r} \hline  & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**Transformer**} \\ \hline parameter & RE \& WV & BE & RE \& WV & BE \\ \hline layers & 4 & 4 & 6 & 2 \\ number of heads & - & - & 8 & 8 \\ dimension of & & & & \\ layer output & 1000 & 1000 & - & - \\ embedding & 512 & 768 & 512 & 768 \\ attention & 1000 & 1000 & 1000 & 1000 \\ model [16] & - & - & 512 & 768 \\ feed forward [16] & - & - & 2048 & 2048 \\ total parameters & 168 M & 373 M & 120 M & 303 M \\ \hline batch size & 64 & 32 & 64 & 32 \\ voca. of encoder & 50,002 & 119,547 & 50,002 & 119,547 \\ voca. of decoder & 50,004 & 119,547 & 50,004 & 119,547 \\ \hline \end{tabular}
\end{table}
Table 7: Model settings for robustness analysis (M: million, voca.: vocabulary).  
Figure 4 shows the layer-wise mean and standard deviation of absolute weight parameters of encoder and decoder models in training. In the cases of the mean, the difference between structures is small because of the very large number of parameters. Thus, the small change may be induced by significantly large changes of some parameters. In the zoomed boxes of the subgraph (a), the mean values of the source-side layers were higher than the no-matching case and the target-side layers in the early epochs, although they were lower after sufficient training. In the subgraph (b), source-side layers maintained the mean value of the decoder models, but target-side layers showed significantly larger values. Subgraph (c) and (d) showed the similar superiority of source-side layers as the mean cases.  
In the overall results, source-side layers provided higher pressure at the early stage, but caused less conflict at the final stage of training compared to target-side layers. In the early stage, the gradient for the matching cost is split into the encoder and the matching layer in target-side layers, while source-side layers use the full gradient for training the encoder and increase training speed. In the final stage, the matching mechanism restricts the training of the correct output vector distribution of the encoder, so that the decoder models need to train more information and therefore increase the model complexity. In this stage, source-side layers are more flexible regarding changing the output distribution, because a neural network layer is a many-to-one mapping between the input and output. Any movement of the output distribution is regarded as the change of the input in the source-side layers, but it pushes the output vectors of the matching layer in target-side layers. Thus, target-side layers are more reluctant to change and restrict the encoder to move toward the correct distribution for minimizing the translation cost. In sum,  
* Source-side layers have a higher impact on improving performance;
* Source-side layers have stronger pressure and less conflict.  
Figure 4: Layer-wise statistics \(\mu_{L}\) and \(\sigma_{L}\) of weight parameters for a matching layer structure in training (cosine distance was used and the results were averaged over all hidden dimension settings).  
Figure 3: Token-level precision by dimensions of hidden layer for structure types.