## 4 Analysis and Discussion
### Analysis of Cross-Attentions  
Cross-attentions can be seen as a reordering model as well as an aligning model between two languages from the perspective of machine translation [18]. Although cross-attentions are known to be a core component of deep learning networks for translation, few studies have analyzed the relationships between cross-attentions and a language pair. In this study, we analyze the properties of cross-attentions between a source language and a target language. Prior to analysis, we first define a few terminologies about attentions used in this work. Most NMT models are implemented based on an encoder-decoder architecture. In the encoder-decoder architecture, a cross-attention mechanism is devised so that the decoder can decide which part of a source sentence to pay attention to when generating the next token. The cross-attention mechanism can relieve the encoder of the burden to represent all the information of a source sentence in a fixed-size vector. The cross-attention of each decoding step indicates which source part should be used to translate into a target language. Therefore, in this work, we define the source token with the highest attention score at each decoding step as an aligning token for the current target token. Assuming that a source and a target are the same, the sequence of aligning tokens for the target sequence monotonically increases, as shown in Figure 1a. In this figure, the arrow \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**Embeddings**} & \multicolumn{3}{c}{**MASS Encoder**} \\  & & **JA \(\rightarrow\) KO** & **EN \(\rightarrow\) KO** \\ \hline \multirow{2}{*}{Unsupervised NMT} & Initial embeddings & 6.10 & 1.05 \\  & Final embeddings & 20.90 & 2.95 \\ \hline \multirow{2}{*}{Supervised NMT} & Initial embeddings & 1.15 & 0.90 \\  & Final embeddings & 76.85 & 40.20 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The performances of sentence-translation retrievals. points to an aligning token in a source sequence. We name it a _fixed cross-attention_ and denote it by \(A^{fixed}_{L\to L}\) from a language \(L\) to the same language \(L\). First, we analyze cross-attentions between the same sentences in NMT models. Since MASS can train an NMT model bidirectionally, the NMT model implemented using MASS in this study can translate sentences in both directions. Therefore, it is possible to examine cross-attentions between the same sentences. Ideally, the cross-attentions would be nearly the same as the fixed cross-attentions because the source and the target sentences are the same in this case. However, the cross-attentions between the same sentences may be different from the fixed cross-attentions because the NMT model learns two languages together. Therefore, the other language's properties, such as word order and morphology, can affect the cross-attentions within the same language. Let a source sequence be {BOS, \(s_{1},s_{2},\dots,s_{n}\), EOS} and a target sequence be {BOS, \(t_{1},t_{2},\dots,t_{m}\), EOS}. We align each token of a target sequence with a token of a source sequence. For each target token \(t_{i}\), we can find a source token \(s_{j}\) that has the highest attention score and then denote the alignment by \((t_{i},s_{j})\) for the target token \(t_{i}\) and its aligning token \(s_{j}\). In this work, a target token (except the final one) of a target sequence is said to be _non-aligned_ when its aligning token in a source sequence is either 'BOS' or 'EOS'. For a given a alignment \((t_{i},s_{j})\), we can find the largest index \(k\) of the alignment \((t_{k},s_{i})\), such that \(k<i\) and \(s_{l}\
otin\) {BOS, EOS}. When the indexes of two alignments are \(l>j\), the target token \(t_{i}\) is said to be _aligned in the reverse order_. In the example of Figure 1b, the target tokens \(t_{2},t_{3}\), and \(t_{7}\) are non-aligned because they attend to 'BOS' or 'EOS'.