# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 4 Linguistic Knowledge
### Part-of-Speech
In the third step, we use the masking model for translation and count the total number of correctly translated words of POS type \(p\) in the translation generated by the masking model as  
\[N_{p,head}=\Sigma_{j=1}^{s}\Sigma_{i=1}^{m}min\Big{(}n_{w_{i},ref},n_{w_{i}, mask_{head}}\Big{)} \tag{6}\]  
In the end, we obtain the insights of the attention head learning POS information:  
\[acc_{p,model}=\frac{N_{p,model}}{total_{p,ref}} \tag{7}\]  
\[acc_{p,head}=\frac{N_{p,head}}{total_{p,ref}} \tag{8}\]  
These formulas represent the accuracy of the complete model and the model with masked attention heads in correctly translating words of each POS, relative to the reference translation, where \(acc_{p,model}\) represents the accuracy of the complete model in correctly translating words of POS \(p\), serving as the baseline for POS experiment. \(acc_{p,head}\) represents the accuracy of the model with masked attention heads in correctly translating words of POS \(p\). Figure 5 displays the experimental results, which show how masking important attention heads, unimportant attention heads, and random attention heads affects various POS information. Overall, for each POS, the accuracy of the words' correct translations decreased when important heads were masked, demonstrating that important attention heads have the propensity to engage in a lot of information about the POS. The accuracy of the words' correct translations of each POS remained stable when masking the unimportant attention heads. When randomly masking attention heads, the change in the correct translation accuracy for words of different parts of speech falls between the accuracy changes caused by masking important and unimportant attention heads, a phenomenon that aligns with general patterns. To better understand the findings of our experiment, we selected an example from the dataset we used. As shown in Table 5, we display the reference translation, the translation generated by the complete model, the translation generated when important attention heads are masked, and the translation generated when unimportant attention heads are masked. It can be observed that when important attention heads are masked, words of various POS, such as the verb \"provides\", the adjective \"theoretical\", and the noun \"checking\", are not translated correctly. In addition, when unimportant attention heads are masked, there is almost no change in the translation results. \begin{table}
\begin{tabular}{c c} \hline \hline Reference & The analysis method provides some theoretical foundation for the safe assessment on welded joint strength in a rotation blade. \\ Transformer & This method provides a theoretical basis for the strength-checking of rotary welded joints. \\ Mask Important Attention heads & The analysis method is used to verify the welding strength of the joints. \\ Mask Unimportant Attention heads & This method provides a theoretical basis for the strength-checking of the rotary joint. \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The example of reference translation, the translation generated by the complete model, the translation generated when important attention heads are masked, and the translation generated when unimportant attention heads are masked, and the translation generated when unimportant attention heads are masked. Figure 5: The effect of masking attention heads on each piece of POS information. \(x\)-axis denotes the number of masked attention heads. \(y\)-axis denotes the accuracy of correct translation. (**a–f**) denote different types of POS. We discovered that the decreasing trend of the words' correct translations was different for each type of POS information while masking important attention heads. We examine the alteration in the accuracy of the words' correct translations when masking the important attention head to better understand how the model learns various POS information, as shown in Figure 6. It can be seen that all POS information is important for model decisions.