# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 4 Linguistic Knowledge
### Syntax Tree  
In this section, we studied the change in the accuracy of syntactic tree information in the translation while masking several important attention heads and unimportant attention heads to evaluate the impact of attention heads on the model's processing of multiple-word information and overall syntactic structure [35]. A syntactic tree is an abstract syntactic structure that represents the structure of a sentence. In the first step, we constructed the syntactic analysis of the reference translation using stanfordCoreNLP, a tool that constructs syntactic trees of sentences. We summarize the patterns of the bottom three nodes in the syntactic tree paths of each word to better examine the syntactic structure knowledge acquired by the attention head because there are plenty of ways to combine syntactic tree paths. For this study, we focused on the five syntactic tree path patterns that are most frequently used: NP-NP-NN, NP-PP-IN, PP-NP-NN, NP-NP-DT, and VP-PP-IN. We statistically counted the number of words of each syntactic tree path pattern in the reference translation, as shown in Table 4, and the total number of words of each syntactic tree path pattern \(path\) in the reference translation was indicated as \(total_{path,ref}\). \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Complete** & \multicolumn{6}{c}{**Pruned Model**} \\  & **Model** & **5** & **10** & **15** & **20** & **25** & **30** \\ \hline root & 32.92 & 2.36 & 3.61 & 9.47 & 14.26 & 17.69 & 20.49 \\ nsubj & 22.57 & 3.62 & 4.79 & 9.98 & 12.91 & 14.58 & 16.72 \\ obj & 19.17 & 3.50 & 4.54 & 8.62 & 11.72 & 13.45 & 14.76 \\ adymod & 19.02 & 7.95 & 9.08 & 14.12 & 15.02 & 15.37 & 15.83 \\ det & 36.72 & 9.04 & 11.19 & 12.40 & 15.58 & 19.18 & 22.35 \\ amod & 31.31 & 7.80 & 9.96 & 22.08 & 24.86 & 26.09 & 26.55 \\ nmod & 23.03 & 7.64 & 9.84 & 12.06 & 16.10 & 18.00 & 19.03 \\ compound & 33.93 & 4.60 & 7.22 & 14.58 & 19.99 & 22.22 & 24.12 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: The changes in the accuracy of correct translations for word combinations for each dependency between the complete model and the pruned model. All data in the table are represented as percentages. The leftmost column lists the dependency. The “Complete Model” column shows each part’s translation accuracy, while the “Pruned Model” column, labeled “5-30,” displays the percentage decrease in accuracy after masking 5 to 30 attention heads. Figure 8: Masking important attention heads on each piece of dependency information. In the second step, we constructed syntactic trees for the translations generated by the complete model. We counted the total number of words in the translation generated by the complete model and the reference translation whose word paths have the same syntactic tree path pattern, denoted as \(N_{path,model}\), with the following formula:  
\[N_{path,model}=\Sigma_{j=1}^{s}\Sigma_{i=1}^{m}\varphi\Big{(}PATH_{ref,j},PATH_{ model,j,i}\Big{)} \tag{13}\]  
where \(PATH_{ref,j}\) denotes the word where the syntactic tree path pattern is _path_ in the \(j\)th sentence of the reference translation, and \(PATH_{model,j,i}\) denotes the \(i\)th word where the syntactic tree path pattern is _path_ in the \(j\)th sentence of the translation generated by the complete model. The other parameters were consistent with the experimental settings of dependency.