# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 4 Linguistic Knowledge
### Dependency  
In this section, we explored the knowledge of various dependencies learned by the attention heads by masking several important attention heads and unimportant attention heads to better understand the information about inter-word relationships learned by each attention head in the model, as in the experiments on POS. In the first step, we used stanfordCoreNLP to extract dependencies from reference translations, and this tool can extract all dependencies of a sentence. Dependency is linguistic knowledge that reveals the dominant relationship between two words. A sentence has a root that does not depend on other words, all other words depend on one word of the sentence, and no one word can depend on two or more words. For this study, we focused on the eight dependency types that are most frequently used: root, nsubj, obj, advmod, det, amod, nmod, and compound. We statistically counted the number of word combinations of each dependency type in the reference translation, as shown in Table 4, and the total number of word combinations of dependency type _dep_ in the reference translation was indicated as \(total_{dep,ref}\). In the second step, we extracted the dependencies from the translations generated by the complete model. The number of correct translations for the word combination of each dependency in the translation is counted as \(N_{dep,model}\), and the formula is as follows:  
\[N_{dep,model}=\Sigma_{j=1}^{s}\Sigma_{i=1}^{m}\varphi\Big{(}DEP_{ref,j},DEP_{ modd,j,i}\Big{)} \tag{9}\]  
where \(s\) denotes the number of sentences in the dataset and \(m\) denotes a word combination of dependency _dep_ in a sentence. \(\varphi()\) is an indicator function, \(DEP_{ref,j}\) denotes all word combinations where dependency is _dep_ in the \(j\)th sentence of the reference translation, and \(DEP_{model,j,i}\) denotes the _ith_ word combinations where dependency is _dep_ in the \(j\)th sentence of the translation generated by the complete model. If the word combination of \(DEP_{model,j,i}\) exists in \(DEP_{ref,j}\), the word combination is considered to be translated correctly and the output of the indicator function is 1, otherwise the output is 0. In the third step, we extracted the dependencies from the translations generated by the masking model and calculated the total number of correctly translated word combinations whose dependencies are _dep_ as  
\[N_{dep,model}=\Sigma_{j=1}^{s}\Sigma_{i=1}^{m}\varphi\Big{(}DEP_{ref,j},DEP_{ mask_{band},j,i}\Big{)} \tag{10}\]  
In the end, we obtained the insights of attention head learning dependency information:  
\[acc_{dep,model}=\frac{N_{dep,model}}{total_{dep,ref}} \tag{11}\]  
\[acc_{dep,head}=\frac{N_{dep,head}}{total_{dep,ref}} \tag{12}\]  
These formulas represent the accuracy of the complete model and the model with masked attention heads in correctly translating word combinations for each dependency, relative to the reference translation, where \(acc_{dep,model}\) represents the accuracy of the complete model in correctly translating word combinations for dependency _dep_, serving as the baseline for dependency experiment. \(acc_{dep,head}\) represents the accuracy of the model with masked attention heads in correctly translating word combinations for dependency _dep_. The effect of masking the attention heads on the accuracy of word combinations that correspond to each dependency in the translation reveals the attention head's learning of knowledge of various dependencies. The experimental results are shown in Figure 7. When the important attention heads were masked, the accuracy of the correct translation of each dependent word combination decreased, demonstrating that the attention head had become acquainted with the dependence. Similarly, we analyzed the variation in the accuracy of each dependency when masking important attention heads, as shown in Figure 8 andTable 7. We discovered that attention heads learned more inter-word information about adjectival modifiers (amod).