# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 4 Linguistic Knowledge
### Part-of-Speech  
In this experiment, we masked several important attention heads and unimportant attention heads according to attention heads' importance to identify different POS knowledge learned by these attention heads. Specifically, we evaluate the POS information learned by the attention head in the translation accuracy of each POS in the translations generated by the masking model. In the first step, according to the POS types in Penn Treebank [33], we used Stanford-CoreNLP to POS annotate the reference translation. Penn Treebank is a large-scale corpus that provides precise POS and syntactic annotations, which are extremely useful for the training and evaluation of language models. It is widely referenced in natural language processing research and serves as a benchmark dataset for numerous algorithms and models. Therefore, we chose it for our experiments. POS information represented the function of the individual word information in a sentence for NMT. For this study, we focused on the six POS types that are most frequently used: the noun (N), verb (V), adjective (ADJ), adverb (ADV), preposition or subordinating conjunction (IN), and determiner (DT). Since various POS are divided into POS subclasses, for example, general adjectives, adjectival comparatives, and adjectival maxims are three categories of adjectives, we cluster nouns, verbs, adjectives, and adverbs individually [34]. We statistically counted the number of words of each POS type in the reference translation, as shown in Table 4, and the total number of words of POS type \(p\) in the reference translation was indicated as \(total_{p,ref}\). In the second step, we compare the number of correct translations of each POS word in the translation generated by the complete model with the reference translation. The total number of correctly translated words of POS type \(p\) in the translations generated by the complete model is as follows:  
\[N_{p,model}=\Sigma_{j=1}^{s}\Sigma_{i=1}^{m}min\left(n_{w_{i},ref},n_{w_{i}, model}\right) \tag{5}\] \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Types** & **Number** \\ \hline \multirow{6}{*}{Part-of-speech} & None & 1209801 \\  & Verb & 639937 \\  & Adjective & 325112 \\  & Adverb & 168872 \\  & Preposition & 521772 \\  & Determiner & 421079 \\ \hline \multirow{6}{*}{Dependency} & Root & 253109 \\  & Nsubj & 300879 \\  & Obj & 200308 \\ \cline{1-1}  & Advmod & 183645 \\ \cline{1-1}  & Det & 401812 \\ \cline{1-1}  & Amod & 285112 \\ \cline{1-1}  & Nmod & 229550 \\ \cline{1-1}  & Compound & 242266 \\ \hline \multirow{6}{*}{Syntax tree} & NP-NP-NN & 185474 \\ \cline{1-1}  & NP-PP-IN & 139905 \\ \cline{1-1}  & PP-NP-NN & 148864 \\ \cline{1-1}  & NP-NP-DT & 126033 \\ \cline{1-1}  & VP-PP-IN & 131974 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Number of words in each POS category, number of combinations of dependent words, and number of syntactic tree path patterns in the reference translation. where \(s\) denotes the number of sentences in the dataset and \(m\) denotes a non-repeating word of POS \(p\) in a sentence. For the \(i\)th non-repeating word \(w_{i}\) of POS \(p\) in the \(j\)th sentence of the reference translation, the number in the reference sentence \(n_{w_{i},ref}\) and the number in the complete model-generated sentence \(n_{w_{i},model}\) are counted.