# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 3 Attention Heads' Importance
### Correlation Analysis  
Previous research has identified that important attention heads frequently possess specific and interpretable functions [18]. To verify whether the attention heads' importance is consistent with the interpretable function, we analyzed the correlation between attention head importance and attention head POS contribution. The attention head contributes to the accurate translation of each POS word, which is the attention head POS contribution. Specifically, we evaluate the impact on the translation accuracy of each POS word when masking the attention head. For more details see Section 4.1. We employed various statistical methods to evaluate these correlations, including Pearson correlation and KL divergence. The Pearson correlation measures the linear correlation between two variables and the KL divergence is an asymmetric measure of the difference between two probability distributions. The value of the Pearson correlation is easy to understand and directly reflects the strength of the linear relationship between variables. The KL divergence can sensitively capture subtle differences between two probability distributions and is widely used in machine learning. Since the data contain negative values,  
Figure 4: The distribution of attention head importance. (**a**) The number of important attention heads. (**b**) The number of unimportant attention heads.  
Figure 3: The importance of each layer’s attention head. X-axis denotes the attention layer. Y-axis denotes the impact on the BLEU score (absolute value).  
we must normalize it to [0, 1] before computing the KL divergence. We use two normalization methods. The first is \(KL_{Norm}=(x_{i}-x_{min})/(x_{max}-x_{min})\). The second method takes the logarithm of the data \(a=[x_{1},\ldots,x_{i},\ldots,x_{n}]\) in the interval \((-\infty,+\infty)\) to obtain the data \(b=[\ln x_{1},\ldots,\ln x_{i},\ldots,\ln x_{n}]\) in the interval \((0,+\infty)\), then normalizes the data \(KL_{Softmax}=x_{i}/sum(x)\).  
The correlation between attention head importance and attention head POS contribution among three distinct relevance evaluation criteria is shown in Table 3. We have established a connection between human-understandable POS information and the importance of attention heads, significantly enhancing the model's interpretability and aiding in our understanding of why some attention heads are important. Moreover, the experiment employs statistical analysis of a large number of sentences to ensure the accuracy and validity of the results. Overall, there was a discernible positive correlation between attention head importance and their contributions to various POS categories, indicating that the important attention heads we identified possess specific and interpretable functions. This aligns with the research findings of [PERSON] [18]. A more substantial correlation was observed between attention head importance and the contributions to nouns and verbs. This could be attributed to the pivotal role of nouns and verbs in Chinese-English machine translation, suggesting that important attention heads have a deeper understanding of these POS. Such insights motivate us to refine the model design to facilitate enhanced learning of nouns and verbs by the attention heads, potentially improving the model's overall performance.