# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 5 Conclusions  
In this paper, we explore the importance of attention heads in understanding NMT. We confirmed the correctness of the importance of the attention head and investigated its distribution. The correlation between attention head importance and their contribution to POS was assessed using three evaluative criteria. Our investigation covered how attention heads acquire language knowledge at three different granularities: POS, dependency relationships, and syntactic trees, enhancing our understanding of their learning processes. We find that not all attention heads are important, and the important attention heads are more distributed in the encoder self-attention layer and the encoder-decoder attention layer. These findings have propelled us towards a deeper understanding of the internal structure of NMT models and have opened up new research directions for subsequent related studies. Specifically, these discoveries can guide us in conducting in-depth analyses of critical components within the model to enhance its performance. Additionally, they also support pruning non-critical parts of the model to reduce parameters and improve operational efficiency. In addition, the important attention heads in Chinese-English machine translation contribute more to the generation of nouns and verbs in their translations. Finally, we discovered that the attention heads had varying degrees of POS, dependence, and syntax tree learning. These findings offer valuable guidance for future model design. For instance, when designing Chinese-English machine translation models, we should  
Figure 10: Masking important attention heads on each piece of syntactic tree information. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **Complete** & \multicolumn{6}{c}{**Pruned Model**} \\ \cline{3-8}  & **Model** & **5** & **10** & **15** & **20** & **25** & **30** \\ \hline NP-NP-NN & 34.20 & 7.51 & 9.82 & 11.11 & 15.01 & 18.84 & 20.69 \\ NP-PP-IN & 59.12 & 14.00 & 18.39 & 8.00 & 5.83 & 7.68 & 15.01 \\ PP-NP-NN & 37.83 & 11.84 & 13.13 & 13.51 & 16.18 & 19.91 & 20.49 \\ NP-NP-DT & 56.55 & 13.54 & 17.69 & 9.64 & 7.83 & 9.26 & 19.48 \\ VP-PP-IN & 41.43 & 14.31 & 15.67 & 17.53 & 19.25 & 22.05 & 24.11 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: The changes in the accuracy of correct translations for words for each syntactic tree path pattern between the complete model and the pruned model. All data in the table are represented as percentages. The leftmost column lists the syntactic tree path pattern. The “Complete Model” column shows each part’s translation accuracy, while the “Pruned Model” column, labeled “5-30”, displays the percentage decrease in accuracy after masking 5 to 30 attention heads. pay closer attention to the processing of linguistic elements such as nouns and adjectives. This ensures that the attention mechanism functions more effectively, thereby enhancing translation quality. Overall, we investigated the multigranularity of linguistic knowledge within the context of attention heads making decisions. For future work, we plan to explore the role of attention heads in learning other linguistic knowledge, such as morphology. Additionally, we will conduct comparisons using other salience methods, including layer-wise relevance propagation (LRP) and gradient-based approaches, to examine the decisions made by important attention heads from different perspectives. Conceptualization, J.Z. ; methodology, Z.Z. and J.Z. ; validation, Z.Z. ; resources, J.Z. ; writing--original draft, Z.Z. and J.Z. ; writing--review and editing, J.Z. and W.L. All authors have read and agreed to the published version of the manuscript. Supported by the National Natural Science Foundation of China (grant No. 62166022; 62066022) and the General Project of Yunnan Fundamental Research Programs (grant No. 202101 AT07 0077; 202301 AT070015). Not applicable. The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.