# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 3 Attention Heads' Importance
### Attention Head Analysis  
We mask several important attention heads, unimportant attention heads, and random attention heads based on the attention heads' importance we acquired to show that it is accurate. By looking at the degradation in translation performance, we evaluate the validity of attention heads' importance. Figure 2 illustrates the effect of masking multiple attention heads on translation performance. There is a significant degradation in translation performance when masking several important attention heads, while masking multiple unimportant attention heads results in minimal impact. This outcome validates the accuracy of our attention head importance ranking. To ensure the reliability of our experimental results, we assessed the distribution of attention head importance and its correlation with the contribution of attention heads to POS. [PERSON] (2018) posits that certain components of the Transformer model are more important than others, particularly within the attention layer. They suggest that the decoder self-attention layers are the least critical, whereas the upper encoder-attention layers in the decoder are more significant than the lower ones. We conducted two experiments to investigate the distribution of attention head importance and verify if it aligns with these suggested properties. First, we want to know which layers in the model are more important and which are less important. This can help us understand the importance of the attention layers. Therefore, the initial experiment aims to assess the importance of the attention heads in each layer. Specifically, the translation performance is evaluated as a function that aggregates the contributions of all attention heads within a layer. If this function demonstrates a significant influence, the layer is deemed crucial. Otherwise, it is considered non-essential. Then, we want to know whether the important attention heads are concentrated in a particular layer and whether retaining only a few attention heads are more important. Figure 1: Attention heads’ importance. The heatmap reflects the impact of each attention head in the model on translation performance. The x-axis from zero to seven represents the eight attention heads in each layer of the model. The y-axis represents the different layers of the three attention mechanisms in the model. Specifically, 0–5 represent the 1–6 layers of encoder self-attention, 6–11 represent the 1–6 layers of decoder self-attention, and 12–17 represent the 1–6 layers of encoder–decoder attention. The numbers represent the impact of each attention head on translation performance. A higher value indicates that the attention head is less important, while a lower value indicates that the attention head is more important. The color reflects the importance of the attention head, with the darker one indicating more importance. heads in a certain layer can allow the model to function normally. Therefore, the second experiment examines the distribution of important and unimportant attention heads. We specifically count the 30 most important and the 30 least important attention heads in the model, based on their importance. A layer is considered important if it contains a high number of crucial attention heads and a low number of non-essential ones. Conversely, it is deemed unimportant if this is not the case. Figure 3 represents the importance of attention layers, which is the sum of the impacts of all attention heads in each layer of the three types of attention mechanisms in the model on translation performance. We find that the encoder self-attention (0-5) and encoder-decoder attention (12-17) are more important, while the decoder self-attention (6-11) is less important. Overall, we observe that higher layers are more crucial than lower layers across different attention layers.