# Towards Understanding Neural Machine Translation with Attention Heads' Importance
## 3 Attention Heads' Importance
### Multi-Headed Attention  
NMT is a technique for translating one natural language \(x=\{x_{1},\dots,x_{M}\}\) into another natural language \(y=\{y_{1},\dots,y_{N}\}\), In autoregressive NMT, each word of the target sentence is generated step by step. Therefore, the generation of the nth target word \(y_{n}\) is influenced by the source sentence \(x\) and the part of the target sentence that has already been generated, denoted as \(y_{<n}\):  
\[P(y|x)=\prod_{n=1}^{N}P(y_{n}|y_{<n},x) \tag{1}\]  
where \(x\) is the source sentence, \(y\) is the target sentence, \(y_{n}\) is the generation of the nth target word, and \(y_{<n}\) is the \(n-1\) target words that have already been generated. The model generates an output word based on the source sentence \(x\) and the partial translation \(y_{<n}\).  
The output values of numerous attention heads are combined to produce MHA. The following is the attention mechanism formula:  
\[Attention(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{2}\]  
where \(Q\), \(K\), and \(V\) are three linearly transformed matrices, and \(d_{k}\) is the vector dimension of \(Q\), \(K\). First, calculate the degree of similarity between each _query_ and each _key_. Second, \(softmax\) is employed to derive the weights. Third, the weights are then averaged with the corresponding \(value\).  
Finally, the output values of MHA are then obtained by concatenating the output values of each layer's attention heads:  
\[MultiHead(Q,K,V)=Concat(h_{1},\dots,h_{i},\dots,h_{n})W^{O} \tag{3}\]  
where \(h_{i}\) denotes the output value of the \(ith\) attention head in the MHA. \(W^{O}\) is a linear transformation matrix used to transform the concatenated output of all attention heads.  
In machine translation, masking is a crucial technique that helps the model learn and predict data more effectively. The application of this technique markedly boosts the model's efficiency and accuracy, rendering it an essential component of contemporary NMT systems. Masking enables the model to better understand and process sentences of varying lengths and to utilize contextual information more effectively, thereby enhancing the quality of translation. It is worth mentioning that in our research, we investigated the importance of attention heads by employing masking to obscure them.  
In this paper, we attribute the importance of attention heads to the effect on translation performance. We modified the attention formula to implement the masking operation of the attention heads:  
\[h=\varphi Attention(Q,K,V) \tag{4}\]  
where \(\varphi\) is a discrete value, which can be 0 or 1. When \(\varphi\) is 0, it indicates that the attention head is masked; when \(\varphi\) is 1, it means that the attention head is retained in the complete model.  
Our experiments were implemented in the Transformer, which has three different attention mechanisms: Enc-Enc, Dec-Dec, and Enc-Dec. The impact of each attention head on translation performance serves as the attention heads' importance. We masked each attention head in each attention mechanism in turn [31]. We used the heatmap to display the information regarding the importance of attention heads that we obtained. It provides an intuitive representation of the distribution of attention head importance. We used the Matplotlib library to implement the drawing of the heatmap. We used the change in BLEU score values due to masking each attention head as an input and set the parameter minimum value to \(-1.0\).  
Figure 1 illustrates the heatmap of the attention heads' importance in the Transformer, which reflects the impact of each attention head on the translation performance. In contrast to the findings of [PERSON] [7], not all attention heads are important; some are and some are not.