# Analyzing and Controlling Inter-Head Diversity in Multi-Head Attention
## 6 Experiments on Controlling Inter-Head Similarity Methods
### Quantitative Evaluation  
We report BLEU scores on every language pairs in Tables 4 and 6. These results support our hypothesis that a multi-head attention model can extend its own capability by controlling inter-head diversity with our suggested methods. Models with all three suggested methods applied (+All) show the best performances on every language pair.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Language Pairs**} \\ \cline{2-3}  & **De\(\rightarrow\)En** & **En\(\rightarrow\)De** \\ \hline Baseline Transformer & 33.67 & 29.76 \\ + DROHEAD ONLY & 34.26 & 30.13 \\ + ORTHO ONLY & 34.03 & 30.27 \\ + HSIC ONLY & 34.43 & 30.32 \\ +ALL & **34.53** & **30.38** \\ \hline \hline \end{tabular}
\end{table}
Table 4: BLEU evaluation with controlled inter-head similarity on En-De IWSLT17 corpus.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**SVCCA**} & **CKA** \\ \hline
**Baseline Transformer** & **0.397** & **0.199** \\ \hline +DROHEAD & _0.1_ & 0.415 & 0.207 \\ +DROHEAD & _0.3_ & 0.534 & 0.317 \\ +DROHEAD & _0.5_ & 0.709 & 0.527 \\ \hline +ORHO & _0.1_ & 0.408 & 0.208 \\ +ORHO & _1.0_ & 0.407 & 0.223 \\ +ORHO & _10.0_ & 0.420 & 0.366 \\ \hline +HSIC & \(10^{-8}\) & 0.364 & 0.182 \\ +HSIC & \(10^{-7}\) & 0.338 & 0.158 \\ +HSIC & \(10^{-6}\) & 0.325 & 0.125 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Controlled inter-head similarity with suggested methods.  
We also verify the effect of our suggested methods on language modeling task in order to show that our methods can be applied to tasks other than machine translation. Table 7 shows perplexity score on language modeling task with PTB corpus. As well as on the encoder-decoder Transformer, our methods applied to the decoder-only Transformer also increases its performance on the language modeling task. Applying +HSIC only shows the best performance, even better than applying all methods. Nevertheless, all of our methods clearly improve the perplexity of the decoder-only Transformer. The experimental results show that our methods can easily be applied to various model architectures that use multi-head attention. Note that our suggested methods and our analyses in Section 5 do not relate to the size of the model (i.e., the hidden size or the number of layers). We strongly believe that our methods can be applied to larger language models such as BERT [2] or XLM-R [33], because they also exploit the multi-head attention as the same way as the Transformer model in our experiments.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{6}{c}{**Language Pairs**} \\ \cline{2-7}  & **De\(\rightarrow\)En** & **En\(\rightarrow\)De** & **Ru\(\rightarrow\)En** & **En\(\rightarrow\)Ru** & **Zh\(\rightarrow\)En** & **En\(\rightarrow\)Zh** \\ \hline Baseline Transformer & 31.47 & 25.69 & 52.68 & 44.62 & 52.21 & 47.08 \\ \(+\)Drophead only & 31.69 & 25.73 & 52.90 & 44.82 & 52.60 & 47.09 \\ \(+\)Ortho only & 31.63 & 25.86 & 52.92 & 44.86 & 52.55 & 47.28 \\ \(+\)HSIC only & 31.57 & 25.89 & 52.89 & 44.82 & 52.54 & 47.16 \\ \(+\)ALL & **31.76** & **25.91** & **53.02** & **45.23** & **52.69** & **47.33** \\ \hline \hline \end{tabular}
\end{table}
Table 6: BLEU evaluation on various language pairs with controlled inter-head similarity on WMT17 corpus and UN corpus.  
Figure 4: SVCCA coefficient curves of inter-head similarity with controlling methods.