EIT: Enhanced Interactive Transformer (2024)

Abstract: Two principles: the complementary principle and the consensus principle are widely acknowledged in the literature of multi-view learning. However, the current design of multihead self-attention, an instance of multi-view learning, prioritizes the complementarity while ignoring the consensus. To address this problem, we propose an enhanced multi-head selfattention (EMHA). First, to satisfy the complementary principle, EMHA removes the oneto-one mapping constraint among queries and keys in multiple subspaces and allows each query to attend to multiple keys. On top of that, we develop a method to fully encourage consensus among heads by introducing two interaction models, namely inner-subspace interaction and cross-subspace interaction. Extensive experiments on a wide range of language tasks (e.g., machine translation, abstractive summarization and grammar correction, language modeling), show its superiority, with a very modest increase in model size. Our code would be available at: https://github.com/zhengkid/EI T-Enhanced-Interactive-Transformer.

heading: 1 Introduction
text: Transformer architectures (Vaswani et al., 2017) have yielded promising results on a wide range of natural language processing tasks (Devlin et al., 2019; Brown et al., 2020). A key factor contributing to their success is the multi-head self-attention network (MHSA), which enables efficient modeling of global dependencies among tokens in parallel. Notably, instead of utilizing a single attention mechanism, MHSA uses an ensemble of attention models, each models a small subspace, and finally aggregates these results to the final one. The core idea is similar to subspace learning (Blum and Mitchell, 1998) or multi-view learning (Chaudhuri et al., 2009).
* Equal Contribution. † Corresponding author.
In the realm of multi-view learning, two fundamental principles guide the research: the complementary principle and the consensus principle (Xu et al., 2013). The complementary principle emphasizes that each data view may possess unique knowledge not present in other views, prompting the use of multiple views for a comprehensive and accurate data description. On the contrary, the consensus principle aims to maximize the agreement on multiple distinct views. However, in the context of MHSA design, most studies predominantly focus on the complementary principle. This oversight is evident in their encouragement of diverse information capture by different heads (Li et al., 2018; Cui et al., 2019; Zheng et al., 2024) and the adoption of complex aggregation operations (Li et al., 2019; Wang and Tu, 2020). Some studies (Michel et al., 2019; Clark et al., 2019; Voita et al., 2019; Behnke and Heafield, 2020) even consider the high similarity among attention heads as a significant problem referred to as attention redundancy.
Although diversity is crucial in multi-view learning, Dasgupta et al. (2001) has shown that simply fusing diverse outputs does not guarantee improved results: the probability of a disagreement of two independent hypotheses upper bounds the error rate of either hypothesis. The consensus principle highlights the need to minimize disagreement among views to achieve better outcomes. In response to the consensus principle, several studies (Kumar and III, 2011; Kumar et al., 2011) have focused on minimizing disagreement among views to achieve better outcomes. However, in the context of MHSA research, there is a tendency to prioritize complementarity over consensus among different attention heads. Here we ask a question: Can balancing these two principles benefit the design of MHSA mechanisms?
However, encouraging such a consensus in multihead self-attention is challenging. In our preliminary experiments, we found that directly utilizing
7734
regularization terms can achieve this goal but cannot improve performance. Drawing inspirations from human behavior where group discussions and interactions foster consensus, we propose introducing interactions among different subspaces in MHSA to achieve consensus.
To this end, we propose a new multi-head self-attention variant: Enhanced Multi-Head SelfAttention, which encourages consensus among attention heads while guaranteeing to contain sufficient information. To ensure information sufficiency, we propose a novel many-to-many mapping scheme to generate numerous high-quality initial attention maps. This can generate more attention maps without suffering low-bottleneck problems (Bhojanapalli et al., 2020). On top of these sufficient attention maps, we propose two interaction components: inner-subspace interaction (ISI) and cross-subspace interaction (CSI). These hierarchical interaction modules fully encourage consensus among attention maps of different heads.
The outcome of this work is an Enhanced Interactive Transformer (EIT) architecture in that MHSA is replaced with Enhanced Multi-Head Attention (EMHA). Our proposed EIT has been demonstrated to be simple to implement and highly parameter efficient, yet it produces consistent performance improvements across a diverse set of tasks, including machine translation, grammar error correction, abstractive summarization, and language modeling. In addition, we have developed a computationally efficient variant of EIT, which, while still maintaining strong performance on several tasks, is better suited for low-latency industrial applications.

heading: 2 Preliminary: Multi-Head Self-Attention
text: Multi-head self-attention (MHSA) is an efficient operation that can capture the interactions among tokens. Given an embedded input sequence X ∈ RT×d, MHSA is defined as follows:
Ai = Softmax( (XWiQ)(XW i K) T
dk ) (1)
O =
M∑
i=1
AiXWiV W i O (2)
where T denotes the sequence length, d is the input embedding dimension, dk is the head dimension, M is the number of head partition on representations, WiQ,W i K ,W i O ∈ Rd×dk , WiO ∈ Rdk×d. Ai represents the attention distribution of i-th head and O represents the output
of multi-head self-attention mechanism. Without the special declaration, we use Qi,Ki,Vi to refer to XWiQ,XW i K ,XW i V , respectively, which denotes the query, key and value in i-th head.

heading: 3 Enhanced Interactive Transformer
text: We design a novel Enhanced Interactive Transformer (EIT) in which we replace the multi-head self-attention with an Enhanced Multi-Head Attention mechanism (EMHA) that encourages consensus among different attention heads. Our method mainly modified Eq. (1) but otherwise follows the standard Transformer.

heading: 3.1 Many-to-Many Mapping Scheme
text: Intuitively, to achieve better consensus, multi-head self-attention should first contain as much information as possible. To achieve this goal, a natural idea is to employ more attention heads in multi-head self-attention. However, multi-head self-attention with too many heads suffers from low bottleneck problems (Bhojanapalli et al., 2020), resulting in performance deterioration in practical applications.
Although various strategies like attention expansion (Shazeer et al., 2020; Zhou et al., 2021b) have been proposed, the information captured in their attention maps remains limited due to an additional linear transformation step, which can introduce redundancy among the maps.

heading: Generated Attention Maps: 16
text: To alleviate this problem, we propose a novel many-to-many (M2M) mapping scheme that enables each query to attend to M keys instead of a single key. As illustrated in Figure 1, four queries and four keys can serve as two components in a bipartite graph, and each element in a component, e.g., Q1, can interact with any elements in another component, e.g., K1, . . . ,K4. Formally, supposing one with M heads, the i-th attention map can be formally calculated as:
Si = Q⌊(i−1)/M+1⌋(K(i−1)%M+1)T√
dk (3)
where i ∈ {1, . . . ,M2}, Si ∈ RT×T is the attention maps without softmax, ⌊⌋ is the round down operation and % is the mod operation. For example, S4 is computed by Q1 and K4 when M = 4.
Discussion. M2M demonstrates an increased capacity to generate M times the number of attention maps when given identical input. This enhanced capability can be attributed to the effective utilization of a many-to-many mapping strategy by M2M, which fully leverages the original head features, such as Q and K. Notably, this approach successfully avoids the production of similar attention maps by employing a dot-multiplication strategy to directly generate the attention maps (See Figure 10). By avoiding the generation of redundant attention maps, M2M improves its ability to capture diverse and distinct patterns in the input data. As a result, it facilitates the subsequent creation of more comprehensive and informative representations. This module can also be viewed as a strategy to enhance complementary principle.

heading: 3.2 Dual Enhanced Interaction
text: As aforementioned, M2M enlarges the information capacity, which provides a prerequisite for encouraging consensus among different heads. To encourage consensus, a simple idea is to directly add a linear transformation among attention maps (Shazeer et al., 2020; Zhou et al., 2021b; Wang and Tu, 2020). While these methods can achieve performance improvements in vanilla Transformer settings, they are unsuitable in our framework. One key factor is that our framework contains a substantial amount of information; however, it also incorporates certain elements of noise. Such a coarse interaction fails to attain a satisfactory consensus.
To address this problem, we propose a more refined solution that is able to differentiate between relevant and irrelevant information, discarding the latter while fully utilizing the former. Two kinds of interactions among those attention maps are introduced hierarchically, the inner-subspace interaction and cross-subspace interaction.
Two Relationships. We begin with identifying two important relationships: inner-subspace interaction (ISI) relationship and cross-subspace interaction (CSI) relationship. As illustrated in Figure 1, the inner-subspace interaction (ISI) relationship describes the connection among the attention maps generated by the same query, e.g., attention maps in the block of the same color. These attention maps
own a closer relationship. The cross-subspace interaction (CSI) relationship describes the collaboration among different heads, which exists in the attention maps generated by different queries, e.g., attention maps from blocks of different colors.
Inner-Subspace Interaction Modeling. One can adopt the standard convolution operation via batch transformation. However, such a way ignores the difference among the ISI relationship constrained by different queries, e.g., the ISI relationship in red block and blue block in Figure 1. It is desirable to preserve and enhance this distinction. To more efficiently model the interaction within subspaces, we therefore adopt group convolutions (Krizhevsky et al., 2012), which use separate parameters to process features from different groups.
Suppose f(·) as a single layer group convolution. As illustrated in Figure 2, given the output of M2M, namely S, as input, ISI sub-module computed as:
Ṡ = f (1)(ReLU(f (0)(S))) (4)
where Ṡ ∈ RM×T×T is the output of the ISI submodule. We use MHisi to represent the intermediate number of heads in ISI sub-module and set the number of groups in group convolutions to M .
Finally, we can obtain M high-quality attention maps that effectively retain the benefits of using a larger number of attention heads while discarding irrelevant information. Such a process is another key for Transformer to benefit from more heads and is unique to our work.
Cross-Subspace Interaction Modeling. To efficiently model the cross-subspace interaction, we adopt two-layer convolutions accompanied by the ReLU activation to consist this sub-module.
Let us denote g(·) as a single layer convolution. As illustrated in Figure 2, given the output of ISI, namely Ṡ, as input, CSI sub-module computed as:
S̈ = g(1)(ReLU(g(0)(Ṡ))) (5)
where S̈ ∈ RM×T×T is the output of the CSI submodule. We use MHcsi to represent the intermediate number of heads in CSI sub-module. Finally, we can obtain M final attention maps that fully leverage the benefits of each head.

heading: 3.3 Efficient Version of EIT
text: Despite the theoretical computational efficiency and parametric efficiency of group convolutions, they slow down the training in practice (Ma et al., 2018). To alleviate this issue, we provide another efficient version of EIT, namely E-EIT, by simplifying the design of dual enhanced interaction. As illustrated in Figure 3, both ISI and CSI adopt a single-layer operation. Formally, the dual enhanced interactions are computed as:
S̈ = g(0)(ReLU(f (0) (S))), (6)
where ReLU(f (0) (S)), namely as Ṡ, ∈ RMH×T×T and S̈ ∈ RM×T×T and MH is a hyper-parameter, e.g. we set it as 32 for the base configuration. In this way, E-EIT avoids parts of memory consumption and somehow improves the computational efficiency.

heading: 4 Experiment Settings
text: We evaluated our EIT on four widely used benchmarks1: 1) Machine Translation, 2) Grammar Error Correction, 3) Abstractive Summarization, and 4) Language Modeling. The detailed architecture setups, training setups and evaluation setups were presented in Appendix A.

heading: 4.1 Machine Translation
text: Dataset. We selected four widely used corpus: WMT’14 English-German (En-De) translations2 (a large-scale dataset, 4.5M training sentence pairs), WMT’16 English-Romanian (En-Ro) translations
1We also evaluated our EIT variants on some task beyond natural language processing in Appendix.
2https://github.com/facebookresearch/fairseq/ tree/main/examples/scaling_nmt.
(a small-scale dataset, 610K training sentence pairs), WMT’17 English-German (En-De) translations and WMT’17 German-English (De-En) translations3. The validation and test sets are newstest2013 and newstest2014, respectively. For the En-Ro task, it consists of 610K training sentence pairs. We preprocessed the data as the setups in Mehta et al. (2021)4. We performed shared BPE operations on both datasets to overcome the out-ofvocabulary (OOV) problem. Concretely, we set the size of BPE operations to 32K and 20K for En-De and En-Ro datasets, resulting in a shared vocabulary with sizes of 34040 and 19064, respectively.
Models. Our model architectures were based on Transformer (Vaswani et al., 2017). We provided three configurations, namely base, big in Vaswani et al. (2017), and deep in Li et al. (2020, 2021). We adopted a pre-normalization strategy (Wang et al., 2019) considering training stability under different configurations.
Training & Evaluation. We implemented our models using Fairseq (Ott et al., 2019). Training employed 8 GEFORCE RTX 3090 cards for WMT’14 En-De and 4 cards for WMT’16 En-Ro, with batch sizes of 65536 and 16384, respectively. In the En-De task, we completed 50K updates for base, 50K for deep, and 100K for big models. We utilized Adam (Kingma and Ba, 2015) with adamβ (0.9, 0.997) as the optimizer, an invert sqrt learning rate scheduler with a rate of 0.002 and 16000 warmup updates, and 0.1 label smoothing for all experiments. During evaluation, we used 4 beams with a length penalty of 0.6 for En-De and 5 beams with a length penalty of 1.3 for En-Ro. For evaluation metrics, we utilized multi-BLEU5 (Papineni et al., 2002), COMET-226 (Rei et al., 2022) and sacreBLEU (Post, 2018) scores. We ran each experiment three times and reported the average score.

heading: 4.2 Grammar Error Correction
text: Dataset. We also assessed EIT’s effectiveness for grammar error correction, a crucial natural language processing application. Our experiments were conducted on the CONLL dataset, comprising 827K training sentences. Following the setup
3https://data.statmt.org/wmt17/translation-t ask/preprocessed/
4https://github.com/sacmehta/delight/blob/mas ter/readme_files/nmt/wmt16_en2ro.md
5https://github.com/moses-smt/mosesdecoder/bl ob/master/scripts/generic/multi-bleu.perl
6https://github.com/Unbabel/COMET
in (Chollampatt and Ng, 2018), we incorporated the word-level dropout technique (Sennrich et al., 2016) to mitigate overfitting. We configured BPE operations to 30K.
Models. We selected the Transformer (Vaswani et al., 2017) and SURFACE (Liu et al., 2021) for comparison. These architectures adhere to the Transformer-base configuration outlined in Vaswani et al. (2017).
Training & Evaluation. We trained grammar error correction models on 8 GEFORCE RTX 3090 cards, using a batch size of 65536 and completing 14K total updates. Further training specifics can be found in Table 11. For testing, we configured the number of beams to 6 and the length penalty to 0.6.

heading: 4.3 Abstractive Summarization
text: Dataset. We also tested the effectiveness of EIT on abstractive summarization task, a task relying on the ability of modeling long dependency. Shared BPE operations of 30K were applied to the training data, resulting in a vocabulary of 32,584 words.
Models. Our models were all under base configuration, e.g., embedding dimension, hidden dimension, M are set to 512, 2048 and 8, respectively.
Training & Evaluation. We trained abstractive summarization models on 8 GEFORCE RTX 3090
cards, utilizing a batch size of 131,072 and completing 30,000 total updates, the same setting in Li et al. (2022a). We incorporated a weight decay strategy with a ratio of 0.0001. We set warming updates to 16000. For testing, we configured 4 beams and a length penalty of 2.0, with minimum and maximum lengths set to 55 and 140, respectively.

heading: 4.4 Language Modeling
text: Dataset. We assessed EIT in a language modeling task using WikiText-103 to investigate its capacity for handling long dependencies. The training, validation, and test sets encompass 103 million words (from 28,000 articles), 218,000 words, and 246,000 words, respectively. We adhered to the official preprocessing procedure (Ott et al., 2019)7.
Models. We chose the Adaptive Input Transformer (Baevski and Auli, 2019) as the baseline model. All models are 8-layer models with 8 heads.
Training & Evaluation. The training and evaluation settings adhered to the standard PyTorch (Ott et al., 2019) language modeling guidelines. We trained both the baseline and EIT with 286,000 updates. During evaluation, we selected the checkpoint with the best performance on the validation set. Parameters such as max-tokens, max-
7https://github.com/facebookresearch/fairseq/ tree/main/examples/language_model

heading: Method WMT17 En-De WMT17 De-En
text: sentences, and context-window were set to 3072, 1, and 2560, respectively.

heading: 5 Experiments Results

heading: 5.1 Machine Translation
text: Results of WMT’14 and WMT’16. Table 1 displays the results on WMT’14 En-De and WMT’16 En-Ro tasks. First, we can see that our EIT variants demonstrate superior performance compared to the vanilla Transformer across various configurations on both tasks. This indicates the effectiveness of EIT variants. Notably, E-EIT, an alternative to satisfy the low latency of industrial applications, can deliver competitive results compared with the full version while maintaining fast processing speeds.
Besides, Our EIT can beat all selected methods of head modification and localness modeling, including the latest methods such as MoA (Zhang et al., 2022), Fishformer (Nguyen et al., 2022), UMST (Li et al., 2022b), on both datasets. This highlights the fact that focusing on a single aspect, such as complementarity, is inadequate for achieving optimal results. By contrast, considering both complementarity and consensus leads to better performance.
Results of WMT’17. Table 2 displays the results on WMT’17 En-De and De-En tasks. We can make similar observations as with the WMT’14 and WMT’16 tasks. Specifically, EIT can achieve BLEU points of 29.58 and 35.62 on WMT’17 EnDe and De-En tasks, respectively, outperforming vanilla Transformer by BLEU points of 0.99 and BLEU points of 0.58. COMET-22 scores can further support this observation.

heading: Model RG-1 RG-2 RG-L

heading: 5.2 Grammar Error Correction
text: Table 3 presents the results of the CONLL dataset’s test set. Both EIT and E-EIT outperform the standard Transformer, showing improvements of 0.87 and 1.13 in terms of F0.5, respectively. Compared to the strong baseline SURFACE, our methods (EIT and E-EIT) still outperform it by 0.45 and 0.71 F0.5 points, respectively. Importantly, both EIT and EEIT require negligible extra parameters, less than 0.1M, indicating their enhanced expressive power. Notably, the Talking Heads model underperforms, possibly due to imperfect hyper-parameters, needing more fine-tuning.
An interesting observation is that EIT variants seem to trade recall for precision. This behavior is due to EIT’s foundation in both complementary and consensus principles, which naturally generate more precise attention maps by filtering out uncertain information. As a result, EIT primarily makes corrections where it is most confident.

heading: 5.3 Abstractive Summarization
text: Table 4 shows results on the test set of CNNDailyMail. We can see EIT can achieve scores of 41.62 ROUGE-1 points, 18.70 ROUGE-2 points and 38.33 ROUGE-L points, outperforming the standard Transformer by 0.78, 0.70 and 0.75 in terms of ROUGE-1, ROUGE-2 and ROUGE-L points, respectively. Compared with other strong baselines, our EIT can still show superiority on these datasets in terms of ROUGE-1 points, e.g., EIT surpasses SURFACE, DMAN, BOTTOM-UP, Talking-Head by 0.62, 0.64, 0.40 and 0.36 in terms of ROUGE-1 points, respectively. Notably, our E-EIT can achieve comparable performance with EIT.

heading: 5.4 Language Modeling
text: Table 5 presents the perplexity scores of various models on the WikiText-103 test set. Our EIT and
E-EIT models outperform the baseline with PPL scores of 1.11 and 0.92, respectively. These results highlight the high expressiveness of our methods, as the improvements are achieved with only a negligible increase in parameters. Also, these results demonstrate the universality of our approach as applying our approach to the decoder side can also achieve improvement.

heading: 6 Ablation Studies
text: Settings. We gave detailed description about the settings of ablation studies.
• EIT - M2M: We directly applied ISI and CSI modules to the attention maps generated by Eq. (1). Notably, in both ISI and CSI, we maintain a consistent ratio between hidden size and input size, e.g., 2 and 8 for ISI and CSI, respectively, mirroring the EIT settings.
• EIT - ISI: we directly applied CSI module to the M2 attention maps generated by M2M.
• EIT - CSI: we only applied ISI module to the M2 attention maps generated by M2M.
Results. Table 6 summarizes the impacts of removing each module on En-De and En-Ro tasks, respectively. First, we found removing any module (or sub-module) results in obvious performance degradation (#3,4,5 vs. #2). These evidences indicate the indispensability of these modules.
Notably, when removing the M2M module (#2 vs. #3), we observe an obvious decline in performance on two translation tasks, indicating the
importance of M2M module. Within our EIT framework, the M2M module, motivated by the complementary principle, serves the critical purpose of supplying necessary information for subsequent interactions. Therefore, its absence impedes the effectiveness of our two interaction models.
Furthermore, the omission of the ISI sub-module (#2 vs. #4) results in a significant and noticeable decrease in BLEU scores. One possible explanation is that while increasing the number of heads enhances the information capacity, it also introduces a certain degree of irrelevant information (noise) into the attention maps. Consequently, a direct fusion of these heads fails to yield satisfactory outcomes. However, our EIT framework overcomes this challenge by incorporating the ISI sub-module, which provides an effective mechanism for discarding irrelevant information while retaining the benefits of the previous heads. This unique and innovative design sets our approach apart from the attention expansion technique (Zhou et al., 2021b).

heading: 7 Analysis on Attention Heads Behavior

heading: 7.1 EIT owns Higher Consensus
text: As depicted in Figure 4, it is evident that EIT exhibits the highest average similarity among attention maps from various heads, surpassing all other models. This finding suggests that EIT demonstrates a greater consensus among attention heads. We attribute this achievement to the significant role played by M2M and dual-enhanced interaction. M2M facilitates the generation of rich information, while dual enhanced interaction efficiently lever-
ages and refines the available information from different attention heads.
Discussions. This phenomenon is contradictory to the findings of previous studies about head interaction (Wang et al., 2022a). We speculate that this is because our interactions are more efficient, not only relying on an adequate number of attention heads but also operating in a hierarchical manner. These characteristics result in a consensus among the attention maps.

heading: 7.2 Benefits of High Consensus
text: EIT Learns High-quality Representations We further investigate how consensus affects the layer representations. Following (Gong et al., 2021; Dong et al., 2021; Shi et al., 2022; Wang et al., 2022b), we adopt the token correlation T C to measure the quality of features (the lower, the better). The token correlation is computed by the Pearson correlation coefficient (Benesty et al., 2009).
Figure 5 exhibits the results on the test set of the En-De task. Notably, the features learned by EIT exhibit lower token correlation compared to the vanilla Transformer across all configurations. This indicates that EIT effectively learns improved layer representations.
Furthermore, we observe that the vanilla Transformer consistently maintains a relatively high token correlation from the first layer. This observation aligns with prior study (Shleifer et al., 2021), suggesting that lower layers struggle to optimize effectively in pre-normalization Transformers. However, our EIT approach alleviates this issue.
EIT Makes Head Pruning Easier To further explore the possibility of pruning the consensus attention maps, we introduce a simple head mask mechanism for head pruning during the inference phase as follows:: O = ∑M i=1 ηiA iXWiV W i O, where ηi ∈ {0, 1}. Table 7 exhibits the results on En-De tasks. Note that the head selection process is done in a straightforward manner, such as selecting heads by index, without considering their

heading: Model θ (M) MACs Time Memory BLEU
text: relative importance as highlighted in previous studies (Michel et al., 2019). Additionally, the head pruning operations are exclusively applied to the encoder side. It is evident that EIT exhibits a high tolerance for head pruning without experiencing significant deterioration in performance. Such phenomenon sheds light on the research of head pruning and inference speeding.

heading: 8 Analysis of Computational Efficiency
text: MACs Comparison. Table 8 displayed MACs comparison between EIT variants and transformer baselines. We can see that EIT can achieve an improvement of 0.87 BLEU points with an increase of just 0.1B MACs and 0.07M parameters. This indicated the efficiency of our EIT architecture. Besides, the efficient version of EIT. the E-EIT can achieve similar improvements with even fewer extra resource consumption.
Resource Comparison. In addition to theoretically exploring the efficiency of EIT variants, we also measured the practical computational consumption during the training process. Without losing generality, we focused on the model for the base configuration. We can see that EIT consumes 8% more memory consumption and 45% more training costs than the baseline with a depth of 6. To mitigate this, we have proposed the E-EIT which only costs 5% more memory consumption and 10% more training costs than the baseline but delivered similar performance compared to EIT. Notably, as shown in Table 1, the performance gap between EIT and E-EIT decreases as the model capacity increases.

heading: 9 Related Work
text: Low Bottleneck in Multi-Head Attention The “Low Bottleneck” issue in Multi-head SelfAttention (MHSA) occurs when adding more heads to Transformers, which does not correspondingly improve performance. Bhojanapalli et al. (2020)
first identified this issue, attributing it to the diminishing head dimension as the number of heads increases, which limits the creation of precise attention maps. In response, Shazeer et al. (2020) introduced “talking-head attention,” using two linear transformations around the SoftMax function to address this bottleneck. Later, Zhou et al. (2021b) proposed a framework involving “ghost heads” to enrich attention patterns, differentiating from talkinghead attention in the positioning of linear transformations and the number of ghost heads. Our approach introduces a many-to-many mapping in MHSA, using existing queries and keys for more attention maps through direct query-key multiplication.
Improved Multi-Head Mechanism Previous work has shown that multi-head attention can be further enhanced by encouraging individual attention heads to extract distinct information (Li et al., 2018; Cui et al., 2019; Sukhbaatar et al., 2019; Guo et al., 2020; Hao et al., 2019). Another branch of research is designing more complex interactive modeling to make better use of the multiple subspace information (Shazeer et al., 2020; Wang and Tu, 2020; Li et al., 2019). Besides, Voita et al. (2019) empirically demonstrates that some heads in attention are useless and can be pruned without performance degradation. Along this line, researchers investigate how to efficiently cut off redundant heads (Michel et al., 2019; Behnke and Heafield, 2020). Different from theirs, our study utilized the benefits of both diversity and consistency.

heading: 10 Conclusions
text: In this paper, we propose EIT, an alternative to the Transformer architecture. It further advances the multi-head schema by fully leveraging two principles in multi-view learning: the complementary principle and the consensus principle. In addition, E-EIT can serve as another choice considering the trade-off between performance and computation efficiency. Experimental results on four widely-used tasks demonstrate the effectiveness of EIT-variants, which deliver consistent improvements to the standard Transformer.

heading: Acknowledgments
text: This work was supported in part by the National Science Foundation of China (No.62276056), the Natural Science Foundation of Liaoning Province
of China (2022-KF-16-01), the Fundamental Research Funds for the Central Universities (Nos. N2216016 and N2316002), the Yunnan Fundamental Research Projects (No. 202401BC070021), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009).

heading: Limitations
text: Besides the advantages endowed by EIT, there still exists a shortcoming that the computational efficiency of the group convolution cannot be satisfactory, although it is computationally efficient in theory. This is due to the lack of high-efficiency CUDA kernel support. We will release a more efficient optimization of group convolutions in the soon future.

heading: A Detailed Setups of Experiments

heading: A.1 Machine Translation Task
text: Dataset We evaluated our approach on two widely used machine translation datasets: WMT’14 En-De and WMT’16 En-Ro. The En-De dataset contains approximately 4.5M tokenized training sentence pairs. We selected newstest2013 and newstest2014 as the validation and test data, respectively. As for the En-Ro dataset, it consists of 0.6M tokenized training sentence pairs. We performed shared BPE operations on both datasets to overcome the out-of-vocabulary (OOV) problem. Concretely, we set the size of BPE operations to 32K and 20K for En-De and En-Ro datasets, resulting in a shared vocabulary with sizes of 34040 and 19064, respectively.
Model Configuration Our model architectures are based on Transformer (Vaswani et al., 2017). We provided three basic configurations, namely base, deep, and big which follow the configurations in Vaswani et al. (2017). We adopted a prenormalization strategy (Wang et al., 2019) considering training stability under different configurations. The detailed settings of hyper-parameters are given in Table 10.
Training & Evaluation Our implementations are based on Fairseq (Ott et al., 2019). Our experiments are performed on the GEFORCE RTX 3090 cards. We use 8 GEFORCE RTX 3090 cards to train models for the WMT’14 En-De task. As for the models on the WMT’16 En-Ro task, we train them on 4 GEFORCE RTX 3090 cards. The batch sizes for En-De and En-Ro tasks are 65536 and 16384, respectively. The total updates are 50K, 50K and 100K for base, deep and big in En-De task, respectively. We adopt Adam (Kingma and Ba, 2015) as an optimizer with an adamβ of (0.9, 0.997). The learning rate scheduler is invert sqrt with a learning rate of 0.002 and warmup updates of 16000. We also adopt label smoothing with a ratio of 0.1 in all the experiments. More details are exhibited in Table 11. During the evaluation process, we set the beam number to 4 and the length penalty to 0.6 for the En-De task. As for the EnRo task, the number of beams is 5 and the length penalty is 1.3.

heading: A.2 Abstractive Summarization Task
text: Dataset For abstractive summarization, we conduct experiments on a widely used corpus, e.g.,
CNN/DailyMail dataset. It consists of 287K training documents. Shared BPE operations with a size of 30K are performed on all the training data, resulting in a vocabulary of 32584.
Model Configuration We only provide the base configuration of our EIT and E-EIT for abstractive summarization. The details are presented in Table 10.
Training & Evaluation We train models for an abstractive summarization task on 8 GEFORCE RTX 3090 cards with a batch size of 131072 and total updates of 30K. We adopt a weight decay strategy with a ratio of 0.0001. Other hyper-parameters are the same as that in machine translation tasks. You can find their settings in Table 11. During testing, the number of beams is set to 4 and the length penalty is set to 2.0. Besides, we set the minimal length and maximum length to 55 and 140, respectively.

heading: A.3 Grammar Error Correction Task
text: Dataset For the grammar error correction task, we select the CONLL dataset to evaluate our approach. The CONLL dataset consists of 827K training sentences. We replicate the setup in Chollampatt and Ng (2018) and adopt the word-level dropout technique (Sennrich et al., 2016) to alleviate the overfitting problem. More details are listed in Table 9.
Model Configuration For the grammar error correction task, we only provide the base configuration of our EIT and E-EIT. The details are presented in Table 10. Notice that the models on this task adopt a post-normalization strategy.
Training & Evaluation We train models for the grammar error correction task on 8 GEFORCE RTX 3090 cards. The batch size is 65536 and the total updates are 14K. More training details are shown in Table 11. During testing, the beams and length penalty are set to 6 and 0.6, respectively.

heading: A.4 Automatic Disease Diagnosis Task
text: Dataset For the automatic disease diagnosis task, we select the ABIDE dataset to evaluate our approach. The ABIDE dataset consists of 1009 brain networks from 1009 real samples of 17 international sits. Due to the heterogeneity of this data, we adopt the shared data with re-standardized data splitting in Kan et al. (2022). Specifically, 70%,
10% and 20% samples are served as the training, validation and test sets, respectively.
Model Configuration For ABIDE task, we still follow the model configuration in Kan et al. (2022). Specifically, we build our BrainNetEITF with a
two-layer encoder. The number of heads M is set to 4 for each layer.
Training & Evaluation We train all models including the BrainNetTF and BrainNetEITF for 200 epochs on a single GEFORCE RTX 3090 card. Each model is trained by 5 times. We adopt Adam (Kingma and Ba, 2015) as an optimizer with an initial learning rate of 10−4 and a weight decay of 10−4. The batch size is set to 64. We adopt the checkpoint of the final epoch for evaluating the test set.

heading: A.5 Language Modeling Task
text: Dataset For the language modeling task, we select the WikiText-103 dataset to evaluate our approach. The training set consists of 103M words from 28K articles. While for the validation and test sets, they are made up of 218K and 246K words, respectively. In detail, we follow the instructions in Fairseq (Ott et al., 2019) to obtain and preprocess the data. The details are listed in Table 9.
Model Configuration For WikiText-103 task, Both the baseline and our model are all 8-layer big models with 8 heads. Note that the baseline we adopted is adaptive input transformer (Baevski and Auli, 2019). In this task, the kernel sizes in DEI are all set to 1.
Training & Evaluation The training and evaluation settings all follow the standard instructions for language modeling in PyTorch (Ott et al., 2019). We train both baseline and EIT with 286000 updates. The details are given in Table 11. As for the evaluation process, we adopt the checkpoint performing best on the validation set. We set the maxtokens, max-sentences, context-window to 3072, 1 and 2560, respectively.

heading: B Details of Metrics

heading: B.1 Calculation of Head Distance
text: Inspired by the attention metrics in Zhou et al. (2021a) and Wang et al. (2022b), we measure the distance between different heads by calculating cosine similarity among attention maps. Notice that our metric focuses on the diversity of attention maps, which is quite different from them. Denote the dataset as D, and the attention map of h-th head of l-th layer of i-th sample denotes as A(h,l,i), the head similarity in l-th layer is computed by averaging the cosine similarity of every two heads in i-th layer across all samples as:
HD(l) = 1|D| 1 M(M − 1) 1 T ×
|D|∑
i=1  M∑
j=1
M∑
k=1
T∑
t=1
C(A (j,l,i) t,: ,A (k,l,i) t,: )−M
 
(7) where |D| denotes the size of dataset, M is the number of partition of features in attention, T is the sequence length and C(·) denotes the cosine similarity function. We set D to the test set of the corresponding task. The obtained head similarity ranges from [0, 1]. The larger the head similarity, the lower the distances between different heads are.

heading: B.2 Calculation of Token Correlation
text: We define a metric T C, which measures the correlation among the representations of different tokens. Denote the dataset as D, and the sequence representation of i-th sample in l-th layer denotes as X(l,i), the token correlation of in l-th layer is computed as:
T C(l) = 1|D| 1 T (T − 1)
|D|∑
i=1
( T∑
j=1
T∑
k=1
ρ(X (l,i) j ,X (l,i) k )− T )
(8)
where ρ(·) denotes the pearson correlation function. Intuitively, the larger the T C is, the higher the token correlation is, degrading the model’s learning capacity (Gong et al., 2021).

heading: B.3 Efficiency Comparison
text: Despite the performance evaluation, memory consumption and computational cost are also two major concerns in the literature. Figure 6 also displays the memory consumption and computational cost of models on the En-De task. EIT only costs 8.5% more memory consumption and 44.4% more training costs than the baseline with a depth of 6. However, the extra consumption goes larger as the depth goes deeper.
Besides, as aforementioned, we elaborately design an efficient version E-Eit that only costs 9.4% more memory consumption and 21.7% more training costs than the baseline under all the configurations on average. In this work, the many-to-many mapping rule is only applied on the encoder side.
This is because the proposed M2M module and the subsequent ISI and CSI sub-modules will significantly enlarge the inference cost due to the heavy use of product attention on the decoder side, although it can attain further benefits in terms of BLEU.
C Visualization of Training and Validation Perplexity
We plot the training and validation perplexity of Transformer and our EIT on the WMT’14 task in Figure 7. We can see that our EIT owns lower training and validation perplexity than Transformer.

heading: D Hyper-Parameters Analysis (Kernel Size and Hidden Size)
text: Since there are several hyper-parameters in both ISI and CSI sub-modules, it is necessary to figure out how they affect performance. Figure 8 (a-d) plots the performance of EIT against the kernel size and the hidden size. We can see that EIT can outperform Transformer in all choice of kernel size and hidden size. This observation can further help us trade off efficiency and performance well. For example, we can set CSI kernel size to 1 or ISI kernel size to 3 or MHisi to M2 or MHcsi to 4M to own a more efficient EIT.

heading: E Local Analysis
text: Local modeling is one of the widely accepted ways to improve the expressiveness of Transformer (Yang et al., 2019; Fan et al., 2021; Li et al., 2022b). In dual enhanced interaction, we apply convolution operations to attention maps, which has the potential to introduce local biases. To figure it out, we measure the localness of attention maps since if there is a local bias, each token will distribute larger attention weights on their neighboring tokens. We adopt the localness metric of
Fan et al. (2021), denoted as C (higher is better). More details are presented in the Appendix.
We plot the C value within a local region w = 0.1 ∗ T + 1, of models in En-De task and CNNDailyMail task in Figure 9. The value is computed over the test set. Due to the long sequence length, we only use a subset of the test set consisting of 1000 samples for the CNN-DailyMail task. The results (mean) show no significant local enhancement phenomena in both tasks. Note that the attention maps in the first layer of EIT on the abstractive summarization have a strong local pattern, but the kernel sizes are set to 1 on this task. So we conclude that the improvements do not come from local enhancement.

heading: F Evaluation on Automatic Brain Disease Diagnosis Task
text: We further inspect the potential of EIT to be severed as a general method beyond language tasks. The automatic brain disease diagnosis, a disease classification task that highly relies on precisely learning relationships among different brain regions has recently been dominated by graph convolution (Wen et al., 2022) and Transformer (Kan et al., 2022). We select a widely used real-world fMRI dataset: Autism Brain Imaging Data Exchange (ABIDE), which consists of 1009 brain networks from 17 international sites, of which 516 samples are autism spectrum disorder patients. We follow the preprocessing setup in Kan et al. (2022) and adopt the CC200 (Craddock et al., 2012) as the Regin-of-
Interest (ROI) partition template. We select two latest methods, the Mvs-GCN (Wen et al., 2022) and BrainNetTF (Kan et al., 2022), as our comparison. The experimental setups and configurations of our BrainNetEITF and BrainNetEEITF are the same as in Kan et al. (2022). Each experiment is conducted 5 times and we report the mean and standard deviation of the four metrics: Accuracy (ACC), AUROC, Sensitivity (SEN) and Specificity (SPE).
Results We exhibit the ACC, AUROC, SEN and SPE of different models in Table 12. We can see that both BrainNetEITF and BrainNetEEITF can outperform all the baselines in terms of all metrics. Similarly, thanks to the little increased parameters of our models, we can conclude that they have stronger expressiveness and can be easily extended to other scenarios.

heading: G Further Analyses

heading: G.1 Effect of Number of EIT Layers
text: Recent research (Shi et al., 2016; Peters et al., 2018; Hao et al., 2019) has demonstrated that various layers in the encoder of a model have a tendency to capture distinct syntax and semantic features. Consequently, each layer may have different requirements for promoting agreement among the representations. In light of this, we examine the
impact of consensus on different layers. The results of the En-De task are presented in Tables 13. The lowest layer clearly benefits from a higher degree of consensus compared to other layers, consistent with prior research (Shleifer et al., 2021) indicating the challenges of optimizing shallow layers within the pre-normalization paradigm. However, by employing the consensus strategy, we enhance the learning of representations in shallow layers, giving them a significant advantage. Additionally, it is observed that incorporating consensus into a small subset of all layers can also yield good results, e.g., 28.08. These findings suggest two insights: 1) Our EMHA is so powerful that can work well even only being applied to a small subset of all layers; and 2) more efficient utilization of consensus may achieve better performance while working more efficiently.

heading: G.2 Dynamics of Attention Map Similarity during Computation
text: Figure 10 exhibits the dynamics of attention map similarity for the EIT 48L model on the En-De test set. The similarity between attention maps initially decreases and then increases as the dual interactions progress. This pattern is attributed to the two stages of our approach. In the ISI phase, interactions are modeled within each group instead of the whole, generating representative attention maps. As these groups operate independently, the similarity among these representatives is lower. Subsequently, in the CSI phase, interactions occur among these representatives, resulting in the final attention maps. This CSI enhances similarity among the attention maps, achieving the consensus.

heading: G.3 Back Translation Experiments
text: In this section, we further investigate the effect of back-translation on our EIT. We conducted experiments on WMT’17 De-En and En-De to avoid misleading evaluation. Concretely, we randomly selected 7M monolingual German monolingual data and filtered the sentences by length (ones longer than 10 and shorter than 200 would be reserved).
Then we used the already trained the WMT’17 DeEn EIT model to generate the translations via top-k sampling (which is a more robust back-translation strategy than the original beam search. The beam is 4 and the length penalty is 1.5). We presented the results in Table 14.
When the back translation was applied, the vanilla Transformer and our EIT achieved BLEU scores of 29.53 and 30.39, respectively. This indicates that even under the back translation setting, our EIT can still achieve consistent performance improvements.

heading: Future Work
text: We will release a more efficient optimization of group convolutions in the soon future