Transformers with Competitive Ensembles of Independent Mechanisms
Authors: Alex Lamb
Di He
Anirudh Goyal
Guolin Ke
Chien-Feng Liao
Mirco Ravanelli
Yoshua Bengio
Venue:  (2021)
Keywords: transformer
mechanism
modularity
modules
independence

Abstract: An important development in deep learning from the earliest MLPs has been a move towards architectures with structural inductive biases which enable the model to keep distinct sources of information and routes of processing well-separated.  This structure is linked to the notion of independent mechanisms from the causality literature, in which a mechanism is able to retain the same processing as irrelevant aspects of the world are changed.  For example, convnets enable separation over positions, while attention-based architectures (especially Transformers) learn which combination of positions to process dynamically.  In this work we explore a way in which the Transformer architecture is deficient: it represents each position with a large monolithic hidden representation and a single set of parameters which are applied over the entire hidden representation.  This potentially throws unrelated sources of information together, and limits the Transformer's ability to capture independent mechanisms.  To address this, we propose Transformers with Independent Mechanisms (TIM), a new Transformer layer which divides the hidden representation and parameters into multiple mechanisms, which only exchange information through attention.  Additionally, we propose a competition mechanism which encourages these mechanisms to specialize over time steps, and thus be more independent.  We study TIM on a large scale BERT model, on the Image Transformer, and on speech enhancement and find evidence for semantically meaningful specialization as well as improved performance.  

Under review as a conference paper at ICLR 2021
TRANSFORMERS WITH COMPETITIVE ENSEMBLES OF INDEPENDENT MECHANISMS
Anonymous authors Paper under double-blind review
ABSTRACT
An important development in deep learning from the earliest MLPs has been a towards architectures with structural inductive biases which enable the model to keep distinct sources of information and routes of processing well-separated. This structure is linked to the notion of independent mechanisms from the causality literature, in which a mechanism is able to retain the same processing as irrele- vant aspects of the world are changed. For example, convnets enable separation over positions, while attention- beased architectures (especially Transformers) learn which combination of positions to process dynamically. In this work we explore a way in which the Transformer architecture is deficient: it represents each posi- tion with a large monolithic hidden representation and single set of parameters which are applied over the entire hidden representation. This potentially throws unrelated sources of information together, and limits the Transformer's ability capture independent mechanisms. To address this, we propose Transformers with Independent Mechanisms (TIM), a new Transformer layer which divides the hidden representation and parameters into multiple mechanisms, which only exchange information through attention. Additionally, we propose a competition mechanism which encourages these mechanisms to specialize over time steps, and thus be more independent. We study TIM on a large-scale BERT model, on the Image Transformer, and on speech enhancement and find evidence for semantically meaningful specialization as well as improved performance.
INTRODUCTION
major theme throughout the history of deep learning has been the introduction of inductive biases in neural architectures, more recently with focus on the ability to dynamically keep distinct types of information separated. While an MLP architecture has one large hidden representation at each layer, convnet keeps different spatial positions' representations separated by default. This separation enables more appropriate reuse of parameters, improving generalization (c.g. compared with a fully connected MIP) by ensuring that some parts of the hidden representation capturing some aspects of the data can remain unchanged when other aspects are changed. Additionally, it is important to be able reuse parameters in all situations where the parameters are relevant, and use parameters in positions where they are irrelevant, and this is where attention mechanisms can be very useful.
While dividing information between different positions (for example time steps spatial positions) is already very useful, it has been recognized from the earliest deep learning work on the notion of disentangling (Bengio) 2009] Glorot al., 2011] Rifai al.) 2012 Mathieu et al., [016 Achille & Soatto 2018) that other features of the data could ad advantagcously be kept well-separated, even over overlapping sets of positions. This has suggested the idea that model can be decomposed into multiple components, which are often called modules, each operating on different set of features. Modularity has been identified as an essential ingredient for generalization in machine learning (Ronco et al.) 1997 Alet et al.) 2018 Goyal et al.) 2019). The motivating intuition is that if the relationship between the modules changes between training and evaluation, then a model which keeps these modules sufficiently separate but can adapt how they are combined could be more robust. It can even be robust to changes where the overall data distribution differs between training and evaluation. This has been studied in the causality literature through the notion of "Independent Mechanisms"

---

Under review as a conference paper at ICLR 2021
Toanhheee layer
Monolttii States
Head
Head?
Atantion
PL41
TIKL Rye AIR
Mechanism
Altition
Mechaniem
Figure show a simplified version the model at a single position to illustrate the difference between heads and mechanisms (left). Heads allow for parallel attention, but the differentiation between heads is transient: it begins with the projection layer and ends immediately following the attention. As a result, most of the parameters are not head-specifie. With independent mechanisms, the information is kept well-separated throughout the entire layer, and all of the layer's 's parameters are specific a single mechanism. Competition patterns of an unsupervised image Transformer on CIFAR images (top right) with 2 mechanisms shows that mechanisms learn to specialize over foreground and background patterns on an early layer (center right) and become more confident in later layer (bottom right).
(Peters et al. 2018: Parascandolo et al. 2018) or causal modules, which can be flexibly re-combined, re-used, and re-purposed.
While modularity and independent mechanisms ideas are closely related, the latter has special focus on the notion that mechanisms should have the ability to remain unchanged when unrelated aspects of the world are changed. In that sense it is a more specific idea which builds on the more general concept of modularity. While the study of independent mechanisms in the context of deep architectures is relatively recent (Goyal et al. 2019 Mittal et al. 2020), a few ideas are considered central. One is that mechanisms are separately parameterized (or dynamically parameterized, with the possibility of separation), which means that the function computed by module remains the same even as other mechanisms need be changed. Another central idea is specialization between mechanisms, which is the idea that mechanisms should seek to only model some parts of the world. One way to help accomplish this is by forcing the mechanisms to compete to explain different positions (in time or space), such that some mechanisms would not be used by the model on positions where they are less relevant.
In this work we explore how the idea of independent mechanisms can be beneficial in the Transformer architecture. Transformers (Vaswani et al., (017)) are based on information sharing across positions controlled dynamically by soft-attention mechanism (Bahdanau et al.) 2014), while still using fully-connected MLP to process the extracted feature vectors (concatenated over a set of attention heads) at each position. An important way in which this improves over convnets is that if this attention becomes sufficiently sparse, then it gains the ability to keep information well-separated between different positions. At the same time, each position, the Transformer stores single monolithic hidden representation, over which it applies its entire set of parameters. For example, if we consider a generative model of images of animals in field, then some of the parameters like those describing how animals have symmetric eyes or a certain number of feet, are only relevant for the positions in the image where the animal is present. A normal Transformer, however, would apply the same parameters to the entire hidden representation at all spatial positions. Additionally. if sources of information need be accessed over multiple positions, it has no way to keep that information well-separated between parts of the hidden representation, unless large fraction of the parameters are set to exactly zero. In practice, models tend not learn these sorts of highly sparse parameter matrices it is not necessary in order to fit the training set. Thus different underlying factors tend to be freely blended together rather than disentangled: we hypothesize and show empirically that this leads to deteriorated generalization when something about some of these factors changes.
Our newly proposed technique, which we call Transformers with Competitive Independent Mecha- nisms (TIM) seeks to address this limitation of the Transformer by dividing the hidden representation and parameters into multiple distinct mechanisms. These mechanisms perform self-attention (over input elements) separately, and information is exchanged sparingly between the mechanisms using
2

---

Under review as a conference paper at ICLR 2021
attention. Thus the model is naturally compelled to keep multiple information signals well-separated, even within a single position. Moreover, only the parameters corresponding to an activated mecha- aspect of the hidden representation. The process of selectively
nism called upon, focusing activating some mechanisms and not others relies on competition between mechanisms, just like in recurrent independent mechanism (RIMs) (Goyal et al. 2019). We hypothesize and show empiri- cally that this provides an inductive bias encouraging the mechanisms to be more independent and specialized, more robust to changes only affecting other mechanisms.
2
TRANSFORMERS WITH COMPETITIVE INDEPENDENT MECHANISMS
2.1 PRELIMINARIES
Multihead Self-attention sub-layer The attention mechanism can be formulated querying dictio- nary with key-value pairs (Bahdanau et al. 2014 Vaswani et al. 2017). e.g., Attention K, V) = softmax QKT/ / (mmdde) V, where dmoodd is the dimensionality of the hidden representations and Q (Query), K (Key). V (Value) are specified as the hidden representations of the previous layer in the so-called self-attention sub-layers in the Transformer architecture. The multi-head variant of attention allows the model to jointly attend information from different representation subspaces, and is defined as Multihead( Q, K. V) = Concat Concat(heady, ... head W with the heads defined as: headk = Attention & KWW A V where W R x xxx W K E Rdmastdx WV E Radma xvv and t Hdd xdmadsi are project parameter matrices, H is the number of heads, and dg and dv are the dimensionalities of Key and Value.
Group Linear Layer: It takes multiple hidden representations, and applies a separately parameterized linear transformation to each. This operation can be efficiently implemented using batched-matrix multiplications. We set the numbers of groups n,, and define weight tensor W E IR IRn.xda> x doue If the input h is shaped as e II Rdx, then we can define the layer as: GroupLinear(t h. W,ne) = [h, W51
2.2 TIM ALGORITHM
We first lay out the parts of a TIM layer and then give more detailed steps in Algorithm [ We then give a high-level detail of how to turn a transformer layer into a TIM layer in a typical implementation (Section) 2.3) An illustration of how independent mechanisms differ from heads given in Figure
2.2.1 COMPETITION BETWEEN DIFFERENT MECHANISMS
Aside from having separate parameters and only exchanging information via mechaneecanism attention, we wanted to create a stronger inductive bias to encourage the mechanisms to specialize. To do this, we created competition system in which each mechanism has a layer which outputs single scalar value (as a function of the current layer's ' representation), and these are passed through softmax over the different mechanisms (this softmax is applied position-wise and separately for each layer). The value of this softmax is then used to weight how much each mechanism is allowed to update its representation after the self-attention. This competition score is computed as c = softmax (GroupLinear(M (h, W W"nn))). mn))). where we note that each mechanism has its own parameters for the layer (hence the use Group Linear layer instead of a normal linear layer). Thus the n, modules have per-step weighting for how much they are able to read during the later self-attention stage. Thus if one mechanism wants to perform attention on a given position. it suppresses the other mechanisms that position. We found that this often improved results and that these softmax scores are fairly interpretable as a measure of specialization. Exact equations for this step are given in Step 1 and used in Step in Algorithm inn the appendix.
2.2.2 EACH MECHANISM SHARES INFORMATION ACROSS TIME AND PROCESSES INFORMATION
This step allows each mechanism to have its own independent dynamics, which are themselves similar normal transformer layer. These independent dynamics allow each mechanism to read information from other time steps using attention and process that information using FFN layers. We modify the self-attention sub-layer and forwad sub-layers (FFN) be mechanism-we wise as well
3

---

Under review as a conference paper at ICLR 2021
as position-wiid wise, with separate parameters for each mechanism. Additionally, the nnyrr-oormalization is modified to be performed separately for each mechanism. The projections and FFN sub-layers can be modified by replacing the linear layers with group linear layers. When performing the self-attention itself, the mechanisms behave the same as heads, and thus we can use the same type of multi-head attention process, so long the total number of heads is divisible by the number of mechanisms. One notable property is if TIMs only consisted of this part of the model (independent dynamics) by itself, then each TIM would be completely independent transformer model with its own forward pass and its own parameters. Steps 2 and 4 in the appendix, Algorithm I
give more detail on this step.
2.2.3 ATTENTION IS USED TO COMMUNICATION INFORMATION BETWEEN DIFFERENT MECHANISMS
Although we allow each TIM to remain independent and process information independently, it is also important to allow the different mechanisms in TIMs to share information between each other (in case the TIMs are not truly fully independent). To do this we use a standard multi-head attention sub-layer to share information between the mechanisms, which is done in position-wise fashion. We made this attention mechanism relatively small, with just 2 heads with 32 units each. This is because we want the different mechanisms be as independent as possible, and thus only share small amounts of high level information. This can be thought of as another attention layer, where we treat the different mechanisms as positions, and perform this attention in parallel over the different steps in the sequence. More details on this are given in Step 3 in the appendix's 's Algorithm
2.3 IMPLEMENTING AND INTEGRATING TIM
The TIM layer is a drop-in replacement for standard Transformer layer and turning an existing Transformer layer into TIM layer surprisingly straightforward. It is a drop-in replacement for single layer which can be flexibly used in variety of models and architectures (both encoders and decoders). A simple strategy is a normal hidden representation is of shape (T, b, dmooee), then our TIM hidden representation should be reshape-able to (T, b, R2. dmodel nn).. First, each layer linear layer in the existing Transformer layer should be replaced by a group-wis wwipewise linear layer implemented using batch matrix multiplication. Second, long as the number of heads is divisible by the number of mechanisms, the self-attention does not need to be changed, since mechanisms behave interchangeably with heads in this part of the model. Third, the inter-mechanism communication can be added as a drop-in module into the Transformer layer. Finally, the competition layer is just a single layer with softmax, which can easily be added.
Although TIM is a drop-in replacement for a normal Transformer layer, there are a few subtleties that must be considered for successful integration. First, if the total size of the hidden representation is kept the same, integrating TIM drastically reduces the total number of parameters in the model because all of the linear layers are replaced by grouped-lineat layers (which can be thought of as having a block-sparse structure). This step by itself reduces the number of parameters by a factor of n., but a TIM layer also adds new parameters to the model through the addition of the Inter-meche mechanism Attention Sub-Layer and Techanism-Compettiine small. In practice a TIM layer usually reduces the number of parameters by about 30-40%, depending on the exact hyperparameters. To compensate for this, in all of our experiments we increased the total hidden size to match the number of parameters of the original model, usually by about 20%.
Sub-Layers, although both of these are rather
Additionally, while we initially thought that it would make sense replace every Transformer layer with a TIM layer, when we analyzed the mechanism mechanism-compettiee competition, we found that it was almost always completely flat the first layer, which suggested to that the first two layers well as the last layer should be kept as normal Transformer layers.
3 RELATED WORK
Specialization and Competition over heads in Transformers. Cui et al. (2019) proposed mixed
multi-head attention mechanism which forces some heads to learn specific patterns, such as attending to precedent/local tokens only. Clark et al. (2019) studied which positions attention heads focus on and found that some heads have specific patterns, such as attending locally. |Vig et al. 20020) showed that the heads in model of protein sequences semantically meaningful. An al.
(020
4

---

Under review as a conference paper at ICLR 2021
considered adding a repulsive foree to the heads in Transformers to try make them more specialized. In our view, this evidence for specialization over heads is complementary with our results.
Independent Mechanisms and Modularity in Transformers. We're not aware of any work which breaks a Transformer's ' hidden representation into multiple mechanisms with separate parameters which interact through attention, though some works hint at this direction. The Group Transformer (Park et al. 2020) replaces the fully-connecte
layers with group-linear layers and uses low-rank layers pass information between the groups. The universal transformer (Dehghani et 2018) shared parameters between layers and updated using gating, and this gating could behave similarly to the competition that we propose but lacks the idea of having multiple mechanisms.
Independent Mechanisms in Recurrent Networks. The idea of independent mechanisms has seen a significant amount of focus in recurrent networks (Goyal al. 2019). The idea to parameterize the model as an ensemble of mechanisms, having their own dynamics, but sparingly interacting with each other using a bottleneck of attention. In the case of recurrent networks, dividing the hidden representation into mechanisms has the advantage that at a particular time-step, only a fraction of mechanisms can be active, and hence computation sparse in time, where in the case of transformers, imposing the idea of independent mechanisms in some higher layers has the added advantage that computation can be sparse both in space (i.c., position ) as well as time.
4 EXPERIMENTS
We seek to answer two questions in our experiments. First, do the mechanisms that we learn with TIM specialize in sensible and semantically meaningful ways? We analyze this both on toy datasets where we have clearly independent mechanisms by construction (Figure 2) and on large-scale realistic speech and NLP tasks (Figure and Figure Our second question how using a model which learns these independent mechanisms leads to better quantitative performance, both on the original task and on transfer learning, which we demonstrate in Figure ] and Table [
4.1 IMAGE TRANSFORMER: EVIDENCE OF SPECIALIZATION
We integrated TIM into the Image Transformer, which is generative model which generates image pixel-by-pixel, with small-scale variant of the GPT-2 architecture (Karpathy) 2020: Radford et al.. 2019). We first considered a pedagogic task in which the dataset consists of two clearly independent mechanisms, Our synthetic task uses MNIST digits (LeCun & Burges 1998) and CIFAR images Krizhevsky (2009) of small realistic images of animals and vehicles. Each example in our constructed dataset consists of an MNIST digit on its left-side and CIFAR image on its right-side, with these two examples selected randomly. It is clear that two sides of the image are independent and have completely different types of content, and thus it is natural for each mechanism to specialize over single side.
When training with TIM on this dataset, we found that we were able to nearly exactly recover a competition pattern which the mechanisms specialize over the two sides of the image (Fig. 2 middle). Intriguingly, this specialization does not appear at the very beginning of training, in which the mechanisms mostly specialize over the lightness or darkness of the pixels. However as training progresses, the two sides of the image become increasingly specialized to one mechanism the other (Figure 2). We also experimented with the CIFAR- 10 datasct, and found that integrating TIM led to superior test-set likelihoods. Moreover we visualized the competition pattern with TIM on CIFAR-10 and found a specialization between foreground and background regions in the images (Fig. [ right).
4.2 SPEECH ENHANCEMENT
Speech enhancement aims to improve the quality of speech recordings. speech signal captured in real environments, in fact, is often corrupted by noise and reverberation that might severely affect its intelligibility. Speech enhancement has long been studied in the research community (Jacob Benesty & Chen 2015). Traditional approaches were based on signal processing techniques such as spectral-subtracee
or Wiener filtering (Boll 1979 Ephraim & Malah| 1984 Scalart & Filho 1996). The idea behind these methods is to estimate the noise in non- speech segments and remove it from speech regions. End-to-end deep learning- based speech enhancement has turned out to significantly outperform traditional signal processing methods, and recently using Transformers
5

---

Under review as a conference paper at ICLR 2021
Mechanism Activation Side Total Variance Rabio
7
6
7
20
Lyve
Lyye
4
4
6
6
0
4
3
9
03
02
kpach
Figure 2: We trained Image Transformer (pixel-by saseel-b-pieelll raster-order generative model) a datasct where we construct new images in which the left-half of cach image is MNIST digit and the right-half of the image is a random CIFAR example (left). We found that with n, = 2, the two mechanisms in TIM learn to specialize over the two sides of the image, with one TIM only activating on the MNIST digit and one TIM only activating on the CIFAR example (activation patterns shown in center). We measured the relative variance in the mechanism-acttivatoon over the two sides of the image, showing that the specialization becomes much stronger over the course of training (right).
has led to promising performance (Kim et al. 2020). We believe that TIM fits well with this task because the traditional technique of decomposing the signal into speech and noisy parts and then analyzing these two signals separately embodies the desiderata of independent mechanisms.
Table (left) compares the performance achieved by TIM with other recent systems on the widely- studied Deep Noise Suppression (DNS) dataset (Reddy et al. 2020). DNS is a large corpus composed of roughly 441 hours of clean and noisy speech samples. The clean speech is artificially corrupted with noise sequences from the audioset database, which contains two million auman-labeled clips drawn from YouTube videos and belong about 600 audio events. Noise is added to the clean speech signal using a random signal-to- sonalltt-nois-raatti (SNR) ranging from 0 to 40 dB. We replaced all Transformer layers except for the first two and the last layer with TIM layers and we increased the total number of hidden units and heads (by about 20%) to match the number of parameters of the baseline, and we used two mechanisms (but achieved slightly worse yet better-than-baseline results with n, = 4). The systems are evaluated with the Perceptual Evaluation of Speech Quality (PESQ) score (Rix et al. 2001). To assess the generalization capability of TIM, we tested our model on the Voicebank test-set as well (see Table r-ight). Voicebank (Thiemann et al.) 2013). in fact, is characterized by noisy conditions different from that of the DNS dataset used for training.
The results, shown in Table [ highlight that TIM slightly outperforms the ecently-proposee PocoNet (Hu et al.] 2020) model, which uses additional data and has times the parameters of TIM. To the best of our knowledge, TIM achieves the best PESQ performance far published in the literature on the DNS dataset. Qualitatively, we found that the competition scores matches our intuition. Indeed, the two mechanisms clearly specialize over speech and non-speech parts of the audio sequence, as shown in Figure 3 Moreover, we intriguingly found that this competition between mechanisms is consistent across layers, starts out with low confidence, and becomes increasingly confident in later layers. Compared to a standard Transformer, TIM shows superior generalization capabilities. This interesting feature can be appreciated in Table I (right), where we tested our model on a different dataset ( (VoiceBank). In mismatch conditions, the competition mechanism seems to play a crucial role. This finding agrees with our intuition, according to which employing specialized and competing modules can make the model less affected by irrelevant changes of the input distribution.
4.3 BERT PRE-TRAINING AND FINE-TUNING
BERT (Devlin et al. 2018 is one of the most popularly used methods to learn the representation of natural language. The BERT model uses a multi-layer Transformer encoder and is trained by the masked language modeling task using Web data corpus (Liu et al. 2019). The pre- pre-trained contextual sentence representations have been shown to be effective in large number of downstream tasks.
For BERT, we replaced all of the transformer layers except for the first two layers and the last layer with TIM layers (we also report result where all layers are TIM layers, showing that it leads to worse performance). We used two mechanisms and evenly increased the number of hidden units and total number of heads across all layers to match the number of parameters in the baseline model.
Pre-training Following Devlin et al. 2018), we used English Wikipedia corpus and BookCorpus for pre-training. By concatenating these two datasets, we obtained a corpus with roughly 3.4 billion

---

Under review as a conference paper at ICLR 2021
Table I:
We trained TIM on the DNS speech enhancement datasct and evaluate on the DNS test-set (left). To assess zcro-shot transfer generalization
evaluate it the voicebank test-sct well (right). The performance reported on wideband PESQ (higher is better). TIM reaches statc-of-the
performance and shows improved generalization capabilities in mismatch conditions. Results with (*) used additional outside datasets for training.
For external baselines (a) is (Choi et al. 2020). (b) is (Koyama et al. 2020), and (c) is (Isik et al. 2020
Models
Param (M)
DNS (PESQ)
(Trained DNS)
Noisy no reverb
n/a
1,582
Models (Trained on DNS)
VoiceBank (PESQ)
U-Net-MultiScalct (a)
3.5
2.710
Conv-TasNet (b)
5.1
2.730
Noisy reverb
1.970
PoCoNet (c)
50.0
2.722
Transformer Baseline
2.517
PoCoNet-SSL (c)
50.0
2.748
TIM-NoComp(n.=4)))
2.503
Transformer Baseline
6.1
2.727
(IM-Comp(1,:2))
2.575
TIM-NoComp(n,=22
6.0
2.754
TIM-Comp(n,=4)
2.540
TIM-Comp(n,=2)
6.0
2.742
TIM-Comp(n,, Comp(n),=p)
6.0
2.730
Correlation Between Layers
0.7
b1.
008
0.5
-07 0.4
Layar
Layer
Figure 3: An examples a speech signal (left) with their respective competition patterns over successive TIM layers (ordered from top to bottom). In the early layers, the competition is uncertain, but becomes more certain in the deeper layers. This is further quantified in correlation matrix of competition over layers (middle) and a plot showing that competition entropy drops later layers, especially at the lowest percentiles (right).
words in total. We trained all model variants with the same procedure and hyperparameters which were tuned on the BERT baseline model. All models were run on 16 NVIDIA Tesla V100 GPUs.
Fine-tuning We used MNLI, MNLI-MM, QNLI, SST-2 and STS-B from the GLUE (General Language Understanding Evaluation) dataset (Wang al. 2018) the downstream tasks to evaluate the performance of the pre- pre-trained models. Ideally the features learned by BERT would remain useful on these distinct tasks which have relatively small training sets.
Results The overall comparison results are shown in Table We found that both TIM-NoComp and TIM-Comp achieve lower perplexities (masked language modeling loss) on the validation dataset compared to the two BERT baselines. We found generally better and more reliable (less variance between seeds) results when fine-tuning experiments with TIM. These empirical results show that our proposed TIM is better model architecture in a wide range of natural language applications.
4.4 DISCUSSION: RNN MODULARITY VS. TRANSFORMER MODULARITY
single layer RNN already fairly powerful model which can have strong priors to inform how to select mechanisms (or modules more generally). However, a single layer Transformer is rather weak model, as can only base its representations on a single round attention based upon the individual tokens and their position encoding. We've consistently found that the quality of the mechanism-coe commmiismmommmtttiiion is poor in the first layer of a Transformer network and that performance is substantially improved by making the early layers ordinary Transformer layers rather than TIM layers. This is in contrast with what has been observed with RNNs, where improvements can be obtained by using multiple modules multiple mechanisms even in single layer model.
7

---

Under review as a conference paper at ICLR 2021
Table 2: We compare the baseline BERT models to TIM with and without competition, both validation likelihood (perplexity) and NLP fine-tuning tasks (reported as accuracy, with median and standard deviation over five fine-tuning trials with different seeds). We also show that it is essential to make the first and last layers normal Transformer layers and not TIM layers TTIM-All-Layery
Result
BERT
BERT-130M
HIM-AllLayer
TIM-NoComp 9/12
TIM-Comp 9/12
TIM Layers
0/12
0/12
12/12
Parameters
110M
130M X
110M
130M X
130M
Competition?
Valid-NLL
2.096
2.040
2.112
2.033
2.027
MNLI-M
84.93 + 0.15
85.37 +: 0.29
84.19 + 0.34
85.89 + 0.17
85.28 + 0.22
MNLI-MM
84.91 0.18
85.28 + 0.27
84.55 0.15
85.80 + 0.07
85.17 + 0.18
QNLI
91.34 0.21
91.84 = 0.32
91.37 = 0.59
91.78 0.14
91.97 + 0.20
SST-2
92.88 = 0.33
92.75 = 0.26
92.52 0.56
92.75 0.13
92.97 + 0.25
STS-B
89.43 + 0.25
89.34 = 0.15
88.20 0.32
88.52 + 0.28
89.63 + 0.05
1000
3.95
0.8
Mechanism Activation
3
0.6
Layer
layar
3.80
0.75
0.2
0.2
3.70
3.65
0.0
they
was
the
0.0
0.2
0.4
0.6
0.8
Layer
Layer
Figure 4: a BERT model, we show the minimum, average, and maximum mechanism competition values of some selected common tokens (left). One mechanism clearly specializes over the period between sentences, yet the high difference between the minimum and maximum values suggest that these differences are contextual and static function of the token. In particular whether is used to mark the end of sentence or whether used as part number a URL. Moreover, the mechanism activation is highly correlated between layers (scatter-plot in center, correlation matrix on right).
found that the modular activation for period depends
CONCLUSION
Scaling extremely large Transformers with very large number of hidden units for each position has become one of the dominant paradigms in applied machine learning. This work explores a new direction in the structure of the Transformer architecture which will become increasingly important as models become larger and researchers seek to model more complex phenomena. Evidence suggests that the Transformer's 's success is result of its use of attention to communicate information between positions, which allows for effective and precise transmission of information even over very long sequences (Kaplan et al. 2020). At the same time, each position within Transformer is still represented with single monolithic hidden representation, and set of parameters which is applied over the entire hidden representation. Our newly proposed technique, TIM, has shown that possible to make the Transformer even more dynamic by breaking the hidden representation and layers into multiple mechanisms which interact via attention and have an inductive bias towards specialization. We show that these mechanisms specialize results across diverse types of data. These results suggest that there is room to improve the structural inductive biases in the Transformer and point towards increasingly central area of future research as maateooftthe-ae Transformers, and the tasks they's 're trained on, become larger and more diverse.
distinct parts of the data and improve
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Hmergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):147-1180 2018.
Ferran Alet, Tomás Lozano-Pérezz and Leslie P Kaelbling. Modular meta-leaming. arXiv preprint arXiv: 10166, 2018.
Bang An, Jie Lyu, Zhenyi Wang, Chunyuan Li, Changwei Hu, Fei Tan, Ruiyi Zhang, Yifan Hu, and Changyou Chen. Repulsive attention: Rethinking multi-head attention as bayesian inference. arXiv preprint arXiv:2009.09364 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.0449, 2014.
Yoshua Bengio. Learning deep architectures for Al. Now Publishers Inc, 2009.
Steven Boll. Suppression of acoustic noise speech using spectral subtraction. IEEE Transactions acoustics, speech, and signal processing, 27(2): 13-120. 1979.
Hyeong-Seok Choi, Hoon Heo, Jie Hwan Lee, and Kyogu Lee. Phase-aware single-stage speech denoising and dereverberation with u-net. arXiv preprint arXiv:2006.00688 00687, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D) Manning. What does bert look at? analysis of bert's attention. arXiv preprint arXiv: /906 190..4441,0 2019.
Hongyi Cui, Shohei lida, Po-Hsuan Hung, Takehito Utsuro, and Masaaki Nagata. Mixed multi-head self-attention for neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 214,214, 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.0388 2018.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 04805, 2018.
Yariv Ephraim and David Malah. Speech Enhancement Using Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator. IEEE Transactions Audio, Speech, and Language Processing, 32(6): 109-112,, 1984.
Szu-Wei Fu, Chien-Feng Liao, Tsun-An Hsich, Kuo-Hsuan Hung, Syu-Siang Wang, Cheng Yu, Heng-Cheng Kuo, Ryandhimas E Zezario, You-Jin Li, Shang-Yt Yi Chuang, et al. Boosting ob- jective scores of speech enhancement model through metriegan post-processing. arXiv preprins arXiv:2006. /0296, 2020.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sentiment classification: deep learning approach. In ICML, 2011.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909. 2019.
Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. Deern: Deep complex convolution recurrent network for phase-aware speech enhancement. arXiv preprint ar0iv:2008. 000664, 2020.
Umut Isik, Ritwik Giri, Neerad Phansalkar, Jean-Mare Valin, Karim Helwani, and Arvindh Krish- naswamy. Poconet: Better speech enhancement with frequency-positionn embeddings, semi- supervised conversational data, and biased loss. arXiv preprint arXiv:2008.04477 2020.
Mads G. Christensen Jacob Benesty, Jesper R. Jensen and Jingdong Chen. Speech Enhancement-A Signal Subspace Perspective. Academic Press, 2015.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001 00036) 2020.

---

Under review as a conference paper at ICLR 2021
Andrej Karpathy, http://////ttbb..ooc////aaaa/a/a
2020.
Jaeyoung Kim, Mostafa El-Khamy, and Jungwon Lee. T-gsa: Transformer with gaussian-weighted self-attention for speech enhancement. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6649-6653. IEEE, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. /RLhtt:///aae attpa/aii.....aaaaa./..
6980
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. ACL, 2007.
Yuichiro Koyama, Tyler Vuong, Stefan Uhlich, and Bhiksha Raj. Exploring the best loss function for dnn-based low-latency speech enhancement with temporal convolutional networks. arXiv preprint arXiv:2005. 16611, 2020.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Cortes LeCun, Y. and C.J. Burges. The mnist database handwritten digits. arXiv preprint arXhv:2001.08361, 1998. UtLhttp: wwtp://wwwccc...ccccocceccccceee
Yinhan Liu, Myle Ont Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using ad adversarial training. In Advances in neural information processing systems, pp. 5040-5048, 2016.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules. arXiv preprint vrxiv:2006.1698/ 2020.
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas- CarullCa and Bernhard Schölkopf. Learning independent causal mechanisms. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 4033 4041, 2018.
Sungrae Park, Geewook Kim, Junyeop Lee, Junbum Cha, and Ji-Hoon Kim Hwalsuk Lee. Group- transformer: Towards lightweight character-level language model, 2020. URL https:// openreview net wwt./fettuurroroeo wwwoooooooiiiiiooooeee IddzxxxexxBPPE
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv:1802.05365. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Mirco Ravanelli and Maurizio Omologo. Contaminated speech training methods for robust DNN- HMM distant speech recognition. In Proc. of Interspeech, pp. 766-760. 2015.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005. 1 13981, 2020.
Salah Rifai, Yoshua Bengio, Aaron Courville, Pascal Vincent, and Mehdi Mirza. Disentangling factors variation for facial expression recognition. European Conference on Computer Vision, pp. 808-822 Springer, 2012.
10

---

Under review as a conference paper at ICLR 2021
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a -a new method for speech quality assessment telephone networks and codecs. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), volume 2, pp. 749-752, 2001.
Eric Ronco, Henrik Gollee, and Peter Gawthrop. Modular neural networks and self-decomposition. Technical Report CSC-96012, 1997.
Pascal Scalart and Jozué V. Filho. Speech enhancement based on priori signal to noise estimation. In Acoustics, Speech, and Signal Processing, 1996. ICASSP-90 Conference Proceedings., 1996 IEEE International Conference on, volume 2, pp. 629-632. IEEE, 1996.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2016.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi-channel acoustic noise database: A database of multichannel environmental noise recordings. The Journal of the Acoustical Society of America, 133(5):35113359) 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jesse Vig, Ali Madani, Lav Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. Bertology meets biology: Interpreting attention protein language models. arXiv preprint 27Xiv:2006.15222, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GL.UE: A multi-task benehmark and analysis platform for natural language understanding. CoRR, abs/1 804.07461, 2018. URL httt:://:/// http://awwwwv.oooaboooooo/ 804. 07461
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv: (00::/00000627 2015.

---

Under review as a conference paper at ICLR 2021
A EXPERIMENT DETAILS
A.1 IMAGE TRANSFORMER DETAILSS
For the Image Transformer, we used a bascline with 6 Transformer layers. As in the other experiments, we made the first 2 layers use TIMs as well as the last layer. We ran each experiment for 30 epochs with a batch size of 24, and otherwise used the the Adam optimizer with warmup, with Betas (0.9, 0.95). Our baseline model had heads (total per-layer) and a layer hidden size of 184, When using TIMs, we increased this (for all layers) to 10 heads and a hidden size of 200. This led to the baseline and the TIM model having roughly the same number of total parameters.
training hyperparameters the minGPT repository (Karpathy 2020). We used
A.2 SPEECH ENHANCEMENT DETAILS
Datasets Neural speech enhancement systems are trained using parallel corpus of noise and clean examples. which are generated by artificially contaminating clean speech with disturbances such as additive noise and reverberation (Ravanelli Omologo 2015). The speech enhancement models considered this work are trained with the DNS dataset (noisy, no reverb) (Reddy et al. (2020), which is a synthetic corpus recently made publicly available by Microsoft. This corpus is extremely suitable for our study because it is quite big (441 hours) and contains large variety of possible noises (from 600 different categories). To the best of our knowledge. it is the biggest open-source speech enhancement dataset. Moreover, has been the object of an international challenge on speech enhancemen This gave us the possibility to compare our TIM with the best systems submitted to this competition.
For evaluation, we used the test sets of the DNS and Voicehank datasets (Thiemann et al. (013). The latter has been adopted to study transfer learning scenario, where different datasets are used for training and evaluation purposes. Voicebank, in fact, generated with noisy sequences different from the one contained in the DNS corpus. Since Voicebank is released at 48 kHz, the original raw waveforms were downsampled from 48kHz to 16kHz.
Model Architecture The proposed TIM is fed with noisy speech and estimates clean speech at the output. More precisely, we estimate the log-spectral magnitude of the clean signal. Mean Squared Error (MSF) between the clean and the corresponding noisy signal is used cost function. The input waveform is transformed with the Short-Tern Fourier Transform (STFT) based on 512 frequency points and window length of 32 ms with 16 ms overlap.
Before adding the transformer layers, we employ four 1D convolutional layers that act as a pre-enonder module. This is done to replace positional encoding from the original transformers and inject relative location information to the frames in the sequence (Kim et al. 2020 Fu et al. 2020). The four convolutional layers are based on 1024, 512. 128, and 256 channels, respectively. The kernel size is 3. After the convolution, each layer applies layernorm followed by LeakyReLU. The Transformer part is composed of 8 encoder blocks with a hidden size of 512. In order to employ approximately the same number of parameters (i.c, 6 million), the baseline transformers used hidden size of 256. We used 16 attention hcads, dropout rate of 0. and LeakyReLU activations. We kept the number of heads the same as the bascline model. To follow the real-time processing restriction in DNS challenge, causal setting is adopted to all our models with access to 32 ms of future frames. Attention masks are also applied to the self-attention layers to prevent using the future information.
Training We followed the exact same training procedure for the baseline model and the TIMs model, with both trained for 50 epochs. We used the standard variant of the Adam optimizer with a batch size of 16. The initial learning rate was set to 0.0002 and halved when the validation score decreased for epochs. We reported test set performance at the epoch with the best validation score, which in practice was near the end of training. Both models train for about 50 hours on a single Nvidia V100 GPU.
A.3 BERT
AND TUN-IG DETAIL.S
BERT (Devlin et al. 2018) is one of the most popularly used methods to learn the representation of natural language. The BERT model uses multi-layer Transformer encoder and is trained by the masked language modeling task using Web data corpus (Liu al. 2019). The pre-trained contextual sentence representations have been shown to be effective in large number of downstream tasks.
To validate our proposed architecture, we conduct experiments to compare TIM with Transformer on the language pre-training task. For our model, we replace all of the transformer layers except for the first two layers and the last layer with TIM layers (we also report result where all layers are TIM layers, showing that it leads to worse performance). We scaled the dimensionality of the hidden nodes and the inner-layer of the FFN sub-layer
https: chal eengeeg......... www.eeewwwwwwwwsae. tee. net/ htwp://eeeeeeeetttteeeeeheoeaa
12

---

Under review as a conference paper at ICLR 2021
to, the number of mechanisms set 2 and the number of heads is set 16. We mainly test two TIM variants, TIM without competition (TIM-NoComp) and TIM with competition (TIM-Comp).
For a fair comparison, we set one baseline as a 12-layer Transformer with 130M parameters (BERT-130M The size of hidden nodes and the inner-layer of the FFN sub-layer are set to 768/4096, and the number of heads is set to 12. We also use the standard BERT-Base model (1 (110M parameters) as another baseline.
Dataset Following Devlin al. (2018), use English Wikipedia corpus and BookCorpus for pre-training By concatenating these two datascts, wc obtain corpus with roughly 3400M words in total, We follow couple consecutive pre-processing steps: segmenting documents into sentences by Spacy normalizing, lower-casing, and tokenizing the texts by Moses decoder (Koehn et al. 2007). and finally, applying byte pair 32,678.
encoding (BPE) SSnnnrich el al. 2016) with setting the vocabulary size
Optimization Following the standard settings used in many previous works Devlin al. 20118) Liu et al. (211)) we train the models for 1000k steps with setting the batch size 256 and the maximum sequence length as 512. For the models to compare, we set the masked probability p to 0.15. We follow previous works to replace 80% of the masked positions by [MASK], 10% by randomly sampled words, and keep the remaining positions unchanged. We choose the most widely used Adam Kingma & Ba (2014) as the optimizer. and set the syper-parameter 3 (0.9, 0.98). The learning rate is set 1c-4 with 10k-step step warm-up stage and then decays linearly to zero, We the dropout probability as 0.1. models are rum on 8 NVIDIA Tesla V100 GPUs
Fine-tuning We use the GLUE (General Language Understanding Evaluation) dataset WWang et al. 2018) as the downstream tasks to evaluate the performance of the pre-trained models. Reporting large-scale task choice. Same the pre-training, we (0.9, 0.98). Following all previous works, apply
performance averaged performance over all tasks depends Adam the optimizer and set the hyper parameter the hyper-paramett search (over /3 and learning rate) during fine-tuning for each downstream task. Each configuration was run for five times with different random seeds, and the median and standard deviation over these five results on the development set was he used as the performance of one configuration.
Results The overall comparison results shown in Table 2 It is easy find that both TIM- IIM-NoComp and TIM-Comp Comp achieve lower perplexities (masked language modeling loss) on validation dataset compared to the two BERT baselines. On the downstream tasks, the two TIM variants are also slightly better than the BERTs on all tasks. Those empirical results show that our proposed TIM is a better model architecture in a wide range of natural language applications.
Similar to previous analysis, further study the competition patterns in the TIM-Comp model to investigate how the competitive module behaves.
phttps://dunt httpp:://mmms..iiiindee orwp .wt.iaarrrg/e//eee
"A the dataset BookCorpus (hhu et al. 2015) is no longer freely distributed, we follow suggestions from
Devlin et al. (2018) to crawl from smashwords com and collect BookCorpus by ourselves.
htttss://spayy:
13

---

Under review as a conference paper at ICLR 2021
B DETAILED ALGORITHM DESCRIPTION
Algorithm A single TIM Encoder-Layer
Hyperparameters: Number of mechanisms rts, key size de, value size du. number of heads for self-attention H, number of heads for inter-mechanism attention H.. We set dmech dmodel n, and fn d
Input: An input hidden representation for a single example of shape (T, bs, dmodet).
Step 1: Compute Mechanism Competition
W C E X x1
softmax (CroupLinear(h, W mna))
Step 2: Mechanism-wise self-attention sub-layer
W WT C. Xdrachh x.dd
W E RR x Hdy
W R Hdy dhane
Q GroupLinear (h, W n.)
K GroupLinear(4, (h, W ns)
= GroupLinear h, W nz)
M = Attention Q K, V,n,H
M GroupLinear M, W ms))
= norm(h c M, ns)
Step 3: Inter-mechunism Attention Sub-Layer
W W RR x druch x Hldd
W. R. H.dy
W3 Reex Xdhmock
Q = GroupLinear(( /,, W (2))
K GroupLinear (h, W. ms))
V = GroupLinear(H h, W3 ne)]
Reshape Q.K, and V to (n.: T # bs, He d).
M Attention Q, V. He)
Reshape (T, bs, n, He * dg).
M = GroupLinear (M, W nss) norm(h M,n.)
Step 4: Mechanism-wise, Position- Wise, PFN Sub-Layer
(1) R. xdy. -m
Wr(2) E Rdg..dd--- Xd
F= (omupp/imar((GGGoopiieeeee W())), W())
norm(h + F,F F.n.)
14


Reviews:
[{"review_id": "XLt205dbsES", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors tackle the problem of making vanilla Transformer better by handling different positions (in data) via independent and competing mechanisms. Formulation is interesting but experimental setup has various deficiencies:\n1. It is quite heuristical to propose to use first two and last layers of Transformer to be vanilla. This suggests that either (1) proposed solution has something deeper going on (uninvestigated), or (2) it is an experimental thing that works for datasets used, or (3) \"competition\" should not happen in initial and later layers of network.\n2. To provide proof-of-concept for independent mechanisms, more complex problems should be picked like image segmentation and speech source separation (more than 2 sources), and they should be investigated with more than 2 mechanisms.\n3. Results should also be provided for TIM without matching its number of parameters to vanilla Transformer. This is to see if TIM still gives good performance or not. Also, in my opinion, increasing number of heads of TIM to match parameters with vanilla is not fair. It usually gives performance improvement. IMO only number of hidden layer units may be increased for matching #parameters.\n4. DNS performance is good but best is achieved without competition mechanism, which shows that independence and competitiveness are not complementary?\n5. Improvement on VoiceBank is decent but baseline chosen in very far away from SOTA. Check: https://arxiv.org/abs/2006.12847", "extra_scores": {}, "extra_text": {}}, {"review_id": "R1vdH15I01f", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an independent mechanism that divides hidden representations and parameters into multiple independent mechanisms. The authors claim that the mechanism benefits the computation of sparse tensors; it does learn better inductive biases than a sizeable monolithic model. This idea is particularly similar to Recurrent Independent Mechanisms (RIM) [1], mentioned in the paper. The main contribution of this work is introducing competition between independent mechanisms. The authors evaluate their models on the image transformer model, speech enhancement, and NLP tasks.\n\nThe main thing that has been missing in this paper is the fine-details of the approaches. \n\nI think the paper has exciting ideas on reconstructing the architecture of the Transformer. However, the proposed models only give marginal improvements; thus it is tough to find the reasons for using this model. \n\nTo have a stronger claim, I suggest adding a comparison to RIM on Sequential MNIST Resolution Task to show that independent mechanisms can benefit the generalization performance.\n\nStrengths:\n- The authors introduce a novel transformer architecture to split the parameters into independent sections for a better inductive bias\n\nWeaknesses:\n- The performance improvement is not consistent in all experiments. The competition mechanism benefits only some tasks.\n- The evaluation results are not convincing\n\nSeveral issues and questions for the authors:\n1. Can you show the dynamics of the competition between mechanisms over time?\n2. In terms of evaluation, I didn't see any baseline for the Image Transformer model. It isn't easy to understand the significance of the proposed method.\n3. Please consider citing the relevant papers [2] [3]\n\n***Post-Rebuttal***\n\n> I want to thank the author for addressing my concerns. However, the authors did not address the issues of the evaluation. I think the paper can be further improved by providing more convincing results and analysis. For me, the significance of the proposed method is minimal. Thus, I will not change my score. \n\nReferences\n1. Recurrent Independent Mechanisms https://arxiv.org/pdf/1909.10893.pdf\n2. LeCun, Y., Cortes, C., & Burges, C. J. (2010). MNIST handwritten digit database.\n3. Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ip8881DKBou", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the potential limitation of Transformers on modelling data with clear non-uniform structure. Authors propose to integrate a useful structural inductive bias which helps to separate the information in hidden states of the Transformer (instead of using a single monolithic hidden representation). \n\nThe experiments on a toy image Transformer and speech enhancement demonstrate the effectiveness of the proposed approach. The experiments on BERT pre-training and fine-tuning do not bring much to the table, as the text data is relatively uniform and does not benefit much from the introduced structural inductive bias. As a result, the accuracy of the proposed model on GLUE benchmark is close to the accuracy of BERT baseline.\n\nOne obvious limitation of the proposed approach is the necessity to know the number of \"independent\" components in the data. It would be interesting to see whether it can be generalized to overcome this limitation, e.g., by choosing the number of independent mechanisms automatically.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ap7AKp8yKlP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary of the paper\nIt is motivated from a perspective that a model should be decomposed into independent modules/mechanisms, such that the model is robust to data distribution shift. A TIM layer is proposed to replace the Transformer encoder layer, where a mechanism competition is introduced to suppress all but one group. Experiments are conducted on different Transformer architectures with a replacement of encoder layers by TIM layers in a) autoregressive image generation with Image Transformer, b) speech enhancement with a customized Transformer and c) BERT on language modeling. \n\n# Pros\nThe idea that grouping the hidden activations by multiple mechanisms is a promising inductive bias towards out-of-distribution generalization. It is an interesting attempt to integrate this idea to Transformers. \n\n# Cons\n1. The presentation is terrible: the method description is basically throwing the code (Algorithm 1) to the audience without much explanation. It is hard to understand the TIM layer if neither diagrams nor equations are supplied. \n2. I didn't get what does \"mechanism\" mean in Algorithm 1. The competition will for sure suppress some activations but how can you guarantee that different groups will learn different behaviors or achieve disentanglement? \n3. Experiments only demonstrate n_s = 2 mechanisms. Is n_s > 2 even working? All results suggest that TIM can make some neurons focusing on the foreground, which isn't that surprising as this is most attention based models aiming at. Do you have results showing that TIM can learn diverse functionalities from data which can be interpreted as mechanisms?  \n4. Figure 1 is confusing: how can we know the difference between \"head-wise self-attention\" and \"Mechanism-wise Self-Attention and Inter-mechanism Attention\", which are all represented by a blackbox. What is the competition patterns? How are they computed? \n5. Algorithm 1: functions such as GroupLinear(), Attention() are used before defining. \n6. Figure 2 (middle): what do you mean \"TIM learn to specialize over the two sides of the image\"? Also, what are these figures showing? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the idea of introducing structural biases in the attention mechanism is interesting but the results and presentation right now is not convincing. Improvements are seen on only some datasets and the comparisons are not exact.
A reject.
Decision: Reject