Synthesizer: Rethinking Self-Attention for Transformer Models

The dot product self-attention is known to be central and indispensable to stateof-the-art Transformer models . But is it really required ? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models . Via extensive experiments , we find that ( 1 ) random alignment matrices surprisingly perform quite competitively and ( 2 ) learning attention weights from token-token ( query-key ) interactions is useful but not that important after all . To this end , we propose SYNTHESIZER , a model that learns synthetic attention weights without token-token interactions . In our experiments , we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks , including machine translation , language modeling , text generation and GLUE/SuperGLUE benchmarks . When composed with dot product attention , we find that Synthesizers consistently outperform Transformers . Moreover , we conduct additional comparisons of Synthesizers against Dynamic Convolutions , showing that simple Random Synthesizer is not only 60 % faster but also improves perplexity by a relative 3.5 % . Finally , we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks . 1 INTRODUCTION . Transformer models ( Vaswani et al. , 2017 ) have demonstrated success across a wide range of tasks . This has resulted in Transformers largely displacing once popular auto-regressive and recurrent models in recent years . At the heart of Transformer models lies the query-key-value dot product attention . The success of Transformer models is widely attributed to this self-attention mechanism since fully connected token graphs , which are able to model long-range dependencies , provide a robust inductive bias . But is the dot product self-attention really so important ? Do we need it ? Is it necessary to learn attention weights via pairwise dot products ? This paper seeks to develop a deeper understanding of the role that the dot product self-attention mechanism plays in Transformer models . The fundamental role of dot product self-attention is to learn self-alignment , i.e. , to determine the relative importance of a single token with respect to all other tokens in the sequence . To this end , there have been memory metaphors and analogies constructed to support this claim . Indeed , the terms query , keys , and values imply that self-attention emulates a content-based retrieval process which leverages pairwise interactions at its very core . Moving against convention , this paper postulates that we can not only do without dot product self-attention but also content-based memory-like self-attention altogether . Traditionally , attention weights are learned at the instance or sample level , where weights are produced by instance-level pairwise interactions . As a result , these instance-specific interactions often fluctuate freely across different instances as they lack a consistent global context . This paper proposes SYNTHESIZER , a new model that learns to synthesize the self-alignment matrix instead of manually computing pairwise dot products . We propose a diverse suite of synthesizing functions and extensively evaluate them . We characterize the source information that these synthesizing functions receive , i.e. , whether they receive information from individual tokens , token-token interactions , and/or global task information . Intuitively , different source inputs to the synthesizing functions should capture diverse views , which may be useful when employed in conjunction . Aside from generalizing the standard Transformer model , we show that it is possible to achieve competitive results with fully global attention weights that do not consider token-token interactions or any instance-level ( local ) information at all . More specifically , a random matrix SYNTHESIZER model achieves a 27.27 BLEU score on WMT 2014 English-German1 . Via a set of rigorous experiments , we observe that the popular and well-established dot-product content-based attention can be approximated with simpler variants such as random matrices or dense layers without sacrificing much performance in some cases . In our experiments , we also show that our relatively simple Synthesizer models also outperform Dynamic Convolutions ( Wu et al. , 2019 ) with a +3.5 % relative improvement in perplexity while being 60 % faster . On encoding tasks , our factorized Synthesizers can outperform other low-rank efficient Transformer models such as Linformers ( Wang et al. , 2020 ) . While simple Synthesizer models are able to perform competitively , our experiments show that the pairwise dot product is still ultimately helpful . When composing our synthesizing functions with dot products , we find that they consistently improve the performance of Transformers . In general , we believe our findings will spur further investigation and discussion about the true role and utility of the self-attention mechanism in Transformer models . Our Contributions Our key contributions are described as follows : • We propose Synthetic Attention , a new way of learning to attend without explicitly attending ( i.e. , without dot product attention or content-based attention ) . Instead , we generate the alignment matrix independent of token-token dependencies and explore a potpourri of parameterized functions for synthesizing attention matrices . • We propose SYNTHESIZER , a new model that leverages Synthetic Attention . The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks , including machine translation and language modeling . • Moreover , we show that ( 1 ) random learnable alignment matrices perform competitively and ( 2 ) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks . • On large-scale masked language modeling on the C4 dataset ( Raffel et al. , 2019 ) and finetuning on SuperGLUE and GLUE benchmarks , we show that simple random Synthesizers can outperform/match Lightweight Dynamic convolutions ( Wu et al. , 2019 ) along with outperforming Transformers and Universal Transformers ( Dehghani et al. , 2018 ) . On two encoding tasks , factorized random Synthesizers outperform low-rank Linformers ( Wang et al. , 2020 ) . 2 RELATED WORK . Attention-based models are used across a wide spectrum of problem domains . Such models are especially popular , due to their effectiveness , in the language and vision domains . Attention models can be traced back to the machine translation models of ( Bahdanau et al. , 2014 ) and ( Luong et al. , 2015 ) , where attention is employed to learn soft word alignments between language pairs . The intuition behind the attention mechanism is deeply-rooted in the notion of memory-based retrieval ( Graves et al. , 2014 ; Weston et al. , 2014 ) , in which soft differentiable addressing of memory was initially proposed . The paradigm of learning self-alignments , also known as self-attention , has been largely popularized by Transformer models ( Vaswani et al. , 2017 ) . This technical narrative has also been explored by a number of other recent studies , including those on intra-attention ( Parikh et al. , 2016 ) , selfmatching networks ( Wang et al. , 2017 ) , and LSTMN ( Cheng et al. , 2016 ) . To this end , Transformer models , which function primarily based on self-attention and feed-forward layers , generally serve as a reliable replacement for autoregressive recurrent models . 1The originally reported result is 27.30 . The self-attention layer itself has been the subject of many recent technical innovations . For example , recent studies have investigated improving the layer ’ s overall efficiency via sparsification and reducing the complexity of computing the alignment matrix ( Child et al. , 2019 ; Kitaev et al. , 2020 ; Huang et al. , 2018 ; Tay et al. , 2020 ; Beltagy et al. , 2020 ) . These methods are tightly coupled with the query-key-value paradigm , employing a form of memory-based content retrieval as an attention mechanism . On the other end of the spectrum , there have been studies that advocate for replacing self-attention with convolution ( Wu et al. , 2019 ) . The recent surge in interest in simplifying the attention mechanism raises important questions about the role and utility of the pairwise dot products , which are one the defining characteristics of self-attention models . Meanwhile , in the image domain , ( Cordonnier et al. , 2019 ) shows connection of Transformers with CNNs . Our work is a new take on the self-attention mechanism in Transformer models . We delve deeper , starting with replacing the pairwise dot products with what we call synthesizing functions that learn attention matrices that may or may not depend on the input tokens . The most closely related work is ( ( Raganato et al. , 2020 ) ) , in which the authors propose using fixed ( i.e. , not learned ) attention patterns in Transformer encoders . However , the scope of their work is limited to encoders and relies on manually defined handcrafted patterns that seem to work well . Our work takes this intuition further and expands on this narrative . 3 THE PROPOSED METHOD . This section introduces our proposed SYNTHESIZER model . At its core , our model is essentially a Transformer model with self-attention modules replaced with our Synthetic Attention modules . Figure 3.1 illustrates the key ideas behind ( a ) Transformer ( b ) Dense Synthesizers and ( c ) Random Synthesizers . 3.1 SYNTHESIZER MODEL . This section introduces Synthetic Attention , our proposed self-attention module . Our model removes the notion of query-key-values in the self-attention module and directly synthesizes the alignment matrix instead . Dense Synthesizer Let us consider the simplest variation of the SYNTHESIZER model which is conditioned on each input token . Overall , our method accepts an input X ∈ R ` ×d and produces an output of Y ∈ R ` ×d . Here , ` refers to the sequence length and d refers to the dimensionality of the model . We first adopt F ( . ) , a parameterized function , for projecting input Xi from d dimensions to ` dimensions . Bi = F ( Xi ) ( 1 ) where F ( . ) is a parameterized function that maps Rd to R ` and i is the i-th token of X and is applied position-wise ( to each vector in the sequence of length ` ) . Intuitively , this can be interpreted as learning a token-wise projection to the sequence length ` . Essentially , with this model , each token predicts weights for each token in the input sequence . In practice , we adopt a simple two layered feed-forward layer with ReLU activations for F ( . ) : F ( Xi ) =W2 ( σR ( W1 ( Xi ) + b1 ) ) + b2 ( 2 ) where σR is the ReLU activation function and W1 ∈ Rd×d and W2 ∈ Rd× ` . Hence , Bi is now of R ` . Given B ∈ R ` × ` , we now compute : Y = Softmax ( B ) G ( X ) ( 3 ) where G ( . ) is another parameterized function of X that is analogous to V ( value ) in the standard Transformer model . This approach eliminates the dot product attention Y = Softmax ( QK > ) V altogether by replacing QK > in standard Transformers with the synthesizing function F ( . ) . Random Synthesizer The previous variant learns synthetic attention by conditioning on each input of X and projecting to ` dimensions . Hence , the Dense Synthesizer conditions on each token independently , as opposed to pairwise token interactions in the vanilla Transformer model . We consider another variation of SYNTHESIZER where the attention weights are not conditioned on any input tokens . Instead , the attention weights are initialized to random values . These values can then either be trainable or kept fixed ( denoted as Fixed ) . Let R be a randomly initialized matrix . The Random Synthesizer is defined as : Y = Softmax ( R ) G ( X ) . ( 4 ) where R ∈ R ` × ` . Notably , each head adds ` 2 parameters to the network . The basic idea2 of the Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples . This is a direct generalization of the recently proposed fixed self-attention patterns Raganato et al . ( 2020 ) . Factorized Models The Dense Synthesizer adds d × ` parameters to the network . On the other hand , the Random Synthesizer adds ` × ` parameters . Here , note that we omit theQ , K projections in the standard Transformer which results in further parameter savings . Despite these savings , synthesized models can be cumbersome to learn when ` is large . Hence , we propose factorized variations of the SYNTHESIZER models and show that these variants perform comparably in practice . Factorized Dense Synthesizer Factorized outputs not only slightly reduce the parameter cost of the SYNTHESIZER but also aid in preventing overfitting . The factorized variant of the dense synthesizer can be expressed as follows : A , B = FA ( Xi ) , FB ( Xi ) ( 5 ) where FA ( . ) projects input Xi into a dimensions , FB ( . ) projects Xi to b dimensions , and a× b = ` . The output of the factorized module is now written as : Y = Softmax ( C ) G ( X ) . ( 6 ) where C = HA ( A ) ∗HB ( B ) where HA , HB are tiling functions and C ∈ R ` × ` . The tiling function simply duplicates the vector k times , i.e. , R ` → R ` ×k . In this case , HA ( · ) is a projection of Ra → Ra×b and HB ( · ) is a projection of Rb → Rb×a . To avoid having similar values within the same block , we compose the outputs of HA and HB . Factorized Random Synthesizer Similar to Factorized Synthesizers , we are also able to factorize R into low rank matrices R1 , R2 ∈ R ` ×k . Y = Softmax ( R1R > 2 ) G ( X ) . ( 7 ) 2We were not expecting this variation to work at all , but it turns out to be a strong baseline . Therefore , it is easy to see that , for each head , this reduces the parameter costs from ` 2 to 2 ( ` k ) where k < < ` and hence helps prevent overfitting . In practice , we use a small value of k = 8 . Mixture of Synthesizers Finally , we note that all of the proposed synthetic attention variants can be mixed in an additive fashion . This can be expressed as : Y = Softmax ( α1S1 ( X ) + · · ·αNSN ( X ) ) G ( X ) . ( 8 ) where S ( . ) is a parameterized synthesizing function and the α ( where ∑ α = 1 ) are learnable weights . In the case of mixing Random Factorized with standard Dense Synthesizers , this is expressed as : Y = Softmax ( α1R1R > 2 + α2F ( X ) ) G ( X ) . ( 9 ) We investigate several Mixture of Synthesizers variants in our experiments . On Parameters Depending on Sequence Length Random and dense Synthesizers both rely on parameters that depend on length ` . In general , we define a maximum length and dynamically truncate to the actual length of each batch . We note that this is in similar spirit to trainable positional encodings which have been common practice in Transformer models . Hence , we do not forsee any issue here . In the case that this is really a problem , one potential solution is to project to a smaller value b and tile b to the maximum sequence length . We leave this exploration to future work .

Summary: This paper challenges the common belief that self-attention with dot product is necessary to train good NLP models. Several variants of the Synthesizer model is proposed. The effectiveness of Synthesizer is surprisingly good, although not beating the dot-product attention. The authors further showed that mixing synthesizer and dot-product attention sometimes achieve better results. The idea is validated on Translation, NLU, Summarization, Dialogue, and Language Modeling.