Synthesizer: Rethinking Self-Attention for Transformer Models
Authors: Yi Tay
Dara Bahri
Donald Metzler
Da-Cheng Juan
Zhe Zhao
Che Zheng
Venue:  (2021)
Keywords: Transformers
Deep Learning
Attention

Abstract: The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose \textsc{Synthesizer}, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexity by a relative $3.5\%$. Finally, we show that simple factorized Synthesizers can outperform Linformers on encoding only tasks. 

Under review as a conference paper at ICLR 2021
SYNTHESIZER: RETHINKING SELF-ATTENTION FOR TRANSFORMER MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
The dot product self-attention is known to be central and indispensable to state- of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mech- anism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose SYNTHESIZER. model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly com- petitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product atten- tion, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons Synthesizers against Dynamic Convolu- tions, showing that simple Random Synthesizer is not only 60% faster but also improves perplexity by a relative 3.5%. Finally, we show that simple factorized Synthesizers outperform Linformers encoding only tasks.
INTRODUCTION
Transformer models (Vaswami et al. 2017) have demonstrated success across a wide range of tasks. This has resulted in Transformers largely displacing once popular auto-regressive and recurrent mod- els in recent years. At the heart of Transformer models lies the query-key-value dot product atten- tion. The success of Transformer models is widely attributed to this self-attention mechanism since fully connected token graphs, which are able to model long-range dependencies, provide robust inductive bias
But is the dot product self-attention really so important? Do we need it? Is it necessary to learn attention weights via pairwise dot products? This paper seeks to develop deeper understanding of the role that the dot product self-attention mechanism plays in Transformer models.
The fundamental role of dot product self-attention is learn self-alignment, i.c.. determine the relative importance of a single token with respect to all other tokens in the sequence. To this end, there have been memory metaphors and analogies constructed to support this claim. Indeed, the terms query, keys, and values imply that self-attention emulates content-based retrieval process which leverages pairwise interactions at its very core.
Moving against convention, this paper postulates that we cannot only do without dot product self-attention but also content-based memory-like self-attention altogether. Traditionally, attention weights are learned at the instance sample level, where weights produced by instance-level pairwise interactions. As a result, these instance-specific interactions often fluctuate freely across different instances as they lack consistent global context.
This paper proposes SYNTHESIZER, a new model that learns to synthesize the self-alignment matrix instead of manually computing pairwise dot products. We propose diverse suite of synthesizing functions and extensively evaluate them. We characterize the source information that these synthe- sizing functions receive, i.e., whether they receive information from individual tokens, token-token

---

Under review as a conference paper at ICLR 2021
interactions, and/or global task information. Intuitively, different source inputs to the synthesizing functions should capture diverse views, which may be useful when employed in conjunction.
Aside from generalizing the standard Transformer model, we show that it is possible to achieve competitive results with fully global attention weights that do not consider -ekentoken interactions any instance-level (local) information at all. More specifically, random matrix SYNTHESIZER model achieves a 27.27 BLEU score on WMT 2014 English-Gerre German" Via a set of rigorous exper- iments, we observe that the popular and well-established dot-product content-based attention can be approximated with simpler variants such as random matrices or dense layers without sacrificing much performance in some cases.
In our experiments, we also show that our relatively simple Synthesizer models also outperform Dynamic Convolutions (Wu al. 2019) with a +3.5% relative improvement in perplexity while being 60% faster. On encoding tasks, our factorized Synthesizers can outperform other low-rank efficient Transformer models such as Linformers (Wang al. 2020))
While simple Synthesizer models are able to perform competitively, our experiments show that the pairwise dot product still ultimately helpful. When composing our synthesizing functions with dot products, we find that they consistently improve the performance of Transformers. In general, we believe our findings will spur further investigation and discussion about the true role and utility of the self-attention mechanism in Transformer models.
Our Contributions Our key contributions are described as follows:
We propose Synthetic Attention, new way of learning to attend without explicitly attend- ing (i.e., without dot product attention or content-based attention). Instead, generate the alignment matrix independent of token-token dependencies and explore a potpourri of parameterized functions for synthesizing attention matrices.
We propose SYNTHESIZER, new model that leverages Synthetic Attention. The model performs competitive to state-of-the-art Transformer models on a wide range of language tasks, including machine translation and language modeling.
Moreover, we show that (1) random learnable alignment matrices perform competitively and (2) token-token dependencies are not necessary to achieve good performance with Transformer models on certain tasks.
On large-scale masked language modeling on the C4 dataset (Raffel et al.) 2019) and fine- tuning on SuperGLUUF and GLUE benchmarks, we show that simple random Synthesizers can outperform/match Lightweight Dynamic convolutions (Wu et al.] 2019) along with outperforming Transformers and Universal Transformers (Dehghani et al. 2018). On two encoding tasks, factorized random Synthesizers outperform low-rank Linformers (Wang ct al. 2020).
RELATED WORK
Attention based models are used across wide spectrum of problem domains. Such models are especially popular, due to their effectiveness, in the language and vision domains. Attention models can be traced back to the machine translation models of (Bahdanau et al., 2014) and (Luong et al.) 2015), where attention is employed to learn soft word alignments between language pairs. The intuition behind the attention mechanism is deeply-rooted in the notion of memory-based retrieval (Graves et al., 2014 Weston et al. 2014), in which soft differenniablee addressing of memory was initially proposed.
The paradigm of learning self-alignments, also known self-attention, has been largely popularized by Transformer models (Vaswani et al.] 2017). This technical narrative has also been explored by number of other recent studies, including those aatra-attention (Parikh et al. 2016). self- matching networks (Wang et al. 2017). and LSTMN (Cheng et al. 2016). To this end, Transformer models, which function primarily based on self-attention and feed-forwart layers, generally serve as reliable replacement for autoregressive recurrent models.
The originally reported result 27.30.
2

---

Under review as a conference paper at ICLR 2021
The self-attention layer itself has been the subject of many recent technical innovations. For exam- ple, recent studies have investigated improving the layer's overall efficiency via sparsification and reducing the complexity of computing the alignment matrix (Child et al. 2019 Kitaev et al. 2020 Huang et al. 2018 Tay et al., 2020 |Blltay et al., 2020). These methods are tightly coupled with the query-key-valien paradigm, employing a form of memory- mechanism. On the other end of the spectrum, there have been studies that advocate for replacing self-attention with convolution (Wu al. 2019). The recent surge in interest in simplifying the attention mechanism raises important questions about the role and utility of the pairwise dot prod- ucts, which are one the defining characteristics of self-attention models. Meanwhile, in the image domain, (Cordonnier et al., 2019) shows connection of Transformers with CNNs.
content retrieval as attention
Our work is a new take on the self-attention mechanism Transformer models. We delve deeper, starting with replacing the pairwise dot products with what we call synthesizing functions that learn attention matrices that may or may not depend on the input tokens. The most closely related work is (Raganato et al. 2020)), in which the authors propose using fixed (i.e., not learned) attention patterns in Translormer encoders. However, the scope of their work is limited to encoders and relies on manually defined handcrafted patterns that seem to work well. Our work takes this intuition further and expands on this narrative.
3 THE PROPOSED METHOD
This section introduces our proposed SYNTHESIZER model. At its core, our model essentially Transformer model with self-attention modules replaced with our Synthetic Attention modules. Figure illustrates the key ideas behind (a) Transformer (b) Dense Synthesizers and (c) Random Synthesizers.
3.1 SYNTHESIZER MODEL
This section introduces Synthetic Attention, our proposed self-attention module. Our model removes the notion of query-key-valuu in the self-attention module and directly synthesizes the alignment matrix instead.
Dense Synthesizer Let us consider the simplest variation of the SYNTHESIZER model which is conditioned on each input token. Overall, our method accepts input X E Rex and produces an output of Y E Reddd Here, refers to the sequence length and d refers to the dimensionality of the model. We first adopt F(),, parameterized function, for projecting input X, from d dimensions to dimensions.
B, B F(X.(X))
where F-)) is parameterized function that maps TRd to R and i is the i-th token of X and is applicd position-wise (to each vector in the sequence of length f). Intuitively, this can be interpreted as learning token-wise projection to the sequence length É. Essentially, with this model, each token predicts weights for each token the input sequence. In practice, we adopt a simple two layered forward layer with ReLU activations for F()):
F(X,) = W2 ( b2)) + by
where OR is the ReLU activation function and W1 E dRd and W2 E _dxe Hence, B, is now of R Given B E gxxé we now compute:
Y Softmax(B)G(X)
where G(.) is another parameterized function of X that is analogous to V (value) in the standard Transformer model. This approach eliminates the dot product attention Y = Softmax (QTT) altogether by replacing QK" in standard Transformers with the synthesizing function F F.).
Random Synthesizer The previous variant learns synthetic attention by conditioning on each in- put of X and projecting to é dimensions. Hence. the Dense Synthesizer conditions each token independently, as opposed to pairwise token interactions in the vanilla Transformer model. We con- sider another variation of SYNTHESIZER where the attention weights are not conditioned on any
3

---

Under review as a conference paper at ICLR 2021
(a) Transformer
(b) Synthesizer (Dense)
(c) Synthesizer (Random)
Capa
Oup
Dupe
Dense Synthesizer
Random Synthesizer
Ooery
Riy
Figure 1: Our proposed SYNTHESIZER model architecture.
input tokens. Instead, the attention weights are initialized to random values. These values can then either be trainable or kept fixed (denoted as Fixed).
Let R be randomly initialized matrix. The Random Synthesizer is defined as:
(4) where R E Rext Notably, each head adds e parameters to the network. The basic idea of the Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather learn a task-specific alignment that works well globally across many samples. This is a direct generalization of the recently proposed fixed self-attention patterns Raganato et al. 2020
Y Softmax
Factorized Models The Dense Synthesizer adds d x & parameters to the network. On the other hand, the Random Synthesizer adds ε parameters. Here, note that we omit the Q, K projections in the standard Transformer which results in further parameter savings. Despite these savings, synthe- sized models can be cumbersome to learn when é is large. Hence, we propose factorized variations of the SYNTHESIZER models and show that these variants perform comparably in practice.
Factorized Dense Synthesizer Factorized outputs not only slightly reduce the parameter cost of the SYNTHESIZER but also aid in preventing overfitting. The factorized variant of the dense synthe- sizer can be expressed as follows:
A, = FA(x), Fn(.))
(5)
where (-) projects input X; into dimensions, (.).) projects X; to b dimensions, and a x b = l. The output of the factorized module now written as:
Y Softmax(C)G(XX
(6)
where C= (A)+ (B) where HA, HB are tiling functions and C' E Rext The tiling function simply duplicates the vector is times, i.e., R + R this case, H-(-) is a projection of Raa Ra and (-) is projection of g& _xxaa To avoid having similar values within the same block, we compose the outputs of HA and H
Factorized Random Synthesizer Similar to Factorized Synthesizers, we are also able to factorize
R into low rank matrices R1, R2 E 1
Y = Softmax R1 RRR, (X().
We were expecting this variation to work all, hut it turns be strong baseline.
4

---

Dot Product Under review as a conference paper at ICLR 2021
Therefore, it is easy to that, for each head, this reduces the parameter costs from e to 2((k) where & << and hence helps prevent overfitting. In practice, we use a small value of k 8.
Mixture of Synthesizers Finally, note that all of the proposed synthetic attention variants be mixed in an additive fashion. This can be expressed as:
ON Softmax(on SN (X))G(())
where S(.) is a parameterized synthesizing function and the a (where La= = 1) are learnable weights. In the case of mixing Random Factorized with standard Dense Synthesizers, this is ex- pressed as:
Ro + (x)(()))(X).
We investigate several Mixture of Synthesizers variants in our experiments.
On Parameters Depending on Sequence Length Random and dense Synthesizers both rely on parameters that depend on length f. In general, define maximum length and dynamically truncate to the actual length of each batch. We note that this in similar spirit to trainable positional encodings which have been common practice in Transformer models. Hence, do not forsee any issue here. In the case that this is really a problem, one potential solution is to project to a smaller value b and tile to the maximum sequence length. We leave this exploration future work.
3.2 DISCUSSION
This paper asks fundamental questions about the attention matrix A and whether it is possible to synthesize by alternate means other than pairwise attention. worth noting that the regular dot product attention also be subsumed by our SYNTHESIZER framework, i.c., SYNTHESIZER generalizes the Transformer model. the case of the Transformer, the synthesizing function in question is S(X) = (x(X) Fx (X)).
Model
S(X)
Condition On
Sample
Interact
0
F
X₃ Vj
Local
Yes
2d
Random
R
N/A
Global
No
Fac. Random
RIR (xx.) (X.FA(X.)) (x(F)(X.))
N/A
Global
No
26k
Dense
X,
Local
No
d² + de
Fac. Dense
X,
Local
No
d d(ki k2)
Table 1: Overview of all Synthesizing Functions.
Table T lists the different model variants explored within SYNTHESIZER framework. The 'con- dition on' column refers to whether the synthesized output is produced as a function of X; or every X,, X; pair. The "sample" column indicates whether given variant leverages local global con- text. Random Synthesizers are global because they share the same global alignment patterns across all samples. Dense Synthesizers are considered makes the alignment pattern dependent on each individual sample. To this end, it is imperative for synthesized models have multiple heads be effective.
be local as they are conditioned on X,, which
EXPERIMENTS
This section outlines our experimental setup and results. We first conduct experiments on five tasks to evaluate the effectiveness of different Synthesizer variants along with how they compare to the vanilla Transformer. Specifically, we conduct experiments on (1) machine translation (EnDe, EnFr) (2) autoregressive language modeling (LM1B) (3) text generation (summarization and di- alogue modeling and (4) multi-task natural language processing (GLUE/SuperGILUE) Details of each experiments can be found in the appendix.
NNte that we primarily interested in making controlled comparisons instead of going for the state-of- thc-art result each task.
5

---

Under review as a conference paper at ICLR 2021
Notation of Variants We R to denote Random, D denote Dense and V to denote vanilla dot product attention. Fix to represent Fixed Random, FR to represent Factorized Random and FD to represent Factorized random. For Mixture Synthesizers, use + to denote that two methods are mixed.
4.1 COMPARING SYNTHESIZER VARIANTS AND TRANSFORMER MODELS
This section dives into detailed study of multiple Synthesizer variants and the base Transformer model.
Experimental Results on MT/LM First, observe that our Random Synthesizer baseline achieves 27.27 on EnDe and 41.12 on EnFr. The non-trainable (i.e., fixed) variant performs sub- stantially worse, but still yields surprisingly strong 2 24 BLEU with fixed random attention weights. Most other SYNTHESIZER variants achieve com- petitive performance, although with slight per- formance degradation compared to Transformers. An interesting finding is that the Mixture model Random + Dense synthesizer performs com- parably to vanilla Transformers on EnDe. When mixing the standard dot product attention, per- formance further increases by +0.88 BLEU points (EnDe). In general, the performance of SYNTHE- Table 2: Experimental Results on WMT 14 English- SIZER variants are competitive with Transformers Genman, WMT' English-Frencd French Machine Trans- for this task. On LMIb, We find that the Random lation tasks and Language Modeling One Billion Synthesizers perform within 1-2 PPL points away (LMIB). i denotes original reported results in from the vanilla Transformer model. The best per- Vaswani et al. 2017).
NMT (BLEU)
LM (PPI.)
Model
[6]
EnDe
EnFr
0
LM
Trans.
67M
27.30
38.10
Trans.
67M
27.67
41.57
70M
38.21
Synthesizer Models
Fix
61M
23.89
38.31
53M
50.52
67M
27.27
41.12
58M
40.60
FR
61M
27.30
41.12
53M
42.40
D
62M
27.43
41.39
53M
40.88
FD
61M
27.32
41.57
53M
41.20
R+D
67M
27.68
41.21
58M
42.35
D+V
74M
27.57
41.38
70M
37.27
R+V
73M
28.47
41.85
70M
40.05
forming model is the Synthesizer (D+V), which achieves the best performance on this setting.
Results on Text Generation For summarization. we find that the (R) and (D) variants do not outperform Transformers. The performance of the (D) model is & 2 Rouge-L points below Transformers. Hence, we postulate that the local sample-wise pairwise interac- tions are important for the summarization task. On the other hand, the utility of synthesized attention can also be observed, i.e., the (R+V) and (R+D) models both outperform Transformers. On the dialogue task, Syn- R+V
Sum.
Dialogue
Model Trans.
RL
B4
RL
Met.
CIDr
35.77
3.20
13.38
5.89
18.94
Synthesizer Models
R
33.10
2.25
15.00
6.42
19.57
33.70
4.02
15.22
6.61
20.54
D+V
36.02
3.57
14.22
6.32
18.87
35.95
2.28
14.79
6.39
19.09
thesizers (R) and (D) both outperform vanilla Trans- formers by a reasonable margin (xe 1-3) points across Table 3: Experimental results on Abstrac-
most/all metrics. The best performing model here is tive Summarization (CNN/Dailymaill and
the (D) variant. Surprisingly, unlike most other tasks, Dialogue Generation (PersonaChat). We re- the (+V) variants do not perform well, signifying that port on RL (Rouge-L), B4 (Bleu-4), Met. dot product self-attention may actually be harmful for (Meteor) and CIDr.
this task.
4.2 COMPARING SYNTHESIZERS WITH DYNAMIC CONVOLUTIONS
To ascertain the competitiveness of Synthesizers, we also compare them with Dynamic convolu- tions (Wu et al., 2019). We compare them (1) pretraining perplexity using the masked language modeling objective on C4 and (2) downtream finetuning results on GLUE and SuperGLUUE.
Results on Masked Language Modeling We also benchmark the speed of these models. order to do so, conduct additional experiments the T5 adaptation of masked language modeling on the C4 dataset (Raffel et al. 2019) by comparing against lightweight dynamic convolutions (Wu et al. 2019) masked language modeling task. We also take this chance benchmark the

---

Under review as a conference paper at ICLR 2021
Model
Glue
CoLA
SST
MRPC
STSB
QQP
MNLI
QNLI
RTE
T3 (Base)
83.5
53.1
92.2
92.0/88.7
89.1/88.9
88.2 88.2/91.2
84.7/85.0
91.7
76.9
TS (Base+)
82,8
54.3
92.9
88.0/83.8
85.2/85,4
88.3/91.2
84.2/84.3
91.4
79.1
DyConv
69.4
33.9
90.6
82.6/72.5
60.7/63.1
84.2/88.2
73.8/75.1
84.4
58.1
Syn (R)
75.1
41.2
91.2
85.9/79.4
74.0/74.3
85.5/89.0
77.6/78.1
87.6
59.2
Syn (D)
72.0
18.9
89.9
86.4/79.4
75.3/75.5
85.2/88.3
77.4/78.1
86.9
57.4
Syn (D+V)
82,6
48.6
92.4
91.2/87.7
88.9/89.0
88.6/91.5
84.3/84.8
91.7
75.1
Syn (R+V)
84.1
53.3
92.2
91.2/87.7
89.3/88.9
88.6/91.4
85.0/84.6
92.3
81.2
Table 5: Experimental results (dev scores) multi-task language understanding (GLUE bench- mark) for small model and en-mix mixture. Note: This task has been co-trained with SuperGLUE.
Model
SGlue
BoolQ
CB
CoPA
MultiRC
ReCoRD
RTE
WIC
WSC
T3 (Base)
70.3
78.2
72.1/83.9
59.0
73.1/32.1
71.1/70.3
77.3
65,8
80.8
T5 (Base+)
70.7
79.3
81.1/87.5
60.0
75.1/34.4
71.7/70.7
80.5
64.6
DyConv
57.8
66.7
65,9/73.2
58.0
57.9/8.71
58.4/57.4
69.0
58.6
73.1
Syn (R)
61.1
69.5
54.6/73.2
60.0
63.0/15.7
58.4/57.4
67.5
64.4
66.3
Syn (D)
58.5
69.5
51.7/71.4
51.0
66.0/15.8
54.1/53.0
67.5
65.2
58.7
Syn (D+V)
69.7
79.3
74.3/85.7
64.0
73.8/33.7
69.9/69.2
78.7
64.3
Syn (R+V)
72.2
79.3
82.7/91.1
64.0
74.3/34.9
70.8/69.9
82.7
64.6
75.0
Table 6: Experimental results (dev scores) on multi-task language understanding (SuperGLUE benchmark) for small model and en-mix mixture. Note: This task has been co-trained with GLUE.
speed of Synthesizers compared with Transformers. Experiments are conducted on Mesh Tensorflow (Shazeer et al. 2018) and ran on 2x2 TPU V3 Chips for approximately 524K steps.
Model
Log PPL.
Steps/Sec
Params
FLOPS
Transformer /Vaswani et al. 2017
1.865
3,90
223M
3.70 x 10
Dynamic Conv (Wu al. 2019)
2.040
2.65
257M
3.93 10
Lightweight Cony (Wu et 2019)
1.972
4.05
224M
3.50
10
Synthesizer (D)
1.965
3.61
224M
3.80
10
Synthesizer (R)
1.972
4.26
254M
3.36
10
Synthesizer (R+V)
1.849
3.79
292M
4.03
10
Synthesizer (D+V)
1.832
3.34
243M
4.20 10
Table 4: Validation perplexity scores C4 dataset (Raffel et al. 2019
All models approxi-
mately similar parameterization.
Results on MIM Table reports the validation set log perplexity on masked language modeling We observe that Synthesizers (R) can outperform Dynamic Convolutions by relative +3.5% while being +600 faster. Against Lightweight Dynamic Convolutions, we match the performance while being +5% faster. Given that this is the simple random Synthesizer baseline, we find this extremely interesting how it is able to outperform dynamic convolutions, a relatively complex model. The Random Synthesizer also has less FLOPS compared to both convolution models. On the other hand, the Mixture Synthesizer models that use the dot product attention improves the performance of the base Transformer model with relatively equal model speed. Finally, similar the earlier results, we see a consistent performance gain of Synthesizer (D+V) and Synthesizer (R+V) outperforming the base Transformer model.
Results on GLUE and SuperGI.UE Tables and 6 report results on the GLUE and SuperGLUU benchmarks. We note that the (R) and (D) variants of SYNTHESIZER do achieve reasonable per- formance. This can be largely attributed to the fact that the encoder self-attention in the T5 setting also functions as cross-sentence attention. For example, the entailment or reading compre- hension tasks, the premise and hypothesis are concatenated together and self-attention effectively acts as cross-sentence attention On datasets like SST, straightforward sentiment classification
Note that this follows the sequence transduction style TS.
*On related note, the perceived success of pairwise self-attention might also be attributed to the fact that these public benchmarks are bias towards pairwise matching tasks. In reality, this is computationally prohibitive for many practical real-world applications (Sco et al. 2018).

---

Under review as a conference paper at ICLR 2021
task, this cross sentence attention is not necessary and therefore Syn (R) and Syn (D) both perform competitively. To this end, Dynamic Convolutions (Wu et al. 2019) also do not have this encoder "cross-attention" and therefore also suffer many of these pairwise matching tasks. Notably, in this "no cross attention' setting, the Random Synthesizers are are 4 to 5 percentage points higher in GLUE/SuperGLLUE score compared to Dynamic Convolutions.
Optimistically, we observe that the mixture model Syn (R+V) outperforms the T5 model by a sub- stantial margin (+1.9 points on SuperGLUE and +0.6 points on GLUE). Naturally, the hybrid mix- ture model also very substantially outperforms Dynamic Convolution. Finally to ensure that the Syn (+V) variations are not outperforming Transforeess due simply having more parameters, we also compared with T5 (Base+) which has equal number of parameters to Syn (+V) variants (ap- proximately R 10M more parameters). Our results show that Synthesizers (+V) still outperform T5 (Base+).
4.3 COMPARING SYNTHESIZERS WITH LINFORMERS
We conduct more experiments comparing factorized random Synthesizers with Linformers. Since Linformer cannot be used to decode, we compare them on two encoding tasks from tensor- flow datasets (AGnews (Zhang et al., 2013) and movie reviews (Maas al. 201T). We use k=32 for both factorized models. We also benchmark Transformers on this task. Note we do not use contextualized embeddings so results are not comparable with other work.
Model
News
Reviews
Steps/Sec
Results We notice that factorized Synthesiz- ers (FR) are competitive with Linformers and Transformers on this task. The accuracy of Syn (FR) competitive with Linformers while
Transformer
88.83
81.34
1.09
Linformer
86.50
82.86
1.09
Syn (FR)
86.53
83.39
1.10
Syn (FR+V)
89.13
84.61
0.80
Syn (FR+V) outperforms both Transformers
Table 7: Results on Encoding only tasks (accu- and Linformers.
racy).
4.4 OVERALL SUMMARY OF QUANTITATIVE RESULTS
This section summarizes overall findings.
Synthetic Attention is competitive even without Dot Product Attention On all evalu- ated tasks, we showed that synthesized attention functions competitively, i.c., achieves performance reasonably close the dot product self-attention. On one task (dialogue gen- eration), the dot product self-attention is found to actually degrade performance. Amongst the other tasks, machine translation is the least affected by the removal of the vanilla dot product. These findings allow us to introspect about whether pairwise comparisons for self-attention are even necessary. On the multi-task language understanding benchmark, the self-attention functions as a form of cross-attention by concatenating sentence pairs. Hence, synthesize attention performance is considerably worse than vanilla Transformers.
Synthetic Attention and Dot Product Attention are highly complementary Overall, we also observe that the dot product attention is very helpful. To this end, synthetic attention is highly complementary to the pairwise dot product attention. While Synthetic Attention can usually achieve competitive and fast performance its own, synthetic attention boosts per- forms, composing multiple synthetic attention (and dot product attention) together shows gains almost all tasks that have investigated. Hence, we believe this to be robust finding.
The simplest Synthesizers such as Random Synthesizers are fast competitive baselines Finally, we note that simple random Synthesizers are competitive with dynamic convolu- tions and Linformers, which recently proposed models. On two encoding task and large-scale masked language modeling task, we show that random (or factorized random) Synthesizers remain competitive to other fast efficient Transformer models.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
This paper proposed SYNTHESIZER, a new Transformer model that employs Synthetic Attention. We conducted a principled study to better understand and evaluate the utility of global alignment and local, instance-wise alignment (e.g., independent token and token-token based) in self-attention. We show that, on multiple tasks such as machine translation, language modeling, dialogue generation, masked language modeling and document classification, synthetic attention demonstrates compet- itive performance compared vanilla self-attention. Moreover, for the dialogue generation task, pairwise interactions actually hurt performance. Notably, we reemphasize that this study refers to self-attention. We found that we are not able to replace cross-attention with simpler variants in most cases. Via a set of additional large-scale experiments, also find that Synthesizers can outperform or match Dynamic Convolutions and Factorized Synthesizers can outperform other low rank Linformer models.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint #rXiv:1409.044. 2014.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-doeument transformer. arXiv:2004.05 150, 2020.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint
2016.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1904. 10509, 2019.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- 2019.
attention and convolutional layers. arXiv preprint /rXiv:
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 933-941. JMLR. org, 2017.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal 2018.
transformers. arXiv preprint
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint 1KK.:180990428.,, 2018
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. preprint arXiv:2001.0445: 2020.
Reformer: The efficient transformer. arXiv
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- 2015.
based neural machine translation. arXiv preprint
Andrew I.. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint arXiv: wet.:60069999 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.

---

Under review as a conference paper at ICLR 2021
Alessandro Raganato, Yves Schemrer and Jörg Tiedemann. Fixed encoder self-attentio patterns in transformer based machine translation. arXiv preprint arXiv:2002.10266) 2020.
Minjoon Seo, Tom Kwiatkowski, Ankur P Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Phrase- indexed question answering: A new challenge for scalable document comprehension. arXiv preprint arXiv:1804.077 2018.
Shikhar Sharma, Layla EI Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised met- rics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1 70s0/7906,,799, 2017. URL http::/ttaaatteee org abs / 1706. 09799
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances Neural Information Processing Systems, pp. 10414- 10423, 2018.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. arXiv preprint arXiv:2002.112000 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 20iv:2006.04768. 2020.
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching networks for reading comprehension and question answering. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 189-198, 2017.
Jason Weston, Sumit Chopra, and Antoine Bordes.
Memory networks.
arXiv preprint
arXiv:1410.3916, 2014.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint
2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
A APPENDIX
A. DETAILED SETUP FOR EXPERIMENTS
Machine Translation implement models Tensor2Tensor, using the standard base hyper- parameter settings. Specifically, we use byte-pair encoding (BPE), 6-layered Transformer networks with hidden size 512, filter size of 2048 and 8 heads. We use label smoothing of 0.1. The maximum sequence length is set to 256. Training is performed using 8 x V100 GPUs. We train all models for 250K steps and report results at the last checkpoint. We use a length penalty of 0.6 and beam size of 4 following the default settings. We also compare with standard Transformer models. In the interest of keeping a consistent, fair evaluation across all model settings, we do not use checkpoint averaging or tune the decoding hyperparameters although this generally leads to better performance. We evaluate BLEU scores using sacrebleu.
Language Modeling We implement our models in Tensor2Tensor using the packed TPU setup of sequence length 256. We train our models on 300K steps on 16 TPU V2 chips. We use the lmx_baas base model setting for fair comparison across all model variations. The model has 6 layers and 8 heads, along with filter width of 2048 and hidden size of 512. We used conv_re lu for the positional feed-forward layers across all baselines since we find them to perform slightly better. We report results (subword level perplexity scores) the test the final checkpoint.
10

---

Under review as a conference paper at ICLR 2021
Summarization For the summarization task, train all models for 300K steps and a batch size of 128. All models use the base size setting. For the dialogue generation task, due to the smaller dataset size, train small model for 20K steps. All results reported on the test set. For the summarization task, we use the well-established metrics, i.e., Rouge-1, Rouge-2 and Rouge-I. Experiments conducted using Mesh Tensorflow.
Dialogue Generation For the dialogue generation task, we train our models on the small size for 20K steps. Experiments are conducted in Tensor2Tensor. We use NLG-Eval (Sharma al,, 2017) and report BLEU-1, BLEU-4, Rouge-L, Meteor, CIDr and Embedding based similarity scores (Hmb).
Multi-Task Language Understanding Our experiments are based the T5 repository imple- mented Mesh Tensorflow (Shazeer et al. 2018). We pre-train the vanilla T5 models our models for 524288 steps using the span denoising objective. We then co-train the model multiple tasks. We co-train on the en mix mixture (SuperGLUE and GLUE) for 100k steps with a constant learning rate of 10-3 Embedding and Softmax output layer parameters mum sequence length is set to 512. We evaluate on the en mi x mixture as defined in the original codebase which is comprised of training GLUE, SuperGLUE and SQuAD in single model.
kept fixed. The maxi-
Pretraining experiments on C4 Experiments are conducted on Mesh Tensorflow. We pretrain for 524288 steps and report the perplexity on the validation We use 2x2 TPU V3 chips for our experiments. The sequence length is 512 and optimizer is Adafactor.
Experiments on Document Classification We run experiments in JAX/FLAX (https:// github. www.co////////// wwwggooooooooooooooooooooooooooo e flax) with base size models of 8 heads, 6 layers, MLP dimensions of 2048 and a hidden size of 512. We use the Adam optimizer with learning rate 0.05 and 8K steps linear warmup. We train for 10K steps and report evaluation results at 10K step. We use batch size of 128. We build a new sentencepiece model for each new dataset comprising of 32K tokens. No pretraining contextualized embeddings used. Experiments
16 TPU v3 chips.
A.2 ADDITIONAL VARIANTS OF SYNTHESIZER
We report results of several additional variants of SYNTHESIZER, most of which we found to have marginal or no improvement over the simple densc/random variations.
Convolution Applying ID convolution instead of layer nonlinear network. We vary the filter width in our experiments.
Bottleneck Converting the 2 layered feed forward network to a bottleneck layer, e.g., 512 + 16 - 512. We also experiment with a convolutional variant of bottleneck, i.c., projecting to low dimension space and then projecting back to high dimensions.
Gated Linear Units (GLU), applying the GLU units of (Dauphin al. 2017) as the Syn- thesizing function.

---

Under review as a conference paper at ICLR 2021
Variant
BLEU
Transformer
27.67
Random
27.27
Dense
27.43
Conv (f = 3) Linear
27.43
ConvReluConv (f =
27.51
ConvReluConv (f 5)
27.56
ConvReluConv (f = 3,
27.49
Bottleneck + Dense
27.43
Bottleneck + ConvReluConv
27.72
GLU
27.43
Table 8: Results for additional SYNTHESIZER variants WMT EnDe (BLEU scores)
A.3 EFFECT OF NUMBER HEADS
We also investigate the impact of the number of heads on performance. We trained three Random Synthesizer models for the small version the machine translation tasks using the T5 framework without pretraining, For simplicity, evaluation is done via greedy decoding. We report scores on the development set. We are mainly interested in relative performance and not absolute numbers. Table 9 reports the results varying the number of heads performance.
Heads
EnDe
EnFr
EnRo
Syn h=2
19.43
34.12
18.67
Syn h=4
20.42
35.26
19.78
Syn h=8
20.88
34.92
20.28
Syn h=16
21.71
35.26
20.43
Syn h=32
21.72
36.01
20.52
Table 9: Effect of number of heads on multi-task MT. Increasing the number of heads improves
performance.
A.4 ANALYSIS
Enc 1.1
Enc L.3
Enc L.5
Dec L.
Dec L.3
Dec L.5
Figure 2: Histogram of Encoder and Decoder Attention Weights on MT (WMT EnDe). L denotes layer number and Enc/Dec denotes encoder or decoder.
12

---

Under review as a conference paper at ICLR 2021
Distribution of Weights We are interested in investigating how the synthetically generated attention weights differ from the dot product attention weights. Figure 2 shows the attention histograms trained Transformer and SYNTHESIZER models. We report histograms at lay- 1, and of 6 layered (Transformer SYNTHESIZER) model at 50K steps. We found that the weight distributions remain relatively identical thereafter. Figure 3 shows the initialization state. We ob- serve that there are distinct differences in the weight distribution of SYNTHESIZER and Transformer models. The variance of the SYN- THESIZER weights tends to be higher. On the other hand, the weights on the Transformer model tends gravitate near and have smaller variance. There are also notable differences across the (R) and (D) SYNTHESIZER variants. Specifically, the (D) model general has greater max values with more values in the 0.1-0.2 range while the values of the R model tends to stay closer to
Figure 3: Init Decoder weights (Reference)
A.5 WHAT PATTERNS DO SYNTHESIZERS LEARN?
In this section, we perform deeper analysis of the SYNTHESIZER model.
Analysis Finally, we are interested to understand what these Synthesizer models are learning. We inspect the random syn- thetic attention weights for language modeling task LMIB and visualise the differences compared to the vanilla atten- tion. We find that, for the LM task, Synthesizers are capa- ble of learning a local window, emulating the vanilla Trans- former quite closely despite starting from completely random. The weights, however, seem smoother and less coarse com- pared to the Transformer. This seems to reflect what we expect since the Synthesizer does not benefit from token specific in- formation. We provide additional analysis and visualisation of weights for the Machine Translation task in the supplementary material.
Synthesizer
Transformer weights on LMIB.
weights on LMIB.
A.6 MORE ATTENTION WEIGHTS ANALYSIS
This section illustrates the attention weights extracted from different variants of Synthesizer on the machine translation (En-De) task. Weights are extracted from lower layers although we do not find any substantial difference in the patterns in early layers and deeper layers. We extract them from Tensorboard midway during training.
Vanilla
Random
Fac. Random
Dense
Fac. Dense
Figure 4: Visual analysis Synthetic Attention (decoder) on WMT EnDe.
13

---

Under review as a conference paper at ICLR 2021
Vanilla
Random
Fac. Random
Dense
Fac. Dense
Figure 5: Visual analysis of Synthetic Attention (encoder) WMT EnDe.
Analysis We first observe that these weights differ lot from the LM weights shown in the main paper in Section 4.5, This shows that the Synthesizer learns very different weights for different tasks. Next, based on the weighs on MT, we observe very different pattern in all variants of Synthesizer. For the decoder weights, the main difference seems to be the overall magnitude and distribution values of the weights. However, we can easily observe the cracks and lines of the factorized variants. For the encoder weights, we observe that the Random and Dense variants are more uniform. On the other hand, there appears to structural/regional clustering of values in the factorized variants.
A.7 CONVERGENCE OF SYNTHESIZERSS VS TRANSFORMERS
0.12
0.08
620k
6B0k
600k
640k
660k
720k
7B0k
Convergence of Synthesizers (Brown) Transformers (Blue)
Figure A.7 shows the convergence of Synthesizers Transformers on SuperGLUE finetuning. In general, Synthesizers and Transformers have pretty similar convergence patterns and this is also observed across other tasks.
14


Reviews:
[{"review_id": "xwitXcVkWoZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=================================\n\nSummary\n\nThis paper challenges the common belief that self-attention with dot product is necessary to train good NLP models. Several variants of the Synthesizer model is proposed. The effectiveness of Synthesizer is surprisingly good, although not beating the dot-product attention. The authors further showed that mixing synthesizer and dot-product attention sometimes achieve better results. The idea is validated on Translation, NLU, Summarization, Dialogue, and Language Modeling.\n\n=================================\n\nReview\n\nI enjoyed reading this paper and I imagine it would benefit many researchers in this community. I can't find any reason to reject this paper. However, it does not propose a new model that completely beats the transformer so I wouldn't give a higher score.\n\nPros\n-\tThe idea of Synthesizer is novel and could inspire the community to rethink self-attention transformers.\n-\tExperiment are thorough and cover a wide range of different tasks.\n-\tPaper is well-written.\n\nCons\n-\tThis model cannot fully replace dot-product attention, although it’s not a big problem in my opinion.\n\n=================================\n\nOther Questions / Suggestions\n\n- Why not finetune from the MLM Synthesizer for GLUE? Does Synthesizer also benefit from MLM pretraining?\n- What D/R/V stand for is not clearly stated in the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RbF80QRFVNK", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new Transformer variant that removes the pair-wise dot-product attention but use pointwise non-linearity instead. Here are my comments.\n\nRegarding Equation 2:\n\n1. In Eqn(2), if W2 is a d \\times l matrix, how can F(X) be an l \\times l matrix? Should there be a transformation? There are two terms b in Eqn 2. Are they the same, and what is the shape of b?\n\n2. It will be great if you can also present the dot-product attention equation together with your equation 2&3. That would be more helpful to the readers.\n\nRegarding Equation 4:\n\n1. The matrix R is randomly initialized. Is it fixed, or is optimized during training? If it is randomly initialized and fixed, Is the performance sensitive to the random seed? If it is randomly initialized and further optimized, could you show some learned patterns of this matrix R?\n\n2. If R is independent of tokens, what is the difference between R and positional encoding (in particular, the relative positional encoding?)\n\n3. Token-token attention can handle cases that exceed length l. For your random attention R, how do you handle length that more than l?\n\n\nRegarding Experimental results:\n\n1. The authors propose a variant called the random synthesizer. However, from the experimental results, we can see that the random synthesizer is worse than the baselines (significantly worse on GLUE benchmarks). I am not sure of the reason behind presenting this variant of networks. There is no gain but only a drop for this choice.\n\n2. I couldn't find a place that defines what the model R+V is and what is D+V.\n\n3. In GLUE, the WSC dataset is usually difficult. Could you explain why your model has a significant drop compared to the baseline model?\n\n4. In both GLUE and superGLUE data, small-task performance (COLA, RTE..) is quite unstable. For large tasks, such as MNLI and QNLI, the performance improvement is not much but just comparative.\n\n\nOverall comments:\n\nDot-product attention is one of the key components in the Transformer model, but the necessity of the dot-product attention is not very clear. The authors make a step to understand the attention mechanism. However, from both theoretical and empirical results, I cannot see any strong motivations behind the modification or any empirical benefits. It seems to me that the new model cannot be a good replacement for the original model. But I am open to further discussions. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "MkRRdIOi0YR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes replacing/combining Transformer self-attention with synthetic attention weights that do not rely on pairwise dependencies between token positions. Synthetic attention relies on either the input at the given position (dense synthesizer) or is altogether randomly initialized (random synthesizer).\n\nThe goal of the paper is to show that synthetic attention is a competitive alternative to self-attention. \n\nSome thoughts:\n1. The claim that combining synthetic attention with self-attention improves performance seems pretty unfair since the synthetic attention adds extra parameters that the baseline using only self-attention doesn’t have. This claim seems pervasive throughout the experiments but doesn’t appear to be well supported. Also, it would be interesting to see what the learned weights for the different types of attention look like - does it mostly just use self-attention?\n2. The fact that synthetic attention is competitive is interesting, but how does it compare to other methods for replacing self-attention (eg: Wu et al, 2019’s work on convolutions)? This question hasn’t really been properly addressed. They report a wide range of experiments in their paper, but the only setting compared here is T5 pre-training, and even there the metric is dev set loss for masked language modeling, which doesn’t objectively mean much. \n3. The random synthesizer results are interesting and surprising. It would be interesting to delve deeper into figuring out why the randomly initialized parameters serve as a reasonable proxy for token dependent weights. Are there any results from the Fixed random synthesizer in the paper? Don’t see any.\n4. Speed is recorded in terms of FLOPS for T5 pre-training, what about model convergence? Does it take longer for the model to converge? Doesn’t have to be on the T5 pre-training, MT or LM or the other tasks are fine too.\n5. Overall, the paper isn’t very clearly written. The motivations are not highlighted - why should one want to use synthetic attention? The notation, language and naming conventions are also a bit sloppy and inconsistent.\n\nIt might be worth considering an analytic framing for the paper where the goal is to study what makes the pairwise interactions replaceable and what going from all pairs to convolutions to dense and then finally random looks like in terms of model behavior and outputs. Also, is self-attention more replaceable in some layers than others? Eg see the ideas in sandwich transformers: https://arxiv.org/pdf/1911.03864.pdf.\n\n__UPDATE AFTER RESPONSE__\n\nHello authors, thanks for your response. After reviewing the updates, I'm still unconvinced. I will raise my score to a 4, but won't be recommending acceptance. I'll provide some suggestions below, but first I didn't note any strengths in my original review which was not right! So I'll start with a list of strengths.\n\nStrengths:  \n- This direction of trying to understand how much value dot product self-attention adds is very interesting. Synthesizing the attention matrix, rather than computing pairwise dependencies is a cool idea.  \n- The experiments are on a range of tasks including machine translation, language modeling, GLUE/SuperGLUE and more.  \n- The performance of the random synthesizer is quite surprising, the fact that it doesn't depend on input tokens but can still achieve non-trivial performance is intriguing.  \n\nSuggestions to improve:  \n- I still think the paper could do a better job of reporting a more complete set of experiments/comparisons. Comparing against the variants of synthetic attention is interesting but not enough given that there are quite a few papers that investigate similar ideas. Dynamic convolutions -- Wu et al. report a range of experiments on machine translation, language modeling etc. Why not compare to them on these tasks as well? Comparing only to self-attention just isn't enough since **synthetic attention is not the first attempt to replace it**.  \n- It seems a bit strange that dynamic convolutions are competitive with self-attention in the original paper, but results on GLUE are so much worse. It might be worth verifying on the sequence generation tasks that results are as expected. For GLUE, Linformer has results in the original paper, why not also compare against it here?  \n- The paper needs some revision to clarify the motivations -- it starts out by talking about how self-attention may not be necessary, but in some of the results synthetic attention has to be combined with dot product self-attention to achieve reasonable performance. **On GLUE, looks like the deterioration from using synthetic attention alone is as large as 10 points on average.** The fact that it improves performance to use self-attention and add some parameters strategically can still be interesting I guess, but the original motivation of the paper starts to fade.  \n- Small note: Everywhere, that the baseline is \"Transformer\" that's a self-attention-only variant (V), so maybe the notation/tables could clarify that point.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KpLm0eoGDG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n---------------------\nThis paper questions the necessity of the standard query-key attention in Transformer layers. It shows that replacing the standard attention mechanism with (1) random (which are learned in a task-specific way), (2) dense (which only depend on the contextualized representation $X_i$ instead of $X_i, X_j$ as usual) attention weights performs almost as well as standard methods. Combining this approach with standard attention via a mixture approach even improves upon the usual dot product attention. I found the results rather surprising, and believe that this would be an interesting and worthwhile contribution to the conference.\n\nStrengths\n---------------------\n- Interesting set of baselines/experiments (random/dense, factored, mixtures, etc.) with appropriate ablations across the various modeling choices.\n- Comprehensive evaluation across various settings: machine translation, language modeling, summarization, dialogue modeling,  and GLUE.\n- Very nice analysis of the learned model/attention in the Appendix. I found some of these fascinating so I encourage the authors to consider including them in the main paper.\n- The paper is very well written and was a pleasure to read.\n\nWeaknesses\n---------------------\n- While interesting, I am not sure what the practical computational benefit of this approach seems to be, since the query/key projection component of the Transformer does not take up much time.\n- While the point of the paper is not to achieve SoTA performance, there is still nontrivial degradation in performance for many tasks if just using the random/dense approach.\n\nQuestions\n---------------------\n- How much of the performance loss is due to longer sequences being less frequently encountered in the training set? (I.e. the $l$-th dimension of $F(X)$ gets worse for increasing $l$). Have you tried an alternative where you fix use the same $l$ for $l$ greater than (for example) 40? \n- Another way to achieve attention that only conditions on $X_i$ (as in the dense approach) would be to train with key/query as usual, and then use the average logits for token $X_i$ (averaged across the training set, for example). Have you considered this approach as a baseline?\n- The computational benefit of this approach is somewhat lessened by the fact that one still needs to perform the dense projection for each $X_i$ (for the dense approach at least). This could be avoided if $F(X_i)$ only depends on the non-contextualized representation of the $i$-th word, which means one could precompute $F(X_i)$ for all words in the vocab after training for faster inference. Given that the random baseline works, I imagine this baseline would work as well. Have you considered this approach?\n\n-----------------\nEDIT after rebuttal: Thank you for the response. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper seeks to answer the question on the necessity of the self-attention matrix in Transformers and whether it is possible to synthesize it by alternate means other than pairwise attention.  
The reviewers appreciated the main general idea and the wide range of experiments conducted. 
However, there are some concerns on clarity and evidence supporting main claims. While authors tried to address certain concerns through revision and response, the results suggests that self-attention matrix is still needed for strong performance and cannot be fully replaced by synthesizers. While authors also acknowledge this in discussions, If we do not consider the combined models (R+V or D+V), the empirical results do not look very convincing on the competitiveness of Synthesizers. They are only competitive on MT/Dialogue while Failing quite considerably on GLUE and Summarization. Overall, I felt very positive about the direction the paper pursues, but the empirical results doesn't seem to fully support the claims.   Quoting some points from reviewer discussions:

> `Comment: Moving towards an analytic framing would necessitate having the bare minimum set of experiments/comparisons before running additional analyses, but the bare minimum is still needed for this paper.

> Comment: I think the paper needs a round of revision and experiments need additional, carefully chosen baselines to adequately present synthetic attention in the context of existing solutions.

> Comment: this is not a reason that some random explored idea should be viewed as a great contribution, given that there are already several theory-grounded papers appeared in ICML (linear-attention..), NIPS (Linformer, follow-up work from linear-attention..), and ICLR (Random-feature attention, Performer..) this year. Compared to those theoretically well-motivated attention modification papers, this work is not that solid.
Decision: Reject