SlowGuess/ACL_2020_OCR huggingface dataset:
Weiqiu You*, Simeng Sun*, Mohit Iyyer
{wyou,simengsun,miyyer}@cs.umass.edu
Recent work has questioned the importance of the Transformer's multi-headed attention for achieving high translation quality. We push further in this direction by developing a "hard-coded" attention variant without any learned parameters. Surprisingly, replacing all learned self-attention heads in the encoder and decoder with fixed, input-agnostic Gaussian distributions minimally impacts BLEU scores across four different language pairs. However, additionally hard-coding cross attention (which connects the decoder to the encoder) significantly lowers BLEU, suggesting that it is more important than self-attention. Much of this BLEU drop can be recovered by adding just a single learned cross attention head to an otherwise hard-coded Transformer. Taken as a whole, our results offer insight into which components of the Transformer are actually important, which we hope will guide future work into the development of simpler and more efficient attention-based models.
The Transformer (Vaswani et al., 2017) has become the architecture of choice for neural machine translation. Instead of using recurrence to contextualize source and target token representations, Transformers rely on multi-headed attention mechanisms (MHA), which speed up training by enabling parallelization across timesteps. Recent work has called into question how much MHA contributes to translation quality: for example, a significant fraction of attention heads in a pretrained Transformer can be pruned without appreciable loss in BLEU (Voita et al., 2019; Michel et al., 2019), and self-attention can be replaced by less expensive modules such as convolutions (Yang et al., 2018; Wu et al., 2019).
In this paper, we take this direction to an extreme by developing a variant of MHA without
Figure 1: Three heads of learned self-attention (top) as well as our hard-coded attention (bottom) given the query word "to". In our variant, each attention head is a Gaussian distribution centered around a different token within a local window.
any learned parameters (Section 3). Concretely, we replace each attention head with a "hard-coded" version, which is simply a standard normal distribution centered around a particular position in the sequence (Figure 1). When we replace all encoder and decoder self-attention mechanisms with our hard-coded variant, we achieve almost identical BLEU scores to the baseline Transformer for four different language pairs (Section 4).
These experiments maintain fully learned MHA cross attention, which allows the decoder to condition its token representations on the encoder's outputs. We next attempt to additionally replace cross attention with a hard-coded version, which results in substantial drops of 5-10 BLEU. Motivated to find the minimal number of learned attention
<sup>1</sup>In Figure 1, the hard-coded head distribution centered on the word "to" (shown in green) is [0.054, 0.24, 0.40, 0.24, 0.054].
$^{2}$ Our code is available at https://github.com/fallcat/stupidNMT
* Authors contributed equally.
7689
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7689-7700 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Most learned attention heads for a Transformer trained on IWSLT16 En-De focus on a local window around the query position. The x-axis plots each head of each layer, while the y-axis refers to the distance between the query position and the argmax of the attention head distribution (averaged across the entire dataset).
parameters needed to make up this deficit, we explore configurations with only one learned cross attention head in total, which performs just slightly worse (1-3 BLEU) than the baseline.
By replacing MHA with hard-coded attention, we improve memory efficiency (26.4% more tokens per batch) and decoding speed (30.2% increase in sentences decoded per second) without significantly lowering BLEU, although these efficiency improvements are capped by other more computationally-expensive components of the model (Section 5). We also perform analysis experiments (Section 6.2) on linguistic properties (e.g., long-distance subject-verb agreement) that MHA is able to better model than hard-coded attention. Finally, we develop further variants of hard-coded attention in Section 6.3, including a version without any attention weights at all.
Our hard-coded Transformer configurations have intuitively severe limitations: attention in a particular layer is highly concentrated on a local window in which fixed weights determine a token's importance. Nevertheless, the strong performance of these limited models indicates that the flexibility enabled by fully-learned MHA is not as crucial as commonly believed: perhaps attention is not all you need. We hope our work will spur further development of simpler, more efficient models for neural machine translation.
In this section, we first briefly review the Transformer architecture of Vaswani et al. (2017) with a focus on its multi-headed attention. Then, we provide an analysis of the learned attention head distributions of a trained Transformer model, which motivates the ideas discussed afterwards.
2.1 Multi-headed Transformer attention
The Transformer is an encoder-decoder model formed by stacking layers of attention blocks. Each encoder block contains a self-attention layer followed by layer normalization, a residual connection, and a feed-forward layer. Decoder blocks are identical to those of the encoder except they also include a cross attention layer, which connects the encoder's representations to the decoder.
To compute a single head of self-attention given a sequence of token representations  $t_{1\dots n}$ , we first project these representations to queries  $q_{1\dots n}$ , keys  $k_{1\dots n}$ , and values  $v_{1\dots n}$  using three different linear projections. Then, to compute the self-attention distribution at a particular position  $i$  in the sequence, we take the scaled dot product between the query vector  $q_{i}$  and all of the key vectors (represented by matrix  $\mathbf{K}$ ). We then use this distribution to compute a weighted average of the values  $(\mathbf{V})$ :
$$
\operatorname {A t t n} \left(\boldsymbol {q} _ {i}, \mathbf {K}, \mathbf {V}\right) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {q} _ {i} \mathbf {K} ^ {\top}}{\sqrt {d _ {k}}}\right) \mathbf {V} \tag {1}
$$
where  $d_{k}$  is the dimensionality of the key vector.
For MHA, we use different projection matrices to obtain the query, key, and value representations for each head. The key difference between self-attention and cross attention is that the queries and keys come from different sources: specifically, the keys are computed by passing the encoder's final layer token representations through a linear projection. To summarize, MHA is used in three different components of the Transformer: encoder self-attention, decoder self-attention, and cross attention.
7690
2.2 Learned heads mostly focus on local windows
The intuition behind MHA is that each head can focus on a different type of information (e.g., syntactic or semantic patterns). While some heads have been shown to possess interpretable patterns (Voita et al., 2019; Correia et al., 2019), other work has cautioned against using attention patterns to explain a model's behavior (Jain and Wallace, 2019). In our analysis, we specifically examine the behavior of a head with respect to the current query token's position in the sequence. We train a baseline Transformer model (five layers, two heads per layer) on the IWSLT 2016 En→De dataset, and compute aggregated statistics on its learned heads.
Figure 2 shows that outside of a few layers, most of the model's heads focus their attention (i.e., the argmax of the attention distribution) on a local neighborhood around the current sequence position. For example, both self-attention heads in the first layer of the encoder tend to focus on just a one to two token window around the current position. The decoder self-attention and cross attention heads show higher variability, but most of their heads are still on average focused on local information. These results beg the question of whether replacing self-attention with "hard-coded" patterns that focus on local windows will significantly affect translation quality.
3 Hard-coded Gaussian attention
While learned attention enables model flexibility (e.g., a head can "look" far away from the current position if it needs to), it is unclear from the above analysis how crucial this flexibility is. To examine this question, we replace the attention distribution computation in Equation 1 (i.e., scaled dot product of queries and keys) with a fixed Gaussian distribution. In doing so, we remove all learned parameters from the attention computation: the mean of the Gaussian is determined by the position  $i$  of the current query token, and the standard deviation is always set to 1. As Transformers contain both self-attention and cross attention, the rest of this section details how we replace both of these components with simplified versions. We will re
fer to experimental results on the relatively small IWSLT16 English-German dataset throughout this section to contextualize the impact of the various design decisions we describe. Section 4 contains a more fleshed out experimental section with many more datasets and language pairs.
3.1 Hard-coded self-attention
In self-attention, the queries and keys are derived from the same token representations and as such have the same length  $n$ . The baseline Transformer (BASE) computes the self-attention distribution at position  $i$  by taking the dot product between the query representation  $\mathbf{q}_i$  and all of the key vectors  $\mathbf{k}_{1\dots n}$ . We instead use a fixed Gaussian distribution centered around position  $i - 1$  (token to the left),  $i$  (the query token), or  $i + 1$  (token to the right). More formally, we replace Equation 1 with
$$
\operatorname {A t t n} (i, \mathbf {V}) = \mathcal {N} (f (i), \sigma^ {2}) \mathbf {V}. \tag {2}
$$
The mean of the Gaussian  $f(i)$  and its standard deviation  $\sigma^2$  are both hyperparameters; for all of our experiments, we set  $\sigma$  to 1 and  $f(i)$  to either  $i - 1$ ,  $i$  or  $i + 1$ , depending on the head configuration. Note that this definition is completely agnostic to the input representation: the distributions remain the same regardless of what sentence is fed in or what layer we are computing the attention at. Additionally, our formulation removes the query and key projections from the attention computation; the Gaussians are used to compute a weighted average of the value vectors.
Instead of learning different query and key projection matrices to define different heads, we simply design head distributions with different means. Figure 1 shows an example of our hard-coded self-attention for a simple sentence. We iterate over different configurations of distribution means  $f(i)$  on the IWSLT16 En-De dataset, while keeping the cross attention learned. Our best validation result with hard-coded self-attention (HC-SA) replaces encoder self-attention with distributions centered around  $i - 1$  and  $i + 1$  and decoder self-attention with distributions centered around  $i - 1$  and  $i$ . This
5The Gaussian distribution is cut off on the borders of the sentence and is not renormalized to sum to one.
6 Preliminary models that additionally remove the value projections performed slightly worse when we hard-coded cross attention, so we omit them from the paper.
7See Appendix for a table describing the effects of varying  $f(i)$  on IWSLT16 En-De BLEU score. We find in general that hard-coded heads within each layer should focus on different tokens within the local window for optimal performance.
3 Yang et al. (2018) implement a similar idea, except the mean and standard deviation of their Gaussians are learned with separate neural modules.
4 Preliminary experiments with other standard deviation values did not yield significant differences, so we do not vary the standard deviation for any experiments in this paper.
7691
model achieves slightly higher BLEU than the baseline Transformer (30.3 vs 30.0 BLEU).
3.2 Alternatives to cross attention
We turn next to cross attention, which on its face seems more difficult to replace with hard-coded distributions. Unlike self-attention, the queries and keys in cross attention are not derived from the same token representations; rather, the queries come from the decoder while the keys come from the encoder. Since the number of queries can now be different from the number of keys, setting the distribution means by position is less trivial than it is for self-attention. Here, we describe two methods to simplify cross attention, starting with a fully hard-coded approach and moving onto a minimal learned configuration.
Hard-coded cross attention: We begin with a simple solution to the problem of queries and keys having variable lengths. Given a training dataset, we compute the length ratio  $\gamma$  by dividing the average source sentence length by the average target sentence length. Then, to define a hard-coded cross attention distribution for target position  $i$ , we center the Gaussian on positions  $\lfloor \gamma i - 1 \rfloor$ ,  $\lfloor \gamma i \rfloor$ , and  $\lfloor \gamma i + 1 \rfloor$  of the source sentence. When we implement this version of hard-coded cross attention and also hard-code the encoder and decoder self-attention as described previously (HC-ALL), our BLEU score on IWSLT16 En-De drops from 30.3 to 21.1. Clearly, cross attention is more important for maintaining translation quality than self-attention. Michel et al. (2019) notice a similar phenomenon when pruning heads from a pretrained Transformer: removing certain cross attention heads can substantially lower BLEU.
Learning a single cross attention head: Prior to the advent of the Transformer, many neural machine translation architectures relied on just a single cross attention "head" (Bahdanau et al., 2015). The Transformer has many heads at many layers, but how many of these are actually necessary? Here, we depart from the parameter-free approach by instead removing cross attention at all but the final layer of the decoder, where we include only a single learned head (SH-X). Note that this is the only learned head in the entire model, as both the encoder and decoder self-attention is hard-coded. On IWSLT16 En-De, our BLEU score improves from 21.1 to 28.1, less than 2 BLEU under the BASE Transformer.
Train	Test	Len SRC	Len TGT
IWSLT16 En-De	196,884	993	28.5	29.6
IWSLT17 En-Ja	223,108	1,452	22.9	16.0
WMT16 En-Ro	612,422	1,999	27.4	28.3
WMT14 En-De	4,500,966	3,003	28.5	29.6
WMT14 En-Fr	10,493,816	3,003	26.0	28.8
Table 1: Statistics of the datasets used. The last two columns show the average number of tokens for source and target sentences, respectively.
4 Large-scale Experiments
The previous section developed hard-coded configurations and presented results on the relatively small IWSLT16 En-De dataset. Here, we expand our experiments to include a variety of different datasets, language pairs, and model sizes. For all hard-coded head configurations, we use the optimal IWSLT16 En-De setting detailed in Section 3.1 and perform no additional tuning on the other datasets. This configuration nevertheless proves robust, as we observe similar trends with our hard-coded Transformers across all of datasets. $^{8}$
We experiment with four language pairs, English  $\leftrightarrow$  {German, Romanian, French, Japanese} to show the consistency of our proposed attention variants. For the En-De pair, we use both the small IWSLT  $2016^{9}$  and the larger WMT 2014 datasets. For all datasets except WMT14  $\mathrm{En}\rightarrow \mathrm{De}$  and WMT14  $\mathrm{En}\rightarrow \mathrm{Fr},^{10}$  we run experiments in both directions. For English-Japanese, we train and evaluate on IWSLT 2017  $\mathrm{En}\leftrightarrow \mathrm{Ja}$  TED talk dataset. More dataset statistics are shown in Table 1.
4.2 Architectures
Our BASE model is the original Transformer from Vaswani et al. (2017), reimplemented in PyTorch (Paszke et al., 2019) by Akoury et al. (2019). To implement hard-coded attention, we only modify the attention functions in this codebase and keep everything else the same. For the two small IWSLT datasets, we follow prior work
$^{8}$ Code and scripts to reproduce our experimental results to be released after blind review.
<sup>9</sup>We report BLEU on the IWSLT16 En-De dev set following previous work (Gu et al., 2018; Lee et al., 2018; Akoury et al., 2019). For other datasets, we report test BLEU.
10 As the full WMT14  $\mathrm{En}\rightarrow \mathrm{Fr}$  is too large for us to feasibly train on, we instead follow Akoury et al. (2019) and train on just the Europarl / Common Crawl subset, while evaluating using the full dev/test sets.
11 https://github.com/dojoteef/synst
7692
BASE	HC-SA	HC-ALL	SH-X
IWSLT16 En-De	30.0	30.3	21.1	28.2
IWSLT16 De-En	34.4	34.8	25.7	33.3
IWSLT17 En-Ja	20.9	20.7	10.6	18.5
IWSLT17 Ja-En	11.6	10.9	6.1	10.1
WMT16 En-Ro	33.0	32.9	25.5	30.4
WMT16 Ro-En	33.1	32.8	26.2	31.7
WMT14 En-De	26.8	26.3	21.7	23.5
WMT14 En-Fr	40.3	39.1	35.6	37.1
Table 2: Comparison of the discussed Transformer variants on six smaller datasets (top) $^{14}$  and two larger datasets (bottom). Hard-coded self-attention (HC-SA) achieves almost identical BLEU scores to BASE across all datasets, while a model with only one cross attention head (SH-X) performs slightly worse.
by using a small Transformer architecture with embedding size 288, hidden size 507, four heads, $^{12}$  five layers, and a learning rate 3e-4 with a linear scheduler. For the larger datasets, we use the standard Tranformer base model, with embedding size 512, hidden size 2048, eight heads, six layers, and a warmup scheduler with 4,000 warmup steps. For all experiments, we report BLEU scores using SacreBLEU (Post, 2018) to be able to compare with other work. $^{13}$
4.3 Summary of results
Broadly, the trends we observed on IWSLT16 En-De in the previous section are consistent for all of the datasets and language pairs. Our findings are summarized as follows:
- A Transformer with hard-coded self-attention in the encoder and decoder and learned cross attention (HC-SA) achieves almost equal BLEU scores to the BASE Transformer.
- Hard-coding both cross attention and self-attention (HC-ALL) considerably drops BLEU compared to BASE, suggesting cross attention is more important for translation quality.
A configuration with hard-coded self
attention and a single learned cross attention head in the final decoder layer (SH-X) consistently performs 1-3 BLEU worse than BASE.
These results motivate a number of interesting analysis experiments (e.g., what kinds of phenomena is MHA better at handling than hard-coded attention), which we describe in Section 6. The strong performance of our highly-simplified models also suggests that we may be able to obtain memory or decoding speed improvements, which we investigate in the next section.
5 Bigger Batches & Decoding Speedups
We have thus far motivated our work as an exploration of which components of the Transformer are necessary to obtain high translation quality. Our results demonstrate that encoder and decoder self-attention can be replaced with hard-coded attention distributions without loss in BLEU, and that MHA brings minor improvements over single-headed cross attention. In this section, we measure efficiency improvements in terms of batch size increases and decoding speedup.
Experimental setup: We run experiments on WMT16 En-Ro with the larger architecture to support our conclusions. $^{15}$  For each model variant discussed below, we present its memory efficiency as the maximum number of tokens per batch allowed during training on a single GeForce RTX 2080 Ti. Additionally, we provide inference speed as the number of sentences per second each model can decode on a 2080 Ti, reporting the average of five runs with a batch size of 256.
Hard-coding self-attention yields small efficiency gains: Table 7 summarizes our profiling experiments. Hard-coding self-attention and preserving learned cross attention allows us to fit  $17\%$  more tokens into a single batch, while also providing a  $6\%$  decoding speedup compared to BASE on the larger architecture used for WMT16 En-Ro. The improvements in both speed and memory usage are admittedly limited, which motivates us to measure the maximum efficiency gain if we only modify self-attention (i.e., preserving learned cross attention). We run a set of upper bound experiments where we entirely remove self-attention in the encoder and decoder. The resulting encoder
<sup>12</sup>For hard-coded configurations, we duplicate heads to fit this architecture (e.g., we have two heads per layer in the encoder with means of  $i + 1$  and  $i - 1$ ).
$^{13}$ SacreBLEU signature: BLEU+case.mixed+lang.LANG +numrefs.1+smooth.exp+test.TEST+tok.intl+version.1.2.11, with LANG  $\in$  {en-de, de-en, en-fr} and TEST  $\in$  {wmt14/full, iwslt2017/tst2013}. For WMT16 En-Ro and IWSLT17 En-Ja, we follow previous work for preprocessing (Sennrich et al., 2016), encoding the latter with a 32K sentencepiece vocabulary (https://github.com/google/sentencepiece) and measuring the de-tokenized BLEU with SacreBLEU.
15Experiments with the smaller IWSLT16 En-De model are described in the Appendix.
7693
Model	BLEU	sent/sec	tokens/batch
BASE	33.0	26.8	9.2K
HC-SA	32.9	28.4	10.8K
SH-X	30.3	34.9	11.7K
BASE/-SA	27.0	30.1	11.8K
SH-X/-SA	15.0	37.6	13.3K
Table 3: Decoding speedup (in terms of sentences per second) and memory improvements (max tokens per batch) on WMT16 En-Ro for a variety of models. The last two rows refer to BASE and SH-X configurations whose self-attention is completely removed.
thus just becomes a stack of feed-forward layers on top of the initial subword embeddings. Somewhat surprisingly, the resulting model still achieves a fairly decent BLEU of 27.0 compared to the BASE model's 33.0. As for the efficiency gains, we can fit  $27\%$  more tokens into a single batch, and decoding speed improves by  $12.3\%$  over BASE. This relatively low upper bound for HC-SA shows that simply hard-coding self-attention does not guarantee significant speedup. Previous work that simplifies attention (Wu et al., 2019; Michel et al., 2019) also report efficiency improvements of similar low magnitudes.
Single-headed cross attention speeds up decoding: Despite removing learned self-attention from both the encoder and decoder, we did not observe huge efficiency or speed gains. However, reducing the source attention to just a single head results in more significant improvements. By only keeping single-headed cross attention in the last layer, we are able to achieve  $30.2\%$  speed up and fit in  $26.4\%$  more tokens to the memory compared to BASE. Compared to HC-SA, SH-X obtains a  $22.9\%$  speedup and  $8.0\%$  bigger batch size.
From our profiling experiments, most of the speed and memory considerations of the Transformer are associated with the large feed-forward layers that we do not modify in any of our experiments, which caps the efficiency gains from modifying the attention implementation. While we did not show huge efficiency improvements on modern GPUs, it remains possible that (1) a more tailored implementation could leverage the model simplifications we have made, and (2) that these differences are larger on other hardware (e.g., CPUs). We leave these questions for future work.
Figure 3: BLEU performance on WMT16 En-Ro before and after removing all feed-forward layers from the models. BASE and HC-SA achieve almost identical BLEU scores, but HC-SA relies more on the feedforward layers than the vanilla Transformer. As shown on the plot, with a four layer encoder and decoder, the BLEU gap between BASE-FF and BASE is 1.8, while the gap between HC-SA and HC-SA-FF is 3.2.