In contrast, standard approaches use a variable-length set of encoder states for attention. At each decoding step, we similarly predict $K$ scores $\beta \in \mathbb {R}^K$ . The final attention context $c$ is a linear combination of the rows in $C$ weighted by the scores. Intuitively, each decoder step predicts how important each of the $K$ attention vectors is.  $$c & = \sum _{i=0}^{K}{\beta _i C_i} \\
\beta & = \text{softmax}(W_\beta h)$$   (Eq. 8)   Here, $h$ is the current state of the decoder, and $W_\beta $ is a learned parameter matrix. Note that we do not access the encoder states at each decoder step. We simply take a linear combination of the attention matrix $C$ pre-computed during encoding - a much cheaper operation that is independent of the length of the source sequence. The time complexity of this computation is $O(KD|T|)$ as multiplication with the $K$ attention matrices needs to happen at each decoding step.

Dropout of 0.2 (0.8 keep probability) is applied to the input of each cell and we optimize using Adam  at a learning rate of 0.0001 and batch size 128. We train for at most 200,000 steps (see Figure 3 for sample learning curves). BLEU scores are calculated on tokenized data using the multi-bleu.perl script in Moses. We decode using beam search with a beam size of 10  . Table 1 shows the BLEU scores of our model on different sequence lengths while varying $K$ . This is a study of the trade-off between computational time and representational power. A large $K$ allows us to compute complex source representations, while a $K$ of 1 limits the source representation to a single vector. We can see that performance consistently increases with $K$ up to a point that depends on the data length, with longer sequences requiring more complex representations. The results with and without position encodings are almost identical on the toy data. Our technique learns to fit the data as well as the standard attention mechanism despite having less representational power. Both beat the non-attention baseline by a significant margin.

Machine Translation	Next, we explore if the memory-based attention mechanism is able to fit complex real-world datasets. For this purpose we use 4 large machine translation datasets of WMT'17 on the following language pairs: English-Czech (en-cs, 52M examples), English-German (en-de, 5.9M examples), English-Finish (en-fi, 2.6M examples), and English-Turkish (en-tr, 207,373 examples). We used the newly available pre-processed datasets for the WMT'17 task. Note that our scores may not be directly comparable to other work that performs their own data pre-processing. We learn shared vocabularies of 16,000 subword units using the BPE algorithm  . We use newstest2015 as a validation set, and report BLEU on newstest2016. We use a similar setup to the Toy Copy task, but use 512 RNN and embedding units, train using 8 distributed workers with 1 GPU each, and train for at most 1M steps. We save checkpoints every 30 minutes during training, and choose the best based on the validation BLEU score.

Introduction	Sequence-to-sequence models  ,  have achieved state of the art results across a wide variety of tasks, including Neural Machine Translation (NMT)  ,  , text summarization  ,  , speech recognition  ,  , image captioning  , and conversational modeling  ,  . The most popular approaches are based on an encoder-decoder architecture consisting of two recurrent neural networks (RNNs) and an attention mechanism that aligns target to source tokens  ,  . The typical attention mechanism used in these architectures computes a new attention context at each decoding step based on the current state of the decoder. Intuitively, this corresponds to looking at the source sequence after the output of every single target token. Inspired by how humans process sentences, we believe it may be unnecessary to look back at the entire original source sequence at each step. We thus propose an alternative attention mechanism (section "Memory-Based Attention Model" ) that leads to smaller computational time complexity. Our method predicts $K$ attention context vectors while reading the source, and learns to use a weighted average of these vectors at each step of decoding. Thus, we avoid looking back at the source sequence once it has been encoded.

Summing $O(KD|S|)$ from encoding and $O(KD|T|)$ from decoding, we have a total linear computational complexity of $O(KD(|S| + |T|)$ . As $D$ is typically very large, 512 or 1024 units in most applications, we expect our model to be faster than the standard attention mechanism running in $O(D^2|S||T|)$ . For long sequences (as in summarization, where |S| is large), we also expect our model to be faster than the cheaper dot-based attention mechanism, which needs $O(D|S||T|)$ computation time and requires encoder and decoder states sizes to match. We also experimented with using a sigmoid function instead of the softmax to score the encoder and decoder attention scores, resulting in 4 possible combinations. We call this choice the scoring function. A softmax scoring function calculates normalized scores, while the sigmoid scoring function results in unnormalized scores that can be understood as gates.

Position Encodings (PE)	In the above formulation, the predictions of attention contexts are symmetric. That is, $C_i$ is not forced to be different from $C_{j\ne i}$ . While we would hope for the model to learn to generate distinct attention contexts, we now present an extension that pushes the model into this direction. We add position encodings to the score matrix that forces the first few context vector $C_1, C_2, ...$ to focus on the beginning of the sequence and the last few vectors $...,C_{K-1}, C_K$ to focus on the end (thereby encouraging in-between vectors to focus on the middle). Explicitly, we multiply the score vector $\alpha $ with position encodings $l_s\in \mathbb {R}^{K}$ :  $$C^{PE} & = \sum _{s=0}^{|S|}{\alpha ^{PE} h_s} \\
\alpha ^{PE}_s & = \text{softmax}(W_\alpha h_s \circ l_s)$$   (Eq.

We show (section "Experiments" ) that this speeds up inference while performing on-par with the standard mechanism on both toy and real-world WMT translation datasets. We also show that our mechanism leads to larger speedups as sequences get longer. Finally, by visualizing the attention scores (section "Visualizing Attention" ), we verify that the proposed technique learns meaningful alignments, and that different attention context vectors specialize on different parts of the source.

$$c_i & = \sum _{j}{\alpha _{ij} s_j} \\
{\alpha }_{i} & = \text{softmax}(f_{att}(h_i, \mathbf {s}))$$   (Eq. 3)  Here, $f_{att}(h_i, \mathbf {s})$ is an attention function that calculates an unnormalized alignment score between the encoder state $s_j$ and the decoder state $h_i$ . Variants of $f_{att}$ used in  and  are: $
f_{att}(h_i, s_j)=
{\left\lbrace \begin{array}{ll}
v_a^T \text{tanh}(W_a[h_i, s_j]),& \emph {Bahdanau} \\
h_i^TW_as_j & \emph {Luong}
\end{array}\right.}
$  where $W_a$ and $v_a$ are model parameters learned to predict alignment. Let $|S|$ and $|T|$ denote the lengths of the source and target sequences respectively and $D$ denoate the state size of the encoder and decoder RNN.

Memory-Based Attention Model	Our proposed model is shown in Figure 1 . During encoding, we compute an attention matrix $C \in \mathbb {R}^{K \times D}$ , where $K$ is the number of attention vectors and a hyperparameter of our method, and $D$ is the dimensionality of the top-most encoder state. This matrix is computed by predicting a score vector $\alpha _t \in \mathbb {R}^K$ at each encoding time step $t$ . $C$ is then a linear combination of the encoder states, weighted by $\alpha _t$ :  $$C_k & = \sum _{t=0}^{|S|}{\alpha _{tk} s_t} \\
\alpha _t & = \text{softmax}(W_\alpha s_t) ,$$   (Eq. 7)   where $W_{\alpha }$ is a parameter matrix in $\mathbb {R}^{K\times D}$ . The computational time complexity for this operation is $O(KD|S|)$ . One can think of C as compact fixed-length memory that the decoder will perform attention over.

Efficient Attention using a Fixed-Size Memory Representation	The standard content-based attention mechanism typically used in sequence-to-sequence models is computationally expensive as it requires the comparison of large encoder and decoder states at each time step. In this work, we propose an alternative attention mechanism based on a fixed size memory representation that is more efficient. Our technique predicts a compact set of K attention contexts during encoding and lets the decoder compute an efficient lookup that does not need to consult the memory. We show that our approach performs on-par with the standard attention mechanism while yielding inference speedups of 20% for real-world translation tasks and more for tasks with longer sequences. By visualizing attention scores we demonstrate that our models learn distinct, meaningful alignments.

Visualizing Attention	A useful property of the standard attention mechanism is that it produces meaningful alignment between source and target sequences. Often, the attention mechanism learns to progressively focus on the next source token as it decodes the target. These visualizations can be an important tool in debugging and evaluating seq2seq models and are often used for unknown token replacement. This raises the question of whether or not our proposed memory attention mechanism also learns to generate meaningful alignments. Due to limiting the number of attention contexts to a number that is generally less than the sequence length, it is not immediately obvious what each context would learn to focus on. Our hope was that the model would learn to focus on multiple alignments at the same time, within the same attention vector. For example, if the source sequence is of length 40 and we have $K=10$ attention contexts, we would hope that $C_1$ roughly focuses on tokens 1 to 4, $C_2$ on tokens 5 to 8, and so on. Figures 6 and 7 show that this is indeed the case. To generate this visualization we multiply the attention scores $\alpha $ and $\beta $ from the encoder and decoder.

That we are able to represent the source sequence with a fixed size matrix with fewer than $|S|$ rows suggests that traditional attention mechanisms may be representing the source with redundancies and wasting computational resources. This makes intuitive sense for the toy task, which should require a relatively simple representation. The last column shows that our technique significantly speeds up the inference process. The gap in inference speed increases as sequences become longer. We measured inference time on the full validation set of 1,000 examples, not including data loading or model construction times. Figure 3 shows the learning curves for sequence length 200. We see that $K=1$ is unable to fit the data distribution, while $K\in \lbrace 32, 64\rbrace $ fits the data almost as quickly as the attention-based model. Figure 3 shows the effect of varying the encoder and decoder scoring functions between softmax and sigmoid. All combinations manage to fit the data, but some converge faster than others. In section "Visualizing Attention" we show that distinct alignments are learned by different function combinations.

Such content-based attention mechanisms result in inference times of $O(D^2|S||T|)$ , as each context vector depends on the current decoder state $h_i$ and all encoder states, and requires an $O(D^2)$ matrix multiplication. The decoder outputs a distribution over a vocabulary of fixed-size $|V|$ :  $$P(y_i \vert y_{<i}, \mathbf {x}) = \text{softmax}(W[s_i; c_i] + b)$$   (Eq. 5)   The model is trained end-to-end by minimizing the negative log likelihood of the target words using stochastic gradient descent.

Conclusion	In this work, we propose a novel memory-based attention mechanism that results in a linear computational time of $O(KD(|S| + |T|))$ during decoding in seq2seq models. Through a series of experiments, we demonstrate that our technique leads to consistent inference speedups as sequences get longer, and can fit complex data distributions such as those found in Neural Machine Translation. We show that our attention mechanism learns meaningful alignments despite being constrained to a fixed representation after encoding. We encourage future work that explores the optimal values of $K$ for various language tasks and examines whether or not it is possible to predict $K$ based on the task at hand. We also encourage evaluating our models on other tasks that must deal with long sequences but have compact representations, such as summarization and question-answering, and further exploration of their effect on memory and training speed.

Model Interpretations	Our memory-based attention model can be understood intuitively in two ways. We can interpret it as "predicting" the set of attention contexts produced by a standard attention mechanism during encoding. To see this, assume we set $K \approx |T|$ . In this case, we predict all $|T|$ attention contexts during the encoding stage and learn to choose the right one during decoding. This is cheaper than computing contexts one-by-one based on the decoder and encoder content. In fact, we could enforce this objective by first training a regular attention model and adding a regularization term to force the memory matrix $C$ to be close to the $T\times D$ vectors computed by the standard attention. We leave it to future work to explore such an objective. Alternatively, we can interpret our mechanism as first predicting a compact $K \times D$ memory matrix, a representation of the source sequence, and then performing location-based attention on the memory by picking which row of the matrix to attend to. Standard location-based attention mechanism, by contrast, predicts a location in the source sequence to focus on  ,  .

Related Work	Our contributions build on previous work in making seq2seq models more computationally efficient.  introduce various attention mechanisms that are computationally simpler and perform as well or better than the original one presented in  . However, these typically still require $O(D^2)$ computation complexity, or lack the flexibility to look at the full source sequence. Efficient location-based attention  has also been explored in the image recognition domain.   presents several enhancements to the standard seq2seq architecture that allow more efficient computation on GPUs, such as only attending on the bottom layer.  propose a linear time architecture based on stacked convolutional neural networks.  also propose the use of convolutional encoders to speed up NMT.  propose a linear attention mechanism based on covariance matrices applied to information retrieval.  enable online linear time attention calculation by enforcing that the alignment between input and output sequence elements be monotonic. Previously, monotonic attention was proposed for morphological inflection generation by  .

11)  To obtain $l_s$ we first calculate a constant matrix $L$ where we define each element as  $$L_{ks} & = (1-k/K)(1-s/\mathcal {S})+\frac{k}{K}\frac{s}{\mathcal {S}},$$   (Eq. 12)   adapting a formula from  . Here, $k\in \lbrace 1,2,...,K\rbrace $ is the context vector index and $\mathcal {S}$ is the maximum sequence length across all source sequences. The manifold is shown graphically in Figure 2 . We can see that earlier encoder states are upweighted in the first context vectors, and later states are upweighted in later vectors. The symmetry of the manifold and its stationary point having value 0.5 both follow from Eq. 12 . The elements of the matrix that fall beyond the sequence lengths are then masked out and the remaining elements are renormalized across the timestep dimension. This results in the jagged array of position encodings $\lbrace l_{ks}\rbrace $ .

Toy Copying Experiment	Due to the reduction of computational time complexity we expect our method to yield performance gains especially for longer sequences and tasks where the source can be compactly represented in a fixed-size memory matrix. To investigate the trade-off between speed and performance, we compare our technique to standard models with and without attention on a Sequence Copy Task of varying length like in  . We generated 4 training datasets of 100,000 examples and a validation dataset of 1,000 examples. The vocabulary size was 20. For each dataset, the sequences had lengths randomly chosen between 0 to $L$ , for $L\in \lbrace 10, 50, 100, 200\rbrace $ unique to each dataset. All models are implemented using TensorFlow based on the seq2seq implementation of  and trained on a single machine with a Nvidia K40m GPU. We use a 2-layer 256-unit, a bidirectional LSTM  encoder, a 2-layer 256-unit LSTM decoder, and 256-dimensional embeddings. For the attention baseline, we use the standard parametrized attention  .

Table 2 compares our approach with and without position encodings, and with varying values for hyperparameter $K$ , to baseline models with regular attention mechanism. Learning curves are shown in Figure 4 . We see that our memory attention model with sufficiently high $K$ performs on-par with, or slightly better, than the attention-based baseline model despite its simpler nature. Across the board, models with $K=64$ performed better than corresponding models with $K=32$ , suggesting that using a larger number of attention vectors can capture a richer understanding of source sequences. Position encodings also seem to consistently improve model performance. Table 3 shows that our model results in faster decoding time even on a complex dataset with a large vocabulary of 16k. We measured decoding time over the full validation set, not including time used for model setup and data loading, averaged across 10 runs. The average sequence length for examples in this data was 35, and we expect more significant speedups for tasks with longer sequences, as suggested by our experiments on toy data. Note that in our NMT examples/experiments, $K\approx T$ , but we obtain computational savings from the fact that $K \ll D$ .

We may be able to set $K \ll T$ , as in toy copying, and still get very good performance in other tasks. For instance, in summarization the source is complex but the representation of the source required to perform the task is "simple" (i.e. all that is needed to generate the abstract). Figure 5 shows the effect of using sigmoid and softmax function in the encoders and decoders. We found that softmax/softmax consistently performs badly, while all other combinations perform about equally well. We report results for the best combination only (as chosen on the validation set), but we found this choice to only make a minor difference.

Figure 8 shows a sample translation task visualization. Figure 6 suggests that our model learns distinct ways to use its memory depending on the encoder and decoder functions. Interestingly, using softmax normalization results in attention maps typical of those derived from using standard attention, i.e. a relatively linear mapping between source and target tokens. Meanwhile, using sigmoid gating results in what seems to be a distributed representation of the source sequences across encoder time steps, with multiple contiguous attention contexts being accessed at each decoding step.

Sequence-to-Sequence Model with Attention	Our models are based on an encoder-decoder architecture with attention mechanism  ,  . An encoder function takes as input a sequence of source tokens $\mathbf {x} = (x_1, ..., x_m)$ and produces a sequence of states $\mathbf {s} = (s_1, ..., s_m)$ .The decoder is an RNN that predicts the probability of a target sequence $\mathbf {y} = (y_1, ..., y_T \mid \mathbf {s})$ . The probability of each target token $y_i \in \lbrace 1, ... ,|V|\rbrace $ is predicted based on the recurrent state in the decoder RNN, $h_i$ , the previous words, $y_{<i}$ , and a context vector $c_i$ . The context vector $c_i$ , also referred to as the attention vector, is calculated as a weighted average of the source states.

Question: How much is the BLEU score?
Answer: Ranges from 44.22 to 100.00 depending on K and the sequence length.