Understanding How Encoder-Decoder Architectures Attend
Year: 2021

Abstract:
Encoder-decoder networks with attention have proven to be a powerful way to solve many sequence-to-sequence tasks. In these networks, attention aligns encoder and decoder states and is often used for visualizing network behavior. However, the mechanisms used by networks to generate appropriate attention matrices are still mysterious. Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forward, etc.) are also not well understood. In this work, we investigate how encoder-decoder networks solve different sequence-to-sequence tasks. We introduce a way of decomposing hidden states over a sequence into temporal (independent of input) and inputdriven (independent of sequence position) components. This reveals how attention matrices are formed: depending on the task requirements, networks rely more heavily on either the temporal or input-driven components. These findings hold across both recurrent and feed-forward architectures despite their differences in forming the temporal components. Overall, our results provide new insight into the inner workings of attention-based encoder-decoder networks.

heading: 1 Introduction
text: Modern machine learning encoder-decoder architectures can achieve strong performance on sequenceto-sequence tasks such as machine translation (Bahdanau et al., 2014; Luong et al., 2015; Wu et al., 2016; Vaswani et al., 2017), language modeling (Raffel et al., 2020), speech-to-text (Chan et al., 2015; Prabhavalkar et al., 2017; Chiu et al., 2018), etc. Many of these architectures make use of attention (Bahdanau et al., 2014), a mechanism that allows the network to focus on a specific part of the input most relevant to the current prediction step. Attention has proven to be a critical mechanism; indeed many modern architectures, such as the Transformer, are fully attention-based (Vaswani et al., 2017). However, despite the success of these architectures, an understanding of how said networks solve such tasks using attention remains largely unknown.
Attention mechanisms are attractive because they are interpretable, and often illuminate key computations required for a task. For example, consider neural machine translation—trained networks exhibit attention matrices that align words in the encoder sequence with the appropriate corresponding position in the decoder sentence (Ghader & Monz, 2017; Ding et al., 2019). In this case, the attention matrix already contains information about which words in the source sequence are relevant for translating a particular word in the target sequence; that is, forming the attention matrix itself
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
constitutes a significant part of solving the overall task. How is it that networks are able to achieve this? What are the mechanisms underlying how networks form attention, and how do they vary across tasks and architectures?
In this work, we study these questions by analyzing three different encoder-decoder architectures on sequence-to-sequence tasks. We develop a method for decomposing the hidden states of the network into a sum of components that let us isolate input driven behavior from temporal (or sequence) driven behavior. We use this to first understand how networks solve tasks where all samples use the same attention matrix, a diagonal one. We then build on that to show how additional mechanisms can generate sample-dependent attention matrices that are still close to the average matrix.
Our Contributions
• We propose a decomposition of hidden state dynamics into separate pieces, one of which explains the temporal behavior of the network, another of which describes the input behavior. We show such a decomposition aids in understanding the behavior of networks with attention.
• In the tasks studied, we show the temporal (input) components play a larger role in determining the attention matrix as the average attention matrix becomes a better (worse) approximation for a random sample’s attention matrix.
• We discuss the dynamics of architectures with attention and/or recurrence and show how the input/temporal component behavior differs across said architectures.
• We investigate the detailed temporal and input component dynamics in a synthetic setting to understand the mechanism behind common sequence-to-sequence structures and how they might differ in the presence of recurrence.
Related Work As mentioned in the introduction, a common technique to gain some understanding is to visualize learned attention matrices, though the degree to which such visualization can explain model predictions is disputed Wiegreffe & Pinter (2019); Jain & Wallace (2019); Serrano & Smith (2019). Input saliency Bastings & Filippova (2020) and attribution-propagation Chefer et al. (2020) methods have also been studied as potential tools for model interpretability.
Complementary to these works, our approach builds on a recent line of work analyzing the computational mechanisms learned by RNNs from a dynamical systems perspective. These analyses have identified simple and interpretable hidden state dynamics underlying RNN operation on text-classification tasks such as binary sentiment analysis (Maheswaranathan et al., 2019; Maheswaranathan & Sussillo, 2020) and document classification (Aitken et al., 2020). Our work extends these ideas into the domain of sequence-to-sequence tasks.
Notation Let T and S be the input and output sequence length of a given sample, respectively. We denote the encoder and decoder hidden states by hEt ∈ Rn with t = 1, . . . , T . Similarly, we denote decoder hidden states by hDs ∈ Rn, with s = 1, . . . , S. The encoder and decoder hidden state dimensions are always taken to be equal in this work. Inputs to the encoder and decoder are denoted by xEt ∈ Rd and xDs ∈ Rd̃. When necessary, we subscript different samples from a test/train set using α, β, γ, e.g. xEt,α for α = 1, . . . ,M .
Outline We begin by introducing the three architectures we investigate in this work with varying combinations of recursion and attention. Next we introduce our temporal and input component decomposition and follow this up with a demonstration of how such a decomposition allows us to understand the dynamics of attention in a simple one-to-one translation task. Afterwards, we apply this decomposition to two additional tasks with increasing levels of complexity and discuss how our decomposition gives insight into the behavior of attention in these tasks.

heading: 2 Setup
text: A schematic of the three architectures we study is shown in Fig. 1 (see SM for precise expressions).
Vanilla Encoder Decoder (VED) is a recurrent encoder-decoder architecture with no attention (Sutskever et al., 2014). The encoder and decoder update expression are hEt = FE(h E t−1,x E t ) and hDs = FD(h D s−1,x D s ), respectively. Here, FD and FE are functions that implement the hidden state updates, which in this work are each one of three modern RNN cells: LSTMs (Hochreiter & Schmidhuber, 1997), GRUs (Cho et al., 2014), or UGRNNs (Collins et al., 2016).
Encoder-Decoder with Attention (AED) is identical to the VED architecture above with a simple attention mechanism added (Bahdanau et al., 2014; Luong et al., 2015). For time step s of the decoder, we compute a context vector cs, a weighted sum of encoder hidden states, cs := ∑T t=1 αsth E t , with αt := softmax (a1t, . . . , aSt) the tth column of the attention matrix and ast := hDs ·hEt the alignment between a given decoder and encoder hidden state. While more complicated attention mechanisms exist, in the main text we analyze the simplest form of attention for convenience of analysis.1
Attention Only (AO) is identical to the AED network above, but simply eliminates the recurrent information passed from one RNN cell to the next and instead adds fixed positional encoding vectors to the encoder and decoder inputs (Vaswani et al., 2017). Due to the lack of recurrence, the RNN functions FE and FD simply act as feedforward networks in this setting.2 AO can be treated as a simplified version of a Transformer without self-attention, hence our analysis may also provide a hint into their inner workings (Vaswani et al., 2017).

heading: 2.1 Temporal and Input Components
text: In architectures with attention, we will show that it is helpful to write the hidden states using what we will refer to as their temporal and input components. This will be useful because each hidden state has an associated time step and input word at that same time step (e.g. s and xDs for h D s ), therefore such a decomposition will often allow us to disentangle temporal and input behavior from any other network dynamics.
We define the temporal components of the encoder and decoder to be the average hidden state at a given time step, which we denote by µEt and µ D s , respectively. Similarly, we define an encoder input component to be the average of all hEt − µEt for hidden states that immediately follow a given input word. We analogously define the decoder input components. In practice, we estimate such averages using a test set of size M , so that the temporal and input components of the encoder are respectively given by
µEt ≈ ∑M α=1 1≤EoS,αh E t,α∑M
β=1 1≤EoS,β , χE (xt,α) ≈
∑M β=1 ∑T t′=1 1xt,α,xt′,β ( hEt′,β − µEt′ ) ∑M γ=1 ∑T t′′=1 1xt,α,xt′′,γ , (1)
where hEt,α the encoder hidden state of the αth sample, 1≤EoS,α is a mask that is zero if the αth sample is beyond the end of sentence, 1xt,α,xt′,β is a mask that is zero if xt,α 6= xt′,β , and we have temporarily suppressed superscripts on the inputs for brevity.3 By definition, the temporal components only vary with time and the input components only vary with input/output word. As such,
1In the SM, we implement a learned-attention mechanism using a scaled-dot product attention in the form of queries, keys, and value matrices (Vaswani et al., 2017). For the AED and AO architectures, we find qualitatively similar results to the simple dot-product attention presented in the main text.
2We train non-gated feedforward networks and find their dynamics to be qualitatively the same, see SM. 3See SM for more details on this definition and the analogous decoder definitions.
it will be useful to denote the encoder and decoder input components by χEx and χ D y , with x and y respectively running over all input and output words (e.g. χEyes and χ D oui). We can then write any hidden state as
hEt = µ E t + χ E x + ∆h E t , h D s = µ D s + χ D y + ∆h D s , (2)
with ∆hEt := h E t − µEt − χEt and ∆hDs := hDs − µDs − χDy the delta components of encoder and decoder hidden states, respectively. Intuitively, we are simply decomposing each hidden state vector as a sum of a component that only varies with time/position in the sequence (independent of input), a component that only varies with input (independent of position), and whatever else is left over. Finally, we will often refer to hidden states without their temporal component, i.e. χEx + ∆h E t and χDy + ∆h D s , so for brevity we refer to these combinations as the input-delta components.
Using the temporal and input components in (2), we can decompose the attention alignment between two hidden states as
ast = ( µDs + χ D y + ∆h D s ) · ( µEt + χ E x + ∆h E t ) . (3)
We will show below that in certain cases several of the nine terms of this expression approximately vanish, leading to simple and interpretable attention mechanisms.

heading: 3 One-to-One Results
text: To first establish a basis of how each of the three architectures learn to solve tasks and the role of their input and temporal components, we start by studying their dynamics for a synthetic one-to-one translation task. The task is to convert a sequence of input words into a corresponding sequence of output words, where there is a one-to-one translation dictionary, e.g. converting a sequence of letters to their corresponding position in the alphabet, {B,A,C,A,D} → {2, 1, 3, 1, 4}. We generate the
input phrases to have variable length, but outputs always have equal length to their input (i.e. T = S). While a solution to this task is trivial, it is not obvious how each neural network architecture will solve the task. Although this is a severely simplified approximation to realistic sequence-to-sequence tasks, we will show below that many of the dynamics the AED and AO networks learn on this task are qualitatively present in more complex tasks.
Encoder-Decoder with Attention. After training the AED architecture, we apply the decomposition of (2) to the hidden states. Plotting the temporal components of both the encoder and decoder, they each form an approximate circle that is traversed as their respective inputs are read in (Fig. 2a).4 Additionally, we find the encoder and decoder temporal components are closest to alignment when s = t. We also plot the input components of the encoder and decoder together with the encoder input-delta components, i.e. χEx + ∆h E t , and the network’s readout vectors (Fig. 2b).
5 We see for the encoder hidden states, the input-delta components are clustered close to their respective input components, meaning for this task the delta components are negligible. Also note the decoder input-delta components are significantly smaller in magnitude than the decoder temporal components. Together, this means we can approximate the encoder and decoder hidden states as hEt ≈ µEt + χEx and hDs ≈ µDs , respectively. Finally, note the readout vector for a given output word aligns with the input components of its translated input word, e.g. the readout for ‘1’ aligns with the input component for ‘A’ (Fig. 2b).6
For the one-to-one translation task, the network learns an approximately diagonal attention matrix, meaning the decoder at time s primarily attends to the encoder’s hidden state at t = s. Additionally, we find the temporal and input-delta components to be close to orthogonal for all time steps, which allows the network’s attention mechanism to isolate temporal dependence rather than input dependence. Since we can approximate the hidden states as hEt ≈ µEt + χEx and hDs ≈ µDs , and the temporal and input components are orthogonal, the alignment in (3) can be written simply as ast ≈ µDs · µEt . This means that the full attention is completely described by the temporal components and thus input-independent (this will not necessarily be true for other tasks, as we will see later).
With the above results, we can understand how AED solves the one-to-one translation task. After reading a given input, the encoder hidden state is primarily composed of an input and temporal component that are approximately orthogonal to one another, with the input component aligned with the readout of the translated input word (Fig. 2b). The decoder hidden states are approximately made up of only a temporal component, whose sole job is to align with the corresponding encoder temporal component. Temporal components of the decoder and encoder are closest to alignment for t = s, so the network primarily attends to the encoder state hEt=s. The alignment between encoder input components and readouts yields maximum logit values for the correct translation.
Attention Only. Now we turn to AO architecture, which is identical to AED except with the recurrent connections cut, and positional encoding added to the inputs. We find that AO has qualitatively similar temporal components that give rise to diagonal attention (Fig. 2c) and the input components align with the readouts (Fig. 2d). Thus AO solves the task in a similar manner as AED. The only difference is that the temporal components, driven by RNN dynamics in AED, are now driven purely by the positional encoding in AO.
Vanilla Encoder-Decoder. After training the VED architecture, we find the encoder and decoder hidden states belonging to the same time step form clusters, and said clusters are closest to those corresponding to adjacent time steps. This yields temporal components that are close to one another for adjacent times, with µET next to µ D 1 (Figs. 2e). Since there is no attention in this architecture, there is no incentive for the network to align temporal components of the encoder and decoder as we saw in AED and AO.
4Here and in plots that follow, we plot the various components using principal component analysis (PCA) projections simply as a convenient visualization tool. Other than observation that in some cases the temporal/input components live in a low-dimensional subspace, none of our quantitative analysis is dependent upon the PCA projections. For all one-to-one plots, a large percentage (> 90%) of the variance is explained by the first 2 or 3 PC dimensions.
5For N possible input words, the encoder input components align with the vetrices of an (N − 1)-simplex, which is similar to the classification behavior observed in Aitken et al. (2020).
6Since in AED we pass both the decoder hidden state and the context vector to the readout, each readout vector is twice the hidden state dimension. We plot only the readout weights corresponding to the context vector, since generally those corresponding to the decoder hidden state are negligible, see SM for more details.
As recurrence is the only method of transferring information across time steps, encoder and decoder hidden states must carry all relevant information from preceding steps. Together, this results in the delta components deviating significantly more from their respective input components for VED relative to AED and AO (Fig. 2f). That is, since hidden states must hold the information of inputs/outputs for multiple time steps, we cannot expect them to be well approximated by µEt + χ E x because, by definition, it is agnostic to the network’s inputs at any time other than t (and similarly for µEs + χ E y). As such, the temporal and input component decomposition gains us little insight into the inner workings of the VED architecture. Additional details of the VED architecture dynamics are discussed in the SM.
Additional Tasks. In this section, we briefly address how two additional synthetic tasks can be understood using the temporal and input component decomposition. First, consider a task identical to the one-to-one task, with the target sequence reversed in time, e.g. {B,A,C,A,D} → {4, 1, 3, 1, 2}. For this task, we expect an attention matrix that is anti-diagonal (i.e. it is nonzero for t = S + 1− s). For the AED and AO networks trained on this task, we find their temporal and input component behavior to be identical to the original one-to-one task with one exception: instead of the encoder and decoder temporal components following one another, we find one trajectory is flipped in such a way as to yield an anti-diagonal attention matrix (Fig. 2g). That is, the last encoder temporal component is aligned with the first decoder temporal component and vice versa.
Second, consider the task of sorting the input alphabetically, e.g. {B,C,A,D} → {A,B,C,D}. For this example, we expect the network to learn an input-dependent attention matrix that correctly permutes the input sequence. Since there is no longer a correlation between input and output sequence location, the average attention matrix is very different from that of a random sample, and so we expect the temporal components to insignificantly contribute to the alignment. Indeed, we find µDs ·µEt to be negligible, and instead ∆hDs · χEx dominates the alignment values (Fig. 2h).

heading: 4 Beyond One-to-One Results
text: In this section we analyze the dynamics of two tasks that have “close-to-diagonal” attention: (1) what we refer to as the extended SCAN dataset and (2) translation between English and French phrases. Since we found temporal/input component decomposition to provide little insight into VED dynamics, our focus in this section will be on only the AED and AO architectures. For both tasks we explore below, parts of the picture we established on the one-to-one task continues to hold. However, we will see that in order to succeed at these tasks, both AO and AED must implement additional mechanisms on top of the dynamics we saw for the one-to-one task.
Extended SCAN (eSCAN) is a modified version of the SCAN dataset (Lake & Baroni, 2018), in which we randomly concatenate a subset of the phrases to form phrases of length 15 to 20 (see SM for
details). The eSCAN tasks is close to one-to-one translation, but is augmented with several additional rules that modify its structure. For example, a common sequence-to-sequence structure is that a pair of outputs can swap order relative to their corresponding inputs: the English words ‘green field’ translate to ‘champ vert’ in French (with ‘field’↔ ‘champ’ and ‘green’↔ ‘vert’). This behavior is present in eSCAN: when the input word ‘left’ follows a verb the output command must first turn the respective direction and then perform said action (e.g. ‘run left’→ ‘LTURN RUN’). The AED and AO models both achieve ≥ 98% word accuracy on eSCAN. Looking at a sample attention matrix of AED, we see consecutive words in the output phrase tend to attend to the same encoder hidden states at the end of subphrases in the input phrase (Fig. 3a). Once again decomposing the AED network’s hidden states as in (2), we find the temporal components of the encoder and decoder form curves that mirror one another, leading to an approximately diagonal attention matrix (Fig. 3b). The delta components are significantly less negligible for this task, as evidence by the fact χEx + ∆h E t aren’t nearly as clustered around their corresponding input component (Fig. 3c). As we will verify later, this is a direct result of the network’s use of recurrence, since now hidden states carry information about subphrases, rather than just individual words.
Training the AO architecture on eSCAN, we also observe non-diagonal attention matrices, but in general their qualitative features differ from those of the AED architecture (Fig. 3d). Focusing on the subphrase mapping ‘run twice’→ ‘RUN RUN’, we see the network learns to attend to the word preceding ‘twice’, since it can no longer rely on recurrence to carry said word’s identity forward. Once again, the temporal components of the encoder and decoder trace out paths that roughly follow one another (Fig. 3e). We see input-delta components cluster around their corresponding input components, indicating the delta components are small (Fig. 3f). Finally, we again see the readouts of particular outputs align well with the input components of their corresponding input word.
English to French Translation is another example of a nearly-diagonal task. We train the AED and AO architectures on this natural language task using a subset of the para_crawl dataset Bañón et al. (2020) consisting of over 30 million parallel sentences. To aid interpetation, we tokenize each sentence at the word level and maintain a vocabulary of 30k words in each language; we train on sentences of length up to 15 tokens.
Since English and French are syntactically similar with roughly consistent word ordering, the attention matrices are in general close to diagonal (Fig. 4a). Again, note the presence of features that require off-diagonal attention, such as the flipping of word ordering in the input/output phrases and multiple words in French mapping to a single English word. Using the decomposition of (2), the temporal components in both AED and AO continue to trace out similar curves (Fig. 4b). Notably, the alignment resulting from the temporal components is significantly less diagonal, with the diagonal behavior clearest at the beginning of the phrase. Such behavior makes sense: the presence of offdiagonal structure means, on average, translation pairs become increasingly offset the further one moves into a phrase. With offsets that increasingly vary from phrase to phrase, the network must rely less on temporal component alignments, which by definition are independent of the inputs. Finally, we see that the the dot product between the input components and the readout vectors implement the translation dictionary, just as it did for the one-to-one task (Fig. 4c, see below for additional discussion).

heading: 4.1 A Closer Look at Model Features
text: As expected, both the AED and AO architectures have more nuanced attention mechanisms when trained on eSCAN and translation. In this section, we investigate a few of their features in detail.
Alignment Approximation. Recall that for the one-to-one task, we found the alignment scores could be well approximated by ast ≈ µDs · µEt , which was agnostic to the details of the input sequence. For eSCAN, the µDs · µEt term is still largely dominant, capturing > 77% of ast in the AED and AO networks (Fig. 5a). A better approximation for the alignment scores is ast ≈ µDs · µEt + µDs · χEx + µDs ·∆hEt , i.e. we include two additional terms on top of what was used for one-to-one. Since χEx and ∆h E t are dependent upon the input sequence, this means the alignment has non-trivial input dependence, as we would expect. In both architectures, we find this approximation captures > 87% of the top alignment scores. For translation, we see the term µDs · µEt makes up a significantly smaller portion of the alignment scores, and in general we find none of the nine terms in (3) dominate above the rest (Fig. 5a). However, at early times in the AED architecture, we again see µDs · µEt is the largest contribution to the alignment. As mentioned above, this matches our intuition that words at the start of the encoder/decoder phrase have a smaller offset from one another than later in the phrase, so the network can rely more on temporal components to determine attention.
Temporal Component Offset. For the one-to-one task, the input sequence length was always equal to the output sequence length, so the temporal components were always peaked at s = t (Fig. 5b). In eSCAN, the input word ‘and’ has no corresponding output, which has a non-trivial effect on how the network attends since its appearance means later words in input phrase are offset from their corresponding output word. This effect also compounds with multiple occurrences of ‘and’ in the input. The AED and AO networks learn to handle such behavior by biasing the temporal component dot product, µDs · µEt , the dominant alignment contribution, to be larger for time steps t further along in the encoder phrase, i.e. t > s (Fig. 5b). It is possible to compute the average offset of input and output words in eSCAN training set, and we see the maximum of µDs · µEt follow this estimate quite well. Similarly, in our set of English to French translation phrases, we find the French phrases to be on average ∼ 20% longer than their English counterparts. This results in the maximum of µDs · µEt to gradually move toward t < s, e.g. on average the decoder attends to earlier times in the encoder (Fig. 5b). Additionally, note the temporal dot product falls off significantly slower as a function of offset for later time steps, indicating the drop off for non-diagonal alignments is smaller and thus it is easier for the network to off-diagonally attend.
Word Variance. The encoder hidden states in the one-to-one task had a negligible delta component, so the hidden states could be approximated as hEt ≈ µEt + χEx. By definition, χEx is constant for a given input word, so the variance in the hidden states corresponding to a given input word is primarily contained in the temporal component (Fig. 5c). Since the temporal component is input-independent, this led to a clear understanding of how all of a network’s hidden states evolve with time and input. In the AED and AO architectures trained on eSCAN, we find the variance of the input word’s hidden
states drops by 90% and 95% when the temporal component is subtracted out, respectively (Fig. 5c). Meanwhile, in translation, we find the variance only drops by 8% and 25% for the AED and AO architectures, indicating there is significant variance in the hidden states beyond the average temporal evolution and thus more intricate dynamics.
Input/Readout Alignment. Lastly, recall we saw that in the one-to-one case the input components’ alignment with readouts implemented the translation dictionary (Figs. 2b, d). For eSCAN, the dot product of a given readout is again largest with the input component of its corresponding input word, e.g. the readout corresponding to ‘RUN’ is maximal for the input component of ‘run’ (Fig. 5d). Notably, words that produce no corresponding output such as ‘and’ and ‘twice’ are not the maximal in alignment with any readout vector. Similarly, for translation, we see the French-word readouts have the largest dot product their translated English words (Fig. 4c). For example, the readouts for the words ‘la’, ‘le’, and ‘les’, which are the gendered French equivalents of ‘the’, all have maximal alignments with χEthe.

heading: 4.2 A Closer Look at Dynamics
text: In this section, we leverage the temporal and input component decomposition to take a closer look at how networks trained on the eSCAN dataset implement particular off-diagonal attentions. Many of the sequence translation structures in eSCAN are seen in realistic datasets, so we this analysis will give clues toward understanding the behavior of more complicated sequence-to-sequence tasks.
A common structure in sequence-to-sequence tasks is when an output word is modified by the words preceding it. For example, the phrases ‘we run‘ and ‘they run’ translate to ‘nous courrons’ and ‘ils courent’ in French, respectively (with the second word in each the translation of ‘run’). We can study this phenomenon in eSCAN since the word ‘twice’ tells the network to repeat the command just issued two times, e.g. ‘run twice’ outputs to ‘RUN RUN’. Hence, the output corresponding to the input ‘twice’ changes based on other words in the phrase.
Since an AED network has recurrence, when it sees the word ‘twice’ it can know what verb preceded it. Plotting input-delta components, we see the RNN outputs ‘twice’ hidden states in three separate clusters separated by the preceding word (Fig. 6a). Thus for an occurrence of ‘twice’ at time step t, we have χEtwice + ∆h E t ≈ χEverb + ∆hEt−1. For example, this means the AED learns to read in ‘run twice’ approximately the same as ‘run run’. This is an example of the network learning context.
AO has no recurrence, so it can’t know which word was output before ‘twice’. Hence, unlike the AED case, all occurrences of ‘twice’ are the same input-delta component cluster regardless of what word preceded it. Instead, it has to rely on attending to the word that modifies the output, which in this case is simply the preceding word (Fig. 3d). As mentioned in Sec. 4.1, for the eSCAN task we find the alignment to be well approximated by ast ≈ µDs · hEt . When the word ’twice’ appears in the input phrase, we find µDs · χEtwice + µDs ·∆hEt < 0 for s = t (Fig. 6b). This decreases the value of the alignment as,s, and so the decoder instead attends to the time step with the second largest value
of µDs · µEt , which the network has learned to be t = s− 1. Hence, as,s−1 is the largest alignment, corresponding to the time step before ‘twice’ with the verb the network needs to output again. Unlike the one-to-one case, the encoder input-delta and the decoder temporal components are no longer approximately orthogonal to one another (Fig. 6c). In the case of ‘twice’, χEtwice + ∆h E t is partially antialigned with the temporal component, yielding a negative dot product.
This mechanism generalizes beyond the word ’twice’: in eSCAN we see input-delta components of several input words are no longer orthogonal to the decoder’s temporal component (Fig. 6c). Like ‘twice’, the dot product of the input-delta component for a given word with its corresponding temporal component determines how much its alignment score is increased/decreased. For example, we see χEand + ∆h E t has a negative dot product with the temporal component, meaning it leans away from its corresponding temporal component. Again, this make sense from eSCAN task: the word ‘and’ has no corresponding output, hence it never wants to be attended to by the decoder.
Perhaps contradictory to expectation, χEleft + ∆h E t has a negative dot product with the temporal component. However, note that the alignment of χEx + ∆h E t with the h D s is dependent on both t and s. We plot the dot products of χEx + ∆h E t and h D s as a function of their offset, defined to be the t− s (Fig. 6d). Notably, χEleft + ∆h E t has a larger dot product for larger offsets, meaning it increases its alignment when t > s. This makes sense from the point of view that the word ‘left’ is always further along in the input phrase than its corresponding output ‘LTURN’, and this offset is only compounded by the presence of the word ‘and’. Thus, the word ‘left’ only wants to get noticed if it is ahead of the corresponding decoder time step, otherwise it hides. Additionally, the words ‘and‘ and ‘twice‘ have large negative dot products for all offsets, since they never want to be the subject of attention.

heading: 5 Discussion
text: In this work, we studied the hidden state dynamics of sequence-to-sequence tasks in architectures with recurrence and attention. We proposed a decomposition of the hidden states into parts that are input- and time-independent and showed when such a decomposition aids in understanding the behavior of encoder-decoder networks.
Although we have started by analyzing translation tasks, it would be interesting to understand how said decomposition works on different sequence-to-sequence tasks, such as speech-to-text. Additionally, with our focus on the simplest encoder-decoder architectures, it is important to investigate how much the observed dynamics generalize to more complicated network setups, such as networks with bidirectional RNNs or multiheaded and self-attention mechanisms. Our analysis of the attentiononly architecture, which bears resemblance to the transformer architecture, suggests that a similar dynamical behavior may also hold for the Transformer, hinting at the working mechanisms behind this popular non-recurrent architecture.
Acknowledgments and Disclosure of Funding
We thank Ankush Garg for collaboration during the early part of this work. None of the authors receive third-party funding/support during the 36 months prior to this submission or had competing interests.

Review summaries:
The paper introduces a novel method for understanding how attention works in sequence alignment problems by decomposing the hidden states into temporal and input-driven contributions.  This is applied to three different network architectures, with AO only recurrent connection, AED with both recurrent connections and attention, and AO, with only attention.  They then test on some different synthetic dataset and link the attention matrix to the hidden state decomposition and to the temporal and input component dynamics.

In this work, the authors propose a framework to analyze how different encoder-decoder architectures attend in 3 simulated tasks and 2 real-world tasks. The framework relies on isolating the  following components in the network: 1. Temporal - Average hidden state of the encoder/decoder at a specific timestep, across all the test samples. 2. Input - Average deviation of a word's hidden state from the mean state at the timepoint it occurs in, across all its occurrences in the test set. Using these components, the paper factorizes the attention alignment between eh encoder & decoder's hidden state into 9 different products of the temporal and input states.  The usefulness of the two components in understanding attention mechanisms is demonstrated with 3 tasks: 1. Simulated tasks: Matching 3 characters to their numbered IDs | The authors find that the input and read-out states are highly aligned such that the decoder only needs to find its corresponding temporal match in the encoder. This is apparent from the results in the paper. Further, the non-attention based network doesn't demonstrate similar behavior. 2. eSCAN: Extension of the SCAN dataset formed by concatenating commands to generate longer sequences 3. English $\rightarrow$ French translation  For tasks 2 & 3, the authors find that the networks learn mappings between the input tokens and corresponding actions/French words in the readout layer. Further, the networks attend through a combination of temporal information and input components unlike task 1. Since the output in several cases doesn't depend on just 1 timepjoint in the input, these networks also exhibit 'offset' behavior wherein they attend to information at different time points depending on the input token (thus violating their orthogonality)  Overall, by separating the input and temporal components of the network state, the paper reveals if networks use only one mechanism or a combination to attend and solve a task.

1. Motivation: The paper focuses on understanding the behavior of networks with attention and investigating how encoder-decoder networks solve different sequence-to-sequence tasks. To this end, the authors hypothesize that representations (i.e., hidden states) can be decomposed into "temporal" and "input" components.  2. Approach: In detail, the author presents a decomposition of the hidden states of encoder/decoder into temporal (independent of input) and input-driven (independent of sequence position) parts, where the former is used to explain the temporal behavior of the network and the latter is used to describe the input behavior.   3. Experiments: This paper conducts extensive experimental studies on several common sequence-to-sequence frameworks, including vanilla RNN, RNN with attention, and simplified Transformer networks. They study the sequence-to-sequence networks on both synthetic and real-world tasks, i.e., a synthetic task, eSCAN and English-French translation tasks, and find that the temporal components contribute more when computing the attention.

This paper proposes a novel method to decompose hidden representations of encoder-decoder architectures into two kinds of components: temporal and input-driven. The temporal component is the average hidden representation at a given time step thus it only varies with time, while the input-driven component is defined as the representation of a specific input word removing temporal component thus it is input-dependent but time-independent. Based on the decomposition, the authors show how encoder-decoder architectures attend, providing some insights for future research.