Understanding How Encoder-Decoder Architectures Attend
Authors: Kyle Aitken
Vinay Venkatesh Ramasesh
Yuan Cao
Niru Maheswaranathan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Attention
NLP

Abstract: Encoder-decoder networks with attention have proven to be a powerful way to solve many sequence-to-sequence tasks. In these networks, attention aligns encoder and decoder states and is often used for visualizing network behavior. However, the mechanisms used by networks to generate appropriate attention matrices are still mysterious. Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forward, etc.) are also not well understood. In this work, we investigate how encoder-decoder networks solve different sequence-to-sequence tasks. We introduce a way of decomposing hidden states over a sequence into temporal (independent of input) and input-driven (independent of sequence position) components. This reveals how attention matrices are formed: depending on the task requirements, networks rely more heavily on either the temporal or input-driven components. These findings hold across both recurrent and feed-forward architectures despite their differences in forming the temporal components. Overall, our results provide new insight into the inner workings of attention-based encoder-decoder networks.

Understanding How Encoder-Decoder Architectures Attend
Kyle Aitken Department of Physics University of Washington Seattle, Washington, USA kaitken
Vinay Ramasesh Google Research, Blueshift Team Mountain View, California, USA
Yuan Cao Google, Inc. Mountain View, Califoria, USA
Niru Maheswaranathan Google Research, Brain Team Mountain View, California, USA
Abstract
Encoder-decoder networks with attention have proven be a powerful way to solve many sequence-to-seqqeecee tasks. In these networks, attention aligns encoder and decoder states and is often used for visualizing network behavior. However, the mechanisms used by networks to generate appropriate attention matrices still mysterious. Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forwarde etc.) are also not well understood. In this work, we investigate how encoder-decoder networks solve different sequence-to-sequeee hidden states over a sequence into temporal (independent of input) and imput- driven (independent of sequence position) components. This reveals how attention matrices are formed: depending on the task requirements, networks rely more heavily on either the temporal input-driven components. These findings hold across both recurrent and feed-forwart architectures despite their differences in forming the temporal components. Overall, our results provide new insight into the inner workings of attention- bbased encoder-decoder networks.
tasks. We introduce way of decomposing
Introduction
Modern machine learning encoder-decoder architectures achieve strong performance on sequence-
io-sequence tasks such as machine translation (Bahdanau et al. 2014 Luong et al., 2015 Wu et al.)
2016
Vaswani al. 2017). language modeling (Raffel al., 2020 speech-to-text (Chan al.
2017 Chiu et al. 2018), etc. Many of these architectures make use of 2014). mechanism that allows the network focus on a specific part of the input most relevant to the current prediction step. Attention has proven to be critical mechanism: indeed many modern architectures. such as the Transformer, are fully attention- based (Vaswani et 2017). However, despite the success of these architectures, an understanding of how said networks solve such tasks using attention remains largely unknown.
2015
Prabhavalkar et al.
attention (Bahdanau
Attention mechanisms are attractive because they are interpretable, and often illuminate key com- putations required for task. For example, consider neural machine translation-traln exhibit attention matrices that align words the encoder sequence with the appropriate correspond- ing position in the decoder sentence (Ghader & Monz] 2017 Ding al.) 2019). In this case, the attention matrix already contains information about which words in the source sequence are relevant for translating particular word in the target sequence; that is, forming the attention matrix itself
networks
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We denote the encoder and decoder hidden states by hi E I with t = 1..... T, ..,,,, 1, ...... S. The encoder and decoder hidden state constitutes a significant part of solving the overall task. How is it that networks are able to achieve this? What are the mechanisms underlying how networks form attention, and how do they vary across tasks and architectures?
In this work, study these questions by analyzing three different encoder-decoder architectures on tasks. We develop a method for decomposing the hidden states of the network into a sum of components that let isolate imput driven behavior from temporal (or sequence) driven behavior. We use this to first understand how networks solve tasks where all samples use the same attention matrix, a diagonal one. We then build on that to show how additional mechanisms can generate sample-dependent attention matrices that are still close to the average matrix.
Our Contributions
We propose decomposition of hidden state dynamics into separate pieces, one of which explains the temporal behavior of the network, another of which describes the input behavior. We show such decomposition aids in understanding the behavior of networks with attention.
In the tasks studied, we show the temporal (input) components play larger role in determining the attention matrix as the average attention matrix becomes a better (worse) approximation for random sample's 's attention matrix.
We discuss the dynamics of architectures with attention and/or recurrence and show how the input/temporal component behavior differs across said architectures.
We investigate the detailed temporal and input component dynamics in synthetic setting to understand the mechanism behind common sequence- coreeeeettt--geeeeeee tequeenecce structures and how they might differ in the presence of recurrence.
Related Work
As mentioned in the introduction, a common technique to gain some understanding visualize learned attention matrices, though the degree to which such visualization can explain
model predictions is disputed Wiegreffe & Pinter ): Jain & Wallace (2019); Serrano & Smith
(2019). Input saliency [Battings & Filippova| (2020) and attribution- propiaattoooattiioonn Chefer et al. (2020) methods have also been studied as potential tools for model interpretability.
Complementary to these works, our approach builds recent line of work analyzing the computa- tional mechanisms learned by RNNs from dynamical systems perspective. These analyses have iden- tified simple and interpretable hidden state dynamics underlying RNN operation text-classification tasks such as binary sentiment analysis (Maheswaranathan et al. 2019 Maheswaranathan & Sussillo 2020) and document classification (Aitken al.] 2020). Our work extends these ideas into the domain of sesueence-t--equenne tasks.
Notation Let T and S be the input and output sequence length of given sample, respectively. Similarly, we
denote decoder hidden states by h² E R", with dimensions are always taken to be equal in this work. Inputs to the encoder and decoder are denoted by x E I and xP Rdd When necessary, we subscript different samples from a test/train set using a, a,8, 7, e.g. xf.,, for a I..
Outline We begin by introducing the three architectures we investigate in this work with varying combinations of recursion and attention. Next we introduce our temporal and input component decomposition and follow this up with demonstration of how such a decomposition allows us to understand the dynamics of attention in simple one-to-one translation task. Afterwards, we apply this decomposition to two additional tasks with increasing levels of complexity and discuss how our decomposition gives insight into the behavior of attention in these tasks.
Setup
A schematic of the three architectures study is shown in Fig. I (see SM for precise expressions).
Vanilla Encoder Decoder (VED) is a recurrent encoder-decoder architecture with no attention (Sutskever et al. 2014). The encoder and decoder update expression are h. = Fr xf) and
hp Pp h2 xP), respectively. Here, FD and Fr. are functions that implement the hidden state updates, which in this work are each one of three modern RNN cells: LSTMs (Hochreiter
Schmidhuber 1997). GRUs (Cho al. 2014), UGRNNs (Collins et al. 2016).

---

VED
AED
AO
Encoder
Readout Decoder
Figure Schematic of the three primary architectures analyzed in this work. The orange, purple, and green boxes represent the encoder RNNs, decoder RNNs, and linear readout layers, respectively. Recurrent connections are shown in blue, attention based connections and computational blocks are shown in gold. The grey circles add positional encoding to the inputs.
Encoder-Decoder with Attention (AED) is identical to the VED architecture above with simple attention mechanism added (Bahdanau et al. 2014 Luong et al. 2015). For time step 8 of the decoder, we compute context vector Cs. weighted sum of encoder hidden states, C₂ := > bf, with Ct := softmax (014,... Ast) the hth column of the attention matrix and ast := h² h the aligument between given decoder and encoder hidden state. While more complicated attention mechanisms exist, in the main text we analyze the simplest form of attention for convenience of analysis |
Attention Only (AO) is identical the AED network above, but simply eliminates the recurrent information passed from one RNN cell to the next and instead adds fixed positional encoding vectors to the encoder and decoder inputs (Vaswani et al. 2017). Due to the lack of recurrence, the RNN functions Fg and FD simply act as feedforward networks in this setting AO can be treated as simplified version of a Transformer without self-attention, hence our analysis may also provide a hint into their inner workings (Vaswani et al. 2017).
2.1 Temporal and Input Components
In architectures with attention, we will show that helpful to write the hidden states using what we will refer to as their temporal and input components. This will be useful because each hidden state has an associated time step and input word at that same time step (e.g. s and x5 for hP), therefore such decomposition will often allow to disentangle temporal and input behavior from any other network dynamics.
We define the temporal components of the encoder and decoder to be the average hidden state at given time step, which we denote by iel and μd respectively. Similarly, we define an encoder input component to be the average of all h luō for hidden states that immediately follow given input word. We analogously define the decoder input components. practice, we estimate such averages using a test set of size M, so that the temporal and input components of the encoder are respectively given by
E EM 1 EsS 11 RJ
of 1x, ΣMe 2n Xe",a
(be mB
x (Xr.a)
(I)
ΣE1 1 1
where h the encoder hidden state of the oth sample, 1 1ESS, is a mask that is zero if the oth sample is beyond the end of sentence, xx,, is mask that is zero if Xt,a * Xe, and have temporarily suppressed superseripts on the inputs for brevity components only vary with time and the input components only vary with input/output word. As such,
Fl By definition, the temporal
In the SM, we implement a attentioo mechanism using a scaled-dot product attention in the form of queries, keys, and value matrices (Vaswani et al. (2077). For the AED and AO architectures, we find qualitatively similar results to the simple dot-product attention presented in the main text.
We train non- non-gated feedforward networks and find their dynamics to be qualitatively the same, see SM. See SM for more details on this definition and the analogous decoder definitions.

---

AED
AO
VED
(a)
Enc.
Enc.
(g)
Dec.
Dec.
Dec.
PC2
PC2
PC1
PC
0
-5
-2
-7
0
3
0
7
PC
1
PC 1
PC
PC
(b)
3
(d)
3
(
(h)
Atcestion
PC
0
RO3
PC
3
Dilita Impet
Input: Delta
-3
-3 RO1
-3
-3
3 -3
PC
PC 0'
Figure 2: Summary of attention dynamics on synthetic tasks. (a-f) All three architectures trained on an
N = 3 one-to-one translation task of variable length ranging from 15 to 20. Plots in the top are projected onto the principal components (PCs) of the encoder and decoder temporal components, while those the bottom row are projected onto the PCs of the input components. (a) For AED, the path formed by the temporal components of the encoder (orange) and decoder (purple), joر and μa, We denote the first and last temporal component by square and star, respectively, and the color of said path lighter for earlier times. The inset shows the softmaxed aligument scores for μt, 44ū, which find to be good approximation the full alignment for the one-to-one translation task. (b) The input-delta components of encoder (light) and decoder (dark) colored by word (see labels). The encoder input components, X are represented by dark colored 'X'. The solid lines are the readout vectors (see labels on (d)). Start/end of sentence characters are in purple. (c, d) The same plots for the AO network. (e, f) The same plots for the VED network (with attention inset). (g) task with temporally reversed output sequence. (h) Attention matrices
Temporal components for the for test example from network trained to alphahetically sort a list of letters. Clockwise from top left, the softmaxed attention from the full hidden states (h. hy), temporal components only (uu 4F)) decoder input components and encoder delta components (x) Ah,), and decoder delta components and encoder input components (Ah" x
will be useful to denote the encoder and decoder input components by X and xy- with and y respectively running over all input and output words (e.g. Xyes and X ). We can then write any hidden state as
h =
Ah, h-= + Ah,
(2)
with Ah, h- 1r x and Ah, = h- R.P -x, the delta components of encoder and decoder hidden states, respectively. Intuitively, we are simply decomposing each hidden state vector sum of component that only varies with time/position in the sequence (independent of input), a component that only varies with input (independent of position), and whatever else is left over. Finally, we will often refer hidden states without their temporal component, i.e. XE Ah, and xD Ah so for brevity we refer to these combinations as the input-delta components.
Using the temporal and input components in 2p. we can decompose the attention alignment between two hidden states as
O,t (12 x Ah?) (n. + X Aht)
We will show below that in certain cases several of the nine terms of this expression approximately vanish, leading to simple and interpretable attention mechanisms.
3 One-to-One Results
To first establish basis of how each of the three architectures learn to solve tasks and the role of their input and temporal components, we start by studying their dynamics for a synthetic one-to-one translation task. The task is to convert sequence of input words into a corresponding sequence of output words, where there a one-to-one translation dictionary, e.g. converting a sequence of letters to their corresponding position in the alphabet, (B,A, A, C, A. D) + 12, 1, 1,4 We generate the

---

input phrases to have variable length, but outputs always have equal length to their input (i.e. T S). While a solution to this task is trivial, it not obvious how each neural network architecture will solve the task. Although this is a severely simplified approximation to realistic tasks, we will show below that many of the dynamics the AED and AO networks learn on this task are qualitatively present in more complex tasks.
Encoder-Decoder with Attention. After training the AED architecture, we apply the decomposi- tion of (2) to the hidden states. Plotting the temporal components of both the encoder and decoder, they each form an approximate circle that is traversed as their respective inputs are read in (Fig. 2 Additionally, we find the encoder and decoder temporal components are closest to alignment when t. We also plot the input components of the encoder and decoder together with the encoder input-delta components, i.c. X Ah,, and the network's 's readout vectors (Fig. 2 We see for the encoder hidden states, the input-delta components are clustered close to their respective input components, meaning for this task the delta components are negligible. Also note the decoder input-delta components are significantly smaller in magnitude than the decoder temporal components. Together, this means we can approximate the encoder and decoder hidden states as h f +x and h. x po. respectively. Finally, note the readout vector for a given output word aligns with the input components of its translated input word, e.g. the readout for 'I' aligns with the input component for 'A' (Fig. 2b) 6
For the one-to-one translation task. the network learns an approximately diagonal attention matrix, meaning the decoder at time s primarily attends to the encoder's 's hidden state at s. Additionally, we find the temporal and input-delta components to be close to orthogonal for all time steps, which allows the network's attention mechanism to isolate temporal dependence rather than input dependence. Since we can approximate the hidden states as h = p. + E and h2 ~ nd, and the temporal and input components are orthogonal, the alignment in (3) can be written simply as Ogt s μ μe.
This means that the full attention is completely described by the temporal components and thus input-independent (this will not necessarily be true for other tasks, as we will see later).
With the above results, we can understand how AED solves the one-to-one translation task. After reading a given input, the encoder hidden state is primarily composed of an input and temporal component that are approximately orthogonal to one another, with the input component aligned with the readout of the translated input word (Fig. pp). The decoder hidden states are approximately made up of only temporal component, whose sole job is to align with the corresponding encoder temporal component. Temporal components of the decoder and encoder are closest to alignment for s, so the network primarily attends to the encoder state h. The alignment between encoder input components and readouts yields maximum logit values for the correct translation.
Attention Only. Now we turn to AO architecture, which is identical to AED except with the recurrent connections cut, and positional encoding added to the inputs. We find that AO has qualitatively similar temporal components that give rise to diagonal attention (Fig. 2b) and the input components align with the readouts (Fig. Dd... Thus AO solves the task in a similar manner as AED. The only difference is that the temporal components, driven by RNN dynamics in AED, are now driven purely by the positional encoding AO.
Vanilla Encoder-Decoder. After training the VED architecture, we find the encoder and decoder hidden states belonging to the same time step form clusters, and said clusters are closest those corresponding to adjacent time steps. This yields temporal components that are close to one another for adjacent times, with f next to su (Figs. 2p. Since there is attention in this architecture, there is no incentive for the network to align temporal components of the encoder and decoder as we saw in AED and AO.
"Here and in plots that follow, we plot the various components using principal component analysis (PCA) projections simply as convemient visualization tool. Other than observation that in some cases the temporal/input components live in a low-dimensional subspace, none of our quantitative analysis is dependent upon the PCA projections. For all one-to-one plots, a large percentage (> 90%) of the variance is explained by the first or 3 PC dimensions.
'For N possible input words, the encoder input components align with the vetrices of an (N 1) simplex, which is similar to the classification behavior observed in Aitken et al. (2020).
"Since in AED we pass both the decoder hidden state and the context vector to the readout, each readout vector is twice the hidden state dimension. We plot only the readout weights corresponding to the context vector, since generally those corresponding to the decoder hidden state negligible, see SM for more details.
5

---

AED
A
(e)
6
Decoder
Enc.
UURN
RUM
LTURN
WALK
WALK
JUMP
UIMP
PC
Dec.
UURN
BUM
LTURN
WALK
WALK
JUMP
JUMP
PC2
Dec.
0
run
run
lot
lot
Encoder
and
and
walk
let
PC 0
lit
PC
RUN
LTURN
and
PC
jump
and
PC
WALK
twice
JUMP
twice
<.>
-4-
LEUS
PC 0'
Figure 3: Summary of dynamics for AED and AO architectures trained on eSCAN. (a) Example attention matrix for the AED architecture. (b) AED network's temporal components, with the inset showing the attention matrix from said temporal components. Once again, encoder and decoder components are orange and purple, respectively and we are projecting onto the temporal component PCs. (c) AED network's input-delta components, input components, and readouts. all colored by their corresponding input/output words (see labels). All quantities projected onto input component PCs. (d, f) The same plots for AO.
As recurrence is the only method of transferring information across time steps, encoder and de- coder hidden states must carry all relevant information from preceding steps. Together, this results in the delta components deviating significantly more from their respective input components for VED relative to AED and AO (Fig. 2D. That is, since hidden states must hold the information of inputs/outputs for multiple time steps, we cannot expect them to be well approximated by μ. x because, by definition, it is agnostic to the network's 's inputs at any time other than t (and similarly for f. + x As such, the temporal and input component decomposition gains us little insight into the inner workings of the VED architecture. Additional details of the VED architecture dynamics are discussed in the SM.
Additional Tasks. In this section, we briefly address how two additional synthetic tasks can be understood using the temporal and input component decomposition. First, consider a task identical to the one-to-one task, with the target sequence reversed in time, e.g. {B, A, C, D) For this task, we expect an attention matrix that is anti-diagonal (i.e. is nonzero for t 5 s). For the AED and AO networks trained on this task, we find their temporal and input component behavior be identical to the original one-to-one task with one exception: instead of the encoder and decoder temporal components following one another, we find one trajectory is flipped in such way as to yield an anti-diagonal attention matrix (Fig. Zg. That is, the last encoder temporal component is aligned with the first decoder temporal component and vice versa.
Second, consider the task of sorting the input alphabetically, e.g. (B,C,A, + (, D, For this example, we expect the network to learn an input-dependent attention matrix that correctly permutes the input sequence. Since there no longer correlation between input and output sequence location, the average attention matrix is very different from that of a random sample. and so we expect the temporal components insignificantly contribute to the aligmment. Indeed, we find nd! μ be negligible, and instead Ah, Xx dominates the alignment values (Fig. 2h).
4 Beyond One-to-One Results
In this section we analyze the dynamics of two tasks that have "close-to-diagone attention: (1) what we refer to as the extended SCAN dataset and (2) translation between English and French phrases. Since dynamics, our focus in this section will be on only the AED and AO architectures. For both tasks we explore below, parts of the picture established the one-to-one task continues to hold. However, we will see that in order to succeed at these tasks, both AO and AED must implement additional mechanisms on top of the dynamics
found temporal/input component decomposition provide little insight into VED
for the one-to-one task.
Extended SCAN (eSCAN) is a modified version of the SCAN dataset (Lake Baroni 2018). in which randomly concatenate subset of the phrases to form phrases of length 15 to 20 (see SM for

---

les
numéros
du
lata
viking
mercredl
-25
marg
2015
Hoteries
<eos>>
Enc.
Hi
lles
dee
pour
dens
uun
una
que
est
sur
viking
PC
Encoder
Dec.
the
lotto
numbers
wednesday
25
-2
Decoder
for
2015
-3
0
3
with
lotteries
PC
1
on
Figure 4: Summary features for AO trained English French translation. (a) Sample attention matrix. (b) The encoder (orange) and decoder (purple) temporal components, with square and star marking the first and last time step, respectively. Once again, quantities are projected onto the temporal component PCs. The inset shows the attention matrix from the temporal components, i.e. the softmax of μu. μe.- (c) The dot product between the most common output word readouts and the most common input word input components, X2
details). The eSCAN tasks is close to one-to-one translation, but is augmented with several additional rules that modify its structure. For example, a common pair of outputs can swap order relative to their corresponding inputs: the English words 'green field' translate 'champ vert' in French (with 'field' <> 'champ' and 'green' +> 'vert'). ). This behavior present in eSCAN: when the imput word 'left' follows verb the output command must first turn the respective direction and then perform said action (c.g. 'run left' + 'LTURN RUN').
structure is that
The AED and AO models both achieve 2 98% word accuracy on eSCAN. Looking at sample attention matrix of AED, we see consecutive words the output phrase tend to attend to the same encoder hidden states at the end of subphrases in the input phrase (Fig. 33). Once again decomposing the AED network's 's hidden states as in (2. we find the temporal components of the encoder and decoder form curves that mirror one another, leading to an approximately diagonal attention matrix (Fig-Bb). The delta components are significantly less negligible for this task, as evidence by the fact x + Ahद aren't nearly as clustered around their corresponding input component (Fig. 3b). As we will verify later, this is a direct result of the network's 's use of recurrence, since now hidden states earry information about subphrases, rather than just individual words.
Training the AO architecture on eSCAN, we also observe non-diagonal attention matrices, but in general their qualitative features differ from those of the AED architecture (Fig. 3 ft). Focusing on the subphrase mapping 'run twice' - 'RUN RUN', we see the network learns to attend to the word preceding "twice', since it can no longer rely on recurrence to carry said word's identity forward. Once again, the temporal components of the encoder and decoder trace out paths that roughly follow one another (Fig. 3e). We see input-delta components cluster around their corresponding input components, indicating the delta components are small (Fig. 3 Finally, we again see the readouts of particular outputs align well with the input components of their corresponding input word.
English to French Translation is another example of nearly-diagonal task. We train the AED and AO architectures on this natural language task using a subset of the para_ carraa_ralllll dataset Bañón al. 2020) consisting of over 30 million parallel sentences. To aid interpetation, we tokenize each sentence at the word level and maintain a vocabulary of 30k words in each language; we train on sentences of length up to 15 tokens.
Since English and French are syntactically similar with roughly consistent word ordering, the attention matrices are in general close diagonal (Fig. D). Again, note the presence of features that require off-diagonal attention, such as the flipping of word ordering in the input/output phrases and multiple words in French mapping to a single English word. Using the decomposition of (2). the temporal components in both AED and AO continue to trace out similar curves (Fig. 4b). Notably, the
alignment resulting from the temporal components significantly less diagonal, with the diagonal behavior clearest at the beginning of the phrase. Such behavior makes sense: the presence of off- diagonal structure means, on average, translation pairs become increasingly offset the further one moves into a phrase. With offsets that increasingly vary from phrase to phrase, the network must rely less on temporal component alignments, which by definition are independent we see that the the dot product between the input components and the readout vectors implement the translation dictionary, just as it did for the one-to-one task (Fig. 4c, see below for additional discussion).
the inputs. Finally,
7

---

(a) 1.0
(b)
12
10a
(d)
run
walk
jump
left
and
twice
<,>
DeC. Time
Var. Ratio
10
RUN
0.5
Align
WALK
10
JUMP
0.0
10
LTURN
AED
AO
Offset
AED
AO
Figure 5: Temporal and imput component features. In the first three plots, the data shown red, blue, and green corresponds to networks trained on the one-to-one, eSCAN, and English to French translation tasks, respectively. (a) Breakdown of the nine terms that contribute to the largest alignment scores (see (3) averaged across the entire decoder sequence for each task/architecturc combination (sce SM for details). For cach bar, from top bottom, the alignment contributions from μ f (dark), μ²= X + u Ah, (medium), and the remaining six terms (light). (b) For the AO architecture. the dot product of the temporal components, uu! 14s as a function of the offset, f shown at different decoder times. Each offset is plotted from [-5. 5] and the dotted lines show the theoretical prediction for maximum offset as function of decoder time, s. Plots for the AED architecture are qualitatively similar.
(c) For all hidden states corresponding variance of hf f to h². (d) For AO trained on eSCAN, the dot product input components, X with each of the readouts (AED is qualitatively similar).
input word, the ratio of
4.1 A Closer Look at Model Features
As expected, both the AED and AO architectures have more nuanced attention mechanisms when trained on eSCAN and translation. In this section, we investigate few of their features in detail.
Alignment Approximation. Recall that for the one-to-one task, we found the alignment scores could be well approximated by ast R 12 fr, which was agnostic to the details of the input sequence. For eSCAN, the ub fe term is still largely dominant, capturing the AED and AO networks (Fig. 12 ur + 1pp X + uh Aht i.e. we include two additional terms on top of what was used for one-to-one. Since Xx and Ah; are dependent upon the input sequence, this means the alignment has non-trivial input dependence, as we would expect. In both architectures, we find this approximation captures > 87% of the top alignment scores. For translation, we see the term μu f4; makes up significantly smaller portion of the alignment scores, and in general we find none of the nine terms in (3) dominate above the rest (Fig. 5p). However, at early times in the AED architecture, we again see u. F. the largest contribution to the alignment. that words at the start of the encoder/decoder phrase have smaller offset from one another than later in the phrase, so the network can rely more on temporal components to determine attention.
77% of ast in A better approximation for the alignment scores is aat RV
mentioned above, this matches our intuition
Temporal Component Offset. For the one-to-one task, the input sequence length was always equal the output sequence length, so the temporal components were always peaked at s (Fig. In eSCAN, the input word 'and' has no corresponding output, which has a non-trivin con-trivial effect on how the network attends since its appearance means later words in input phrase are offset from their corresponding output word. This effect also compounds with multiple occurrences of 'and' in the input. The AED and AO networks learn handle such behavior by biasing the temporal component dot product, 4 1.F the dominant alignment contribution, to be larger for time steps further along in the encoder phrase, i.e. > s (Fig.55) It is possible to compute the average offset of input and output words in eSCAN training set, and we see the maximum of uP u follow this estimate quite well. Similarly, in our set of English to French translation phrases, we find the French phrases to be on average 20% longer than their English counterparts. This results in the maximum of 1uP n to gradually move toward t < 8, c.g. on average the decoder attends to earlier times in the encoder frig.5p), Additionally, note the temporal dot product falls off significantly slower function of offset for later time steps, indicating the drop off for non-diagonal alignments is smaller and thus it is easier for the network to off-diagonally attend.
The encoder hidden states in the one-to-one task had negligible delta component, so the hidden states could be approximated as h. ~ μ. + +xE By definition, X is constant for given input word, so the variance in the hidden states corresponding to given input word is primarily contained in the temporal component (Fig. 5p). Since the temporal component is inpur-independent, this led to a clear understanding of how all of a network's 's hidden states evolve with time and input. In the AED and AO architectures trained on eSCAN, find the variance of the input word's 's hidden
Word Variance.
8

---

(a)
walk
(b)
10
Temp "Jemp
(c)
3
(d)
3
5
PC
Dot
Dot
0
Dot
0
left
twice
Ewice
-5
Temp Delta)
-3
and
-3
0
+3
run walk
Jume left
and twice
Offset
Offset
Figure 6: How AO and AED networks implement off-diagonal attention in the eSCAN dataset. (a) For AED, the input-delta components various words and subphrases. (b) For AO, the alignment values, ast. are shown in black when the input word 'twice' is at f s. Three contributions to the alignment, 4 1 (gold), y.. Xx + n.. Ah, (pink), and aut ue= h (grey) are also plotted.
To keep the offsct between "twice" and the output location of the repeated word constant, this plot was generated but observe the decoder's 's temporal component, ph for 8. (d) How the dot product of X Ah; and μ. changes as a function of their offset, 8, for a few select input words. The vertical gray slice represents the data in (c) and the input word colors are the same.
subset of eSCAN with T = S. qualitative features when T 2 S. (c) The dot product between X Ah, and the
states drops by 90% and 95% when the temporal component is subtracted out, respectively (Fig. 5p). Meanwhile, in translation, we find the variance only drops by 8% and 25% for the AED and AO architectures, indicating there is significant variance in the hidden states beyond the average temporal evolution and thus more intricate dynamics.
Input/Readout Alignment. Lastly, recall we saw that in the one-to-one case the input components' alignment with readouts implemented the translation dictionary (Figs. Pb, d). For eSCAN, the dot product of a given readout is again largest with the input component of its corresponding input word, e.g. the readout corresponding Notably, words that produce no corresponding output such as 'and' and 'twice' are not the maximal in alignment with any readout vector. Similarly, for translation, we see the French-word readouts have the largest dot product their translated English words (Fig. 4c). For example, the readouts for the words 'la', 'le", and 'les', which are the gendered French equivalents of 'the', all have maximal alignments with Xahe
'RUN' is maximal for the input component of 'run' (Fig. (d)).
4.2 A Closer Look at Dynamics
In this section, we leverage the temporal and input component decomposition to take a closer look at how networks trained the eSCAN dataset implement particular off-diagonal attentions. Many of the sequence translation structures in eSCAN are seen in realistic datasets, so we this analysis will give clues toward understanding the behavior of more complicated eqquenceto--sqquence tasks.
common structure in sequence-to-ssqueeee tasks is when an output word is modified by the words preceding it. For example, the phrases 'we run' and 'they run't translate to 'nous courrons' and 'ils courent' in French, respectively (with the second word each the translation of 'run'). We can study this phenomenon in eSCAN since the word 'twice" tells the network repeat the command just issued two times, e.g. 'run twice' outputs to 'RUN RUN'. Hence, the output corresponding to the input 'twice' changes based other words in the phrase.
Since an AED network has recurrence, when it sees the word 'twice" it can know what verb preceded it. Plotting input-delta components, we see the RNN outputs 'twice' hidden states in three separate clusters separated by the preceding word (Fig. (4). Thus for an occurrence of 'twice' at time step f, we have E + AhE x X + AhF_, For example, this means the AED learns to read in 'run twice' approximately the same as 'run run" This is an example of the network learning context.
AO has no recurrence, so it can't know which word was output before 'twice'. Hence, unlike the AED case, all occurrences of 'twice" are the same input-delta component cluster regardless of what word preceded it. Instead, it has to rely on attending to the word that modifies the output, which in this case simply the preceding word (Fig. 3d). As mentioned in Sec. J.I. for the eSCAN task we find the alignment to be well approximated by input phrase, we find pnl X + n. Ah < 0 for s = (Fig. (p)) This decreases the value of the alignment ag.e. and the decoder instead attends to the time step with the second largest value
n1 h. When the word 'twice' appears in the

---

of p2, yu, which the network has learned to be t = s- 1. Hence, 0,,a-1 is the largest alignment, corresponding the time step before 'twice' with the verb the network needs oulput again. Unlike the one-to-one case, the encoder input-delta and the decoder temporal components are no longer approximately orthogonal to one another (Fig. (c). In the case of 'twice', Xiwice + Ah, is partially antialigned with the temporal component, yielding a negative dot product.
This mechanism generalizes beyond the word 'twice": in eSCAN we see input-delta components of several input words are no longer orthogonal to the decoder's 's temporal component (Fig. ( Like 'twice', the dot product of the input-delta component for given word with its corresponding temporal component determines how much its alignment score is increased/decrent Xand + Ah has a negative dot product with the temporal component, meaning it leans away from its corresponding temporal component. Again, this make sense from eSCAN task: the word 'and' has no corresponding output, hence never wants to be attended to by the decoder.
For example, we see
Perhaps contradictory to expectation, Xlen + Ah, has a negative dot product with the temporal component. However, note that the alignment of x + Aht with the h2 dependent on both t and s. function of their offset, defined to be the
We plot the dot products of xF + Ah; and h? (Fig.60) Notably, Xict + Ah; has a larger dot product for larger offsets, meaning it increases its alignment when > This makes sense from the point of view that the word 'left' is always further along in the input phrase than its corresponding output 'LTTRNN, and this offset is only compounded by the presence of the word 'and'. Thus, the word 'left' only wants get noticed if it is ahead of the corresponding decoder time step, otherwise it hides. Additionally, the words 'and' and 'twice' have large negative dot products for all offsets, since they never want be the subject of attention.
Discussion
In this work, we studied the hidden state dynamics of sequence-to-sequee to-sequence tasks architectures with recurrence and attention. We proposed a decomposition of the hidden states into parts that are input- and time-independent and showed when such a decomposition aids in understanding the behavior of encoder-decoder networks.
Although we have started by analyzing translation tasks, it would be interesting to understand how said decomposition works on different eqquence-to-sqqueee tasks, such as speech-to-text. Additionally, with our focus on the simplest encoder-decoder architectures, it is important investigate how much the observed dynamics generalize to more complicated network setups, such as networks with bidirectional RNNs or multiheaded and self-attention mechanisms. Our analysis of the attention- only architecture, which bears resemblance to the transformer architecture, suggests that a similar dynamical behavior may also hold for the Transformer, hinting at the working mechanisms behind this popular non-recurrent architecture.
Acknowledgments and Disclosure of Funding
We thank Ankush Garg for collaboration during the early part of this work. None of the authors receive third parrtpprty funding/support during the 36 months prior to this submission or had competing interests.
References
Aitken, K., Ramasesh, V. V., Garg, A., Cao, Y., Sussillo, D., and Maheswaranathan, N. The geometry of integration in text classification mnss arXiv preprini arXiv:2010. 2020.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and 1409.4009000 0473, 2014.
translate. arXiv preprint arXiv: arXiv:1409.0
Bañón, M., Chen, P., Haddow, B., Heafield, K., Hoang, H., Esplà-Gomis, M., Forcada, M. L., Kamran, A., Kirefu, F., Koehn, P., Ortiz Rojas, S., Pla Sempere, L.,, Ramirez-Sáncher G., Sarrías, E., Strelec, M., Thompson, B., Waites, W., Wiggins, D., and Zaragoza, J. ParaCrawl: Web-scale acquisition of parallel corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4555-4567, Online, July 2020. Association for
10

---

Computational Linguistics. doi: 10.18633w1/2022aclmmmibbb URL https::/www.a.. aclweb org/ htg/anthology// /2000.2200.000 acl-main.......mmaa.aa.aamamma.a maim.miii777.....
Bastings, and Filippova, K. The elephant the interpretability room: Why use attention as explanation when we have saliency methods?, 2020.
Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. Listen, attend and spell, 2015.
Chefer, H., Gur, S., and Wolf, L., Transformer interpretability beyond attention visualization, 2020.
Chiu, C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, Chen, Z., Kannan, A.. Weiss, R. J., Rao, K., Gonina, E., Jaitly, N., Li, B., Chorowski, J., and Bacchiani, M. State-of-the-art models. 2018 IEEE International Conference on
speech recognition with sequence-to-sequene Acoustics, Speech and Signal Processing (ICASSP), pp. 4774-4778 2018. doi: 10.1109/CCASSP. 2018.8462105.
Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1 Abs/1406.100 1078. 2014.
Collins, J., Sohl-Dickstein, J., and Sussillo, Capacity and trainability recurrent neural networks, 2016.
Ding, S., Xu, H., and Koehn, P. Saliency-driven word alignment interpretation for neural machine translation. arXiv preprint arXiv: /906. /0282, 2019.
Ghader, H. and Monz, C. What does attention neural machine translation pay attention to? arXiv preprint arXiv:1710.0334 2017.
Hochreiter, and Schmidhuber, Long short-term memory. Neural Computation, 98)):1335-1780 1997.
Jain, S. and Wallace, B. C. Attention is not explanation, 2019.
Lake, B. and Baroni, M. Generalization without systematicity: On the compositional skills of scquence-to-sequenee recurrent networks. In International Conference on Machine Learning, pp- 2873-2882. PMLR, 2018.
Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention- neural machine translation. arXiv preprint arXiv: 04025, 2015.
Maheswaranathan, N. and Sussillo, D. How recurrent networks implement contextual processing in sentiment analysis. arXiv preprint arXiv:2004.08013, 2020.
Maheswaranathan, N., Williams, A., Golub, M., Ganguli, S., and Sussillo, D. Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. In Advances in Neural Information Processing Systems 32, pp. 15705. Curran Associates, Inc., 2019.
Prabhavalkar, R., Rao, K., Sainath, T., Li, B., Johnson, L., and Jaitly, N. A comparison of sequence- to-sequence models for speech recognition. 2017. URL http://www.isca.t
org/
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.
Exploring the limits of transfer learning with unified text-to-text transformer, 2020.
Serrano, S. and Smith, N. A. attention interpretable? arXiv preprint 1eXiv:1906.0327 2019.
Sutskever, Vinyals, O., and Le, Q. Sequence sequence learning with neural networks. arXiv 2014.
preprint
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E., and Polosukhin, Attention is all you need. Advances in neural information processing systems, pp- 5998-6008, 2017.
Wiegreffe, S. and Pinter, Y. Attention is not not explanation, 2019.

---

Wu, Y., Schuster, M., Chen, Z., Le, Q. V. Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q.. and al., K. M. Google's neural machine translation system: Bridging the gap between human and machine translation, 2016.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL.? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [No]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/]
12


Reviews:
[{"review_id": "wt1Lb7Ivcuq", "reviewer": "Reviewer_bArF", "summary": "In this work, the authors propose a framework to analyze how different encoder-decoder architectures attend in 3 simulated tasks and 2 real-world tasks. The framework relies on isolating the  following components in the network:\n1. Temporal - Average hidden state of the encoder/decoder at a specific timestep, across all the test samples.\n2. Input - Average deviation of a word's hidden state from the mean state at the timepoint it occurs in, across all its occurrences in the test set.\nUsing these components, the paper factorizes the attention alignment between eh encoder & decoder's hidden state into 9 different products of the temporal and input states.\n\nThe usefulness of the two components in understanding attention mechanisms is demonstrated with 3 tasks:\n1. Simulated tasks: Matching 3 characters to their numbered IDs | The authors find that the input and read-out states are highly aligned such that the decoder only needs to find its corresponding temporal match in the encoder. This is apparent from the results in the paper. Further, the non-attention based network doesn't demonstrate similar behavior.\n2. eSCAN: Extension of the SCAN dataset formed by concatenating commands to generate longer sequences\n3. English $\\rightarrow$ French translation\n\nFor tasks 2 & 3, the authors find that the networks learn mappings between the input tokens and corresponding actions/French words in the readout layer. Further, the networks attend through a combination of temporal information and input components unlike task 1. Since the output in several cases doesn't depend on just 1 timepjoint in the input, these networks also exhibit 'offset' behavior wherein they attend to information at different time points depending on the input token (thus violating their orthogonality)\n\nOverall, by separating the input and temporal components of the network state, the paper reveals if networks use only one mechanism or a combination to attend and solve a task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post discussion: I have read the author responses to both my comments and those of the other reviewers. The authors have highlighted the complexity of explaining attention mechanisms in large architectures and other real-world tasks. Overall, the proposed decomposition seems like a promising approach to study attention in encoder-decoder architectures and I would be interested in seeing how it can be used to improve model performance. I stand by my score.\n\n###################################\n\nOverall, I find this direction of research interesting and enjoyed reading several analyses in the paper. However, the current version of the paper lacks clarity and organization that could/should highlight main findings for each task & architecture and also highlight the usefulness of the component characterization beyond existing work (visualizing attention, attribution methods etc.). My confidence in the paper's merits would increase if the authors could point at a clear takeaway on how the networks behave. In its current form, it is hard to gauge the framework's usefulness for more complicated tasks that don't necessarily exhibit local alignment or easy 1-1 correspondence between tokens.\n\n- While the SM contains clear definitions, lines 94-100 are confusingly worded despite being central to the paper. I’d recommend moving equations from SM to main text for readability. For instance, just from the main text it was not evident to me that $\\mu_t^E$ was averaged across the test samples.\n- Fig. 2:\n    - There is no mention of PCA being used or why 2 PCs explain enough variance in the data that we can rely on them. \n    - Why only 3 tokens?\n    - What is the takeaway from the result that they traverse a circular path? It should be explicitly stated (in addition to highlighting it much later on Pg-5). I agree that they have _similar_ patterns across time, but it would be helpful to visualize in a 3D space (see Cai et al., ICLR 2021 for example) It is hard to discern if the patterns are similar because of the dimensionality or the learned structure.\n    - What is the purple ‘X’ in the input component visualization?\n    - There is a singular mention to ‘readout’ layers in Fig. 1 caption and no mention of it again, except directly in Fig. 2. From the footnote (and not main text) it is apparent that the readout vector is the weights of that layer on the context vector. If the readout is also a point in the space, depicting it as a line is confusing. In my opinion, it would be easier to understand the structure in the lower panels if everything was visualized as a dot/blob.\n    - Line 137: It would be helpful to elaborate on why the temporal and input terms are orthogonal.\n    - Overall, some of the explanation is missing/confusing, but the characterization of network behavior in Lines 140-147 is interesting!\n    - Clarify the attention matrix headings for row 2 in Fig 1h.\n- Fig. 3 & 4 & 5:\n    - It would be helpful to demonstrate attentional differences between the two architectures in Fig. 3 by using the same input sequence.\n    - Add discussion on Fig 4c to main text.\n    - Overall, I think the visualizations are interesting but I’m afraid I don’t understand how they inform our understanding of the network beyond visualizing attention matrices. In other words, can the authors elaborate on what splitting the network representations into temporal and input components tells us in addition to attention matrices?\n- Where are the details of this?\n\t> It is possible to determine the average theoretical offset of input and output words in eSCAN\n- I agree with several inferences made in the real-world experiments regarding the nature of attention and the use of temporal information. But if we were looking at a task that didn’t have local alignment like in eSCAN or English$\\rightarrow$French, it would be much harder to interpret under the proposed framework. As a simple example, it is not apparent to me what Fig. 4b vs. its inset indicate. One says that the temporal components are aligned in some fashion from start to end but the softmax shows that the network is relying on temporal components farther in the past as the translations get ‘offset’. What does this tell us about how the network solves the task? How is this building on our understanding beyond say, visualizing raw attention values as a heat map and averaging across samples in a test set? While these inferences fit into a clear possible explanation of the network in the simulated task, it seems a much harder ask in the English-to-French translation settings. I would appreciate if the authors can clarify the usefulness of the proposed temporal & input components framework given the intuitive results that in a non one-to-one task that relies on multi-word information like translation, the input plays a significant role and the network doesn’t rely on pure temporal alignment. Having said that, I found the characterization in Fig. 5 interesting and would be excited about future directions that can provide more insight into the model.\n- I extend my comments on how interpretation may still be hard with this setup for more complex tasks. While the analysis on ‘twice’ in Section 4.2 throws light on how the input and temporal components both play a role to output required behavior (and do this differently between recurrent and position-encoding based architectures), I disagree that the analysis is equivalent to and as simple as the translation case. While _run twice_$\\rightarrow$`RUN RUN` only needs to attend to the past word and copy it over, there could be examples of translation wherein there is a paraphrase/ difference in case based on meaning etc. wherein the one-to-one mapping is lost like in _we_$\\rightarrow$_nous_ or _ils_$\\rightarrow$_they_. To this end, what is the scope of this method?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss any limitations or societal impact of their approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "qEWev2_hdw0", "reviewer": "Reviewer_GYkA", "summary": "This paper proposes a novel method to decompose hidden representations of encoder-decoder architectures into two kinds of components: temporal and input-driven. The temporal component is the average hidden representation at a given time step thus it only varies with time, while the input-driven component is defined as the representation of a specific input word removing temporal component thus it is input-dependent but time-independent. Based on the decomposition, the authors show how encoder-decoder architectures attend, providing some insights for future research.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think the proposed decomposition method is very interesting and intuitive, which provides a new way for understanding model behaviors of (attention-based) sequence-to-sequence learning models. However, I think it still has the following weaknesses:\n\n***1.\tThe studied architectures are unrepresentative.***\nIn this paper, the authors study the architectures of the recurrence-based encoder-decoder architecture without any attention (VED), the attention-based encoder-decoder architecture (AED), and the attention-based encoder-decoder architecture without any recurrence (AO). However, the most widely used Transformer architecture has not been studied. The authors claim that AO bears resemblance to Transformer, but I think both the analyses and conclusions will be significantly changed in Transformer due to its much stronger layer-to-layer abstraction. The analyses and conclusions on Transformer might make the proposed decomposition method more convincing and more widely used, bringing more benefits to the community. Therefore, I think that the experiments on Transformer should be indispensable. The audiences will be less interested in a paper on sequence-to-sequence learning without Transformer experiments.\n\n***2.\tThe studied tasks are unrepresentative.***\nThis paper only studies some very simple sequence-to-sequence learning tasks, thus a question is raising that do the contributions listed in the introduction part can really inspire current research on sequence-to-sequence learning, like machine translation, question answering, and text summarization? Do the observations of the temporal and input components really exist in the more complicated task? Although the authors have conducted experiments on the English-French translation task, the uncommon settings (e.g., only training on sentences of length up to 15 tokens) make it less convincing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper should focus more on mainstream sequence-to-sequence learning architectures and tasks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "PXQTsAoFARs", "reviewer": "Reviewer_mZex", "summary": "1. Motivation: The paper focuses on understanding the behavior of networks with attention and investigating how encoder-decoder networks solve different sequence-to-sequence tasks. To this end, the authors hypothesize that representations (i.e., hidden states) can be decomposed into \"temporal\" and \"input\" components.\n\n2. Approach: In detail, the author presents a decomposition of the hidden states of encoder/decoder into temporal (independent of input) and input-driven (independent of sequence position) parts, where the former is used to explain the temporal behavior of the network and the latter is used to describe the input behavior. \n\n3. Experiments: This paper conducts extensive experimental studies on several common sequence-to-sequence frameworks, including vanilla RNN, RNN with attention, and simplified Transformer networks. They study the sequence-to-sequence networks on both synthetic and real-world tasks, i.e., a synthetic task, eSCAN and English-French translation tasks, and find that the temporal components contribute more when computing the attention.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well written. The studied problem is interesting and important. Considering the attention-based encoder-decoder networks have been widely used for sequence-to-sequence problems, I think this paper involves an interesting topic, aiming to understand the mechanism behind the attention-based encoder-decoder networks.\n\n2. The paper gives a very detailed analysis of various aspects of the input and temporal components, and observes some interesting phenomena regarding how the attention patterns are formed by the hidden state dynamics under different tasks.\n\n3. The experiments, consisting of both synthetic and real-world tasks, are extensive.\n\n\nWeaknesses:\n1. It's not clear what generalizable knowledge we should take away from the fact that in tasks where attention *can* be computed without looking at input content, attention *is* computed without looking at input content.\n\n2. The three tasks used for experiments and analyses are relatively simple. Most real-world tasks are not like this.\n- First, among the three tasks, the English-French phrase translation is the most complex, but most experimental analyses are concentrated on the eSCAN task. It's interesting to see the analysis of hidden state dynamics can discover that the attention for the one-to-one task is only driven by temporal components, and for eSCAN, some input components such as \"twice\" also contribute to the formation of the attention. But it is not surprising given the simplistic nature of these two tasks, and it is not clear what indication it might have over more realistic tasks. It would be better if more analysis is done on the phrase translation tasks (especially the sequence-to-sequence networks are applied to machine translation tasks mostly), and some discussion on how the analysis could generalize to other tasks would be nice. Therefore, I recommend the authors concentrate more on the analyses of more MT tasks in addition to the English-French phrase translation.\n- Second, for the English-French phrase translation task, the delta component of the hidden states becomes relatively large. I am concerned that the validity and usefulness of the \"input\"/\"temporal\" analysis if the delta dominates the hidden states when applied to more realistic tasks. Besides, I suggest the authors report BLEU scores for the translation task so that readers can check and understand the performance of the trained translation models.\n- More analyses on the complicated tasks would significantly add support to your claims.\n\n3. Although extensive experimental results are provided, it is hard for me to conclude the main contribution of this paper. The authors do have summed up some conclusions, such as the temporal component contributes more to the attention, which, however, does not lead to some deeper insights. For example, I would expect that the findings of this paper could be utilized to improve the sequence-to-sequence networks and promote their performance. The paper only provides some intuitive conclusions to the sequence-to-sequence networks, without taking a step forward and proposing some novel improvements to them. Can we get any benefits from the observation, e.g., guiding us to design better models?\n\n4. Some parts of the paper can be made clearer for easier reading. \n- The definition of temporal and input components is essential in understanding the paper, and it should be in the main paper, not in the supplementary material. \n- The axis in Figure 2 should be explained in the paper, even if the reader might be able to guess its meaning. \n- It would be better if there is an outline paraphrase at the end of the first section.\n- The authors should report BLEU scores for the translation task so that readers can check and understand the performance of the trained translation models.\n\nOverall:\nI think this is a promising start, but the paper would benefit from both a clearer statement of the generalizable machine learning knowledge, fewer monotonic tasks, and a more detailed discussion of how to interpret findings with large \\Delta. Considering the new insights and observations, I would like to give a positive suggestion for this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "-vGWELr_d0", "reviewer": "Reviewer_pVaT", "summary": "The paper introduces a novel method for understanding how attention works in sequence alignment problems by decomposing the hidden states into temporal and input-driven contributions.  This is applied to three different network architectures, with AO only recurrent connection, AED with both recurrent connections and attention, and AO, with only attention.  They then test on some different synthetic dataset and link the attention matrix to the hidden state decomposition and to the temporal and input component dynamics.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The analysis and decomposition appear to be novel, the paper is cites relevant papers and contrast their contributions to their analyses.\n\nImpact:  Understanding how neural networks attend could have a suggest methods for improving performance and increasing interpretability of neural networks with attention.  Some of the main insights seem to be that the AED attention can be approximated by looking at the AO attention and assuming an implicit attention is present on the previous local elements in the sequence.  Another insight is that AED behaves closely to AO as opposed to behaving more like the VED, but it is not explored in more depth why this is more efficient.  The results and analysis are thorough and confirm that the networks are solving the synthetic/translation problems in the potentially expected manner given the dataset setup.  However, the insights provided by the analysis don't suggest further enhancements to how to improve attention and how networks are designed, with would increase the papers impact.  \n\nClarity:  Paper is generally well written, but there are some minor issues with the figures.  In particular, Figures 3 and 4 would be easier to compare if the same example was used for both systems.  It needs detailed study by the reader to see the differences in attention in the systems due to needing to map the different attention system to different problems.  \n\nThe presentation of the results (in particular on the synthetic sections) could end up being easier to follow if the results on the simpler architecture were introduced first, namely AO not AED, as it is a simpler system.\n\nLine 279 is missing \"with\"\nLine 289 in supplementary has a typo\n\nQuality: The submission appears to be technically sound, with sufficient technical detail to replicate the experiments, and the claims are well supported.  The paper does a good job of making the different architectures comparable, and the investigation into the hidden state is well visualized in various manners in the figures.  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative social impacts are addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three of four reviewers rated this paper as a 6, one reviewer still assesses this paper as being below threshold.
The concerns that remain relate to the general applicability of the method, but I think the work is in good enough shape for a poster.
The AC recommends accept as poster.

Decision: Accept (Poster)