Understanding How Encoder-Decoder Architectures Attend

1 Introduction . Modern machine learning encoder-decoder architectures can achieve strong performance on sequenceto-sequence tasks such as machine translation ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Wu et al. , 2016 ; Vaswani et al. , 2017 ) , language modeling ( Raffel et al. , 2020 ) , speech-to-text ( Chan et al. , 2015 ; Prabhavalkar et al. , 2017 ; Chiu et al. , 2018 ) , etc . Many of these architectures make use of attention ( Bahdanau et al. , 2014 ) , a mechanism that allows the network to focus on a specific part of the input most relevant to the current prediction step . Attention has proven to be a critical mechanism ; indeed many modern architectures , such as the Transformer , are fully attention-based ( Vaswani et al. , 2017 ) . However , despite the success of these architectures , an understanding of how said networks solve such tasks using attention remains largely unknown . Attention mechanisms are attractive because they are interpretable , and often illuminate key computations required for a task . For example , consider neural machine translation—trained networks exhibit attention matrices that align words in the encoder sequence with the appropriate corresponding position in the decoder sentence ( Ghader & Monz , 2017 ; Ding et al. , 2019 ) . In this case , the attention matrix already contains information about which words in the source sequence are relevant for translating a particular word in the target sequence ; that is , forming the attention matrix itself 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constitutes a significant part of solving the overall task . How is it that networks are able to achieve this ? What are the mechanisms underlying how networks form attention , and how do they vary across tasks and architectures ? In this work , we study these questions by analyzing three different encoder-decoder architectures on sequence-to-sequence tasks . We develop a method for decomposing the hidden states of the network into a sum of components that let us isolate input driven behavior from temporal ( or sequence ) driven behavior . We use this to first understand how networks solve tasks where all samples use the same attention matrix , a diagonal one . We then build on that to show how additional mechanisms can generate sample-dependent attention matrices that are still close to the average matrix . Our Contributions • We propose a decomposition of hidden state dynamics into separate pieces , one of which explains the temporal behavior of the network , another of which describes the input behavior . We show such a decomposition aids in understanding the behavior of networks with attention . • In the tasks studied , we show the temporal ( input ) components play a larger role in determining the attention matrix as the average attention matrix becomes a better ( worse ) approximation for a random sample ’ s attention matrix . • We discuss the dynamics of architectures with attention and/or recurrence and show how the input/temporal component behavior differs across said architectures . • We investigate the detailed temporal and input component dynamics in a synthetic setting to understand the mechanism behind common sequence-to-sequence structures and how they might differ in the presence of recurrence . Related Work As mentioned in the introduction , a common technique to gain some understanding is to visualize learned attention matrices , though the degree to which such visualization can explain model predictions is disputed Wiegreffe & Pinter ( 2019 ) ; Jain & Wallace ( 2019 ) ; Serrano & Smith ( 2019 ) . Input saliency Bastings & Filippova ( 2020 ) and attribution-propagation Chefer et al . ( 2020 ) methods have also been studied as potential tools for model interpretability . Complementary to these works , our approach builds on a recent line of work analyzing the computational mechanisms learned by RNNs from a dynamical systems perspective . These analyses have identified simple and interpretable hidden state dynamics underlying RNN operation on text-classification tasks such as binary sentiment analysis ( Maheswaranathan et al. , 2019 ; Maheswaranathan & Sussillo , 2020 ) and document classification ( Aitken et al. , 2020 ) . Our work extends these ideas into the domain of sequence-to-sequence tasks . Notation Let T and S be the input and output sequence length of a given sample , respectively . We denote the encoder and decoder hidden states by hEt ∈ Rn with t = 1 , . . . , T . Similarly , we denote decoder hidden states by hDs ∈ Rn , with s = 1 , . . . , S. The encoder and decoder hidden state dimensions are always taken to be equal in this work . Inputs to the encoder and decoder are denoted by xEt ∈ Rd and xDs ∈ Rd̃ . When necessary , we subscript different samples from a test/train set using α , β , γ , e.g . xEt , α for α = 1 , . . . , M . Outline We begin by introducing the three architectures we investigate in this work with varying combinations of recursion and attention . Next we introduce our temporal and input component decomposition and follow this up with a demonstration of how such a decomposition allows us to understand the dynamics of attention in a simple one-to-one translation task . Afterwards , we apply this decomposition to two additional tasks with increasing levels of complexity and discuss how our decomposition gives insight into the behavior of attention in these tasks . 2 Setup . A schematic of the three architectures we study is shown in Fig . 1 ( see SM for precise expressions ) . Vanilla Encoder Decoder ( VED ) is a recurrent encoder-decoder architecture with no attention ( Sutskever et al. , 2014 ) . The encoder and decoder update expression are hEt = FE ( h E t−1 , x E t ) and hDs = FD ( h D s−1 , x D s ) , respectively . Here , FD and FE are functions that implement the hidden state updates , which in this work are each one of three modern RNN cells : LSTMs ( Hochreiter & Schmidhuber , 1997 ) , GRUs ( Cho et al. , 2014 ) , or UGRNNs ( Collins et al. , 2016 ) . Encoder-Decoder with Attention ( AED ) is identical to the VED architecture above with a simple attention mechanism added ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) . For time step s of the decoder , we compute a context vector cs , a weighted sum of encoder hidden states , cs : = ∑T t=1 αsth E t , with αt : = softmax ( a1t , . . . , aSt ) the tth column of the attention matrix and ast : = hDs ·hEt the alignment between a given decoder and encoder hidden state . While more complicated attention mechanisms exist , in the main text we analyze the simplest form of attention for convenience of analysis.1 Attention Only ( AO ) is identical to the AED network above , but simply eliminates the recurrent information passed from one RNN cell to the next and instead adds fixed positional encoding vectors to the encoder and decoder inputs ( Vaswani et al. , 2017 ) . Due to the lack of recurrence , the RNN functions FE and FD simply act as feedforward networks in this setting.2 AO can be treated as a simplified version of a Transformer without self-attention , hence our analysis may also provide a hint into their inner workings ( Vaswani et al. , 2017 ) . 2.1 Temporal and Input Components . In architectures with attention , we will show that it is helpful to write the hidden states using what we will refer to as their temporal and input components . This will be useful because each hidden state has an associated time step and input word at that same time step ( e.g . s and xDs for h D s ) , therefore such a decomposition will often allow us to disentangle temporal and input behavior from any other network dynamics . We define the temporal components of the encoder and decoder to be the average hidden state at a given time step , which we denote by µEt and µ D s , respectively . Similarly , we define an encoder input component to be the average of all hEt − µEt for hidden states that immediately follow a given input word . We analogously define the decoder input components . In practice , we estimate such averages using a test set of size M , so that the temporal and input components of the encoder are respectively given by µEt ≈ ∑M α=1 1≤EoS , αh E t , α∑M β=1 1≤EoS , β , χE ( xt , α ) ≈ ∑M β=1 ∑T t′=1 1xt , α , xt′ , β ( hEt′ , β − µEt′ ) ∑M γ=1 ∑T t′′=1 1xt , α , xt′′ , γ , ( 1 ) where hEt , α the encoder hidden state of the αth sample , 1≤EoS , α is a mask that is zero if the αth sample is beyond the end of sentence , 1xt , α , xt′ , β is a mask that is zero if xt , α 6= xt′ , β , and we have temporarily suppressed superscripts on the inputs for brevity.3 By definition , the temporal components only vary with time and the input components only vary with input/output word . As such , 1In the SM , we implement a learned-attention mechanism using a scaled-dot product attention in the form of queries , keys , and value matrices ( Vaswani et al. , 2017 ) . For the AED and AO architectures , we find qualitatively similar results to the simple dot-product attention presented in the main text . 2We train non-gated feedforward networks and find their dynamics to be qualitatively the same , see SM . 3See SM for more details on this definition and the analogous decoder definitions . it will be useful to denote the encoder and decoder input components by χEx and χ D y , with x and y respectively running over all input and output words ( e.g . χEyes and χ D oui ) . We can then write any hidden state as hEt = µ E t + χ E x + ∆h E t , h D s = µ D s + χ D y + ∆h D s , ( 2 ) with ∆hEt : = h E t − µEt − χEt and ∆hDs : = hDs − µDs − χDy the delta components of encoder and decoder hidden states , respectively . Intuitively , we are simply decomposing each hidden state vector as a sum of a component that only varies with time/position in the sequence ( independent of input ) , a component that only varies with input ( independent of position ) , and whatever else is left over . Finally , we will often refer to hidden states without their temporal component , i.e . χEx + ∆h E t and χDy + ∆h D s , so for brevity we refer to these combinations as the input-delta components . Using the temporal and input components in ( 2 ) , we can decompose the attention alignment between two hidden states as ast = ( µDs + χ D y + ∆h D s ) · ( µEt + χ E x + ∆h E t ) . ( 3 ) We will show below that in certain cases several of the nine terms of this expression approximately vanish , leading to simple and interpretable attention mechanisms .

Summary: In this work, the authors propose a framework to analyze how different encoder-decoder architectures attend in 3 simulated tasks and 2 real-world tasks. The framework relies on isolating the  following components in the network: 1. Temporal - Average hidden state of the encoder/decoder at a specific timestep, across all the test samples. 2. Input - Average deviation of a word's hidden state from the mean state at the timepoint it occurs in, across all its occurrences in the test set. Using these components, the paper factorizes the attention alignment between eh encoder & decoder's hidden state into 9 different products of the temporal and input states.  The usefulness of the two components in understanding attention mechanisms is demonstrated with 3 tasks: 1. Simulated tasks: Matching 3 characters to their numbered IDs | The authors find that the input and read-out states are highly aligned such that the decoder only needs to find its corresponding temporal match in the encoder. This is apparent from the results in the paper. Further, the non-attention based network doesn't demonstrate similar behavior. 2. eSCAN: Extension of the SCAN dataset formed by concatenating commands to generate longer sequences 3. English $\rightarrow$ French translation  For tasks 2 & 3, the authors find that the networks learn mappings between the input tokens and corresponding actions/French words in the readout layer. Further, the networks attend through a combination of temporal information and input components unlike task 1. Since the output in several cases doesn't depend on just 1 timepjoint in the input, these networks also exhibit 'offset' behavior wherein they attend to information at different time points depending on the input token (thus violating their orthogonality)  Overall, by separating the input and temporal components of the network state, the paper reveals if networks use only one mechanism or a combination to attend and solve a task.