Understanding How Encoder-Decoder Architectures Attend

1 Introduction . Modern machine learning encoder-decoder architectures can achieve strong performance on sequenceto-sequence tasks such as machine translation ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Wu et al. , 2016 ; Vaswani et al. , 2017 ) , language modeling ( Raffel et al. , 2020 ) , speech-to-text ( Chan et al. , 2015 ; Prabhavalkar et al. , 2017 ; Chiu et al. , 2018 ) , etc . Many of these architectures make use of attention ( Bahdanau et al. , 2014 ) , a mechanism that allows the network to focus on a specific part of the input most relevant to the current prediction step . Attention has proven to be a critical mechanism ; indeed many modern architectures , such as the Transformer , are fully attention-based ( Vaswani et al. , 2017 ) . However , despite the success of these architectures , an understanding of how said networks solve such tasks using attention remains largely unknown . Attention mechanisms are attractive because they are interpretable , and often illuminate key computations required for a task . For example , consider neural machine translation—trained networks exhibit attention matrices that align words in the encoder sequence with the appropriate corresponding position in the decoder sentence ( Ghader & Monz , 2017 ; Ding et al. , 2019 ) . In this case , the attention matrix already contains information about which words in the source sequence are relevant for translating a particular word in the target sequence ; that is , forming the attention matrix itself 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . constitutes a significant part of solving the overall task . How is it that networks are able to achieve this ? What are the mechanisms underlying how networks form attention , and how do they vary across tasks and architectures ? In this work , we study these questions by analyzing three different encoder-decoder architectures on sequence-to-sequence tasks . We develop a method for decomposing the hidden states of the network into a sum of components that let us isolate input driven behavior from temporal ( or sequence ) driven behavior . We use this to first understand how networks solve tasks where all samples use the same attention matrix , a diagonal one . We then build on that to show how additional mechanisms can generate sample-dependent attention matrices that are still close to the average matrix . Our Contributions • We propose a decomposition of hidden state dynamics into separate pieces , one of which explains the temporal behavior of the network , another of which describes the input behavior . We show such a decomposition aids in understanding the behavior of networks with attention . • In the tasks studied , we show the temporal ( input ) components play a larger role in determining the attention matrix as the average attention matrix becomes a better ( worse ) approximation for a random sample ’ s attention matrix . • We discuss the dynamics of architectures with attention and/or recurrence and show how the input/temporal component behavior differs across said architectures . • We investigate the detailed temporal and input component dynamics in a synthetic setting to understand the mechanism behind common sequence-to-sequence structures and how they might differ in the presence of recurrence . Related Work As mentioned in the introduction , a common technique to gain some understanding is to visualize learned attention matrices , though the degree to which such visualization can explain model predictions is disputed Wiegreffe & Pinter ( 2019 ) ; Jain & Wallace ( 2019 ) ; Serrano & Smith ( 2019 ) . Input saliency Bastings & Filippova ( 2020 ) and attribution-propagation Chefer et al . ( 2020 ) methods have also been studied as potential tools for model interpretability . Complementary to these works , our approach builds on a recent line of work analyzing the computational mechanisms learned by RNNs from a dynamical systems perspective . These analyses have identified simple and interpretable hidden state dynamics underlying RNN operation on text-classification tasks such as binary sentiment analysis ( Maheswaranathan et al. , 2019 ; Maheswaranathan & Sussillo , 2020 ) and document classification ( Aitken et al. , 2020 ) . Our work extends these ideas into the domain of sequence-to-sequence tasks . Notation Let T and S be the input and output sequence length of a given sample , respectively . We denote the encoder and decoder hidden states by hEt ∈ Rn with t = 1 , . . . , T . Similarly , we denote decoder hidden states by hDs ∈ Rn , with s = 1 , . . . , S. The encoder and decoder hidden state dimensions are always taken to be equal in this work . Inputs to the encoder and decoder are denoted by xEt ∈ Rd and xDs ∈ Rd̃ . When necessary , we subscript different samples from a test/train set using α , β , γ , e.g . xEt , α for α = 1 , . . . , M . Outline We begin by introducing the three architectures we investigate in this work with varying combinations of recursion and attention . Next we introduce our temporal and input component decomposition and follow this up with a demonstration of how such a decomposition allows us to understand the dynamics of attention in a simple one-to-one translation task . Afterwards , we apply this decomposition to two additional tasks with increasing levels of complexity and discuss how our decomposition gives insight into the behavior of attention in these tasks . 2 Setup . A schematic of the three architectures we study is shown in Fig . 1 ( see SM for precise expressions ) . Vanilla Encoder Decoder ( VED ) is a recurrent encoder-decoder architecture with no attention ( Sutskever et al. , 2014 ) . The encoder and decoder update expression are hEt = FE ( h E t−1 , x E t ) and hDs = FD ( h D s−1 , x D s ) , respectively . Here , FD and FE are functions that implement the hidden state updates , which in this work are each one of three modern RNN cells : LSTMs ( Hochreiter & Schmidhuber , 1997 ) , GRUs ( Cho et al. , 2014 ) , or UGRNNs ( Collins et al. , 2016 ) . Encoder-Decoder with Attention ( AED ) is identical to the VED architecture above with a simple attention mechanism added ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) . For time step s of the decoder , we compute a context vector cs , a weighted sum of encoder hidden states , cs : = ∑T t=1 αsth E t , with αt : = softmax ( a1t , . . . , aSt ) the tth column of the attention matrix and ast : = hDs ·hEt the alignment between a given decoder and encoder hidden state . While more complicated attention mechanisms exist , in the main text we analyze the simplest form of attention for convenience of analysis.1 Attention Only ( AO ) is identical to the AED network above , but simply eliminates the recurrent information passed from one RNN cell to the next and instead adds fixed positional encoding vectors to the encoder and decoder inputs ( Vaswani et al. , 2017 ) . Due to the lack of recurrence , the RNN functions FE and FD simply act as feedforward networks in this setting.2 AO can be treated as a simplified version of a Transformer without self-attention , hence our analysis may also provide a hint into their inner workings ( Vaswani et al. , 2017 ) . 2.1 Temporal and Input Components . In architectures with attention , we will show that it is helpful to write the hidden states using what we will refer to as their temporal and input components . This will be useful because each hidden state has an associated time step and input word at that same time step ( e.g . s and xDs for h D s ) , therefore such a decomposition will often allow us to disentangle temporal and input behavior from any other network dynamics . We define the temporal components of the encoder and decoder to be the average hidden state at a given time step , which we denote by µEt and µ D s , respectively . Similarly , we define an encoder input component to be the average of all hEt − µEt for hidden states that immediately follow a given input word . We analogously define the decoder input components . In practice , we estimate such averages using a test set of size M , so that the temporal and input components of the encoder are respectively given by µEt ≈ ∑M α=1 1≤EoS , αh E t , α∑M β=1 1≤EoS , β , χE ( xt , α ) ≈ ∑M β=1 ∑T t′=1 1xt , α , xt′ , β ( hEt′ , β − µEt′ ) ∑M γ=1 ∑T t′′=1 1xt , α , xt′′ , γ , ( 1 ) where hEt , α the encoder hidden state of the αth sample , 1≤EoS , α is a mask that is zero if the αth sample is beyond the end of sentence , 1xt , α , xt′ , β is a mask that is zero if xt , α 6= xt′ , β , and we have temporarily suppressed superscripts on the inputs for brevity.3 By definition , the temporal components only vary with time and the input components only vary with input/output word . As such , 1In the SM , we implement a learned-attention mechanism using a scaled-dot product attention in the form of queries , keys , and value matrices ( Vaswani et al. , 2017 ) . For the AED and AO architectures , we find qualitatively similar results to the simple dot-product attention presented in the main text . 2We train non-gated feedforward networks and find their dynamics to be qualitatively the same , see SM . 3See SM for more details on this definition and the analogous decoder definitions . it will be useful to denote the encoder and decoder input components by χEx and χ D y , with x and y respectively running over all input and output words ( e.g . χEyes and χ D oui ) . We can then write any hidden state as hEt = µ E t + χ E x + ∆h E t , h D s = µ D s + χ D y + ∆h D s , ( 2 ) with ∆hEt : = h E t − µEt − χEt and ∆hDs : = hDs − µDs − χDy the delta components of encoder and decoder hidden states , respectively . Intuitively , we are simply decomposing each hidden state vector as a sum of a component that only varies with time/position in the sequence ( independent of input ) , a component that only varies with input ( independent of position ) , and whatever else is left over . Finally , we will often refer to hidden states without their temporal component , i.e . χEx + ∆h E t and χDy + ∆h D s , so for brevity we refer to these combinations as the input-delta components . Using the temporal and input components in ( 2 ) , we can decompose the attention alignment between two hidden states as ast = ( µDs + χ D y + ∆h D s ) · ( µEt + χ E x + ∆h E t ) . ( 3 ) We will show below that in certain cases several of the nine terms of this expression approximately vanish , leading to simple and interpretable attention mechanisms .

Summary: This paper proposes a novel method to decompose hidden representations of encoder-decoder architectures into two kinds of components: temporal and input-driven. The temporal component is the average hidden representation at a given time step thus it only varies with time, while the input-driven component is defined as the representation of a specific input word removing temporal component thus it is input-dependent but time-independent. Based on the decomposition, the authors show how encoder-decoder architectures attend, providing some insights for future research.