Area Attention

1 INTRODUCTION . Attentional mechanisms have significantly boosted the accuracy on a variety of deep learning tasks ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Xu et al. , 2015 ) . They allow the model to selectively focus on specific pieces of information , which can be a word in a sentence for neural machine translation ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) or a region of pixels in image captioning ( Xu et al. , 2015 ; Sharma et al. , 2018 ) . An attentional mechanism typically follows a memory-query paradigm , where the memory M contains a collection of items of information from a source modality such as the embeddings of an image or the hidden states of encoding an input sentence , and the query q comes from a target modality such as the hidden state of a decoder model . In recent architectures such as Transformer ( Vaswani et al. , 2017 ) , self-attention involves queries and memory from the same modality for either encoder or decoder . Each item in the memory has a key and value ( ki , vi ) , where the key is used to compute the probability ai regarding how well the query matches the item ( see Equation 1 ) . ai = exp ( fatt ( q , ki ) ) ∑|M | j=1 exp ( fatt ( q , kj ) ) ( 1 ) The typical choices for fatt include dot products qki ( Luong et al. , 2015 ) and a multilayer perceptron ( Bahdanau et al. , 2014 ) . The output OMq from querying the memory M with q is then calculated as the sum of all the values in the memory weighted by their probabilities ( see Equation 2 ) , which can be fed to other parts of the model for further calculation . During training , the model learns to attend to specific piece of information , e.g. , the correspondance between a word in the target sentence and a word in the source sentence for translation tasks . OMq = |M |∑ i=1 aivi ( 2 ) Attention mechanisms are typically designed to focus on individual items in the entire memory , where each item defines the granularity of what the model can attend to . For example , it can be a character for a character-level translation model , a word for a word-level model or a grid cell for an image-based model . Such a construction of attention granularity is predetermined rather than learned . While this kind of item-based attention has been helpful for many tasks , it can be fundamentally limited for modeling complex attention distribution that might be involved in a task . In this paper , we propose area attention , as a general mechanism for the model to attend to a group of items in the memory that are structurally adjacent . In area attention , each unit for attention calculation is an area that can contain one or more than one item . Each of these areas can aggregate a varying number of items and the granularity of attention is thus learned from the data rather than predetermined . Note that area attention subsumes item-based attention because when an area contains a single item , it is equivalent to regular attention mechanisms . Area attention can be used along multi-head attention ( Vaswani et al. , 2017 ) . With each head using area attention , multi-head area attention allows the model to attend to multiple areas in the memory . As we show in the experiments , the combination of both achieved the best results . Extensive experiments with area attention indicate that area attention outperforms regular attention on a number of recent models for two popular tasks : machine translation ( both token and character-level translation on WMT ’ 14 EN-DE and EN-FR ) , and image captioning ( trained on COCO and tested for both in-domain with COCO40 and out-of-domain captioning with Flickr 1K ) . These models involve several distinct architectures , such as the canonical LSTM seq2seq with attention ( Luong et al. , 2015 ) and the encoder-decoder Transformer ( Vaswani et al. , 2017 ; Sharma et al. , 2018 ) . 2 RELATED WORK . Item-grouping has been brought up in a number of language-specific tasks . Ranges or segments of a sentence , beyond individual tokens , have been often considered for problems such as dependency parsing or constituency parsing in natural language processing . Recent works ( Wang & Chang , 2016 ; Stern et al. , 2017 ; Kitaev & Klein , 2018 ) represent a sentence segment by subtracting the encoding of the first token from that of the last token in the segment , assuming the encoder captures contextual dependency of tokens . The popular choices of the encoder are LSTM ( Wang & Chang , 2016 ; Stern et al. , 2017 ) or Transformer ( Kitaev & Klein , 2018 ) . In contrast , the representation of an area ( or a segment ) in area attention , for its basic form , is defined as the mean of all the vectors in the segment where each vector does not need to carry contextual dependency . We calculate the mean of each area of vectors using subtraction operation over a summed area table ( Viola & Jones , 2001 ) that is fundamentally different from the subtraction applied in these previous works . Lee et al . proposed a rich representation for a segment in coreference resolution tasks ( Lee et al. , 2017 ) , where each span ( segment ) in a document is represented as a concatenation of the encodings of the first and last words in the span , the size of the span and an attention-weighted sum of the word embeddings within the span . Again , this approach operates on encodings that have already captured contextual dependency between tokens , while area attention we propose does not require each item to carry contextual or dependency information . In addition , the concept of range , segment or span that is proposed in all the above works addresses their specific task , rather than aiming for improving general attentional mechanisms . Previous work have proposed several methods for capturing structures in attention calculation . For example , Kim et al . used a conditional random field to directly model the dependency between items , which allows multiple `` cliques '' of items to be attended to at the same time ( Kim et al. , 2017 ) . Niculae and Blondel approached the problem , from a different angle , by using regularizers to encourage attention to be placed onto contiguous segments ( Niculae & Blondel , 2017 ) . In image captioning tasks , Pedersoli et al . enabled a model to attend to object proposals on an image ( Pedersoli et al. , 2016 ) while You et al . applied attention to semantic concepts and visual attributes that are extracted from an image ( You et al. , 2016 ) . Compared to these previous works , area attention we propose here does not require to train a special network or sub-network , or use an additional loss ( regularizer ) to capture structures . It allows a model to attend to information at a varying granularity , which can be at the input layer where each item might lack contextual information , or in the latent space . It is easy to apply area attention to existing single or multi-head attention mechanisms . By enhancing Transformer , an attention-based architecture , ( Vaswani et al. , 2017 ) with area attention , we achieved state-of-art results on a number of tasks . 3 AREA-BASED ATTENTION MECHANISMS . An area is a group of structurally adjacent items in the memory . When the memory consists of a sequence of items , a 1-dimensional structure , an area is a range of items that are sequentially ( or temporally ) adjacent and the number of items in the area can be one or multiple . Many language-related tasks are categorized in the 1-dimensional case , e.g. , machine translation or sequence prediction tasks . In Figure 1 , the original memory is a 4-item sequence . By combining the adjacent items in the sequence , we form area memory where each item is a combination of multiple adjacent items in the original memory . We can limit the maximum area size to consider for a task . In Figure 1 , the maximum area size is 3 . When the memory contains a grid of items , a 2-dimensional structure , an area can be any rectangular region in the grid ( see Figure 2 ) . This resembles many image-related tasks , e.g. , image captioning . Again , we can limit the maximum size allowed for an area . For a 2-dimensional area , we can set the maximum height and width for each area . In this example , the original memory is a 3x3 grid of items and the maximum height and width allowed for each area is 2 . As we can see , many areas can be generated by combining adjacent items . For the 1-dimensional case , the number of areas that can be generated is |R| = ( L − S ) S + ( S + 1 ) S/2 where S is the maximum size of an area and L is the length of the sequence . For the 2-dimensional case , there are an quadratic number of areas can be generated from the original memory : |R| = |Rv||Rh| . |Rv| = ( Lv −H ) H + ( H + 1 ) H/2 and |Rh| = ( Lh−W ) W + ( W + 1 ) W/2 where Lv and Lh are the height and width of the memory grid and H and W are the maximum height and width allowed for a rectangular area . To be able to attend to each area , we need to define the key and value for each area that contains one or multiple items in the original memory . As the first step to explore area attention , we define the key of an area , µi , simply as the mean vector of the key of each item in the area . µi = 1 |ri| |ri|∑ j=1 ki , j ( 3 ) where |ri| is the size of the area ri . For the value of an area , we simply define it as the the sum of all the value vectors in the area . vrii = |ri|∑ j=1 vi , j ( 4 ) With the keys and values defined , we can use the standard way for calculating attention as discussed in Equation 1 and Equation 2 . Note that this basic form of area attention ( Eq.3 and Eq.4 ) is parameterfree—it does not introduce any parameters to be learned . 3.1 COMBINING AREA FEATURES . Alternatively , we can derive a richer representation of each area by using features other than the mean of the key vectors of the area . For example , we can consider the standard deviation of the key vectors within each area . σi = √√√√ 1 |ri| |ri|∑ l=1 ( ki , l − µi ) 2 ( 5 ) We can also consider the height and width of each area , hi,1 ≤ hi ≤ H and wi,1 ≤ wi ≤W , as the features of the area . To combine these features , we use a multi-layer perceptron . To do so , we treat hi and wi as discrete values and project them onto a vector space using embedding ( see Equation 6 and 7 ) . ehi = 1 ( hi ) E h ( 6 ) ewi = 1 ( wi ) E w ( 7 ) where 1 ( hi ) and 1 ( wi ) are the one-hot encoding of hi and wi , and Eh ∈ RH×S and Ew ∈ RW×S are the embedding matrices . S is the depth of the embedding . We concatenate them to form the representation of the shape of an area . ei = [ e h i , e w i ] ( 8 ) We then combine them using a single-layer perceptron followed by a linear transformation ( see Equation 9 ) . kri = φ ( µiWµ + σiWσ + eiWe ) Wd ( 9 ) where φ is a nonlinear transformation such as ReLU , andWµ ∈ RD×D , Wσ ∈ RD×D , We ∈ R2S×D and Wd ∈ RD×D . Wµ , Wσ , We and Wd are trainable parameters .

Summary: Paper “AREA ATTENTION” extends the current attention models from word level to “area level”, i.e., the combination of adjacent words. Specifically, every $r_i$ adjacent words are first merged into a new item; next a key and the value for this item is calculated based on Eqn.(3 or 7) and Eqn. (4), and then the conventional attention models are applied to these new items. The authors work on (char level) NMT and image captioning to verify the algorithm.