Hierarchical Attention: What Really Counts in Various NLP Tasks

1 INTRODUCTION . In recent years , long short-term memory , convolutional and recurrent networks with attention mechanisms have been very successful on a variety of NLP tasks . Most of the tasks In NLP can be formulated as sequence to sequence problems . Moreover , encoders and decoders are vital components in sequence to sequence models while processing the inputs and outputs . An Attention mechanism works as a connector between the encoders and decoders and help the decoder to decide which parts of the source text to pay attention to . Thus an attention mechanism can integrates sequence models and transduction models , so it is able to connect two words in a single passage or paragraph without regard to their positions . Attention mechanisms have become integral components in various NLP models . For example , in machine translation tasks , attention mechanism-based models [ 1 , 2 , 3 ] have ever been the state-ofthe-art ; in sentence embedding , self attention based model is now the state-of-the-art [ 4 ] ; in machine reading comprehension , almost every recently-published model , such as BIDAF [ 5 ] , Match-LSTM [ 6 ] , Reinforcement Ranker Reader [ 7 ] , and R-NET [ 8 ] , contains attention mechanism ; in abstractive summarization model [ 12 ] which has also once been the state-of-the-art , attention mechanism is very necessary , and in poem generation [ 9 ] , attention mechanism is also widely used . More surprisingly , Vaswani , et al . ( 2017 ) showed that their model Transformer which relies solely on the attention mechanisms can outperform those RNN or LSTM-based existing models in machine translation tasks . Thus , they stated that `` Attention is all you need '' . However , we note that the potential issue with the existing attention mechanisms is that the basic attention mechanism learns only the low-level features while the multi-level attention mechanism learns only the high-level features . This may make it difficult for the model to capture the intermediate feature information , especially when the source texts are long . In order to address this issue , we present Ham which introduces a hierarchical mechanism into the existing multi-level attention mechanisms . Each time when we perform a multi-level attention , instead of using the result of the last attention level only , we use the weighted sum of the results of all the attention levels as the final output . We show that Ham can learn all levels of features among the tokens in the input sequence and give a proof of its monotonicity and convergency . This work presents the design and implementation of Ham and our implementation performs well on a range of tasks by replacing the existing attention mechanisms in different models of different tasks . We are able to achieve results comparable to or better than existing state-of-the-art model . On Chinese poem generation , our model scores 0.246 BLUE , an improvement of 21.78 % from a RNN-based Poem Generator model . On machine reading comprehension task , our implementation is more effective , model with Ham has achieved an average improvement of 6.5 % compared to previous models . The implementation of the Hierarchical Attention Mechanism is not difficult and the code will be available on http : //github.com after the acceptance . 2 ATTENTION MECHANISMS . The attention mechanism can be described as a function whose input is a query and a set of key-value pairs , where the query and keys are vectors with the same dimension ( denoted dk ) , and the values are defined as dv-vectors . Note that in most types of attention mechanisms , the values are equal to the keys . Through the mapping of the attention mechanism , the input can be mapped to a single vector , which is as the output . 2.1 THE VANILLA ATTENTION MECHANISM ( VAM ) . Given a query q ∈ Rdk and an input sequence K = [ k1 , k2 , . . . , kn ] ∈ Rdk×n where ki ∈ Rdk denotes the word embedding of the i-th word of the sequence , the vanilla attention mechanism aims at using a compatibility function f ( ki , q ) to compute a relativity score between the query q and each word ki . This score is treated as the attention value of q to ki . Then we have n attention scores f ( ki , q ) for i = 1 , 2 , . . . , n. Now we apply the softmax function to define a categorical distribution : p ( i|K , q ) = softmax ( f ( ki , q ) ) = exp ( f ( ki , q ) ) ∑n j=1 exp ( f ( kj , q ) ) . Futher , we compute the output which is represented as the weighted sum of the input sequence : s = n∑ i=1 p ( i|K , q ) ki . The attention mechanism above is the original version which was firstly proposed by Bahdanau , et al . ( 2014 ) . In the Scaled Dot-product Attention Mechanism , the compatibility function f is defined as the scaled dot product function f ( ki , q ) = < ki , q > √dk . Here the scaling factor 1√ dk is used to prevent the dot product from growing too large in magnitude . 2.2 SOFT , HARD AND LOCAL ATTENTION MECHANISMS . There are three different types of mechanisms : soft , hard and local . The main difference between them is the region where attention function is calculated . The VAM belongs to soft attention where the categorical distribution p ( i|K , q ) is computed over the whole input sequence of words . Thus it is also referred to as global attention . The resulting distribution can reflect the relatedness or importance between the query and every word in the input sequence , and we use these importance scores as the weights of the output sum . Soft attention takes every word in the input sequence , no matter what kind of word it is , into consideration . Soft attention is differentiable and parameter-free , but is computationally expensive and less accurate . In order to overcome the weakness of soft attention , hard attention is a natural alternative . Contrary to the widely-studied soft attention , hard attention locates accurately to only one key ki0 . In other words , the probability of getting the special key ki0 is 1 and others be 0 . This implies that the choice of the one key means everything to the performance of the model . The action of choosing is not differentiable , so one uses reinforcement learning methods instead , such as policy gradient method . As we have seen , soft and hard attentions are two extreme cases . Xu , et al . ( 2015 ) proposed a hybrid attention mechanism . Instead of choosing every key or only one key , one chooses a subset of all the keys from the input sequence . When computing the attention , one can just focus on the important part of the keys and discard the rest , thus it is also referred to as local attention . This attention mechanism combines the wideness and accuracy when choosing keys . The subset-choosing process is non-differentiable and reinforcement learning methods are also needed . 2.3 MULTI-HEAD ATTENTION AND MULTI-LEVEL ATTENTION MECHANISMS . The multi-head attention mechanism proposed by Vaswani , et al . ( 2017 ) plays an important role in the Transformer model which is state-of-the-art in neural machine translation . Instead of calculating a single attention function with queries , keys and values , it linearly projects the queries , keys and values h times to dk , dk and dv dimensions , respectively . On each version of linear projections , the attention function is performed in parallel and yields several versions of dv-dimensional scaled dot-product attentions . Subsequently , these attention values are concatenated and once again projected , resulting in the final value as the output of the multi-head attention mechanism . That is , Attention ( Q , K , V ) = softmax ( QKT√ dk ) V , MultiHead ( Q , K , V ) = Concat ( head1 , · · · , headh ) WO , where headi = Attention ( QW Q i , KW K i , V W V i ) and W Q i , W K i , W V i , W O i are all projection matrices . The multi-level attention mechanism is another variety of attention mechanisms . Instead of increasing the number of heads , the multi-level attention increases the number of levels . For example , in a two-level attention mechanism , we calculate the attention value of the query q and the keys , which is represented as Attention ( q , K , K ) . This output has the same dimension as the query , giving the first level . In the second level , we treat the output as the new query and calculate the attention value with the input sequence ( or keys ) K again . The result can be represented as Attention ( Attention ( q , K , K ) , K , K ) . The second attention can learn a higher level of internal features among the words of the input text . Based on the long line of previous attempt , Cui et al . ( 2017 ) proposed a novel way of treating various documents in neural machine translation . They used a self-attention mechanism to encode the words in every document and then used a second attention over different documents to learn a higher level of features among the words of different documents . Yang , et al . ( 2016 ) also proposed a 2-level hierarchical attention network for document classification task . In recently proposed Transformer mode [ 1 ] , the authors repeated the attention mechanism N times over the input sequence in order to learn the higher level feature . This is an N -level attention mechanism through which the input sequences can be changed again and again into sequences much more suitable for feature extraction and decoder input . This is why so-called Transformer . 2.4 SELF ATTENTION MECHANISM . In the self attention mechanism the query and key are the same . In other words , the query q stems from the input sequence K itself . Using self attention mechanism , we are able to learn the relatedness of different parts of the input sequence , no matter what their distance is . With self-attention , a long text can be encoded to a more suitable input for the decoder . Similar to the original attention mechanism , self-attentions also have three different types : soft , hard and local , as well as have multi-head version and multi-level version . 3 HIERARCHICAL ATTENTION MECHANISM ( HAM ) . In this section we present two Hierarchical Attention models built on the vanilla attention and self attention , respectively . 3.1 HIERARCHICAL VANILLA ATTENTION MECHANISM ( HAM-V ) . We have mentioned above that multi-level attention mechanisms can learn a deeper level of features among all the tokens of the input sequence and the query . In our model , we use multi-level for reference but different from the existing Multi-Level Attention Mechanisms . Our Ham-V focus on all the intermediate attention results rather than just the result of the last attention level . As shown in Figure1 , given the query and the input sequence which consists of n keys , we calculate the Vanilla Attention Mechanism result of them and get Query 1 . And then we continue to calculate the attention result of Query 1 and the keys and get Query 2 . Repeat this calculation d times . Thus , we form a d-depth attention . Finally , the output of our Ham-V is the weighted sum of the above d attention results , where the d weights are the softmax values of d trainable parameters . The softmax is used to convert these d weights into the probabilities . These weights can tell us the relative importance of the d intermediate attention results Query i . In other words , the relative importance of the d attention levels .

Summary: The paper proposes to enhance existing multi-level attention (self-attention) mechanism by obtaining query and key vectors (= value vectors) from all levels after weighted-averaging them. The paper claims that this is also theoretically beneficial because the loss function will converge to zero as the number of layers increase. It claims that the proposed architecture outperforms existing attention-based models in English MRC test (SQuAD), Chinese MRC test, and Chinese poem generation task.