Title: HIERARCHICAL ATTENTION: WHAT REALLY COUNTS IN VARIOUS NLP TASKS

Abstract: Attention mechanisms in sequence to sequence models have shown great ability and wonderful performance in various natural language processing (NLP) tasks, such as sentence embedding, text generation, machine translation, machine reading comprehension, etc. Unfortunately, existing attention mechanisms only learn either high-level or low-level features. In this paper, we think that the lack of hierarchical mechanisms is a bottleneck in improving the performance of the attention mechanisms, and propose a novel Hierarchical Attention Mechanism (Ham) based on the weighted sum of different layers of a multi-level attention. Ham achieves a state-of-the-art BLEU score of 0.26 on Chinese poem generation task and a nearly 6.5% averaged improvement compared with the existing machine reading comprehension models such as BIDAF and Match-LSTM. Furthermore, our experiments and theorems reveal that Ham has greater generalization and representation ability than existing attention mechanisms.

1. INTRODUCTION:

In recent years, long short-term memory, convolutional and recurrent networks with attention mechanisms have been very successful on a variety of NLP tasks. Most of the tasks In NLP can be formulated as sequence to sequence problems. Moreover, encoders and decoders are vital components in sequence to sequence models while processing the inputs and outputs.

An Attention mechanism works as a connector between the encoders and decoders and help the decoder to decide which parts of the source text to pay attention to. Thus an attention mechanism can integrates sequence models and transduction models, so it is able to connect two words in a single passage or paragraph without regard to their positions.

Attention mechanisms have become integral components in various NLP models. For example, in machine translation tasks, attention mechanism-based models [1, 2, 3] have ever been the state-ofthe-art; in sentence embedding, self attention based model is now the state-of-the-art [4] ; in machine reading comprehension, almost every recently-published model, such as BIDAF [5] , Match-LSTM [6] , Reinforcement Ranker Reader [7] , and R-NET [8] , contains attention mechanism; in abstractive summarization model [12] which has also once been the state-of-the-art, attention mechanism is very necessary, and in poem generation [9] , attention mechanism is also widely used. More surprisingly, Vaswani, et al. (2017) showed that their model Transformer which relies solely on the attention mechanisms can outperform those RNN or LSTM-based existing models in machine translation tasks. Thus, they stated that "Attention is all you need". However, we note that the potential issue with the existing attention mechanisms is that the basic attention mechanism learns only the low-level features while the multi-level attention mechanism learns only the high-level features. This may make it difficult for the model to capture the intermediate feature information, especially when the source texts are long. In order to address this issue, we present Ham which introduces a hierarchical mechanism into the existing multi-level attention mechanisms. Each time when we perform a multi-level attention, instead of using the result of the last attention level only, we use the weighted sum of the results of all the attention levels as the final output.

We show that Ham can learn all levels of features among the tokens in the input sequence and give a proof of its monotonicity and convergency. This work presents the design and implementation of Ham and our implementation performs well on a range of tasks by replacing the existing attention mechanisms in different models of different tasks.

We are able to achieve results comparable to or better than existing state-of-the-art model. On Chinese poem generation, our model scores 0.246 BLUE, an improvement of 21.78% from a RNN-based Poem Generator model. On machine reading comprehension task, our implementation is more effective, model with Ham has achieved an average improvement of 6.5% compared to previous models.

The implementation of the Hierarchical Attention Mechanism is not difficult and the code will be available on http://github.com after the acceptance.

2. ATTENTION MECHANISMS:

The attention mechanism can be described as a function whose input is a query and a set of key-value pairs, where the query and keys are vectors with the same dimension (denoted d k ), and the values are defined as d v -vectors. Note that in most types of attention mechanisms, the values are equal to the keys. Through the mapping of the attention mechanism, the input can be mapped to a single vector, which is as the output.

2.1. THE VANILLA ATTENTION MECHANISM(VAM):

Given a query q ∈ R d k and an input sequence

K = [k 1 , k 2 , . . . , k n ] ∈ R d k ×n where k i ∈ R d k

denotes the word embedding of the i-th word of the sequence, the vanilla attention mechanism aims at using a compatibility function f (k i , q) to compute a relativity score between the query q and each word k i . This score is treated as the attention value of q to k i . Then we have n attention scores f (k i , q) for i = 1, 2, . . . , n. Now we apply the softmax function to define a categorical distribution:

p(i|K, q) = softmax(f (k i , q)) = exp(f (k i , q)) n j=1 exp(f (k j , q))

.

Futher, we compute the output which is represented as the weighted sum of the input sequence:

s = n i=1 p(i|K, q)k i .

The attention mechanism above is the original version which was firstly proposed by Bahdanau, et al. (2014) . In the Scaled Dot-product Attention Mechanism, the compatibility function f is defined as the scaled dot product function f

(k i , q) = <ki,q> √ d k

.

Here the scaling factor 1

√ d k

is used to prevent the dot product from growing too large in magnitude.

2.2. SOFT, HARD AND LOCAL ATTENTION MECHANISMS:

There are three different types of mechanisms: soft, hard and local. The main difference between them is the region where attention function is calculated. The VAM belongs to soft attention where the categorical distribution p(i|K, q) is computed over the whole input sequence of words. Thus it is also referred to as global attention. The resulting distribution can reflect the relatedness or importance between the query and every word in the input sequence, and we use these importance scores as the weights of the output sum. Soft attention takes every word in the input sequence, no matter what kind of word it is, into consideration. Soft attention is differentiable and parameter-free, but is computationally expensive and less accurate.

In order to overcome the weakness of soft attention, hard attention is a natural alternative. Contrary to the widely-studied soft attention, hard attention locates accurately to only one key k i0 . In other words, the probability of getting the special key k i0 is 1 and others be 0. This implies that the choice of the one key means everything to the performance of the model. The action of choosing is not differentiable, so one uses reinforcement learning methods instead, such as policy gradient method.

As we have seen, soft and hard attentions are two extreme cases. Xu, et al. (2015) proposed a hybrid attention mechanism. Instead of choosing every key or only one key, one chooses a subset of all the keys from the input sequence. When computing the attention, one can just focus on the important part of the keys and discard the rest, thus it is also referred to as local attention. This attention mechanism combines the wideness and accuracy when choosing keys. The subset-choosing process is non-differentiable and reinforcement learning methods are also needed.

2.3. MULTI-HEAD ATTENTION AND MULTI-LEVEL ATTENTION MECHANISMS:

The multi-head attention mechanism proposed by Vaswani, et al. (2017) plays an important role in the Transformer model which is state-of-the-art in neural machine translation. Instead of calculating a single attention function with queries, keys and values, it linearly projects the queries, keys and values h times to d k , d k and d v dimensions, respectively. On each version of linear projections, the attention function is performed in parallel and yields several versions of d v -dimensional scaled dot-product attentions. Subsequently, these attention values are concatenated and once again projected, resulting in the final value as the output of the multi-head attention mechanism. That is,

Attention(Q, K, V ) = softmax( QK T √ d k )V, MultiHead(Q, K, V ) = Concat(head 1 , • • • , head h )W O , where head i = Attention(QW Q i , KW K i , V W V i ) and W Q i , W K i , W V i , W O i are all projection matrices.

The multi-level attention mechanism is another variety of attention mechanisms. Instead of increasing the number of heads, the multi-level attention increases the number of levels. For example, in a two-level attention mechanism, we calculate the attention value of the query q and the keys, which is represented as Attention(q, K, K). This output has the same dimension as the query, giving the first level. In the second level, we treat the output as the new query and calculate the attention value with the input sequence (or keys) K again. The result can be represented as Attention(Attention(q, K, K), K, K). The second attention can learn a higher level of internal features among the words of the input text.

Based on the long line of previous attempt, Cui et al. (2017) proposed a novel way of treating various documents in neural machine translation. They used a self-attention mechanism to encode the words in every document and then used a second attention over different documents to learn a higher level of features among the words of different documents. Yang, et al. (2016) also proposed a 2-level hierarchical attention network for document classification task. In recently proposed Transformer mode [1] , the authors repeated the attention mechanism N times over the input sequence in order to learn the higher level feature. This is an N -level attention mechanism through which the input sequences can be changed again and again into sequences much more suitable for feature extraction and decoder input. This is why so-called Transformer.

2.4. SELF ATTENTION MECHANISM:

In the self attention mechanism the query and key are the same. In other words, the query q stems from the input sequence K itself. Using self attention mechanism, we are able to learn the relatedness of different parts of the input sequence, no matter what their distance is. With self-attention, a long text can be encoded to a more suitable input for the decoder. Similar to the original attention mechanism, self-attentions also have three different types: soft, hard and local, as well as have multi-head version and multi-level version.

3. HIERARCHICAL ATTENTION MECHANISM (HAM):

In this section we present two Hierarchical Attention models built on the vanilla attention and self attention, respectively.

3.1. HIERARCHICAL VANILLA ATTENTION MECHANISM (HAM-V):

We have mentioned above that multi-level attention mechanisms can learn a deeper level of features among all the tokens of the input sequence and the query. In our model, we use multi-level for 

3.2. HIERARCHICAL SELF ATTENTION MECHANISM (HAM-S):

In Self Attention Mechanisms, the query stems from the input sequence k itself. So it can be treated as a special case of attention mechanisms. Similarly, the self-version of Hierarchical Attention Mechanisms, which is shown in Figure 2 In Vanilla Attention Mechanism, we have that min

1 i n k i 2 Attention(q, K, K) 2 max 1 i n k i 2 .

This result is very obvious according to the definition of Vanilla Attention Mechanism, Attention(q, K, K) is the weighted sum of k 1 , k 2 , • • • , k n and the weights are nonnegative with their sum 1. Denote the weights as

α 1 , • • • , α n . Then Attention(q, K, K) 2 = n i=1 α i k i 2 n i=1 α i k i 2 max 1 i n k i 2 .

The left-hand side of the inequality can be proved similarly. This theorem tells us that through multiple attention layers, the vectors of intermediate attention levels will neither explode nor vanish as the increase of attention depth d.

Theorem 2. Let A d = min ci,θ L(Ham(c 1 , • • • , c d )

, X, θ) be the global minimal value of the loss function. Then {A d }| +∞ d=1 is a monotonically decreasing sequence and it will converge. It is easy to note that:

A d = min ci(1 i d),θ L(Ham(c 1 , • • • , c d ), X, θ) = min ci(1 i d),θ L(Ham(c 1 , • • • , c d , c d+1 = -∞), X, θ) min ci(1 i d+1),θ L(Ham(c 1 , • • • , c d , c d+1 ), X, θ) = A d+1 .

This means the monotonicity of the sequence {A d }| +∞ d=1 . On the other hand, since our loss function always has positive values, sequence {A d } has 0 as its lower bound. Therefore, the monotonically decreasing sequence {A d }| +∞ d=1 will converge.

5. EXPERIMENTS:

Attention mechanisms are widely used in various NLP tasks. In our experiments, we would like to replace existing attention mechanisms with our novel Ham-V and replace existing self-attention mechanisms with our Ham-S to show the powerfulness and generalization ability of our model. We conduct our experiments on two different NLP tasks, Chinese Poem Generation and Machine Reading Comprehension.

5.1. MACHINE READING COMPREHENSION:

The first NLP task we used to test our Ham is the machine reading comprehension (MRC). We conduct our experiment on both English MRC and Chinese MRC. The baseline models we use include BIDAF [5] , Match-LSTM [6] , R-NET [8] . Here, BIDAF is for both Chinese and the other two are for English. They have a major similarity that all of them use attention mechanism as the connection between their encoders and decoders and they are all open source. Their code is available on http://github.com/baidu/DuReader, https://github. com/MurtyShikhar/Question-Answering and https://github.com/NLPLearn/ R-net. What we will do is to replace their attention mechanism with Ham and compare the difference of their performance. Specially, the R-NET model contains two attention mechanisms when doing question-passage matching and passage self-matching. Here, we replace them both with Ham and Ham-S.

The Chinese dataset we use for MRC experiments is DuREADER which is introduced by He, et al. ( 2017) and the English dataset we use include SQUAD which can be downloaded from https: //rajpurkar.github.io/SQuAD-explorer and MS-MARCO which can be downloaded from http://www.msmarco.org. We randomly choose 10 percent of question-answer data as testing set and the rest as training set. The evaluation method we use is BLEU-4 and ROUGE-L for Chinese, ExactMatch(EM) and F1 score for English, where EM measures the percentage of how much the prediction of the model matches ground truth exactly and F1 measures the overlap between prediction and ground truth. During our experiments, we set also different attention depths d to show the influence of attention depth.

5.2. CHINESE POEM GENERATION:

In In our experiment, we replace the attention part of PPG from a Vanilla Attention Mechanism to our Ham-V and set the compatibility function f to be the scaled dot product function, while keeping other parts and dataset unchanged as PPG except the evaluation part. The evaluation of poem generation in PPG is done by experts and we can not keep evaluation method unchanged since it is not convincing to find experts for evaluation. Our evaluation algorithm is based on BLEU-2 score which is calculated as

BLEU = 1 3 3 i=1 BLEU i ,

where BLEU i denotes the BLEU-2 score computed for the next (i + 1)th lines given the previous i goldstandard lines. This averaged BLEU can tell us how much correlated the lines of a generated quatrain are. We will show some quatrains generated by our Ham-based PPG in the appendix.

6 EXPERIMENTAL RESULTS AND QUALITATIVE ANALYSIS

As clearly visible in Table 1 , the proposed model is much better than conventional model at Chinese and English machine reading comprehension. This is likely due to the fact that our human language has a kind of hierarchical relationship within itself both structurally and semantically. And our hierarchical attention mechanism is much easier to capture the inherent structural and semantical hierarchical relationship in the source texts because of their innate similarity. We also set the attention depths d to be 1 (which is equivalent to ordinary models without using Ham), 2, 5, 10 and 20.

From Table 1 , we can find that our Ham plays a significant role of the whole model. With the increase of attention depth d, the performance rises quickly at first and starts to converge when d grows larger.

The biggest improvements on these three models are 7.26%, 7.76% and 5.64% respectively. Their average is over 6.5% which is a huge progress.

The results of our BLEU-based evaluation are summarized in When attention depth d is not large enough, the larger d is, the better performance our model will achieve. PPG with 10-level Ham has almost 25% of improvement on the averaged BLEU score compared with ordinary PPG. On the other hand, through the last two rows of the table above and Theorem 2, we know that with the continuing increase of d, our performance will converge sooner or later. So in order to balance between performance and training cost, we suggest attention depth d to be between 5 and 10. We tested the proposed model Ham on the task of Chinese poem generation and machine reading comprehension. The experiment revealed that the proposed Ham outperforms the conventional models significantly, achieving the state-of-the-art results. In the future, we would like to study more applications of Ham on other NLP tasks such as neural machine translation, abstractive summarization, paraphrase generalization and so on.

Recall that Ham belongs to soft attention where every token of input sequences is calculated by attention function. We will extend Ham to hard attention and local attention, to show whether the performance can be better and whether Ham can fit reinforcement learning environment better. Also, we will attempt to extend Multi-head Attention Mechanism of Vaswani, et al. (2017) to its hierarchical version and apply to neural machine translation.

:



Figures and Tables:

Figure 1: Hierarchical Vanilla Attention Mechanism

, takes only the sequence k as input. We calculate the self-attention results of the input sequence for d times consecutively. Finally, the output of our Ham-S is the weighted sum of these d attention results, where the d weights are the softmax values of d trainable parameters w 1 , w 2 , • • • , w d which is the same as Ham-V. Through the d levels of self-attention, our model can learn different levels of deep features among all the tokens of the input sequence, and through the d trainable parameters and the weighted sum mechanism, our model can learn the relative importance of the d self-attention levels. 4 ANALYSIS We present theoretical analysis of our Ham mainly in two aspects: representation ability and convergence. The representation ability of Ham is obviously higher than Vanilla Attention Mechanisms and Multi-Level Attention Mechanisms. Because the two attention mechanisms are just two extreme cases of Ham. When we set w 1 = 1, w 2 = w 3 = • • • = w d = 0, our Ham model is equivalent to the former. When we set w d = 1, w 1 = w 2 = • • • = w d-1 = 0, our model is equivalent to the latter. Thus Ham is much more general and the representation ability is much higher. Using the

Figure 2: Hierarchical Self Attention Mechanism

Figure 3: Three quatrains generated by 10-level Ham PPG model

this work we generate Chinese quatrains, each of whose lines has the same length of 5 or 7 characters. The baseline model we use is Planning based Poetry Generation (PPG) proposed byWang, et al. (2016), which generates Chinese poetries with a planning based neural network. Once we input a Chinese text, this model will generate a highly-related Chinese quatrains as follows.Firstly, the model extracts keywords from this input text with TextRank algorithm proposed byMihalcea, et al. (2004). Next, if the number of extracted keywords is not enough for a whole quatrain, more keywords will be created by Knowledge-based method. Then it comes to the final step, poem generation. The quatrain is generated line by line and each line corresponds a keyword. When generating a single line, one uses a bi-directional Gated Recurrent Unit (GRU) model proposed by Cho, et al. (2014 ) as encoder and another GRU model as decoder. Between encoder and decoder, an attention mechanism is used for connection.It is worth emphasizing that the dataset used by PPG consists of 76,859 quatrains from the Internet and PPG randomly chooses 2000 quatrains for testing, 2000 for validation and the rest for training. In the encoder part of PPG, the word embedding dimensionality is set as 512 and initialized by word2vec(Mikolov, et al. (2013)). In both GRU models, the hidden layers also contain 512 hidden units but they are initialized randomly. For more details, please readWang, et al. (2016). The code and dataset of PPG model can be found from https://github.com/Disiok/poetry-seq2seq. Evaluation results for MRC models

Table content: <table><tr><td>Model</td><td>Dataset</td><td colspan="2">BLEU-4 ROUGE-L</td></tr><tr><td>BIDAF</td><td>DuReader</td><td>33.95</td><td>44.20</td></tr><tr><td>BIDAF with 2-level Ham</td><td>DuReader</td><td>34.02</td><td>44.39</td></tr><tr><td>BIDAF with 5-level Ham</td><td>DuReader</td><td>34.79</td><td>45.10</td></tr><tr><td>BIDAF with 10-level Ham</td><td>DuReader</td><td>35.96</td><td>47.33</td></tr><tr><td>BIDAF with 20-level Ham</td><td>DuReader</td><td>35.79</td><td>47.41</td></tr><tr><td>Model</td><td>Dataset</td><td>EM</td><td>F1</td></tr><tr><td>Match-LSTM</td><td>SQUAD</td><td>54.29</td><td>66.87</td></tr><tr><td>Match-LSTM with 2-level Ham</td><td>SQUAD</td><td>54.41</td><td>66.99</td></tr><tr><td>Match-LSTM with 5-level Ham</td><td>SQUAD</td><td>55.29</td><td>69.03</td></tr><tr><td>Match-LSTM with 10-level Ham</td><td>SQUAD</td><td>58.37</td><td>70.70</td></tr><tr><td>Match-LSTM with 20-level Ham</td><td>SQUAD</td><td>58.47</td><td/></tr><tr><td>Model</td><td>Dataset</td><td colspan="2">BLEU-1 ROUGE-L</td></tr><tr><td>R-NET</td><td>MSMARCO</td><td>41.29</td><td>43.38</td></tr><tr><td>R-NET with 2-level Ham</td><td>MSMARCO</td><td>41.37</td><td>43.89</td></tr><tr><td>R-NET with 5-level Ham</td><td>MSMARCO</td><td>43.13</td><td>45.67</td></tr><tr><td>R-NET with 10-level Ham</td><td>MSMARCO</td><td>43.48</td><td>45.75</td></tr><tr><td>R-NET with 20-level Ham</td><td>MSMARCO</td><td>43.62</td><td>45.78</td></tr></table>

We also set different attention depths in order to learn the relationship between overall performance and the attention depth d.

Table content: <table><tr><td>Model</td><td colspan="2">BLEU 1</td><td colspan="2">BLEU 2</td><td colspan="2">BLEU 3</td><td colspan="2">BLEU</td></tr><tr><td/><td colspan="8">5-Char 7-Char 5-Char 7-Char 5-Char 7-Char 5-Char 7-Char</td></tr><tr><td>SMT</td><td>0.056</td><td>0.124</td><td>0.052</td><td>0.150</td><td>0.054</td><td>0.176</td><td>0.054</td><td>0.150</td></tr><tr><td>RNNPG</td><td>0.058</td><td>0.187</td><td>0.062</td><td>0.210</td><td>0.067</td><td>0.207</td><td>0.062</td><td>0.202</td></tr><tr><td>PPG</td><td>0.061</td><td>0.185</td><td>0.069</td><td>0.193</td><td>0.073</td><td>0.198</td><td>0.068</td><td>0.192</td></tr><tr><td>5-level Ham PPG</td><td>0.063</td><td>0.210</td><td>0.070</td><td>0.237</td><td>0.075</td><td>0.226</td><td>0.070</td><td>0.224</td></tr><tr><td colspan="2">10-level Ham PPG 0.062</td><td>0.217</td><td>0.075</td><td>0.267</td><td>0.076</td><td>0.259</td><td>0.072</td><td>0.244</td></tr><tr><td colspan="2">20-level Ham PPG 0.062</td><td>0.221</td><td>0.074</td><td>0.258</td><td>0.076</td><td>0.260</td><td>0.071</td><td>0.246</td></tr></table>

BLEU-based evaluation results

Table content: <table/>

Peer review (iclr2019):
 The paper proposes to enhance existing multi-level attention (self-attention) mechanism by obtaining query and key vectors (= value vectors) from all levels after weighted-averaging them. The paper claims that this is also theoretically beneficial because the loss function will converge to zero as the number of layers increase. It claims that the proposed architecture outperforms existing attention-based models in English MRC test (SQuAD), Chinese MRC test, and Chinese poem generation task. I find three major issues in the paper. 1. I think the proposed hypothesis lacks the novelty that ICLR audience seeks for. Through many existing architectures (ResNet, ELMo), we already know that skip connection between CNN layers or weighted average of multiple LSTM layers could improve model significantly. Perhaps this could be an application paper that brings existing methods to a slightly different (attention) domain, but not only such paper is less suitable for ICLR, but also it would require strong experimental results. But as I will detail in the second point, I also have some worries about the experiments. 2. The experimental results have problems. For English MRC experiment (SQuAD), the reproduced match-LSTM score is ~10% below the reported number in its original paper. Furthermore, it is not clear whether the improvement comes from having multiple attention layers (which is not novel) or weighted-averaging the attention layers (the proposed method). BiDAF and match-LSTM have single attention layers, so it is not fair to compare them with multi-layer attention. 3. Lastly, I am not sure I understood the theoretical section correctly, but it is not much interesting that having multiple layers allow one to approach closer to zero loss. In fact, any sufficiently large model can obtain close-to-zero loss on the training data. This is not a sufficient condition for a good model. We cannot guarantee if the model has generalized well; it might have just overfit to the training data. A few minor issues and typos on the paper: - First para second sentence: In -> in - First para second sentence: sequence to sequence -> sequence-to-sequence - Second last para of intro: sentence fragment - Figure 3: would be good to have English translation.