Title: A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING
Authors: Zhouhan Lin
Minwei Feng
Cicero Nogueira dos Santos
Mo Yu
Bing Xiang
Bowen Zhou
Yoshua Bengio
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Supervised Learning

Abstract: This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

TL;DR: a new model for extracting an interpretable sentence embedding by introducing self-attention and matrix representation.

Review discussion:
How do you use the sentence representation matrix M as the input to a classifier? Do you simply concatenate the rows into one long vector, and feed that into a softmax, or is there any kind of weight-sharing between the classifier parameters for different rows?
On another note, how do you see this work relating to the LSTMN? Your work has clear differences with theirs, but they share the idea of attending over a sequence without any outside input to the attention process, and they both work well on similar tasks.
Paper (EMNLP '16): https://arxiv.org/pdf/1601.06733v7.pdf
How do you use the sentence representation matrix M as the input to a classifier? Do you simply concatenate the rows into one long vector, and feed that into a softmax, or is there any kind of weight-sharing between the classifier parameters for different rows?
- There can be weight sharing or not. The simplist way is to concatenate all the rows together and then feed them into an MLP as a regular vector, which is what we did in those results shown in the paper.
Otherwise, we can also take advantage of the 2D structure, multiplying a same weight matrix for each row, and producing a 2D hidden representation. We actually tried that, and it turns out to have yielded similar performance (but with much fewer parameters of course). We didn't mention that in the paper majorly because we want this paper to foucs on the sentence embedding itself.
On another note, how do you see this work relating to the LSTMN? Your work has clear differences with theirs, but they share the idea of attending over a sequence without any outside input to the attention process, and they both work well on similar tasks.
- The attention of LSTMN is sort of an "online updating" attention, which is a more fine-grained attention that targets at discovering lexical correlations between a certain word and its previous words. It achieves extra language understanding by discovering lexical correlations.
On the contrary, the attention we are using happens after the LSTM processes the whole sentence, so it is less fine-grained, focuses directly on the semantics that makes sense for discriminating the targets. It is less focused on relations between words, but more on the semantics of the whole sentence that each word contributes to.
Because of the aforementioned reason, we can also say that they are different levels of attention.
Moreover, LSTMN still needs a maxpooling step to compress all the contents in a tape into a fixed length vector afterward.
Thus they can be used together: LSTMN but with our attention substituting the maxpooling part.
Another difference that resulted from this granularity is the scalability. LSTMN computes an annotation vector over all of its previous words each time when the LSTMN computes its next step, and that is O(n^2) complexity.
So when the sequence to be processed becomes very long (like the long reviews in the author profiling dataset), it becomes more and more expensive to compute its next time step. It makes it hard for this model to scale up to longer contents, like long reviews, paragraphs, etc. In that respect, our model is only O(n) complexity, which scales up better.
You mention that using KL divergence as a diversity regularizer is too computationally expensive, but presumably this is not true if r (the number of rows) is sufficiently small. Have you tried decreasing r to a small enough value so that you can utilize the KL penalty? If so, does it out perform the Frobenius norm penalty with the same (or larger) value of r?
We tried but we haven't got it work correctly. There includes a p(i) / q(i) operation in the KL divergence expression, and since we are maximizing (instead of minimizing) the KL divergence, We are optimizing the annotation matrix A to have a lot of sufficiently small or even zero values at different softmax output units (which maximizes the "mismatch" between two distributions).
Thus it makes the training not very stable. Maybe there is something more to play with to make the KL term stable.
So that's why we moved on to devise another more stable and computationally cheaper penalty term instead.
On another note, if we use the KL divergence, then a KL divergence should be computed between every pair of distributions, i.e., all possible pairs of rows in the A (annotation) matrix.  For each pair we need to compute \Sigma{i}{p(i)log{\frac{p(i)}{q(i)}}}, which takes 3n multiplications. (Inheriting notations, n here means the number of tokens in the sentence.) To compare KL for all possible pairs, we thus need 3n*r^2 multiplications.
Plus the Frobenius norm you will still need, it sums up to (3n+1)*r^2 multiplications.
However, it only takes (n+1)*r^2 multiplications for the penalty we currently use. It’s like one-third of the computation, no matter how large the “r” is. Since there are specific GEMM for matrix multiplications in GPU, the run time difference will be much more than 3 times.
There is another feature that KL doesn’t provide but we want, which is, we want each individual row to focus on a single aspect of semantics, so we want the attention annotation to be more focused.
That is enforced by forcing the diagonal of AA^T to be close to 1, but with KL penalty we can’t encourage that.
This paper introduces a sentence encoding model (for use within larger text understanding models) that can extract a matrix-valued sentence representation by way of within-sentence attention. The new model lends itself to (slightly) more informative visualizations than could be gotten otherwise, and beats reasonable baselines on three datasets.
The paper is reasonably clear, I see no major technical issues, and the proposed model is novel and effective. It could plausibly be relevant to sequence modeling tasks beyond NLP. I recommend acceptance.
There is one fairly serious writing issue that I'd like to see fixed, though: The abstract, introduction, and related work sections are all heavily skewed towards unsupervised learning. The paper doesn't appear to be doing unsupervised learning, and the ideas are no more nor less suited to unsupervised learning than any other mainstream ideas in the sentence encoding literature.
Details:
- You should be clearer about how you expect these embeddings to be used, since that will be of certain interest to anyone attempting to use the results of this work.
In particular, how you should convert the matrix representation into a vector for downstream tasks that require one. Some of the content of your reply to my comment could be reasonably added to the paper.
- A graphical representation of the structure of the model would be helpful.
- The LSTMN (Cheng et al., EMNLP '16) is similar enough to this work that an explicit comparison would be helpful. Again, incorporating your reply to my comment into the paper would be more than adequate.
- Jiwei Li et al. (Visualizing and Understanding Neural Models in NLP, NAACL '15) present an alternative way of visualizing the influence of words on sentence encodings without using cross-sentence attention. A brief explicit comparison would be nice here.
Thanks for your review and comments. We made 4 changes according to your suggestion:
1. Now we have made the related works more complete by adding several new citations to supervised learning.
2. As we responded in the former comment, there are two ways to use the matrix embeddings. We have made it clear in the paper and added a whole appendix about the other parameter-saving method. One gives you better performance but with a larger number of parameters, the other reduces the parameters a lot but at the cost of reducing performance a little bit.
3. We also added graphical representations of the models both in the paper and in the supplementary material.
4. We discussed LSTMN in the related work section, which we also think is a very important related work.
The visualization introduced in Jiwei et. al. 2015 is an interesting and related work. It focuses on understanding how the LSTM states function in NLP applications. The closest visualization among all the visualization methods proposed in that paper is the Figure 9 in their paper.
However we still haven't found a good enough way to compare this: it seems more similar to LSTMN's type of attention.
So in this revision, we haven't included it.
Thanks again for your contributive comments and suggestions, and we sincerely hope our revision addresses your concerns satisfyingly.
This paper proposes a method for representing sentences as a 2d matrix by utilizing a self-attentive mechanism on the hidden states of a bi-directional LSTM encoder. This work differs from prior work mainly in the 2d structure of embedding, which the authors use to produce heat-map visualizations of input sentences and to generate good performance on several downstream tasks.
There is a substantial amount of prior work which the authors do not appropriately address, some of which is listed in previous comments. The main novelty of this work is in the 2d structure of embeddings, and as such, I would have liked to see this structure investigated in much more depth.
Specifically, a couple important relevant experiments would have been:
* How do the performance and visualizations change as the number of attention vectors (r) varies?
* For a fixed parameter budget, how important is using multiple attention vectors versus, say, using a larger hidden state or embedding size?
I would recommend changing some of the presentation in the penalization term section.
Specifically, the statement that "the best way to evaluate the diversity is definitely the Kullback Leibler divergence between any 2 of the summation weight vectors" runs somewhat counter to the authors' comments about this topic below.
In Fig. (2), I did not find the visualizations to provide particularly compelling evidence that the multiple attention vectors were doing much of interest beyond a single attention vector, even with penalization. To me this seems like a necessary component to support the main claims of this paper.
Overall, while I found the architecture interesting, I am not convinced that the model's main innovation -- the 2d structure of the embedding matrix -- is actually doing anything important or meaningful beyond what is being accomplished by similar attentive embedding models already present in the literature.
Further experiments demonstrating this effect would be necessary for me to give this paper my full endorsement.
We thank the reviewer for the detailed review and suggestion. We have made 3 major changes and some other minor changes in our paper to address the former two concerns you raised.
1. We rewrite the "related works" section in the revised paper. We cited those works mentioned in the other comment, discussed and compared them with our proposed approach.
2.
Also, we now have added new experiments in Section 4.4.2 that shows the influence of attention vector numbers ($r$).
3. To address the concern about parameters, we add a supplementary material which introduced weight pruning by taking advantage of the matrix structure. This weight pruning mechanism allows us to drastically reduce the number of parameters, by sacrificing a bit of performance.
On the other hand, we want to mention that all of the hyperparameters for baseline models are either independently optimized or cited from other papers.
Thus those baseline results are almost the best these models can achieve, even if you add more parameters to it.
So in terms of performance, we think having more parameters doesn’t necessarily give our model advantage.
Quote:"Specifically, the statement that "the best way to evaluate the diversity is definitely the Kullback-Leibler divergence between any 2 of the summation weight vectors" runs somewhat counter to the authors' comments on this topic below.
That statement was not precise and detailed enough. Now we have updated the statement about KL divergence in the paper, integrating some of the contents in the discussion made on this webpage.
About concerns on Figure 2: We agree that it is very hard to find evidence to support the idea of using multiple vectors just by looking at the heatmap plots. Now we have a new subsection in the explorative experiments part (Section 4.4.2) to provide quantitative evidences to support the idea of using multiple vectors. Figure 2 by itself is trying to show that having the penalty term forces the model to learn a set of attentions that is more variant. Figure 2 left every hops of attention appears very similar, which in the right the model's attention exhibits more variance.
Now in the new 4.4.2 section, we can tell that the model performance is indeed influenced by $r$ values. Having a matrix embedding brings a significant advantage over using vector embeddings (i.e., when $r=1$). We can figure that $r$ is not the larger the better (and so do the parameters), a value between 5~20 turned out to be the most suitable. We also show in table 1 that matrix sentence outperforms LSTM and CNN baselines. In Table 2, we show that matrix embedding outperforms a bunch of other sentence embedding models by a significant margin, only slightly worse than NSE.
Thanks again for your contributive comments and suggestions, and we sincerely hope our revision addresses your concerns satisfyingly.
I like the idea in this paper that use not just one but multiple attentional vectors to extract multiple representations for a sentence. The authors have demonstrated consistent gains across three different tasks Age, Yelp, & SNLI.
However, I'd like to see more analysis on the 2D representations (as concerned by another reviewer) to be convinced.
Specifically, r=30 seems to be a pretty large value when applying to short sentences like tweets or those in the SNLI dataset. I'd like to see the effect of varying r from small to large value. With large r value, I suspect your models might have an advantage in having a much larger number of parameters (specifically in the supervised components) compare to other models. To make it transparent, the model sizes should be reported. I'd also like to see performances on the dev sets or learning curves.
In the conclusion, the authors remark that "attention mechanism reliefs the burden of LSTM". If the 2D representations are effective in that aspect, I'd expect that the authors might be able to train with a smaller LSTM. Testing the effect of LSTM dimension vs $r$ will be helpful.
Lastly, there is a problem in the presentation of the paper in which there is no training objective defined. Readers have to read until the experimental sections to guess that the authors perform supervised learning and back-prop through the self-attention mechanism as well as the LSTM.
* Minor comments:
Typos: netowkrs, toghter, performd
Missing year for the citation of (Margarit & Subramaniam)
In figure 3, attention plotswith and without penalization look similar.
We thank the reviewer for the detailed review, and for your interest in our multiple-vector representation.
We do agree that we are lacking a set of experiments about how the $r$, i.e. number of rows in the matrix representation, would affect the model’s performance. In the reviewed version of the text, we added 2 sets of experiments in Section 4.4.2, one for yelp and the other for SNLI, that shows how the model’s performance is related to $r$. In both experiments, it turned out that using multiple vector representation does have a significant impact on the performance on each task.
Additionally, although we were using $r=30$ in previous experiments, we also show that a smaller $r$ would also yield a similar result.
As for the model size, we do agree that our model ends up with a much larger number of parameters.
Like in convnets, 90% of the weights are in the output MLP (the MLP connecting the matrix representation to softmax). We do have a weight pruning mechanism that significantly reduces the number of parameters in the big output MLP by taking advantage of the matrix representation. We didn’t include that in the first version of the paper because we wanted to use the limited space to focus on the sentence embedding and penalization term. Now we added a full supplementary material on how these weight sharing can significantly reduce the parameters. Compared to our original model, we can still get similar results after structurally pruning most of the weights.
On the other hand, the improvement one can reach by naively adding more and more parameters to a certain model is really marginal. Hyperparameters in all baseline models in our paper are either optimized independently or directly cited from other papers, which means the reported baseline results are almost the best these models can do, even if you add more parameters to it.
So in terms of performance, we think having more parameters doesn’t necessarily give our model advantage.
But we do agree that having more parameters means taking more memory space, more inference time, etc.
In addition, the new Appendix A we attached to the end of the paper provides a satisfying method to overcome this problem.
Quote:"In the conclusion, the authors remark that "attention mechanism reliefs the burden of LSTM". If the 2D representations are effective in that aspect, I'd expect that the authors might be able to train with a smaller LSTM. Testing the effect of LSTM dimension vs $r$ will be helpful.
Thanks for pointing that out. It seems that we didn’t state it in a clear enough way. The amount of information for each LSTM hidden state to carry is not changed, since each hidden state should still roughly carry the amount of information contained in a word embedding. What is “relieved” is that the LSTM is not expected to carry all information through all of its hidden states, which is the case if one picks the last LSTM state as sentence embedding. Just providing some correlational information around each hidden state’s peripheral time steps would be sufficient for the self-attention model.
Of course the latter requires the hidden states to cover more content and be more abstractive, thus is much harder to learn.
So the “relief” here is more about relieving the long term dependencies, which cannot be solved by changing hidden state numbers. Now we have rephrased our reasoning in the corresponding part of the paper.
We thought that since the sentence embedding is for supervised tasks, the matrix sentence embedding model can be combined with any downstream applications.
So the objective is thus dependent on what tasks the downstream application belongs to. To make the architecture presentation clearer, we added graphical representations of the structure of the model.
Figure 3 is intended to show that with the penalty term, the "overall attention" becomes more focused. Both models trained with and without the penalty term will learn to focus on keywords that contribute to the sentiment of the sentence, but the difference we want to show is that the one trained with penalty will pay even less attention to less related parts of the sentence. This is reflected in the plot that many of the shallow red parts are even shallower when using penalty term.
Thanks again for your contributive comments and suggestions, and we sincerely hope our revision addresses your concerns satisfyingly.

Meta-review:
role: PC
data: comment: A solid paper about sentence representation learning using "internal attention" (representing sentences as a self-attention matrix rather than a vector). The idea is novel, well presented, and has potential applications to a variety of sequence-based problems. The main criticism in reviews was about minor points needing clarification, but I am suitably satisfied that these have been acknowledged and incorporated into the current revision. Reviewer 2 is concerned that the experiments do not fully support the claim that this model will improve end task performance over more standard attention mechanisms. Glancing over the paper, and in light of other reviews and discussion, I am not convinced that this objection should halt acceptance, but invite the authors to take this suggestion onboard for future work (or in the final version of their paper). This paper should be accepted to the conference.

Decision: Accept (Poster)