Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a sentence encoding model (for use within larger text understanding models) that can extract a matrix-valued sentence representation by way of within-sentence attention. The new model lends itself to (slightly) more informative visualizations than could be gotten otherwise, and beats reasonable baselines on three datasets.

The paper is reasonably clear, I see no major technical issues, and the proposed model is novel and effective. It could plausibly be relevant to sequence modeling tasks beyond NLP. I recommend acceptance.

There is one fairly serious writing issue that I'd like to see fixed, though: The abstract, introduction, and related work sections are all heavily skewed towards unsupervised learning. The paper doesn't appear to be doing unsupervised learning, and the ideas are no more nor less suited to unsupervised learning than any other mainstream ideas in the sentence encoding literature.

Details:
- You should be clearer about how you expect these embeddings to be used, since that will be of certain interest to anyone attempting to use the results of this work. In particular, how you should convert the matrix representation into a vector for downstream tasks that require one. Some of the content of your reply to my comment could be reasonably added to the paper.
- A graphical representation of the structure of the model would be helpful.
- The LSTMN (Cheng et al., EMNLP '16) is similar enough to this work that an explicit comparison would be helpful. Again, incorporating your reply to my comment into the paper would be more than adequate. 
- Jiwei Li et al. (Visualizing and Understanding Neural Models in NLP, NAACL '15) present an alternative way of visualizing the influence of words on sentence encodings without using cross-sentence attention. A brief explicit comparison would be nice here.
review_title: Strong, but some framing issues

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method for representing sentences as a 2d matrix by utilizing a self-attentive mechanism on the hidden states of a bi-directional LSTM encoder. This work differs from prior work mainly in the 2d structure of embedding, which the authors use to produce heat-map visualizations of input sentences and to generate good performance on several downstream tasks.

There is a substantial amount of prior work which the authors do not appropriately address, some of which is listed in previous comments. The main novelty of this work is in the 2d structure of embeddings, and as such, I would have liked to see this structure investigated in much more depth. Specifically, a couple important relevant experiments would have been:

* How do the performance and visualizations change as the number of attention vectors (r) varies?
* For a fixed parameter budget, how important is using multiple attention vectors versus, say, using a larger hidden state or embedding size?

I would recommend changing some of the presentation in the penalization term section. Specifically, the statement that "the best way to evaluate the diversity is definitely the Kullback Leibler divergence between any 2 of the summation weight vectors" runs somewhat counter to the authors' comments about this topic below.

In Fig. (2), I did not find the visualizations to provide particularly compelling evidence that the multiple attention vectors were doing much of interest beyond a single attention vector, even with penalization. To me this seems like a necessary component to support the main claims of this paper.

Overall, while I found the architecture interesting, I am not convinced that the model's main innovation -- the 2d structure of the embedding matrix -- is actually doing anything important or meaningful beyond what is being accomplished by similar attentive embedding models already present in the literature. Further experiments demonstrating this effect would be necessary for me to give this paper my full endorsement.
review_title: Interesting embedding method, lacking in analysis of 2d structure

final_score: rating: null
review_content: You mention that using KL divergence as a diversity regularizer is too computationally expensive, but presumably this is not true if r (the number of rows) is sufficiently small. Have you tried decreasing r to a small enough value so that you can utilize the KL penalty? If so, does it out perform the Frobenius norm penalty with the same (or larger) value of r?
review_title: Penalization term

final_score: rating: null
review_content: How do you use the sentence representation matrix M as the input to a classifier? Do you simply concatenate the rows into one long vector, and feed that into a softmax, or is there any kind of weight-sharing between the classifier parameters for different rows?

On another note, how do you see this work relating to the LSTMN? Your work has clear differences with theirs, but they share the idea of attending over a sequence without any outside input to the attention process, and they both work well on similar tasks.

Paper (EMNLP '16): https://arxiv.org/pdf/1601.06733v7.pdf
review_title: Two questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: I like the idea in this paper that use not just one but multiple attentional vectors to extract multiple representations for a sentence. The authors have demonstrated consistent gains across three different tasks Age, Yelp, & SNLI. However, I'd like to see more analysis on the 2D representations (as concerned by another reviewer) to be convinced. Specifically, r=30 seems to be a pretty large value when applying to short sentences like tweets or those in the SNLI dataset. I'd like to see the effect of varying r from small to large value. With large r value, I suspect your models might have an advantage in having a much larger number of parameters (specifically in the supervised components) compare to other models. To make it transparent, the model sizes should be reported. I'd also like to see performances on the dev sets or learning curves.

In the conclusion, the authors remark that "attention mechanism reliefs the burden of LSTM". If the 2D representations are effective in that aspect, I'd expect that the authors might be able to train with a smaller LSTM. Testing the effect of LSTM dimension vs $r$ will be helpful.

Lastly, there is a problem in the presentation of the paper in which there is no training objective defined. Readers have to read until the experimental sections to guess that the authors perform supervised learning and back-prop through the self-attention mechanism as well as the LSTM.

* Minor comments:
Typos: netowkrs, toghter, performd
Missing year for the citation of (Margarit & Subramaniam)
In figure 3, attention plotswith and without penalization look similar.
review_title: Interesting idea but need additional work to be convincing

Meta-review: A solid paper about sentence representation learning using "internal attention" (representing sentences as a self-attention matrix rather than a vector). The idea is novel, well presented, and has potential applications to a variety of sequence-based problems. The main criticism in reviews was about minor points needing clarification, but I am suitably satisfied that these have been acknowledged and incorporated into the current revision. Reviewer 2 is concerned that the experiments do not fully support the claim that this model will improve end task performance over more standard attention mechanisms. Glancing over the paper, and in light of other reviews and discussion, I am not convinced that this objection should halt acceptance, but invite the authors to take this suggestion onboard for future work (or in the final version of their paper). This paper should be accepted to the conference.
Decision: Accept (Poster)