Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends the "order matters" idea in (Vinyals et al., 2015) from the sentence level to an interesting application on discourse level. Experiments in this paper show the capacity of the proposed model on both order discrimination task and sentence ordering.

I think the problem is interesting and the results are promising. However, there are some problems about technical details:

- Why there are two components of LSTM hidden state (h_{enc}^{t-1},c_{ent}^{t-1}), what information is captured by each of these hidden states? Refer to (Vinyals et al. 2015a)?
- Some notations in this paper are confusing. For example, what is the form of W in the feed-forward scoring function? Does it have the same form as the W in the bilinear score function?
- What is the connection between the encoder and decoder in the proposed model? How to combine them together? I read something relevant from the caption of Figure 1, but it is still not clear to me.

review_title: interesting problem and promising results

final_score: rating: null
review_content: It seems that the only information from the encoder used in the decoder is the final hidden state, which is used as the initial state in the decoder. Is this correct? Would it make sense to "merge" the encoder and decoder, by directly using h_{t, enc} in Eq. 7 instead of h_{t, dec}?

What scoring function (defined in section 3.1) is used in the experiments? (I believe it is the bilinear function, but cannot find it in the text). Is there a big difference between the two scoring functions?

In section 4.3.4, the authors mention that "when the sentence encoder is initialized with a good set of parameters [...] this compensates for the lack of an encoder and leads to better performance." Does this setting corresponds to the "RNN Decoder" line in Table 3? Does this mean that the difference of performance between "RNN Decoder" and "Proposed model" is due to better representations?
review_title: Intuition for encoder / decoder

final_score: rating: null
review_content: For the sentence ordering task, I am wondering whether you can compare with the document-context language models proposed in the following work: https://arxiv.org/pdf/1511.03962v4.pdf
review_title: Comparison

final_score: rating: 7: Good paper, accept
review_content: The main contribution of this paper is to introduce a new neural network model for sentence ordering. This model is presented as an encoder-decoder, and uses recent development for neural network (pointer networks and order invariant RNNs).

The first processing step of the model is to encode each sentence using a word level LSTM recurrent network. An order invariant encoder (based on Vinyals et al. (2015)) is then used to obtain a representation of the set of sentences. The input at each time step of this encoder is a "bag-of-sentences", over which attention probabilities are computed. The last hidden representation of the encoder is then used to initialize the decoder. This decoder is a pointer network, and is used to predict the order of the input sentences.

The model introduced in this paper is then compared to standard method for sentence ordering, as well as a model with the decoder only. The encoder-decoder approach outperforms the other methods on the task of ordering a given set of sentences. The authors also show that the model learns relatively good sentence representations (e.g. compared to Skip-thought vectors).

This paper is relatively well written, and easy to follow. The model described in the paper does not really introduce anything new, but is a natural combination of existing techniques to solve the task of sentence ordering. In particular, it uses an order-invariant encoder to get a representation of the set of sentences, and a pointer network to predict the ordering. While I am not entirely convinced by the task of sentence ordering, this approach seems promising to learn sentence representations.

Overall, this is a pretty solid and well executed paper.

pros:
 - sound model for sentence ordering
 - strong experimental results
cons:
 - might be a bit incremental
 - usefulness of sentence ordering
review_title: accept

final_score: rating: null
review_content: The definition of "accuracy" in Table 3 is different from that of "accuracy" in Table 2 in that the former is computed over each sentence's absolute predicted position while the later is computed over pairwise ranking between two alternative orderings. The potential downside of accuracy as defined in Table 3 is that it conflates two different sources of errors: (1) whether the model can correctly rank one ordering of paragraph over an alternative ordering (which corresponds to the accuracy in Table 2) and (2) whether the model can generate the sequence of ideal ordering in a greedy manner (through beam search). Given that most other baseline models are not designed to perform well for greedy sequential generation (i.e., (2)), it seems natural that they will perform poorly when the accuracy and Kendall's tau are computed over the ordering generated by each baseline method. I think this may be also why there's surprisingly big performance gap in Table 3, while the differences are much more modest in Table 2. Is it possible to report Table 2 style accuracy in Table 3 as well, by creating an identical set of pairwise ranking tasks across different methods?
review_title: accuracy

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper presents an empirical study on sentence ordering using RNN variants.

I’m not sure how strong novelty this paper brings in terms of technical contributions. The proposed method closely follows the read, process, and write framework of Vinyals et al. (2015) that has been developed for set-to-sequence type tasks. Sentence ordering is a good fit for set-to-sequence formulation, as the input sentences are rather a set than a sequence. Thus, the main contribution of this work can be viewed as a new application of an existing model rather than a new model development.

That said, the empirical evaluation of the paper is very solid and the authors have updated the supplementary material to strengthen the evaluation even further in response to the QAs.


review_title: null

Meta-review: Sentence ordering is central to a number of NLP tasks (e.g., summarization), and this paper introduces a very sensible application of existing techniques, experimentations is also very solid. However, the technical contributions seems quite limited.
 
 Positive:
 
 -- An important NLP problem (I would disagree here with Reviewer1), a good match between the problem and the methods
 -- Solid experiments 
 -- A well-written paper
 
 Negative:
 
 -- A straightforward application of existing methods
 
 Unfortunately, the latter, I believe, is a problem. Though application papers are welcome, I believe they should go beyond direct applications and provide extra insights to the representation learning community.
Decision: Reject