Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents simple but useful ideas for improving sentence embedding by drawing from more context. The authors build on the skip thought model where a sentence is predicted conditioned on the previous sentence; they posit that one can obtain more information about a sentence from other "governing" sentences in the document such as the title of the document, sentences based on HTML, sentences from table of contents, etc. The way I understand it, previous sentence like in SkipThought provides more local and discourse context for a sentence whereas other governing sentences provide more semantic and global context.

Here are the pros of this paper:
1) Useful contribution in terms of using broader context for embedding a sentence.
2) Novel and simple "trick" for generating OOV words by mapping them to "local" variables and generating those variables.
3) Outperforms SkipThought in evals.

Cons:
1) Coreference eval: No details are provided for how the data was annotated for the coreference task. This is crucial to understanding the reliability of the evaluation as this is a new domain for coreference. Also, the authors should make this dataset available for replicability. Also, why have the authors not used this embedding for eval on standard coreference datasets like OntoNotes. Please clarify.
2) It is not clear to me how the model learns to generate specific OOV variables. Can the authors clarify how does the decoder learns to generate these words.

Clarifications:
1) In section 6.1, what is the performance of skip-thought with the same OOV trick as this paper?
2) What is the exact heuristic in "Text Styles" in section 3.1? Should be stated for replicability.
review_title: Good paper on extracting more signal from document structure

final_score: rating: 5: Marginally below acceptance threshold
review_content: 1) This paper proposes a method for learning the sentence representations with sentences dependencies information. It is more like a dependency-based version skip-thought on the sentence level. The idea is interesting to me, but I think this paper still needs some improvements. The introduction and related work part are clear with strong motivations to me. But section 4 and 6 need a lot of details. 

2) My comments are as follows:
i) this paper claims that this is a general sentence embedding method, however, from what has been described in section 3, I think this dependency is only defined in HTML format document. What if I only have pure text document without these HTML structure information? So I suggest the authors do not claim that this method is a "general-purpose" sentence embedding model.

ii) The authors do not have any descriptions for Figure 3.  Equation 1 is also very confusing.

iii) The experiments are insufficient in terms of details. How is the loss calculated? How is the detection accuracy calculated?

review_title: borderline

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper extends the idea of forming an unsupervised representation of sentences used in the SkipThought approach by using a broader set of evidence for forming the representation of a sentence. Rather than simply encoding the preceding sentence and then generating the next sentence, the model suggests that a whole bunch of related "sentences" could be encoded, including document title, section title, footnotes, hyperlinked sentences. This is a valid good idea and indeed improves results. The other main new and potentially useful idea is a new idea for handling OOVs in this context where they are represented by positional placeholder variables. This also seems helpful. The paper is able to show markedly better results on paraphrase detection that skipthought and some interesting and perhaps good results on domain-specific coreference resolution.

On the negative side, the model of the paper isn't very excitingly different. It's a fairly straightforward extension of the earlier SkipThought model to a situation where you have multiple generators of related text. There isn't a clear evaluation that shows the utility of the added OOV Handler, since the results with and without that handling aren't comparable. The OOV Handler is also related to positional encoding ideas that have been used in NMT but aren't reference. And the coreference experiment isn't that clearly described nor necessarily that meaningful. Finally, the finding of dependencies between sentences for the multiple generators is done in a rule-based fashion, which is okay and works, but not super neural and exciting.

Other comments:
 - p.3. Another related sentence you could possibly use is first sentence of paragraph related to all other sentences? (Works if people write paragraphs with a "topic sentence" at the beginning.
 - p.5. Notation seemed a bit non-standard. I thought most people use \sigma for a sigmoid (makes sense, right?), whereas you use it for a softmax and use calligraphic S for a sigmoid....
 - p.5. Section 5 suggests the standard way to do OOVs is to average all word vectors. That's one well-know way, but hardly the only way. A trained UNK encoding and use of things like character-level encoders is also quite common.
 - p.6. The basic idea of the OOV encoder seems a good one. In domain specific contexts, you want to be able to refer to and re-use words that appear in related sentences, since they are likely to appear again and you want to be able to generate them. A weakness of this section however is that it makes no reference to related work whatsoever. It seems like there's quite a bit of related work. The idea of using a positional encoding so that you can generate rare words by position has previously been used in NMT, e.g. Luong et al. (Google brain) (ACL 2015). More generally, a now quite common way to handle this problem is to use "pointing" or "copying", which appears in a number of papers. (e.g., Vinyals et al. 2015) and might also have been used here and might be expected to work too. 
 - p.7. Why such an old Wikipedia dump? Most people use a more recent one!
 - p.7. The paraphrase results seem good and prove the idea works. It's a shame they don't let you see the usefulness of the OOV model.
 - p.8. For various reasons, the coreference results seem less useful than they could have been, but they do show some value for the technique in the area of domain-specific coreference.


review_title: A good practical extension of SkipThought, but fairly straightforward

Meta-review: The paper presents an interesting extension of the SkipThought idea by modeling sentence embeddings using several document-structure related information.  Out of the various kinds of evaluations presented, the coreference results are interesting -- but, they fall short by a bit (as noted by Reviewer 2) because they don't compare with recent work by Kenton Lee et al.  In summary, the idea provides an interesting bit on building sentence embeddings, but the experimental results could have been stronger.
Decision: Reject