Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This work is interesting and fairly thorough. The ablation studies at the end of the paper are the most compelling part of the argument, more so than achieving SoTa. Having said that, since their studies on performance with a low dataset size are the most interesting part of the paper, I would have liked to see results on smaller datasets like RTE. Additionally, it would be useful to see results on MultiNLI which is more challenging and spans more domains; using world knowledge with MultiNLI would be a good test of their claims and methods. 
I'm also glad that the authors establish statistical significance! I would have liked to see some additional analysis on the kinds of sentences the KIM models succeeds at where their baseline ESIM fails. I think this would be a compelling addition.

Pros:-
- Thorough experimentation with ablation studies; show success of method when using limited training data.
- Establish statistical significance.
- Acheive SoTa on SNLI.

Cons:-
- Authors make the broad claim of world knowledge being helpful for textual entailment, and show usefulness in a limited datasize setting, but don't test their method on other datasets RTE (which has a lot less data). If this helps performance on RTE then this could be a technique for low resource settings.
- No results for MultiNLI shown. MultiNLI has many more domains of text and may benefit more from world knowledge.
- Authors don't provide a list of examples where KIM succeeds and ESIM fails.
review_title: Good paper, needs some more experiments.

final_score: rating: 3: Clear rejection
review_content: This paper adds WordNet word pair relations to an existing natural language inference model. Synonyms, antonyms, and non-synonymous sister terms in the ontology are represented using indicator features. Hyponymy and hypernymy are represented using path length features. These features are used to modify inter sentence attention, the final post-attention word representations, and the pooling operation used to aggregate the final sentence representations prior to inference. All of these three additions help, especially in the low data learning scenario. When all of the SNLI training data is used this approach adds 0.6% accuracy on the SNLI 3-way classification task. 

I think that the integration of structured knowledge representations into neural models is a great avenue of investigation. And I'm glad to see that WordNet helps. But very little was done to investigate different ways in which these data can be integrated. The authors mention work on knowledge base embeddings and there has been plenty of work on learning WordNet embeddings. An obvious avenue of exploration would compare the use of these to the use of the indicator features in this paper. Another avenue of exploration is the integration of more resources such as VerbNet, propbank, WikiData etc. An approach that works with all of these would be much more impressive as it would need to handle a much more diverse feature space than the 4 inter-dependent features introduced here.

Questions for authors:

Is the WordNet hierarchy bounded at a depth of 8? If so please state this and if not, what is the motivation of your hypernymy and hyponymy features?
review_title: This paper marginally improves performance on SNLI using a limited set of features indicating WordNet relations. The result is nice but predictable and the methods are not obviously applicable to other external forms of information. This contribution is not sufficient for ICLR.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This is a very interesting paper!
We are finally back to what has been already proven valid for NLI also know as RTE. External knowledge is important to reduce the amount of training material for NLI. When dataset were extremely smaller yet more complex, this fact has been already noticed and reported in many systems. Now, it is extremely important that it is has been started to be true also in NN-based models for NLI/RTE.

Hovever, the paper fails in describing the model with respect to the vast body of research in RTE. In fact, alignment is one of the basis for building RTE systems. Attention models are in fact extremely related to alignment and "KNOWLEDGE-ENRICHED CO-ATTENTION" is somehow a model that incorporates what has been already extensively used to align word pairs. 
Hence, models such as those described in the book "Recognizing textual entailment" can be extremely useful in modeling the same features in NN-based models, for example, "A Phrase-Based Alignment Model for Natural Language Inference", EMNLP 2008, or "Measuring the semantic similarity of texts", 2005 or "Learning Shallow Semantic Rules for Textual Entailment", RANLP, 2007.

The final issue is the validity of the initial claim. Is it really the case that external knowledge is useful? It appears that external knowledge is useful only in the case of restricted data (see Figure 3). Hence, it is unclear whether this is useful for the overall set. One of the important question here is then if the knowledge of all the data is in fact replicating the knowledge of wordnet. If this is the case, this may be a major result. 

Minor issues
=====
The paper would be easier to read if Fig. 1 were completed with all the mathematical symbols.
For example, where are a^c_i and b^c_i ? Are they in the attention box?
review_title: Alignment is back in NN

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update:

The response addressed all my major concerns, and I think the paper is sound. (I'm updating my confidence to a 5.) So, the paper makes an empirically *very* small step in an interesting line of language understanding research. This paper should be published in some form, but my low-ish score is due simply to my worry that ICLR is not the venue. I think this would be a clear 'accept' as a *ACL short paper, and would probably be viable as a *ACL long paper, but it will definitely have less impact on the overall field of representation learning than will the typical ICLR paper, so I can recommend it only with reservations.

--

This paper presents a method to use external lexical knowledge (word–word relations from WordNet) as an auxiliary input when solving the problem of textual entailment (aka NLI). The idea of accessing outside commonsense knowledge within an end-to-end trained model is one that I expect to be increasingly important in work on language understanding. This paper does not make that much progress on the problem in general—the methods here are quite specific to words and to NLI—and the proposed methods yields only yield large empirical gains in a reduced-data setting, but the paper serves as a well-executed proof of concept. In short, the paper is likely to be of low technical impact, but it's interesting and thorough enough that I lean slightly toward acceptance.

My only concern is on fair comparison: Numbers from this model are compared with numbers from the published ESIM model in several places (Table 3, Figure 2, etc.) as a way to provide evidence for the paper's core claim (that the added knowledge in the proposed model helps). This only constitutes clear evidence if the proposed model is identical to ESIM in all of its unimportant details—word representations, hyperparameter tuning methods, etc. Can the authors comment on this?

For what it's worth, the existence of another paper submission on roughly the same topic with roughly the same results (https://openreview.net/pdf?id=B1twdMCab) makes more confident that the main results in this paper are sound, since they've already been replicated, at least to a coarse approximation.

Minor points:

For TransE, what does this mean:"However, these kind of approaches usually need to train a knowledge-graph embedding beforehand."

You should say more about why you chose the constant 8 in Table 1 (both why you chose to hard code a value, and why that value).

There's a mysterious box above the text 'Figure 1'. Possibly a figure rendering error?

The LSTM equations are quite widely known. I'd encourage you to cite a relevant source and remove them.

Say more about why you choose equation (9). This notably treats all five relation types equally, which seems like a somewhat extreme simplifying assumption.

Equation (15) is confusing. Is a^m a matrix, since it doesn't have an index on it?

What is "early stopping with patience of 7"? Is that meant to mean 7 epochs?

The opening paragraph of 5.1 seems entirely irrelevant, as do the associated results in the results table. I suspect that this might be an opportunity for a gratuitous self-citation.

There are plenty of typos: "... to make it replicatibility purposes."; "Specifically, we use WordNet to measure the semantic relatedness of the *word* in a pair"; etc.
review_title: Interesting direction; modest positive results

Meta-review: the reviewers seem to agree that this submission could be much more strengthened if more investigation is done in two directions: (1) the effect of different, available resources (e.g., in the comment, the authors mentioned WikiData didn't improve, and this raises a question of what kind of properties of external resources are necessary to help) and (2) alternatives to incorporating external knowledge (e.g., as pointed out by one of the reviewers, this is certainly not the only way to do so, and external knowledge has been used by other approaches for RTE earlier. how does this specific way fare against those or other alternatives?) addressing these two points more carefully and thoroughly would make this paper much more appreciated.
Decision: Invite to Workshop Track