Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes an interesting approach towards learning NLI via parameter free operations over pairs of sentence embeddings. The authors propose entailment and contradiction operators that learn entailment and contradiction scores while training the parameters of the sentence encoders.

This is an interesting approach, however the experiments make me doubt the effectiveness of the proposed method. Admittedly, the authors do point out that at the cost of fewer training parameters, the proposed approach attains the same performance as NLI encoders with MLP based or attention based classifiers. However, the question becomes, how many fewer parameters are being learned to accept a performance that is in the same ball park but that does not exceed the SOTA.

The authors should have tried to an ablation type of approach in equation 1, to check if concatenation alone, element wise dot product alone or absolute difference alone or a combination of any two would work better with the scoring function.

This paper while taking a step in the right direction, seems a little premature for publication. That being said, the reported results my be of some value after all. It is hard to narrow down on the exact contributions of this paper.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
***Update***
I'd like to thank the authors for responding to my questions and for the additional experiments. I think the new sentence embedding experiments make the paper quite a bit stronger - it would be interesting to scale them up to using SNLI + MNLI to see how much further they can go (right now they are still below Sentence-BERT which also was trained on SNLI in addition to MNLI). I think this paper is an interesting idea, and my only other main concern is the transfer results to other NLI datasets. I think it would be a good idea to confirm that the difference between the two approaches is due to biases perhaps though an error analysis. I am borderline on this paper, but I feel enough improvement has been made to raise my rating.


This paper proposes a new way to train sentence embedding models using NLI data such that very few parameters are used for classification. This is in contrast to prior work where an MLP is used. They define an interaction layer where operations are applied to the sentence embeddings to produce just 5 scores, which are then fed into a softmax layer for the final prediction.

The reason for their approach is they hypothesize that the the classification layers are encoding some of the information, presumably lessening the amount of information in the sentence embeddings and also preventing them from having a direct interpretation.

Their model does surprisingly well on the entailment datasets, only about 1-1.5 points or so worse than using a MLP, and so they indeed demonstrate that the sentence embeddings contain a lot of entailment information. However, I do have some concerns about their claims that this helps in transfer learning to other NLI tasks. Their main results show transfer performance comparing their approach and using an MLP, but it seems that overall, on all datasets, their approach transfers more poorly. Two datasets that they do worse on they try to discount for either having terrible, below random performance (which is true) and the other for having the same biases as MNLI. However, if that was the case, I don't see why their model would perform worse since both theirs and the baseline would benefit from having these biases, but their model performs about 11 points less.

So therefore, I don't find the transfer experiments convincing, though it is interesting how different the models do on some of these tasks - model performance is surprisingly task dependent 

What I propose is for the authors to investigate if their sentence embeddings are in fact noticeably different than the ones trained in the more conventional matter. They could evaluate on sentence embedding and probing tasks (like SentEval) and see how the two models compare. It would be interesting to see wha encoded information differs between the models.

Im summary, I think this is an interesting experiment and it's nice to see that the MLP isn't doing a lot of heavy lifting (which also might be slightly counter to their hypothesis about the MLP containing a lot of entailment information). However, I find the transfer experiments unconvincing and the paper is short on analysis about when their model does better on transfer and when having an MLP helps, or how the learned sentence embeddings of the two models differ.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a few heuristic scorers to model entailment and contradiction, based on encoded sentence embeddings.

These scores include an entailment score, a contradiction score, a neutral score, and two similarity scores. They are defined heuristically, e.g., entailment score = geometric avg of such thing: 1 - sigma(premise not satisfied) * sigma(hypothesis satisfied). This is similar to fuzzy logic (for example, https://en.wikipedia.org/wiki/Fuzzy_logic) and some citations are needed in this regard.

Different from fuzzy logic, this paper learns whether an anonymous feature is true or false by NN encoders end-to-end. Thus, the model actually has enough power to extract those features suitable for fuzzy logic-like heuristic matching.

The experiments are well designed. I especially appreciate the comparison to random matching heuristics, which already exhibits non-trivial performance. This is very reasonable because the neural network underlying random matching heuristics is still learnable. However, the proposed matching heuristics achieve 7% improvement compared with random ones, showing the effectiveness of the approach. 

The authors also have ablation test and experiments on out-of-domain datasets. 

I have two concerns:

1. One limitation of this paper is that the heuristic matching scorers are pretty ad hoc to the inference task. The two similarity scores are not too novel, for example, sim_diff is the L1-distance between two vectors. Entailment, contradiction, and neutral scores are interesting, but hardly generalize to other sentence matching tasks (e.g., various IR applications). 

2. I have a feeling that the importance of NLI is over-estimated. While logical reasoning is important in AI, NLI datasets are somehow degenerated, and existing solutions are basically connecting neural edges. As mentioned in the paper, NLI models do not transfer well to out-of-domain NLI samples, not to mention non-NLI tasks. It would be interesting to see if the well-designed heuristic matching scores could ease the underlying model, so that it learns more generic sentence embeddings in general. 


Minor:

In references: Williams, Nagnia, Bowman: duplicate entry

review_title: Official Blind Review #3

Meta-review: This paper proposes a method for learning sentence embeddings such that entailment and contradiction relationships between sentence pairs can be inferred by a simple parameter-free operation on the vectors for the two sentences.

Reviewers found the method and the results interesting, but in private discussion, couldn't reach a consensus on what (if any) substantial valuable contributions the paper had proven. The performance of the method isn't compellingly strong in absolute or relative terms, yielding doubts about the value of the method for entailment applications, and the reviewers didn't see a strong enough motivation for the line of work to justify publishing it as a tentative or exploratory effort at ICLR.
Decision: Reject