Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Overall, the paper is well-written and the proposed model is quite intuitive. Specifically, the idea is to represent entailment as a product of continuous functions over possible worlds. Specifically, the idea is to generate possible worlds, and compute the functions that encode entailment in those worlds. The functions themselves are designed as tree neural networks to take advantage of logical structure. Several different encoding benchmarks of the entailment task are designed to compare against the performance of the proposed model, using a newly created dataset. The results seem very impressive with > 99% accuracy on tests sets.

One weakness with the paper was that it was only tested on 1 dataset. Also, should some form of cross-validation be applied to smooth out variance in the evaluation results. I am not sure if there are standard "shared" datasets for this task, which would make the results much stronger.
Also how about the tradeoff, i.e., does training time significantly increase when we "imagine" more worlds. Also, in general, a discussion on the efficiency of training the proposed model as compared to TreeNN would be helpful.
The size of the world vectors, I would believe is quite important, so maybe a more detailed analysis on how this was chosen is important to replicate the results.
This problem, I think, is quite related to model counting. There has been a lot of work on model counting. a discussion on how this relates to those lines of work would be interesting.


After revision

I think the authors have improved the experiments substantially.
review_title: The paper proposes a new model to use deep models for detecting logical entailment

final_score: rating: 7: Good paper, accept
review_content: SUMMARY 

The paper is fairly broad in what it is trying to achieve, but the approach is well thought out. The purpose of the paper is to investigate the effectiveness of prior machine learning methods with predicting logical entailment and then provide a new model designed for the task. Explicitly, the paper asks the following questions: "Can neural networks understand logical formula well enough to detect entailment?", and "Which architectures are best at inferring, encoding, and relating features in a purely structural sequence-based problem?". The goals of the paper is to understand the learning bias of current architectures when they are tasked with learning logical entailment. The proposed network architecture, PossibleWorldNet, is then viewed as an improvement on an earlier architecture TreeNet.

POSITIVES 

The structure of this paper was very well done. The paper attempts to do a lot, and succeeds on most fronts. The generated dataset used for testing logical entailment is given a constructive description which allows for future replication. The baseline benchmark networks are covered in depth and the reader is provided with a deep understanding on the limitations of some networks with regard to exploiting structure in data. The PossibleWorldNets is also given good coverage, and the equations provided show the means by which it operates.
• A clear methodological approach to the research. The paper covers how they created a dataset which can be used for logical entailment learning, and then explains clearly all the previous network models which will be used in testing as well as their proposed model.
• The background information regarding each model was exceptionally thorough. The paper went into great depth describing the pros and cons of earlier network models and why they may struggle with recognizing logical entailment.
• The section describing the creation of a dataset captures the basis for the research, learning logical entailment. They describe the creation of the data, as well as the means by which they increase the difficulty for learning.
• The paper provides an in depth description of their PossibleWorldNet model, and during experimentation we see clear evidence of the models capabilities.

NEGATIVES

One issue I had with the paper is regarding the creation of the logical entailment dataset. Not so much for how they explained the process of creating the dataset, that was very thorough, but the fact that this dataset was the only means to test the previous network models and their new proposed network model. I wonder if it would be better to find non-generated datasets which may contain data that have entailment relationships. It is questionable if their hand crafted network model is learned best on their hand crafted dataset.

The use of a singular dataset for learning logical entailment. The dataset was also created by the researchers for the express purpose of testing neural network capacity to learn logical entailment. I am hesitant to say their proposed network is an incredible achievement since PossibleWorldNet effectively beat out other methods on a dataset that they created expressly for it.

RELATED WORK 

The paper has an extensive section dedicated to covering related work. I would say the research involved was very thorough and the researchers understood how their method was different as well as how it was improving on earlier approaches.

CONCLUSION 

Given the thorough investigation into previous networks’ capabilities in logical entailment learning, I would accept this paper as a valid scientific contribution. The paper performs a thorough analysis on the limitations that previous networks face with regard to exploiting structure from data. The paper also covers results of the experiments by not only pointing out their proposed network’s success, but by analyzing why certain earlier network models were able to achieve competitive learning results. The structure of the PossibleWorldNet was also explained well, and during ex- perimentation demonstrated its ability to learn structure from data. The paper would have been improved through testing of multiple datasets, and not just on there self generated dataset, but the contribution of their research on their network and older networks is still justification enough for this paper.
review_title: Deep learning with entailment 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This is a wonderful and a self-contained paper. In fact, it introduces a very important problem and it solves it. 

The major point of the paper is demonstrating that it is possible to model logical entailment in neural networks. Hence, a corpus and a NN model are introduced. The corpus is used to demonstrate that the model, named PossibleWorld, is nearly perfect for the task. A comparative analysis is done with respect to state of the art recurrent NN. So far, so good.

Yet, what is the take home message? In my opinion, the message is that generic NN should not be used for specific formal tasks whereas specific neural networks that model the task are desirable. This seems to be a trivial claim, but, since the PossibleWorld nearly completely solves the task, it is worth to be investigated. 

The point that the paper leaves unexplained is: what is in the PossibleWorld Network that captures what we need? The description of the network is in fact very criptic. No examples are given and a major effort is required to the reader. Can you provide examples and insights on why this is THE needed model?

Finally, the paper does not discuss a large body of research that has been done in the past by Plate. Plate has investigated how symbolic predicates can be described in distributed representations. This is strictly related to the problem this paper investigates. As discussed in "Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey", 2017, the link between symbolic and distributed representations has to be better investigated in order to propose innovative NN models. Your paper can be one of the first NN model that takes advantage of this strict link.
review_title: A solved task?

Meta-review: This paper studies the problem of modeling logical structure in a neural model.  It introduces a data set for probing various existing models and proposes a new model that addresses shortcomings in existing ones.  The reviewers point out that there is a bit of a tautology in introducing a new task and a new model that solves it.  The revised version addresses some of those concerns.  Overall, it is a thought-provoking and well-written study that will be interesting to discuss at ICLR.
Decision: Accept (Poster)