Can recursive neural tensor networks learn logical reasoning?
Authors: Samuel R. Bowman
ICLR 2014

Abstract: Recursive neural network models and their accompanying vector representations for words have seen success in an array of increasingly semantically sophisticated tasks, but almost nothing is known about their ability to accurately capture the aspects of linguistic meaning that are necessary for interpretation or reasoning. To evaluate this, I train a recursive model on a new corpus of constructed examples of logical reasoning in short sentences, like the inference of 'some animal walks' from 'some dog walks' or 'some cat walks,' given that dogs and cats are animals. The results are promising for the ability of these models to capture logical reasoning, but the model tested here appears to learn representations that are quite specific to the templatic structures of the problems seen in training, and that generalize beyond them only to a limited degree.


Peer review:
[Review]:
Thanks for your comments. I am updating the paper now with some clarifications and typo repairs, and I'm in the process of setting up a few follow up experiments.

Section 2: 

Thanks for pointing out the unclear bits here, especially that “some dogs bark” example, which I seem to have broken during some hasty final revisions. I'll post an updated version with this fixed shortly. 

To clarify some details here:
- “Some' is, in fact, upward monotone in both arguments.
- D is the domain of containing all possible objects of the type being compared.
- The “^” symbol in column three was typeset incorrectly, and is meant to represent logical AND.

Section 3: 

I did not try initializing the vectors with those used in any previous experiments (e.g. Socher’s or Mikolov’s). While that kind of initialization sounds promising in general, I think that the unambiguous fragment of English that I use is so different from ordinary English usage that it is unlikely that outside information from these sources would be helpful to the task. The pretraining settings that I experimented with involved first training the model on some or all of the pairs of individual words from Appendix B, annotated with the relations between them.

I'm certainly sensitive to the concern that the model might be overparameterized, and I will see about getting a training curve together in the next week or two.

Section 4: 

The numbers referenced in those subsections do refer to Table 2, but the '(as in 2)' reference is a mistake. Example 2 in Table 2 corresponds to 'Monotonicity with quantiﬁer substitution.” Thanks for catching that, and expect a fix soon.

Section 5:

I agree that the all-split result is unsurprising, though I think it is useful as a sanity check to ensure that the model structure is usable for the task, and that the model isn't dramatically *under*parameterized.

The three target datasets were chosen by hand: the choice of a fairly small number was necessary due to resource constraints, but the choices were arbitrary. I chose to focus on quantifier substation datasets so as to render the three settings (the last three columns of Table 4) most easily comparable across the three target datasets.

The reference to 'potentially other similar datasets” could have been better put, but it refers to the fact that in each of the three experimental settings reported in Table 4, different criteria are used to decide which datasets are held out in training, and that all of these criteria involve how similar a given dataset is to the target dataset.

You raise an important point about reproducibility. I would appreciate any suggestions about better ways to report results given the fluctuations during training. I may try to report statistics over the model’s performance over a range of iterations, or statistics over the model’s performance at a given iteration over several random re-initializations, and I am experimenting further with different ways of encouraging the model to converge. 

I would like to suggest that even the current results do show a broader reproducible pattern: high performance on SET-OUT and SUBCL.-OUT is possible but subject to instabilities in the training algorithm, whereas high performance on PAIR-OUT cannot be demonstrated with this model as configured.