Can recursive neural tensor networks learn logical reasoning?
Authors: Samuel R. Bowman
ICLR 2014

Abstract: Recursive neural network models and their accompanying vector representations for words have seen success in an array of increasingly semantically sophisticated tasks, but almost nothing is known about their ability to accurately capture the aspects of linguistic meaning that are necessary for interpretation or reasoning. To evaluate this, I train a recursive model on a new corpus of constructed examples of logical reasoning in short sentences, like the inference of 'some animal walks' from 'some dog walks' or 'some cat walks,' given that dogs and cats are animals. The results are promising for the ability of these models to capture logical reasoning, but the model tested here appears to learn representations that are quite specific to the templatic structures of the problems seen in training, and that generalize beyond them only to a limited degree.


Peer review:
[Review]:
In this work the author investigates how effective the vector representation of words is for the task of logical inference. A set of seven entailment relations from MaCartney are used, and a data set of 12,000 logical statements (of pairs of sentences) are generated from these relations and from 41 predicate tokens.  The task is multiclass classification, where given two sentences, the system must output the correct relation between them. A simple recursive tensor network is used. The study is limited to considering quantifiers like 'some' and 'all', which have clear monotonicity properties.  Results show that the model can learn but that generalization is limited.  Unfortunately because the training process converges to an inferior model, results are given after very early stopping, in which subsequent iterations can give widely different results.

This is an exciting direction for research and it's great to see it being tackled. Unfortunately, however, the paper is unclear in crucial places, and the training methodology is questionable.  I would encourage the author to clarify the paper (especially for the likely non-linguist audience) and strengthen the training algorithm (in order to demonstrate usefully reproducible results).  Even if the results remain negative, this would then still be of significant value to the community.

Specific comments:

Section 2
---------

Your example of 'some dogs bark' seems confused.  For both arguments of 'some' (not just the first), the inference works if the argument is replaced by something more general. You write that 'some' is downward monotonic in its second argument, but your examples show upward monotonicity in both. (Specifically, you write 'The quantifier 'some' is upward monotone in its first argument because it permits substitution of more general terms, and downward monotone in its second argument because it permits the substitution of more specific terms.' - but in the same paragraph you also write that 'some' is upward monotonic in both arguments.) Readers who are asked to expend mental energy on disentangling unnecessary confusions like this can quickly lose motivation.

Table 1: this table is central to your work, but it needs more explanation.  What is calligraphic D?  You seem to be using the 'hat' operator in two different senses (column 2 versus column 3). What does 'else' mean in column 3 - how exactly is independence defined? 

The whole paper rests on MacCartney's framework, so I think it's necessary to explain more about this scheme here. In particular, I do not understand your 'no animals bark | some dogs bark' example (and I fear most others won't, too).

Typo: much hold --> must hold

Section 3
---------

'Several pretraining regimes meant to initialize the word vectors to reflect the relations between them were tried, but none offered a measurable improvement to the learned model' - please say which were tried.  In particular, did you try fixed, off-the-shelf vectors, for example from Socher's work, or trained using a large unlabeled dataset using Mikolov's Word2Vec?

I counted 4624 parameters in the composition parameters, 13,005 for the comparison layer (with dimension 45), and 800 for the 50 (16 dimensional) word vectors, giving a total of 18,429 parameters.  Your training set size is quite a bit smaller than this, and regularization can only help so much.  I wonder if the limited results and difficulty of training (converging, but to poor solutions) just indicates the need for more training data. You can test this by generating a training curve - that is, plot performance on a validation set for various training set sizes (when trained to convergence, and using whatever regularization you settle on).  If the curve is still steep when using all the training data, then more data will help.  If it's flat, then the task may not be learnable with the model used.

Section 4
---------

Your description of the 'basic monotonicity' datasets was unclear to me.  Does 1, 2 refer to Table 2?  If so it's not clear how 'In some of the datasets (as in 1), this alternation is in the first argument, in some the second argument (as in 2), and in some both.'

Section 5
---------

It is not very surprising that the model can learn the all-split data, since the training data is so tightly constrained.  Set-out is also very close to the training data.

I found that exactly how the data splits were done, and what was tested on, was unclear.

typo: 'the it is'

'I choose one of three target datasets' - how did you choose the three?  (From the 200?)

'potentially other similar datasets...' is imprecise.  How did you choose?

'The model did not converge well for any of these experiments: convergence can take hundreds or thousands of passes through the data, and the performance of the model at convergence on test data was generally worse than its performance during the first hundred or so iterations. To sidestep this problem somewhat, I report results here for the models learned after 64 passes through the data.'  I'm afraid that this greatly reduces the value of these results (they are close to being irreproducible).  The training algorithm should at least converge, or be more reproducible than this.  (If the test error is still fluctuating wildly on the stopping iteration, other small changes, e.g. in the data, may give completely different results).

Section 6
---------

'Pessimistically... Optimistically... ' this is speculation (neither is supported by the experiments) and so I don't think it adds much value.