Can recursive neural tensor networks learn logical reasoning?
Authors: Samuel R. Bowman
ICLR 2014

Abstract: Recursive neural network models and their accompanying vector representations for words have seen success in an array of increasingly semantically sophisticated tasks, but almost nothing is known about their ability to accurately capture the aspects of linguistic meaning that are necessary for interpretation or reasoning. To evaluate this, I train a recursive model on a new corpus of constructed examples of logical reasoning in short sentences, like the inference of 'some animal walks' from 'some dog walks' or 'some cat walks,' given that dogs and cats are animals. The results are promising for the ability of these models to capture logical reasoning, but the model tested here appears to learn representations that are quite specific to the templatic structures of the problems seen in training, and that generalize beyond them only to a limited degree.


Peer review:
[Review]:
I have a fairly major unexpected update to report. In attempting to respond to 8e44's concerns about using early stopping to get around convergence issues, I discovered a mistake in my implementation of AdaGrad. In short, fixing that mistake led to much more consistent convergence and better results, including strong performance on the PAIR-OUT test settings, suggesting that the model is much more capable than I had previously suggested at generalizing to unseen reasoning patterns. 

I realize that this is somewhat late in the review process to make substantial changes, but a new version of the paper is pending on ArXiv and should be live by Monday. The results table and the discussion section have been replaced. I will also be updating the source code linked to above and in the paper before Monday to reflect this bug fix, and a couple of small improvements to the way that cost and test error is reported during training.

If you are interested in what went wrong: I accidentally set up SGD with AdaGrad in such a way that it the sum of squared gradients after every full pass of the data, equivalent to every few hundred gradient updates. Since this sum is used to limit the size of the gradient updates, resetting it this often prevented the model from reliably converging without hurting gradient accuracy or preventing it from converging occasionally.