Can recursive neural tensor networks learn logical reasoning?
Authors: Samuel R. Bowman
ICLR 2014

Abstract: Recursive neural network models and their accompanying vector representations for words have seen success in an array of increasingly semantically sophisticated tasks, but almost nothing is known about their ability to accurately capture the aspects of linguistic meaning that are necessary for interpretation or reasoning. To evaluate this, I train a recursive model on a new corpus of constructed examples of logical reasoning in short sentences, like the inference of 'some animal walks' from 'some dog walks' or 'some cat walks,' given that dogs and cats are animals. The results are promising for the ability of these models to capture logical reasoning, but the model tested here appears to learn representations that are quite specific to the templatic structures of the problems seen in training, and that generalize beyond them only to a limited degree.


Peer review:
[Review]:
The paper tries to determine whether representations constructed with recursive embeddings can be used to support simple reasoning operations.  The essential idea is to train an additional comparison layer that takes the representations of two sentences and produces an output that describes the relation between the two sentences (entailment, equivalence, etc.)  This approach is in fact closely related to the 'restricted entailment operator' suggested near the end of  Bottou's white paper http://arxiv.org/pdf/1312.6192v3.pdf.  Experiments are carried out using a vastly simplified language and Socher's supervised training technique. According to the author, the results are a mixed bag. On the one hand, the system can learn to reason on sentences whose structure matches that of the training sentences.  On the other hand, performance quickly degrades when using sentences whose structure did not appear in the training set.

My reading of these results is much more pessimistic. I find completely unsurprising that the system can learn to 'reason' on sentences with known structure. On the other hand, the inability of the system to reason on sentences with new structure indicates that the recursive embedding network did not perform what was expected.  The key of the recursive structure is to share weights across all applications of the grouping layer. This weight sharing was obviously insufficient to induce a bias that helps the system generalize to other structures. Whether this is a simple optimization issue or a more fundamental problem remains to be determined.

My understanding is that the author always trains the system using the correct parsing structure in a manner similar to Socher's initial work (please confirm).  It would be very interesting to investigate whether one obtains substantially different results if one trains the system using incorrect parsing structures (either a random structures or a left-to-right structure). Worse results would indicate that the structure of the recursive embeddings matters.  Similar results would confirm the results reported in http://arxiv.org/abs/1301.2811 and strongly suggest that recursive embeddings do not live up to expectations. This would of course be a negative results, but negative results are sometimes more informative than mixed bags (and in my opinion very worth publishing.)