Learning Type-Driven Tensor-Based Meaning Representations
Authors: Tamara Polajnar, Luana Fagarasan, Stephen Clark
ICLR 2014

Abstract: This paper investigates the learning of 3rd-order tensors representing the semantics of transitive verbs. The meaning representations are part of a type-driven tensor-based semantic framework, from the newly emerging field of compositional distributional semantics. Standard techniques from the neural networks literature are used to learn the tensors, which are tested on a selectional preference-style task with a simple 2-dimensional sentence space. Promising results are obtained against a competitive corpus-based baseline. We argue that extending this work beyond transitive verbs, and to higher-dimensional sentence spaces, is an interesting and challenging problem for the machine learning community to consider.


Peer review:
[Review]:
This paper introduces a method to train very low 2-dimensional representations for different syntactic types and evaluates it on a benchmark of selectional preferences.


Regarding
'assume that words, phrases and sentences all live in the
same vector space. The tensor-based semantic framework is more flexible, in that it allows different
spaces for different grammatical types, which results from it being tied more closely to a type driven
syntactic description; however, this flexibility comes at a price, since there are many more
paramaters to learn.'
In fact, for non-toy tasks, the number of parameters will be too many. Realistically, to capture cooccurence counts of single words, single vectors need to be at least 100 dimensions, which will then require 1000000 parameters to represent a simple adverb.

Furthermore, one can argue that this particular tensor based framework is actually less flexible in that it could not compare the semantic similarity between simple phrases such as 'a rainy day', 'raining all day' and 'sunshine' or 'swim', 'swimmer' and 'swimming'. People are able to easily determine how similar or related these concepts are and a framework that assigns them representations of different dimensionality is not flexible enough to do this.

Regarding:
'plausibility space to have two dimensions, one corresponding to plausible and one corresponding to implausible.'
This could be simply captured by a single number that is high for plausible and low for implausible?
In fact, in standard machine learning a 2 class softmax (which is being used in this paper) is usually collapsed to a single sigmoid unit.

Since the MEN dataset has just been introduced 2 years ago and it is not very well known, it would be great if you described it in your paper.

Section 5.1 mentions problems with rare words.
How could this approach be extended to learn representations for rare combinations or words? People can certainly understand a word from very few examples.
On a related note, could this approach ever learn representations for all syntactic types in setups that move beyond trigrams? Presumably, the counts for most such n-tuples will be extremely sparse.

Since you used techniques from neural networks to train your model, why not compare to neural networks to predict the same statistics?
How was your SVD word vector representations cross validated? Did you try alternative word vectors?

This paper works in the interesting area of combining discrete syntactic information with distributed representations.
It is well written but some doubts remain about the approach.

While the paper is not very novel (approach A + approach B) it may expose the ml community to type and syntactic approaches.