Learning Type-Driven Tensor-Based Meaning Representations
Authors: Tamara Polajnar, Luana Fagarasan, Stephen Clark
ICLR 2014

Abstract: This paper investigates the learning of 3rd-order tensors representing the semantics of transitive verbs. The meaning representations are part of a type-driven tensor-based semantic framework, from the newly emerging field of compositional distributional semantics. Standard techniques from the neural networks literature are used to learn the tensors, which are tested on a selectional preference-style task with a simple 2-dimensional sentence space. Promising results are obtained against a competitive corpus-based baseline. We argue that extending this work beyond transitive verbs, and to higher-dimensional sentence spaces, is an interesting and challenging problem for the machine learning community to consider.


Peer review:
[Review]:
The authors investigate learning tensor representations of verbs using fixed vector representations of nouns, building on Steedman's CCGs and Coecke's work on modeling meaning representations using linear maps (so that, for example, a verb 3-tensor is right-contracted with its object noun vector and left-contracted with its subject noun vector, to output a sentence living in another vector space). In particular, they learn the noun/verb/noun mapping, turning the problem into a binary classification problem (where the binary labels correspond to legal combinations, and impostor combinations), using L2 regularized logistic regression.  Results are given on ten verbs, chosen for varying levels of frequency and 'concreteness', and using up to 2000 positive and 2000 negative N/V/N training sentences, constructed using the Google N-Gram corpus and Wikipedia.  The results are compared to a baseline where the verb tensors are constructed using the outer product of the vector representations of their subject and object nouns, for legal uses of that verb. Throughout the work the nouns are represented by fixed vectors using a statistical co-occurrence technique based on that of Turney and Pantel. The proposed technique beats the proposed baseline in accuracy.

This is an interesting and useful direction for research: to try to construct linear maps that more faithfully represent the functional aspects of language (e.g. a verb is a function that takes two nouns and outputs a sentence).  I wonder, however, why the authors didn't just treat the problem as a standard classification problem.  Why map to a two dimensional space, one axis of which models plausibility, and the other, implausibility? How is the point (1,1) in this space different from the point (0.5, 0.5) (i.e. what does simultaneously being 'more plausible' and 'less plausible' mean)?  Wouldn't it be more natural, and simpler, to just map to the real line?  (These questions also apply to Clark's paper from which this idea is inherited.)  Also the baseline seems quite weak, as it is constructed from only positive examples.  Furthermore the generalization performance of the resulting system was split by sentence, not by noun - that is, nouns that appear in the training set can also appear in the test set, suggesting other, simpler ways of constructing baselines (see below).  I do think the paper has some value as a step towards investigating more general linear maps to model language; the link between CCGs and linear maps seems intuitive.


Specific Comments
*****************

Numbers are section numbers.

1.0

It seems a bit of a stretch to call the simple gradient descent method used here, 'deep learning'.

2.0

Clarity: I think it would help the reader's understanding to explain how the string 'with' in 'eat with a fork' maps to the category ((SNP)(SNP))/NP.

3.0

Again, why not use a one dimensional representation for plausibility?  What is the intuition for using two dimensions?

3.1

The problem appears to decompose into very small decoupled optimization problems (an objective function for each verb) since the verb parameters are independent and the noun vectors are fixed (right?).  It would have been interesting to allow the noun vectors to be learned also, perhaps starting from your initial values (but a much harder optimization problem).

3.2

The baseline - cosine similarity on the matrix elements of the Kronecker product matrix and that of the target subject/object pair - seems very ad-hoc, and so it's hard to see what we learn by comparing against it.

4.2

A little more clarification is needed here.  From what I understand, you took, for each noun, the top N most relevant context words, each of which is also a vector, thus forming the 'noun context matrix'.  But how were the vectors for the context words computed?  In the same way as the nouns, thus giving a 10,000 dimensional vector for each context word? Also, when you do the SVD, you will wind up with a matrix representation (constructed from either the left singular vectors or the right, depending on how the problem was set up). How do you wind up with a 20 dimensional vector?  If you just used the top 20 singular values, that would seem to throw away crucial information. 

5.0

There appears to be a flaw in the experimental protocol: no attempt was made (or at least, mentioned) to keep the nouns that appear in the test set, different from those that appear in the training set.  If this is so it makes the results weaker, as methods that leverage such coincidences may do much better.  For example, we could construct a simple baseline for a given N1-V-N2 test triple by saying that if N1-V appears in the training set, and also if V-N2 does too, then output '1', else output '0'.  At least I think it's important to break results out by such a 'did noun occur in training set' stratification.