Learning Type-Driven Tensor-Based Meaning Representations
Authors: Tamara Polajnar, Luana Fagarasan, Stephen Clark
ICLR 2014

Abstract: This paper investigates the learning of 3rd-order tensors representing the semantics of transitive verbs. The meaning representations are part of a type-driven tensor-based semantic framework, from the newly emerging field of compositional distributional semantics. Standard techniques from the neural networks literature are used to learn the tensors, which are tested on a selectional preference-style task with a simple 2-dimensional sentence space. Promising results are obtained against a competitive corpus-based baseline. We argue that extending this work beyond transitive verbs, and to higher-dimensional sentence spaces, is an interesting and challenging problem for the machine learning community to consider.


Peer review:
[Review]:
Dear Reviewers,

Thank you very much for your insightful comments. We respond directly
below to the comments of Reviewer 1, but these also cover most of the
points made by Reviewer 2.

> Why map to a two dimensional space, one axis of which models
> plausibility, and the other, implausibility? How is the point (1,1)
> in this space different from the point (0.5, 0.5) (i.e. what does
> simultaneously being 'more plausible' and 'less plausible' mean)? Wouldn't it be more natural, and simpler, to just map to the real
> line? (These questions also apply to Clark's paper from which this
> idea is inherited.)

In the compositional framework that we base our experiments on
(Coecke et al.), the sentence space is represented as a vector
resulting in a tensor representation for the verb, so part of the
reason we chose an overparameterised representation is to remain
faithful to this framework.

Both reviewers ask why we did not represent the verb as a matrix, with
the sentence space being a scalar. In fact we did try these
experiments, with the verb as a matrix and the output a scalar value
transformed by a sigmoid, resulting in a single-output two-class
classifier. This approach was not as effective as the softmax
overparameterised version, and to make the paper more concise we
omitted these results. We can add a line or two to the final version
to clarify this point.

> Also the baseline seems quite weak, as it is constructed from only
> positive examples. Furthermore the generalization performance of the
> resulting system was split by sentence, not by noun - that is, nouns
> that appear in the training set can also appear in the test set,
> suggesting other, simpler ways of constructing baselines (see
> below).

This baseline was chosen as it has precedent in the paper of
Grefenstette and Sadrzadeh (2011), the first paper discussing
experimental approaches for this compositional framework. In general,
methods in distributional semantics typically use positive-only data,
and hence we consider this to be a reasonable baseline. Furthermore,
we have implemented the alternative verb-as-a-matrix model, as
discussed above, and can present results for it if necessary.

> It seems a bit of a stretch to call the simple gradient descent
> method used here, 'deep learning'.

In the context of this simple experiment with the single logistic
regression node, we agree that the claim that this is deep learning is
overstated. Nevertheless, the aim of this work is just an initial
investigation, the ultimate goal of which is to investigate longer
sentences whose more complicated sentence structures would necessitate
the use of 'deep learning' techniques with more complex networks.

> Clarity: I think it would help the reader's understanding to explain
> how the string 'with' in 'eat with a fork' maps to the category
> ((SNP)(SNP))/NP.

We will clarify this point.

> The problem appears to decompose into very small decoupled
> optimization problems (an objective function for each verb) since
> the verb parameters are independent and the noun vectors are fixed
> (right?). It would have been interesting to allow the noun vectors
> to be learned also, perhaps starting from your initial values (but a
> much harder optimization problem).

This is an interesting suggestion, and has been explored in related
work (Andreas and Gharamani, 2013) and anecdotally by Socher.

> A little more clarification is needed here. From what I understand,
> you took, for each noun, the top N most relevant context words, each
> of which is also a vector, thus forming the 'noun context
> matrix'. But how were the vectors for the context words computed? In
> the same way as the nouns, thus giving a 10,000 dimensional vector
> for each context word? Also, when you do the SVD, you will wind up
> with a matrix representation (constructed from either the left
> singular vectors or the right, depending on how the problem was set
> up). How do you wind up with a 20 dimensional vector? If you just
> used the top 20 singular values, that would seem to throw away
> crucial information.

We constructed the vectors in a standard way for distributional
semantics, so we did not describe the method in great detail. The
context words are just used as indices (or rather basis vectors) of
the target word vectors. Furthermore, using SVD is a standard
technique to reduce noise and uncover the latent dimensions (e.g. see
Turney and Pantel). 20 is perhaps a little low compared with other
work (e.g. Socher uses 50), but with additional techniques including
context selection and normalisation (see our EACL-14 paper) we can get
away with 20 dimensions. We found that adding more dimensions did not
consistently improve results.

> There appears to be a flaw in the experimental protocol: no attempt
> was made (or at least, mentioned) to keep the nouns that appear in
> the test set, different from those that appear in the training
> set. If this is so it makes the results weaker, as methods that
> leverage such coincidences may do much better.  For example, we
> could construct a simple baseline for a given N1-V-N2 test triple by
> saying that if N1-V appears in the training set, and also if V-N2
> does too, then output '1', else output '0'. At least I think it's
> important to break results out by such a 'did noun occur in training
> set' stratification.

We ensure that the subject-verb-object *triples* are unique, but not
allowing any subject or object nouns in the test data to occur in the
training data we think is too restrictive (and would most likely
result in a small training set). Such a scenario would also be
unrealistic when moving to longer sentences.

Finally, we wish to stress the point that this paper is a first
attempt at implementing the tensor-based compositional framework, with
a simple sentence space. In future work our intention is to move to
more complex sentences and sentence spaces, for which the learning
methods used here will become more indispensable.