Learning Type-Driven Tensor-Based Meaning Representations
Authors: Tamara Polajnar, Luana Fagarasan, Stephen Clark
ICLR 2014

Abstract: This paper investigates the learning of 3rd-order tensors representing the semantics of transitive verbs. The meaning representations are part of a type-driven tensor-based semantic framework, from the newly emerging field of compositional distributional semantics. Standard techniques from the neural networks literature are used to learn the tensors, which are tested on a selectional preference-style task with a simple 2-dimensional sentence space. Promising results are obtained against a competitive corpus-based baseline. We argue that extending this work beyond transitive verbs, and to higher-dimensional sentence spaces, is an interesting and challenging problem for the machine learning community to consider.


Peer review:
[Review]:
This paper presents a method for learning 3-modes tensor for representing meaning of transitive verbs. The choice of this representation for verbs is motivated by the Combinatory Categorical Grammar framework; nouns are represented by low-dimensional continuous vectors obtained by a modified SVD on a co-occurrence matrix. Verbs tensors are learned for 10 verbs using subject-verb-object triples extracted from Wikipedia  and Google N-grams. Evaluation is carried out by asking the system to predict the verb given a pair (subject, object).

---

The paper is clearly written and tackles an important and hot topic. The introduction and section 2 are actually very interesting to motivate this field of research: attempting to combine distributional and combinatorial semantics. However, I find the message a bit too ambitious and misleading w.r.t. the content of the remainder of the paper, and hence its actual contributions. Indeed, the paper ends up by learning representations for 10 verbs (nouns representations being obtained using methods developed in previous work). This might be an important first step, but this is not exactly what the introduction and title indicate.

The sentence of the introduction 'One goal of this paper is to introduce the problem of learning tensor-based semantic representations to the machine learning community' is a bit strong, since there exist many tensor learning work, perhaps not exactly in the same context but still, as well as the paper by Krishnamurthy and Mitchell from last year (cited in the paper). 

Besides, the learned tensors are quite strange since one slice has only 2 dimensions for predicting plausible/implausible probabilities. What is different from using a single value to predict the plausibility only? This would imply simply representing verbs by matrices, but I wonder if the result would be much different. In test, I assume that the ranking of verbs given pairs of (subject, object) is carried out by sorting using the plausibility probability. The role of the second value destined to encode for implausibility is unclear (and hence the role of tensors with a mode of dimension 2).

Even if using such tensors was fully justified, the paper needs to propose a comparison with a baseline encoding verbs as matrices (the current baseline is too weak). For instance, the paper by Jenatton et al. 'A latent factor model for highly multi-relational data' (NIPS 2012) proposes very similar experiments by also training on subject-verb-object extracted from Wikipedia to learn matrices for representing verb meanings. I think this system or an adaptation of it would make a much better baseline.

Other remarks:
- Which norm is used in Equation 1? L2-norm for tensors or matrix is ambiguous. How was set lambda?
- In Equation 2, f_{w_i , c_j} should be defined.
- It seems that there is a single N (and not 1 per noun). This should be made more clear.
- Figure 2 is rather unclear. What is K? The size of the noun vectors?
- In the beginning of Section 5, for the second experiments shouldn't it be 52 instead of 42?