A Simple Model for Learning Multilingual Compositional Semantics
Authors: Karl Moritz Hermann, Phil Blunsom
ICLR 2014

Abstract: Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. In particular, recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning compositional representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.


Peer review:
[Review]:
This paper introduces an interesting model to learn single word vector embeddings for 2 languages simultaneously.
It is applied to a classification task.

The paper is very clear and well written.

It does not seem to actually learn compositional semantics in the usual sense:

http://en.wikipedia.org/wiki/Principle_of_compositionality
Principle of Compositionality is the principle that the meaning of a complex expression is determined by the meanings of its constituent expressions and the rules used to combine them.

Certainly, averaging all words in a bag of words is not a compositional rule that would allow people to retrieve the meaning. From wikipedia: 
'The principle of compositionality states that in a meaningful sentence, if the lexical parts are taken out of the sentence, what remains will be the rules of composition. Take, for example, the sentence 'Socrates was a man'. Once the meaningful lexical items are taken away—'Socrates' and 'man'—what is left is the pseudo-sentence, 'S was a M'. The task becomes a matter of describing what the connection is between S and M.'
The connection between S and M would not be retrievable from a bag of words representation.

On a related note, the model could not be used for (presumable the final goal of) machine translation in its current form.

It would be great to see a comparison with the work from the same lab of Kalchbrenner and Blunsom.

Despite its problems, it seems an interesting paper.