A Simple Model for Learning Multilingual Compositional Semantics
Authors: Karl Moritz Hermann, Phil Blunsom
ICLR 2014

Abstract: Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. In particular, recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning compositional representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.


Peer review:
[Review]:
The paper considers learning cross-lingual representations of words using parallel data aligned at the level of sentences. A representation of a sentence is just a sum of embeddings of words in the sentence. These representations for pairs of sentences are learned to be similar. Specifically,  a ranking objective is used: for every sentence x in L1, the representation of the aligned sentence in L2 should be closer to the representation of x  than to representations of (a random sample of) other sentences in L2.  The resulting representations are used in transferring  a document classifier across languages (without retraining - i.e.  so called 'direct transfer').  Interestingly, the results are (mostly) better than these with cross-lingual word embeddings of Klementiev et al. (2012) learned using automatically word aligned sentences (as well as significantly better than a machine translation baseline -- which applies the classifier to automatically translated documents). 

I find the paper quite interesting and well written. The results are fairly impressive as well. 

However, I am not entirely convinced that calling this 'multilingual compositional semantics' is very appropriate. Though I agree that a more complex compositional model is probably not necessary for the document classification task (after all, a unigram model achieves competitive results on classifying RCV documents), it seems a bit misleading to call a bag-of-words approach compositional. After all, Klementiev et al. also sum word representation to yield a representation of a document. (However, such added compositional model, of course, have been considered in the past and called compositional -- e.g., Mitchell and Lapata (2008)).  From my perspective, the interesting aspect here is learning word representation without using word alignment information. In this way, the work is similar to the paper of Lauly et al. presented at the NIPS Deep Learning workshop (http://arxiv.org/abs/1401.1803). However, their results are not directly comparable as they used a different test set.  My concern is that a different learning objective would be needed if more expressive compositional models are used (perhaps combining both similarity across languages and the reconstruction error as in Socher (EMNLP 2011)). 

Minor:
-- it would be interesting to see results on other language pairs (e.g., French was already used in training, but not in testing, even though RCV contains articles in French)
-- also I am wondering how performance varies depending on the size of parallel data (as this might be a concern for low resource languages where direct transfer approaches are especially attractive)
-- It was not entirely clear how negative examples are sampled (formula 6):  are they chosen at every epoch  (as, e.g., in Rendle et al. (UAI 2009)), or chosen once for every sentence pair and then kept fixed during training?
-- section 2.1, par 1:  'multi-agent learning' -> 'multi-task learning' ?