A Simple Model for Learning Multilingual Compositional Semantics
Authors: Karl Moritz Hermann, Phil Blunsom
ICLR 2014

Abstract: Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. In particular, recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning compositional representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.


Peer review:
[Review]:
We would like to thank the anonymous reviewers for their thorough reviews and comments regarding our submission.

Several reviewers took issue with our use of the term 'compositional semantics' in relation to an additive composition function (i.e. a bag-of-words model). I agree with reviewer 57c7, who pointed out that such models have been described as compositional in the past, but that the term might be misleading in light of more recent development in compositional semantics. I will update the paper to improve the wording regarding this term, and to clarify the limitations of our model versus fuller compositional approaches.

As pointed out by reviewers 9aaf and 048b, it would be interesting to see how our objective function performs in combination with a more complex model. I agree that the current BoW model is likely to be insufficient for tasks such as machine translation. The key contribution of the current submission are two: first, learning word embeddings without word alignment, and second the multilingual objective function that we use to achieve this and its parallel use for semantic transfer. I agree that it would be interesting to develop this further by using more complex composition models and also by attempting other tasks such as machine translation - we are currently studying these possibilities for further work.

Also, thank you for pointing out the Lauly et al. paper from the NIPS workshop; this seems indeed very relevant and I will incorporate and reference this paper accordingly.

To answer the question raised about sampling by reviewer 57c7: we sampled negative examples at every epoch. We will update the paper to address this and the other minor remarks made in the three reviews, and submit an updated paper to ArXiv in the next few days.