A Simple Model for Learning Multilingual Compositional Semantics
Authors: Karl Moritz Hermann, Phil Blunsom
ICLR 2014

Abstract: Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. In particular, recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning compositional representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.


Peer review:
[Review]:
Dear Ryan,

Thank you for your comments on our submission. I think you are raising a valid point concerning the dimensionality of our embeddings - this is something we should have explained better in the paper. Effectively, we started using d=128 in the beginning (chosen somewhat arbitrarily based on prior work on distributed representations). Subsequently, we tried tuning this parameter a little bit, but it didn't make much of a difference and so we stuck with the original setting.

Following your comments I have re-run all experiments with d=40. The results for this are as follows (matching Table 1):

BiCVM        83.7    71.4
BiCVM+      86.2    76.9

If you think this would be useful, we are happy to include these and other results with changing dimensionalities in the paper.

Best
Karl Moritz