A Simple Model for Learning Multilingual Compositional Semantics
Authors: Karl Moritz Hermann, Phil Blunsom
ICLR 2014

Abstract: Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. In particular, recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning compositional representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.


Peer review:
[Review]:
This paper proposes a simple model to learn word embeddings in a
bilingual setting. The model learns embeddings at the sentence-pair
level, where aligned sentences are similarly represented. This simple
model does not rely on word alignments or MT system.

The paper is well written and presents convincing results. My only
concern is why the authors use the term 'Models of Compositional
Distributed Semantics' for a model that is more related to annotation
transfer. Moreover, the continuous representation of a sentence is the
sum of word embeddings. This more related to a bag of word model than
a model that can handle compositionality. A minor remark about the
results: the authors observe that BICVM+ outperforms the BICVM model
when training on English data, but performs worse in the opposite
direction. Could you comment on that.

The authors could read the paper of Lauly et al. at the last NIPS
workshop on Deep Learning. This is clearly related to this
work. Finally, I wonder what could be the performances with more
complex model. For instance, the same authors proposed a translation
model based on recurrent Net that could be used for this task.