Title: Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context
Authors: Shyam Upadhyay
Kai-Wei Chang
James Zou
Matt Taddy
Adam Kalai
Venue: ICLR 2017
Keywords: Natural language processing

Abstract: Word embeddings, which represent a word as a point in a vector space, have become ubiquitous to several NLP tasks. A recent line of work uses bilingual (two languages) corpora to learn a different vector for each sense of a word, by exploiting crosslingual signals to aid sense identification. We present a multi-view Bayesian non-parametric algorithm which improves multi-sense word embeddings by (a) using multilingual (i.e., more than two languages) corpora to significantly improve sense embeddings beyond what one achieves with bilingual information, and (b) uses a principled approach to learn a variable number of senses per word, in a data-driven manner. Ours is the first approach with the ability to leverage multilingual corpora efficiently for multi-sense representation learning. Experiments show that multilingual training significantly improves performance over monolingual and bilingual training, by allowing us to combine different parallel corpora to leverage multilingual context. Multilingual training yields com- parable performance to a state of the art monolingual model trained on five times more training data.

TL;DR: Using multilingual context for learning multi-sense embeddings helps.

Review discussion:
- you do not model the polysemy in the foreign language. how can the model be extended in that direction?
- have you considered exploiting the morphology in foreign languages?
- why is table 2 missing Russian and Spanish?
MINOR issues:
- dirichlet process reference has a problem
- page 6 english is lowercase.
- table 3 instead of using (1) you can use $\Delta$
We have made changes to correct the minor issues in the new revision.
1) Morphology and Foreign Language Polysemy
It should be possible to extend our model to incorporate morphology,although we did not attempt it. For bilingual/multilingual training, to handle
polysemy in both english and foreign language, we can have a separate dirichlet process (with different parameters) on the foreign language side similar to english.
2) Missing Ru and Es in Table 2?
Note that the datasets in table 2 and table 3 are different (see caption for tab 3), because in table 3 we wanted to control for change in domain. In our preliminary experiments the corpora used in Table 3 did signficantly worse on the WSI task (avg. ari < 0.45 for each of the 4 languages) than those used for table 2.
Indeed, even after pairing languages in Table 3, the avg ARI is still far from Table 2.
I found the model section of the paper a bit hard to follow. If I understood correctly, the model is a Bayesian variant of the skipgram model, where the context words are from both the source and the aligned sentences. Word polysemy on the source side is captured by the latent variables z, indicating which vector representation to use for a given context (thus, p(y | z, x) \propto exp(<u_y, v_x,z>)).
Is the probability distribution P(y | x_e, x_f) is defined as a probability over all the word of the vocabulary, or as the probability of presence (or absence) of the word y given x_e and x_f?
In practice, I believe that the word x_f from the "foreign" language cannot be aligned with the context word y_e in English. Why not consider a factor Psi between aligned words (to enforce that aligned words have similar representation)?
Is polysemy only considered on the English side?
What is the difference between the one-sided and full models?
What is the metric reported in tables 2/3?
Your interpretation is correct. We will make changes to improve the presentation in the model section.
Is the probability distribution P(y | x_e, x_f) is defined as a probability over all the word of the vocabulary, or as the probability of presence (or absence) of the word y given x_e and x_f?
We define the probability as over all words in the vocabulary, and compute it using hierarchical softmax.
In practice, I believe that the word x_f from the "foreign" language cannot be aligned with the context word y_e in English.Why not consider a factor Psi between aligned words (to enforce that aligned words have similar representation)?
We are not aligning the pivot word x_f (which aligns to english pivot x_e) in the foreign language to the context word y_e in english, but only  predicting y_e using x_f. In principle, we can also enforce the aligned words to have similar representation, using a different factorization than what we presented.
Is polysemy only considered on the English side?
Yes, we only model the polysemy on the english side.
What is the difference between the one-sided and full models?
The one-sided model does not contain the Psi(y^e, x^f) term which tries to predict the english context using the foreign aligned word, as discussed in Section 4. It is meant to show the value of the Psi(y^e, x^f) term when using the multilingual setting.
What is the metric reported in tables 2/3?
We report adjusted rand index (see experimental setup Section 5)
In this paper, the authors propose a Bayesian variant of the skipgram model to learn word embeddings. There are two important variant compared to the original model.
First, aligned sentences from multiple languages are used to train the model.
Therefore, the context words of a given target word can be either from the same sentence, or from an aligned sentence in a different language. This allows to learn multilingual embedding. The second difference is that each word is represented by multiple vectors, one for each of its different senses. A latent variable z models which sense should be used, given the context.
Overall, I believe that the idea of using a probabilistic model to capture polysemy is an interesting idea. The model introduced in this paper is a nice generalization of the skipgram model in that direction.
However, I found the paper a bit hard to follow. The formulation might probably be simplified (e.g. why not consider a target word w and a context c, where c is either in the source or target language. Since all factors are independent, this should not change the model much, and would make the presentation easier). The performance of all models reported in Table 2 & 3 seem pretty low.
Overall, I like the main idea of the paper, which is to represent word senses by latent variables in a probabilistic model. I feel that the method could be presented more clearly, which would make the paper much stronger. I also have some concerns regarding the experimental results.
Pros:
Interesting extension of skipgram to capture polysemy.
Cons:
The paper is not clearly written.
Results reported in the paper seems pretty low.
Thanks for the comments and suggestions.
We will simplify the model description per your suggestion.
this work aims to address representation of multi-sense words by exploiting multilingual context. Experiments on word sense induction and word similarity in context show that the proposed solution improves over the baseline.
From a computational linguistics perspective, the fact that languages less similar to English help more is intriguing. I see following problem with this work:
- the paper is hard to follow and hard to see what's new compared to the baseline model [1]. A paragraph of discussion should clearly compare and contrast with that work.
- the proposed model is a slight variation of the previous work [1] thus the experimental setup should be designed in a way so that we compare which part helps improvement and how much.
thus MONO has not been exposed the same training data and we can't be sure that the proposed model is better because MONO does not observe the data or lacks the computational power. I suggest following baseline: turning multilingual data to monolingual one using the alignment, then train the baseline model[1] on this pseudo monolingual data.
- the paper provides good benchmarks for intrinsic evaluation but the message could be conveyed more strongly if we see improvement in a downstream task.
[1] https://arxiv.org/pdf/1502.07257v2.pdf
Thank you for your comments. We will improve the writing and experimental comparison in the next version.
We would like to note that MONO is exposed to same amount of training data, because we ensure that the number of words in the context window are same (by using a bigger context parameter for MONO).
Indeed, MONO is unable to process multilingual data, but that is a limitation of the modeling itself. We will try the pseudo monolingual experiment you suggested.
This paper discusses multi-sense embedddings and proposes learning those by using aligned text across languages.
Further, the paper suggests that adding more languages helps improve word sense disambiguation (as some ambiguities can be carried across language pairs).
While this idea in itself isn't new, the authors propose a particular setup for learning multi-sense embeddings by exploiting multilingual data.
Broadly this is fine, but unfortunately the paper then falls short in a number of ways. For one, the model section is unnecessarily convoluted for what is a nice idea that could be described in a far more concise fashion.
Next (and more importantly), comparison with other work is lacking to such an extent that it is impossible to evaluate the merits of the proposed model in an objective fashion.
This paper could be a lot stronger if the learned embeddings were evaluated in downstream tasks and evaluated against other published methods. In the current version there is too little of this, leaving us with mostly relative results between model variants and t-SNE plots that don't really add anything to the story.
Thank you for you comments.
We will work to make the model description easier to understand. We will also evaluate our embeddings on other downstream tasks  to improve the comparison with existing works.

Meta-review:
role: PC
data: comment: While the reviewers find the core idea intriguing, the method needs a clearer explanation and a more thorough comparison to related work.

Decision: Reject