Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - you do not model the polysemy in the foreign language. how can the model be extended in that direction?

- have you considered exploiting the morphology in foreign languages?

- why is table 2 missing Russian and Spanish?


MINOR issues:
- dirichlet process reference has a problem
- page 6 english is lowercase.
- table 3 instead of using (1) you can use $\Delta$
review_title: the polysemy & morphology in the foreign language

final_score: rating: 4: Ok but not good enough - rejection
review_content: this work aims to address representation of multi-sense words by exploiting multilingual context. Experiments on word sense induction and word similarity in context show that the proposed solution improves over the baseline.

From a computational linguistics perspective, the fact that languages less similar to English help more is intriguing. I see following problem with this work:
- the paper is hard to follow and hard to see what's new compared to the baseline model [1]. A paragraph of discussion should clearly compare and contrast with that work.

- the proposed model is a slight variation of the previous work [1] thus the experimental setup should be designed in a way so that we compare which part helps improvement and how much. thus MONO has not been exposed the same training data and we can't be sure that the proposed model is better because MONO does not observe the data or lacks the computational power. I suggest following baseline: turning multilingual data to monolingual one using the alignment, then train the baseline model[1] on this pseudo monolingual data.

- the paper provides good benchmarks for intrinsic evaluation but the message could be conveyed more strongly if we see improvement in a downstream task.

[1] https://arxiv.org/pdf/1502.07257v2.pdf
review_title: needs better comparison

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors propose a Bayesian variant of the skipgram model to learn word embeddings. There are two important variant compared to the original model. First, aligned sentences from multiple languages are used to train the model. Therefore, the context words of a given target word can be either from the same sentence, or from an aligned sentence in a different language. This allows to learn multilingual embedding. The second difference is that each word is represented by multiple vectors, one for each of its different senses. A latent variable z models which sense should be used, given the context.

Overall, I believe that the idea of using a probabilistic model to capture polysemy is an interesting idea. The model introduced in this paper is a nice generalization of the skipgram model in that direction. However, I found the paper a bit hard to follow. The formulation might probably be simplified (e.g. why not consider a target word w and a context c, where c is either in the source or target language. Since all factors are independent, this should not change the model much, and would make the presentation easier). The performance of all models reported in Table 2 & 3 seem pretty low.

Overall, I like the main idea of the paper, which is to represent word senses by latent variables in a probabilistic model. I feel that the method could be presented more clearly, which would make the paper much stronger. I also have some concerns regarding the experimental results.

Pros:
Interesting extension of skipgram to capture polysemy.
Cons:
The paper is not clearly written.
Results reported in the paper seems pretty low.
review_title: Review

final_score: rating: null
review_content: I found the model section of the paper a bit hard to follow. If I understood correctly, the model is a Bayesian variant of the skipgram model, where the context words are from both the source and the aligned sentences. Word polysemy on the source side is captured by the latent variables z, indicating which vector representation to use for a given context (thus, p(y | z, x) \propto exp(<u_y, v_x,z>)).

Is the probability distribution P(y | x_e, x_f) is defined as a probability over all the word of the vocabulary, or as the probability of presence (or absence) of the word y given x_e and x_f?

In practice, I believe that the word x_f from the "foreign" language cannot be aligned with the context word y_e in English. Why not consider a factor Psi between aligned words (to enforce that aligned words have similar representation)?

Is polysemy only considered on the English side?

What is the difference between the one-sided and full models?

What is the metric reported in tables 2/3?
review_title: Model clarification

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses multi-sense embedddings and proposes learning those by using aligned text across languages. Further, the paper suggests that adding more languages helps improve word sense disambiguation (as some ambiguities can be carried across language pairs). While this idea in itself isn't new, the authors propose a particular setup for learning multi-sense embeddings by exploiting multilingual data.

Broadly this is fine, but unfortunately the paper then falls short in a number of ways. For one, the model section is unnecessarily convoluted for what is a nice idea that could be described in a far more concise fashion. Next (and more importantly), comparison with other work is lacking to such an extent that it is impossible to evaluate the merits of the proposed model in an objective fashion. 

This paper could be a lot stronger if the learned embeddings were evaluated in downstream tasks and evaluated against other published methods. In the current version there is too little of this, leaving us with mostly relative results between model variants and t-SNE plots that don't really add anything to the story.
review_title: Interesting idea, execution needs more work

Meta-review: While the reviewers find the core idea intriguing, the method needs a clearer explanation and a more thorough comparison to related work.
Decision: Reject