Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describe a new method to capture word polysemy with word embeddings. In order to disambiguate a word in a given sentence, the word is represented by the subspace spanned by the word vectors of the context in which it appears. This departs from a traditional approach were the context is represented as a (weighted) sum of the word vectors. A clustering algorithm (very similar to k-means), is then used to cluster the different usages of a given word, and discover the different senses (each sense corresponding to a cluster). The proposed method is evaluated on various word sense induction datasets. It is compared to other word embedding techniques which model word polysemy.

The method proposed in the paper to represent words in context is really interesting, simple to apply and seems very effective, based on the strong experimental results reported in the paper. My main concern about this paper is the writing, which is sometimes a bit verbose, making it hard to follow the description of the method. Some of the justification ("intersection hypothesis", "polysemy intersection hypothesis") might feel a bit like hand waving.

Overall, the work presented in the paper looks solid.

Pros:
 - I really liked the idea of representing a word in context by a subspace (as opposed to a weighted sum). Indeed, such representations captures much more information than a single vector.
 - The proposed method also obtain very good results, compared to existing polysemous word embeddings.
 - It can be used with any word vectors, making its application very easy.

Cons:
 - I felt that the paper is sometimes a bit verbose and some justifications might be a bit hand waving.
 - I am also wondering how much of the improvement over existing approaches is due to the quality of the word2vec embeddings, or due to the proposed approach. It would therefore be nice to have a comparison with a regular k-means approach, where context are represented as sum of word vectors using the same embeddings.
review_title: REVIEW

final_score: rating: 7: Good paper, accept
review_content: On the plus side, the paper proposes a mathematically interesting model for a context of a word (i.e., a Grassmanian manifold).

On the minus side, the paper mostly ignores the long history of Word Sense Induction (WSI) and Word Sense Disambiguation (WSD), citing and comparing only some relatively recent papers. The experiments in this paper done on SemEval-2010 are not very persuasive. (It's difficult to evaluate the experiments done on the 2016 data, since they are not directly comparable to published results).

For example, going back to the SemEval-2010 WSI task in [1], the best system seems to be UoY [2]. The F-measure seems to be a poor metric: always assigning one sense to every word ("MFS") yields the highest F-measure of 63.5%. The paper's result with "2 clusters" (with an average of about 1.9) seems to be close to MFS. So I don't think we can use F-measure to compare.

The V-measure seems to be tilted towards systems that have high number of senses per word. UoY has V=15.7%, while the paper (with "5 clusters") has 14.4%. That isn't very convincing that the proposed method has captured the geometry of polysemy.

In general, I have often wondered why people work on pure unsupervised WSI and WSD. The assessment is very difficult (as described above). More importantly, some very weakly supervised systems (with minimal labels) can work pretty well to bootstrap. See, e.g., the classic paper [3].

If the authors used the Grassmannian idea to solve higher-level NLP problems directly (such as analogies), that would be very persuasive. However, that's a very different paper than what was submitted. For an example of application of Grassmannian manifolds to analogies, see [4].

References:
1. Manandhar, Suresh, et al. "SemEval-2010 task 14: Word sense induction & disambiguation." Proceedings of the 5th international workshop on semantic evaluation. Association for Computational Linguistics, 2010.
2. Korkontzelos, Ioannis, and Suresh Manandhar. "Uoy: Graphs of unambiguous vertices for word sense induction and disambiguation." Proceedings of the 5th international workshop on semantic evaluation. Association for Computational Linguistics, 2010.
3. Yarowsky, David. "Unsupervised word sense disambiguation rivaling supervised methods." Proceedings of the 33rd annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, 1995.
4. Mahadevan, Sridhar,  and Sarath Chandar Reasoning about Linguistic Regularities in Word Embeddings using Matrix Manifolds https://arxiv.org/abs/1507.07636 
review_title: Doesn't do a thorough job comparing to long history of WSI and WSD

final_score: rating: null
review_content: In the case where the contexts are represented by a subspace of dimension 1, the algorithm described in the "sense induction" paragraph is equivalent to k-means. Is that correct?

I also have several questions regarding the experiments described in the paper:

(1) How are the word vectors from Huang et al. (2012) and Neelakantan et al. (2015) used to perform word sense disambiguation?

(2) Why do the authors compare their method trained on a snapshot of Wikipedia from 2015 to word representation trained  on a snapshot of Wikipedia from 2010? The 2010 snapshot of Wikipedia, introduced by (Shaoul and Westbury, 2010) is available at http://www.psych.ualberta.ca/~westburylab/downloads/westburylab.wikicorp.download.html.

(3) How does the proposed algorithm compared to k-means, where contexts are represented as a tf-idf weighted sum of word vectors from the context?

(4) What is the influence of the dimension N of the subspace representing the context?

(5) Why aren't there results for the method of Arora et al. (2016) in Table 1?

(6) In Table 1, Huang 2012 should be in bold in the column Make-Sense-2106/V-Measure.
review_title: Questions about experiments

final_score: rating: 7: Good paper, accept
review_content: This paper presents a study of the spaces around existing word embeddings. I proposes something unorthodox: instead of representing a word token by a vector, represent it by the subspace spanned by embeddings of the context word types around that token. These subspaces are fairly low-dimensional and are shown to capture some notions of polysemy (subspaces for tokens of the same sense should all roughly intersect in the same direction). While thinking about the subspace spanned by the context is fairly similar to thinking about a linear combination of the context embeddings, the subspace picture allows for a little more information to be preserved which can improve downstream semantic tasks.

The paper is a little dense reading at times, and some things are hard to understand, but the perspective is original enough and the results are good enough that I think it belongs in ICLR.
review_title: Weak accept

final_score: rating: null
review_content: In the argument on section 2 around figure 1, it's not at all clear why it is surprising that the variance is explained with the top few principal components. I also don't understand the weird distribution plot, instead of the usual plot about the decay of the eigenvalues, since eigenvalues dropping quickly would lead to the same low-dimensional subspace justification (and it is unsurprising since this drop has been observed in many places in language). Indeed, given that the word vectors were initially trained by maximizing their dot products with other co-occurring context word vectors, it is unsurprising that they group in subspaces.

How do the baseline algorithms compare in the task of disambiguating artificial polysemous words? This task is frequently used in word sense disambiguation research and a lack of a baseline makes those results hard to trust.

Where is the actual algorithm used to compute the subspaces and their embeddings? The main body of the paper should contain one. 


review_title: Questions about the justification

Meta-review: The paper considers an important problem largely ignored by continuous word representation learning: polysemy. The approach is mathematically grounded and interesting and well explored.
Decision: Accept (Poster)