Title: Geometry of Polysemy
Authors: Jiaqi Mu
Suma Bhat
Pramod Viswanath
Venue: ICLR 2017
Keywords: Natural language processing

Abstract: Vector representations of words have heralded a transformational approach to classical problems in NLP; the most popular example is word2vec. However, a single vector does not suffice to model the polysemous nature of many (frequent) words, i.e., words with multiple meanings.    In this paper, we propose a three-fold approach for unsupervised polysemy modeling: (a) context representations, (b) sense induction and disambiguation and (c) lexeme (as a word and sense pair) representations. A key feature of our work is the finding that  a sentence containing a target word is  well represented by a low-rank subspace,  instead of a point in a vector  space. We then show that the subspaces associated with a particular sense of the target word tend to intersect over a line (one-dimensional subspace), which we use to disambiguate senses using a  clustering algorithm that harnesses the Grassmannian geometry of the representations. The disambiguation algorithm, which we call $K$-Grassmeans, leads to a procedure to label the different senses of the target word in the corpus -- yielding lexeme vector representations, all in an unsupervised manner starting from a large (Wikipedia) corpus in English. Apart from several prototypical target (word,sense) examples and a host of empirical studies to intuit and justify  the various geometric representations,  we validate our algorithms on standard sense induction and disambiguation datasets and present new state-of-the-art results.


Review discussion:
In the argument on section 2 around figure 1, it's not at all clear why it is surprising that the variance is explained with the top few principal components. I also don't understand the weird distribution plot, instead of the usual plot about the decay of the eigenvalues, since eigenvalues dropping quickly would lead to the same low-dimensional subspace justification (and it is unsurprising since this drop has been observed in many places in language).
Indeed, given that the word vectors were initially trained by maximizing their dot products with other co-occurring context word vectors, it is unsurprising that they group in subspaces.
How do the baseline algorithms compare in the task of disambiguating artificial polysemous words? This task is frequently used in word sense disambiguation research and a lack of a baseline makes those results hard to trust.
Where is the actual algorithm used to compute the subspaces and their embeddings? The main body of the paper should contain one.
Thank you for your suggestions and comments!
- Regarding Figure 1 in Section 2:
1. Given a context $c$, let $\sigma_1(c) > . > \sigma_M(c)$ be the variance ratios for $M$ principle components. If I understand you correctly, then the plot you suggested is $\sigma_m(s)$ against $m$ and if there is a quick drop in the plot, our hypothesis can be validated. Yes, we agree this can also be an empirical validation if we plot many curves for many contexts. We did not see a good way to plot all of them nicely in one figure and therefore we chose to report the histogram of the total variance ratio (then that's one number for one context, i.
e. , $\sum_{m=1}^N \sigma_m(s)$) captured by rank-$N$ PCA. The two plots (the one we reported and the one you suggested) are inherently highly correlated: if enough energy is captured by the first few principal components, then there will be a quick drop and the total variance ratios will also be large (as the histogram shows).
2. We haven't seen a plot of the singular values of the row-stacked vectors of words in a context/sentence in the literature. It is possible we misunderstand each other -- perhaps the reviewer can kindly point out a few citations to such  plots referred in the Question?
- Regarding our hypothesis:
Note that although the word vectors were trained by maximizing inner product between neighboring word vectors, the overall objective involves *averaging* over very many sentences.
Thus, it is not at all obvious why the grouping in subspaces should occur for *most* contexts/sentences (this is the point of Figure 1).
Indeed, prior work [1] posits that the word vectors are isotropic (i.e., angular symmetry), which also makes our finding not obvious. The observation of low-dimensional nature of the context vectors is novel (to the best of our knowledge), and the key to the representations in this paper and the corresponding algorithms which yield state-of-the-art results on standard polysemy datasets.
- The algorithm to compute subspace embeddings is just 2 lines of code (stack context word vectors and then do PCA) and hence we moved the formal description to Appendix J. We are happy to move this (very short) appendix to the main text.
- The suggestion on the baseline for the synthetic task is a very good one. Our experiments are running and we will report the results shortly. For now, we note that our algorithms outperformed the state of the art ones on the standardized datasets (Table 1, Page 8).
[1] Arora, Sanjeev, et al. "Rand-walk: A latent variable model approach to word embeddings." arXiv preprint arXiv:1502.03520 (2015).
The comparison between the baseline algorithm and our proposed algorithm for the synthetic task is now reported in Figure 4(c).
Re (2), it'd be interesting to check how quickly do the eigenvalues of 2W random word vectors decay. I expect that would decay more slowly.
That said, the findings from the prior work do not contradict your findings, since all word vectors being isotropic does not imply that particular non-random subsets of them are isotropic as well (and I see no reason for them not to be). Since these context vectors are non random and optimized to have a high dot product with each other, you'd expect tem to lie in a low-dimensional space.
In the case where the contexts are represented by a subspace of dimension 1, the algorithm described in the "sense induction" paragraph is equivalent to k-means. Is that correct?
I also have several questions regarding the experiments described in the paper:
(1) How are the word vectors from Huang et al. (2012) and Neelakantan et al. (2015) used to perform word sense disambiguation?
(2) Why do the authors compare their method trained on a snapshot of Wikipedia from 2015 to word representation trained  on a snapshot of Wikipedia from 2010? The 2010 snapshot of Wikipedia, introduced by (Shaoul and Westbury, 2010) is available at http://www.psych.ualberta.ca/~westburylab/downloads/westburylab.wikicorp.download.html.
(3) How does the proposed algorithm compared to k-means, where contexts are represented as a tf-idf weighted sum of word vectors from the context?
(4) What is the influence of the dimension N of the subspace representing the context?
(5) Why aren't there results for the method of Arora et al. (2016) in Table 1?
(6) In Table 1, Huang 2012 should be in bold in the column Make-Sense-2106/V-Measure.
Thank you for your comments and suggestions!
In the case where the contexts are represented by a subspace of dimension 1, the algorithm described in the "sense induction" paragraph is equivalent to k-means. Is that correct?
Not really.
First, there are no norm constraints in k-means while our k-Grassmeans always asks for vectors of unit length.
Second, even if we add the norm constraint on k-means, the objective is still  different: let $u(c)$ be the rank-1 subspace representing $c$, then the objectives are as follows:
- k-means: \arg\min_{\|u\|_2 = 1} \sum_{c} \|u - u(c)\|_2^2 = \arg\min_{\|u\|_2 = 1} \sum_{c} (\|u\|_2^2 + \|u(c)\|_2^2 - 2u^T u(c)) = \arg\max_{\|u\|_2 = 1} \sum_{c} u^Tu(c)
- k-Grassmeans: \arg\max_{\|u\|_2 = 1} \sum_{c} (u^T u(c))^2.
I also have several questions regarding the experiments described in the paper:
(1) How are the word vectors from Huang et al. (2012) and Neelakantan et al. (2015) used to perform word sense disambiguation?
For Huang et al. (2012) we use the code and vectors provided on the author's website (http://ai.stanford.edu/~ehhuang/). For Neelakantan et al. (2015), we also use their vectors provided at http://iesl.cs.umass.edu/downloads/vectors/release.tar.gz, and implement their disambiguation algorithm (c.f. line 4 - line 7 in Algorithm 1 in their paper) on our own.
(2) Why do the authors compare their method trained on a snapshot of Wikipedia from 2015 to word representation trained  on a snapshot of Wikipedia from 2010? The 2010 snapshot of Wikipedia, introduced by (Shaoul and Westbury, 2010) is available at http://www.psych.ualberta.ca/~westburylab/downloads/westburylab.wikicorp.download.html.
Very perspicacious! We missed this when running our experiments. We have set up an experiment using the 2010 Wikipedia corpus and will get back to you with the statistics in the next few days.
(3) How does the proposed algorithm compared to k-means, where contexts are represented as a tf-idf weighted sum of word vectors from the context?
Thanks for the suggestion. Huang 2012 is actually using idf weighting.  MSSG is using unweighted average, and it is not clear why their didn't try tf-idf weighting and how the corresponding performance would change. We submit that it is onerous for us to adapt their algorithm to the reweighing scheme in the short time frame ahead.
Further, this is tangential to the main contributions of our paper -- which include the various geometrical findings involving subspace representations).
(4) What is the influence of the dimension N of the subspace representing the context?
This is a tradeoff between the energy and the noise one wants to keep. Our experiments show that N=3,4,5 lead to similar performances.
(5) Why aren't there results for the method of Arora et al. (2016) in Table 1?
Arora et al. (2016) provide only sense representations (as global atoms of discourse, 2000 of which are shared across the entire vocabulary). They didn't provide a disambiguation algorithm in their paper (explicitly leaving it as future work) and we are not sure how to incorporate their embeddings in this experiment either. Our results on  their police lineup task in Appendix F.2  show a direct comparison (where we perform uniformly better over the precision-recall curve).
(6) In Table 1, Huang 2012 should be in bold in the column Make-Sense-2106/V-Measure.
Sorry, we made a typo here. The statistics for Huang 2012 should be 15.50 (V-measure), 47.40 (F-score) and 6.15 (# cluster).  We will update the paper soon.
We redo the sense induction and disambiguation task using the 2010 snapshot of Wikipedia. The statistics for Table 1 is provided below. From this table, we can see the performance does not change much with a corpus of a different year, since the domains are the same.
|Sem-Eval-2010                      |Make-Sense-2016                    |
|V-measure  |F-score    |#cluster   |V-measure  |F-score    |#cluster   |
MSSG.300D.30K   |9.00       |47.26      |2.88       |19.40      |54.49      |2.88       |
MSSG.300D.6K    |6.90       |48.43      |2.45       |14.40      |57.91      |2.35       |
NP-MSSG.300D.6K |6.50       |52.45      |2.56       |15.50      |55.39      |3.05       |
#cluster=2      |7.00       |55.24      |1.90       |29.30      |64.79      |1.96       |
#cluster=5      |15.00      |43.36      |4.44       |33.40      |56.54      |4.62       |
This paper presents a study of the spaces around existing word embeddings. I proposes something unorthodox: instead of representing a word token by a vector, represent it by the subspace spanned by embeddings of the context word types around that token. These subspaces are fairly low-dimensional and are shown to capture some notions of polysemy (subspaces for tokens of the same sense should all roughly intersect in the same direction).
While thinking about the subspace spanned by the context is fairly similar to thinking about a linear combination of the context embeddings, the subspace picture allows for a little more information to be preserved which can improve downstream semantic tasks.
The paper is a little dense reading at times, and some things are hard to understand, but the perspective is original enough and the results are good enough that I think it belongs in ICLR.
Dear AnonReviewer1,
Thank you for your review and comments!
Yes, representing words by subspaces instead of vectors might also be a promising approach. We haven't dived deep into this yet, and we are happy to discuss this idea with you.
Thanks, Jiaqi
We took your suggestion to heart and have revised our manuscript to make it more readable -- the focus is to get to the methods as directly as possible (by moving the intuition and examples and empirical validation of the hypotheses to the supplementary material). The space created allowed us to move back our results on lexeme (i.e., (word,sense) pair) representations to the main text. Thank you very much for these  suggestions. We would appreciate hearing any further comments you might have on this revision.
This paper describe a new method to capture word polysemy with word embeddings. In order to disambiguate a word in a given sentence, the word is represented by the subspace spanned by the word vectors of the context in which it appears. This departs from a traditional approach were the context is represented as a (weighted) sum of the word vectors. A clustering algorithm (very similar to k-means), is then used to cluster the different usages of a given word, and discover the different senses (each sense corresponding to a cluster). The proposed method is evaluated on various word sense induction datasets. It is compared to other word embedding techniques which model word polysemy.
The method proposed in the paper to represent words in context is really interesting, simple to apply and seems very effective, based on the strong experimental results reported in the paper. My main concern about this paper is the writing, which is sometimes a bit verbose, making it hard to follow the description of the method. Some of the justification ("intersection hypothesis", "polysemy intersection hypothesis") might feel a bit like hand waving.
Overall, the work presented in the paper looks solid.
Pros:
- I really liked the idea of representing a word in context by a subspace (as opposed to a weighted sum).
Indeed, such representations captures much more information than a single vector.
- The proposed method also obtain very good results, compared to existing polysemous word embeddings.
- It can be used with any word vectors, making its application very easy.
Cons:
- I felt that the paper is sometimes a bit verbose and some justifications might be a bit hand waving.
- I am also wondering how much of the improvement over existing approaches is due to the quality of the word2vec embeddings, or due to the proposed approach. It would therefore be nice to have a comparison with a regular k-means approach, where context are represented as sum of word vectors using the same embeddings.
Dear AnonReviewer2,
Thank you for your review and comments!
We admit that our writing is somewhat different from mainstream NLP papers. Given the eight-page limit, we decided to focus more on the insight, the hypothesis, the geometry and the model instead of the algorithm. Once the model is well validated (so we did lots of experiments to check the rationale of the hypothesis and the model), $k$-Grassmeans is one of the inference algorithms. Due to the combinatorial nature of clustering and the inherent randomness in the samples, there is no *best* algorithm/method universally for all instances.
The proposed $k$-Grassmeans is computationally simple and turns out to perform well enough (i. e. , statistically significantly better than baselines) in both synthetic tasks (c. f. the end of page 5 and Figure 4(c)) and standard polysemy tasks (c. f. the word sense induction task in Section 5, the word similarity task in Appendix F. 1, and the police line-up task in Appendix F. 2). Alternative inference algorithms are possible in the context of the same model.
For example, one can also do $K$-means on the subspace basis -- though we didn't check if this approach works well or not. We are happy to add more discussions on the comparison between $K$-Grassmeans and the other alternative algorithms.
We did a direct comparison on our synthetic tasks (c.f. the end of page 5 and Figure 4(c) in the updated version) between $k$-Grassmeans of subspaces and $k$-means of average word vectors. It turns out that $k$-Grassmeans+subspaces performs better (in a statistically significant way) than $k$-means+averages. We also did an indirect comparison on standard polysemy tasks: (a) we redid our experiment on Dec 5, 2016 as you requested, our algorithm performs better than Huang et al. (2012) and Neelakantan et al. (2015) using the same 2010 snapshot of Wikipedia corpus; (b) as reported in their papers, their algorithms are better than a simple (weighted) average of word vectors.
Specifically, MSSG in Neelakantan et al. (2015) has the same flavor of $k$-means on average word vectors. As indicated by (a) and (b), we are confident that subspaces capture more information than averages, and that $k$-Grassmeans on subspaces is more robust than $k$-means on averages in standardized tasks and datasets as well.
Thanks, Jiaqi
We took your suggestion to heart and have revised our manuscript to make it more readable -- the focus is to get to the methods as directly as possible (by moving the intuition and examples and empirical validation of the hypotheses to the supplementary material). The space created allowed us to move back our results on lexeme (i.e., (word,sense) pair) representations to the main text. Thank you very much for these  suggestions. We would appreciate hearing any further comments you might have on this revision.
On the plus side, the paper proposes a mathematically interesting model for a context of a word (i.e., a Grassmanian manifold).
On the minus side, the paper mostly ignores the long history of Word Sense Induction (WSI) and Word Sense Disambiguation (WSD), citing and comparing only some relatively recent papers. The experiments in this paper done on SemEval-2010 are not very persuasive. (It's difficult to evaluate the experiments done on the 2016 data, since they are not directly comparable to published results).
For example, going back to the SemEval-2010 WSI task in [1], the best system seems to be UoY [2]. The F-measure seems to be a poor metric: always assigning one sense to every word ("MFS") yields the highest F-measure of 63.5%. The paper's result with "2 clusters" (with an average of about 1.9) seems to be close to MFS.
So I don't think we can use F-measure to compare.
The V-measure seems to be tilted towards systems that have high number of senses per word. UoY has V=15.7%, while the paper (with "5 clusters") has 14.4%. That isn't very convincing that the proposed method has captured the geometry of polysemy.
In general, I have often wondered why people work on pure unsupervised WSI and WSD. The assessment is very difficult (as described above). More importantly, some very weakly supervised systems (with minimal labels) can work pretty well to bootstrap. See, e.g., the classic paper [3].
If the authors used the Grassmannian idea to solve higher-level NLP problems directly (such as analogies), that would be very persuasive.
However, that's a very different paper than what was submitted. For an example of application of Grassmannian manifolds to analogies, see [4].
References:
1. Manandhar, Suresh, et al. "SemEval-2010 task 14: Word sense induction & disambiguation." Proceedings of the 5th international workshop on semantic evaluation. Association for Computational Linguistics, 2010.
2. Korkontzelos, Ioannis, and Suresh Manandhar. "Uoy: Graphs of unambiguous vertices for word sense induction and disambiguation." Proceedings of the 5th international workshop on semantic evaluation. Association for Computational Linguistics, 2010.
3. Yarowsky, David. "Unsupervised word sense disambiguation rivaling supervised methods." Proceedings of the 33rd annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, 1995.
4. Mahadevan, Sridhar,  and Sarath Chandar Reasoning about Linguistic Regularities in Word Embeddings using Matrix Manifolds https://arxiv.org/abs/1507.07636
Dear AnonReviewer3,
Thanks for your review and comments!
- Regarding the experiments on SemEval-2010:
As far as we understand, the comments focus on two aspects: (a) we are not comparing against participating systems in SemEval-2010 (for instance UoY). (b) the F-measure is a poor metric.
Both these aspects are readily fixed, especially in the innovative format of ICLR where editing the submission is encouraged based on clarification questions and open reviews. Our revised manuscript addresses both these issues. We also have the following comments.
1. We redid $K$-Grassmeans using the exact same setup as SemEval-2010 and have reported the results in Table 2 (quick summary: we outperform SemEval 2010 results in all four metrics).
2. We did not explicitly compare to the reported systems in [2] in the originally submitted version since the task settings are different. In the shared task, the participating systems are trained on contexts given in the training set (ABC/CNN news), but we trained directly using (the mismatched) Wikipedia corpus, so the comparison is not fair.
Nevertheless, our unsupervised (and mismatched domain) performance is fairly close to the top performing systems of SemEval 2010.
On the other hand, relatively recent researches we compared to  (i.e., Huang 2012 and Neelakantan 2014), are of broadly the  same flavor as our ours (i.e., unsupervised methods using word embeddings) and and make natural comparison targets.
Second, our approach yields lexeme representations as a direct downstream application, and do not want to limit the domain based on the training data from SemEval-2010.
3.
In terms of the best system from SemEval-2010, Hermit  is the best in terms of V-Measure, and MFS is the best in terms of paired F-score. UoY is the best in terms of supervised evaluations (which we have now compared against in the revised manuscript -- our performance is better than UoY in all four metrics).  Re whether F-score is appropriate: we point out that this  metric was one used in the SemEval-2010 system (and thus a natural measure to make comparisons).
Regarding the evaluation metric,  V-Measure favors those with a larger number of cluster and paired F-score favors those with a smaller number of cluster (as discussed in Section 3 of our text) -- one notices that UoY had a high V-measure score but at the expense of a large number of clusters (roughly 11) and we perform a bit better than UoY with much smaller number of clusters (roughly 4).
- Regarding the related work: our work is at the intersection of a large number of topical areas (word vectors, sense disambiguation, sentence representations) and given the page constraints, we could  only cite the most relevant papers. We have now added citations to the key performing systems of  some WSI and WSD evaluation tasks.
- Regarding higher-level NLP tasks (analogies):
Thank you for the reference (we very much appreciate the camaraderie). Based on our detailed reading of this article, we find the focus quite tangential to our paper: their representations are lexical-level while our goals are in disambiguating the sense by innovative representations of the contexts surrounding a given word.
Although there is a similarity in terms of certain representations (subspaces), we submit that the similarity is only superficial: the representations are for different objects and the algorithms significantly different too.
- Regarding the unsupervised setting:
We believe unsupervised approaches are of central scientific and practical interest to NLP: to quote from our introduction: "since hand-crafted lexical resources sometimes do not reflect the actual meaning of a target word in a given context and, more importantly, such resources are lacking in many languages, we focus on the second approach (i.e., unsupervised approach) in this paper; such an approach is inherently scalable and potentially plausible with the right set of ideas.
Indeed, a human expects the contexts to cue in on the particular sense of a specific word, and successful unsupervised sense representation and sense extraction algorithms would represent progress in the broader area of representation of natural language. Such are the goals of this work."
The idea of bootstrapping from minimal training data is a useful one (thank you for the citation to Yarrowsky's  classical paper) -- but again, it it not clear how to obtain the training data in a principled way.
Indeed, Yarrowsky's work used WordNet, which we (and others) have found to have several senses that are too fine-grained (see Appendix G of our paper for a concrete example).
Thanks, Jiaqi
Thanks for adding Table 2 and comparing to the best systems in SemEval-2010. The fact that you're trying to be domain-independent is a positive factor in your paper.
Adding the extra experiments (e.g., k-means) and reorganizing definitely improved the paper.
We appreciate the useful comments and suggestions from the anonymous reviewers and have revised our paper in the following ways:
1. We moved the detailed description of our hypothesis and model into the appendix. Now the main text focuses on our methods with  intuition justifications provided in the background (detailed experimental validations of the hypotheses have been moved to the supplement).  We thank AnonReviewer 2 and AnonReviewer3 for their comments that directed us to make these (salutary) changes.
2. We had conducted detailed experiments on lexeme, i.e., (word,sense), representations  to the main text (previously Appendix F). Learning lexeme representations is also one of our main contributions and was perhaps not as centrally featured in the main text earlier. The space created by removing the detailed description of the hypotheses surrounding our method allows us to bring this material into the main text. Again, thanks to  AnonReviewer 2 and AnonReviewer3 for their suggestions on this regard.
3. We added one baseline algorithm -- specifically a standard k-means approach, where contexts are represented as a sum of word vectors -- and evaluated its performance for both the synthetic experiment in Section 2.2 on page 4 and the standardized WSI experiment in Section 3. We thank AnonReviewer2 for this suggestion.  Quick summary of the results: our algorithms performs a fair amount better than this  the baseline. The surprise is that this baseline performs similar to other works we compared to in the literature (especially the MSSG method, which itself is based on jointly learning the sense representations and average-word-vector context representations).
4. We added a few key baselines from SemEval-2010 participating systems, and redid our WSI experiment to compare against them. Table 2 has the details, but a quick summary: we perform superior to all the baselines in all four metrics of evaluation. We thank AnonReviewer1 for this suggestion.

Meta-review:
role: PC
data: comment: The paper considers an important problem largely ignored by continuous word representation learning: polysemy. The approach is mathematically grounded and interesting and well explored.

Decision: Accept (Poster)