Title: A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax
Authors: Fenfei Guo
Mohit Iyyer
Leah Findlater
Jordan Boyd-Graber
Venue: ICLR 2019
Keywords: unsupervised representation learning
sense embedding
word sense disambiguation
human evaluation

Abstract: We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources. It jointly learns how to disambiguate senses given local context and how to represent senses using hard attention. Unlike previous multi-prototype models, our model approximates discrete sense selection in a differentiable manner via a modified Gumbel softmax. We also propose a novel human evaluation task that quantitatively measures (1) how meaningful the learned sense groups are to humans and (2) how well the model is able to disambiguate senses given a context sentence. Our model outperforms competing approaches on both human evaluations and multiple word similarity tasks.

TL;DR: Disambiguate and embed word senses with a differentiable hard-attention model using Scaled Gumbel Softmax

Review discussion:
This paper proposes GASI to disambiguate different sense identities and learn sense representations given contextual information.
The main idea is to use scaled Gumbel softmax as the sense selection method instead of soft or hard attention, which is the novelty and contribution of this paper.
In addition, the authors proposed a new evaluation task, contextual word sense selection, which can be used to quantitatively evaluate the semantic meaningfulness of sense embeddings.
The proposed model achieves comparable performance on traditional word/sense intrinsic evaluation and word intrusion test as previous models, while it outperforms baselines on the proposed contextual word sense selection task.
While the scaled Gumbel softmax is the claimed novelty, it is more like an extension of the original MUSE model (Lee and Chen, 2017), which proposed the sense selection and representation learning modules for learning sense-level embeddings.
The only difference between the proposed one and Lee and Chen (2017) is Gumbel softmax instead of reinforcement learning between sense selection and representation learning modules.
Therefore, the idea from the proposed model is similar to Li and Jurafsky (2015), because the sense selection is not one-hot but a distribution.
The novelty of this paper is limited because the model is relatively incremental.
From my perspective, the more influential contribution is that this paper points out the importance of evaluating sense selection capability, which is ignored by most prior work.
Therefore, I expect to see more detailed evaluation on the selection module of the model.
Also, because the task of this paper is multi-sense embeddings, the traditional word similarity (without contexts) task seems unnecessary.
Moreover, there is no error analysis about the result on the proposed contextual word sense selection task, which may shed more light on the strength and weakness of the model.
Finally, I suggest the authors remove the word-level similarity task and try the recently released Word in Context (WiC) dataset, which is a binary classification task that determines whether the meaning of a word is different given two contexts.
It would be better to see that GASI performs well on this task given its better sense selection module.
Overall, the contribution is somewhat incremental and the evaluation/discussion should focus more on the sense selection module.
Considering the issues mentioned above, I will expect better quality for an ICLR paper.
We thank the reviewer for taking time to read our paper and the useful suggestions! We address the reviewer’s concerns and suggestions as follows:
1) Additional evaluation with recently released WiC dataset
Finally, I suggest the authors remove the word-level similarity task and try the recently released Word in Context (WiC) dataset
We thank the reviewer for their suggestion! We add an evaluation on the recently released WiC dataset in the revision. We focus the evaluation on the sense selection module of the model and classify the senses in an unsupervised fashion. Our model achieves the highest accuracy among competing models (Table 2), except for DeConf which is a supervised sense model that annotates senses on the same lexical resource (WordNet) that was used to build WiC.
We believe that the word similarity tasks demonstrate that each sense-specific embedding learned by our model captures good semantics in addition to better sense disambiguation ability. The high quality of each sense embeddings demonstrates the benefits of using Gumbel softmax.
Therefore we decide to keep this evaluation in the revision, but it is more meaningful alongside the WiC results.
2) Novelty
The only difference between the proposed one and Lee and Chen (2017) is Gumbel softmax instead of reinforcement learning between sense selection and representation learning modules.
We appreciate that the reviewer noticed the similarity between our models with MUSE by Lee and Chen (2017), as both try to improve the sense selection module with hard attention.
However, the overall structure of our model (Figure 1) is quite different, in addition to using Gumbel Softmax (GS) instead of RL for hard attention, we’d like to explain the differences on two key aspects:
Model structure and parameters: MUSE learns *four* sets of parameters: sense representations for target words U, collocation context representations V, and two additional matrix P, Q to estimate sense selection distribution for both target words and contexts (both target and context have multiple senses).
In contrast, ours learns *two* sets of parameters (Section 3.1): sense representations for target words S and global context representations C. We use C to disambiguate senses of target words S instead of using additional parameters like in MUSE, which reduces the number of total parameters in our model.
Furthermore, we update S and C in both the sense selection and context prediction modules, as these two modules are “symmetric” (predicting senses by context and predict context by word sense) and both help to capture the semantics in words.
Moreover, similar to Neelakantan et al. (2014), we do not disambiguate senses for context words (one global vector per context word) to further reduce the parameter size.
Optimization function: We use the (scaled) GS instead of straight-through (scaled) GS to have a stronger error signal (update not only the senses that are chosen but also the ones that are not).  To use a distribution instead of a one-hot selection and reduce the computational cost by negative sampling, we optimize the lower bound of the original negative sampling Skip-Gram objective with marginalization and Jensen's Inequality. Using straight-through (scaled) GS learns worse sense embeddings (lower word similarity score and human-model consistency) than (scaled) GS. Due to space limitations, we didn’t include the comparison in the paper. RL methods are similar to ST-GS since they also make a hard selection each time and update the selected senses but not others.
the idea from the proposed model is similar to Li and Jurafsky (2015), because the sense selection is not one-hot but a distribution.
Li and Jurafsky (2015) sample one-hot senses during the training with Chinese Restaurant Process (CRP) and model the CRP with a distribution;
while we directly use the distribution and implement the standard skip-gram objective with marginalization over senses.
3) Error analysis
Moreover, there is no error analysis about the result on the proposed contextual word sense selection task, which may shed more light on the strength and weakness of the model.
We appreciate the reviewer’s suggestion! We add the error analysis on the crowdsourced contextual word sense selection task in the revision (Section 6.2 Error Analysis).
After seeing the author responses, my score is updated.
The revised paper includes more experiments (almost all I can think of) and detailed analysis.
The difference between the proposed model and the prior work can be better elaborated for explicitly pointing out the novelty.
Also, the improved performance seems convincing for the proposed model, and it will be better to see the published code for encouraging researchers to easily follow up this direction.
To sum up, this is a good paper that can motivate the following research in the related field.
We thank the reviewer for the new comments! We'll elaborate on the differences between our model with prior works with a little more detail in the next revision after the decision been made. And we'll release our code and data for the human evaluation as soon as possible.
* Summary
This paper extends the skipgram model using one vector per sense of a word. Based on this, the paper proposes two models for training sense embeddings: One where the word senses are marginalized out with attention over the senses, and the second where only the sense with highest value of attention contributes to the loss. For the latter case, the paper uses a variant of Gumbel softmax for training. The paper shows evaluations on benchmark datasets that shows that the Gumbel softmax based method is competitive or better than other methods. Via a crowdsourced evaluation, the paper shows that the method also produces human interpretable clusters.
* Review
This paper is generally well written and presents a plausible solution for the problem of discovering senses in an unsupervised fashion.
If \beta=0, then we get SASI, right? How well does this perform on the non-contextual word similarity task?
Also, on the crowd sourced evaluation? The motivation for the hard attention/Gumbel softmax is to learn sense representations that are distinguishable.
But do the experiments test this?
There's something strange about Eq 6. If I understand this correctly, \tilde{c_i} is the context and c_j^i is the j^th context word.
Then P(c_j^i | w, \tilde{c_i}) should be 1 because the context is given, right?
While the motivation for the right hand side makes sense, the notation could use work.
The description of how the number of senses is pruned in section 3.1 seems to be a bit of a non sequitur. It is not clear whether this is used in the experiments and if so, how it compares. The appendix gives more details, but it seems a bit out of place even then because the evaluations don't seem to use it.
* Minor comments
There are some places where the writing could be cleaned up.
- Eq 16 changes the notation for the sense embeddings and the context words from earlier, say Eq 12.
- Parenthetical citations would be more appropriate in some places Eg: above Eq 3, in footnote 3
- Page 6, above 6.2: Figure-Figure?
- Page 9, Agreement paragraph: hight -> highest
We thank the reviewer for taking time to read our paper and the useful suggestions on improving our writing! We address the specific points from the reviewer as follows:
If \beta=0, then we get SASI, right? How well does this perform on the non-contextual word similarity task?
Also, on the crowdsourced evaluation?
We thank the reviewer for this suggestion! We’ve added the results in Table 3 in the revision. SASI generally performs poorly on the word similarity tasks, so we focus our comparison between our main model GASI-beta with the baseline models given limited space.
The motivation for the hard attention/Gumbel softmax is to learn sense representations that are distinguishable.
But do the experiments test this?
Our crowdsourced contextual sense selection task evaluates this property. The raters need to distinguish between the learned senses in order to make a selection (Section 6.2, sense disambiguation and interpretability). We also add more detail to these experiments in the additional error analysis in the revision.
There's something strange about Eq 6. ……,  While the motivation for the right hand side makes sense, the notation could use work.
We address the notation issue in the revision.
The description of how the number of senses is pruned in section 3.1 seems to be a bit of a non sequitur.
We thank the reviewer’s suggestion. Since it’s not the focus of our paper, in our revision we move the descriptions of pruning to the appendix.
The paper presents a method for deriving multi sense word embeddings. The key idea behind this method is to learn a sense embedding tensor using a skip-gram style training objective. The objective defines the probability of contexts marginalised over latent sense embeddings. The paper uses Gumbel-softmax reparametrization trick to approximate sampling from the discrete sense distributions. The method also uses a separate hyperparameter to help scale the dot product appropriately.
Strengths:
1. The technique is a well-motivated solution for a hard problem that builds on the skip-gram model for learning word embeddings.
2. A new manual evaluation approach for comparing sense induction approaches.
3. The empirical advance while relatively modest appears to be significant since the technique seems to yield better results than multiple baselines across a range of tasks.
Suggestions:
1. The number of senses is fixed to three. This is a bit arbitrary, even though it is following some precedence. I like the information in the appendix that shows how to handle cases when there are duplicate senses induced for words that dont have many senses. It would be useful to know how to handle the cases where a word can have more than three senses. Given that the authors have a way of pruning duplicate senses, it would have been interesting to try a few basic methods that select the number of senses per word dynamically.
2. The evaluation includes word similarity task and crowdsourcing for sense intrusion and sense selection. These provide a measure of intrinsic quality of the sense based embeddings.
However, as Li and Jurafsky (2015) point out, typically applications use more powerful models that use a wide context. It is not clear how these improvements to sense embeddings will translate in these settings. It would have been useful to have at least one or two end applications to illustrate this.
3. Given that the empirical gains are not quite consistent, I would encourage the authors to specifically argue why this particular method should be favoured over other existing methods. The related work discussion merely highlights methodological differences.
For example, the contrast with Lee and Chen (2017) seems to be only that of differentiability. Is the claim that differentiability is desirable because this allows for fine tuning in applications? If this is the case then it will be nice to have this verified.
4. The lower bound on the log likelihood objective is good but what are we supposed to take away from it? Is it that there is an interpretation that allows us to get away with negative sampling?
Overall I like the paper. It presents an application of the Gumbel-softmax trick for sense embeddings induction and shows some empirical evidence for the usefulness of this idea, including some manual evaluation.
I think the evaluation could be strengthened with some end applications and much crisper arguments on why the method is preferable over other methods that achieve comparable performance.
References:
[Li and Jurafsky., EMNLP 2015] Do Multi-Sense Embeddings Improve Natural Language Understanding?
We thank the reviewer for taking time to read our paper and the useful suggestions! We address the suggestions from the reviewer as follows:
1）dynamic number of senses and evaluation on downstream applications.
We thank the reviewer for these two suggestions! The simplest way to model words that have more than 3 senses is to initialize all words with more senses and prune aggressively; we set K=3 mainly for purpose of comparison. We think both implementing a dynamic number of senses (e.g., by setting a threshold to split senses) and evaluating on end tasks are great ideas; given the limited space, we’ll address these in future work.
2）benefits of differentiability
In addition to updating the sense selection module and context prediction module at the same time, full differentiability allows updates to flow to all senses, not only the ones chosen by the attention, which results in stronger error signals and better sense selection ability.
While approximating hard attention still guarantees that the model will focus on specific senses so that each sense captures good semantics (Table 3) and is interpretable to humans (Section 6), the Gumbel-softmax trick helps to guarantee both with the original objective, and we don’t need additional parameters for the policy network in RL.
For example, the contrast with Lee and Chen (2017) seems to be only that of differentiability.
We also contrast the sense selection module with Lee and Chen (2017) in the related work and Section 3.1. The overall structure of the two models are actually different, Lee and Chen (2017) learn *four* set of parameters while we learn *two*. Given limited space, we don’t elaborate further in our paper, but we discuss this with more details in our response to Reviewer 3.
3) negative sampling and lower bound
The negative sampling (NS) is for reducing the computational cost. To still optimize our original objective while implementing NS, we deduce the lower bound by Jensen’s Inequality.

Meta-review:
role: AC
data: metareview: 
Pros:

*  High quality evaluation across different benchmarks, plus human eval

*  The paper is well written (though one could quibble about the motivation for the method, see Cons)

Cons:

*  The approach is incremental, the main contribution is replacing marginalization or RL with G-S. G-S has already been studied in the context of VAEs with categorical latent variables, i.e. very similar models.

*  The main technical novelty is varying amount of added noise (i.e. downscaling Gumbel noise). In principle, the Gumbel relaxation is not needed here as exact marginalization can be done (as) effectively. Unlike the standard strategy used to make discrete r.v. tractable in complex models, samples from G-S are not used in this work to weight input to the 'decoder' (thus avoiding expensive marginalization) but to weight terms corresponding to reconstruction from individual latent states (in constract, e.g., to SkimRNN of Seo et al (ICLR 2018)). Presumably adding noise to softmax helps to force sharpness on the posteriors (~ argmax in previous work) and stochasticity may also help exploration.  

(Given the above, "to preserve differentiability and circumvent the difficulties in training with reinforcement learning, we apply the reparameterization trick with Gumbel softmax" seems slightly misleading)


*  With contextualized embeddings, which are sense-disambiguated given the context, learning discrete senses (which are anyway only coarse approximations of reality) is less practically important

Two reviewers are somewhat lukewarm (weak accept) about the paper (limited novelty), whereas one reviewer is considerably more positive. I do not believe that the reviews diverge in any factual information though.





Decision: Reject