Title: A DIFFERENTIABLE SELF-DISAMBIGUATED SENSE EM-BEDDING MODEL VIA SCALED GUMBEL SOFTMAX

Abstract: We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources. It jointly learns how to disambiguate senses given local context and how to represent senses using hard attention. Unlike previous multi-prototype models, our model approximates differentiable discrete sense selection via a modified Gumbel softmax. We also propose a novel human evaluation task that quantitatively measures (1) how meaningful the learned sense groups are to humans and (2) how well the model is able to disambiguate senses given a context sentence-an evaluation ignored by previous models. Our model not only discovers distinct, interpretable embeddings but is competitive against previous models on word similarity tasks.
Machine learning models for natural language processing applications often represent words with real valued vector embeddings. Popular word embedding models such as Word2Vec (Mikolov et al., 2013a; b) and GloVe (Pennington et al., 2014) enabled state-of-the-art results on myriad NLP tasks such as sentiment analysis (

:

However, for polysemous words (those with multiple senses), learning a single vector for each word type conflates different meanings (e.g., "A hydrogen bond exists between water molecules." vs. "Do you want to buy this bond?"). This is not a new problem- Schütze (1998) demonstrates the deficiency of assigning just one vector per word-but it is more pernicious in modern models, as conflated senses can pull semantically unrelated words toward each other in the embedding space (Neelakantan et al., 2014; Pilehvar & Collier, 2016; Camacho-Collados & Pilehvar, 2018) . To disentangle distinct senses in word embeddings and learn finer-grained semantic clusters, multi-prototype word embedding models learn multiple sense-specific embeddings for a single word (Section 7).

But what makes a good multisense word embedding? While word similarity is the most common evaluation, it has many detractors (Faruqui et al., 2016; Gladkova & Drozd, 2016) : similarity is subjective and is hard to be differentiate from word relatedness. Moreover, word similarity taskswith the exception of Stanford Contextual Word Similarity (Huang et al., 2012, SCWS) -ignore polysemous cases or are tied to specific sense inventories (Boyd-Graber et al., 2006) .

Moreover, these evaluations ignore a key component of learning sense inventories: do they make sense to a human? Previous multisense embedding papers present nearest neighbors to claim their representations are interpretable and useful. Like topic models, these implicit interpretability claims need to be rigorously verified. In Section 6, we adapt techniqes for evaluating topic models (Chang et al., 2009) to measure whether learned sense groups are internally coherent and whether humans can consistently match a learned sense vector to a word in context. Just like topic models, word embedding models that win conventional evaluations do not always make sense to humans.

We present a simple method that not only correlates well with traditional word similarity evaluations (Section 5) but also discovers interpretable (measured by human evaluations) sense embeddings (Section 6). Our model extends the Skip-Gram Word2Vec model and simultaneously learns (1) automatic sense induction given local context and (2) sense-specific embeddings. To learn disentangled sense representations (i.e., avoid sense mixing), we approximate hard attention and preserve 2). This modeling contribution-Scaled Gumbel Softmax-is critical for disambiguating senses.

2. FOUNDATIONS: SKIP-GRAM AND GUMBEL SOFTMAX:

Our model extends Skip-Gram Word2Vec (Mikolov et al., 2013a; b) , which jointly learns word embeddings W ∈ R |V |×d and context embeddings C ∈ R |V |×d . More specifically, given a vocabulary V and embedding dimension d, it maximizes the likelihood of the context words c i j that surround a given center word w i in a context window ci ,

EQUATION

where P (c i j | w i ) is estimated by a softmax over all possible context words, i.e, the vocabulary,

EQUATION

In practice, log P (c i j | w i ) is approximated by negative sampling to reduce computational cost.

2.1. GUMBEL SOFTMAX:

The Gumbel softmax (Jang et al., 2016; Maddison et al., 2016) approximates the sampling of discrete random variables. Given a discrete random variable X with P (X = k) ∝ α k , α k ∈ (0, ∞), the Gumbel-max (Gumbel & Lieblein, 1954; Maddison et al., 2014) refactors the sampling of X into

EQUATION

where the Gumbel noise g k = -log(-log(u k )) and u k are i.i.d samples drawn from Uniform(0, 1).

The Gumbel softmax approximates sampling one hot(arg max k (log

EQUATION

3. GUMBEL-ATTENTION SENSE INDUCTION (GASI):

Building on these foundations, we now introduce our model, GASI, and along the way introduce a soft-attention stepping-stone (SASI); afterward, we will compare models on both traditional evaluation metrics and interpretability. The critical component of our model is that we model the sense selection probability, which can be interpreted as sense attention over contexts, into the Skip-Gram model while preserving the original objective through marginalization (Figure 1 ). By using Gumbel Softmax, our model both approximates discrete sense selection and is differentiable. Previous models are either non-differentiable or otherwise complicate inference through hard attention with reinforcement learning methods (Lee & Chen, 2017) .

3.1. ATTENTIONAL SENSE INDUCTION FRAMEWORK:

Embedding Parameters We learn a context embedding matrix C ∈ R |V |×d and a sense embedding tensor S ∈ R |V |×K×d . Unlike previous work (Neelakantan et al., 2014; Lee & Chen, 2017) , no extra embeddings are kept for sense induction. Sense Attention in Objective Function Assuming a center word w i has senses {s i 1 , s i 2 , . . . , s i K }, the original Skip-Gram likelihood can be written as marginal distribution over all senses of w i with the sense induction probability P (s i k | w i ), we focus on the sense disambiguation given local context ci and estimate

EQUATION

Replacing P (c i j | w i ) in Equation 1with Equation 5 gives our objective function

EQUATION

Lower Bound the Objective for Negative Sampling Like the Skip-Gram objective (Equation 2), we model the likelihood of a context word given the center sense P (c i j | s i k ) using softmax,

EQUATION

where the bold symbol s i k is the vector representation of sense s j k from S, and c j is the context embedding of word c j from C.

Computing the softmax over the vocabulary is time-consuming. We want to adopt negative sampling to approximate log P (c i j | s i k ), which does not exist explicitly in our objective function (Equation 6).2 However, given the concavity of the logarithm function, we can apply Jensen's inequality,

EQUATION

and create a lower bound of the objective. Maximizing this lower bound gives us a tractable objective,

EQUATION

where log P (c i j | s i k ) is estimated by negative sampling Mikolov et al. (2013b) ,

EQUATION

u k ∼ U(0, 1) c⊤ i s i k g k : Gumbel noise

Figure 3: Our hard attention mechanism is approximated with Gumbel softmax on the context-sense dot product c i s i k (Equation 13), whose mean and std plotted here as a function of iteration. The shadowed area shows that it has a smaller scale than the gumbel noise g k , such that g k , rather than the embeddings, dominates the sense attention.

Modeling Sense Attention We can model the attention term, contextual sense induction distribution, with soft attention; we call the resulting model soft-attention sense induction (SASI); although it is a stepping stone to our final model, we compare against it in our experiments as it helps isolate the contributions of hard attention. In SASI, the sense attention is conditioned on the entire local context ci with softmax:

EQUATION

)

where ci is the mean of the context vectors in ci .

3.2. SCALED GUMBEL SOFTMAX FOR SENSE DISAMBIGUATION:

To reduce separate senses and learn distinguishable sense representations, we implement hard attention in our full model, GASI. To preserve differentiability and circumvent the difficulties in training with reinforcement learning (Sutton & Barto, 1998), we apply the reparameterization trick with Gumbel softmax (Section 2.1) to our sense attention function (Equation 11) and make a continuous relaxation.

Vanilla Gumbel Attention:

The discrete sense sampling from Equation 11 can be refactored by

EQUATION

and the hard attention is approximated with

EQUATION

Scaled Gumbel Softmax for Sense Disambiguation Gumbel softmax learns a flat distribution over senses even with low temperatures (Figure 2 ): the dot product c i s i k is too small compared to the Gumbel noise g k (Figure 3 ). 3 Thus we use a scaling factor β to reduce the randomness,4 and tune it as a hyperparameter.5 

EQUATION

We use GASI-β to identify the GASI model with scaling factor. This modification is critical for learning distinguishable senses (Figure 2 , Table 1, and Table 5 ).

Final Objective Function:

The objective function of our GASI-β model is Trained on same dataset and vocabulary, both models learn three vectors per word. Here, word i represent the i-th vector for word. Our GASI (right) learns three distinct senses of "bond" while MUSE (left) learns overlapping senses.

EQUATION

4. TRAINING SETTINGS:

For fair comparisons, we try to remain consistent with previous work (Huang et al., 2012; Neelakantan et al., 2014; Lee & Chen, 2017 ) in all aspects of training. In particular, we train GASI on the same April 2010 Wikipedia snapshot (Shaoul C., 2010) with 1B tokens the same vocabulary released by Neelakantan et al. (2014) ; set the number of senses K = 3 and dimension d = 300 for each word unless otherwise specified. More details are in Appendix A. We fix the temperature τ = 0.5,6 and tune the scaling factor β from {0.1, 0.2, ...,0.9} on the AvgSimC measure for the contextual word similarity task (Section 5). The optimal scaling factor β is 0.4. If not reprinted, numbers for competing models are either computed with pre-trained embeddings released by authors or trained on released code.7 

5. WORD SIMILARITY EVALUATION:

We first compare our GASI and GASI-β model with previous work on standard word similarity tasks before turning to interpretability experiments. Each task has word pairs with a similarity/relatedness score. For evaluation, we measure Spearman's rank correlation ρ (Spearman, 1904) between word embedding similarity and the gold similarity judgements: higher scores imply the model captures semantic similarities consistent with the trusted similarity scores.

Contextual Word Similarity Tailored for sense embedding evaluation, Stanford Contextual Word Similarities (Huang et al., 2012, SCWS) has 2003 word pairs and similarity scores with sentential context. Moreover, the word pairs and their contexts reflect homonymous and polysemous words. Therefore, we use this dataset to tune our hyperparameters.

To compute the word similarity with senses we use two metrics Reisinger & Mooney (2010) that take context and sense disambiguation into account: MaxSimC computes the cosine similarity cos(s * 1 , s * 2 ) between the two most probable senses s * 1 and s * 2 that maximizes P (s i k | w i , ci ). AvgSimC weights average similarity over the combinations of all senses

K i=1 K i=j P (s 1 i | w 1 , c1 )P (s 2 j | w 2 , c2 ) cos(s 1 i s 2 j ).

We compare variants of our model with multi-prototype sense embedding models (Table 1 ), including two previous state-of-the-art models: the clustering-based Multi-Sense Skip-Gram model (Neelakantan et al., 2014, MSSG) 2 ). GASI-β has the best AvgSimC and a competitive MaxSimC. While MUSE has a higher MaxSimC than GASI-β, it fails to distinguish senses as well (Figure 4 , Section 6). The Probabilistic FastText Gaussian Mixture (Athiwaratkun et al., 2018, PDF-GM) is SOTA on multiple non-contextual word similarity tasks (Table 3 ). Without sense selection module given context, we evaluate PDF-GM on MaxSim (Equation 16), which is 66.4. Our GASI-β has the same on MaxSim, and better correlation on AvgSimC (69.5). (Miller & Fellbaum, 1998, WordNet) used to build WiC.

Word Sense Selection in Context:

This evaluation, however, does not reflect the interpretability of the senses themselves. We address this in Section 6.

Non-Contextual Word Similarity:

To evaluate the semantics captured by each sense-specific embeddings, we compare the models on the non-contextual word similarity datasets: RG-65 (Rubenstein & Goodenough, 1965); SimLex-999 (Hill et al., 2015) ; WS-353 (Finkelstein et al., 2002) ; MEN-3k (Bruni et al., 2014) ; MC-30 (Miller & Charles, 1991) ; YP-130 (Yang & Powers, 2006) ; MTurk-287 (Radinsky et al., 2011) ; MTurk-771 (Halawi et al., 2012) ; RW-2k (Luong et al., 2013) . Similar to Lee & Chen (2017) and Athiwaratkun et al. (2018) , we compute the word similarity based on senses by MaxSim (Reisinger & Mooney, 2010) , which maximizes the cosine similarity over the combination of all sense pairs and does not require local contexts,

EQUATION

GASI-β has better correlation on three datasets, is competitive on the rest (Table 3 ), and remains competitive without scaling. GASI is better than MUSE, the other hard-attention multi-prototype model, on six datasets and worse on three. Our model can reproduce word similarities as well or better than existing models through our sense selection. GASI-β outperforms the other models on three datasets are competitive on the others. Note that PFT-GM are trained with two components/senses while other models learn three senses.

6. CROWDSOURCING EVALUATION:

GASI can capture word similarity (Section 5), but do the learned representations make sense? Could a human use them to help build a dictionary? If you show a human the senses, can they understand why a model would assign a sense to that context? In this section we evaluate whether the representations make sense to human consumers of multisense models.

Qualitive analysis Previous papers use nearest neighbors of a few examples to qualitatively argue that their models have captured meaningful senses of words. We also give an example in Figure 4 , which provides an intuitive view on how the learned senses are clustered by visualizing the nearest neighbors of word "bond" using t-SNE projection (Maaten & Hinton, 2008) . Our proposed model (right) disentangles the three sense of "bond" clearly and learns three distinct sense vectors.

However, the examples can be cherry-picked and lack standards. This problem also bedeviled topic modeling until the introduction of rigorous human evaluation (Chang et al., 2009) . We adapt both aspects Chang et al's evaluations: word intrusion (Schnabel et al., 2015) to evaluate whether individual senses are coherent and topic intrusion-rather sense intrusion in this case-to evaluate whether humans agree with models' sense assignments in context. Both crowdsourcing tasks collect human inputs on Figure-Eight. We compare our models with two previous state-of-the-art multi-prototype sense embeddings models that disambiguate senses given local context, i.e., MSSG (Neelakantan et al., 2014) and MUSE (Lee & Chen, 2017 ).10 

6.1 WORD INTRUSION FOR SENSE COHERENCE Schnabel et al. (2015) suggests a "good" word embedding should have coherent neighbors and evaluate coherence by word intrusion. They presents crowdworkers four words: three are close in embedding space while one of which is an "intruder". If the embedding makes sense, contributors will easily spot the word that "does not belong".

Similarly, we examine the coherence of ten nearest neighbors of senses in the contextual word sense selection task (Section 6.2) and replace one neighbor with an "intruder" (Figure 5 ). We generate three intruders for each sense and collect three judgements per intruder. We consider the "intruder" to be correctly selected if at least two judgements are correct. Like Chang et al. (2009) , we want the "intruder" to not be too different in terms of frequency to the target set but not too similar semantically. For sense s m i of word type w i , we randomly select a word from the neighbors of another sense s n of w i but with a low threshold, i.e., any words that has cosine similarity larger than 0.0 can be viewed as a neighbor.

Result and Analysis:

All models have comparable model accuracy. GASI-β learns senses that have the highest coherency among top ten nearest neighbors while MUSE learns more sense mixtures.

Inter-rater Agreement We use the aggregated confidence score provided by Figure-Eight to estimate the level of agreement between multiple contributors. 11 The agreements are high for all models and our GASI-β has the highest agreement, suggesting that the senses learned by GASI-β are easier to interpret.

6.2. CONTEXTUAL WORD SENSE SELECTION:

The previous task measures whether individual senses are coherent. In this task, we measure whether the learned senses by sense embedding models make sense human and evaluate the models' ability to disambiguate senses in context.

Task Description Given a target word in context, we ask a crowdworker to select which sense group best fits the sentence. Each sense group is described by its top ten distinct nearest neighbors (Figure 6 ).12 Data Collection We select fifty nouns with five sentences from SemCor 3.0 (Miller et al., 1994) . We first filter all word types with fewer than ten sentences and select the fifty most polysemous nouns from WordNet (Miller & Fellbaum, 1998) among the remaining senses. For each noun, we randomly select five sentences.

Metrics For each model, we collect three judgements for each question. We consider a model correct if at least two crowdworkers select the same sense as the model. We also consider the probability P assigned to the human choices by the model, indicating the model's confidence in sense selection. P = 1/3 indicates the model learns flat, uniform sense induction distribution is unable to disambiguate senses.

Sense disambiguation and interpretability:

If humans consistently pick the same sense as the model: 1) humans can interpret the nearest neighbor words (as measured by the previous experiment); 2) the senses are distinguishable to human; 3) the human's choice is consistent with the model's.

Results and Analysis GASI-β selects senses that are most consistent with humans; it has the highest accuracy and assigns the largest probability assigned to the human choices (Table 5 ). Thus, GASI-β produces sense embeddings that are both more interpretable and distinguishable. GASI without a scaling factor, however, has low consistency and flat sense distribution.

Inter-rater Agreement We use the confidence score computed by Figure-Eight to estimate the rater's agreement for this task as well. Our GASI-β achieves the highest human-model agreement while both MUSE and GASI without scaling have the lowest. Table 6 : Similarities of human and model choices when they disagree (error) vs. similarities between the senses that both human and model select with other senses in the same word (correct). Human agrees with the model when the senses are distinct.

Error Analysis Next, we attempt to answer why crowdworkers disagree with the model although they can interpret most senses (measured by the word intruder task, Table 4 ). Is it that the model has learned duplicate senses that both the users and model cannot distinguish or is it that crowdworkers agree with each other but disagree with the model? The former relates to the model's ability in learning human distinguishable senses; while the latter relates to the model's ability in contextual sense selection.

Two trends reveal that duplicated senses that are not distinguishable to humans are one of the main causes of human-model disagreement. First, users agree with the model when the senses are distinct (Table 6 , correct), while disagreement rises with more similar senses (Table 6 , error); second, more distinct senses allows higher inter-rater agreement (Figure 7 ). We measure distinctness both by counting the number of shared nearest neighbors and the average cosine simlarities of GloVe embeddings. 13 Specifically, MUSE learns duplicate senses for most words, preventing users from choosing appropriate senses and results in random human-model agreement. GASI-β learns some duplicated senses and some distinguishable senses. MSSG appears to learn the least similar senses, but they are not distinguishable enough for humans. For MSSG, small neighbor overlaps do not necessarily help humans to distinguish between senses: users disagree with each other (agreement 0.33) even when the number of overlaps is very small (Figure 7 ). An intuitive example is shown in Table 7 , which demonstrates the necessity of human evaluation. If we use rater agreement to measure how distinguishable the learned senses are to humans, GASI-β learns the most distinguishable senses (histogram in Figure 8 ).

Figure 8 also shows that the model is more likely to agree with humans when humans agree more with each other (as a result of more distinct senses), i.e., human-model consistency correlates with rater agreement (Figure 8 ). MSSG disagrees with humans more even when raters agree with each other, indicating worse sense selection ability.

6.3. WORD SIMILARITY VS. SENSE DISAMBIGUATION:

The evaluation results on word similarity tasks (Section 5) and human evaluations (Section 6) are inconsistent for several models. GASI, GASI-β model and the MUSE model are all competitive in word similarity (Table 1 and Table 3 ), but only GASI-β also does well in the human evaluations (Table 5 ). Both GASI without scaling and MUSE fail to learn distinguishable senses and cannot disambiguate senses given local context. High word similarities do not necessarily indicate "good" sense embeddings quality; our human evaluation-contextual word sense selection-is complementary.

7 RELATED WORK Schütze (1998) introduces context-group discrimination for senses and uses the centroid of context vectors as a sense representation. Other work induces senses by context clustering (Purandare & Pedersen, 2004) or probabilistic mixture models (Brody & Lapata, 2009) . Reisinger & Mooney (2010) first introduce multiple sense-specific vectors for each word, inspiring other multi-prototype sense embedding models. Generally, to address polysemy in word embeddings, some previous work trained on annotated sense corpora (Iacobacci et al., 2015) or external sense inventories (Labutov & Lipson, 2013; Chen et al., 2014; Jauhar et al., 2015; Chen et al., 2015; Wu & Giles, 2015; Pilehvar & Collier, 2016; Mancini et al., 2017) ; Rothe & Schütze (2015; 2017) extend word embeddings to lexical resources without training; others induce senses via multilingual parallel corpora (Guo et al., 2014; Šuster et al., 2016; Ettinger et al., 2016) .

We contrast our GASI to unsupervised monolingual multi-prototype models along two dimensions: sense induction methodology and differentiability. 1), Huang et al. (2012) 

8. CONCLUSION:

The goal of multi-sense word embeddings is not just to win word sense evaluation datasets; rather, they should also describe language: given millions of tokens of a language, what are the patterns in the language that can help a lexicographer or linguist in day-to-day tasks like building dictionaries or understanding semantic drift. Our differentiable Gumbel Attention Sense Induction (GASI) offers a best of both worlds: comparable word similarities while also learning more distinguishable, interpretable senses. A TRAINING DETAILS

During training, we fix the window size to five and the dimensionality of the embedding space to 300 for comparison to previous work. We initialize both sense and context embeddings randomly within U(-0.5/dim, 0.5/dim) as in Word2Vec. We set the initial learning rate to 0.01; it is decreased linearly until training concludes after 5 epochs. The batch size is 512, and we use five negative samples per center word-context pair as suggested by Mikolov et al. (2013a) . The subsample threshold is 1e-4. We train our model on the GeForce GTX 1080 Ti, and our implementation (using pytorch 3. 

B NUMBER OF SENSES:

For simplicity and consistency with most of previous work, we present our model with a fixed number of senses K.

B.1 POST-TRAINING PRUNING:

For words that do not have multiple senses or have most senses appear very low-frequently in corpus, our model (as well as many previous models) learns duplicate senses. We can easily remove such duplicates by pruning the learned sense embeddings with a threshold λ. Specifically, for each word w i , if the cosine distance between any of its sense embeddings (s i m , s i n ) is smaller than λ, we consider them to be duplicates. After discovering all duplicate pairs, we start pruning with the sense s i k that has the most duplications and keep pruning with the same strategy until no more duplicates remain.

Model-specific pruning We estimate a model-specific threshold λ from the learned embeddings instead of deciding it arbitrary. Therefore, this pruning methods is also applicable to other sense embedding models. We first sample 100 words from the negative sampling distribution over the vocabulary. Then, we retrieve the five nearest neighbors (from all senses of all words) to each sense of each sampled word. If one of a word's own senses appears as a nearest neighbor, we append the distance between them to a sense duplication list D dup . For other nearest neighbors, we append their distances to the word neighbor list D nn . After populating the two lists, we want to choose a threshold that would prune away all of the sense duplicates while differentiating sense duplications with other distinct neighbor words. Thus, we compute λ = 1 2 (mean(D dup ) + mean(D nn )).

(17)

We can prune the duplicated senses for words that have senses less than K, details in Appendix B. We can also set different number of senses based on word frequency in the training, details in Appendix B.3.

Deriving the negative sampling requires the logarithm of a softmax(Goldberg & Levy, 2014).

Float32 precision, the saturation of log(σ(•)) and gradient vanishing result in a small range of c i s i k .

Normalizing c i s i k or directly using log P (s i k | wi, ci) results in a similar outcome.

Learning β instead of fixing it as a hyperparameter does not successfully disambiguate senses.

This is similar to the experiment settings for Gumbel softmax inMaddison et al. (2016)

We adopt the numbers forLi & Jurafsky (2015) fromLee & Chen (2017) and tune the PDF-GM(Athiwaratkun et al., 2018) model on the same 1B corpus and vocabulary as previous works using https://github.com/ benathi/multisense-prob-fasttext with suggested hyperparameters and select the best results.

Unavailable as of November 2018 at https://pilehvar.github.io/wic/

For words not in vocabulary or only have one sense learned, we chose randomly.

MSSG has two settings; we run human evaluation with MSSG-30K which has higher correlation with MaxSimC on SCWS.

https://success.figure-eight.com/hc/en-us/articles/201855939-How-to-Calculate-a-Confidence-Score

We shuffle the choices for questions with the same target word.

Different models learn different representations; we use GloVe for a uniform basis of comparison.

Figures and Tables:

You only live twice, Mr. Bond . . .

Figure 1: Network struture with an example of our GASI model which learns a set of global context embeddings C and a set of sense embeddings S

Figure 2: As the scale factor β increases, the sense selection distribution for "bond" given examples from SemCor 3.0 for synset "bond.n.02" becomes flatter, indicating less disambiguated sense vectors.

Figure4: t-SNE projections of nearest neighbors for "bond" by hard-attention models: 1) previous SOTA model MUSE(RL-based); 2) our proposed GASI-β. Trained on same dataset and vocabulary, both models learn three vectors per word. Here, word i represent the i-th vector for word. Our GASI (right) learns three distinct senses of "bond" while MUSE (left) learns overlapping senses.

Figure 5: Word intrusion task prompt

Figure 6: An example (target: bond) of the contextual word sense selection task; each option contains top ten nearest neighbors of a sense embedding learned by the model; senses in this example are from our GASI-β (1. 007; 2. chemical; 3. financial).

Figure 7: More distinct senses within each word lead to higher inter-rater agreement

Figure9: Histogram of number of senses left after post-training pruning for two models: GASI-0.4 initialized with three senses and GASI-0.4 initialized with five senses. We rank the number of senses of words by their frequency from high to low.

0) takes ∼ 6 hours to train one epoch on the April 2010 Wikipedia snapshot Shaoul C. (2010) with 100k vocabulary. For comparison, our implementation of Skip-Gram on the same framework takes ∼ 2 hours each epoch.

on AvgSimC metric and the RL-based Modularizing Unsupervised Spearman's correlation 100ρ on SCWS (trained 1B token, 300d vectors except forHuang et al.)

Table content: <table><tr><td>Model</td><td colspan="2">MaxSimC AvgSimC</td><td/><td/></tr><tr><td>Huang et al. (2012)-50d</td><td>26.1</td><td>65.7</td><td>Model</td><td>Accuracy(%)</td></tr><tr><td>MSSG-6K</td><td>57.3</td><td>69.3</td><td colspan="2">Unsupervised Multi-prototype models</td></tr><tr><td>MSSG-30K Tian et al. (2014) Li &amp; Jurafsky (2015) Qiu et al. (2016)</td><td>59.3 63.6 66.6 64.9</td><td>69.2 65.4 66.8 66.1</td><td>MSSG-30K MUSE Boltzmann GASI-β</td><td>54.00 52.14 55.27</td></tr><tr><td>Bartunov et al. (2016)</td><td>53.8</td><td>61.2</td><td colspan="2">Multi-prototype models</td></tr><tr><td>MUSE Boltzmann</td><td>67.9</td><td>68.7</td><td colspan="2">with external lexical resources</td></tr><tr><td>SASI</td><td>55.1</td><td>67.8</td><td>DeConf</td><td>58.55</td></tr><tr><td>GASI(w/o scaling)</td><td>68.2</td><td>68.3</td><td>SW2V</td><td>54.56</td></tr><tr><td>GASI-β</td><td>66.4</td><td>69.5</td><td/><td/></tr></table>

Unsupervised sense selection accuracy on Word in ContextSense Embeddings(Lee & Chen, 2017, MUSE)  on MaxSimC. All three are better than the baseline Skip-Gram model (65.2 using the word embedding). GASI better captures similarity than SASI, corroborating that hard attention aids word sense selection.

Table content: <table/>

SCWS evaluates models' ability of sense selection indirectly. We further compare GASI-β with previous SOTA, MSSG-30K and MUSE, on the Word in Context dataset(Pilehvar & Camacho-Collados, 2018, WiC)  which requires the model to identify whether a word has the same sense in two contexts. Lacking ground truth for the development set, 8 to reduce the variance in training and to focus on evaluating the sense selection module, we use an evaluation suited for unsupervised models: if the model selects different sense vectors given contexts, we mark that the word has different senses.9 For MUSE, MSSG and GASI-β, we use each model's sense selection module; for DeConf(Pilehvar & Collier, 2016) and SW2V(Mancini et al., 2017), we follow Pilehvar &Camacho-Collados (2018)  andPelevina et al. (2016) by selecting the closest sense vectors to the context vector. Results on DeConf are comparable to supervised results (59.4± 0.7). Our GASI-β has the best result apart from DeConf itself, which uses the same sense inventory

Table content: <table/>

Spearman's correlation 100ρ on non-contextual word similarity measured by MaxSim.

Table content: <table><tr><td>Dataset</td><td>MSSG-30K</td><td>MSSG-6K</td><td>MUSE Boltzmann</td><td>SASI</td><td>GASI</td><td>GASI-β</td><td>PFT-GM</td></tr><tr><td>SimLex-999</td><td>31.80</td><td>28.65</td><td>39.61</td><td colspan="2">31.56 40.14</td><td>41.68</td><td>40.19</td></tr><tr><td>WS-353</td><td>65.69</td><td>67.42</td><td>68.41</td><td colspan="2">58.31 68.49</td><td>69.36</td><td>68.6</td></tr><tr><td>MEN-3k</td><td>65.99</td><td>67.10</td><td>74.06</td><td colspan="2">65.07 73.13</td><td>72.32</td><td>77.40</td></tr><tr><td>MC-30</td><td>67.79</td><td>76.02</td><td>81.80</td><td colspan="2">70.81 82.47</td><td>85.27</td><td>74.63</td></tr><tr><td>RG-65</td><td>73.90</td><td>64.97</td><td>81.11</td><td colspan="2">74.38 77.19</td><td>79.77</td><td>79.75</td></tr><tr><td>YP-130</td><td>40.69</td><td>42.68</td><td>43.56</td><td colspan="2">48.28 49.82</td><td>56.34</td><td>59.39</td></tr><tr><td>MT-287</td><td>65.47</td><td>64.04</td><td>67.22</td><td colspan="2">64.54 67.37</td><td>66.13</td><td>69.66</td></tr><tr><td>MT-771</td><td>61.26</td><td>58.83</td><td>64.00</td><td colspan="2">55.00 66.65</td><td>66.70</td><td>68.91</td></tr><tr><td>RW-2k</td><td>42.87</td><td>39.24</td><td>48.46</td><td colspan="2">45.03 47.22</td><td>47.69</td><td>45.69</td></tr></table>



Table content: <table/>

Human-model consistency on contextual word sense selection; P is the average probability assigned by the model to the human choices. GASIβ is most consistent with human.

Table content: <table><tr><td>Model</td><td>Accuracy</td><td>P</td><td>Agreement</td><td/><td/><td>MUSE</td><td>MSSG</td><td>GASI-β</td></tr><tr><td>MUSE MSSG-30K</td><td>28.0 44.5</td><td>0.33 0.37</td><td>0.68 0.73</td><td>word overlaps</td><td>correct error</td><td>4.78 5.43</td><td>0.39 0.98</td><td>1.52 6.36</td></tr><tr><td>GASI (no β)</td><td>33.8</td><td>0.33</td><td>0.68</td><td>cosine sim</td><td>correct</td><td>0.86</td><td>0.33</td><td>0.36</td></tr><tr><td>GASI-β</td><td>50.0</td><td>0.48</td><td>0.75</td><td>by Glove</td><td>error</td><td>0.88</td><td>0.57</td><td>0.81</td></tr></table>

and Neelakantan et al. (2014) induce senses by context clustering; Tian et al. (2014) model a corpus-level sense distribution; Li & Jurafsky (2015) model the sense assignment as a Chinese Restaurant Process; Qiu et al. (2016) induce senses by minimizing an energy function on a context-depend network; Bartunov et al. (2016) model the sense assignment as a steak-breaking process; Nguyen et al. (2017) model the sense embeddings as a weighted combination of topic vectors with pre-computed weights by topic models; Athiwaratkun & Wilson (2017) and Athiwaratkun et al. (2018) model word representations as Gaussian Mixture embeddings where each Gaussian component captures different senses; Lee & Chen (2017) computes sense distribution by a separate set of sense induction vectors; while our GASI marginalizes the likelihood of contexts over senses and induces senses by local context vectors; the most similar sense selection module is a bilingual model( Šuster et al., 2016)  except that it does not introduce lower bound for negative sampling but uses weighted embeddings, which results in more sense mixture. 2), most sense selection models are non-differentiable and discretely select senses, with two exceptions:Šuster et al. (2016)  use weighted vectors over senses;Lee & Chen (2017) implement hard attention with RL to mitigate the non-differentiability. In contrast, GASI keeps full differentiability by reparameterization and approximates discrete sense sampling with scaled Gumbel softmax.

Table content: <table/>

The real question is -how are those four years used and what is their value as training?

Table content: <table/>

Peer review (iclr2019):
 This paper proposes GASI to disambiguate different sense identities and learn sense representations given contextual information. The main idea is to use scaled Gumbel softmax as the sense selection method instead of soft or hard attention, which is the novelty and contribution of this paper. In addition, the authors proposed a new evaluation task, contextual word sense selection, which can be used to quantitatively evaluate the semantic meaningfulness of sense embeddings. The proposed model achieves comparable performance on traditional word/sense intrinsic evaluation and word intrusion test as previous models, while it outperforms baselines on the proposed contextual word sense selection task. While the scaled Gumbel softmax is the claimed novelty, it is more like an extension of the original MUSE model (Lee and Chen, 2017), which proposed the sense selection and representation learning modules for learning sense-level embeddings. The only difference between the proposed one and Lee and Chen (2017) is Gumbel softmax instead of reinforcement learning between sense selection and representation learning modules. Therefore, the idea from the proposed model is similar to Li and Jurafsky (2015), because the sense selection is not one-hot but a distribution. The novelty of this paper is limited because the model is relatively incremental. From my perspective, the more influential contribution is that this paper points out the importance of evaluating sense selection capability, which is ignored by most prior work. Therefore, I expect to see more detailed evaluation on the selection module of the model. Also, because the task of this paper is multi-sense embeddings, the traditional word similarity (without contexts) task seems unnecessary. Moreover, there is no error analysis about the result on the proposed contextual word sense selection task, which may shed more light on the strength and weakness of the model. Finally, I suggest the authors remove the word-level similarity task and try the recently released Word in Context (WiC) dataset, which is a binary classification task that determines whether the meaning of a word is different given two contexts. It would be better to see that GASI performs well on this task given its better sense selection module. Overall, the contribution is somewhat incremental and the evaluation/discussion should focus more on the sense selection module. Considering the issues mentioned above, I will expect better quality for an ICLR paper.