A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax

We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources . It jointly learns how to disambiguate senses given local context and how to represent senses using hard attention . Unlike previous multi-prototype models , our model approximates differentiable discrete sense selection via a modified Gumbel softmax . We also propose a novel human evaluation task that quantitatively measures ( 1 ) how meaningful the learned sense groups are to humans and ( 2 ) how well the model is able to disambiguate senses given a context sentence—an evaluation ignored by previous models . Our model not only discovers distinct , interpretable embeddings but is competitive against previous models on word similarity tasks . 1 SENSE-SPECIFIC EMBEDDING . Machine learning models for natural language processing applications often represent words with real valued vector embeddings . Popular word embedding models such as Word2Vec ( Mikolov et al. , 2013a ; b ) and GloVe ( Pennington et al. , 2014 ) enabled state-of-the-art results on myriad NLP tasks such as sentiment analysis ( Kim , 2014 ; Tai et al. , 2015 ) and textual entailment ( Chen et al. , 2017 ) . However , for polysemous words ( those with multiple senses ) , learning a single vector for each word type conflates different meanings ( e.g. , “ A hydrogen bond exists between water molecules. ” vs. “ Do you want to buy this bond ? ” ) . This is not a new problem—Schütze ( 1998 ) demonstrates the deficiency of assigning just one vector per word—but it is more pernicious in modern models , as conflated senses can pull semantically unrelated words toward each other in the embedding space ( Neelakantan et al. , 2014 ; Pilehvar & Collier , 2016 ; Camacho-Collados & Pilehvar , 2018 ) . To disentangle distinct senses in word embeddings and learn finer-grained semantic clusters , multi-prototype word embedding models learn multiple sense-specific embeddings for a single word ( Section 7 ) . But what makes a good multisense word embedding ? While word similarity is the most common evaluation , it has many detractors ( Faruqui et al. , 2016 ; Gladkova & Drozd , 2016 ) : similarity is subjective and is hard to be differentiate from word relatedness . Moreover , word similarity tasks— with the exception of Stanford Contextual Word Similarity ( Huang et al. , 2012 , SCWS ) —ignore polysemous cases or are tied to specific sense inventories ( Boyd-Graber et al. , 2006 ) . Moreover , these evaluations ignore a key component of learning sense inventories : do they make sense to a human ? Previous multisense embedding papers present nearest neighbors to claim their representations are interpretable and useful . Like topic models , these implicit interpretability claims need to be rigorously verified . In Section 6 , we adapt techniqes for evaluating topic models ( Chang et al. , 2009 ) to measure whether learned sense groups are internally coherent and whether humans can consistently match a learned sense vector to a word in context . Just like topic models , word embedding models that win conventional evaluations do not always make sense to humans . We present a simple method that not only correlates well with traditional word similarity evaluations ( Section 5 ) but also discovers interpretable ( measured by human evaluations ) sense embeddings ( Section 6 ) . Our model extends the Skip-Gram Word2Vec model and simultaneously learns ( 1 ) automatic sense induction given local context and ( 2 ) sense-specific embeddings . To learn disentangled sense representations ( i.e. , avoid sense mixing ) , we approximate hard attention and preserve You only live twice , Mr . Bond . . . c̄i ci1 c i m ... G M predict contexts G M Gumbel softmax Marginalization context words center word context embeddings lookup sense embeddings lookup sense attention C S ... wic̃i P ( sik|wi , c̃i ) chemical 007 financial si1 si2 siK P ( cij |wi ) Figure 1 : Network struture with an example of our GASI model which learns a set of global context embeddings C and a set of sense embeddings S differentiability via a scaled variant of the Gumbel Softmax function ( Section 3.2 ) . This modeling contribution—Scaled Gumbel Softmax—is critical for disambiguating senses . 2 FOUNDATIONS : SKIP-GRAM AND GUMBEL SOFTMAX . Our model extends Skip-Gram Word2Vec ( Mikolov et al. , 2013a ; b ) , which jointly learns word embeddings W ∈ R|V |×d and context embeddings C ∈ R|V |×d . More specifically , given a vocabulary V and embedding dimension d , it maximizes the likelihood of the context words cij that surround a given center word wi in a context window c̃i , J ( W , C ) ∝ ∑ wi∈V ∑ cij∈c̃i logP ( cij |wi ; W , C ) , ( 1 ) where P ( cij |wi ) is estimated by a softmax over all possible context words , i.e , the vocabulary , P ( cij |wi ; W , C ) = exp ( cij > wi ) ∑ c∈V exp ( c > wi ) . ( 2 ) In practice , logP ( cij |wi ) is approximated by negative sampling to reduce computational cost . 2.1 GUMBEL SOFTMAX . The Gumbel softmax ( Jang et al. , 2016 ; Maddison et al. , 2016 ) approximates the sampling of discrete random variables . Given a discrete random variable X with P ( X = k ) ∝ αk , αk ∈ ( 0 , ∞ ) , the Gumbel-max ( Gumbel & Lieblein , 1954 ; Maddison et al. , 2014 ) refactors the sampling of X into X = arg max k ( logαk + gk ) , ( 3 ) where the Gumbel noise gk = − log ( − log ( uk ) ) and uk are i.i.d samples drawn from Uniform ( 0 , 1 ) . The Gumbel softmax approximates sampling one hot ( arg maxk ( logαk + gk ) ) by yk = softmax ( ( logαk + gk ) /τ ) . ( 4 ) 3 GUMBEL-ATTENTION SENSE INDUCTION ( GASI ) . Building on these foundations , we now introduce our model , GASI , and along the way introduce a soft-attention stepping-stone ( SASI ) ; afterward , we will compare models on both traditional evaluation metrics and interpretability . The critical component of our model is that we model the sense selection probability , which can be interpreted as sense attention over contexts , into the Skip-Gram model while preserving the original objective through marginalization ( Figure 1 ) . By using Gumbel Softmax , our model both approximates discrete sense selection and is differentiable . Previous models are either non-differentiable or otherwise complicate inference through hard attention with reinforcement learning methods ( Lee & Chen , 2017 ) . 3.1 ATTENTIONAL SENSE INDUCTION FRAMEWORK . Embedding Parameters We learn a context embedding matrix C ∈ R|V |×d and a sense embedding tensor S ∈ R|V |×K×d . Unlike previous work ( Neelakantan et al. , 2014 ; Lee & Chen , 2017 ) , no extra embeddings are kept for sense induction . Number of Senses For simplicity and consistency with most previous work , our model has a fixed number of senses K.1 Sense Attention in Objective Function Assuming a center word wi has senses { si1 , si2 , . . . , siK } , the original Skip-Gram likelihood can be written as marginal distribution over all senses of wi with the sense induction probability P ( sik |wi ) , we focus on the sense disambiguation given local context c̃i and estimate P ( cij |wi ) = K∑ k=1 P ( cij | sik ) P ( sik |wi ) ≈ K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) ︸ ︷︷ ︸ attention , ( 5 ) Replacing P ( cij |wi ) in Equation 1 with Equation 5 gives our objective function J ( S , C ) ∝ ∑ wi∈V ∑ cij∈c̃i log K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) . ( 6 ) Lower Bound the Objective for Negative Sampling Like the Skip-Gram objective ( Equation 2 ) , we model the likelihood of a context word given the center sense P ( cij | sik ) using softmax , P ( cij | sik ) = exp ( cij > sik ) ∑|V | j=1 exp ( c > j s i k ) , ( 7 ) where the bold symbol sik is the vector representation of sense s j k from S , and cj is the context embedding of word cj from C. Computing the softmax over the vocabulary is time-consuming . We want to adopt negative sampling to approximate logP ( cij | sik ) , which does not exist explicitly in our objective function ( Equation 6 ) .2 However , given the concavity of the logarithm function , we can apply Jensen ’ s inequality , log K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) ≥ K∑ k=1 P ( sik |wi , c̃i ) logP ( cij | sik ) , ( 8 ) and create a lower bound of the objective . Maximizing this lower bound gives us a tractable objective , J ( S , C ) ∝ ∑ wi∈V ∑ cij∈c̃i K∑ k=1 P ( sik |wi , c̃i ) logP ( cij | sik ) , ( 9 ) where logP ( cij | sik ) is estimated by negative sampling Mikolov et al . ( 2013b ) , log σ ( cij > sik ) + n∑ j=1 Ecj∼Pn ( c ) [ log σ ( −c > j s j k ) ) ] , ( 10 ) 1We can prune the duplicated senses for words that have senses less than K , details in Appendix B . We can also set different number of senses based on word frequency in the training , details in Appendix B.3 . 2Deriving the negative sampling requires the logarithm of a softmax ( Goldberg & Levy , 2014 ) . c̄⊤i s i k gk : Gumbel noise Modeling Sense Attention We can model the attention term , contextual sense induction distribution , with soft attention ; we call the resulting model soft-attention sense induction ( SASI ) ; although it is a stepping stone to our final model , we compare against it in our experiments as it helps isolate the contributions of hard attention . In SASI , the sense attention is conditioned on the entire local context c̃i with softmax : P ( sik |wi , c̃i ) = exp ( c̄ > i s i k ) ∑K k=1 exp ( c̄ > i s i k ) , ( 11 ) where c̄i is the mean of the context vectors in c̃i . 3.2 SCALED GUMBEL SOFTMAX FOR SENSE DISAMBIGUATION . To reduce separate senses and learn distinguishable sense representations , we implement hard attention in our full model , GASI . To preserve differentiability and circumvent the difficulties in training with reinforcement learning ( Sutton & Barto , 1998 ) , we apply the reparameterization trick with Gumbel softmax ( Section 2.1 ) to our sense attention function ( Equation 11 ) and make a continuous relaxation . Vanilla Gumbel Attention The discrete sense sampling from Equation 11 can be refactored by zi = one hot ( arg max k ( c̄i > sik + gk ) ) , ( 12 ) and the hard attention is approximated with yik = softmax ( ( c̄i > sik + gk ) /τ ) . ( 13 ) Scaled Gumbel Softmax for Sense Disambiguation Gumbel softmax learns a flat distribution over senses even with low temperatures ( Figure 2 ) : the dot product c̄ > i s i k is too small compared to the Gumbel noise gk ( Figure 3 ) .3 Thus we use a scaling factor β to reduce the randomness,4 and tune it as a hyperparameter.5 γik = softmax ( ( c̄i > sik + βgk ) /τ ) , ( 14 ) We use GASI-β to identify the GASI model with scaling factor . This modification is critical for learning distinguishable senses ( Figure 2 , Table 1 , and Table 5 ) . Final Objective Function The objective function of our GASI-β model is J ( S , C ) ∝ ∑ wi∈V ∑ wc∈ci K∑ k=1 softmax ( ( c̄i > sjk + βgk ) /τ ) logP ( wc | s i k ) . ( 15 ) 3Float32 precision , the saturation of log ( σ ( · ) ) and gradient vanishing result in a small range of c̄ > i sik . 4Normalizing c̄ > i s i k or directly using logP ( s i k |wi , c̃i ) results in a similar outcome . 5Learning β instead of fixing it as a hyperparameter does not successfully disambiguate senses .

Summary:   This paper extends the skipgram model using one vector per sense of a word. Based on this, the paper proposes two models for training sense embeddings: One where the word senses are marginalized out with attention over the senses, and the second where only the sense with highest value of attention contributes to the loss. For the latter case, the paper uses a variant of Gumbel softmax for training. The paper shows evaluations on benchmark datasets that shows that the Gumbel softmax based method is competitive or better than other methods. Via a crowdsourced evaluation, the paper shows that the method also produces human interpretable clusters.