A Differentiable Self-disambiguated Sense Embedding Model via Scaled Gumbel Softmax

We present a differentiable multi-prototype word representation model that disentangles senses of polysemous words and produces meaningful sense-specific embeddings without external resources . It jointly learns how to disambiguate senses given local context and how to represent senses using hard attention . Unlike previous multi-prototype models , our model approximates differentiable discrete sense selection via a modified Gumbel softmax . We also propose a novel human evaluation task that quantitatively measures ( 1 ) how meaningful the learned sense groups are to humans and ( 2 ) how well the model is able to disambiguate senses given a context sentence—an evaluation ignored by previous models . Our model not only discovers distinct , interpretable embeddings but is competitive against previous models on word similarity tasks . 1 SENSE-SPECIFIC EMBEDDING . Machine learning models for natural language processing applications often represent words with real valued vector embeddings . Popular word embedding models such as Word2Vec ( Mikolov et al. , 2013a ; b ) and GloVe ( Pennington et al. , 2014 ) enabled state-of-the-art results on myriad NLP tasks such as sentiment analysis ( Kim , 2014 ; Tai et al. , 2015 ) and textual entailment ( Chen et al. , 2017 ) . However , for polysemous words ( those with multiple senses ) , learning a single vector for each word type conflates different meanings ( e.g. , “ A hydrogen bond exists between water molecules. ” vs. “ Do you want to buy this bond ? ” ) . This is not a new problem—Schütze ( 1998 ) demonstrates the deficiency of assigning just one vector per word—but it is more pernicious in modern models , as conflated senses can pull semantically unrelated words toward each other in the embedding space ( Neelakantan et al. , 2014 ; Pilehvar & Collier , 2016 ; Camacho-Collados & Pilehvar , 2018 ) . To disentangle distinct senses in word embeddings and learn finer-grained semantic clusters , multi-prototype word embedding models learn multiple sense-specific embeddings for a single word ( Section 7 ) . But what makes a good multisense word embedding ? While word similarity is the most common evaluation , it has many detractors ( Faruqui et al. , 2016 ; Gladkova & Drozd , 2016 ) : similarity is subjective and is hard to be differentiate from word relatedness . Moreover , word similarity tasks— with the exception of Stanford Contextual Word Similarity ( Huang et al. , 2012 , SCWS ) —ignore polysemous cases or are tied to specific sense inventories ( Boyd-Graber et al. , 2006 ) . Moreover , these evaluations ignore a key component of learning sense inventories : do they make sense to a human ? Previous multisense embedding papers present nearest neighbors to claim their representations are interpretable and useful . Like topic models , these implicit interpretability claims need to be rigorously verified . In Section 6 , we adapt techniqes for evaluating topic models ( Chang et al. , 2009 ) to measure whether learned sense groups are internally coherent and whether humans can consistently match a learned sense vector to a word in context . Just like topic models , word embedding models that win conventional evaluations do not always make sense to humans . We present a simple method that not only correlates well with traditional word similarity evaluations ( Section 5 ) but also discovers interpretable ( measured by human evaluations ) sense embeddings ( Section 6 ) . Our model extends the Skip-Gram Word2Vec model and simultaneously learns ( 1 ) automatic sense induction given local context and ( 2 ) sense-specific embeddings . To learn disentangled sense representations ( i.e. , avoid sense mixing ) , we approximate hard attention and preserve You only live twice , Mr . Bond . . . c̄i ci1 c i m ... G M predict contexts G M Gumbel softmax Marginalization context words center word context embeddings lookup sense embeddings lookup sense attention C S ... wic̃i P ( sik|wi , c̃i ) chemical 007 financial si1 si2 siK P ( cij |wi ) Figure 1 : Network struture with an example of our GASI model which learns a set of global context embeddings C and a set of sense embeddings S differentiability via a scaled variant of the Gumbel Softmax function ( Section 3.2 ) . This modeling contribution—Scaled Gumbel Softmax—is critical for disambiguating senses . 2 FOUNDATIONS : SKIP-GRAM AND GUMBEL SOFTMAX . Our model extends Skip-Gram Word2Vec ( Mikolov et al. , 2013a ; b ) , which jointly learns word embeddings W ∈ R|V |×d and context embeddings C ∈ R|V |×d . More specifically , given a vocabulary V and embedding dimension d , it maximizes the likelihood of the context words cij that surround a given center word wi in a context window c̃i , J ( W , C ) ∝ ∑ wi∈V ∑ cij∈c̃i logP ( cij |wi ; W , C ) , ( 1 ) where P ( cij |wi ) is estimated by a softmax over all possible context words , i.e , the vocabulary , P ( cij |wi ; W , C ) = exp ( cij > wi ) ∑ c∈V exp ( c > wi ) . ( 2 ) In practice , logP ( cij |wi ) is approximated by negative sampling to reduce computational cost . 2.1 GUMBEL SOFTMAX . The Gumbel softmax ( Jang et al. , 2016 ; Maddison et al. , 2016 ) approximates the sampling of discrete random variables . Given a discrete random variable X with P ( X = k ) ∝ αk , αk ∈ ( 0 , ∞ ) , the Gumbel-max ( Gumbel & Lieblein , 1954 ; Maddison et al. , 2014 ) refactors the sampling of X into X = arg max k ( logαk + gk ) , ( 3 ) where the Gumbel noise gk = − log ( − log ( uk ) ) and uk are i.i.d samples drawn from Uniform ( 0 , 1 ) . The Gumbel softmax approximates sampling one hot ( arg maxk ( logαk + gk ) ) by yk = softmax ( ( logαk + gk ) /τ ) . ( 4 ) 3 GUMBEL-ATTENTION SENSE INDUCTION ( GASI ) . Building on these foundations , we now introduce our model , GASI , and along the way introduce a soft-attention stepping-stone ( SASI ) ; afterward , we will compare models on both traditional evaluation metrics and interpretability . The critical component of our model is that we model the sense selection probability , which can be interpreted as sense attention over contexts , into the Skip-Gram model while preserving the original objective through marginalization ( Figure 1 ) . By using Gumbel Softmax , our model both approximates discrete sense selection and is differentiable . Previous models are either non-differentiable or otherwise complicate inference through hard attention with reinforcement learning methods ( Lee & Chen , 2017 ) . 3.1 ATTENTIONAL SENSE INDUCTION FRAMEWORK . Embedding Parameters We learn a context embedding matrix C ∈ R|V |×d and a sense embedding tensor S ∈ R|V |×K×d . Unlike previous work ( Neelakantan et al. , 2014 ; Lee & Chen , 2017 ) , no extra embeddings are kept for sense induction . Number of Senses For simplicity and consistency with most previous work , our model has a fixed number of senses K.1 Sense Attention in Objective Function Assuming a center word wi has senses { si1 , si2 , . . . , siK } , the original Skip-Gram likelihood can be written as marginal distribution over all senses of wi with the sense induction probability P ( sik |wi ) , we focus on the sense disambiguation given local context c̃i and estimate P ( cij |wi ) = K∑ k=1 P ( cij | sik ) P ( sik |wi ) ≈ K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) ︸ ︷︷ ︸ attention , ( 5 ) Replacing P ( cij |wi ) in Equation 1 with Equation 5 gives our objective function J ( S , C ) ∝ ∑ wi∈V ∑ cij∈c̃i log K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) . ( 6 ) Lower Bound the Objective for Negative Sampling Like the Skip-Gram objective ( Equation 2 ) , we model the likelihood of a context word given the center sense P ( cij | sik ) using softmax , P ( cij | sik ) = exp ( cij > sik ) ∑|V | j=1 exp ( c > j s i k ) , ( 7 ) where the bold symbol sik is the vector representation of sense s j k from S , and cj is the context embedding of word cj from C. Computing the softmax over the vocabulary is time-consuming . We want to adopt negative sampling to approximate logP ( cij | sik ) , which does not exist explicitly in our objective function ( Equation 6 ) .2 However , given the concavity of the logarithm function , we can apply Jensen ’ s inequality , log K∑ k=1 P ( cij | sik ) P ( sik |wi , c̃i ) ≥ K∑ k=1 P ( sik |wi , c̃i ) logP ( cij | sik ) , ( 8 ) and create a lower bound of the objective . Maximizing this lower bound gives us a tractable objective , J ( S , C ) ∝ ∑ wi∈V ∑ cij∈c̃i K∑ k=1 P ( sik |wi , c̃i ) logP ( cij | sik ) , ( 9 ) where logP ( cij | sik ) is estimated by negative sampling Mikolov et al . ( 2013b ) , log σ ( cij > sik ) + n∑ j=1 Ecj∼Pn ( c ) [ log σ ( −c > j s j k ) ) ] , ( 10 ) 1We can prune the duplicated senses for words that have senses less than K , details in Appendix B . We can also set different number of senses based on word frequency in the training , details in Appendix B.3 . 2Deriving the negative sampling requires the logarithm of a softmax ( Goldberg & Levy , 2014 ) . c̄⊤i s i k gk : Gumbel noise Modeling Sense Attention We can model the attention term , contextual sense induction distribution , with soft attention ; we call the resulting model soft-attention sense induction ( SASI ) ; although it is a stepping stone to our final model , we compare against it in our experiments as it helps isolate the contributions of hard attention . In SASI , the sense attention is conditioned on the entire local context c̃i with softmax : P ( sik |wi , c̃i ) = exp ( c̄ > i s i k ) ∑K k=1 exp ( c̄ > i s i k ) , ( 11 ) where c̄i is the mean of the context vectors in c̃i . 3.2 SCALED GUMBEL SOFTMAX FOR SENSE DISAMBIGUATION . To reduce separate senses and learn distinguishable sense representations , we implement hard attention in our full model , GASI . To preserve differentiability and circumvent the difficulties in training with reinforcement learning ( Sutton & Barto , 1998 ) , we apply the reparameterization trick with Gumbel softmax ( Section 2.1 ) to our sense attention function ( Equation 11 ) and make a continuous relaxation . Vanilla Gumbel Attention The discrete sense sampling from Equation 11 can be refactored by zi = one hot ( arg max k ( c̄i > sik + gk ) ) , ( 12 ) and the hard attention is approximated with yik = softmax ( ( c̄i > sik + gk ) /τ ) . ( 13 ) Scaled Gumbel Softmax for Sense Disambiguation Gumbel softmax learns a flat distribution over senses even with low temperatures ( Figure 2 ) : the dot product c̄ > i s i k is too small compared to the Gumbel noise gk ( Figure 3 ) .3 Thus we use a scaling factor β to reduce the randomness,4 and tune it as a hyperparameter.5 γik = softmax ( ( c̄i > sik + βgk ) /τ ) , ( 14 ) We use GASI-β to identify the GASI model with scaling factor . This modification is critical for learning distinguishable senses ( Figure 2 , Table 1 , and Table 5 ) . Final Objective Function The objective function of our GASI-β model is J ( S , C ) ∝ ∑ wi∈V ∑ wc∈ci K∑ k=1 softmax ( ( c̄i > sjk + βgk ) /τ ) logP ( wc | s i k ) . ( 15 ) 3Float32 precision , the saturation of log ( σ ( · ) ) and gradient vanishing result in a small range of c̄ > i sik . 4Normalizing c̄ > i s i k or directly using logP ( s i k |wi , c̃i ) results in a similar outcome . 5Learning β instead of fixing it as a hyperparameter does not successfully disambiguate senses .

Summary: The paper presents a method for deriving multi sense word embeddings. The key idea behind this method is to learn a sense embedding tensor using a skip-gram style training objective. The objective defines the probability of contexts marginalised over latent sense embeddings. The paper uses Gumbel-softmax reparametrization trick to approximate sampling from the discrete sense distributions. The method also uses a separate hyperparameter to help scale the dot product appropriately.