Learn Interpretable Word Embeddings Efficiently with von Mises-Fisher Distribution

Word embedding plays a key role in various tasks of natural language processing.1 However , the dominant word embedding model don ’ t explain what information2 is carried with the resulting embeddings . To generate interpretable word embed-3 dings we intend to replace the word vector with a probability density distribution.4 The insight here is that if we regularize the mixture distribution of all words to5 be uniform , then we can prove that the inner product between word embeddings6 represent the point-wise mutual information between words . Moreover , our model7 can also handle polysemy . Each word ’ s probability density distribution will gen-8 erate different vectors for its various meanings . We have evaluated our model in9 several word similarity tasks . Results show that our model can outperform the10 dominant models consistently in these tasks.11 1 INTRODUCTION12 Word embedding is a widespread technique in boosting the performance of modern NLP systems13 by learning a vector for each word as its semantic feature . The general idea of word embedding is to14 assign each word with a dense vector having lower dimensionality than the vocabularies ’ cardinal-15 ity . In a qualified word embedding model , the vector-similarity tends to reflect the word-similarity.16 Therefore , feeding these vectors as features of words into the other NLP systems will always boost17 the performance of them in many downstream tasks ( Turian et al. , 2010 ; Socher et al. , 2013 ) .18 One such qualified model is the skip-gram with negative sampling ( SGNS ) model proposed in19 word2vec ( Mikolov et al. , 2013 ; Joulin et al. , 2016 ) , which is very popular in various NLP tasks20 with expressive performance . The SGNS model propose to represent each word with vectors and21 estimate these vectors by applying maximum likelyhood estimation method . It implicitly factorize22 a word-context matrix containing a co-occurrence statistic . This would assign each word wc in the23 vocabulary with a ” word ” vector vc ∈ Rd and a ” context ” vector uc ∈ Rd , so as to model the con-24 ditional probabilities p ( wk|wc ) separately for different words . By maximizing the log likelyhood of25 p ( wk|wc ) , the SGNS model can estimate the ” word ” vector and ” context ” vector of each word.26 However , the SGNS model ’ s main problem is that it doesn ’ t build interpretable model for the em-27 beddings themself , and therefore , people don ’ t understand how word vectors can express useful28 information . For example , previous work emphasized that the inner product between one word ’ s29 ” word ” vector and another word ’ s ” context ” vector represents the point-wise mutual information30 between the two words ( Levy & Goldberg , 2014 ) . However , people never use the ” word-context ” 31 inner product in practise . Instead , people will simply chose ” word ” vectors as the word embeddings32 and drop the other one or vice versa . Therefore , we should care about the behaviour of ” word-word ” 33 or ” context-context ” inner product , which are rarely analyzed and are never guaranteed to have any34 good properties.35 In this paper , we propose a variational inference based framework to learn more interpretable word36 embeddings . That is , we would estimate a probability density distribution for each word instead of37 estimating a vector from the training corpus . To be specific , we propose to replace the ” word ” vector38 with a probability density distribution , namely the von Mises-Fisher ( vMF ) distribution , and keep39 the ” context ” vector for each word . As what we will show in this paper , representing the word with40 a probability density distribution can result in more interpretable word embeddings . Besides , the41 probability density representations can provide other benefits too . For example , such representations42 can model the polysemy phenomenon when we train our model . To be specific , we can sample a43 vector from the vMF distribution to represent the specific meaning of this word in a particular context44 during training.45 The main inspiration for this work is the analysis in Ma ( 2017 ) ’ s work , which can also be found in46 Arora et al . ( 2016 ) ’ s paper . They assumed that the word vectors would obey the uniform distribution47 over a sphere during the analysis . This assumption is critical in their analysis and yet it turns out to48 be wrong as reported by Mimno & Thompson ( 2017 ) . In fact , the word frequency obey the Zipf ’ s49 law , which means that it ’ s impossible for word vectors to obey the uniform distribution when we50 represent each word by a vector . However , when we represent each word by a probability density51 distribution , it ’ s possible that the mixture distribution of all words is uniform if we adjust each52 word ’ s probability density distribution ’ s position and shape carefully.53 To estimate each word ’ s probability density representation , we need to adopt the Bayesian varia-54 tional inference technique , and this would result in a Bayesian version of SGNS model . We are55 not the first to propose a Bayesian version of word embedding model ( Zhang et al. , 2014 ; Sakaya56 et al. , 2015 ; Barkan , 2017 ) . Among them , the state-of-the-art model is the BSG model introduced57 by Bražinskas et al . ( 2017 ) , and it is also the most related work with us . However , our model is dif-58 ferent from the BSG model in many aspects . The BSG model represents each word with a Gaussian59 distribution while we adopt the vMF distribution . It ’ s important to notice that the vMF distribution60 is defined over a unit hyper-sphere , which means that we will sample a unit vector for each word61 during training . Further more , the BSG model is based on the variational autoencoding framework62 ( VAE ) . As reported by Davidson et al . ( 2018 ) , it ’ s impossible for the native VAE to work well when63 the prior is defined over a hyper-sphere . In contrast , we didn ’ t adopt the VAE framework , and yet we64 would still talk about the reparameterization tricks . At last but not least , the BSG model focuses on65 how to build a more reasonable model using the VAE techniques , while we focus on how to generate66 more interpretable word embeddings using the vMF distribution.67 Our main contributions can be summarized as follows . First , we proposed to represent each word68 with a vMF distribution . Second , we generated highly interpretable word embeddings , and show69 that our model ’ s ” context-context ” inner-product would represent the point-wise mutual information70 between words . At last , our word embeddings out-perform the dominant models and the state-of-71 the-art model in various tasks.72 2 OUR FRAMEWORK73 In this section , we intend to generate interpretable word embeddings by adopting the vMF represen-74 tation for each word . Specifically , we will show that the ” context-context ” inner-product between75 word embeddings would represent the point-wise mutual information between words.76 2.1 MODEL DEFINITION77 The SGNS model intends to maximize the probability of a context word appearing around a center78 word . It assumes this probability is proportional to the inner product between two fixed vectors . In79 contrast , we assume that the probability of a context word appearing around a center word should80 be the average probability when the center word take different meanings . That is , we are suggesting81 a generative model as pictured in 1 . Assuming there are T words wc in a training corpus , then for82 each word we would like to predict the possible words wk appearing around it within a K length83 word window . However , it ’ s hard to determine what exactly the current word wc means , therefore,84 we propose to take the average probability over all possible v̂c for each observed wk.85 The training objective of our model is to find vector representations and probability density repre-86 sentations for words that are useful for predicting the surrounding words in a sentence . Given a87 sequence of training words w1 , · · · , wT , we are meant to maximize the average log likelyhood88 argmax θ 1 T T∑ t=1 K/2∑ j=−K/2 log ∫ p ( wt+j |v̂t , wt ; θ ) p ( v̂t|wt ; θ ) dv̂ + L ( θ ) , j 6= 0 , ( 1 ) where θ is the set of all parameters to be optimized . What ’ s more , v̂t ∈ Rd denotes a vector89 representation for a patential meaning of wt , and p ( v̂t|wt ; θ ) is the probability density representaton90 for wt . We sample v̂t according to p ( v̂t|wt ; θ ) . At last , p ( wt+j |v̂t , wt ; θ ) denotes the probability of91 word wt+j appearing given v̂t , and L ( θ ) denotes the possible regularization term.92 Unfortunately , it ’ s intractable to calculate the integration in ( 1 ) . Therefore , we propose to git rid of93 the integration by applying the jensen inequality considering that log is concate and we are doing94 maximization . By doing so , our model can be defined as95 argmax θ 1 T T∑ t=1 n∑ j=−n E v̂t∼p ( v̂mt |wt ) log p ( wt+j |v̂mt , wt ; θ ) + L ( θ ) , j 6= 0 . ( 2 ) More details can be found in the appendix . We will call the p ( v̂mt |wt ) as a prior for each word96 wt . Although the equation ( 2 ) is very similar to the ELBo in the standard varitional inference97 auto-encoder ( VAE ) technique , the actual meaning of it is quite different from ELBo . First , the ex-98 pectation term in ( 2 ) doesn ’ t involve a ” encoder ” as what VAE would do . Second , the regularization99 term in ( 2 ) is also different from the VAE ’ s KL-divergence term , as what we will show.100 2.2 EXPECTATION TERM101 The most troublesome component of equation ( 2 ) is the epctation term . For each v̂t sampled from102 word wt ’ s prior , we choose the softmax function to calculate the odds of word wt+j appearing103 around it . That is , we can decompose the expectation term into104 E v̂t∼p ( v̂mt |wt ) log p ( wt+j |v̂mt , wt ; θ ) = E v̂t∼p ( v̂mt |wt ) [ log exp ( u > t+jv̂t ) ∑|V | i=1 exp ( u > i v̂t ) ] , ( 3 ) where ui ∈ Rd denotes the ” context ” vector for word wi , and |V | is the cardinality of our vocabu-105 lary . As what ’ s been suggested in the SGNS model , we extend the negative sampling technique to106 our model to avoid the computation of the softmax function ’ s denominator . It ’ s worth to notice that107 different words would have different denominators in theory . We will use Zt to denote the denom-108 inator of word wt . According to the theory proved by Gutmann & Hyvärinen ( 2012 ) , maximizing109 objective ( 3 ) is equivalent to minimizing110 E v̂t∼p ( v̂mt |wt ) [ log σ ( u > t+jv̂t ) +NEi∼p ( wi ) log σ ( −u > i v̂t ) ] , ( 4 ) where ui is the context vector of word wi , and wi is the negative word sampled according to the111 empirical unigram probability p ( wi ) . There are N negative samples.112 At last , we also need to sample v̂t , and we choose von Mises-Fisher ( vMF ) distribution to calculate113 the probability of wt taking this particular meaning . The vMF distribution is an analogy of Gaussian114 distribution over the unit sphere . It ’ s parameterized by a mean vector and a concentration parameter115 κ ≥ 0 . For word wt , it ’ s corresponding mean vector vt can be interpreted as the vector repre-116 sentation for its average meaning . Formally , the probability of wt taking one particular meaning117 is118 p ( v̂t|wt ) = cd ( κt ) exp ( κtv > t v̂t ) , cd ( κt ) = κ d/2−1 t ( 2π ) d/2Id/2−1 ( κt ) , ( 5 ) where d is the dimension of embedding space and Id/2−1 ( · ) denotes the modified Bessel function119 of the first kind Sra ( 2016 ) .120 2.3 REGULARIZATION TERM121 We will introduce an regularization term in this subsection , and show that this term will link the122 ” context-context ” inner-product directly to the point-wise mutual information . The general idea is123 to regularize the mixture distribution of all words to be uniform , then we can achieve our goal . This124 can be done by simply applying the maximum entropy algorithm which would result in a uniform125 distribution naturally . Therefore , the regularization term we are seeking for is126 argmax θ p ( v̂ ) log p ( v̂ ) , where p ( v̂ ) = ∑|V | i=1 p ( wi ) p ( v̂|wi ) . ( 6 ) Objective ( 4 ) + ( 6 ) is the final objective that we want to optimize . We claim that by doing so , for each pair of words wc and wk , we have u > k uc d ≈ PMI ( wk , wc ) : = log ( p ( wk , wc ) p ( wk ) p ( wc ) ) , where PMI ( wk , wc ) is called the point-wise mutual information ( PMI ) between word wk and word127 wc , and p ( wk , wc ) denotes the probability for them to appear in the same word window.128 Before we show why our claim holds , we would like to emphasize why PMI is important . PMI indi-129 cates how much more possible that word wk , wc co-occur than by chance ( Church & Hanks , 1990 ) .130 Then , people developed the notion of word window to help define when two words ” co-occur ” , i.e.,131 wk and wc co-occur , only when they appear in the same word window . Most of the word embedding132 model will take advantage of such co-occurrence statistics . Indeed , the PMI evaluated from co-133 occurrence counts has a strong linear relationship with human semantic similarity judgments from134 survey data ( Hashimoto et al. , 2016 ) . In conclusion , it ’ s reasonable to relate word embedding with135 the point-wise mutual information.136 Then , we will show how to link the ” context-context ” inner-product directly to the PMI by two steps.137 The first step is to show that138 log p ( wk , wc ) ≈ ‖uk+uc‖ 2 2 2d − 2 log ( Z ) , log p ( wk ) ≈ ‖uk‖ 2 2 2d − log ( Z ) , log p ( wc ) ≈ ‖uc‖ 2 2 2d − log ( Z ) , ( 7 ) where Z is the constant that most Zc approximate to . This is a conclusion proved by Ma ( 2017 ) . The second step is quite obvious because PMI ( wk , wc ) = log p ( wk , wc ) − log p ( wk ) − log p ( wc ) ≈ u > k uc d . We will show ( 7 ) by a series equations briefly , and reveal why the regularization term is very impor-139 tant . More details can be found in the appendix . We start with p ( wk , wc ) 140 p ( wk , wc ) = |V |∑ i=1 p ( wi ) p ( wk , wc|wi ) = |V |∑ i=1 p ( wi ) p ( wk|wi ) p ( wc|wi ) = E i∼p ( wi ) p ( wk|wi ) p ( wc|wi ) = E i∼p ( wi ) ∫ eu > k v̂ ′ i Zi p ( v̂′i|wi ) ds′ ∫ eu > c v̂i Zi p ( v̂i|wi ) ds ≈ 1Z2 E i∼p ( wi ) ∫ ∫ eu > k v̂ ′ ieu > c v̂ip ( v̂′i|wi ) p ( v̂i|wi ) dsds′ ≈ 1Z2 E i∼p ( wi ) ∫ exp [ ( uk + uc ) > v̂i ] p ( v̂i|wi ) ds = 1Z2 |V |∑ i=1 p ( wi ) ∗ { ∫ exp [ ( uk + uc ) > v̂i ] p ( v̂i|wi ) ds } = 1Z2 ∫ exp [ ( uk + uc ) > v̂ ] [ ∑|V | i=1 p ( wi ) p ( v̂|wi ) ] ds = 1Z2 ∫ exp [ ( uk + uc ) > v̂ ] p ( v̂ ) ds = 1Z2 E v̂∼p ( v̂ ) { exp [ ( uk + uc ) > v̂ ] } = 1Z2Ex∼N ( 0 , ‖uk+uc‖22/d ) { exp ( x ) } ≈ 1Z2 exp ( ‖uk+uc‖22 2d ) . ( 8 ) Step ten is why we need a regularization term , but before that , we would like to explain all the141 equations above . The first step of ( 8 ) says that wk and wc co-occur iff they appear in another word142 wi ’ s word window together . This is true for the Skip-gram model . The second step is also true143 when we considering the definition of p ( wk|wi ) and p ( wc|wi ) in the Skip-gram model . Step four144 is just by definition and we use slightly different notations here to indicate that v̂i , v̂′i are different145 variable . Step six is a strong claim which needs rigorous prove . We put this prove in the appendix.146 The key insight is that v̂i , v̂′i obey the same vMF distribution , which means that when this vMF is147 concentrate enough , then the probability of v̂i , v̂′i being very different is small . In the eighth step , we148 omit i to emphasize that every word can generate the same vector v̂ and that ’ s why the summation149 and integration can exchange with each other in this way.150 If we regularize the v̂ to be uniformly distributed over the unit sphere in step ten , then ( uk +uc ) > v will obey Gaussian distribution approximately ( Ma , 2017 ) . This means that step eleven holds . More details can be found in the appendix . The last step is the result of a famous calculation practise Ex∼N ( 0 , σ2 ) { exp ( x ) } = exp ( σ2/2 ) . Obviously , by replacing the ” word ” vector with the vMF distribution , we can eliminate the assump-151 tion that p ( wi ) being uniform.152 2.4 REPARAMETERIZATION TRICK153 There is one problem to solve before our model becomes practical . p ( v̂t|wt ) is difficult to optimize154 because the operation of sampling is nondifferentiable . We can solve this problem by applying the155 reparameterization trick . The vMF distribution ’ s reparameterization trick is usually discussed in156 the context of hyperspherical variational auto-encoders ( Davidson et al. , 2018 ; Xu & Durrett , 2018 ; 157 Guu et al. , 2018 ) . To simplify our model , we propose to fix the concentration parameter κt as a158 constant during training for each word wt . This is because the gradient estimation of κ is complex159 and computational expensive.160 When it comes to each word wt ’ s mean vector vt , we follow the technique used in Xu & Durrett ’ s work . Firstly , we sample an auxiliary random variable ω according to the rejection sampling scheme of Wood ( 1994 ) . The distribution of ω is controlled by κ . Specifically , the probability of ω being sampled is p ( ω ; κ ) ∝ exp ( ωκ ) ( 1− ω2 ) . Then we draw our v̂t in the following way v̂t = ωvt + z √ 1− ω2 , where z is a random unit vector and is tangent to the unit sphere Sd−1 at vt . Figure 3 illustrates the161 geometric vision . Because of applying this trick , we can take gradient with respect to vt as usual.162 2.5 FINAL ALGORITHM163 Putting all the details together , we have the following objective to minimize log σ ( u > k v̂ ) +NEi∼p ( wi ) [ log σ ( −u > i v̂ ) ] ︸ ︷︷ ︸ L1 − p ( v̂ ) log p ( v̂ ) ︸ ︷︷ ︸ L2 , 164 where v̂ = ωvc + z √ 1− ω2 , ‖vc‖2 = 1 ω ∼ p ( ω ; κc ) ∝ exp ( ωκc ) ( 1− ω2 ) , ‖z‖2 = 1 , z is tangent to Sd−1 at vc , p ( v̂ ) = ∑|V | j=1 p ( wj ) p ( v̂|wj ) p ( v̂|wj ) = cd ( κj ) exp ( κjv > j v̂ ) , cd ( κj ) = κ d/2−1 j ( 2π ) d/2Id/2−1 ( κj ) . ( 9 ) Based on our final model , we propose algorithm ( 1 ) to train the word embeddings . The for loop165 started from line 1 controls how many times we will go through the entire training corpus . Then , for166 each epoch , we would iterate over each word in the corpus as what the for loop in line 2 suggests . In167 line 3 , we will take wc ’ s mean vector vc and concentrate parameter κc from dictionaries . The third168 layer of for loop will iterate over all the words in a word window centered around wc . For each word169 wk in this word window , we will take its context vector uk from the dictionary , and sample a v̂ from170 wc ’ s prior with the help of our reparameterization trick ( line 5 to line 6 ) . Then form line 7 to line171 10 , we will sample N negative samples for each wk , and then calculate the gradients of L1 . From172 line 11 to line 13 , we will calculate L2 based on the v̂ , and update all word ’ s prior accordingly.173 noend 1 PDF ( N , ui , vi , κi , p ( wi ) , i = 1 , · · · , |V | ) 1 : for epoch in epochs do 2 : for wc in corpus do 3 : vc ← { vi } i=1 , ··· , |V | , κc ← { κi } i=1 , ··· , |V | 4 : for wk in Window ( wc ) do 5 : ω ∼ p ( ω ; κc ) , z ∼ U ( Sd−2 ) 6 : v̂← ωvc + z √ 1− ω2 , uk ← { ui } i=1 , ··· , |V | 7 : for i in range ( N ) do 8 : wi ∼ p ( wi ) , ui ← { ui } i=1 , ··· , |V | 9 : L1 ← L1 ( uk , ui , v̂ ) 10 : Update vc , uk according to L1 ’ s gradient . 11 : L2 ← L2 ( { vj } j=1 , ··· , |V | , v̂ ) 12 : for j in range ( |V | ) do 13 : Update vj according to L2 ’ s gradient . 3 EXPERIMENTS174 We will now experimentally validate our embedding by comparing the performance of our model175 with the dominant word embedding models on the word similarity tasks . Subsection 3.1 presents all176 the experiment settings for different training corpus and different word embedding models . Subsec-177 tion 3.2 introduces several word similarity benchmarks , and how we evaluate model ’ s performance178 on them . Subsection 3.3 compares our model with the STOA model on several benchmarks.179 3.1 EXPERIMET SETTINGS180 We use part of massachusetts bay transportation authority ( mbta ) web crawled corpus because it181 is well cleaned and is large enough . The mbta corpus contains about 600 million tokens . We182 preprocessed all corpora by removing non-English characters , numbers and lower-casing all the183 text . The vocabulary was restricted to the 100K most frequent words in each corpus.184 We trained embeddings using three methods : word2vec Mikolov et al . ( 2013 ) , GloVe Pennington185 et al . ( 2014 ) , and our model . This is because these models are implemented with C/C++ , and186 therefore are fast enough to be evaluated on mbta corpus . For fairness we fix all hyperparameters187 for word2vec , GloVe and our model . Specifically , we trained for 5 epochs for each model using 75188 threads for parallel computation ; the word embedding dimension is 100 ; the window size is 5.189 For the word2vec , and our model , the negative sampling number is 5 ; We adopt the Hogwild ! algo-190 rithm to train models , and the learning rate decays linearly from 0.0025 to 0 . The noise distribution191 is set as the same as used in Mikolov et al. , pn ( w ) ∝ p ( ww ) 0.75 . We also use a rejection threshold192 of 10−4 to subsample the most frequent words.193 For the GloVe model , we follow the original inplementation ’ s default settings for the other hyperpa-194 rameters . This means the initial learning rate is 0.05.195 We also use the news corpus1 with about 15 million tokens to evaluate the BSG model . This is196 because the original implementation of BSG is based on theano , and it ’ s slow to be trained on the197 mbta corpus . Given the small volume of this corpus , we trained for 25 epochs for each model using198 12 threads . we also restrict each model ’ s vocabulary to be 10K . For both the BSG , word2vec and199 our model , we set the negative number to be 10 ; the window width is 10.200 3.2 PERFORMANCE EVALUATION201 We test the quality of the word embeddings by checking if our word embeddings agree with the202 human judgement on word similarity / relatedness.203 For the mbta corpus , we test the performance of our embeddings on three benchmarks : the Word204 Similarity353 data set Finkelstein et al . ( 2002 ) , the RG65 data set Luong et al . ( 2013 ) , and the MEN205 data set Bruni et al . ( 2014 ) . Taking the WS353 data set for example , it contains 353 word pairs along206 with their similarity scores assigned by 29 subjects . These subjects possessed near-native command207 of English and they are instructed to estimate the relatedness of the words in pairs on a scale from 0208 to 10 . During experiments , we will cumpute the Spearman ’ s rank correlation coefficient Spearman209 ( 1904 ) between human judgement and the inner product between the vector representations . The210 larger this coefficient is , the better this embedding is and the mximume value is 1 . Some words in211 1 https : //drive.google.com/file/d/1QWC2x6qq8KyHFUCgyvVJJoGHexZrw7gO/view these testing datasets do not appear in our training corpora , and this means we can ’ t calculate the212 inner product between vectors for those words . In order to provide comparable results , we propose213 to use the mean vectors of the rest words for these missing words . Also , the mean vector of each214 word ’ s prior is similar to the ” word ” vector in the SGNS model , and we also test the performance of215 this vector too.216 We also evaluate the performance over more Benchmarks . Table 2 presents similarity results com-217 puted using the online tool of Faruqui & Dyer ( 2014 ) . Since the BSG model can only generate the218 mean vectors from its prior , we only evaluate the ” word ” or mean vectors in these experiments.219 3.3 COMPARE WITH THE DOMINANT MODELS220 Table 1 shows the results on these benchmarks . As we can see , our ” context ” word embedding can221 constantly outperform the counterpart of the dominant word embedding models by a large margin.222 This is exactly what we expect according to our theory . It ’ s interesting that the ” word ” vector of our223 model can still outperform ours ” context ” vector . Another exception is that on the RG65 test set , the224 SGNS model can outperform our model by a large margin . We argue this may be caused by its bias225 – it only contains 65 noun pairs after all . Besides , our model ’ s ” context ” vector can still outperform226 SGNS on this test set.227 Also , the gap between the ” context ” vectors and the ” word ” vectors in our model is much smaller228 than the dominant models ’ gaps . These results demonstrate that our model is more specific about229 which part of our embedding contains the useful information.230 3.4 COMPARE WITH THE BSG MODEL231 First , we observe that our model can out-perform the BSG model for almost every task except for the232 RG65 data set as before . Although the BSG model can perform better than our model in the WS353-233 Rel data set , we would like to point out that this data set is a subset of WS353-ALL . Therefore , this234 may also be caused by the bias of small data set . Second , the performance of BSG model is weaker235 than the SGNS model for some data sets . Given that we used their released implementation directly,236 this may be the result of small training data set , i.e. , the SGNS model may perform better in the case237 of small training data set because of its simplicity.238 4 CONCLUSIONS239 We generated interpretable word embeddings by represent each word with a von Mises-Fisher dis-240 tribution . We have demonstrated that our word embeddings can be linked to the point-wise mutual241 information directly without making any unrealistic assumptions . The experiments over different242 training and testing data sets demonstrate that our model can outperform both the dominant and the243 STOA models . We argue that our insight into the interpretable word embeddings is important . For244 example , as we are sure that the unit word vectors can encode the semantic similarity between words,245 it ’ s possible to encode the syntactic information between words into the norm of word vectors.246 REFERENCES247 Sanjeev Arora , Yuanzhi Li , Yingyu Liang , Tengyu Ma , and Andrej Risteski . A latent variable model248 approach to pmi-based word embeddings . Transactions of the Association for Computational249 Linguistics , 4:385–399 , 2016.250 Oren Barkan . Bayesian neural word embedding . In Thirty-First AAAI Conference on Artificial251 Intelligence , 2017.252 Arthur Bražinskas , Serhii Havrylov , and Ivan Titov . Embedding words as distributions with a253 bayesian skip-gram model . arXiv preprint arXiv:1711.11027 , 2017.254 Elia Bruni , Nam-Khanh Tran , and Marco Baroni . Multimodal distributional semantics . Journal of255 Artificial Intelligence Research , 49:1–47 , 2014.256 Kenneth Ward Church and Patrick Hanks . Word association norms , mutual information , and lexi-257 cography . Computational linguistics , 16 ( 1 ) :22–29 , 1990.258 Tim R Davidson , Luca Falorsi , Nicola De Cao , Thomas Kipf , and Jakub M Tomczak . Hyperspheri-259 cal variational auto-encoders . arXiv preprint arXiv:1804.00891 , 2018.260 Manaal Faruqui and Chris Dyer . Community evaluation and exchange of word vectors at wordvec-261 tors . org . In Proceedings of 52nd Annual Meeting of the Association for Computational Linguis-262 tics : System Demonstrations , pp . 19–24 , 2014.263 Lev Finkelstein , Evgeniy Gabrilovich , Yossi Matias , Ehud Rivlin , Zach Solan , Gadi Wolfman , and264 Eytan Ruppin . Placing search in context : The concept revisited . ACM Transactions on informa-265 tion systems , 20 ( 1 ) :116–131 , 2002.266 Michael U Gutmann and Aapo Hyvärinen . Noise-contrastive estimation of unnormalized statistical267 models , with applications to natural image statistics . Journal of Machine Learning Research , 13268 ( Feb ) :307–361 , 2012.269 Kelvin Guu , Tatsunori B Hashimoto , Yonatan Oren , and Percy Liang . Generating sentences by270 editing prototypes . Transactions of the Association for Computational Linguistics , 6:437–450,271 2018.272 Tatsunori B Hashimoto , David Alvarez-Melis , and Tommi S Jaakkola . Word embeddings as metric273 recovery in semantic spaces . Transactions of the Association for Computational Linguistics , 4:274 273–286 , 2016.275 Armand Joulin , Edouard Grave , Piotr Bojanowski , and Tomas Mikolov . Bag of tricks for efficient276 text classification . arXiv preprint arXiv:1607.01759 , 2016.277 Omer Levy and Yoav Goldberg . Neural word embedding as implicit matrix factorization . In Ad-278 vances in neural information processing systems , pp . 2177–2185 , 2014.279 Thang Luong , Richard Socher , and Christopher Manning . Better word representations with recursive280 neural networks for morphology . In Proceedings of the Seventeenth Conference on Computational281 Natural Language Learning , pp . 104–113 , 2013.282 Tengyu Ma . Non-convex Optimization for Machine Learning : Design , Analysis , and Understanding.283 PhD thesis , Princeton University , 2017.284 Tomas Mikolov , Ilya Sutskever , Kai Chen , Greg S Corrado , and Jeff Dean . Distributed represen-285 tations of words and phrases and their compositionality . In Advances in neural information pro-286 cessing systems , pp . 3111–3119 , 2013.287 David Mimno and Laure Thompson . The strange geometry of skip-gram with negative sampling . In288 Empirical Methods in Natural Language Processing , 2017.289 Jeffrey Pennington , Richard Socher , and Christopher Manning . Glove : Global vectors for word290 representation . In Proceedings of the 2014 conference on empirical methods in natural language291 processing ( EMNLP ) , pp . 1532–1543 , 2014.292 Joseph Hosanna Sakaya et al . Scalable bayesian induction of word embeddings . 2015.293 Richard Socher , John Bauer , Christopher D Manning , et al . Parsing with compositional vector294 grammars . In Proceedings of the 51st Annual Meeting of the Association for Computational295 Linguistics ( Volume 1 : Long Papers ) , pp . 455–465 , 2013.296 Charles Spearman . The proof and measurement of association between two things . American297 journal of Psychology , 15 ( 1 ) :72–101 , 1904.298 Suvrit Sra . Directional statistics in machine learning : a brief review . 2016.299 Joseph Turian , Lev Ratinov , and Yoshua Bengio . Word representations : a simple and general method300 for semi-supervised learning . In Proceedings of the 48th annual meeting of the association for301 computational linguistics , pp . 384–394 . Association for Computational Linguistics , 2010.302 Andrew TA Wood . Simulation of the von mises fisher distribution . Communications in statistics-303 simulation and computation , 23 ( 1 ) :157–164 , 1994.304 Jiacheng Xu and Greg Durrett . Spherical latent spaces for stable variational autoencoders . arXiv305 preprint arXiv:1808.10805 , 2018.306 Jingwei Zhang , Jeremy Salwen , Michael Glass , and Alfio Gliozzo . Word semantic representations307 using bayesian probabilistic tensor factorization . In Proceedings of the 2014 Conference on Em-308 pirical Methods in Natural Language Processing ( EMNLP ) , pp . 1522–1531 , 2014.309

Summary: The paper addresses the problem the problem in word embeddings where "word-word" or "context-context" inner products are relied upon in practice when the embeddings are usually only optimized for good properties of "word-context" inner products. The new approach to training word embeddings addresses interpretability in the sense of having a theoretically grounded interpretation for the good properties that the inner products should possess -- namely that they capture the PMI between words. By changing the embedding training procedure (to include a step that samples vectors from each word's distribution representation) this interpretation is made possible without problematic theoretical assumptions. Now trained for the purpose they are intended to be used for, the approach yields strong SOTA results on the non-small challenge datasets.