Title: LEARN INTERPRETABLE WORD EMBEDDINGS EFFI-CIENTLY WITH VON MISES-FISHER DISTRIBUTION

Abstract: Word embedding plays a key role in various tasks of natural language processing. However, the dominant word embedding model don't explain what information is carried with the resulting embeddings. To generate interpretable word embeddings we intend to replace the word vector with a probability density distribution.
The insight here is that if we regularize the mixture distribution of all words to be uniform, then we can prove that the inner product between word embeddings represent the point-wise mutual information between words. Moreover, our model can also handle polysemy. Each word's probability density distribution will generate different vectors for its various meanings. We have evaluated our model in several word similarity tasks. Results show that our model can outperform the dominant models consistently in these tasks.

1. INTRODUCTION:

Word embedding is a widespread technique in boosting the performance of modern NLP systems by learning a vector for each word as its semantic feature. The general idea of word embedding is to assign each word with a dense vector having lower dimensionality than the vocabularies' cardinality. In a qualified word embedding model, the vector-similarity tends to reflect the word-similarity.

Therefore, feeding these vectors as features of words into the other NLP systems will always boost the performance of them in many downstream tasks (Turian et al., 2010; Socher et al., 2013) .

One such qualified model is the skip-gram with negative sampling (SGNS) model proposed in word2vec (Mikolov et al., 2013; Joulin et al., 2016) , which is very popular in various NLP tasks with expressive performance. The SGNS model propose to represent each word with vectors and estimate these vectors by applying maximum likelyhood estimation method. It implicitly factorize a word-context matrix containing a co-occurrence statistic. This would assign each word w c in the vocabulary with a "word" vector v c ∈ R d and a "context" vector u c ∈ R d , so as to model the conditional probabilities p(w k |w c ) separately for different words. By maximizing the log likelyhood of p(w k |w c ), the SGNS model can estimate the "word" vector and "context" vector of each word.

However, the SGNS model's main problem is that it doesn't build interpretable model for the embeddings themself, and therefore, people don't understand how word vectors can express useful information. For example, previous work emphasized that the inner product between one word's "word" vector and another word's "context" vector represents the point-wise mutual information between the two words (Levy & Goldberg, 2014) . However, people never use the "word-context" inner product in practise. Instead, people will simply chose "word" vectors as the word embeddings and drop the other one or vice versa. Therefore, we should care about the behaviour of "word-word" or "context-context" inner product, which are rarely analyzed and are never guaranteed to have any good properties.

In this paper, we propose a variational inference based framework to learn more interpretable word embeddings. That is, we would estimate a probability density distribution for each word instead of estimating a vector from the training corpus. To be specific, we propose to replace the "word" vector with a probability density distribution, namely the von Mises-Fisher (vMF) distribution, and keep the "context" vector for each word. As what we will show in this paper, representing the word with a probability density distribution can result in more interpretable word embeddings. Besides, the probability density representations can provide other benefits too. For example, such representations can model the polysemy phenomenon when we train our model. To be specific, we can sample a The main inspiration for this work is the analysis in Ma (2017)'s work, which can also be found in Arora et al. (2016) 's paper. They assumed that the word vectors would obey the uniform distribution over a sphere during the analysis. This assumption is critical in their analysis and yet it turns out to be wrong as reported by Mimno & Thompson (2017) . In fact, the word frequency obey the Zipf's law, which means that it's impossible for word vectors to obey the uniform distribution when we represent each word by a vector. However, when we represent each word by a probability density distribution, it's possible that the mixture distribution of all words is uniform if we adjust each word's probability density distribution's position and shape carefully.

To estimate each word's probability density representation, we need to adopt the Bayesian variational inference technique, and this would result in a Bayesian version of SGNS model. We are not the first to propose a Bayesian version of word embedding model (Zhang et al., 2014; Sakaya et al., 2015; Barkan, 2017) . Among them, the state-of-the-art model is the BSG model introduced by Bražinskas et al. (2017) , and it is also the most related work with us. However, our model is different from the BSG model in many aspects. The BSG model represents each word with a Gaussian distribution while we adopt the vMF distribution. It's important to notice that the vMF distribution is defined over a unit hyper-sphere, which means that we will sample a unit vector for each word during training. Further more, the BSG model is based on the variational autoencoding framework (VAE). As reported by Davidson et al. (2018) , it's impossible for the native VAE to work well when the prior is defined over a hyper-sphere. In contrast, we didn't adopt the VAE framework, and yet we would still talk about the reparameterization tricks. At last but not least, the BSG model focuses on how to build a more reasonable model using the VAE techniques, while we focus on how to generate more interpretable word embeddings using the vMF distribution.

Our main contributions can be summarized as follows. First, we proposed to represent each word with a vMF distribution. Second, we generated highly interpretable word embeddings, and show that our model's "context-context" inner-product would represent the point-wise mutual information between words. At last, our word embeddings out-perform the dominant models and the state-ofthe-art model in various tasks.

2. OUR FRAMEWORK:

In this section, we intend to generate interpretable word embeddings by adopting the vMF representation for each word. Specifically, we will show that the "context-context" inner-product between word embeddings would represent the point-wise mutual information between words.

2.1. MODEL DEFINITION:

The SGNS model intends to maximize the probability of a context word appearing around a center word. It assumes this probability is proportional to the inner product between two fixed vectors. In contrast, we assume that the probability of a context word appearing around a center word should be the average probability when the center word take different meanings. That is, we are suggesting a generative model as pictured in 1. Assuming there are T words w c in a training corpus, then for each word we would like to predict the possible words w k appearing around it within a K length word window. However, it's hard to determine what exactly the current word w c means, therefore, we propose to take the average probability over all possible vc for each observed w k .

The training objective of our model is to find vector representations and probability density representations for words that are useful for predicting the surrounding words in a sentence. Given a sequence of training words w 1 , • • • , w T , we are meant to maximize the average log likelyhood

EQUATION

)

where θ is the set of all parameters to be optimized. What's more, vt ∈ R d denotes a vector representation for a patential meaning of w t , and p(v t |w t ; θ) is the probability density representaton for w t . We sample vt according to p(v t |w t ; θ). At last, p(w t+j |v t , w t ; θ) denotes the probability of word w t+j appearing given vt , and L(θ) denotes the possible regularization term.

Unfortunately, it's intractable to calculate the integration in (1). Therefore, we propose to git rid of the integration by applying the jensen inequality considering that log is concate and we are doing maximization. By doing so, our model can be defined as

EQUATION

More details can be found in the appendix. We will call the p(v m t |w t ) as a prior for each word w t . Although the equation ( 2) is very similar to the ELBo in the standard varitional inference auto-encoder (VAE) technique, the actual meaning of it is quite different from ELBo. First, the expectation term in (2) doesn't involve a "encoder" as what VAE would do. Second, the regularization term in ( 2) is also different from the VAE's KL-divergence term, as what we will show.

2.2. EXPECTATION TERM:

The most troublesome component of equation ( 2) is the epctation term. For each vt sampled from word w t 's prior, we choose the softmax function to calculate the odds of word w t+j appearing around it. That is, we can decompose the expectation term into

EQUATION

where 

u i ∈ R d denotes

EQUATION

where u i is the context vector of word w i , and w i is the negative word sampled according to the empirical unigram probability p(w i ). There are N negative samples.

At last, we also need to sample vt , and we choose von Mises-Fisher (vMF) distribution to calculate the probability of w t taking this particular meaning. The vMF distribution is an analogy of Gaussian distribution over the unit sphere. It's parameterized by a mean vector and a concentration parameter κ ≥ 0. For word w t , it's corresponding mean vector v t can be interpreted as the vector representation for its average meaning. Formally, the probability of w t taking one particular meaning is

EQUATION

where d is the dimension of embedding space and I d/2-1 (•) denotes the modified Bessel function of the first kind Sra (2016) .

Figure 2 : By adjusting the position and shape of each word's vMF distribution, the mixture distribution of all words can be uniform.

2.3. REGULARIZATION TERM:

We will introduce an regularization term in this subsection, and show that this term will link the "context-context" inner-product directly to the point-wise mutual information. The general idea is to regularize the mixture distribution of all words to be uniform, then we can achieve our goal. This can be done by simply applying the maximum entropy algorithm which would result in a uniform distribution naturally. Therefore, the regularization term we are seeking for is

EQUATION

Objective (4) + ( 6) is the final objective that we want to optimize. We claim that by doing so, for each pair of words w c and w k , we have

u k u c d ≈ P M I(w k , w c ) := log p(w k , w c ) p(w k )p(w c ) ,

where P M I(w k , w c ) is called the point-wise mutual information (PMI) between word w k and word w c , and p(w k , w c ) denotes the probability for them to appear in the same word window.

Before we show why our claim holds, we would like to emphasize why PMI is important. PMI indicates how much more possible that word w k , w c co-occur than by chance (Church & Hanks, 1990) .

Then, people developed the notion of word window to help define when two words "co-occur", i.e.,

w k and w c co-occur, only when they appear in the same word window. Most of the word embedding model will take advantage of such co-occurrence statistics. Indeed, the PMI evaluated from cooccurrence counts has a strong linear relationship with human semantic similarity judgments from survey data (Hashimoto et al., 2016) . In conclusion, it's reasonable to relate word embedding with the point-wise mutual information.

Then, we will show how to link the "context-context" inner-product directly to the PMI by two steps.

The first step is to show that

EQUATION

)

where Z is the constant that most Z c approximate to. This is a conclusion proved by Ma (2017) . The second step is quite obvious because

P M I(w k , w c ) = log p(w k , w c ) -log p(w k ) -log p(w c ) ≈ u k u c d .

We will show (7) by a series equations briefly, and reveal why the regularization term is very important. More details can be found in the appendix. We start with p(w k , w c )

EQUATION

)

Step ten is why we need a regularization term, but before that, we would like to explain all the equations above. The first step of (8) says that w k and w c co-occur iff they appear in another word Step six is a strong claim which needs rigorous prove. We put this prove in the appendix.

The key insight is that vi , v i obey the same vMF distribution, which means that when this vMF is concentrate enough, then the probability of vi , v i being very different is small. In the eighth step, we omit i to emphasize that every word can generate the same vector v and that's why the summation and integration can exchange with each other in this way.

If we regularize the v to be uniformly distributed over the unit sphere in step ten, then (u k + u c ) v will obey Gaussian distribution approximately (Ma, 2017) . This means that step eleven holds. More details can be found in the appendix. The last step is the result of a famous calculation practise

E x∼N (0,σ 2 ) {exp(x)} = exp(σ 2 /2).

Obviously, by replacing the "word" vector with the vMF distribution, we can eliminate the assumption that p(w i ) being uniform.

2.4. REPARAMETERIZATION TRICK:

There is one problem to solve before our model becomes practical. p(v t |w t ) is difficult to optimize because the operation of sampling is nondifferentiable. We can solve this problem by applying the reparameterization trick. The vMF distribution's reparameterization trick is usually discussed in the context of hyperspherical variational auto-encoders (Davidson et al., 2018; Xu & Durrett, 2018; Guu et al., 2018) . To simplify our model, we propose to fix the concentration parameter κ t as a constant during training for each word w t . This is because the gradient estimation of κ is complex and computational expensive.

When it comes to each word w t 's mean vector v t , we follow the technique used in Xu & Durrett's work. Firstly, we sample an auxiliary random variable ω according to the rejection sampling scheme of Wood (1994) . The distribution of ω is controlled by κ. Specifically, the probability of ω being sampled is p(ω; κ) ∝ exp(ωκ)(1 -ω 2 ). Then we draw our vt in the following way

vt = ωv t + z 1 -ω 2 ,

where z is a random unit vector and is tangent to the unit sphere S d-1 at v t . Figure 3 illustrates the geometric vision. Because of applying this trick, we can take gradient with respect to v t as usual.

2.5. FINAL ALGORITHM:

Putting all the details together, we have the following objective to minimize

EQUATION

Based on our final model, we propose algorithm (1) to train the word embeddings. The for loop started from line 1 controls how many times we will go through the entire training corpus. Then, for each epoch, we would iterate over each word in the corpus as what the for loop in line 2 suggests. In line 3, we will take w c 's mean vector v c and concentrate parameter κ c from dictionaries. The third layer of for loop will iterate over all the words in a word window centered around w c . For each word w k in this word window, we will take its context vector u k from the dictionary, and sample a v from w c 's prior with the help of our reparameterization trick (line 5 to line 6). Then form line 7 to line 10, we will sample N negative samples for each w k , and then calculate the gradients of L 1 . From line 11 to line 13, we will calculate L 2 based on the v, and update all word's prior accordingly.

noend 1 PDF (N, u i , v i , κ i , p(w i ), i = 1, • • • , |V |) 1: for epoch in epochs do 2:

for w c in corpus do 3:

v c ← {v i } i=1,••• ,|V | , κ c ← {κ i } i=1,••• ,|V | 4: for w k in W indow(w c ) do 5: ω ∼ p(ω; κ c ), z ∼ U(S d-2 ) 6: v ← ωv c + z √ 1 -ω 2 , u k ← {u i } i=1,••• ,|V | 7:

for i in range(N ) do 8:

w i ∼ p(w i ), u i ← {u i } i=1,••• ,|V | 9: L 1 ← L 1 (u k , u i , v) 10:

Update v c , u k according to L 1 's gradient.

11:

L 2 ← L 2 ({v j } j=1,••• ,|V | , v) 12:

for j in range(|V |) do 13:

Update v j according to L 2 's gradient. 

3. EXPERIMENTS:

We will now experimentally validate our embedding by comparing the performance of our model with the dominant word embedding models on the word similarity tasks. Subsection 3.1 presents all the experiment settings for different training corpus and different word embedding models. Subsection 3.2 introduces several word similarity benchmarks, and how we evaluate model's performance on them. Subsection 3.3 compares our model with the STOA model on several benchmarks.

3.1. EXPERIMET SETTINGS:

We use part of massachusetts bay transportation authority (mbta) web crawled corpus because it is well cleaned and is large enough. The mbta corpus contains about 600 million tokens. We preprocessed all corpora by removing non-English characters, numbers and lower-casing all the text. The vocabulary was restricted to the 100K most frequent words in each corpus.

We trained embeddings using three methods: word2vec Mikolov et al. (2013) , GloVe Pennington et al. (2014) , and our model. This is because these models are implemented with C/C++, and therefore are fast enough to be evaluated on mbta corpus. For fairness we fix all hyperparameters for word2vec, GloVe and our model. Specifically, we trained for 5 epochs for each model using 75 threads for parallel computation; the word embedding dimension is 100; the window size is 5.

For the word2vec, and our model, the negative sampling number is 5; We adopt the Hogwild! algorithm to train models, and the learning rate decays linearly from 0.0025 to 0. The noise distribution is set as the same as used in Mikolov et al., p n (w) ∝ p(w w ) 0.75 . We also use a rejection threshold of 10 -4 to subsample the most frequent words.

For the GloVe model, we follow the original inplementation's default settings for the other hyperparameters. This means the initial learning rate is 0.05.

We also use the news corpus1 with about 15 million tokens to evaluate the BSG model. This is because the original implementation of BSG is based on theano, and it's slow to be trained on the mbta corpus. Given the small volume of this corpus, we trained for 25 epochs for each model using 12 threads. we also restrict each model's vocabulary to be 10K. For both the BSG, word2vec and our model, we set the negative number to be 10; the window width is 10.

3.2. PERFORMANCE EVALUATION:

We test the quality of the word embeddings by checking if our word embeddings agree with the human judgement on word similarity / relatedness.

For the mbta corpus, we test the performance of our embeddings on three benchmarks: the Word Similarity353 data set Finkelstein et al. (2002) , the RG65 data set Luong et al. (2013) , and the MEN data set Bruni et al. (2014) . Taking the WS353 data set for example, it contains 353 word pairs along with their similarity scores assigned by 29 subjects. These subjects possessed near-native command of English and they are instructed to estimate the relatedness of the words in pairs on a scale from 0 to 10. During experiments, we will cumpute the Spearman's rank correlation coefficient Spearman (1904) between human judgement and the inner product between the vector representations. The larger this coefficient is, the better this embedding is and the mximume value is 1. Some words in these testing datasets do not appear in our training corpora, and this means we can't calculate the inner product between vectors for those words. In order to provide comparable results, we propose to use the mean vectors of the rest words for these missing words. Also, the mean vector of each word's prior is similar to the "word" vector in the SGNS model, and we also test the performance of this vector too.

We also evaluate the performance over more Benchmarks. Table 2 presents similarity results computed using the online tool of Faruqui & Dyer (2014) . Since the BSG model can only generate the mean vectors from its prior, we only evaluate the "word" or mean vectors in these experiments.

3.3. COMPARE WITH THE DOMINANT MODELS:

Table 1 shows the results on these benchmarks. As we can see, our "context" word embedding can constantly outperform the counterpart of the dominant word embedding models by a large margin. This is exactly what we expect according to our theory. It's interesting that the "word" vector of our model can still outperform ours "context" vector. Another exception is that on the RG65 test set, the SGNS model can outperform our model by a large margin. We argue this may be caused by its bias -it only contains 65 noun pairs after all. Besides, our model's "context" vector can still outperform SGNS on this test set. Also, the gap between the "context" vectors and the "word" vectors in our model is much smaller than the dominant models' gaps. These results demonstrate that our model is more specific about which part of our embedding contains the useful information.

3.4. COMPARE WITH THE BSG MODEL:

First, we observe that our model can out-perform the BSG model for almost every task except for the RG65 data set as before. Although the BSG model can perform better than our model in the WS353-Rel data set, we would like to point out that this data set is a subset of WS353-ALL. Therefore, this may also be caused by the bias of small data set. Second, the performance of BSG model is weaker than the SGNS model for some data sets. Given that we used their released implementation directly, this may be the result of small training data set, i.e., the SGNS model may perform better in the case of small training data set because of its simplicity.

4. CONCLUSIONS:

We generated interpretable word embeddings by represent each word with a von Mises-Fisher distribution. We have demonstrated that our word embeddings can be linked to the point-wise mutual information directly without making any unrealistic assumptions. The experiments over different training and testing data sets demonstrate that our model can outperform both the dominant and the STOA models. We argue that our insight into the interpretable word embeddings is important. For example, as we are sure that the unit word vectors can encode the semantic similarity between words, it's possible to encode the syntactic information between words into the norm of word vectors.

:

https://drive.google.com/file/d/1QWC2x6qq8KyHFUCgyvVJJoGHexZrw7gO/view

Figures and Tables:

Figure 1: Each word w c would generate a latent meaning vector vc , and then w k is picked by vc .

i 's word window together. This is true for the Skip-gram model. The second step is also true when we considering the definition of p(w k |w i ) and p(w c |w i ) in the Skip-gram model. Step four is just by definition and we use slightly different notations here to indicate that vi , v i are different variable.

Figure 3: The illustration about our reparameterization trick.

Results on the word similarity tasks.

Table content: <table><tr><td colspan="6">Model Embedding WS353 WS353-SIM WS353-REL RG65</td><td>MEN</td></tr><tr><td>SGNS</td><td>context</td><td>0.5648</td><td>0.6184</td><td>0.4832</td><td>0.491</td><td>0.479</td></tr><tr><td>GloVe</td><td>context</td><td>0.5686</td><td>0.6219</td><td>0.4864</td><td colspan="2">0.4959 0.4921</td></tr><tr><td>Ours</td><td>context</td><td>0.6378</td><td>0.6932</td><td>0.5961</td><td>0.522</td><td>0.646</td></tr><tr><td>SGNS</td><td>word</td><td>0.6606</td><td>0.7071</td><td>0.6237</td><td>0.669</td><td>0.676</td></tr><tr><td>GloVe</td><td>word</td><td>0.6436</td><td>0.7085</td><td>0.5865</td><td colspan="2">0.6606 0.666</td></tr><tr><td>Ours</td><td>mean</td><td>0.6637</td><td>0.7444</td><td>0.6404</td><td>0.596</td><td>0.68</td></tr></table>

Results on the word similarity tasks.

Table content: <table><tr><td>Datasets</td><td>Ours</td><td>BSG</td><td>GloVe SGNS</td></tr><tr><td>MC-30</td><td colspan="3">0.6190 0.5818 0.4524 0.4524</td></tr><tr><td>MEN-Tr-3k</td><td colspan="3">0.4905 0.3937 0.3607 0.5182</td></tr><tr><td>MTurk-287</td><td colspan="3">0.5343 0.5147 0.3334 0.5351</td></tr><tr><td>MTurk-771</td><td colspan="3">0.4221 0.3693 0.2647 0.4177</td></tr><tr><td>RG65</td><td colspan="3">0.5727 0.6036 0.3455 0.4000</td></tr><tr><td colspan="4">RW-STNFRD 0.4172 0.3830 0.2710 0.3725</td></tr><tr><td colspan="4">SIMLEX-999 0.2110 0.1717 0.1914 0.2339</td></tr><tr><td>VERB-143</td><td colspan="3">0.3127 0.1371 0.0712 0.2376</td></tr><tr><td colspan="4">WS353-ALL 0.4567 0.4350 0.2379 0.4387</td></tr><tr><td>WS353-Rel</td><td colspan="3">0.3485 0.4197 0.1583 0.3591</td></tr><tr><td colspan="4">WS353-SIM 0.6070 0.5018 0.3277 0.5549</td></tr></table>

Peer review (iclr2020):
The paper addresses the problem the problem in word embeddings where *word-word* or *context-context* inner products are relied upon in practice when the embeddings are usually only optimized for good properties of *word-context* inner products. The new approach to training word embeddings addresses interpretability in the sense of having a theoretically grounded interpretation for the good properties that the inner products should possess -- namely that they capture the PMI between words. By changing the embedding training procedure (to include a step that samples vectors from each word*s distribution representation) this interpretation is made possible without problematic theoretical assumptions. Now trained for the purpose they are intended to be used for, the approach yields strong SOTA results on the non-small challenge datasets. The word is well exceptionally well motivated (we should directly optimize for the properties we want!) and situated in the literature. The opposition of Arora 2016 (getting at *interpretable* word embeddings with a latent variable model) and Mimno & Thompson 2017 (problematic assumptions in skip-gram embeddings) is particularly convincing. The vMF distribution is underexamined as tool for analyzing high-dimensional semantic vector distributions, and it is an excellent fit for the purposes of this project. To back up claims of theoretical interpretability, a derivation (building on Ma 2017) proceeds without problematic leap thanks to a property introduced by a carefully selected (if straightforward) regularization term. This reviewer did not read the appendix (not sure where this would be located -- first time reviewing with OpenReview here), but the intuition doesn*t seem to be much different than Ma. To back up claims of applied performance, appropriate experiments are conducted on multiple evaluation datasets. The results seem to be fairly interpreted. This reviewer decides to accept this paper for its balance of theoretical and empirical contributions and for the role it might play in reducing dependence on mysticisms in word embeddings (relying on accidental / uncharacterized properties of previous embedding strategies). Suggestions for improvement: - Run a spell checker. It will catch a large number of problems that weren*t ever big enough to hurt my appreciation of the paper. - Consider the creation of an adversarial dataset (of ~3 words in ~3 contexts) where techniques that optimize for the wrong thing will catastrophically fail but the proposed approach succeeds. - Write one or two more sentences about the fate of kappa_c -- is it ever updated or am I just missing something? Making a bigger point about leaving kappa un-optimized shows there is room for additional depth in this line of thinking. (If you start learning concentration parameters, check out the Kent distribution for an even more expressive distribution on the unit hypersphere: https://en.wikipedia.org/wiki/Kent_distribution) - Figure out what happened to your appendix. - Watch out that readers/reviewers of similar work may try to read the word *interpretable* in the title as a reference to the much broader topic of interpretability in ML related to explaining a model*s behavior. Is there a synonym for *interpretable* that won*t raise this false link?