Learn Interpretable Word Embeddings Efficiently with von Mises-Fisher Distribution
Authors: Minghong Yao
Liansheng Zhuang
Houqiang Li
Jian Yang
Shafei Wang
Venue:  (2020)
Keywords: word embedding
natural language processing

Abstract: Word embedding plays a key role in various tasks of natural language processing. However, the dominant word embedding models don't explain what information is carried with the resulting embeddings. To generate interpretable word embeddings we intend to replace the word vector with a probability density distribution. The insight here is that if we regularize the mixture distribution of all words to be uniform, then we can prove that the inner product between word embeddings represent the point-wise mutual information between words. Moreover, our model can also handle polysemy. Each word's probability density distribution will generate different vectors for its various meanings. We have evaluated our model in several word similarity tasks. Results show that our model can outperform the dominant models consistently in these tasks.

Under review as a conference paper at ICLR 2020
LEARN INTERPRETABLE WORD EMBEDDINGS EFFI- CIENTLY WITH VON MISES-FISHER DISTRIBUTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Word embedding plays key role in various tasks of natural language processing. However, the dominant word embedding model don't explain what information is carried with the resulting embeddings. To generate interpretable word embed- dings we intend to replace the word vector with a probability density distribution. The insight here that if we regularize the mixture distribution of all words be uniform, then we can prove that the inner product between word embeddings represent the point-wise mutual information between words. Moreover, our model can also handle polysemy. Each word's probability density distribution will gen- erate different vectors for various meanings. We have evaluated our model several word similarity tasks. Results show that our model can outperform the dominant models consistently these tasks.
2
11
12
INTRODUCTION
Word embedding is widespread technique in boosting the performance of modem NLP systems 14. by learning vector for each word as its semantic feature. The general idea of word embedding is to
15 assign each word with a dense vector having lower dimensionality than the vocabularies' cardinal-
18 ity. In a qualified word embedding model, the vector-similarity tends to reflect the word-similarity.
17 Therefore, feeding these vectors as features of words into the other NLP systems will always boost
18 the performance of them in many downstream tasks (Turian et al. 2010 Socher et al. 2013).
19 One such qualified model the skip-gram with negative sampling (SGNS) model proposed in word2vec (Mikolov et al., 2013 Joulin et al.] 2016), which is very popular in various NI.P tasks with expressive performance. The SGNS model propose to represent each word with vectors and estimate these vectors by applying maximum likelyhood estimation method. It implicitly factorize a word-context matrix containing a co-occurrence statistic. This would assign each word w, in the vocabulary with "word" vector Ve E gd and a "context" vector u, E Rd so to model the con- ditional probabilities p( wx (wa) separately for different words. By maximizing the log likelyhood of p(w)/wa). the SGNS model can estimate the "word" vector and "context" vector of each word.
20
21
22
23
25
27 However, the SGNS model's main problem is that it doesn't build interpretable model for the em- beddings themself, and therefore, people don't 't understand how word vectors express useful 29 information. For example, previous work emphasized that the inner product between one word's
30 "word" vector and another word's "context" vector represents the point-wise mutual information
31 between the two words (Levy & Goldberg, 2014). However, people never use the "word-contexs
32 inner product in practise. Instead, people will simply chose "word" vectors as the word embeddings
33 and drop the other one or vice versa. Therefore, we should care about the behaviour of "word-word"
34 or "context-context" inner product, which are rarely analyzed and are never guaranteed to have any
35 good properties.
36 In this paper, we propose variational inference based framework to learn more interpretable word
37 embeddings. That is, we would estimate probability density distribution for each word instead of estimating a vector from the training corpus. To be specific, we propose to replace the "word" vector with a probability density distribution, namely the von Mises-Fisher (vMF) distribution, and keep 40 the "context" vector for each word. As what we will show in this paper, representing the word with probability density distribution result in more interpretable word embeddings. Besides, the 42 probability density representations can provide other benefits too. For example, such representations sample model the polysemy phenomenon when we train our model. To be specifie,
38

---

Under review as a conference paper at ICLR 2020
W
A- Lo
Figure Each word w/ would generate a latent meaning vector v.. and then wy picked by Ve-
vector from the vMF distribution to represent the specific mcaning of this word in particular context during training.
45
The main inspiration for this work is the analysis in Ma (2017)'s work, which also be found in Arora et al. (2016)'s paper. They assumed that the word vectors would obey the uniform distribution over sphere during the analysis. This assumption is critical in their analysis and yet it turns out to be wrong as reported by Mimno & Thompson (017). In fact, the word frequency obey the Zipf"s law, which means that it's impossible for word vectors to obey the uniform distribution when we
47
48
49
50
51 represent each word by vector. However, when we represent each word by probability density
52 distribution, it's possible that the mixture distribution of all words uniform we adjust each word's probability density distribution's 's position and shape carefully.
53
54 To estimate each word's probability density representation, we need adopt the Bayesian varia- tional inference technique, and this would result in a Bayesian version of SGNS model. We are not the first to propose a Bayesian version of word embedding model (Zhang et al. 2014 Sakaya et al. 2015 Barkan 2017). Among them, the state-of-the-art model is the BSG model introduced by Bražinskas et al. 2017). and it is also the most related work with us. However, our model is dif- ferent from the BSG model in many aspects. The BSG model represents each word with a Gaussian distribution while we adopt the vMF distribution. It's important to notice that the vMF distribution defined over unit hyper-sphere, which means that will sample unit vector for each word during training. Further more, the BSG model is based on the variational autoencoding framework (VAE). As reported by Davidson al. (2018), it's impossible for the native VAE work well when the prior is defined over a hyper-sphere. In contrast, we didn't adopt the VAE framework, and yet we would still talk about the reparameterization tricks. At last but least, the BSG model focuses on how to build a more reasonable model using the VAE techniques, while we focus on how to generate more interpretable word embeddings using the vMF distribution.
55
56
58
60
82
63
64
65
66
67
68
Our main contributions can be summarized as follows. First, we proposed to represent each word with a vMF distribution. Second, we generated highly interpretable word embeddings, and show that our model's "context-context" inner-product would represent the point-wise mutual information between words. At last, our word embeddings out-perfort the dominant models and the state-of- the-art model in various tasks.
69
71
73 OUR FRAMEWORK
74 In this section, we intend to generate interpretable word emheddings by adopting the vMF represen- tation for each word. Specifically, will show that the "context-context" inner-product between word embeddings would represent the point-wise mutual information between words.
78
77 2.1 MODEL DEFINITION
The SGNS model intends to maximize the probability of context word appearing around a center 79 word. It assumes this probability is proportional to the inner product between two fixed vectors. In contrast, assume that the probability of context word appearing around center word should 81 be the average probability when the center word take different meanings. That is, we are suggesting generative model pictured in [ Assuming there are T words w. training corpus, then for
2

---

Under review as a conference paper at ICLR 2020
each word we would like to predict the possible words wy appearing around it within K length word window. However, it's hard to determine what exactly the current word we means, therefore, propose to take the average probability over all possible v. for each observed wy.
84
86
The training objective of our model is to find vector representations and probability density repre- sentations for words that are useful for predicting the surrounding words in sentence. Given sequence of training words W1," :.. Wr. we are meant to maximize the average log likelyhood
88
T
K/2
arg max
Σ
Σ
log
p(artt/Ve Ve, we 0)p( Vel 0) + L(0), j *
(I)
T j=-K/2
where A is the set of all parameters to be optimized. What's more, v E Rd denotes vector 90 representation for a patential meaning of wt, and ():;)) is the probability density representaton
91 for w,- We sample v, according to p, Jwy; 8). At last, p(W,+ //,,, Wr; 0) denotes the probability of
92 word We+j appearing given v,, and L(0) denotes the possible regularization term.
Unfortunately, it's intractable calculate the integration in (T). Therefore, we propose to git rid of 94 the integration by applying the jensen inequality considering that log is concate and we are doing maximization. By doing so, our model can be defined as
95
T
arg MaX T
1
Σ
log p(we+s (iT w;; + L(0), 40 t=1 j=-n (
Σ.
E
(2)
96
More details can be found in the appendix. We will call the p(v;" as a prior for each word Wt. Although the equation (2) is very similar to the ELBo in the standard varitional inference auto-encoder (VAE) technique, the actual meaning of it is quite different from BLBo. First, the ex- pectation term in (2) doesn't 't involve "encoder" as what VAE would do. Second, the regularization term in 2) is also different from the VAE's KL-divergence term, as what we will show.
88
99
100
101
2.2
EXPECTATION TERM
102
The most troublesome component of equation (2) is the epctation term. For each v, sampled from word wy's 's prior, we choose the softmax function to calculate the odds of word W appearing around That is, we can decompose the expectation term into
103
104
exp(u ve)
E Pemp(Pp'o boc)
log p(we++ V." 0) =
E (w
log
(3)
ΣM exp(u v.)
where u, Rd denotes the "context" vector for word wy, and [V] is the cardinality of our vocabu- lary. As what's been suggested in the SGNS model, we extend the negative sampling technique to our model avoid the computation of the softmax function's denominator. It's worth to notice that
105
108
107
108
different words would have different denominators in theory. We will use Z; to denote the denom-
109
inator of word w;, According to the theory proved by Gutmann & Hyvärinen (2012), maximizing
110
objective (3) is equivalent to minimizing
E Verpp(or we)
[log [logo(u,, v:) + NE log (-u. vc)]
where u, is the context vector of word w;, and w; is the negative word sampled according to the empirical unigram probability p(w:). There are N negative samples.
112
At last, also need to sample v,, and we choose von Mises-Fisher (vMF) distribution to calculate
114 the probability of wy taking this particular meaning. The vMF distribution is an analogy of Gaussian distribution over the unit sphere. It's parameterized by mean vector and concentration parameter
116 x 2 O. For word Wi, it's corresponding mean vector vi can be interpreted as the vector repre-
117 sentation for its average meaning. Formally, the probability of w, taking one particular meaning
118 is
p(Velwi) =Ca(Ks) (((K))))p(((NN))) N. x4/2-1
eXp(Rev, Ve).
(5)
(a())) (27)4/A1a/2-11m)) (N.))
119 where d is the dimension of embedding space and Ia/2-1 (-) denotes the modified Bessel function
120 of the first kind Sra (2016).
3

---

Under review as a conference paper at ICLR 2020
D1.
0 (())ww)
~ U(S
Figure 2: By adjusting the position and shape of each word's vMF distribution, the mixture distri- bution of all words can be uniform.
121
2.3
REGULARIZATION TERM
122
We will introduce an regularization term this subsection, and show that this term will link the "context-context" inner-product directly to the point-wise mutual information. The general idea is to regularize the mixture distribution of all words to be uniform, then we can achieve our goal. This can be done by simply applying the maximum entropy algorithm which would result in uniform distribution naturally. Therefore, the regularization term we are seeking for is
124
128
arg max p(v) lo)pp()()) where p(v) MM p(u)))((\/ss)
(6)
(6) is the final objective that we want to optimize. We claim that by doing so, for each pair of words w, and wk. we have
Objective 4
p(ukk we) n uh so PMI(Wk, we) = log (p(w)))((we))
127
where PM]( w.) is called the point-wise mutual information (PMJ) between word wy and word Wes and p(wk; w.) denotes the probability for them appear in the same word window.
129
Before we show why our claim holds, we would like to emphasize why PMI is important. PMI indi-
130
cates how much more possible that word wk, We co-occur than by chance (Church & Hanks 1990).
Then, people developed the notion of word window to help define when two words "co-occur" I.c.,
132
Wk and w. co-occur, only when they appear in the same word window. Most of the word embedding model will take advantage of such co-occurrence statistics. Indeed, the PMI evaluated from co-
133
134
occurrence counts has a strong linear relationship with human semantic similarity judgments from
survey data (Hashimoto et al.) 2016). In conclusion, it's reasonable to relate word embedding with the point-wisc wise mutual information.
136
Then, will show how to link the "context-context" inner-product directly to the PMI by two steps. The first step is to show that
138
log p pwr,, we)
lu -2 2log(2)
2d
log p(rk)
|u
log (7),
2d
log (ww)
log Z),
where Z is the constant that most Z, approximate to. This is a conclusion proved by Ma 2017) The second step is quite obvious because
u. u
PMI(Wk. we) = log p(wk. we) - log p(w)))))) - log ((w)) x
d
4

---

Under review as a conference paper at ICLR 2020
We will show 7 by series equations briefly, and reveal why the regularization term is very impor- tant. More details can be found in the appendix. We start with p(w;, we)
140
p(w,, we) = p(w;)p(wx: welwx)
Σ p( (ww))p(wwww
E
pwwx (oW))((ww)))
implass)
=
E
J
2: p(
12 ( (Vs/ws)dd
(mpp(ws)
2²
E implw))
JS ceul Viee 8: "pp(///w,)) hppppppppp ( ws)ds
E
S fexp [uus +
To)T u
p(
(8)
iup((w))
V
z² Σ p(w;): _xxexp/ [u + ue) T 9.] p(ps)w;)ds)
1
exp [u + uu)) [+] Σ/' p p(w))p((1 ds
= 2
I exp (u: + u.) Tv] p( dds
E (exp [uu u.)
}]]
= 2 smup())
= Z Ez-A tu (3/d) (exp(x))
||
u..
= 22 exp
2d
Step ten is why need regularization term, but before that, would like to explain all the equations above. The first step of (8) says that uy and w. co-occur iff they appear in another word wy's word window together. This true for the Skip-gram model. The second step is also true when we considering the definition of p(w, (w;) and p(we| (w;) in the Skip-gram model. Step four just by definition and we use slightly different notations here to indicate that v, v are different variable. Step six is strong claim which needs rigorous prove. We put this prove in the appendix. The key insight that v, V obey the same vMF distribution, which means that when this vMF is concentrate enough, then the probability of v. V being very different is small. In the eighth step, we omit to emphasize that every word can generate the same vector and that's why the summation and integration can exchange with each other in this way.
142
143
144
145
148
147
148
149
If we regularize the v to be uniformly distributed over the unit sphere in step ten, then (ug + u.)) T will obey Gaussian distribution approximately (Ma) 2017). This means that step eleven holds. More details can be found in the appendix. The last step is the result of a famous calculation practise
E (0. (xxp(x)) exp(o² /2).
Obviously, by replacing the "word" vector with the vMF distribution, can eliminate the assump- tion that p(w;) being uniform.
152
153
2.4
REPARAMETERIZATTIN TRICK
154
There is one problem to solve before our model becomes practical. p(V, /w:) is difficult to optimize because the operation of sampling is nondifferentiable. We can solve this problem by applying the
155
156
reparameterization trick. The vMF distribution's reparameterization trick usually discussed in the context of hyperspherical variational auto-encoders (Davidson et al. 2018 Xu & Durrett, [0188] Guu et al. 2018). To simplify our model, we propose to fix the concentration parameter Ki as constant during training for each word w.. This is because the gradient estimation of is complex and computational expensive.
158
180
When it comes to each word wy's mean vector Va, follow the technique used in Xu & Durrett's s work. Firstly, we sample an auxiliary random variable w according to the rejection sampling scheme of Wood (1994). The distribution of w is controlled by Specifically, the probability of w being sampled is
p(w;k) exp(wn)(1 w²).
5

---

Under review as a conference paper at ICLR 2020
Figure 3: The illustration about our reparameterization trick.
Then we draw our v, in the following way
V. 2V1 w
where z is random unit vector and is tangent to the unit sphere sd-1 at Figure illustrates the geometric vision. Because of applying this trick, we can take gradient with respect to V, as usual.
162
165
2.5
FINAL ALGORITIIM
Putting all the details together, have the following objective minimize
log σ (u, v) NERpp((ee)
[log u- +)]
p( log p
Le
164
where
wVe + 2 w² l/vel2
w plw; Ki)) x exp(wxic)1 w
||z||2 1,, langent to Sd-1 al Ves
p(V) = Σ} ((w))(())) (()W)
p(((10)) = ca(k)) exp(px v),
ca(ks)) = (24)4/21a12 1(5))"
185
Based on our final model, we propose algorithm (1) train the word embeddings. The for loop started from line I controls how many times we will through the entire training corpus. Then, for each epoch, we would iterate over each word in the corpus as what the for loop in line suggests. In line 3, we will take w.'s mean vector V. and concentrate parameter Ke from dictionaries. The third layer of for loop will iterate over all the words in word window centered around w.. For each word in this word window, we will take its context vector from the dictionary, and sample a V from w.'s prior with the help of our reparameterization trick (line 5 to line 6). Then form line 7 to line 10, we will sample N negative samples for each Wi. and then calculate the gradients of L, From
186
187
188
169
170
171
173
line 11 to line 13, we will calculate L2 based on the v, and update all word's 's prior accordingly.
noend PDF (N, ug V., Kig p(w;) 1, [V)
1:
for epoch in epochs do
2: for we corpus do
3:
Ve +- [vi)i=1,.-..IV1 |V Ro (
4:
for we in Window w.) do
5:
p(w; Ke), US
6:
< wv. + zv1 -h, us +- {u (ui for i in range(NN) do
7:
8:
wi ~ p(w;), u; + {u (u.)i=,,....M Li + Li (uR; u, v) Update ve, according to L gradient. v)
9:
10:
Ta < (22((,))=_
for j in range((V]) do
13:
Update v, according to L3 gradient.

---

Under review as a conference paper at ICLR 2020
Model
Embedding
WS353
WS353-SIM
WS353-REL
RG65
MEN
SGNS
context
0.5648
0.6184
0.4832
0.491
0.479
GeoVe
context
0.5686
0.6219
0.4864
0.4959
0,4921
Ours
context
0.6378
0.6932
0.5961
0.522
0.646
SGNS
word
0.6606
0.7071
0.6237
0.669
0.676
GloVe
word
0.6436
0.7085
0,5865
0.6606
0,666
Ours
mean
0.6637
0.7444
0.6404
0.596
0.68
Table Results on the word similarity tasks.
174
3
EXPERIMENTS
175
We will now experimentally validate our embedding by comparing the performance of our model with the dominant word embedding models the word similarity tasks. Subsection 3.1 presents all the experiment settings for different training corpus and different word embedding models. Subsec- tion 3.2 introduces several word similarity benchmarks, and how we evaluate model's performance on them. Subsection 3.3 compares our model with the STOA model on several benchmarks.
178
177
178
179
180
3.1
EXPERIMET SETTINGS
We use part of massachusetts bay transportation authority (mbta) web crawled corpus because it well cleaned and is large enough. The mbta corpus contains about 600 million tokens. We preprocessed all corpora by removing non-English characters, numbers and lower-casing all the
182
183
184
text. The vocabulary was restricted to the 100K most frequent words in each corpus.
We trained embeddings using three methods: word2vec Mikolov et al. (0033), Glo Pennington et al. (0144), and our model. This is because these models are implemented with CIC++, and therefore are fast enough to be evaluated
186
mbta corpus. For fairness for word2vee, GloVe and our model. Specifically, we trained for epochs for each model using 75 threads for parallel computation; the word embedding dimension is 100; the window size is 5.
fix all hyperparameters
188
190
For the word2vec, and our model, the negative sampling number is 5; We adopt the Hogwild! algo- rithm to train models, and the learning rate decays linearly from 0.0025 O. The noise distribution is set as the same as used Mikolov et al., Pr, (w) x p(w,,, 0.75 We also use a rejection threshold of 10-4 subsample the most frequent words.
192
183
194
For the GloVe model, we follow the original inplementation's 's default settings for the other hyperpa- rameters. This means the initial learning rate is 0.05.
195
196
We also use the news corpus] with about 15 million tokens to evaluate the BSG model. This is because the original implementation of BSG is based theano, and it's slow to be trained on the mbta corpus. Given the small volume of this corpus, we trained for 25 epochs for each model using 12 threads. we also restrict each model's vocabulary be 10K. For both the BSG, word2vec and our model, we set the negative number to be 10; the window width is 10.
197
198
199
200
3.2
PERFORMANCE EVALUATION
202
We test the quality of the word embeddings by checking if our word embeddings agree with the human judgement on word similarity relatedness.
203
For the mbta corpus, we test the performance of our embeddings on three benchmarks: the Word Similarity353 data set Finkelstein al. (2002), the RG65 data Let|Luong al. (2013), and the MEN 205
204
data set Bruni et al. (2014). Taking the WS353 data set for example, it contains 353 word pairs along with their similarity scores assigned by 29 subjects. These subjects possessed near-native command
207
of English and they instructed to estimate the relatedness of the words in pairs on a scale from 0 to 10. During experiments, we will cumpute the Spearman's 's rank correlation coefficient SSpearman
209
1904 between human judgement and the inner product between the vector representations. The aargrr this coefficient is, the better this embedding is and the mximume value is 1. Some words in
211

---

Under review as a conference paper at ICLR 2020
Datasets
Ours
BSG
GloVe
SGNS
MC-30
0.6190
0.5818
0.4524
0.4524
MEN-Tr-3k
0,4905
0.3937
0.3607
0.5182
M'Turk-287
0.5343
0.5147
0.3334
0.5351
M'Jurk-771
0.4221
0.3693
0.2647
0.4177
RG65
0.5727
0.6036
0.3455
0,4000
RW-STNERD
0.4172
0.3830
0.2710
0.3725
SIMLEX-99 9999
0.2110
0.1717
0.1914
0.2339
VERB- 143
0.3127
0.1371
0.0712
0.2376
WS353-ALI.
0.4567
0.4350
0.2379
0.4387
WS353-Rel
0.3485
0.4197
0.1583
0.3591
WS353-SIM
0,6070
0,5018
0.3277
0,5549
Table 2: Results on the word similarity tasks.
these testing datasets do not appear in our training corpora, and this means we can't calculate the inner product between vectors for those words. In order to provide comparable results, we propose 213
to use the mean vectors of the rest words for these missing words. Also, the mean vector of each word's prior is similar to the "word" vector in the SGNS model, and we also test the performance of this vector too.
215
217
We also evaluate the performance over more Benchmarks. Table presents similarity results com- puted using the online tool of FFruquu & Dyer (0014). Since the BSG model can only generate the mean vectors from its prior, we only evaluate the "word" or mean vectors in these experiments.
219
220
3.3
COMPARE WITH THE DOMINANT MODELS
Table T shows the results on these benchmarks. As we can see, our "context" word embedding can constantly outperform the counterpart of the dominant word embedding models by large margin. This is exactly what we expect according to our theory. It's interesting that the "word" vector of our model can still outperform ours "context" vector. Another exception is that on the RG65 test set, the
222
223
it only contains 65 noun pairs after all. Besides, model's "context" vector can still outperform 225
SGNS model can outperform our model by large margin. We argue this may caused by its bias
227
SGNS on this test set.
Also, the gap between the "context" vectors and the "word" vectors in model is much smaller than the dominant models' gaps. These results demonstrate that our model is more specific about which part of our embedding contains the useful information.
229
3.4
COMPARE WITH THE BSG MODEL
232
First, we observe that our model can out-perform the BSG model for almost every task except for the
233
RG65 data set before. Although the BSG model can perform better than our model in the WS353-
234
Rel data set, we would like to point out that this data set is a subset of WS353-ALL. Therefore, this
235
may also caused by the bias of small data set. Second, the performance of BSG model is weaker
than the SGNS model for some data sets. Given that we used their released implementation directly, this may the result of small training data set, i.e., the SGNS model may perform better in the case of small training data set because of its simplicity.
237
239 4
CONCLUSIONS
We generated interpretable word embeddings by represent each word with von Mises-Fisher dis- tribution. We have demonstrated that our word embeddings can linked to the point-wise mutual information directly without making any unrealistic assumptions. The experiments over different training and testing data sets demonstrate that our model can outperform both the dominant and the
241
243
STOA models. We argue that our insight into the interpretable word embeddings is important. For example, as we are sure that the unit word vectors can encode the semantic similarity between words, it's possible encode the syntactic information between words into the norm of word vectors.
245
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
248
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
249
250
Oren Barkan. Bayesian neural word embedding. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
252
253
Arthur Bražinskas, Serhii Havrylov, and Ivan Titov. Embedding words as distributions with a bayesian skip-gram model. arXiv preprint arXiv:1711.11027, 2017.
254
255
Elia Bruni, Nam- Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1-47, 2014.
Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexi- cography. Computational linguistics, 16(1):22-29, 1990.
258
259
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspheri- cal variational auto-encoders. arXiv preprint arXiv: 00891, 2018.
260
281
Manaal Faruqui and Chris Dyer. Community evaluation and exchange word vectors at wordvee- tors. org. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguis- fics: System Demonstrations, pp. 19-24, 2014.
263
264
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search context: The concept revisited. ACM Transactions on informa- tion systems, 20(1):1 1,,161,
265
266
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13 (Feb):307-361, 2012.
288
270
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450, 2018.
272
273
Tatsunori Hashimoto, David Alvarez-Melis, and Tommi Jaakkola. Word embeddings as metric recovery in semantic spaces. Transactions of the Association for Computational Linguistics, 4: 273-286, 2016.
275
276
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016.
277
278
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Ad- vances in neural information processing systems, pp. 2177-2185, 2014.
Thang Luong, Richard Socher, and Christopher Manning. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 104-113, 2013.
281
282
283
Tengyu Ma. Non-convex Optimizatioo for Machine Learning: Design, Analysis, and Understanding. PhD thesis, Princeton University, 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
286
287
288
David Mimno and Laure Thompson. The strange geometry of skip-gram with negative sampling. In Empirical Methods in Natural Language Processing, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
291

---

Under review as a conference paper at ICLR 2020
Joseph Hosanna Sakaya et al. Scalable bayesian induction of word embeddings. 2015.
Richard Socher, John Bauer, Christopher D Manning, et al. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 455-465, 2013.
295
296
297
Charles Spearman. The proof and measurement of association between two things. American journal of Psychology, 15(1):72-101, 1904.
Suvrit Sra. Directional statistics in machine learning: brief review. 2016.
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 384-394 Association for Computational Linguisties, 2010.
301
303
Andrew TA Wood. Simulation of the von mises fisher distribution. Communications in statistics- simulation and computation, 23(1):157-164, 1994.
304
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. arXiv preprint arXiv: 100000 2018
306
307
Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Gliozzo. Word semantic representations using bayesian probabilistic tensor factorization. In Proceedings of the 2014 Conference Ëm- pirical Methods in Natural Language Processing (EMNLP), pp. 1522-1531, 2014.
309
10


Reviews:
[{"review_id": "ryg1XGKOcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about learning word embeddings. In contrast to previous work (word2vec's Skipgram) where a word is represented by a fixed word and context vector, here each word is represented by a von Mises-Fisher distribution and a context vector. \nThe paper claims that the resulting embeddings are more interpretable, and that the inner product of two context vectors represents their point-wise mutual information.\n\nMy main concerns are the following:\n\n- Representing words by distributions is not a novel idea; it was previously done by Brazinskas et al. (2017): Embedding words as distributions with a bayesian skip-gram model (BSG). They however used Gaussian distributions and not von Mises-Fisher. BSG is acknowledged in the paper, but only small scale comparisons are performed (15 million) while the BSG paper uses a lot larger data sets. There is therefore not a meaningful comparison to the most relevant previous work.\n- Word similarity experiments are not enough to justify this approach. BSG at least showed the strength of using distributions to represent words by showing that different samples could constitute different word senses/meanings. There is no such analysis here.\n- The claim that the resulting representations are more \"interpretable\" is not backed up by any evidence at all, even though the word \"interpretable\" is in the title and the list of contributions.\n\nGenerally this paper could benefit from proof reading and editing.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rkxk7rD6tS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem the problem in word embeddings where \"word-word\" or \"context-context\" inner products are relied upon in practice when the embeddings are usually only optimized for good properties of \"word-context\" inner products. The new approach to training word embeddings addresses interpretability in the sense of having a theoretically grounded interpretation for the good properties that the inner products should possess -- namely that they capture the PMI between words. By changing the embedding training procedure (to include a step that samples vectors from each word's distribution representation) this interpretation is made possible without problematic theoretical assumptions. Now trained for the purpose they are intended to be used for, the approach yields strong SOTA results on the non-small challenge datasets.\n\nThe word is well exceptionally well motivated (we should directly optimize for the properties we want!) and situated in the literature. The opposition of Arora 2016 (getting at \"interpretable\" word embeddings with a latent variable model) and Mimno & Thompson 2017 (problematic assumptions in skip-gram embeddings) is particularly convincing. The vMF distribution is underexamined as tool for analyzing high-dimensional semantic vector distributions, and it is an excellent fit for the purposes of this project.\n\nTo back up claims of theoretical interpretability, a derivation (building on Ma 2017) proceeds without problematic leap thanks to a property introduced by a carefully selected (if straightforward) regularization term. This reviewer did not read the appendix (not sure where this would be located -- first time reviewing with OpenReview here), but the intuition doesn't seem to be much different than Ma.\n\nTo back up claims of applied performance, appropriate experiments are conducted on multiple evaluation datasets. The results seem to be fairly interpreted.\n\nThis reviewer decides to accept this paper for its balance of theoretical and empirical contributions and for the role it might play in reducing dependence on mysticisms in word embeddings (relying on accidental / uncharacterized properties of previous embedding strategies).\n\nSuggestions for improvement:\n- Run a spell checker. It will catch a large number of problems that weren't ever big enough to hurt my appreciation of the paper.\n- Consider the creation of an adversarial dataset (of ~3 words in ~3 contexts) where techniques that optimize for the wrong thing will catastrophically fail but the proposed approach succeeds.\n- Write one or two more sentences about the fate of \\kappa_c -- is it ever updated or am I just missing something? Making a bigger point about leaving \\kappa un-optimized shows there is room for additional depth in this line of thinking. (If you start learning concentration parameters, check out the Kent distribution for an even more expressive distribution on the unit hypersphere: https://en.wikipedia.org/wiki/Kent_distribution)\n- Figure out what happened to your appendix.\n- Watch out that readers/reviewers of similar work may try to read the word \"interpretable\" in the title as a reference to the much broader topic of interpretability in ML related to explaining a model's behavior. Is there a synonym for \"interpretable\" that won't raise this false link?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg0bMQ5YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposed a variational word embedding method, by using vMF distribution as prior and adding an entropy regularization term.\n\nStrengths:\n[+] In the experiment parts, the authors describe the experiment settings in detail.\n[+] Detailed descriptions for each equation is given.\n\nWeaknesses:\n[-] Template: It seems that the authors use the wrong template, which is not the template for ICLR2020.\n[-] Appendix: Although the author mentioned 'appendix' many times, I cannot see the appendix.\n[-] Motivation: The connection between motivation and proposed method seems weak. The authors argue 'interpretable' in their title and abstract, but their method and experiments do not show this point explicitly. \n\nQuestions\n[.] The experiments seem a little weak. The vocabulary size is 10K and the corpus is not so big, and I wonder whether the performance of the proposed method will be better for large corpus and longer training time.\n[.] For Equation 8, we should have a guarantee on the concentration of partition functions. Is it still true for vMF distribution? \n[.] What is the advantage of vMF distribution? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to learning interpretable word embeddings. The reviewers put this in the lower half of the submissions. One reason seems to be the size of the training corpora used in the experiments, as well as the limited number of experiments; another that the claim of interpretability seems over-stated. There's also a lack of comparison to related work. I also think it would be interesting to move beyond the standard benchmarks - and either use word embeddings downstream or learn word embeddings for multiple languages [you should do this, regardless] and use Procrustes analysis or the like to learn a mapping: A good embedding algorithm should induce more linearly alignable embedding spaces. 

NB: While the authors cite other work by these authors, [0] seems relevant, too. Other related work: [1-4]. 

[0] https://www.aclweb.org/anthology/Q15-1016.pdf
[1] https://www.aclweb.org/anthology/Q16-1020.pdf
[2] https://www.aclweb.org/anthology/W19-4329.pdf
[3] https://www.aclweb.org/anthology/D17-1198/
[4] https://www.aclweb.org/anthology/D15-1183.pdf
Decision: Reject