Word embedding re-examined: is the symmetrical factorization optimal?
Authors: Zhichao Han
Jia Li
Xu Li
Hong Cheng
Venue:  (2020)
Keywords: word embedding
matrix factorization
linear transformation
neighborhood structure

Abstract: As observed in previous works, many word embedding methods exhibit two interesting properties: (1) words having similar semantic meanings are embedded closely; (2) analogy structure exists in the embedding space, such that ''emph{Paris} is to \emph{France} as \emph{Berlin} is to \emph{Germany}''. We theoretically analyze the inner mechanism leading to these nice properties. Specifically, the embedding can be viewed as a linear transformation from the word-context co-occurrence space to the embedding space. We reveal how the relative distances between nodes change during this transforming process. Such linear transformation will result in these good properties. Based on the analysis, we also provide the answer to a question whether the symmetrical factorization (e.g., \texttt{word2vec}) is better than traditional SVD method. We propose a method to improve the embedding further. The experiments on real datasets verify our analysis.

Under review as a conference paper at ICLR 2020
WORD EMBEDDING RE-EXAMINED: THEORETICAL ANALYSIS ABOUT WORD SIMILARITY AND ANALOGY STRUCTURE
Anonymous authors Paper under double-blind review
ABSTRACT
As observed in previous works, many word embedding methods exhibit two in- teresting properties: (1) words having similar semantic meanings are embedded closely; (2) analogy structure exists in the embedding space, such that "Paris is to France as Berlin is to Germany". We theoretically analyze the inner mecha- nism leading to these nice properties. Specifically, the embedding can be viewed as low rank transformation from the word-context co-occurrence space to the embedding space. Such embedding transformation will preserve the relative dis- tances among words. Furthermore, previous studies empirically observed that the parameter a has a strong influence on the performance of word embedding but did not provide theoretical explanation. We provide theoretical explanation for this behavior, and derive a method to automatically find its optimal value. The experiments real datasets verify our analysis.
INTRODUCTION
Word embedding fundamental task in natural language processing. Many different approaches have been proposed, including LSA (Deerwester et al. 1990). SGNS (Mikolov et al., 2013a by, GloVe (Pennington et al., 2214) and others. These methods have achieved huge success in formation retrieval (Salton & Buckley 1988), entity recognition (Lample et al., 2016), sentiment analysis (Socher et al. 2013), machine translation (Sutskever et al. 2014) and so on. Previous studies have demonstrated three well-known facts about the word embedding: (1) Levy & Goldberg (2014b) showes many different kinds of word embeddings (c.g. SGNS) can be understood the matrix factorization framework. For example, Levy & Goldberg (014b) proves that SGNS implicitly factorizing the shifted pointwise mutual information (PMI) matrix. (2) Mikolov et al.
(2013)); [Pennington al.) (2014); Levy et al. (2015): Tian et al. (0016) and other works observe that word embedding exhibits two nice properties: the word similarity such that words with similar semantic meanings are embedded closely in the embedding space, and the analogy structure such that "woman is to queen as man is to king". (3) Previous studies, including (Caron 2001 Tumnyy
2012 Bullinaria & Levy 2012: Levy et al.) 2015 Artetxe et al., 2018), empirically observe that a in word embedding E = U. 2 has an important influence on the quality of word embedding. But these works did not provide theoretical explanation for this behavior of or practical guide to set this parameter.
In this paper, we show the word embedding can understood as a low rank transformation process. Furthermore, we investigate the change of the relative distances between words which is the inner mechanism that leads to these two nice properties in the word embedding. Particularly, we find the parameter a has influence on the change of relative distance between words. Based such analysis, we provide theoretical explanation the behavior of a and derive a method to automatically find its optimal value. To summarize, this research is motivated by questions:
1. What is the inner mechanism resulting in these two good properties of word embedding?
2. How does the parameter influence the word embedding? Can find method to deter- mine its optimal value?
We will explain the meaning of these variables (c.g., Ua) in the following sections.

---

Under review as a conference paper at ICLR 2020
We make the following contributions in this paper:
We reveal how the relative distances between words change during the embedding process which can be seen as a low rank transformation process. The nice properties of word embeddings are inherited from the original word-context co-occurrence matrix as the result of such low rank transformation.
Compared with previous studies which empirically observed the influence of the parameter a on the word embedding, theoretically explain the influence of parameter a the word embedding by revealing it controls the change of relative distances between words. Besides, we derive method to automatically find its optimal value. We conduct experi- ments on real datasets to verify our ideas.
The following sections are organized follows: section rattionn@inrooduccee the background knowledge and necessary preliminaries; section B shows how the relative distances between words change during this linear transformation and it the inner reason incurring these nice properties of word embed- ding. Based on such analysis, section 4 shows that symmetric factorization can be suboptimal and how to improve the word embedding model with asymmetric factorization. Section | contains the experiments on real datasets and section 6 discusses the related work. Finally, section ?? is the conclusion and describes a potential work.
BACKGROUND AND PRELIMINARIES
The goal of word embedding is to find good vector representation for every word in corpus. As summarized in Yin & Shen (2018), most existing word embedding methods can be formulated as low rank matrix approximations. So can analyze the word embedding algorithms (e.g., SGNS) within the matrix factorization framework. Suppose M E R is the word-context matrix where M. represents some statistics between word uy and its context word w.j These word embedding algorithms seeking a d-dimension word embedding matrix E Raxd accompanied with a context embedding matrix C E Rnxd to approximate the word-context matrix M, such that M & EC The matrix M can have different forms due to the design of objective function in every embedding method. For example, M is the shifted pointwise mutual information matrix in the SGNS.
Explicitly performing singular value decomposition. This kind of methods can be summarized as follows. Firstly, word-context matrix M is constructed to capture some co-occurrence statistics of every word and its context. For example, M can be the pointwise mutual information (PMI) matrix (Church & Hanks 1990), positive PMI (PPMI) matrix (Bullinaria & Levy 2012) and Shifted PPMI (SPPMI) matrix (Levy & Goldberg 2014b). Then, SVD used to factorize the word-context matrix M as M = U Σ. FiFallyy, only the top d singular values in Σ and the corresponding columns in U and V are kept, producing Σd, Ua and Va respectively. The word embedding E gaxd is obtained by E Ua E Ua Σg-
Caron 2001). Turnny (2012). [Bullinaria & Levy (2012). Levy et al. (2015) and Artetxe et al. (2018) discussed more general approach which adds a parameter a to the truncated diagonal matrix Dg and obtain the word embedding E = U2 Σg They observe a has an important influence on the embedding and suggest this parameter should be tuned. However, the reason is not theoretically clear and no clear method has been provided to tune yet.
Implicitly performing matrix factorization. Recently, word embedding methods based on neural networks have been proposed (c.g., Bengio et al. (2003)). Particularly, Mikolov posed a popular word embedding approach SGNS based on the skip-gram with negative-sampling and it achieves the state-of-the-ae results in different tasks. Besides, GloVe is another widely used word embedding method (Pennington et al., 2014). While SGNS and GloVe learn word em- beddings by optimizing some objective functions using stochastic gradient methods, it has been shown that these two methods are implicitly performing matrix factorizations. Specifically, Levy & Goldberg (2014b) showed SGNS is implicitly factorizing the shifted Pointwise Mutual Information matrix. Levy et al.) (2015) showed the GLoVe is implicitly factorizing the log-count word-context co-occurrence matrix. So these two neural network based embedding methods can also be formu-
Particularly,MMkkloov et al. (2013a b) pro-
2

---

Under review as a conference paper at ICLR 2020
lated within the matrix factorization framework, and the parameter equals to 0.5 as they doing symmetric factorization of some word-context co-occurrence matrix M.
To summarize, same the first assumption in Yin & Shen 2018), our analysis assumes the word embeddings can be formulated as low rank matrix approximations, either explicitly or implicitly. Specifically, the embedding matrix E equals to Ua- Σg where Ud. Eg and are as defined before.
ANALYSIS: THE REASON WHY THE EMBEDDING EXHIBITS NICE PROPERTIES
Our start point is the relation between word embedding and matrix factorization. As mentioned in section 2 almost all word embedding methods can be formulated as explicit or implicit matrix fac- torization (e.g., SGNS is implicitly doing symmetric factorization information matrix). We adopt the same notation in previous papers (Levy et al. 2015 Yin & Shen, 2018 write M UEVT as the SVD of some word-context co-occurrence matrix M. For example, M can be the pointwise mutual information matrix. Besides. Σn is the truncated diagonal matrix containing top singular values. Ua and Va are the corresponding truncated U and truncated V respectively. The d-dimension word embedding matrix E is obtained by multiplying Ua with power of Ed,
the shifted pointwise mutual
E Ua Σ
3.1 WORD EMBEDDING AS LOW RANK TRANSFORMATION
The embedding E in equation I is indeed low rank transformation from the original word-context co-occurrence space M. Let define another diagonal matrix E Epse (lag((),...,.00 Td, o,.... 0) whose first elements are the singular values in Σ and the remaining diagonal elements are zeros. We have the following observation.
Observation 1. E is the first d columns of E Epsc
E pse M V D(a-1)
The proof is in Appendix A.1
Observation reveals the process to compute E: (1) first multiplying the matrix V to the right side of M; (2) then multiplying the diagonal matrix E(hra) on the intermediate result MV; (3) finally removing the last d columns of M V D/sr (a-1)
pase The first step in this transformation process is a rotation because the matrix V is a unitary matrix. So the relative distances between words are not changed during the first transformation. In the second step, the matrix Syse is diagonal matrix and is multiplied on the right side of MV. So the second step scales every dimension the rotated matrix. Because we keep the first d dimensions in the final step, we only care about the scaling in the first d dimensions which are determined by the corresponding singular values with a power a.
By explictly expressing the process which transforms the word-context co-occurrence matrix M to the embedding matrix E, we can find fact that the relative distances between words only changes within the scaling step. Considering vectors of all words are scaled the same at every dimension, seems that the relative distances among words should be preserved some extent. In fact, we have the following two theorems:
Theorem 1. For any two words wi and wy, if M,, MM < 5 (6 0), then E Ez,0l12 S (a-1) d
Theorem 2. For any two words wi and wj, if Ei. Egall2 < 8 (8 2 0), then |M Miall2 S
01 (1-a) 8 + 2 2/2/
The formal proof of theorem and theorem 2 is in appendix A.2 and A.3
3

---

Under review as a conference paper at ICLR 2020
These two theorems describe how the relative distances (measured in Euclidean distance) among words change during the embedding transformation. Particularly, theorem T shows that if the distance between two words W; and W in M space is no more than 6, their distance in the emhedding space is also within a constraint. This bound is related with the singular value da and the parameter a. So theorem [ implies that the relative distances between words will not change dramatically during this transformation from M to E with suitable a. Theorem Theorem@lemmonsee demonstrates that if we observe two words are embedded closely in E Ei: Ej small), we can infer these two words are also close in the original M space. To sum up. the above two theorems indicate the neighborhood structure (relative distances to other words) for every word will be preserve some extent during the transformation from M to E. In other words, matrix E inherits the neighborhood structure that exists M. As the result, E will inherit the properties in M that are related with the relative distances among words. Now we can answer the question raised in the introduction: What the inner mechanism resulting in the good properties of word embedding? ? In fact, we will show that these properties are the result of inheritance of neighborhood structure.
3.2 THE INNER MECHANISM RESULTING IN THE NICE PROPERTIES IN THE EMBEDDING
Let us focus on two important properties in E: (1) words with similar semantic meanings are likely to be embedded closely in the embedding space; (2) the word embedding tends exhibit the analogy structure, e.g., "woman is to queen as man is to king". Note that these two properties are related with the relative distances among words. So, considering the previous analysis which shows matrix E' inherits the neighborhood structure existing in M, these two properties in E also inherits from M as the result.
Words with similar semantic meanings are embedded closely. Harris 1954) indicated that if two words have almost identical environments we say that they are synonyms... In other words, two words will have similar neighbors they have similar semantic meanings. It means the corresponding rows in the word-context co-occurrence matrix M should be close for these words. So the words with similar semantic meanings are also close in the M space which has been empirically verified in Levy et al. (015). Remember that theorem [ shows if the distance between two words in M is close (within a small range 8), these two words in E will also be close due to this low rank transformation process from M to E. So the low rank transformation in fact makes the embedding E inherit this property existing in the original word-context matrix M.
The analogy structure in the embedding. Different from word similarity which is about the relative distance between two words (e.g., king and queen), the analogy structure related with the relative distance between two pairs (e.g.. (king. queen) and (man, woman)). So let us first examine how the relative distance between two pairs change during the embedding process.
Theorem 3. Let Mn.:. M M and Mæ.; be four rows in M matrix. Denote M, (M,.: - M and M3 = (M - M If ||,, - MB.ll2 S 8 (6 2 0), then
EE, - Ep..l2 S o (1-
8.
be four rows M matrix. Denote M, (M My2s) and Mg (M..: M. If EI - Es. 5 (8 2 0), then
Theorem 4. Let MM MAn MA.: and M
||M,. - MB..ll S (1-a) 8 4 4(E 22) 0221/2
The proof of theorem 3 and theorem 4 is in appendix proof of theorem and theorem 2
A.4 and
A.5 which is quite similar to the
Theorems show that the relative distance between two pairs will not change dramatically dur- ing the embedding process either. In other words, the relative distance between pairs of words will be preserved during the embedding transformation from M E. Furthermore, Levy & Goldberg (2144) revealed a fact that the original word-context co-occurrence M (e.g., M can be the PPMI
matrix also exhibits the analogy structure, such that Maing - M Mqueen & Mman M So
Thhe word-context co-occurrence matrix M is called explicit representation in Levy Goldberg 2014a
4

---

Under review as a conference paper at ICLR 2020
coldestesst
winters Spring
precipitation
temperaturShonson shonsoon soonn
temperaturEsuprmest romees freezing
Jhumldity winters
snowfall .rainkindy
coldest weekend BURNY
winters windy dim Rumigainy
seasons year
precipitation
Summers
weekend nights
snowfall ,seasons
temperatures warmestt
monsoonights
summer
wirter autumn
days yeaar
days dty"humidttwwinn .summe e -wndy
Sur
winter
.rainy
spring
(a) 0.0
(b) 0.5
(c) 1.0
Figure 1: Illustration of the local neighborhood structure in the embedding space E = Us Σ with different From (a) (c): t-SNE (Maaten Hinton 2008) visualization of the nearest 20 neighbors of an arbitrarily chosen word summer with a equals to 0.0, 0.5 and 1.0 respectively. More results corresponding other a can be founded in appendix A.9
theorems 3 and 4 together with this fact indicate the analogy structure in E also inherits from M as the result of distance preserving in such low rank transformation process.
To summarize, we prove the bound about the change of relative distances of words during the embed- ding process. With appropriate a, the neighborhood structure (relative distance) of every word will not change much during the embedding process. So the embedding E inherites the word similarity and the analogy structure from M. We also theoretically explain how the parameter influences the word embedding by controling the change of relative distances among words (the bounds in theorem
So consideration that we can optimize these bounds by choosing suitable a. We show the derivation of finding the suitable a with respect to the bounds in appendix A.7 It shows the suitable depends on σ₁ and dd, together with how weight the two bounds in theorem [ (3) and theorem 2 However, the optimization over the bound only gives rough estimation of the possible range of optimal a. the next section, we optimize a by directly minimizing the change neighborhood of words instead of optimizing over the bound.
4 IMPROVEMENT: WORD EMBEDDING PRESERVING THE DISTANCE STRUCTURE BETTER
In previous section, reveal the inner mechanism why the embedding exhibits such nice prop- erties. Namely, the embedding is in fact a low rank transformation from M to E and the relative distances between words are preserved during such process. As result, the embedding matrix E inherits these nice properties existing in the original word-context co-occurrence matrix M (e.g.. the PMI matrix). Particularly, theorem influence on how the relative distances between words change. For example, we arbitrarily select word summer in the corpus and plot its nearest 20 neighbors with equal to 0.0, 0.5 and 1.0 respectively. Figure [ shows that the neighborhood of summer changes with different This ob- servation gives us an inspiration that we can learn the embedding a, thus preserving the relative distances among words during the embedding transformation process.
and 4 show that the parameter has important
Ua Σg by selecting good
Preserving neighborhood structure better by choosing optimal for embedding E = U. Eg. Let gi (w) represent the neighbors of word w in A' space, e.g., X' can be the word-context co- occurrence matrix M the embedding matrix E. Let I(...) be a correlation function measuring the similarity of two different neighborhoods. This measurement captures how much the neighborhood structure changes incurred by the embedding process. For example, we consider the top knm nearest neighbors as (-) and ignore the order of these nearest neighbors. g (w) can be viewed as a set and can a similarity measurement defined on sets (c.g., Jaccard Similarity). Otherwise, consider the order of neighbors, x (w) can be a ranking list such that the neighbors ranked
We will introduce the corpus in section
5

---

Under review as a conference paper at ICLR 2020
by their relative distance w, and I now can be measurement about the correlation between two rankings. We use to denote the aggregated similarity score over all words
1 Σ Mg (w), (w))
I. L=
WEW
where W the word vocabulary consider and its size n. Based on previous analysis, we argue we can select an optimal a for embedding E' = -Ua E to preserve the relative distances among words during the embedding transformation, such that
arg max
Σ NEW
(4)
Leveraging the effect of a into the original SGNS network structure. The above analysis is based on the view that we compute the embedding matrix E by SVD such that E = -Ug- Eg. We can also think about to incorporate the effect of a into the original SGNS neural network structure. The orig- inal SGNS implicitly performing a symmetric factorization, thus implying the equal to 0.5. Here, we consider to leverage the influence of a directly into the SGNS structure. Consider the embedding Tog([le:11ã)
== U 2g and the context C >0-=)) Ví learnt by SVD, we have log([le; 11
where e; and C; are the word vector wy and the context vector of w; respectively. We put it as regularization term in the original SGNS objective function add the influence of different the embedding learnt by SGNS. The detailed derivation is in appendix A.6
To summarize, the a parameter in the embedding matrix E Ua Σ3 influences how the relative distances between words change during the embedding process, thus influencing the quality of the learnt embedding matrix. We propose method to choose the optimal by preserving the relative distance among words during the embedding transformation. Besides, we come up with an idea to incorporate the effect of a into the original SGNS architecture. the next section, we will test these ideas real experiments.
EXPERIMENTS
In this section, we conduct experiments to verify our previous analysis. The first test is verify that a good embedding does maximize the correlation defined in equation 3 which measures to which extent the neighborhood structure is preserved during the embedding transformation. The second is to test if the proposed method to incorporate into original SGNS architecture really has improvements.
5.1 EXPERIMENT SETTINGS
Corpus We use the Text9 corpus (Mahoney 2011] which is standard benchmark used for various natural language tasks. We follow the same pre-processing steps in Levy et al. (2015).
Word similarity task The purpose of the experiments is verify whether the method proposed in section to find the optimal a matches the real case. We adapt the same experiment as Levy the word similarity and analogy structure. There six
al. (2015) to test the word embedding
datasets acting as the ground truth for the word similarity: WordSim353 (Finkelstein et al.) 2002) and its two subsets WS Similarity and WS Relatedness (Zesch et al.) 2008 Agire al. 2009
Bruni MEN (Bruni et al. 2012). Radinsky Mechanical Turk (Radinsky et al.) 2011) and Luong Rare Words (Luong al. 2013). These dataset contaii word pairs together with human-assigned similarity scores. Same as Levy et al. (2015). word embedding is evaluated by the correlation between the similarity computed by the embedding vector and the numan-assignee similarity score. The function to compute correlation is the Spearman's 's correlation (same as Levy et al. (2015) Note
"Suppose we only consider words for which ||e; 1| ||cll * 1, to avoid case that zero appears in denominator.
These datasets be downloaded from

---

Under review as a conference paper at ICLR 2020
that discard the pairs in the test file if it contains word that does not appear in the tex19 corpus. We use the word similarity task as an example for verifying the proposed method (equation 3 and equation 4 to learn the optimal for word embedding E Ua Eg. and the proposed method to improve SGNS.
The codes used for the SVD embedding is adapted from https://b.tbucket org/ onerleey/hyperworee
and the codes used to learn the SGNS embedding is adapted from https: /ttl/ggtttt gigttb...u.ug...g http://hhhwwwww//eeeeeeee box folder for the double-blind review.
sgns
We store the codes in shared drop-
brunt
0.51
Loong
rarfosky
correlation score
0.76
0.75
correltton score
0.630-
0.79-
0.49
0.820
0.575
0.600-
0.70
0.46
0.48
0.45
0.500
1.0-0.5-06-0.4--22
0.4 0.6 0.5 1.0
parameter a
parameter
parameter a
(a) bruni men testset
(b) Luong testset
radinsky testset
0.70
uS relatedness
0.81
similantty
0.75
Ms:353
correlation score
0..0
carrelation score
0.80
0.79-
0.72
0.64
0.62
0.60
0.60
1.0-0.00..00......0000 0'2
1.0
10-00.--0-004-002
1.0-08-0.-0044-022000 0.2
parameter a
parameter
parameter Oi
(d) relatedness testset
(c) similarity testset
ws353 testset
Figure 2: Compare the best a (black vertical line) in terms of the performance on the word similarity task and the best a* (blue vertical line) in terms of equation 4 o' is searched from [-1, with 0.1 as increment. From (a) to (e): y-axis is the correlation between the embedding score and the human assignend score. x-axis is different a.
Table 1: The performance and similarity corresponding to a = 0 (SVD embedding) and a = 0.5 (symmetric embedding). Left is the peformance on every testset and right is the similarity computed by equation 3 considering the words appearing in that testset.
bruni men
Luong
radinsky
ws relatedness
similarity
Wx353
0.746
0.492
0.507 0.469
0.588 0,507
0.667 00552
0.786 0.533
0.737 / 0.549
0.5
0.759 0.5188
0,509 0.475
0.666 / 0..25
0.701 / /0.573
0.812 0.555
0.750
5.2
RESULTS
Firstly, let us verify that if the embedding E = Ua Σg is good (measured by the performance on the word similarity testsets), the embedding E should preserve the relative distances between words better (measured by equation 3}. We the "vanilla" setting in Levy al. (2015) to construct the positive PMI (PPMI) matrix as M: window size is 2, no dynamic context window and no subsampling. the number of negative samples is and the context distribution smoothing parameter 1.0. For equation 3 we only consider the nearest neighbor for g and use Jaccard Similarity as the I(..)). Note that each testset only contains subset of words in tex19, and also compute the overall correlation (the L in equation considering the words in the corresponding testset. The
TThe drophox MANUe//FUpxUcBuggFF1gg
link is
httpp::/wwwddrppoo..coo// Aptheesttto/t

---

Under review as a conference paper at ICLR 2020
result is in figure [ Figure shows that the performance (measured by these six word similarity groundtruth) and the proposed measurement are highly correlated, such that, for every testset, achieves high score word similarity test, the corresponding measurement in equation is also high. More details can be found in appendix [A_8] Besides we test whether the a* in equation ! really corresponds
that achieves the best performance the word similarity task. We plot the performance of E = U- Σ with different a in figure The black vertical line corresponds to a that achieves best accuracy and the blue vertical line corresponds the a learnt by equation We can see that o" is very near to the optimal point in almost all datasets. The only exception is the Luong testset, which contains many rare words. We infer the reason is that the corpus cannot effectively capture this testset because more than two thirds words in Luong testsets are not in the tex19 corpus.
Table 2: The results of incorporating to the original SGNS.
bruni men
Luong
radinsky
ws relatedness
similarity
ws353
0.3
O. 0,70680
0.45004
0.67119
0,63262 0.63733
0.77532
0.71247
0.5
0.
0.43613
0.67177
0.77679
0.71309
Secondly, we test the effect that incorporating a into SCNS. Table 2 the performance of the original SGNS with a equals to 0.3 and 0.5 According to the result (the gold line) in figure 2 if we push the a from 0.5 to 0.3, the performance in the Luong testset should become better and the performance in all other testsets should drop. The results in table i same with this inference. The only exception the bruni men testset. We infer the reason is that the performances with a = 0.3 and = 0.5 are very close (see gold line in figure 2 (a)).
6 RELATED WORK
Explanation about the analogy structure. Mikolov et al. (2013c) observed the analogy structure existing in the embedding created by neural network. Levy & Goldberg (014a) afterwards found the same property also exists in the original word-context co-occurrence matrix. Levy & Goldberg (2014a) infers neural embedding process not discovering novel patterns, but ruther is doing a remarkable job al preserving the pullerns inherent in the wonl-context co-occurrence matrix. However, it is not clear how the neural embedding process preserves such property. Later, several explanations were been proposed to explain the reason (Arora et al.) 20166) Gittens et al., 2017 Allen & Hospedales 2019 Ethayarajh al. 2018). For example, Gittens et al. (2017) and Allen & Hospedales (2019) referred to the paraphrasing to explain the analogy structure. Different from these previous studies, we analyze word embedding from the low rank transformation perspective and we reveal the inner mechanism how this property is inherited during the embedding transformation process.
Explanation about the a. The influence of a on the quality of word embedding has been observed in (Caron] 2001 Turney 2012; Bullinaria & Levy| 2012; Levy al. 2015 Artetxe et al. 2018). These works empirically find that a has an important influence to the embedding and suggest this parameter should be tuned. However, these works do not provide theoretical explanation and no clear method has been provided to tune the a. YYn & Shen
(0188) is the only work we know which discusses the meaning of from the pairwise inner product (PIP) loss perspective. But their findings have some contradiction to the real experiments in Caron (200T) Turney (2012), Bullinaria & Levy (2012), Levy al. (2015) and Artetxe et al. (2018). Specifically. the analysis in Yin & Shen (2018) suggests small a is casier to result in over- fittingting and lead to the performace drop, which implies that larger a is better. However, the real experiments in other papers show the performance also drops with very large a. To summarize, we explain the meaning of a in the word embedding transformation which once was not theoretically clear. Futhermore, we propose method to find the best for word embedding.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. A study on similarity and relatedness using distributional and wordnet-based approaches. In Pro- ceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 19-27 Association for Computa- tional Linguistics, 2009.
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embed- dings. In Proceedings of the 36th International Conference on Machine Learning, pp. 223-231.
PMLR, 2019.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
Mikel Artetxe, Gorka Labaka, lñigo Lopez-Gazpio, and Eneko Agirre. Uncovering divergent linguistic information word embeddings with lessons for intrinsic and extrinsic evaluation. In Proceedings of the 22nd Conference Computational Natural Language Learning, pp. 282-291, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v 10K/0//////K1--// 1028. httpttss:: www www.wb.w.........ccc........ww.w argganttologyy -
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. neural probabilistic language model. Journal of machine learning research, 3(Feb): 137-1155, 2003.
Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. Distributional semantics in tech- nicolor. In Proceedings of the 50th Annual Meeting of the Association for Computational Lin- guistics: Long Papers-Volume Volume pp. 136-145. Association for Computational Linguistics, 2012.
John A Bullinaria and Joseph P Levy. Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd. Behavior research methods, 44(3):890-905 9000099777700000000 2012.
John Caron. Experiments with LSA scoring: Optimal rank and basis. In Proceedings of the SIAM Computational Information Retrieval Workshop, pp. 157-169, 2001.
Kenneth Ward Church and Patrick Hanks. Word association norms, mutual information, and lexi-
cography. Computational linguistics, 16(1):22-29, 1990.
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-49) 407, 1990.
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analo- gies. arXiv preprint arXiv: 18/0.04882, 2018
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search context: The concept revisited. ACM Transactions on informa- tion systems, 20(1):116-13)) 2002.
Alex Gittens, Dimitris Achlioptas, and Michael W Mahoney. Skip-gram- zipf+ uniform= vector additivity. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 69-76, 2017.
Zellig S Harris. Distributional structure. Word, 10(2-3): 146-162, 1954.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 260-270, 2016.
Omer Levy and Yoav Goldberg. Linguistic regularities in sparse and explicit word representations. In Proceedings of the eighteenth conference on computational natural language learning. pp- 171 171-180, 2014a.

---

Under review as a conference paper at ICLR 2020
Omer Levy and Yoav Goldberg. Neural word embedding implicit matrix factorization. In Ad- vances in neural information processing systems, pp. 2177-2185, 2014b.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:21 1- 225, 2015.
Thang Luong, Richard Socher, and Christopher Manning. Better word representations with recursive neural networks for morphology. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 104-113, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9Novv: 2008.
Matt Mahoney. Large text compression benchmark. URL: http://www. mattmahoney. net/text/text. html, 2011.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 1301.3788 30.3a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119, 2013b.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751,
2013c.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. A word time: computing word relatedness using temporal semantic analysis. In Proceedings of the 20th inter- national conference on World wide web, pp. 337-346. ACM, 2011.
Gerard Salton and Christopher Buckley. werih-igg approaches in automatic text retrieval. Information processing & management, 24(5):513-523, 1988.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Ilya Sutskever, Oriol Vinyals, and Quoc V Lc. Sequence to sequence learning with neural networks. In Advances neural information processing systems, pp. 3104-3112, 2014.
Fei Tian, Bin Gao, En-Hong Chen, and Tie- Yan Liu. Learning better word embedding by asymmet- ric low-rank projection of knowledge graph. Journal of Computer Science and Technology, 31 (3):624-634, 2016.
Peter D Turney. Domain and function: A dual-space model of semantic relations and compositions.
Journal of Artificial Intelligence Research, 44:533-585, 2012.
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. In Advances in Neural Information Processing Systems, pp. 887-898, 2018.
Torsten Zesch, Christof Müller, and Iryna Gurevych. Using wiktionary for computing semantic relatedness. In AAAI, volume pp. 861-866, 2008
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 PROOF OF OBSERVATION
T
Proof. Firstly, have
E =M 2
σ1
U U.,,..,, Und
od
0i U. 00.1002 o0 U U.g.....
Und
(a)
Besides,
E pase =M V E(a-1)
= E V
V. (a-11)
pse
=U Σ I Σ (a-1)
(V is unitary matrix, V V == n
pse
= Σ (a-1)
pse
(a-1)
01
1)
=IU.,, U.2,...... U,,]]
dd+1
01
(a)
-IU.,, U.2, U.n]
d
=10 (00 ( U2 U....... o 1n) Und 0, 0.....00)
So E corresponds to the first d columns of Epse.
A.2 THE PROOF OF THEOREM I
We need the following lemma for the proof of theorem T
Lemma 1.
Unitary matrix does not change the norm of a vector, such that
| = ||a||2
where is unitary matrix and a E IR." is a n-dimension column vector.
Proof. The norm be expressed by using the Hermitian product, (: [a (Ua,Ua)) (a, U'Va) (a,a) ||a||
Ual|2 ||a||
Now we can prove the theorem 1
For any two words wi and Wy, if M M3alla s 6, then
Er, Egoll2 < (0) (a-1) (010-1)
x 6.

---

Under review as a conference paper at ICLR 2020
Proof. Let E denote the Epse defined in section It is obvious that | E3.111
E.: Esi since E is the first d columns of E and the last (n d) dimensions in E are zero.
Let h (/1, hn) denote the (M, M, We have |/2 II V
According to lemma [ Mr.)V V1l (M: - M2 So we have hl]2
[I,M, V1l2 M..- M M2
Bi, Ej,112
E.. Ej:
M V. 1)0-1)
M V
Σ
1(a-1)
pse
2
(M.. Mí: V)
{a-1))
pec
h I(a-1)
2
=( =(o= ((0(0-1)/ h1)² + (02 h22 +... (0 (a))-1)aa)2++ + +...... ... + 0
(a-1)
200(11)) (h² h2 ... h22)
(note au the smallest singular value)
2(0 ((-11)
|1| ?) |
1/2 S d
(a 1) 6
A.3 THE PROOF OF THEOREM
Proof. Let Mpse be UEpse VT. It is easy to show that Mpsc equals to Ua Eg Vf-
2
Let M denote Mn pase We first show that M
M.
(for every i) is constrained by the singular
values of M. Firstly, we have
M M 1|
(E (Mes 23,)3)2/2
Σ
(Mk. M2,)) M M
So we have
M Myi
<
M M
2
UEVT -UE pse V
2
UUE -Epse
= -IEE Epse)ll2 (lemma 1
= =0 + +02+2 +... +..) 2221/2
Then we show if E. Ey. 12, < 6, then | - MM,1|
-M.
s _00 [0o(--))
M M1,111
(Er. E,,) (01 a1 V2J
2
[E,. - EE,) (11-a)
(lemma)
12

---

Under review as a conference paper at ICLR 2020
Let h (h1, ...... hn) denote B Ey:; here. We have ||h||| above equation becomes
E
Ej;il12 6. So the
BL. Eji)) 2/ (1-m)
(h1, h2 ................................ ,,.), E
(A/001 2(1-a) + 12002 2(1-0) +... h300 22/0))
<(1n + 113 +....
+ h2) (01 /1/
$ ||h||2 of (1-a)
_ō0 (1-a)
With above preparations, we can now prove theorem
M - M3all2
(M M..) - (M, -M..) 1/
(M. M.
+
(M,; - -M... 112
(M.. M.
(M, My.)) - (M.. M3:3)11
2
(M. M.
(M, MM.)) / + -M Mi.)]
S(Et=a+1 02)1/2
S(Eieati 72)//2
560
<ōo (1-0) (1-a) +
Σ
0221/2
1=d+1
A.4 THE PROOF OF THEOREM
Proof. Similar the proof in section
A.2 let E denote the Ense and h
hi,,,i.... by denote the ((M My
(Ma Ma We have ||/1|12
II((M, M...)) - (Ma.: M VVl, = |I M - (Ma.. - M& S 6.
Besides, [[(E,,: Emp..) - (Es,. EB,.))12 (Eyy.: Emp.)) (E EBs.)) since E
the first d columns of É and the last (n d) dimensions in É are zero.
(E Exa:) - (Es,. EB2:) l12
(Éri:: Eyai) (Eg..: Ess::)
(M My..) .V. I(a-1) (Ma,; M. V (a-1)
pse
psc
2
(( -M - (Mg. - Mg
((a-1)
pse
h x(-=
12
-(01 (a-1) /11)² + (a2 (a-1)
h22² +... + (o (a-1))
ha)² + +0+ +.....+0)?**
<od (a-1) (h₂ h +... 1/2 (note ag the smallest singular value)
(a-1). [11l] I 0
(a-1) 5
<o²
13

---

Under review as a conference paper at ICLR 2020
A.5 THE PROOF OF THEOREM 4
The proof is quite similar to the proof in section A.3 Let Mpsc be U. VT. According to section
we know Mpse equals to Ua Ed Va-
Let M denote Mpse According to section A.3 //// (for every i).
we have M - Min S (02+1 + +o2+2 +...
Then with the same process section A.3 we show [| - Emgi)) - (E - EBaiill2 S 6,
then (M,, - My2.)) - (Ms... Mon.)) S (0(=-a)
(Mn. Myni) (Ma MBa.))
((ET:: Ex2.) - (EB Ege.)) 2 x1(-a) V
2
(ETT.: - Erai)) - (EBn. - EBgn)) Σ/ (1]-a) (lemma
Let h = (2,,12.... ...., .m)) denote (BE E - (E E here.
We have ||
[I( (E, Ex2.)= (Ea.. Eggn))ll S So the above equation becomes
(Bn,:: - Bxgn)) - (Es.. - Eg2.)) 1/ (1
101,112 ..., hn) 211-a11 12
(Af02(2 2(1-a) + 11202 aa) +...
+ 12
2(1 a))-)))111 1/22
<(G + h1 +...
+ 1/3 01 (1-a)
||h||, 01 (1-a)
0o0 (l-a)
With above preparations, we can now prove theorem
Proof.
[M, Ma.
(M M...) - (Ma, - NM,
(M.. My.))
+ (MMp - My.) 1111
2
MM My,))
+
\Mss My.) - (M, M.. 112
<
M M
+
|M Ma || 2
+ (MT - Nd.-) 1111
(M. M Mn
(M... My ||, (Ma.. Mon.) (Maz. Mo..) >11 + -MM-((M,. - -M.))l
<
(M., MM.,)
+ (M.. Myar)
+ (Ma MAan)) ||, + (M MMar) ||, - Ma (ha(1-N)
S(ET= 02)1/2
S(EP= 02/122
SCER 02)1/2
SiER 07)1/2
=001- (1-0) + 1( Σ 2221/2
k=d+1
14

---

Under review as a conference paper at ICLR 2020
A.6
THE PROOF OF ALPHA REGULARIZATION
Proof. Firstly, we have
E U D
= 11 X U2
X u.
C
= 1-0 Vd
A Fa
Since U and V orthogonal, we have
1220 i=
e
(0
it
{ { 20
i i=j
cf
0,
i i
Then, if (1le:12. [lc:122 + 1, we have
log( e,; 12)
log(Ile:13 [c:112)
A.7 THE OPTIMAL a IN TERMS OF MINIMIZING THE WEIGHTED SUMMATION OF THE BOUNDS IN THEOREM 2ND
We assume that the optimal should minimize the outcome bounds in both Theorem I (B1) and 2 (B2) The reason is that two words are close in the original M space, these two words should also be close in the E space because do not want the relative distances of words change dramatically, and vice versa. Thus we derive the optimal a by minimizing the weighted summation of the bounds
B =B + (1 A)B2
(a-1) d
6, a 5
B
(a-1)) 6, a21
(1-a) 6,
a
B2
(1-a) a21
where À is the weight coefficient in the summation.
We derive the optimal a in three cases, i.c. Case 1: 01 2 08 1, Case 01 2 1 2 od. Case 3:
aa I $1.
For Case 1, when a 2 1, we have
B = - 8 + (1 -)0
0B =100f-11n In - (1 2) 1 In
da
Easy to prove that B convex function, so we derive a by solving the equation simplicity, we use k: =
de 0, for
1-2 x
0B
0 \o0 Ino, = 2 In
da
In klasa Inds (((0)))
a
TThe optimal with respect to theorem and theorem 4 can be derived in the way.
15

---

Under review as a conference paper at ICLR 2020
Similarly, when I 1, we have
B 2 100----4 8 +1+11 (1- - 0B =0=a=14
In kl lndd In (1
da
Then we have the optimal a in terms of k as
kindd
In Inor 2 1,
k>
(d1ed)
In
a
1,
In
In < S
In Ind
Aln
$ 1, k In
(ajoa)
ln
For Case 2, from the formulation of B, easily derive the optimal is 1.
For Case 3, similar with the derivation of Case we have the optimal a in terms of as,
In di (110a)
Ino. lno
S1 1,
k
1,
In
< k
S
lno
lu
lnd
In klaa
1 (o100)
2 1,
k
lnoj
ln
A.8 PERFORMANCE ON WORD SIMILARITY TASK AND THE CORRELATION SCORE DEFINED IN EQUATION WITH DIFFERENT
Table 3: The performance and similarity corresponding to different a. Left is the peformance on every testset and right the similarity computed by equation considering the words appearing in that testset.
bruni
Luong
radinsky
relatedness
similarity
ws353
1.0
0.686 0.492
0.480 / 0.457
0.504 / 0.487
0.591 0.532
0.759 0.522
1.69470..55
0.99
0.691 / 0.490
0.482 / 0.458
0.508 / 70.495
0,597 / 0.532
0.761 / 0.522
0.697 / 0.535
0.8
0.697 / 0.4499
0.484 / 0.459
0.513 0.493
0.605 0.538
0.762 / 0.526
0.701 / 0.539
0.7
0.703 / 0.490
0.487 / 0.460
0.520 0.5001
0.612 0,5538
0.765 0,526
0.706 / 0.539
0.6
0.709 / 0.488
0.491 / 004422
0.526 / 0..99
0.621 / 0.538
0.765 / 0.522
0.711 / 0.537
0.5
0.715
0.488
0.493 / 0.462
0.534 0.499
0.63070.55
0.766 / 0.518
0.715 7 0.535
0.4
0.721
0.488
0.496 / 0.462
0.540 / 0,501
0.637 0.544
0.769 7
/
0.3
0.728
/ 0.489
0.499 / 0.460
0.551 / 0.503
0.645 / 0.544
0.774 0.522
0.724 7 0.539
0.2
0.734
0.492
0,502 / 0.467
0.562 0.509
0.651 0.547
0.779
0.529
0.729/0.33
-0.1
0.740
0.489
0.504 0.466
0.576 / 0.509
0,659 0259/0.55 0.549
0.782 0,533
4.333/05546
0.0
0.746 / 0.492
0.507 / 0.469
0.588 / 0.507
0.667 / 7.5522
0.786 / 0.533
0.737 / 0.549
0.1
0.751
0.497
0,509
0.469
0.601 0.509
0.676 / 0.558
0.791 / 0.536
0.742 / 0.553
0.2
0.755 / 0,504
0.510/0.477
0.618 / 0.513
0.685 / 00.558
0.796 / 0.540
0.747 / 0.553
0.3
0.758
0.520
0.512/00 0.447
0.635 7 0.521
0.693 70.570
0.802 0.802/0.555 0.555
0.7507 / 0.563
0.4
0.759
0.519
0,511 0.475
0.652 0.525
0.698 / 0.570
0.807 0.555
0.751 / 0.565
0.5
0.759/
0.518
0.509 / 0.475
0.666 / 0.525
0.701 0.5533
0.8127 0.558
0.750/ 0.569
0.6
0.756
0.515
0.504 / 0.478
0.675
0.517
0.700 / 0.576
0.816 / 0.558
0.748 / 0.572
0.7
0.750
0.518
0.496 / 0.481
0.675 0.517
0.693 / 0.567
0.813 / 0.562
0.741 0.565
0.8
0.740/
0.527
0.485 / 0.472
0.672 0.511
0.677 70.576
0.805 7 0.573
0.728 / 0.572
0.9
0.726 0.516
0.468 / 0.465
0.663 / 0.517
0.649 / 0.567
0.794 / 0.555
0.708 / 0.558
1.0
0.707 / 0.505
0.448 / 0.456
0.649 0.513
0.619 / 0.561
0.780/ 0.547
0.687 0.556
16

---

Under review as a conference paper at ICLR 2020
A.9
THE NEIGHBORHOOD OF summer WITH DIFFERENT
top 20 nearest words
winter, olympics, autumn, during, and, spring, annual, season, in, at night. year, on, weather, after, rainy, day, including, festival, seasonal
PMI
a
top 20 nearest words
winter, autumn, spring, rainy, monsoon, summers, winters, warmest, nights, coldest seasons. precipitation, temperatures, humidity, year, snowfall. freezing, months, average, climates winter, autumn, spring, rainy, monsoon, summers, winters, warmest, nights, coldest scasons, precipitation, lemperatures.uumidie
-1.0
-0.9
snowfall, year, freezing, months, average, climates winter, autumn, spring, rainy, monsoon, summers, winters, warmest, nights, coldest seasons, precipitation, snowfall, temperatures, humidity, year, freezing, climates, months, average winter, autumn, spring, rainy, summers, monsoon, winters, warmest, nights, coldest seasons, precipitation, snowfall, humidity, temperatures, year, freezing, climates, humid, months
-0.8
-0.7
winter, autumn, spring, rainy, summers, monsoon, winters, warmest, nights, coldest snowfall, precipitation, seasons, humidity, temperatures, year, freezing, climates, humid, months
-0.6
winter, autumn, rainy, spring, summers, monsoon, winters, warmest, nights, coldest snowfall, precipitation, seasons, humidity, temperatures, year, freezing, climates, humid, months
-0.5
winter, autumn, rainy, spring, summers, monsoon, winters, warmest, nights, coldest snowfall. precipitation, humidity, seasons, temperatures, year, freezing, climates, humid, months
-0.4
winter, autumn, rainy, spring, summers, monsoon, winters, nights, warmest, coldest snowfall, precipitation, humidity, temperatures, seasons, year, climates, freezing, humid, windy winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, humidity, temperatures, seasons, year, climates, humid, freezingg windy winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, humidity, temperatures, seasons, climates, year, humid, freezing, windy winter, autumm, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall. precipitation, humidity, temperatures, seasons, climates, humid, year, windy, freezing winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, temperatures, humidity, seasons, year, humid, climates, windy, freezing
-0.3
-0.2
-0.1
0.0
0.1
winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, coldest snowfall, precipitation, temperatures, humidity, seasons, year, humid, climates, windy, sunny winter, autumn, rainy, spring, summers, winters, monsoon, nights, warmest, snowfall coldest, precipitation, temperatures, seasons, humidity, year, humid, windy, climates, sunny winter, autumn, rainy, spring, summers, winters, nights, monsoon, snowfall, warmest coldest, precipitation, seasons, temperatures, year, humidity, windy, humid, climates, days winter, autumn, rainy, spring, summers, nights, winters, monsoon, snowfall, warmest coldest, seasons, precipitation, year, days, temperatures, weekend, windy, sunny, humidity winter, autumn, rainy, spring, nights, summers, winters, monsoon, snowfall, warmest coldest, seasons, year, days, weekend, precipitation, annual, sunny, temperatures, month winter, autumn, rainy, spring, nights, summers, winters, monsoon, snowfall, warmest days. coldest, year, seasons, weekend, annual, week, sunny, month, snow
0.2
0.3
0.4
0.5
0.6
0.7
winter, autumn, spring, rainy, nights, winters, summers, days, monsoon, year weekend, seasons, snowfall, annual, week, night, warmest, coldest, snow, month winter, autumn, spring, nights, rainy, days, winters, summers, night, year weekend, annual, week, seasons, monsoon, snow, holiday, rain, month, day winter, autumn, spring, nights, rainy, days, night, week, annual, year weekend, seasons, winters, snow, summers, day, holiday, rain, month, weather
0.8
0.9
1.0
17


Reviews:
[{"review_id": "SJx5aT16tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n=======\nThis paper provides a closer look at the well-studied problem of learning word embeddings. In particular, it looks at the set of embedding methods that explicitly or implicitly perform a matrix factorization and tries to understand why the word embeddings exhibit analogy structure and why words that are semantically similar get embedded close together. The mechanism it comes up with has to do with the alpha parameter that represents the powers of singular values of the matrix that was factorized to estimate the embeddings. It turns out that alpha controls the distance between the words in the embedding transformation process. Next the paper discusses how to choose/estimate alpha to get better quality embeddings. Results are shown on several word similarity tasks. \n\n\nComments:\n=======\nThe paper offers fresh insights into the well studied problem of learning word embeddings. The impact of the alpha parameter is definitely interesting w.r.t the quality of embeddings learned. That said, the paper does have a few problems. First, though the paper is well motivated and puts itself nicely in context of previous work, it needs a copy-editor as there are many language/grammar issues some of which I highlight below. \n\n\nSecond, and the main problem with the paper, is that the properties of the alpha parameter are intriguing but the experimental evaluation is underwhelming. The paper also needs to show the impact of the alpha parameter on the quality of embeddings learned for some downstream task e.g. NER, POS Tagging. Just showing results on word similarity tasks and computing correlations is not very insightful or useful. \n\n\nGrammar issues (subset):\n\nPage 1: \"Word embedding is a very important task\"\n\nPage 1 : \"...which value should alpha be?\"\n\nPage 1: \"..has an important influence to the...\"\n\nPage 6: \"The first is to verify....\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgNUbOiFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the role of the implicit alpha parameter when learning word embeddings. More concretely, word embeddings work by either implicitly or explicitly factorizing a co-occurrence matrix, and the underlying parameter alpha controls how the singular values are weighted between the word and the context vectors. The authors provide theoretical insights on the role of alpha in relation with the original co-occurrence matrix, and propose a new method to find its optimal value.\n\nI think that this is overall a solid work. The paper provides a new perspective in the workings of word embeddings that I find interesting, it is theoretically well-founded (although I did not check all derivations in detail), and the presentation is clear.\n\nHowever, I think that the paper does a poor job in putting its contributions into context in relation to previous work. In particular, the role of alpha in word embeddings was already studied empirically by Artetxe et al. (CoNLL'18, https://www.aclweb.org/anthology/K18-1028.pdf) for both word analogy and word similarity tasks, to the extent that Figure 2 in both papers is showing the exact same curves. However, the authors do not even cite it. As acknowledged in the paper, other authors like Levy et al. (TACL'15) also observed that the value of alpha was important in their experiments.\n\nI think that the right narrative for the paper should more in the line of \"previous work showed that alpha behaves this and that way; we provide a theoretical explanation for this behavior, and derive a method to automatically find its optimal value\". However, starting from the title (\"word embedding re-examined: is the symmetrical factorization optimal?\", when it was already known that it wasn't) and the abstract (where only the statement that \"we propose a method to find the optimal alpha\" corresponds to a novel contribution), the paper does a poor job in identifying and properly contextualizing its real contributions. More importantly, the paper does not try to establish any connection between the authors own theory and the empirical findings from previous work.\n\nIn terms of the actual content, the authors constantly claim that word2vec is performing a symmetric factorization (e.g. \"the original word2vec is implicitly performing a symmetric factorization, thus implying the alpha equal to 0.5) as if it was something obvious or well-known. I might be missing something here, but I do not see why this is the case. Following your notation, let's say that word2vec is implicitly factorizing M = E*C^T, where E are the word embeddings and C are the context embeddings. One could multiply E with any arbitrary invertible matrix W, and C by the transpose of its inverse W^-T, which could be chosen to completely break any symmetry, yet the objective value of word2vec would not change at all, as (E*W)*(C*W^-T)^T = E*C^T. In other words, there is nothing in the training objective of word2vec that forces a symmetric factorization, and there is always an optimal solution with respect to this training objective that is arbitrarily asymmetric.\n\nAnother point that raises concerns to me is that the optimal value of alpha is determined by the vocabulary of the evaluation task. It would make sense if the optimal alpha depended on the nature of the task (e.g. syntactic vs semantic), but I do not have any intuition (nor do the authors provide) as of why the vocabulary would be anyhow relevant. More importantly, this does not seem generalizable beyond a few intrinsic tasks as, in the general case, one wants good embeddings for the full vocabulary. In either case, I think that this point deserves more attention in the paper.\n\nI also find the experimental evaluation to be somewhat weak. In particular, the proposed theory focuses in two phenomena (word similarity and word analogy) as stated in the abstract itself, but the empirical evaluation is limited to the word similarity task.\n\nAlso, this is a minor detail and it did not influence my score, but I dislike that the authors use \"word2vec\" to refer to skip-gram with negative sampling throughout the paper. I would suggest to either use SGNS (which is quite standard) or simply skip-gram.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1gw6EE7qr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study the word embedding, with a particular emphasize on the word2vec or similar strategies. To this end, the authors consider the matrix factorization framework, previously introduced in the literature, and also study the influence of an hyperparameter denoted by alpha. Roughly speaking, there are two major parts in the paper. On one hand, it explains the reasons why the word embedding schemas provide nice properties, by defining the embedding as a low rank transformation mechanism. On the other hand, they propose to choose optimally the hyperparameter alpha in order to ameliorate word embedding by better preserving the distance structure. Conducted experiments are convincing.\n\nThe paper is well written, and derivations seem correct.\n\nWe think that the major issue in this work is that it does not provide significant contributions with respect to the state of the art. It seems that the contributions are rather incremental compared to related works, such as Levy et al. 2015 and Yin & Shen 2018. In the latter, it is proven that most existing word embedding schemas can be formulated as low rank matrix approximations, either explicitly or implicitly. The submitted paper does not provide new significant results.\n\nMoreover, the authors have failed to provide connections to other related works, or even cite them, including papers that consider word embedding as asymmetric low-rank projections. See for example:\nFei Tian, Bin Gao, Enhong Chen, Tie-Yan Liu\nLearning Better Word Embedding by Asymmetric Low-Rank Projection of Knowledge Graph\nJ. Comput. Sci. Technol. (2016) 31: 624. \nhttps://doi.org/10.1007/s11390-016-1651-5\nAlso available on ArXiv: https://arxiv.org/abs/1505.04891\n\n\n------\nReply to Rebuttal\n\nWe thank the authors for modifying the paper and the reply to out comments and suggestions. However, we still think that the paper is of low quality, due to straightforward extension to the paper of Levy et al from 2015.\n\nThe authors have added a small section on related works, as recommended. However, they have removed the \"Conclusion\" section. The paper no longer has a conclusion and potential work that ends the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies word embeddings using the matrix factorization framework introduced by Levy et al 2015. The authors provide a theoretical explanation for how the hyperparameter alpha controls the distance between words in the embedding and a method to estimate the optimal alpha.  The authors also provide experiments showing the alpha found using their method is close to the alpha that gives the highest performance on the word-similarity task on several datasets. 

The paper received 2 weak rejects and 1 weak accept.  The reviews were unchanged after the rebuttal, with even the review for weak accept (R2) indicating that they felt the submission to be of low quality.  Initially, reviewers commented that while the work seemed solid and provided insights into the problem of learning word embeddings, the paper needed to improve their positioning with respect to prior work on word embeddings and add missing citations.  In the revision, the authors improved the related work, but removed the conclusion.

The current version of the paper is still low quality and has the following issues
1. The paper exposition still needs improvement and it would benefit from another review pass
Following R3's suggestions, the authors have made various improvements to the paper, including modifying the terminology and contextualizing the work.  However, as R3 suggests, the paper still needs more rewriting to clearly articulate the contribution and how it relates to prior work throughout the paper.  In addition, the conclusion was removed and the paper still needs an editing pass as there are still many language/grammar issues.

Page 5: "inherites" -> "inherits"
Page 5: "top knn" -> "top k"

2. More experimental evaluation is needed.
For instance, R1 suggested that the authors perform additional experiments on other tasks (e.g. NER, POS Tagging).  The authors indicated that this was not a focus of their work as other works have already looked at the impact of alpha on other task.  While prior works has looked at the correlation of alpha vs performance on the task, they have not looked at whether alpha estimated the method proposed by the author will give good performance on these tasks as well.  Including such analysis will make this a stronger paper.

Overall, there are some promising elements in the paper but the quality of the paper needs to be improved.  The authors are encouraged to improve the paper by adding more experimental evaluation on other tasks, improving the writing, as well as incorporating other reviewer comments and resubmit to an appropriate venue. 

Decision: Reject