Norm of Word Embedding Encodes Information Gain
Authors: Momose Oyama
Sho Yokoi
Hidetoshi Shimodaira
Venue: EMNLP 2023 Main (2023)
Keywords: Word embedding
Euclidean norm
Skip-gram with Negative Sampling
Softmax function
Kullback-Leibler divergence
Information geometry
Exponential family of probability distributions
TL;DR: With the theory of information geometry, we show that the norm of word embedding approximates the KL divergence and confirm it experimentally.

Abstract: Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution.
Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer.
We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.

vergence of the co-occurrence distribution p(-lw) of a word w for fxed-widtt window to the uni- gram distribution p(-) of the corpus, denoted as Norm of Word Embedding Encodes Information Gain
Momose Oyama 1,3
Sho Yokoi
Hidetoshi Shimodaira
Kyoto University
Tohoku University w.t.eeyys..kyooo-uu.
SRIKEN AIP
oyama. momose@sys. i kyoto-u. ac. yokoi@tohoku.. ac.jp, shimo@i www.k-oooooc....o ac.jp
Abstract
Top 10
Bottom 10
word
KL(w)
word
KL(w)
Distributed representations of words encode lexical semantic information, but what type of information encoded and how? Focusing on the skip-gran with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain con- veyed by the word; the information gain is de- fined by the Dullback-Leible divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are ex- plained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contex- tualized word embeddings in language models any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide useful metric of the informativeness a word in tasks such as key word extraction, proper-noun discrimination, hypernym dis- crimination.
rajonas
11.31
the 0.04
rajons
10.82
in 0.04
dicrostonyx
10.31
and 0.04
dasyprocta
10.27
of 0.05
stenella
10.24
0.07
pesce
10.22
to 0.09
audita
10.09
by 0.09
landosverband
10.05
with 0.10
auditum
9.96
for 0.10
factum
9.84
0.10
Table 1: Top 10 words and bottom 10 words sorted by the value of KL(w) in the text8 corpus with word frequency n. 2 10.
norm of embedding for word w. denoted as ||w ll+ is closely related the Kullback-Leibler (KL) di-
KL(w) == kL(p(-1w) p(-))-
1 Introduction
In Bayesian inference, the expected KL divergence called information gain. In this context, the prior distribution p(-). and the posterior distribution is p(-[w). The information gain represents how much information we obtain about the context word dis- tribution when observing w. Table shows that the 10 highest values of KL(w) are given by context- specific informative words, while the 10 lowest values given by context-independent words. Fig. shows that || is almost linearly re- lated to KL(w); this relationship holds also for a larger corpus Wikipedia dump as shown in Appendix G. We prove Section 4 that the square of the norm of the word embedding with whitening- like transformation approximates the KL divergence!. The main results are explained
The strong connection between natural language processing and deep learning began with word em- beddings (Mikolov et al., 2013; Pennington et al.. 2014; Bojanowski et al., 2017; Schnabel et al., 2015). Even in today's complex models, each word initially converted into vector in the first layer. One of the particularly interesting empir- ical findings about word embeddings that the norm represents the relative importance of the word while the direction represents the meaning of the word (Schakel and Wilson, 2015; Khodak et al., 2018; Arefyev et al., 2018: Pagliardini et al., 2018; Yokoi et al., 2020).
This study focuses on the word embeddings ob- tained by the skip-gram with negative sampling (SGNS) model (Mikolov et al., 2013). We show theoretically and experimentally that the Euclidean
Readers who are interested in information-theoreti mea-

---

10c
We showed theoretically and empirically that the squared norm of word embedding obtained by the SGNS model approximates the infor- mation gain of a word defined by the KL di- vergence. Furthermore, we have extended this theory to encompass contextualized embed- dings in language models.
10a
104
KL(w]
3
2
10a
y 0.048x
0.098
10a
R² = 0.831
10a
We empirically showed that the bias-corrected KI divergence and the norm of word embed- ding similarly good metric of word informativeness.
60
!|
Figure Linear relationship between the KL diver- gence and the squared norm of word embedding for the text8 corpus computed with 100 epochs. The color rep- resents word frequency The- Plotted for all vocabulary words, but those with The < 10 were discarded. A re- gression line was fitted to words with aw 10 Other settings are explained in Section 4.2 and Appendix A.
After providing related work (Section 2) and the- oretical background (Section 3), prove the the- oretical main results in Section 4. In Section 5, we extend this theory to contextualized embed- dings. We then explain the word frequency bias (Section 6) and evaluate KL(w) and || as a metric of word informativeness in the experiments of Section 7.
by the theory of the exponential family of distribu- tions (Barndorff-Nielle 2014; Efron, 1978, 2022; Amari, 1982).
Empirically, the KL. divergence, and thus the norm of word embedding, are helpful for some NLP tasks. In other words, the notion of informa- tion gain, which is defined in terms of statistics and information theory, be used directly metric of informativeness in language. We show this through experiments on the tasks of keyword extraction, proper-noun discrimination, and hyper- nym discrimination in Section 7.
2 Related work
2.1
Norm of word embedding
Several studies empirically suggest that the norm of word embedding encodes the word informativeness. According to the additive compositionality of word vectors (Mitchell and Lapata, 2010), the norm of word embedding is considered to represent the im- portance of the word in sentence because longer vectors have a larger influence on the vector sum. Moreover, it has been shown in Yokoi et al. (2020) that good performance of word mover's distance is achieved in semantic textual similarity (STS) task when the word weights are set to the norm of word embedding, while the transport costs are set to the cosine similarity. Schakel and Wilson (2015) claimed that the norm of word embedding and the word frequency represent word significance and showed experimentally that proper nouns have em- beddings with larger norms than function words. Also, it has been experimentally shown that the norm of word embedding is smaller for less in- formative tokens (Arefyev et al., 201 8; Kobayashi et al., 2020).
In addition, we perform controlled experiments that correct for word frequency bias to strengthen the claim. The KL divergence is heavily influenced by the word frequency Tw. the number of times that word w appears in the corpus. Since the cor- pus size finite, although often very large, the KL divergence calculated from the co-occurrence matrix of the corpus is influenced by the quanti- zation error and the sampling error, especially for low-frequency words. The same is also true for the norm of word embedding. This results in bias due word frequency, and spurious relationship is observed between word frequency and other quan- tities. Therefore, in the experiments, we correct the word frequency bias of the KL divergence and the norm of word embedding.
2.2 Metries of word informativeness
The contributions of this paper are follows:
Keyword extraction. Keywords are expected have relatively large amounts of information. Key- word extraction algorithms often use a metric of the "importance of words in a document" calcu- lated by some methods, such as TF-IDF or word
sures other than KL. divergence are referred to Appendix B. The KL divergence is more strongly related to the norm of word embedding than the Shannon entropy of the co- occurrence distribution (Fig. 7) and the self-information - -log((w) (Fig. 8).

---

co-occurrence (Wartena et al., 2010). Matsuo and Ishizuka (2004) showed that the x² statistics com- puted from the word co-occurrence are useful for keyword extraction. The x² statistic is closely re- lated to the KI. divergence (Agresti, 2013) since x2 statistic approximates the likelihood-ratio chi- squared statistic G2 = (n,,L((w) when each doc-
p(-lw), Since these are probability distributions. Ewer p(w) Eweey p(w"[w) 1, where V is the vocabulary set of the corpus. The frequency- weighted average of p(-lw) is again the unigram distribution p(.), that is,
p(+) = Σ p(w)p((lw).
(1)
WEV
ument is treated corpus.
Embeddings. SGNS learns two different embed- dings with dimensions d for each word in V: word embedding the E Idd for w V and context em- bedding Vev' E I for w' E V. We denote the frequency-weighred averages of uw and uar as
The identification of hypernyms (superordinate words) and hyponyms (subordinate words) in word pairs, c.g., cal and munchkin, has been actively studied. Recent un- supervised hypernym discrimination methods are based on the idea that hyponyms are more informa- tive than hypernyms and make discriminations by comparing a metric of the informativeness of words. Several metrics have been proposed, including the KL divergence of the co-occurrence distribution to the unigram distribution (Herbelot and Gane- salingam, 2013), the Shannon entropy (Shwartz al., 2017), and the median entropy of context words (Santus et al., 2014).
Hypernym discrimination.
u = Σ p(w)uw: v = Σ p(w')o
(2)
WEV
ar E'EV
We also use the centered vectors
th Uw u, Dul == Uwa V.
3.2 KL divergence measures information gain
The distributional semanties (Harris, 1954; Firth, 1957) suggests that "similar words will appear similar contexts" (Brunila and La Violette, 2022). This implies that the conditional probability distri- bution p(-1w) represents the meaning of a word w.
Word frequency bias. Word frequency is strong baseline metric for unsupervised hypernym discrimination. Discriminations based on several unsupervised methods with good task performance are highly correlated with those based simply on word frequency (Bott et al.. 2021). KL diver- gence achieved 80% precision but did not out- perform the word frequency (Herbelot and Gane- salingam, 2013). WeedsPrec (Weeds et al., 2004) and SL.QS Row (Shwartz et al., 2017) correlate strongly with frequency-basee predictions, calling for the need to examine the frequency bias in these methods.
The difference between p(-1w and the marginal dis- tribution p(-) can therefore capture the additional information obtained by observing in a corpus.
A metrie for such discrepancies of information is the KI. divergence of p(- ((-(w) to p(-). defined as
p(w'(w) (L(p(-lw) I p(-)) = Σ p(w' (w) log p(w")
w'eV
In this paper, we denote it by KT(ww) and call it the KL divergence of word w. Since p(-) is the prior distribution and p(-lw) the posterior distribution given the word w, KL(w) can be interpreted as the information gain of word w (Oladyshkin and Nowak, 2019). Since the joint distribution of w' and w is p(w', w) (wwl)))((w)) the expected value of KL(w) is expressed as
3 Theoretical background
In this section, we describe the KL divergence (Sec- tion 3.2), the probability model of SGNS (Sec- tion 3.3), and the exponential family of distribu- tions (Section 3.4) that are the background of our theoretical argument in the next section.
Σ (LW) WEV
Σ p(w', w) log p(w',u) w) p(w' ((w))(())
Σ> WeV W'EV
3.1 Preliminary
Probability distributions. We denote the prob- ability of a word w in the corpus as p(w) and the unigram distribution of the corpus as p(-). Also, we denote the conditional probability of a word w' co-occurring with w within a fixed-width window as p(w"lw), and the co-occurrence distribution as
This is the mutual information I(W", W) of the two random variables W' and W that correspond to w' and w. respectively?. I(W", W) is often called information gain in the literature.
In the following, w' and wo represent W' and W by abuse of notation.

---

3.3 The probability model of SGNS
for u uw with constraints ((u)) log " for w E V; the model (5) is curved exponential family when the parameter value constrained as (²))) - - llog v. but we do not assume in the following argument.
The SGNS training utilizes the Noise Contrastive Estimation (NCE) (Gutmann and Hyvärinen, 2012) to distinguish between p(-lw) and the negative sampling distribution g(-) x p(-)3/4 For each co-occurring word pair (w, w") in the corpus. negative samples {w]] are generated, and aim to classify the 1 samples (w', w":.... w as either positive sample generated from w' p(u(lw) or a negative sample generated from w" q(u"). The objective of SGNS (Mikolov et al.,
This section outlines some well-known basic properties of the exponential family of distribu- tions, which have been established in the litera- ture (BarndofffNielsenn, 2014; Efron, 1978, 2022; Amari, 1982). For ease of reference, we provide the derivations of these basic properties Appendix
2013) involves computing the probability of w being a positive sample using a kind of logistic re-
The expectation and the covariance matrix of Uw with respect to w' ~ -p(wllu) are calculated as the first and second derivatives of (u(),, respectively. Specifically, we have
gression model, which is expressed as follows (Gut-
mann and Hyvärinen, 2012):
p(w'lu) (w)
@vp(u) Σ p(w'lu)uw'
n(u) = Iu W'EV
(6)
p(w'|w) vq(w")
1+ e (M
To gain better understanding of this formula, we can cross-multiply both sides of (3) by the denomi- nators:
G(u) = S2
Dudu
Σ W'EV
((u))(uw n()))
(7)
p(w"lw) (1) (1+ + cm(m::wr)) = p(w'l(u) (w) + vg(w"),
and rearrange it to obtain:
The KL. divergence of p(-lu)) to p(-(u2) for two parameter values u1, u2 E Rd is expressed as
p(w'lw) ()) = vg( w/) "r(ww")llwwneg (e(
(4)
KL(p(-lun) | P(-lw2))
We assume that the co-occurrence distribution sat- isfies the probability model (4). This is achieved when the word embeddings } and {owr per- feetly optimize the SGNS's objective, whereas holds only approximately in reality.
(wi - u2, m7(1) (uu1) + (uu)).
(8)
The KL divergence is interpreted as the squared distance between two parameter values when they are not very far from each other. In fact, the KL divergence (8) is expressed approximately as
3.4 Exponential family of distributions
We can generalize (4) by considering an instance of the exponential family of distributions (Lehmann and Casella, 1998; Garndofff-Nielsett 2014; Efron, 2022), given by
(KL(((- || p(- p(-[u2))
~ (w) - uz) G(u,) (ui - u2)
(9)
for i = 1,2. Here, the equation holds approxi-
p(w'lu)= = q(w') exp((u, vnr) (u)),, (5)
mately by ignoring higher order terms of O(Iuu
u2ll3). For more details, refer to Amari (1982. p. 369), Efron (2022, p. 35). More generally, G(u) is the Fisher information metric, and (9) holds for wide class of probability models (Amari, 1998).
where u E Rd is referred to as the natural parame- ter vector, Vn E Rd represents the sufficient statis- tics (treated as constant vectors here, while tunable parameters in SGNS model), and the normalizing function is defined as
Squared norm of word embedding approximates KL divergence
4 (u) == log Σ q(1o') exp((u, v)))), w'EV
In this section, we theoretically explain the linear relationship between KL(w) and || observed in Fig. 1 by elaborating on additional details of the exponential family of distributions (Section 4.1) and experimentally confirm our theoretical results (Section 4.2).
ensuring that Ev'cv p(w'lu) (u) 1 for any E Id. The SGNS model (4) is interpreted as a special case of the exponential family
p(w'lw) p(w'|uw)

---

4.1 Derivation of theoretical results
10c
10
We assume that the unigram distribution is repre- sented by a parameter vector up E Rd and
10a
8
2KL(w)
6
-00
p(w') p(w'luo).
(10)
4
10a
By substituting 4n and u2 with tw and uo respec-
2
y = 0.909x 0.626
10a
tively in (9), we obtain
0.743
10a
0.0
(kL(w) = (uw -u)) To (uw uo). (11)
(U, Ug) G(u, Us)
Here G = G(u)) is the covariance matrix of ow with respect to w' ~ p(w'), and we can easily compute it from (7) as
Figure 2: Coufirmation of (11). The slope coefficient of 0.909, which is close to 1, indicates the validity of the theory.
108
G Σ plw (tw T) W'EV
10
105
8
because 7((o)= = v from (2) and (6). However, it is important note that the value of ug trained in practice, and thus we need an estimate of uo to compute the uo on the right-hand side of (11).
2KL(w)
10
6
4
103
2
y = 1.384x 0.531
10a
We argue that uw - ug in (11) can replaced
R 0.705
by Une ū = Un so that
10a
||
2KL(w) = ū iww-
(12)
Figure 3: Confirmation of (12) and (14). The slope coefficient of 1.384, which is close to 1, suggests the validity of the theory.
For formal derivation of (12), see Appendix K. Intuitively speaking, ū approximates uo, because ū corresponds to p(-) in the sense that is the weighted average of tw as seen in (2), while p(-) is the weighted average of p(-[uw)) as seen in (1). To approximate ug. could also use the of some representative words instead of using We expect ug to be very close to some uw of stop- words such as 'a' and 'the" since their (ww) are expected to be very close to p(-).
4.2 Experimental confirmation of theory
The theory explained so far was confirmed by an experiment on real data.
Settings. We used the text8 corpus (Mahoney,
201 1) with the size of 17.0 106 tokens and VI = 254 x 10 vocabulary words. We trained 300-dimensional word embeddings (wwr) and (Ww)e'ev by optimizing the objective of SGNS model (Mikolov et al., 2013). We also com- puted the KL divergence (KI(w))wer from the co-occurrence matrix. These embeddings and KL divergence are used throughout the paper. See Ap- pendix for the details of the settings.
Let us define a linear transform of the centered embedding as
Uw : GZ
(13)
i.e., the whitening of uw with the context embed- ding?. then (12) is now expressed*
2KL(w) ~ |w
(14)
Details of Fig. 1. First, look at the plot of KL(w)) and (w ||2 in Fig. 1 again. Although the are raw word embeddings without the transformation (13), confirm good linearity ||u x KL(w). A regression line was fitted words with nhe 103 where low-frequency words were not very stable and ignored. The coefficient of determination R² = 0.831 indicates a very good fitting.
Therefore, the square of the norm of the word em- bedding with the whitening--ie like transformation in (13) approximates the KL divergence.
Note that the usual whightening is Cov(u) - tu.. but we call (13) as "whitening" for convenience in this paper.
((12) and (14) equivalent, because || @... 11² u.
(G² ) G² aı = G. G ih a.G

---

Adequacy of theoretical assumptions. In Fig. 1, the minimum value of KL(w) is observed to be very close to zero. This indicates that (((ww) for the most frequent w is very close to p(-) in the corpus, and that the assumption (10) in Section 4.1 is adequate.
the co-occurrence distribution, p(-lu) now signi- fies the word distribution specific token posi- tion provided with the contextualized embedding u. Instead of the frequency- Em p(w) we redefine 7 == Σ>= W//N as the average over the contextualized embeddings {u, ) calculated from the training corpus of the language model. Here, u, denotes the contextu- alized embedding computed for the i-th token of the training set of size N. The information gain of contextualized embedding u is
average
Confirmation of the theoretical results. To con- firm the theory stated in (II), we thus estimated up as the frequency-weiihted average of word vectors corresponding to the words {the, of. and]. These three words were selected as they are the top three words in the word frequency nww Then the correct- ness of (11) was verified in Fig. 2, where the slope coefficient is much closer to than 0.048 of Fig. 1. Similarly, the fitting in Fig. confirmed the theory stated in (12) and (14), where we replaced uo by u.
KL(u)= == (L(p(-lu) || p()))
With these modifications, all the arguments pre- sented in Sections 3.4 and 4.1, along with their respective proofs, remain applicable in the same manner (Appendix L), and we have the main result (14) extended to contextualized embeddings
Experiments on other embeddings. In Ap- pendix G, the theory was verified by performing experiments using a larger corpus of Wikipedia dump (Wikimedia Foundation, 2021). In Ap- pendix H. we also confirmed similar results using pre-trained fastText (Bojanowski et al., 2017) and SGNS (Li et al., 2017) embeddings.
2KL(u) ~ |
(16)
where the contextualized version of the centering and whitening are expressed as := u - ū and ū == G û, respectively.
5
Contextualized embeddings
5.2 Experimental confirmation of theory
The theory developed for static embeddings of the SGNS model is extended to contcxtualized embed- dings in language models, or any neural networks with the softmax output layer.
RoBERTa
Uama
0.557
A² 0.112
cos 0.977
0.902
5.1 Theory for language models
The final layer of language models with weights owr E Rd and bias byr E RR is expressed for contex- tualized embedding u E Rd as
[0
[0-00]
Figure 4: Linear relationship between the KL diver- gence and the squared norm of contextualized embed- ding for RoBERTa and Ilama The color represents token frequency.
Yw = (u, Ver) + bw'
and the probability of choosing the word w/ V is calculated by the softmax function
We have tested four pre-trained language mod- els: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT-2 (Radford et al., 2019), and Llama 2 (Touvron et al., 2023) from Hugging Face transformers library (Wolf et al., 2020). Since the assumption (10) may not be appropri- ate for these models, we first computed up argmin KL(u), and used p(-luo) as a substitute for p(-) when verifying the linear rela- tionship between KL(u) and ||u uol??. Fig. 4 demonstrates that the lincar relationship holds ap- proximately for RoBERTa and Llama 2. All results,
evlu'
Psoftmax Psoftmax(wollv (u)
(15)
EwEy e8w
eun
Comparing (15) with (5), the final layer is actu- ally interpreted as the exponential family of dis- tributions with qww)) =d'u/ /tw//Ewcy db so that Psoltmax (w/lu) [u] pu)) Thus, the theory for SGNS based on the exponential family of distribu- tions should hold for language models.
However, we need the following modifications to interpret the theory. Rather than representing

---

corpus. Note that the denominator of p(w" (w) is En"ev Rww"" 2hn if the endpoints of the cor- pus are ignored.
raw
10
shuffle
lower 3 percentile
round
KL(w)
Sampling error ('shuffle"). Now we explain how word frequency directly influences the KL diver- gence. Consider randomly shuffled corpus, i.e., words are randomly reordered from the original corpus (Montemurro and Zanette, 2010; Tanaka- Ishii, 2021). The unigram information, i.e., now and p(-), remains unchanged after shuffling the corpus. On the other hand, the bigram informa- tion, i.c., Nw and p(-lw), computed for the shuf- fled corpus is independent of the co-occurrence of words in the original corpus. In the limit of N >> oo, p(-lw) = p(-) holds and KT.(w) = 0 for all w E V in the shuffled corpus. For finite corpus size N, however, p(-lw: deviates from p(-) because ) ew is approximately interpreted as a sample from the multinomial distribution with parameter p(-) and 2hm
2
10a 10a 10a 104
10a
10c
fgr
Figure 5: KL divergence computed with four different procedures plotted against word frequency for the same words Fig. I. 'raw'. 'shuffle', , and 'round' are KL(w), KL(w), and KLa(w), respectively. 'lower percentile" is the lower 3- 3-percentile point of KL(w)) at each word frequency bin.
including those for BERT and GPT-2, as well as ad- ditional details, are described in Appendix I. While not as distinct as the result from SGNS in Fig. 1, was observed that the theory suggested by (16) ap- proximately holds true in the case of contextualized embeddings from language models.
In order to estimate the error caused by the di- rect influence of word frequency, we generated 10 sets of randomly shuffled corpus and computed the average of KL(w), denoted as ([(w), which is shown as 'shuffle' in Fig. 5. KT((w) does not convey the bigram information of the original cor- pus but does represent the sampling error of the multinomial distribution. For sufficiently large N. expect (T,(w) 0 for all c V. However, KL (w) is very large for small nw in Fig.
6 Word frequency bias in KL divergence
The KL divergence is highly correlated with word frequency. In Fig. 5, 'raw' shows the plot of KL(w)) (w) against fh. The KL divergence tends to be larger for less frequent words. A part of this tendency represents the true relationship that rarer words are more informative and thus tend to shift the co-occurrence distribution from the corpus distri- bution. However, large part of the tendency, par- ticularly for low-frequency words, comes from the error caused by the finite size N of the corpus. This introduces spurious relationship between KT.(w) and nu causing direct influence of word frequency. The word informativeness can be better measured by using the KL divergence when this error is adequately corrected.
Sampling error ("lower 3 percentile"). Another computation of KL(w) faster than 'shuffle" was also attempted indicated as 'lower percentile' in Fig. 5. This represents the lower 3-percentile point of KT((w) Na. First, 200 bins were equally spaced on a log- arithmic scale in the interval from to max(n Next, each bin was checked in order of decreas- ing The and merged so that each bin had at least 50 data points. This method allows for faster and more robust computation of KL(w) directly from KL((w) of the original corpus without the need for shuffling.
narrow bin of word frequency
6.1 Estimation of word frequency bias
Preliminary. The word distributions p(-) and p(-lw) are calculated from a finite-length corpus. The observed probability of word w is p(w)= / where N = Ewev Rw. The observed prob- ability of context word w' co-occurring with is p(w"[w) Thw,w'/ Ew'ev Tw,,", where Dww the co-occurrence matrix. We computed nyy as the number of times that w' appears within a window of th around w in the
Quantization error (fround"). There is another word frequency bias due to the fact that the co- occurrence matrix only takes integer values; it is indicated as 'round' in Fig. 5. This quantization error is included in the sampling error estimated by KL(w), so there no need for further correction. See Appendix C for details.

---

6.2 Correcting word frequency bias
Settings. We used 15 public datasets for keyword extraction for English documents. Treating each document as a "corpus", vocabulary words were or- dered by a measure of informativeness, and Mean Reciprocal Rank (MRR) was computed as an eval- uation metric. When keyword consists of two or more words, the worst value of rank was used. We used specific metrics, namely 'random', fh nu H (w) and (T))), our baselines. These metrics are computed only from each document without relying on external knowledge, such as a dictionary of stopwords or a set of other documents, For this reason, we did not use other metrics, such as TF-IDE, as our baselines. Note that ||w/// 1? was not included in this experiment because embed- dings cannot be trained from very short "corpus".
We simply subtracted (L(w) from [L(w). The sampling error (T(w) was estimated by either 'shuffle' or 'lower 3 percentile'. We call
(KL(w) = - - KL(w)
(17)
the bias-corrected KL divergence. The same idea using the random word shuffling has been ap- plied to an entropy-like word statistic in an existing study (Montemurro and Zanette, 2010).
7 Experiments
In the experiments, we first confirmed that the KL divergence is indeed good metric of the word in- formativeness (Section 7.1). Then we confirmed that the norm of word embedding encodes the word informativeness as well as the KI. divergence (Sec- tion 7.2). Details of the experiments are given in Appenices D, E, and F. As one of the baseline methods, we used the Shannon entropy of p(-lw), defined as
Results and discussions. Table shows that Thy (L))) performed best in many datasets. There- fore, keywords tend to have a large value of Th (wr),, and thus p(-lw) is significantly differ- ent from p(-). This result verifies the idea that key- words have significantly large information gain.
[[w)=
Σ p(w'lll log p(w'[w). w'ev
Norm of word embedding encodes the 7.2
word informativeness
It also represents the information conveyed by w explained in Appendix B.
We confirmed through proper-nour discrimination tasks (Section 7.2.1) and hypernym discrimination tasks (Section 7.2.2) that the norm word em- bedding, as well as the KL divergence, encodes the word informativeness. and also confirmed that correcting the word frequency bias improves it. In these experiments, examined the proper- ties of the raw word embedding the instead of the whitening- transformed word embedding uw. From practical standpoint, we used thes but ex- periments using بي exhibited a similar trend.
Dataset
random
Ye (ww)
Me KL(w)
Krapivin2009
0.86
6.17
6.13
9,59
theses100 100
0.97
9.69
9.79
12.31
fao780
1.61
11.77
11.84
15.39
SemEva12010
1.67
9.52
9.50
11.10
Nguyen2007
1.90
10.56
10.57
12.84
PubMed
2.89
8.28
8.25
11.93
citeulike188
180
4.01
18.20
18.18
17.98
wiki20
4.15
9.32
9.23
19,90
fao30
4.92
15.92
17.05
36,88
Schutz2008
8.36
22.32
22,83
20.93
kdd
10.14
18.27
18.24
Correcting word frequency bias. In the same 10.08
Inspec
10.54
16.31
16,22
14.61
way as (17), correct the bias of embedding norm and denote the bias-corrected squared norm as A ||uu,1122 := Mne 2 I u,.ll² We used the 'lower percentile" method of Section 6. for A |!w/l because the recomputation of embeddings for the shuffled corpus is prohibitive. Other bias-corrected quantities, such as AKL(w) and (H(w,, were computed from 10 sets of randomly shuffled cor- pus.
www
12.08
21.20
21.11
12.76
SemEva12017
14.16
19.86
19.62
20.85
KPCrowd 39.64
25.73
25.82
40.47
Table 2: MRR of keyword extraction experiment. For complete results on MRR and P@5,, see Tables 6 and 7, respectively, in Appendix D.
7.1 KL divergence represents the word informativeness
7.2.1 Proper-noun discrimination
Through keyword extraction tasks, we confirmed that the KL divergence indeed a good metric of the word informativeness.
Settings. We used 10561 proper nouns, 123 func- tion words, 4771 verbs, and 2695 adjectives that appeared in the text8 corpus not less than 10 times.

---

Rt
((w)
KL(w)
||u=|l
(H(w)
AKL(w)
A Allua ||
proper vs. verbs
0.519
0.582
0.651
0.656
0.715
0.826 0.699
0.842
proper nouns vs, adjectives
0.543
0.581
0.613
0.626
0.645
0.728
Table 3: Binary classification of part-of-speech. Appendix for histograms of measures.
Values are the ROC-AUC (higher is better). See Fig. in
3.0
60
2.5
proper
proper nouns
50
AKL(w)
1.5
2.0
function words
verbs
Alluw]!?
40
30
verbs
function words
1.0
20
0.5
10
0.0
0
-0.5
-10
10
10a
10a
10
10
10
10
10
103
104
10a
10
Figure 6: The bias-corrected KL divergence (K) and the birsecod squared norm of word embedding are plotted against word frequency nw- Each dot represents word; 10561 proper nouns (red dots), 123 function words (blue dots), and 4771 1771 verbs (green dots). The same plot for adjectives, which is omitted in the figure, produced scatterplot that almost overlapped with the verbs.
1 Al|uw/l
7.2.2 Hypernym discrimination
Thyper/ Phypo <
>
Settings. We used English sypernym-hyoonye pairs extracted from four benchmark datasets for hypernym discrimination: BLESS (Baroni and Lenci, 2011), EVALution (Santus et al., 2015), Lenci/Benotto (Lenci and Benotto, 2012), and Weeds (Weeds al., 2014). Each dataset was di- vided into two parts by comparing nu of hyper- nym and hyponym to remove the effect word frequency. In addition to 'random' and Rus we used WeedsPrec (Weeds and Weir, 2003; Weeds et al., 2004), SLQS Row (Shwartz et al., 2017) and SLQS (Santus et al., 2014) as baselines.
random
50.00
50.00
50.00
Rw
100.00
0.00
50.00
WeedsPrec
95.05
7.61
51.33
STQQS Row
95.20
13.70
54.45
SLQS
82.69
42.82
62.76
KL(ww)
96.46
17.84
57.15
|uwlll
94.07
24.89
59.48
WeedsPrec
46.53
51.88
49.20
ASLQS Row
59.75
43.14
51.44
ASLQS
50.41
69.06
59.74
(KL(w)
65.90
62.94
64.42
Al|ua..||"
75.86
61.81
68.84
Results and diseussions. Table shows that A / and AKT.(w) were the best and the sec- ond best, respectively, for predicting hypernym hyperyym-hyponym pairs. Correcting frequency bias remedies the difficulty of discrimination for the Phyper < Phypo part, resulting in an improve- ment in the average accuracy.
Table 4: Accuracy of hypernym-hyponym classification; the unweighted average over the four datasets. See Table 9 in Appendix for the complete result.
We used Pios (w),, KL(w), and || 2 as mea- sure for discrimination. The performance of binary classification was evaluated by ROC-AUC.
8 Conclusion
We showed theoretically and empirically that the KL divergence, i.e., the information gain of the word, is encoded in the norm of word embedding. The KI divergence and, thus, the norm of word em- bedding has the word frequency bias, which was corrected in the experiments. We then confirmed that the KL divergence and the norm of word em- bedding work as metric of informativeness NLP tasks.
Results and discussions. Table 3 shows that AKL(W) and 1 can discriminate proper nouns from other parts of speech more effec- tively than alternative measures. A larger value of (kL(w) and A A||/urr! 12 indicates that words ap- pear in more limited context. Fig. illustrates that proper nouns tend to have larger (KT((w) and A |wll values when compared to verbs and func-
tion words.

---

Limitations
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. 2019. Implicit regularization in deep matrix factor- ization. Advances in Neural Information Processing Systems, 32.
The important limitation of the paper is that the theory assumes the skip-gram with nega- tive sampling (SGNS) model for static word embeddings or the softmax function in the final layer of language models for contextual- ized word embeddings.
Giusepppe Attardi. 2015. Wikiextractor. https: // gi githh.ccom/aaee
Isabelle Augenstein, Mrinal Das, Sebastian Riedel. Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10; SciencelEB extracting keyphrases and relations from scientific publications. In Proceedings of the 1 11th International Workshop Semantic Evaluation (SemEval-2017,, pages 546- 555, Vancouver, Canada. Association for Computa- tional Linguistics.
The theory also assumes that the model is trained perfectly, as mentioned in Section 3.3. When the assumption is violated, the theory may not hold. For example, the training is not perfect when the number of epochs insuffi- cient, as illustrated in Appendix G.
Ole Namedorf--ielsenn 2014. Information and expo- nential families: in statistical theory. John Wiley & Sons.
Ethics Statement
This study complies with the ACL Ethics Policy".
Marco Baroni and Alessandro Lenci. 2011. How
BLESSed distributional semantic evaluation. In Pro- ceedings of the GEMS 2011 Workshop on GEomet- rical Models of Natural Language Semantics, pages 1-10, Edinburgh, UK. Association for Computational Linguistics.
Acknowledgements
We would like to thank Junya Honda and Yoichi Ishibashi for the discussion and the anonymous reviewers for their helpful advice. This study was partially supported by JSPS KAKENHI 22H05106 23H03355, JST ACT-X JPMIAX200S, and JST CREST JPMJCR2IN3.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
Thomas Bott, Dominik Schlechtweg, and Sabine Schulte im Walde. 2021. More than just frequency? demasking unsupervised hypernym prediction meth- ods. In Findings of the Association for Computa- rional Linguistics: ACL-IJCNLP 2021, pages I 186- 192, Online. Association for Computational Linguis- tics.
References
Alan Agresti. 2013. Categorical Data Analysis, 3rd edition. John Wiley& & Sons.
Shun-Ichi Amari. 1982. Differential Geometry of Curved Exponential Families-Curvattus and Infor- mation Loss. The Annals of Statistics, 10(2):357 385.
Mikael Brunila and Jack LaViolette. 2022. What com- pany do words keep? revisiting the distributional se- mantics of J.R. firth & zellig Harris. In Proceedings of the 2022 Conference of the North American Chap- tr of the Association for Computational Linguistics: Human Language Technologies, pages 4403-4417, Seattle, United States. Association for Computational Linguistics.
Shun-Ichi Amari. 1998. Natural gradient works effi- ciently in learning. Neural computation, 10:251-276.
Makoto Aushima, Dan Shen, Haipeng Shen, Kazuyoshi Yata, Yi-Hui Zhou, and James S Marron. 2018. A sur- vey of high dimension low sample size asymptotics. Australian & New Zealand journal of statistics, 60:4- 19.
Ciprian Chelba, Tomás Mikolov, Mike Schuster, Qi Ge. Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTER- SPEECH 2014, 15th Annual Conference of the In- ternational Speech Communication Association, Sin- gapore, September 14-18, 2014, pages 2635-2639. ISCA.
Nikolay Arefyev, Pavel Ermolaev, and Alexander Panchenko. 2018. How much does a word weigh? weighting word embeddings for word sense induc- tion. ArXiv 1805.09209.
A. R. Aronson, O. Bodenreider, H. Chang, S. M. Humphrey, J. G. Mork, S. Nelson, T. C. Rindflesch, and W. J. Wilbur. 2000. The NLM indexing initiative. In Proceedings of the AMIA Symposium. pages 17- 21.
Jacob Devlin, Ming- Heig-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages
'https: /tvtps://www..rr www.eb..wbbb.oo. org/pootal//ootenttt acl-code-ethic.

---

4171-4186, Minneapolis, Minnesota, Association for Computational Linguistics.
Methods in Natural Language Processing (EMNLP). pages 7057-7075, Online. Association for Computa- tional Linguisties.
Bradley Efron. 1978. The geometry of exponential families. The Annals of Statistics, 6:362 376.
Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009. Large dataset for keyphrases extrac- tion. Technical Report DISI-09-055, University of Trento.
Bradley Efron. 2022. Exponential Families in Theory and Practice. Cambridge University Press.
Stefan Evert. 2005. The statistics of word cooccur- rences: word puirs and collocations. Ph.D. thesis, University of Stuttgart.
Erich L Lehmann and George Casella. 1998. Theory of point estimation. Springer New York, NY.
Alessandro Lenci and Giulia Benotto. 2012. Identi- fying hypernyms in distributional semantic spaces. In *SEM 2012: The First Joint Conference on Lexi- cal and Computational Semantics Volume 1: Pro- ceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth Interna- tional Workshop on Semantic Evaluation (SemEval 2012), pages 75-79, Montréal, Canada. Association for Computational Linguistics.
J. R. Firth. 1957. A synopsis of linguistic theory 1930- 55. Studies in Linguistic Analysis (special volume of the Philological Society), 1952-59:1-32.
Sujatha Das Gollapalli and Comelia Caragea. 2014. Ex- tracting keyphrases from research papers using cita- tion networks. Proceedings of AAAI Conference on Artificial Intelligence, 28(1).
Michael Gutmann and Aapo Hyvärinen. 2012. Noise- contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn Res., 13:307-361.
Bofang Li, Tao Liu, Zhe Zhao, Buzhou Tang, Aleksandr Drozd, Anna Rogers, and Xiaoyong Du. 2017. In- vestigating different syntactic context types and con- text representations for learning word embeddings. In Proceedings of the 2017 Conference Empiri- cal Methods Natural Language Processing, pages 2421-2431, Copenhagen, Denmark. Association for Computational Linguistics.
Zellig Harris. 1954. Distributional structure. Word, 10(2-3):146-162
Aurélie Herbelot and Mohan Ganesalingam. 2013. Mea- suring semantic content in distributional vectors. In Proceedings of the 51st Annual Meeting of the Associ- ation for Computational Linguistics ( (Volume 2: Short Papers), pages 440- Sofia, Bulgaria. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 197.
Anette Hulth. 2003. Improved automatic keyword ex-
Matt Mahoney, 2011. About the test data. http:// html.
traction given more linguistic knowledge. In Pro- ceedings of the 2003 Conference on Empirical Meth- ods in Natural Language Processing, pages 216-223.
mat tmahoney
Luís Marujo, Márcio Viveiros, and João Paulo da Silva Neto. 2011. Keyphrase cloud generation of broadcast news. Proceedings of Annual Conference of the International Speech Communication Associa- tion, pages 2393-2396.
Sungkyu Jung and J Stephen Marron. 2009. PCA con- sistency in high dimension, low sample size context. The Annals of Statisticx, 37:4104 4130.
Y. Matsuo and M. Ishizuka. 2004. Keyword extraction from single document using word co-occurrence statistical information. International Journal on Arti- ficial Intelligence Tools, 13(01):157-169
Mikhail Khodak, Nikunj Saunshi, Yingyu Liang. Tengyu Ma, Brandon Stewart, and Sanjeev Arora. 2018. A la carte embedding: Cheap but effective in- duction semantic feature vectors. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages 12-22, Melbourne, Australia, Association for Computational Linguistics.
Alyona Medelyan. 2015. datasets. eeyword-extraation-daaeeee
Keyword extraction com//eeanddya/
Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009. Human competiie tagging using automatic keyphrase extraction. In Proceedings of the 2009 Conference Empirical Methods in Natural Lan- guage Processing, pages 1318-138 1327, Singapore. As- sociation for Computational Linguistics.
Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Tim- othy Baldwin. 2010. SemEval-2010 task 5 Auto- matic keyphrase extraction from scientific articles. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 21-26, Uppsala, Sweden. Association for Computational Linguistics.
Olena Medelyan and Ian Witten. 2008 Domain- independent automatic keyphrase indexing with small training sets. Journal of the American Society for Information Science and Technology. 59(7):1026 1040.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical

---

Olena Medelyan, lan H Witten, and David Milne. 2008.
Adriaan M. J. Schakel and Benjamin J. Wilson. 2015. Measuring word significance using distributed repre- sentations of words. ArXiv 1508.0222 508.02297.
Topic indexing with Wikipedia. In Proceedings of the AAAI WikiAl workshop, volume 1, pages 19-24.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean. 2013. Distributed representa- tions of words and phrases and their compositionality In Advances in Neural Information Processing Sys- lems, pages 31 11 11-1199
Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pages 298-307, Lisbon, Portugal. Association for Computational Linguisties.
Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Sci- ence, 34(8): (1:8)):1388 188-1489.
A. T. Schutz. 2008. Keyphrase extraction from single documents in the open domain exploiting linguistic and statistical methods. Master's thesis, National University of Ireland.
Marcelo A. Montemurro and Damiá n H. Zanette. 2010. Towards the quantification of the semantic informa- tion encoded in written language. Advances in Com-
Vered Shwartz, Enrico Santus, and Dominik Schlechtweg. 2017.
plex Systems, 13 13(2)135-1533
Linguistically-motte Hypemyys under siege: artillery for hypernymy detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 65-75, Valencia, Spain. Association for Computational Linguistics.
Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers, pages 317-326, Berlin, Hei- delberg. Springer Berlin Heidelberg.
Sergey Oladyshkin and Wolfgang Nowak. 2019. The connection between bayesian inference and informa- tion theory for model selection, information gain and experimental design. Entropy. 21(11):1081.
Kumiko Tanaka-Ishii. 2021. Statistical Universals of Language. Springer.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 009888.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embed- dings using compositional n- n-gram features. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pages 528-540, New Orleans, Louisiana. Association for Computational Linguis- tics.
Christian Wartena, Rogier Brussee, and Wout Slakhorst. 2010. Keyword extraction using word co-occurrence. In 2010 Workshops on Database and Expert Systems Applications, pages 54-58.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Julie Weeds, Daoud Clarke, Jeremy Reffin, David Weir, and Bill Keller. 2014. Learning to distinguish hyper- nyms and co- hyponyms. In Proceedings of COLING 2014, the 25th International Conference on Compu- tational Linguistics: Technical Papers, pages 2249- 2259, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188::
Julie Weeds and David Weir. 2003. A general frame- work for distributional similarity. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 81-88.
Enrico Santus, Alessandro Lenci, Qin Lu, and Sabine Schulte im Walde. 2014. Chasing hypernyms in vec- tor spaces with entropy. In Proceedings of the 14th Conference of the European Chapter of the Associa- tion for Computational Linguistics, volume 2: Short Papers, pages 38-42, Gothenburg, Sweden. Associa- tion for Computational Linguistics.
Julie Weeds, David Weir, and Diana McCarthy. 2004. Characterising measures of lexical distributional sim- ilarity. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguis- tics, pages 1015-1022 Geneva, Switzerland. COL- ING.
Enrico Santus, Frances Yung, Alessandro Lenci, and Chu-Ren Huang. 2015. EVALution 1.0: an evolving semantic dataset for training and evaluation of distri- butional semantic models. In Proceedings of the 4th Workshop Linked Data in Linguistics: Resources and Applications, pages 64-69, Beijing, China. As- sociation for Computational Linguistics.
Wikimedia Foundation. 2021. English wikipedia dump data. Accessed on: 15-Jume-2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,

---

10c
Dimensionality
300
11
-0.058x
10.547
Epochs
100
R² 0.435
10
Window size h
10
10
Negative samples
5
H(w)
9
10
Learning rate
0.025
10a
Min count
1
7
10a
Table 5: SGNS parameters.
10a
100
||w
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Figure The Shannon entropy and the squared norm of word embedding. Settings are the same as in Fig. 1.
100
14
10a
12
dog pow
10
10
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. 2020. Word rotator's distance. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 2944-2960, Online. Association Computa- tional Linguistics.
-00
y = 0.112x + 7.352
-10
R' 0.373
-10
0
20
100
||
A Settings for computation of word embeddings and KL divergence
Figure 8: The self-information and the squared norm word embedding. Settings are the same as in Fig. 1.
Corpus. We used the text8 (Mahoney, 2011). which is an English corpus data with the size of N = 1 10t tokens and =1 254x 254x 103 vocab- ulary words. We used all the tokens separated by spaces for word embeddings and KL divergence.
Computation of KI. divergence. The value of KL(w) was computed from p(-lw) w)) and p(-) us- ing the definition in Section 3.2 with the conven- tion that Olog 0 O. The word probability p(w") and the co-occurrence probability p(w'lw) were computed from the word frequency nw and the co- occurrence matrix (Nho,wr w. respectively, as described in Section 6. The co-occurrence matrix was computed with the window size h - 10.
Training of the SGNS model. Word embeddings were trained? by optimizing the same objective function used in Mikolov et al. (2013). Parameters used to train SGNS summarized in Table 5. The learning rate shown is the initial value, which decreased linearly the minimum value of 1.0 10-4 during the learning process. The negative sampling distribution was specified as
Word set for visualization. We have used 17 x 10' words with n. 2 10 for the plots of Figs. to 5. Except for Fig. 5, extreme points, up to 0.5% for each axis, were truncated to set the plot range. Word embeddings and KL divergence are very stable for low-frequency words. For this reason, we used 1820 words with n > 103 to fit the sim- ple linear regression model using the least squares method.
(ww) x (nw))
The elements of uwe were initialized by the uniform distribution over 0.5, 0.5] divided by the dimen- sionality of the embedding, and the elements of u. were initialized by zero.
"We manually checked the words used in Table and Table were not personally identifiable or offensive.
B Other quantities of information theory
"We used AMD EPYC 7702 64-Core Processor (64 cores x 2). In this setting, the CPU time is estimated at about 12 hours.
In addition to KL divergence, two other informa- tion theoretic quantities are discussed here.

---

B.1 Shannon entropy
as 'round" in Fig. 5. If there is no rounding errors. p(w" (w) p(w") so that KLo(w) 0. In real- ity, however, (L) non-negligible for words with small nw. and this effect can be corrected by (T](w) (T.((w).
The Shannon entropy of (-(lw), defined as
H(w)= - Σ p(w'l w) log p(w')w), W'EV
also represents information conveyed by In this paper, call it the Shannon entropy of word (w) is closely related to KL(w). The Shannon entropy p(-[w) can be written as
D Details of experiment in Section 7.1
In this experiment, confirmed that human- annotated keywords of documents were observed at the top of the ranking calculated by the discrepancy between p(- (w) and p(-).
(w)) = log V (L(((-[w) un)f
meaning that H (w) measures how much the co- occurrence distribution shifts from the uniform dis- tribution (i.e., unif(w') = 1/IVD. Thus, (w)) and KL(w) have different reference distributions.
Datasets. For the experiment of keyword extrac- tion, we used 15 datasets in English Each entry consists of pair of document and gold keywords. Table includes information on the size (the num- ber of documents) and the type of documents.
B.2 Self-information
A much naive way of measuring the information of a word is the self- oef-iinformtion of the event that the word w is sampled from p(-), defined as
Preparation. Each document in the datasets was tokenized by NLTK's tore_itee func- tion. Then, each word was stemmed using NLTK's PorterStemmer, and all characters were converted to lowercase. The same preprocessing of stem- ming and lowercase was also applied the gold keywords. However, we did not remove stop- words in preprocessing to see if the informativeness measures could remove unnecessary stopwords by themselves. The co-occurrence matrix for each document was computed with the window size h 10. Note that only subset V V of the vocabulary set described below used for stable computation of p(w'l w). w V E V. For constructing V", all the words u E V were sorted in decreasing order of nws and the cumulative fre- quency C: = - Mwy up to the i-th frequent word were computed for i = 1,2, 1,2,....,V0 (V). Then V' = {wi...., } was defined with the smallest i such that 2 N/3.
I (ww) = log p(w).
The expected value Evey p(w) (w) is the Shan- entropy of p(-). Since p(w) was computed as p(w) = The Toe/N,
(ww)= = log N log Thw
actually looks the word frequency nw in the log scale.
B.3 Relation to word embedding
(ww and ((w) were computed with the same set- tings as in Section 4.2 and Appendix A. They were plotted along with ||! as shown in Fig. 7 and Fig. 8, respectively. Compared with KL(w). the re- lationships are less clear with R² x 0.4. From this experiment, we see that KL(w) better represents |l" than H (w) and I(w).
In each document, word ranking lists were created by sorting vocabulary words using the informativeness measures. For 'random', the
Methods.
C Quantization error
The co-oceurrence matrix (Nw wev is sparse with many zero values at rows of w with small nyr The effect of quantization error caused by nu taking only integer values cannot be ig- nored for low-frequency words. This effect is part of the sampling error, but we try to iso- late the quantization error here. Let us redefine nu == round (2hmp((()(((w)) and compute the KL divergence, denoted as KLg(ww, which shown
Datasett for the keyword extraction experiment were obtained from a public repository https://githhe which includes Krapivin2009 (Krapivin et al., 2009), theses100 (Medelyan, 2015). fao780 and fao30 (Medelyan and Witten, 2008), SemEva12010 (Kim et al., 2010), Nguyen2007 (Nguyen and Kan, 2007), PubMed (Aronson et al., 2000), citeu- like180 (Medelyan et al., 2009). wiki20 (Medelyan et al.. 2008). Schutz2008 (Schutz. 2008). kdd (Gollapalli and Caragea, 2014). Inspec (Hulth, 2003), www (Gollapalli and Caragea, 2014), ScmEval2017 (Augenstein et al., 7), and KPCrowd (Marujo et al., 2011).

---

Dataset
Size
Type
random
Tbw
Rwll(w)
x2(w)
m.,L(w)
Krapivin2009
2304
article
0.86
6.17
6.13
8.00
9.59
theses100
100
article
0.97
9.69
9.79
9.31
12.31
fao780
779
article
1.61
11.77
11.84
11.04
15.39
SemEval2010
243
article
1.67
9.52
9.50
8.40
11.10
Nguyen2007
209
article
1.90
10.56
10.57
9.78
12.84
PubMed
500
article
2.89
8.28
8.25
9.91
11.93
citeulike180
183
article
4.01
18.20
18.18
10.03
17.98
wiki20
20
report
4.15
9.32
9.23
12.82
19.90
fao30
30
article
4.92
15.92
17.05
29.47
36.88
Schutz2008
1231
article
8.36
22.32
22.83
13.14
20.93
kdd
755
abstract
10.14
18.27
18.24
9.71
10.08
Inspec
2000
abstract
10.54
16.31
16.22
13.75
14.61
www
1330
abstract
12.08
21.20
21.11
11.67
12.76
SemEval2017
493
paragraph
14.16
19.86
19.62
19.18
20.85
KPCrowd
500
news
39.64
25.73
25.82
39.02
40.47
Table 6: MRR of keyword extraction experiment.
ranking list simply a random shuffle the vocab- ulary words. For Tww (w), words were ranked in increasing order. For other measures, words were ranked in decreasing order. We multiply nww to KT,(W) because G = (www is appropriate for testing the null hypothesis that p(-lw) p(-).
several gold keywords share the same words.
Results. Table shows MRR, and Table shows P@5 of the experiment. Datasets were sorted in the increasing order of MRR of the random baseline in both tables. Table 2 in Section 7. a summary of Table 6. Small values of MRR or P@5 of the random baseline indicate the extent of difficulty of the keyword extraction. Datasets with the article type difficult, and the dataset with the news type is the easiest. In the difficult datasets, n., KLw)) performed best in almost all datasets.
H, (w) is also interpreted as test statistic for testing the null hypothesis that p(-lw) = unif(-).
We also included the x2 statistic (Matsuo and Ishizuka, 2004), which is related to KL(w) as x² = G2 for sufficiently large Nw.
Evaluation metrics. We used MRR and P@5 as evaluation metrics for the keyword prediction task. MRR the average the reciprocals of gold keywords' ranks. The numbers in the tables were multiplied by 100. For each document, we used the best-ranked keyword, i.c., the minimum value of the ranks of correct answers. If keyword is given phrase consisting of two or more words, the rank of the keyword is defined by the worst- ranked word. For example, the rank of "New York" is 10 if the ranks of "new" and "york" are and 10, respectively.
E Details of experiment in Section 7.2.1
In this experiment, we confirmed that proper nouns tend to have larger values of AKT.(w) and A A|!uwl compared to other parts of speech.
Datasets. We used 10561 proper nouns, 123
function words, 4771 verbs, and 2695 adjectives that appeared in the text8 corpus not less than 10 times (nw Z 10). The parts speech these words were identified by NLTK's POS tagger. Proper nouns are tagged as (NN, NNS), verbs are tagged as (VB, VBD, VBG, VBN, VBP, VBZ), adjectives are tagged (JJ, JJS, JJR), ) and function words are tagged (IN, PRP, PRPS, WP, WPS, DT, PDT, WDT, CC, MD, RP). Proper nouns were restricted to those found the 61711 words of the English Proper nouns database".
P@5 is the average percentage of correct an- swers that appear in the top five words of the ranked list. For each document, the number of gold key- words in the top five words was computed and divided by 5. For keyword consisting of two more words, only when all the words are included in the top five words. Thus the percentage can be larger than 100
regarded as a correct answer
httpps: //gi thub. com/ com/jxlwqq/ english- pwrpper-nouss...

---

Dataset
Size
Type
random
(w)
x2(w) (w
Krapivin2009
2304
article
0.11
0.80
0.83
2.37
3.12
theses100
100
article
0.16
3,40
3.60
3.80
5.40
fao780
779
article
0.28
3.70
3.72
3.75
5.52
SemEval2010
243
article
0.23
1.89
1.81
2.63
4.28
Nguyen2007
209
article
0.42
3.44
3.54
4.40
5.74
PubMed
500
article
0.54
2.08
2.00
2.96
3.76
citeulike180
183
article
0.90
12.02
11.69
4.37
8.52
wiki20
20
report
0.70
1.00
1.00
7.00
10.00
fao30
30
article
1.53
9.33
8.67
14.67
18.00
Schutz2008
1231
article
2.37
14.77
15.22
5.20
10.93
kdd
755
abstract
3.07
8.98
9.14
2.12
2.28
Inspec
2000
abstract
2.84
7.32
6.85
5.09
5.68
www
1330
abstract
3.78
10.98
10.89
2.33
3,07
SemEval2017
493
paragraph
4.10
13.35
12.78
8.88
9.33
KPCrowd
500
news
21.75
18.37
18.33
21.25
24.33
Table 7: P@5 of keyword extraction experiment.
(KI(w)
Word Examples
Evaluation metrics. Since the classification de- pends on the threshold value, we used ROC-AUC to evaluate the classification performance. ROC- AUC was computed by Scipy's 's roc_curve function.
HONDA, INTERPOL,
Top (0%~~ ~ 10%)
Gabon, Yin, VAR, IMF, Benin, BO, Bene, GB
Results. Table in Section 7.2.1 shows the ROC- AUC of the classification task, confirming the good performance of AKL(w) and A uw 1|
Pete, Dee, Wine, Tony, Bogart, Alice, Cliff, Madonna, Dover, Leopold
Middle
(45% ~ 55%)
Table 8 shows randomly sampled proper nouns with 10 I The 103 and specific ranges of (KL(w); since our experiment is rase-insensitive, some selected words were actually considered as common nouns, such as storm and haven. We ob- served that common nouns tend to have small KI. values. On the other hand, words with large KL values include context-specific nouns, such com- pany names, suggesting that they are more infor- mative.
storm, haven, sale, miracle, discover, Phillip, duty, prohibition. capitol, comfort
Bottom (90% ~ 100%)
Table Randomly sampled proper nouns each range of informativeness measured by the KL divergence.
Preparation. We computed Nww KL(w) and | from the text8 corpus as described in Ap- pendix A. H (w) was also computed in the same way as KL(w). For their bias-corrected versions, we used the 'shuffle" method in Section 6.1 for AKI(w) and AH and the 'lower percentile' method for A 11 We used these measures for the binary classification of pfrt-of-speech.
F Details of experiment in Section 7.2.2
In this experiment, we confirmed that AKL(w) and A | tend to have smaller value for hypernym in hypernym-hyponym pairs.
Datasets. Among the Hypernym-hyponyee pairs in each dataset, we used those consisting of words that appear in the text8 corpus. Specifically, used 1336 pairs from the 1337 pairs of the BLESS dataset (Baroni and Lenci, 2011), 3635 pairs from the 3637 pairs of the EVALution dataset (San- tus et al., 2015), 1760 pairs from the 1933 pairs of the Lenci/Benotto dataset (Lenci and Benotto,
Methods. Proper nouns tend to have large values of Tew, KL(w) and 12 small values of H (w) in Fig. Therefore, each word classified proper noun if measure is larger (or smaller) than a threshold value. We performed two sets binary classification experiments: proper nouns vs. verbs, and proper nouns vs. adjectives.

---

800
proper nouns
400
STop nouns VOTRO
SPORCT POLRSS
proper TOURS
VOrES
603
200
620
400
200
200
220
100
290
100
10g30
H(W)
KL(w))
||o,||
800
prsper nourT
1292
1500
wetl
peoper nours
700
600
verto
proper nouns
0.00
1000
300
400
500
200
200
100
A/1(w)
AKL(w)
Figure 9: Histogram of each measure used for binary classification of part-of-speech. Plotted for 10561 proper nouns (red) and 4771 verbs (green) in the text8 corpus.
2012), 1427 pairs from the 1427 pairs of the Weeds dataset (Weeds al., 2014). Each dataset was di- vided into two parts: the Phyper Thypo part and the Thyper < Phypo part.
where VwiNwz {w' E V Rui we > 0 A Awzzaw > 0}. w1 is predicted as hypernym if
WeedsPrec(wi, w2)
WeedsPPre(Wu, wi).
Preparation. We computed Thes Durpod H Hww), KL(w), | (H()), AKL(w), and A |u from the text8 corpus described in Appendices and E.
SLQS Row (Shwartz et al., 2017) compares the Shannon entropy. wi predicted as hy- pernym if
SLQS (wi, W2) == 1 H(w)) H(w))
< 0,
Methods. We considered the binary classifica- tion of hypernym given hypernym-hyyonym pair. Using KL(w), 1 AKL(w), A l/ as measure of informativeness, the word with smaller value of the measure was predicted as hy- pernym.
equivalently Hwi) H(2)).
SLQS (Santus et al., 2014) compares the me- dian entropy of context words defined
Baseline methods to predict hypernym given word pair (W1, W2) are described below.
E(w) Medianecc. H(c).
w1 is predicted hypernym if
Random is the random classification. The accuracy 50%.
SLQS(W, w2) 1 F((11) O, F((wz)
Word Frequency chooses the word with larger nw hypernym.
or equivalently F(wi) F(w2)) Note that Co is the set of most strongly associated con- text words of w, as determined by positive local mutual information (Evert, 2005). We used |Cwl 50.
WeedsPrec (Weeds and Weir, 2003; Weeds et al., 2004) based on the distributional inclusion hypothesis that the context of hy- ponym is included in the context of its hyper- nym. The weighted inclusion of word wy in the context of word wi is formulated as
A WredsPrec is the bias-corrected version of WeedsPrec computed by the method in Section 6.2. WeodsPreo(w1,,y) W2) is the average of ftedsPPre((ee (wee(w) W2) for 10 randomly shuf- fled corpora, and A WeeessPrec(w,,, w2) =
En EndeV Dowry
WoodsProo(wj, wz)
Ende Tow.

---

Thyper
Phypo
Thyper
Rhypo
BLESS
EVAL
LB
Weeds
BLESS
EVAL
LB
Weeds 405
average
size
763
2394
1324
1022
573
1241
436
random
50.00
50.00
50.00
50.00
50,00
50.00
50.00
50.00
50.00
frequency
100.00
100.00
100.00
100.00
0.00
0.00
0.00
0.00
50.00
WeedsPrec
93.97
94.78
96.45
95.01
4.54
8.30
8.49
9.14
51.33
SLQS Row
96.46
91.73
96.60
95.99
7.68
21.19
12.84
13.09
54.45
SLQS
87.94
84.04
83.16
75.64
52.53
46.25
40.14
32.35
62.76
KL(w)
98.43
94.74
96.98
95.69
16.93
21.11
16.51
16.79
57.15
Jww
98.17
93.69
94.49
89.92
28.27
27.56
22.25
21.48
59.48
A WWedssPrec
35.78
46.07
50.83
53.42
57.77
49.88
50.00
49.88
49.20
ASLQS Row
57.54
59.19
58.08
64.19
47.64
40.21
41.74
42.96
51.44
ASLQS
55.83
55.93
50.45
39.43
73.30
66.00
72.25
64.69
59.74
(KT(w) 84.80
71.39
58.61
48.83
71.38
56.16
61.24
62.96
64.42
Al/ww]
91.87
75.23
72.73
63.60
74.69
58.26
55.05
59.26
68.84
Table 9: Accuracy hypernym classification. For each method, AMethod is the bias-corrected version. We divided each dataset into two parts based the word frequencies of hypernym (Mayper) and hyponym (Phypo). Dataset EVAL denotes EVALution.
WeodsProo(w), (22) WeedsPreo(w1,, W2).
G Results on Wikipedia dump
ti predicted as hypernym
We used the Wikipedia dump (Wikimedia Foun- dation, 2021) 10 with the size of N 24.0 x 10 tokens and [V = 645 x 10 vocabulary words, which was preprocessed by Wikiextractor (Attardi, 2015). The training of the SGNS model and the computation of KL divergence were performed as in Appendix A using the same setting For plot- ting the results, sampled from the 1, 14,207 vocabulary words with nww 2 10. For fitting the regression line, we used 2,662 words with new 103.
A WeedsPrec(wi,,ue u2)
A WeedsPrec(upg u2 w1).
ASLQS Row is the bias-corrected version of SLQS Row. wn is predicted as hypernym if AH (w1) > (H((w).
used 50,000 words randomly
ASLQS is the bias-corrected version of SLQS. is predicted hypernym if
AE(w)) AE(wz), where
Fig. 10 shows the word embeddings of the Wikipedia dump computed with the same setting that of the text8 corpus. The left panel of Fig. 10 is very similar to Fig. confirming that the result for the text8 corpus reproduced for the Wikipedia dump. The right panel of Fig. 10 corresponds Fig. with the axes exchanged and the log1o Thy axis rescaled. Again, the two plots are very similar.
A (Ew) Median AH(C).
The classification accuracy of each method was computed separately for the Nhyper > Phypo part and for the nhyper Phypo part of each dataset. Then, we calculated the weighted average of the accuracy over the four datasets for each part and for both parts.
Evaluation metrics.
However, the result changes when the epoch of training is reduced, thus the optimization is insuffi- cient. Fig. 11 shows the word embeddings of the Wikipedia dump, but the epoch was reduced to 10.
Results. Table 9 shows the classification accu- racy. Table 4 in Section 7.2.2 is summary of Table 9. Looking at the overall accuracy, A |umll | and AKI.(w) were the best and the second best, respectively, for predicting hypernym in hypemnm-- hyponym pairs.
WWiiipedia dump dataset licensed under the GFDL and the BY-SA
"We used AMD EPYC 7763 (64 cores). For 10 epochs of training, the CPU time is estimated at about 20 hours, and for 100 epnchs of training, the CPU time is estimated at about 8 days.

---

10a
10
0.047x 0.225
R² 0.665
107
100
10c
80
10c
6
KL(w)
10a
60
10a
10a
104
40
2
10a
10a
20
1022
102
0
0
101
10a
25
50
75
100
10'
10*
10
||w!||"
nw
Figure 10: Word embeddings of Wikipedia dump computed with 100 epochs.
10a
10
10
100
10
106
80
10c
6
KL(w)
10
||w,||?
10a
60
10a
104
2
10a
10a
0.055x + 0.100
-102
20
10
0
R²
0.686
101
0
10a
0
25
50
75
100
10a
10
106
||w|||2
nw
Figure 11: Word emheddings of Wikipedia dump computed with 10 epochs.
Pre-trained fastText
Pre-trained SGNS
10
3.0
0.039x 0.076
1.75
0.029x 0.147
10a
R² 0.693
R² 0..733
2.5
1.50
107
10
1.25
2.0
1.00
1.5
10s
10
0.75
1.0
0.50
10
0.5
0.25
0.0
104
0.00
10
20
30 !uwll
40
50
0
10
20
30
40
50
!
Figure 12: Two pre-trained word embeddings. Each regression line was fitted to all the points in the scatterplot.
BERT
RoBERTa
GPT-2
Uama
25
0.183
25
R²
0.557
Ra 0.054
Ra 0.112
cos 0.952
COS 0.977
COS 0.812
COS 0.902
107
100
0
1000
2000
4000 1V
3020 5000
500
2000
1500
52000
120000
150500
12000 33000 30000 XE000 50000
1 6
[4
Figure 13: Linear relationship between the KL divergence and the squared norm of contextualized embedding for BERT, RoBERTa, GPT-2, and Llama 2. The color represents token frequency.

---

In the left panel, the linear relationship was not reproduced. Looking at the right panel, the norm of embedding reduces for low-frequency words with fw < 100; plots of the same shape are also found in the literature (Schakel and Wilson, 2015; Arefyev et al., 2018; Pagliardini et al., 2018; Kho- dak et al., 2018). This is considered consequence of insufficient optimization epochs; the norm of parameters tends to be smaller due to the implicit regularization (Arora et al., 2019), thus the trained parameters do not satisfy the ideal SGNS model (4) very well, particularly for lowfrreqenccy words.
raw
whitened
R²
COS
R²
cos
BERT
0,183
0.952
0.003
0.898
RoBERTa
0.557
0.977
0.196
0.943
GPT-2
0.054
0.812
0.431
0.905
Llama
0.112
0.902
0.127
0.894
Table 10: Linear relationship strength between KL diver- gence and squared norm of language model contextual word embeddings. Presented are coefficients of deter- mination (18²) and uncentered correlation coefficients (cosine similarity) for both raw and whitened embed- dings. Larger values indicate better performance.
Results on pre-trained word embeddings
In this section, we show that the linear relationship between the KL divergence and the squared norm of word embedding holds also for pre-trained word embeddings.
tokens at the beginning and end of tokenized inputs, if any, were excluded.
H.1 Pre-trained fastText embeddings
Results. Looking at the scatterplots in Fig. 13, approximate linear relationships can be observed in BERT, RoBERTa, and Llama 2, but in GPT-2, the linear relationship is somewhat weaker. According to the values in Table 10, whitening improves the linear relationship for GPT-2 and Llama but it worsens for BERT and RoBERTa, and the effect of whitening is not clear-cut. While there still room for discussion, overall, an approximate linear rela- tionship between KI. divergence and the squared norm of contextual embeddings appears to hold.
We used Wiki word vectors provided by Bo- janowski et al. (2017). These 300-dimensional embeddings are trained for 5 epochs on Wikipedia with the fastText model. We used the same KL di- vergence as in Appendix G, which was calculated the Wikipedia dump corpus. Results are shown in the left panel of Figure 12, where we randomly selected 10,000 words that appeared not less than 1 times in the Wikipedia dump.
H.2 Pre-trained SGNS embeddings
We used pre-trained SGNS vectors provided by Li al. (2017). These 500-dimensional embeddings are trained for 2 epochs Wikipedia with the SGNS model. We used the same KL divergence as in Appendix G, which was caleulated on the Wikipedia dump corpus. Results are shown in the right panel of Figure 12, where we randomly se- lected 10,000 words that appeared not less than 10a times in the Wikipedia dump.
Basic properties of the exponential family of distributions
The expectation and covariance matrix. The first and second derivatives of (u) are computed
as
I Results on contextualized embeddings
Settings. For the experiment of contextualized used embeddings obtained
v(u)) θ
Σ
g(w ((w))))
word embeddings, from the final layer of BERT, RoBERTa, GPT-2, and Llama 2. We obtained 2000 sentences from One Billion Word Benchmark (Chelba et al., 2014) and input them into each language model to get contextualized embeddings of all tokens. Special
Iu
Ou
w'CV
=e -((u M(a) E
= =
W'EV
Σ (u))))
wCeV

---

Po(()) ①
dudu = - (c-o(u)
Σ
and substituting into (18), we obtain
du
W'CV
2KL(p(-lun) p( p(-lu2)) = [u1 u2) T G(u1) (u1 - w2),
=c)((u) 0
Σ q( (or
(20)
du
w'EV
showing (9) for i = 1. Considering the Taylor u2, each element of G(u)) is (u)( = Ciz(u2) + O(I - u2 Thus we can rewrite the right hand side of (20) (uy - 2 (G(u2) O[I =zll)) (n) u2) (ui= G((z) ( = There-
De 0e-v(u) v(u)
Σ q( ewr
expansion of G(u) at
du W'EV
m(u) Σ
w W'EV
v(u) -(ua
g(w' q('')u el
fore. we have shown that (9) holds for both and =
du
W'EV
E (Mwll))uwn m())((u)) W'EV
K figh-dimensional random vectors
Σ p((wlu))(u - (( n(u)) W'EV
Random vector setting. In this section, we adopt probabilistic viewpoint and treat the elements of vectors and u' and v² for i = l....,dd d to estimate the orders of magnitude of various quantities, such vector norms, Although the embedding vectors {uw) ] (Ver w are not random variables, the random variable setting is justified when we randomly sam- ple words w and w' from a large corpus and set uy and v Owl To simplify the analysis, we assume that the vector elements are distributed independently. While we could relax this assump- tion by imposing the spherical condition (Jung and Marron, 2009; Aoshima et al., 2018), we leave this extension for future work.
random variables denoted by
showing (6) and (7), respectively.
KI. divergence. For computing the KL. diver-
gence, first note that
log p(w'lu1) p(w'luz)
= =(u1 U2 Uw/) ((1) + ((22)
from (4). Thus, the KL divergence is
(L(p(-lu1) |I p(-lu2)))
Σ p(w'lun) (aus u2, un/) (u1) + ((22))
We aim to discuss the relative magnitudes of vectors, so rescaling the vectors does not affect the argument. Therefore, we assume that each element is proportional to d=1/2, i.c., w' = Op(d-1/2), of Od The squared norm of u | (1) Op(d (/23/) (1), and the Op()). Here (1) means that the magnitude of the vector remains bounded even the dimension d increases. The same applies to e, i.e., ||> Op().. The inner product of and v is also (u, 2) Σd w'v' Op(d (d-1/2)3) = Op(1). Throughout this see- tion, we consider magnitudes up to ((d=)) and ignore higher order terms of Odd33/2) for suffi- ciently large d.
W'V
= (un ((11-u,n((u))) - (((1) + ((22),
(18)
showing (8).
norm itself is also ||u|| = (2)
Approximation of KL divergence. Next, we consider the Taylor expansion of (u)) at u1-
By ignoring higher order terms of O(I un I)), we have
Ou((
('u)= ~ v(u1)
(u - u1)
du
111
S&b((1)) DuduT T
(u- w1)
(u- un).
Inner product with centered vector. Each el- ement of centered vector is u u² Op(d: (d-1/2)2) Op(1). However, the inner
Using (6) and (7), we can rewrite this expression for u = U2 as
b,dd1/2) thus ||u xl|2 Ed
E (u' x²
(u2)) ~ (u1) + (u2 - u1, ((u1))
product
1
+ 2 (u2 u1) [G(()) (u2 u2), (19)
(u ā, =
(21)

---

i.e. it tends to zero as d + xc. Similarly,
the summation, we have
{u, - 0) Op(d-1/3). To show (21), note
{ū 20, v
8)
that B(u² -1) r-) = Ewer (( - -mw) = 0. Thus, E((u' - )'))') = E(u' -T') xT)(("") = 0. The variance is B(((n( - )²) = B[(u²
2
Σ
(uw uo) TT)TGG G(u - uo)p(w)
WEV
a ))))))((())))==
= 0(d-'
= =0(d²)). There-
+ 2
{uw uo, V - v('p((o) 0, (24)
fore, E((u- - d,vl) = 0. and B((u- ū, 2)
WEV
B at )'))") + Eiizy B((u' x") T')e))(((')
1 (u' - x))xt)
Ei= E(((u'
where we have replaced Vn by v to clarify that w' is arbitrary. Here, Ewer(ur - uo) G(uw- uo)p(w) = Op(d=') and EwEr(ur ug, v
T ())+)=
Old.d d 2) + O(d 1). This proves (21).
p)pp() = Opdd"),, thus we have proved (22). In addition to showing (22), we can also obtain an explicit formula for {uD - ū,V
u approximates un. Regarding v, we used only the property 2² = O,(d-1/3 1/2) when deriving (21). So, the result does not change if we replace e by V セ- D: (u ū, V τ) = Op(d 1/2). However, the
8). The second term in (24) is Σwer(
" G(u uo)p(w)
Ewer (uw u0w uo) =o())( trGH, where
result changes if we further replace by uo:
: Σ p(w) - uo) - -uo) (25) WEV
(uo- - u, v - 8) = Op(d'),
(22)
The third term in (24) is Ewev(uw - u0, 2 p)pp(w) = Ewer(" - T) (Uw u0)
meaning that ū approximates ug. To show this, we first prepare another presentation of (5) as follows. Since p(w') = q(w') exp((xo. (xo)))) (5) is expressed as p(w"lu) = p(w") cxp((u uo Uert) eb(u) + ((o)) by canceling out q(w"). We substi- tute (u) by (19) with un = un. u2 = u to obtain
uo (w- = (U-t) -b) H(( - v). Therefore.
we obtain
(wo - ū. 2 - 8) = ( - B) T H(v - 8)
L.GH..
(26)
Interstingly, (26) shows that all the context em- beddings (Tand hotev are constrained to a qudractic surface in R4.
p(w'lu) ~ p(w') exp((u - uo, Uwr 1) - - uo) G(u - uo)). (23)
Proof of (12). First note that
In the above, (u - u0, Enr - 6) = Op(d-1/2). ) and T G(u un) = Eweer(u uo) (Fw
(u uo) TG(u uo)
(u - un)
= (u - u + ū - un) TG - u u ū - -uo)
e) v) T(u (u vo)p(w') = Evrev(u
(u- - ū) G ū) + (ū- (u) TG uo)
no, Vws p)3p(w')
Old 1), because (u uo. Vwa' D) 1/23-1
+ +(u T) G( -w))
Next, we consider (1) and let p(w'lw) ((w/luw) with (23).
Using (22), the magnitude of the remaining
terms obtained as follows. (u - TG
uo) = Ewev( - ua) Two - () 6) (a
p(w'
Σ WEV
pl (uw)p(()))))
no)p(w') = Eweev (ū no, Vw p(') =
((d 1/2) = =0(d Similarry (u 8) G(ū
uo) E Ew'er(u - uT v)(ww( v) (u
= p(w')
Σ exp
(ww - uo, Unr 8)
uo)p(w") = Ewevv (u Uw - (U 0, ū uo)p(w') = O 1/2 d = Odd3/22) There-
WeV
(uw - uo) G(u -
fore, we have shown that
(u - uo) G(u - uo)
This holds for any w', thus Ewev .pp(:....(w))))) x
(u ú) T ū) + O,
1. By considering the Taylor expansion of the summand above, we have exp[--.] 1 + (ww
where the magnitude of (u un) T G(u un) and (u 1)) G(u - u) is Op(d-l), and u is arbitrary Uhy Combining this with (11) proves (12).
uo, Vww -6) { un) G(un uo) 1 (uw
uo Ver 7)2 + Od 3/2). Therefore, by taking

---

L Technical details of the contextualized embeddings
We need only the following additional modifica-
tions. The equation (I) for the unigram distribution
p(w) is replaced by
N
p(-) = p(-w))/N
1=1
The definition (25) for the matrix H in Appendix K is replaced by
U4 u0 uo) /N.
These modifications simply replace the average weighted by word frequency p(w) on the vocabu- lary set V with the simple average over {u: For sufficiently large corpus size N of the train- ing set, the distribution of {u,)/r is approxi- mated by density function (u) of contextual- ized embedding u. Therefore, the simple average is interpreted as the expectation with respect to T (u)). Consequently, we can also employ an alter- nate approach to the definition: ū = u
(/r(()) (u) du, p(-) = J (() du and H f(u (u uo) (u) du.


Reviews:
[{"review_id": "Sm92iOmNBx", "reviewer": "Reviewer_Lew5", "summary": "", "questions": "- In the derivation, (9) ignores the higher order terms with the condition that u_1 is close to u_2. Similarly, (10) considers u_w is close to u_0 with u_0 set to be stopwords? Then, u_0 is replaced by u-head for practical computation. Can we justify that u_w is close to u-head, or even u_0?\n- The derivation shows that KL(w) of a word correlates to the square of the norm of the word embedding with the whitening-like transformation. However, this transformation is not used in the following experiments. The authors should consider adding an explanation on the choice.\n- Is it possible to apply this method for the explainability of (larger) language models? Can we trace the \"strength\" of informativeness of the single embedded tokens through the layers of a network?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\n\n---\n\n# Question about Generality\n\n> It is not clear if the finding still holds for different parameters choices of training the embeddings.\n\nThank you for your suggestion.\n\nWhen training word vectors, one can choose different parameters. I understood your question to be whether our findings are general findings that are robust to such changes. The answer is yes.\n\nFirst, theoretically, the proof consists only of parameter-independent quantities such as inner products and conditional probabilities. The theoretical property holds regardless of dimensionality or word frequency distribution.\n\nEmpirically, experiments have also confirmed the property that the norm of some word vectors learned with multiple parameters (different training corpora, different training algorithms, different dimensionality) encodes the KL divergence of a word (see l.373, Appendix H).\n\nIndeed, generality is a point of interest to many readers. In the camera ready version, I would like to add my reply to Section 4.2, Experiments on other embeddings.\n\n---\n\n# Questions regarding the Contextualized Embeddings and Language Models\n\n> Since most work nowadays focuses on \"contextualized embeddings\", this would be maybe more interesting to a bigger community -- however, using contextualized versions of words might make it more complicated.\n\n> Is it possible to apply this method for the explainability of (larger) language models? Can we trace the \"strength\" of informativeness of the single embedded tokens through the layers of a network?\n\nThank you for your question. We understood your question about whether the norm can be interpreted the same way for embeddings in the language model. Our answer is YES. We have performed additional experiments to confirm this. In addition, we would like to discuss some theoretical implications.\n\n## Additional Experiments\n### Result: \nA linear relationship between the norm of BERT embedding and the KL divergence is confirmed. We report the correlation coefficient between the norm of BERT embedding and the KL divergence. \n\n|  | Layer 0 (static) | Layer 12 (dynamic) |\n|----------|----------|----------|\n| BERT | 0.59 | 0.63 |\n\n\n### Settings: \n\n- KL divergence: KL(word) was computed based on the co-occurrence information of the word in the Wikipedia dump. (The corpus is different, yes, but I used a substitute.) The words are space-separated.\n- BERT embeddings: The BERT embeddings for a word are computed based on the respective tokenizer. For example, if the word is tokenized as wo, ##rd, then the average of the two embeddings is used as the word embedding. The embeddings of a word change depending on the context, so we finally averaged them to compute an embedding for a word.\n\n## Theoretical Implications\n\nSince the language model and SGNS are compatible, we expect that the theory in this paper roughly corresponds to the language model. In the final layer of the language model, the probability of each word is calculated from the embedding u in the form of softmax(W u), so each row of W corresponds to v_{w'} for each word w' in SGNS. Since the probability of a word w' is proportional to exp(<v_{w'}, u> + bias_{w'}), the same form can be obtained by writing bias_{w'} = log(nu q(w')) in the SGNS equation (4). Therefore, the SGNS theory in this paper is expected to be valid for language models as well. \n\nThere are technical differences in the learning process, as described below, and we would like to discuss the details again. \n\n- (Difference 1) In the language model, $u$ is conditioned on word sequences before $w'$, but in SGNS, it is conditioned on a surrounding word. \n- (Difference 2) The language model uses the maximum likelihood method and reinforcement learning, but SGNS uses NCE. \n- (Difference 3) In the language model, bias_{w'} is a tunable parameter, but in SGNS, $\\nu$ and $q(w')$ are constants. \n- (Difference 4) The language model is softmax, but SGNS is trained with the restriction that the denominator of softmax is a constant value.\n\nCertainly the interpretability of language models is an important topic and subject to interests of a wider readers. We thank you for stimulating this discussion! In the camera ready version, we would like to emphasize the result that our analysis method can be applied to language models.\n\n---\n\n# Question about Theoretical Analysis\n\n> In the derivation, (9) ignores the higher order terms with the condition that u_1 is close to u_2. Similarly, (10) considers u_w is close to u_0 with u_0 set to be stopwords? Then, u_0 is replaced by u-head for practical computation. Can we justify that u_w is close to u-head, or even u_0?\n\nThank you for pointing out the unclear points in the paper. In the original paper,  \"close\" was used in two different meanings, which was misleading, so I will explain the difference clearly when revising the manuscript. \n\nIn equation (9), u1 and u2 are \"not very far from each other\" due to the approximation that ignores higher-order terms. On the other hand, in equation (10), the stop words u_w and u_0 are \"very close\" and we assume that the stop words u_w = u_0 with very small errors (which is also experimentally verified). What was difficult to understand in the original manuscript was that equation (11), like equation (9), intended that u_w and u_0 for the general words w were \"not very far from each other\". In the camera ready version, this explanation will be added to clear up this last point.\n\n---\n\n# Question regarding the Whitening-like transformation\n\n> The derivation shows that KL(w) of a word correlates to the square of the norm of the word embedding with the whitening-like transformation. However, this transformation is not used in the following experiments. The authors should consider adding an explanation on the choice.\n\nThank you very much for your suggestion. It would certainly be easier to understand if it is explained why the whitening-like transformation used in the theoretical analysis was not used in the following experiments. The explanation is as follows.\n\nIn the experiment, we used embedding without any transformation to ensure that the word informativeness is encoded in the norm of the word vector that is practically used. The experimental results are comparable for both.\n\n---\n\n# Typos\n\n> ll. 477: \".. explained in Appendix B.\"\n\nThank you for pointing this out, we will fix it in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is set up well and presents a clear story, setting the findings of the authors into context.\n- The authors consider the word-frequency bias of KL divergence and work with a bias-correction version of KL and embedding norm. The bias-corrected squared norm of word embedding shows great improvement compared to its baselines, especially on the proper-noun discrimination tasks.", "reasons_to_reject": "- It is not clear if the finding still holds for different parameters choices of training the embeddings.\n- Since most work nowadays focuses on \"contextualized embeddings\", this would be maybe more interesting to a bigger community -- however, using contextualized versions of words might make it more complicated.", "questions_to_authors": "- In the derivation, (9) ignores the higher order terms with the condition that u_1 is close to u_2. Similarly, (10) considers u_w is close to u_0 with u_0 set to be stopwords? Then, u_0 is replaced by u-head for practical computation. Can we justify that u_w is close to u-head, or even u_0?\n- The derivation shows that KL(w) of a word correlates to the square of the norm of the word embedding with the whitening-like transformation. However, this transformation is not used in the following experiments. The authors should consider adding an explanation on the choice.\n- Is it possible to apply this method for the explainability of (larger) language models? Can we trace the \"strength\" of informativeness of the single embedded tokens through the layers of a network?", "paper_topic_and_main_contributions": "The authors investigate the word2vec skip-gram with negative sampling method on the type of information that is encoded.\nThey find that the squared norm of a vector approximates KL divergence. Further, they show that the squared norm of a word vector and KL divergence with bias correction work similarly well as a measure for the information a word carries. They theoretically corroborate their findings and back them further up with experiments.\nUsing their findings, they show that Kullback-Leibler divergence as well as the squared norm of word vectors provide appropriate measures for the information a word carries wrt to specific tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The paper is very well written and follows a clear structure.\n\nTypos:\n\n- ll. 477: \".. explained in Appendix B.\""}, "extra_text": {}}, {"review_id": "S5yiEcVYQp", "reviewer": "Reviewer_w5CX", "summary": "", "questions": "a. Does the norm of other word embeddings, such as GloVe or the word embedding layer in various deep learning models, also encode information gain?\n\nb. Can the theoretical findings in the paper guide pratical work such as model design or model evaluation in the field of NLP?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\n\n---\n\n# Questions about Other Word Embeddings and Deep Learning Models\n\n> Does the norm of other word embeddings, such as GloVe or the word embedding layer in various deep learning models, also encode information gain?\n\nThank you for your question. Our answer is separated in two parts.\n\n## Other Word Embeddings\n\nWe understood your question as whether the norm can be interpreted in the same way for word embeddings obtained by other methods. It could not be shown in the GloVe case, but it could be shown in fastText. Experiments have also confirmed the property that the norm of some word vectors learned with fastText encodes the KL divergence of a word (see l.373, Appendix H).\n\n## Deep Learning Models\n\nWe understood your question as whether the norm can be interpreted in the same way for embeddings in the language model. Our answer is yes. We have performed additional experiments using BERT to confirm this. In addition, we would like to discuss some theoretical implications.\n\n### Additional Experiments\n\n#### Result: \n\nA linear relationship between the norm of BERT embedding and the KL divergence is confirmed. We report the correlation coefficient between the norm of BERT embedding and the KL divergence.\n\n|         | Layer 0 (static) | Layer 12 (dynamic) |\n| ------- | ---------------- | ------------------ |\n| BERT    | 0.59             | 0.63               |\n\n#### Settings\n- KL divergence: KL(word) was computed based on the co-occurrence information of the word in the Wikipedia dump. (The corpus is different, yes, but I used a substitute.) The words are space-separated.\n\n- BERT embeddings: The BERT embeddings for a word are computed based on the respective tokenizer. For example, if the word is tokenized as wo, ##rd, then the average of the two embeddings is used as the word embedding. The embeddings of a word change depending on the context, so we finally averaged them to compute an embedding for a word.\n\n### Theoretical Implications\n\nSince the language model and SGNS are compatible, we expect that the theory in this paper roughly corresponds to the language model. In the final layer of the language model, the probability of each word is calculated from the embedding u in the form of softmax(W u), so each row of W corresponds to v_{w'} for each word w' in SGNS. Since the probability of a word w' is proportional to exp( + bias_{w'}), the same form can be obtained by writing bias}_{w'} = log(nu q(w')) in the SGNS equation (4). Therefore, the SGNS theory in this paper is expected to be valid for language models as well.\nThere are technical differences in the learning process, as described below, and we would like to discuss the details again.\n\n- (Difference 1) In the language model, $u$ is conditioned on word sequences before $w'$, but in SGNS, it is conditioned on only one surrounding word.\n- (Difference 2) The language model uses maximum likelihood and reinforcement learning, but SGNS uses NCE.\n- (Difference 3) In the language model, $\\mathrm{bias}_{w'}$ is a tunable parameter, but in SGNS, $\\nu$ and $q(w')$ are constants.\n- (Difference 4) The language model is softmax, but SGNS learns with the restriction that the denominator of softmax is a constant value.\n\nCertainly the interpretability of language models is an important topic and subject to interests of a wider readers. We thank you for stimulating this discussion! In the camera ready version, we would like to emphasize the result that our analysis method can be applied to language models.\n\n---\n\n# Question regarding the Practical Applicability\n\n> Can the theoretical findings in the paper guide pratical work such as model design or model evaluation in the field of NLP?\n\nThank you for your question. We think that the norm of word embeddings provides information about the word informativeness, which could be used when interpreting the behavior of the model. How exactly to use it is a subject for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Previous works start from an empirical point of view, and believes that the norm of word embeddings represents the importance of words. In this paper, the authors theoretically analyze the relationship between the norm of word embeddings and the KL divergence of the \nconditional probability distribution of the corresponding words in the context to the marginal distribution of the context. This idea is novel.\n\n- The authors prove their argument from theory and experiments, providing sufficient support for their claim. \n\n- Experiment results in the paper could be easily reproduced.", "reasons_to_reject": "- As stated at the end of the paper, all theoretical analysis and experiments in the paper are limited to SGNS word embeddings. However, the SGNS method has been proposed for 10 years. In addition, the static word embeddings like SGNS and GloVe is not currently the mainstream method of NLP. So the overall contribution maybe relatively limited. \n\n- The authors conducts theoretical analysis and experimental verification of the properties of SGNS word embeddings. Due to space limitations, it is understandable to only conduct theoretical analysis on SGNS. However, whether word embeddings obtained by other methods have similar properties should be verified in the experiments. \n", "questions_to_authors": "a. Does the norm of other word embeddings, such as GloVe or the word embedding layer in various deep learning models, also encode information gain?\n\nb. Can the theoretical findings in the paper guide pratical work such as model design or model evaluation in the field of NLP?", "paper_topic_and_main_contributions": "In this paper, the authors find that the squared norm of SGNS word embeddings encodes the information gain of a word, which is defined by the KL divergence of the conditional probability distribution to the marginal distribution. The authors explain their findings theoretically through properties of the exponential family distribution, and verify their findings experimentally.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mLaPsmUq89", "reviewer": "Reviewer_Jf3f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions for improving the paper.\n\n---\n\n# Question about Generality\n\n> As mentioned in Limitations, it is not clear whether the findings in this paper hold in general.\n\nThank you for your question! The same properties hold true for general settings and language models.\n\n## General Settings\n\nFirst, theoretically, the proof consists only of parameter-independent quantities such as inner products and conditional probabilities. The theoretical property holds regardless of dimensionality or word frequency distribution.\n\nEmpirically, experiments have also confirmed the property that the norm of some word vectors learned with multiple parameters (different training corpora, different training algorithms, different dimensionality) encodes the KL divergence of a word (see l.373, Appendix H).\n\nIndeed, generality is a point of interest to many readers. In the camera ready version, I would like to add my reply to Section 4.2, Experiments on other embeddings.\n\n## Language Models\n\nWe have performed additional experiments to confirm that norm of BERT embedding also encodes information gain. Also, we discuss some theoretical implications.\n\n### Additional Experiments\n\n#### Result: \nA linear relationship between the norm of BERT embedding and the KL divergence is confirmed. We report the correlation coefficient between the norm of BERT embedding and the KL divergence.\n\n|         | Layer 0 (static) | Layer 12 (dynamic) |\n| ------- | ---------------- | ------------------ |\n| BERT    | 0.59             | 0.63               |\n\n#### Settings\n- KL divergence: KL(word) was computed based on the co-occurrence information of the word in the Wikipedia dump. (The corpus is different, yes, but I used a substitute.) The words are space-separated.\n\n- BERT embeddings: The BERT embeddings for a word are computed based on the respective tokenizer. For example, if the word is tokenized as wo, ##rd, then the average of the two embeddings is used as the word embedding. The embeddings of a word change depending on the context, so we finally averaged them to compute an embedding for a word.\n\n### Theoretical Implications\n\nSince the language model and SGNS are compatible, we expect that the theory in this paper roughly corresponds to the language model. In the final layer of the language model, the probability of each word is calculated from the embedding u in the form of softmax(W u), so each row of W corresponds to v_{w'} for each word w' in SGNS. Since the probability of a word w' is proportional to exp( + bias_{w'}), the same form can be obtained by writing bias}_{w'} = log(nu q(w')) in the SGNS equation (4). Therefore, the SGNS theory in this paper is expected to be valid for language models as well.\nThere are technical differences in the learning process, as described below, and we would like to discuss the details again.\n\n- (Difference 1) In the language model, $u$ is conditioned on word sequences before $w'$, but in SGNS, it is conditioned on only one surrounding word.\n- (Difference 2) The language model uses maximum likelihood and reinforcement learning, but SGNS uses NCE.\n- (Difference 3) In the language model, $\\mathrm{bias}_{w'}$ is a tunable parameter, but in SGNS, $\\nu$ and $q(w')$ are constants.\n- (Difference 4) The language model is softmax, but SGNS learns with the restriction that the denominator of softmax is a constant value.\n\nCertainly the interpretability of language models is an important topic and subject to interests of a wider readers. We thank you for stimulating this discussion! In the camera ready version, we would like to emphasize the result that our analysis method can be applied to language models.\n\n---\n\n# Question about Experimental Settings of Keyword Extraction\n\n> In the experiment about keyword extraction, TFIDF is not applied as a baseline measure.\n\nThank you for pointing that out. TFIDF is indeed a basic method for keyword extraction, but we did not use it as a baseline this time because we experimented in a special setting to investigate the specific properties of the KL divergence. We would like to provide an additional explanation for this.\n\nIn order to show that the KL divergence includes the information necessary for keyword extraction, we conducted the experiment using the setting: \"information on a specific document only, without referring to the document set\". Therefore, the TFIDF that refers to the document set was not used as a baseline.\n\nIn the camera ready version, we would like to reflect our reply at the beginning of Section 6.1.\n\n---\n\n# Missing Reference\n\n> The following paper is strongly relative to this paper. Keyword Extraction Using Word Co-occurrence. Christian Wartena, Rogier Brussee, Wout Slakhorst. 2010 Workshops on Database and Expert Systems Applications. https://ieeexplore.ieee.org/document/5592000\n\nThank you for pointing this out. The literature you mentioned is indeed very relevant to the metrics of word informativeness in this paper, since this study is based on the idea that words with special co-occurrences are considered informative words. In the camera ready version, we cite it in Section 2.1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well structured. The claims are clearly explained and easy to read.\n- Discussing from both theoretical and empirical aspects.\n", "reasons_to_reject": "- As mentioned in Limitations, it is not clear whether the findings in this paper hold in general.\n- In the experiment about keyword extraction, TFIDF is not applied as a baseline measure.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses a characteristic analysis of word embeddings. The main findings are as follows:\n- (1) The squared norm of word embedding approximates the information gain.\n- (2) The bias-corrected KL divergence and the norm of word embedding are suitable as a metric of word informativeness.\nThe experiments about three tasks, keyword extraction, proper noun discrimination, and hypernym discrimination, are conducted to verify the second point.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following paper is strongly relative to this paper. \n\nKeyword Extraction Using Word Co-occurrence.\nChristian Wartena, Rogier Brussee, Wout Slakhorst.\n2010 Workshops on Database and Expert Systems Applications.\nhttps://ieeexplore.ieee.org/document/5592000\n"}}]

Author rebuttal:

Meta-review: This paper demonstrates a relationship between the norm of word embeddings, the extent to which a word's co-occurrence probabilities diverge from the unigram distribution ("information gain"), and the informativeness of the word with respect to several tasks. 

The reviewers agreed that the research was sound. Opinions were generally positive on "excitement." One concern, which I shared, was that the results pertain only to static word embeddings. During the discussion period, the authors added a follow-up experiment showing similar results for contextualized embeddings (presumably averaged?).
Decision: Accept-Main