Conditional Word Embedding and Hypothesis Testing via
Bayes-by-Backprop
Rujun Han
Information Sciences Institute
University of Southern California
rujunhan@usc.edu

Michael Gill
Center for Data Science
New York University
mzgill@nyu.edu

Arthur Spirling
Center for Data Science
New York University
arthur.spirling@nyu.edu

Kyunghyun Cho
Center for Data Science
New York University
CIFAR Global Scholar
kyunghyun.cho@nyu.edu

Abstract
Conventional word embedding models do notleverage information from document metadata, and they do not model uncertainty.
We address these concerns with a modelthat incorporates document covariates to estimate conditional word embedding distributions. Our model allows for (a) hypothesis
tests about the meanings of terms, (b) assessments as to whether a word is near or far fromanother conditioned on different covariate values, and (c) assessments as to whether estimated differences are statistically significant.

1

Introduction

Whether a word’s meaning varies across contexts
has become a major focus of NLP, linguistics, andsocial science research in recent years. For example, since the early 20th century, the word “gay”
has evolved from describing an emotion to being more aligned with sexual orientation (Hamilton et al., 2016b). Popular word embedding techniques (e.g., Mikolov et al., 2013a; Pennington
et al., 2014) have proven useful for analyzing language evolution. But to use these models for suchresearch, scholars often divide a corpus into distinct training sets (e.g., train independent language
models on different decades of text) and compare
model output across specifications in an ad hocway (Garg et al., 2018). Such splitting inhibits
many within- and across-word comparisons, sinceembeddings are only comparable within a given
model. Additionally, most methods ignore thevariance of words, mechanically treating words
equally regardless of the volatility, or uncertainty,
in their meanings. If one inspects semantics with
only point estimates of embeddings, it is hard
to tell whether embeddings represent meaningfultraits or are simply noise in the data.
We address these concerns in three ways. First,
we estimate a vector for each distinct value of the

document covariates, using a multilayer perceptron (MLP) with a non-linear activation function.
Second, we parametrize the covariance matrix ofeach embedding vector explicitly in the model,
adopting the Bayes-by-Backprop algorithm (Blundell et al., 2015). Third, we utilize Hotelling T 2
statistics (Hotelling, 1931) to assess whether estimated differences in word vectors are statisticallydifferentiable under a null χ2 distribution (Ito,1956). To our knowledge, no prior work evaluatesword embeddings with this statistical framework.

2

Related Work

Drift Analysis using Word Embeddings There
are several ways to measure drifts in word meanings. Hamilton et al. (2016c) propose the use ofcosine similarities of words in different contexts
to detect changes. Hamilton et al. (2016b) provide an alternative measure based on the distanceof words from their nearest neighbors. Rudolph
and Blei (2018) analyze absolute drift of words
using Euclidean distance in (two discrete) slices
of data. All of these methods compute the word
distance based only on the point (i.e., mean) estimates of the word embeddings.
Conditional Word Embedding Rudolph andBlei (2018) estimate dynamic Bernoulli embeddings (DBE), extending the exponential family
embedding (Rudolph et al., 2016) generalizationof Mikolov et al. (2013a), to learn conditional
word embeddings over time. Their amortizedapproach builds a separate neural network thattransforms a global word vector into a covariatespecific vector, and is closely related to our approach in this paper. However, a noticeable omission in their model is that they do not explicitly
model parameter covariance or uncertainty.
Word Embedding with Uncertainty Vilnis and
McCallum (2017) earlier proposed an energy-

based learning framework in which each word isrepresented as a multivariate Gaussian distribution with a diagonal covariance. The energy function is defined by the divergence (e.g., KL) between two Gaussian embeddings, and the margin ranking loss (Weston et al., 2011) is minimized. A related model is the Bayesian skip-gramin Brazinskas et al. (2017), which posits a generative model where words are associated with multivariate Gaussian latent variables that generate context words. The parameters of those prior distributions over the multivariate Gaussian latent variables are estimated by maximizing the variational
lowerbound, and act as word embeddings.
These works replace mean estimates of embeddings with Gaussian distributions, similar to our
proposal here. However, they arrive at this differently; Vilnis and McCallum (2017) from theenergy-based learning (LeCun et al., 2006), andBrazinskas et al. (2017) from generative modeling.
We provide yet another angle: via (approximate)
Bayesian neural networks.

3

Conditional Word Embedding

Adopting Bayes-by-Backprop for Estimation
Given a tuple of a word v, a covariate x and acontext word vc , we define the conditional logprobability as


X
> c
> c
log p(vc |v, x) = θv|x
θvc − log
exp θv|xθvc0 ,
vc0 ∈V

where θv|x and θvcc are the conditional word embedding of v given x and the context embedding of
vc , respectively. V is the vocabulary of all unique
words. To avoid the expensive computation of
the partition function, we use negative sampling(Mikolov et al., 2013b), which stochastically approximates the log-probability above by:> clog p(vc |v, x) ≈ log σ(θv|x
θvc )

(1)

M

+

1 X
> c
log(1 − σ(θv|xθvcm )),
M
m=1

where vcm ∈ V is the m-th negative sample drawn
from a unigram distribution estimated from D.
We define a prior distribution over each parameter θ to be a scaled mixture of two Gaussians, assuggested by Blundell et al. (2015):
log p(θi ) = log uN (θi |0, σ12 )+(1 − u)N (θi |0, σ22 )

(2)


,

where σ1 , σ2 and u are the hyperparameters.
As exactly marginalizing out the parameters θ·and θ·c is not scalable, we maximize the variational lowerbound of the marginal probability.
To do so, we introduce a variational posteriorq(θ|φ) parametrized by its own parameter set φ.
Then, the variational lowerbound is defined as−F(θ, D) = Eq [logP
p(D|θ)] − KL(q(θ)kp(θ)),
where log p(D|θ) = (v,x,vc )∈D log p(vc |v, x) in
our case. This is stochastically approximated by
M

1 X−F(θ, D) ≈log p(D|θ(m) )
M

(3)

m=1

− log q(θ(m) |φ) + log p(θ(m) ),where θ(m) is the m-th sample from the variational
posterior q (Blundell et al., 2015) via the Gaussian
reparametrization in Kingma and Welling (2013).
We formulate the variational posterior as a multivariate Gaussian with diagonal covariance.
We use stochastic gradient descent (SGD) to
minimize F with respect to the variational parameters φ. At each SGD step, we compute the gradient of the following per-example cost given an
example (v, vc , x) ∈ D:
f (θ, (v,vc , x)) ≈ − log p(vc |v, x) + log q(θ̃v|x |φ)+ log q(θ̃vcc ) + log q(θ̃vcc0 ) − log p(θ̃),
where θ̃ is a single sample from the approximate
posterior, and log p(vc |v, x) and log p(θ̃) are from
Eqs. (1)–(2). We then estimate the (approximate)posterior distribution of each conditional wordembedding θv|x rather than its point estimate, by
minimizing F. See Sec. A of the supplementarymaterial for the detailed steps for computing theper-example cost.
Parametrized Conditional Word Embedding
An issue with the approach described so far is thenumber of parameters grows linearly in the size ofthe vocabulary and in the number of covariate partitions, i.e., O(|V | × |C|), where C is the set of allpartitions. This effectively excludes any potential
sharing of structures underlying words across different covariate values and decreases the numberof examples per parameter. To avoid this issue, we
use a single parametrized function to compute thevariational parameters φ of each conditional wordembedding θv|x .
For each covariate-word v|x, there are two variational parameters µv|x and σv|x . We use an MLP

without any hidden layer and tanh output layer,i.e., the affine transformation followed by pointwise tanh, that takes as input both a global word(v)(x)
vector µv and a covariate
vectorh
i µx and outputs(v) (x)µv|x , i.e., µv|x = fψ ( µv ; µx ), where ψ is theparameters of this mean-transformation network.
The diagonal covariance σv|x is parametrized asσv|x = log(1 + exp(ρv )), where ρv is a parameter shared across all covariate configurations.
Weminimize nF w.r.t.
o these parameters ψ,n then o(x)(v).
and µxµv , ρvv∈V

x∈C

This approach of parametrized conditional wordembeddings significantly reduces the number ofparameters from O(|V | × |C|) to O(|V | + |C|),
while maintaining posterior uncertainty of the estimated conditional word embedding θv|x .

4

Divergences for Word Embeddings

As we estimate the approximate posterior uncertainty of conditional word vectors, we can estimate
richer relations between vectors (e.g., KL) in addition to more common comparisons (e.g., cosine or
Euclidean distance). Moreover, we can explicitly
test for whether two vectors are (un)likely to havethe same mean in the population. Below, we introduce how Hotelling’s T 2 may be used for worddrift or across-word hypothesis testing.
Hotelling’s T 2 Statistic We use the estimatedposterior mean vector µv|x and the diagonal covariance vector σv|x of two word-covariate pairsv|xi and v|xj to compute the T 2 statistic, as if
they were estimates from two sets of samples:
T 2 = (µi − µj )> diag(s)−1 (µi − µj ). The pooled(diagonal) covariance s of word pairs is computed(ni −1)·σ 2 +(nj −1)·σ 2ijby s =, where ni and nj areni +nj −2
the numbers of occurrences of v|xi and v|xj in D,
respectively.1 Unlike other divergence measures,
this T 2 statistic explicitly takes into account thefrequencies of the word-covariate pairs.
Under general conditions, e.g., D is large, thesampling distribution of T 2 converges to a χ2d distribution (Ito, 1956) with d equal to the embedding
dimensionality. This allows us to statistically test
such a null hypothesis as Diff(vi |x, vj |x) = 0 andDiff(v|xi , v|xj ) = 0.

5

Application: Political Speech in UK

Data We use U.K. Parliament speech records
from 1935-2012 as our training data (Rheault
1

T 2 is valid only when ni > 1 and nj > 1.

Figure 1: The ranks of “sterling” (solid line) and“pound” (dotted line) w.r.t. “currency” across thedecades according to KL divergence.
et al., 2016). Our conditioning variable of interestis the decade in which a speech occurred. Moredetails are in Sec. B of the supplementary file.
Model and Learning For each word in the corpus, we consider six surrounding words as its context. The size of embedding is set to 100. We
use six negative samples to compute Eq. (1). We
use Adagrad (Duchi et al., 2011) with the initial
learning rate 0.05 for learning.2 For other hyperparameters, see the supplementary material. We
refer to our approach by BBP. For comparison, we
also train analogous DBE embeddings using codefrom the authors.

6

Result and Analysis

Impact of Covariates To demonstrate how document covariates influence conditional word embeddings, we compare the vector for “currency”
against “sterling” and “pound” according to the
KL divergence in each decade, which is shown in
Fig. 1. In each time period we report the ranking ofeach w.r.t “currency”. Here, we observe that pivotal points for both “sterling” and “pound” occur
in the 1970s, which coincides with the moment the
UK began to abandon the ‘sterling area’ (Part III
in Schenk, 2010). As such, this financial policy
appears to have encouraged semantic drift of theword “pound” towards “currency”. See Sec. D in
the supplementary material for more details.
We also show a few more examples in Figure 2
and Figure 3 from the Dictionary Induction section
below.
Dictionary Induction As a quantitative comparison between the proposed approach and theDBE, we take a dictionary of (British) politicalterms by Laver and Garry (2000) and look at the2

https://github.com/rhan1207/ConditionalEmbeddings

Wordsbenefitchildreneducation
healthtransport

Pro-state
Ranks
1437.42431.8716.4
995.8
4246.6

Pctl
5.7%
9.8%
2.9%
4.0%
17.0%

Words
marketmoneyown
privatevalue

Con-state
Ranks
1782.81623.02851.61670.41693.2

Pctl
7.1%
6.5%
11.4%
6.7%
6.8%

Table 1: Average pair-wise rankings of most fre-quent words in “pro-state” and “con-state” from a
British political dictionary.
Figure 2:

The ranks between “market” and“money” across the decades according to KL divergence.

Words
uk
eu
war
councillabour

DBE
Ranks
1
2
6
8
15

No CovarianceL2
cosine
1.60
0.81
1.58
0.84
1.52
0.85
1.66
0.84
1.63
0.82

Covariance
KL
T2
61.4
99.7
44.6
89.2
48.4
96.8
71.0 142.0∗∗
62.4
124.8∗

Table 2: Top word drifts selected based on DBEmodel and estimated by BBP. * and ** indicate
p-value ≤ 0.05 and 0.01, respectively.

Figure 3: The ranks between “benefit” and “chil-dren” across the decades according to KL divergence.
average pair-wise, directional rank in each category (“pro-state”, “con-state” and “neutral-state”).
We only consider the 2,000 most frequent words inthe vocabulary and embeddings with the covariate (decade) set to 2000s. We observed that the
proposed model using KL divergence has significantly smaller average pair-wise ranks in “prostate” (4052 vs. 5047) and “con-state” (2578 vs.
3758) while performs slightly worse than DBE in“neutral-state” category (5414 vs. 5031) suggesting that the proposal approach can cluster words
from similar semantic group into closer neighbors
than DBE.
Furthermore, we pick 5 most frequent words
from “pro-state” and “con-state” and show theiraverage pair-wise rankings and percentile in Table
1. Out of 25K words, our proposed model is able
to rank most chosen words within top 10% percentile.
Statistical Word Drift Analysis Our BBP approach permits meaningful downstream hypothesis tests of word drift, i.e, Diff(v|xi , v|xj ) = 0,
and across-word similarity, i.e., Diff(vi , vj ) = 0.
Among the 2,000 most frequent words in our sam-ple, we perform hypothesis tests of word drifts,
comparing vectors from the 1940s against thosefrom the 2000s. We compare results from BBP
against the top-100 estimated drifts via DBE. We
first observe that most of the top-ranked words by
L2 distance in the DBE model are not statisticallysignificant. With the p-value threshold of α = 0.1,
only eleven words were deemed to have had significant drift, including “council”, “labour”, “european” and “defence”. Sec. E of the supplement
includes entire lists of this drift analysis.
In Table 1, we show results from five illustrative tests, drawn from the top-100 word drifts estimated by the DBE model. We report words’
drift ranks in DBE against their corresponding
L2 distance, cosine similarity, KL divergence andHotelling T 2 using the embeddings estimated in
our BBP model. Based on the distance metrics that
ignore the covariance matrix, these words do notappear to change much over time as their cosine
similarities are fairly large and their L2 distances
are relatively small with little variation across thefive words. This suggests their mean vectors
are projected into close space between 1940s and2000s. However, by taking into account their uncertainty, we observe greater variation in both KL
divergence and T 2 statistic. For example, “council” has the eighth largest drift in DBE by L2, butshows the largest T 2 statistic among the five words
and is statistically significant at α = 0.01. So too,the largest DBE drift (“uk”) is insignificant once
you take into account the covariance structure.

Figure 4: Semantic Graphs with KLD vs. Cosine Similarity

Cosine Similarity vs. KL Divergence In contrast to cosine distance, our proposed method allows computation of the KLD between two vectors that takes into account their covariance. Figure 2 presents semantic graphs estimated in thespirit of Hamilton et al. (2016a). The set of words
is given by the union set of the 10 nearest neighbors, measured by cosine similarity and KLD,
for the five seed words: “currency”, “british”,“health”, “trade” and “labour”. This results in
130 unique words including the seed words andwe compute their pair-wise KLD matrix, WKL
and pairwise cosine similarity matrix, Wcos . We
0=convert WKL to a symmetric matrix as WKLT(WKL + WKL )/2. Both WKL and Wcos have dimensions of 130 × 130.
Edge weights in Figures 2.A and 2.B are
computed by taking a sigmoid transfor0 , i.e.,mation of normalized entries in WKL0σ(normalize(wKLi,j )). Edge weights in 2.C and
2.D are computed by arccos(wcosi,j ), following
Hamilton et al. (2016a). Edges with weights
below 90th percentiles are dropped for visualclarity. Note that with the same number ofedges being eliminated, the KLD charts appear
more clustered around seed words, implying thatincorporating covariance matrix creates useful
segregation of words within local contexts; graphs
constructed via cosine similarity seem to disperse
edge weights in a more diffuse manner.

T 2 -based

Significance In the context ofuncertainty-aware word embeddings, we can use
the T 2 statistic to filter out additional words froma nearest neighbor set. For instance, in Figure
2.B and 2.D, we drop edges for word pairs thatfall below the 90th percentile of computed T 2
statistics. Filtering with Hotelling T 2 results inmore sparse semantic graphs.

7

Conclusion

We proposed an uncertainty-aware conditional
word embedding model that combines two ideas;(1) variational Bayesian learning for estimating
parameter uncertainty, and (2) structured embeddings conditioned on covariates. This provides aprincipled direction to investigate hypothesis testsof word vectors in various forms. We evaluatedvarious aspects of the proposed approach on U.K.
Parliament speech records from 1935-2012. We
believe the proposed approach will serve as a more
rigorous tool in social science and other domains.

8

Acknowledgments

KC thanks the support by eBay, TenCent, NVIDIA
and CIFAR. RH thanks the support by MINDS research group at Information Sciences Institute of
University of California.

References
Charles Blundell,Julien Cornebise,
Koray
Kavukcuoglu, and Daan Wierstra. 2015. Weight
uncertainty in neural networks. arXiv, 1505.05424.
Arthur Brazinskas, Serhii Havrylov, and Ivan Titov.
2017. Embedding words as distributions with abayesian skip-gram model. arXiv, 1711.11027.
John Duchi, Elad Hazan, and Yoram Singer. 2011.
Adaptive subgradient methods for online learningand stochastic optimization. Journal of Machine
Learning Research, 12(Jul):2121–2159.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, andJames Zou. 2018. Word embeddings quantify 100
years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences (Preprint),
pages 1–10.
William L. Hamilton, Kevin Clark, Jure Leskovec, and
Dan Jurafsky. 2016a. Inducing domain-specific sentiment lexicons from unlabeled corpora. Proceedings of the 2016 Conference on Empirical Methods
in Natural Language Processing, pages 595–605.

William L Hamilton, Jure Leskovec, and Dan Jurafsky.
2016b. Cultural shift or linguistic drift? comparing
two computational measures of semantic change. In
Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on
Empirical Methods in Natural Language Processing, volume 2016, page 2116. NIH Public Access.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky.
2016c. Diachronic word embeddings reveal statistical laws of semantic change. arXiv, 1605.09096v4.
Harold Hotelling. 1931. The generalization of student’s ratio. The Annals of Mathematical Statistics,
2(3):360–378.
Koichi Ito. 1956. Asymptotic formulae for the distribution of hotelling’s generalized t20 statistic. The Annals of Mathematical Statistics, 27(4):1091–1105.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes.
arXiv preprint
arXiv:1312.6114.
Michael Laver and John Garry. 2000. Estimating policy positions from political texts. American Journalof Political Science, pages 619–634.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato,
and F Huang. 2006. A tutorial on energy-basedlearning. Predicting structured data.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. arXiv, 1310.4546.
Jeffrey Pennington, Richard Socher, and Christopher
Manning. 2014. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
Ludovic Rheault, Kaspar Beelen, Christopher
Cochrane, and Graeme Hirst. 2016. Measuring
emotion in parliamentary debates with automatedtextual analysis. PloS one, 11(12):e0168843.
Maja Rudolph and David Blei. 2018. Dynamic embeddings for language evolution. In WWW 2018: The
2018 Web Conference, volume April 23–27, 2018,
pages 1003–1011.
Maja Rudolph, Francisco Ruiz, Stephan Mandt, andDavid Blei. 2016. Exponential family embeddings.
In Advances in Neural Information Processing Systems, pages 478–486.
Catherine R Schenk. 2010. The decline of sterling:
managing the retreat of an international currency,
1945–1992. Cambridge University Press.

Luke Vilnis and Andrew McCallum. 2017. Word
representations via gaussian embedding. arXiv,
1412.6623.
Jason Weston, Samy Bengio, and Nicolas Usunier.
2011. Wsabie: Scaling up to large vocabulary image
annotation. In IJCAI, volume 11, pages 2764–2770.