American Political Science Review (2023) 1–20

doi:10.1017/S0003055422001228© The Author(s), 2023. Published by Cambridge University Press on behalf of the American Political
Science Association. This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://
creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article isproperly cited.

Embedding Regression: Models for Context-Specific Description
and Inference
PEDRO L. RODRIGUEZ New York University, United States
ARTHUR SPIRLING New York University, United States
BRANDON M. STEWART Princeton University, United StatesS

ocial scientists commonly seek to make statements about how word use varies over circumstances—including time, partisan identity, or some other document-level covariate. For example, researchers
might wish to know how Republicans and Democrats diverge in their understanding of the term
“immigration.” Building on the success of pretrained language models, we introduce the à la carte on text
(conText) embedding regression model for this purpose. This fast and simple method produces valid
vector representations of how words are used—and thus what words “mean”—in different contexts. We
show that it outperforms slower, more complicated alternatives and works well even with very few
documents. The model also allows for hypothesis testing and statements about statistical significance. We
demonstrate that it can be used for a broad range of important tasks, including understanding US
polarization, historical legislative development, and sentiment detection. We provide open-source software
for fitting the model.

INTRODUCTION

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University PressAll human communication requires common
understandings of meaning. This is nowheremore true than political and social life, where
the success of an appeal—rhetorical or otherwise—
relies on an audience perceiving a message in theparticular way that the speaker seeks to deliverit. Scholars have therefore spent much effort exploring
the meanings of terms, how those meanings are manipulated, and how they change over time and space.
Historically, this work has been qualitative (e.g., Austin
1962; Geertz 1973; Skinner 1969). But in recent times,
quantitative analysts have turned to modeling andmeasuring “context” directly from natural language(e.g., Aslett et al. 2022; Hopkins 2018; Park, Greene,
and Colaresi 2020).
A promising avenue for such investigations has been
the use of “word embeddings”—a family of techniquesthat conceive of meaning as emerging from the distribution of words that surround a term in text (e.g.,
Mikolov et al. 2013). By representing each word as avector of real numbers and examining the relationships

Pedro L. Rodriguez , Visiting Scholar, Center for Data Science,
New York University, United States; and International Faculty,
Instituto de Estudios Superiores de Administración, Venezuela,
pedro.rodriguez@nyu.edu.
Arthur Spirling , Professor of Politics and Data Science, Department of Politics, New York University, United States, arthur.spirling@nyu.edu.
Brandon M. Stewart , Associate Professor, Sociology and Office ofPopulation Research, Princeton University, United States,
bms4@princeton.edu.
Received: June 26, 2021; revised: February 22, 2022; accepted: October 24, 2022.

between vectors for the vocabulary of a corpus,
scholars have uncovered new facts about language
and the people that produce it (e.g., Caliskan, Bryson,
and Narayanan 2017). This is also true in the study ofpolitics, society, and culture (Garg et al. 2018;
Kozlowski, Taddy, and Evans 2019; Rheault andCochrane 2020; Rodman 2020; Wu et al. 2019).
Although borrowing existing techniques has certainly produced insights, for social scientists two problems remain. First, traditional approaches generally
require a lot of data to produce high-quality representations—that is, to produce embeddings that makesense and connote meaning of terms correctly. The
issue is less that our typical corpora are small—though
they are compared with those on the web-scale collections often used in computer science—and more thatterms for which we would like to estimate contexts are
subject-specific and thus typically quite rare. As an
example, there are fewer than twenty parliamentary
mentions of the “special relationship” between the
United States and the United Kingdom in some yearsof the 1980s—despite this arguably being the highwatermark of elite closeness between the two countries. The second problem is one of inference. Although
representations themselves are helpful, social scientists
want to make statements about the statistical properties
and relationships between embeddings. That is, theywant to speak meaningfully of whether language is useddifferently across subcorpora and whether those apparent differences are larger than we would expect bychance. Neither of these problems are well addressedby current techniques. Although there have been
efforts to address inference in embeddings (see, e.g,
Kulkarni et al. 2015; Lauretig 2019), they are typicallydata intensive and computationally intensive.

1https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

We tackle these two problems together in what
follows. We provide both a statistical framework formaking statements about covariate effects on embeddings and one that performs particularly well in cases ofrare words or small corpora. Specifically, we innovateon Khodak et al. (2018), which introduced à la carte
embeddings (ALC). In a nutshell, the method takes
embeddings that have been pretrained on large corpora(e.g., word2vec or GloVe embeddings readily available
online), combined with a small sample of example uses
for a focal word, and then induces a new contextspecific embedding for the focal word. This requires
only a simple linear transformation of the averaged
embeddings for words within the context of the
focal word.
We place ALC in a regression setting that allows forfast solutions to queries like “do authors with thesecovariate values use these terms in a different way thanauthors with different covariate values? If yes, how dothey differ?” We provide three proofs of concept. First,
we demonstrate the strength of our approach by comparing its performance to the “industry standard” aslaid out by Rodman (2020) in a study of a New YorkTimes API corpus, where slow changes over long
periods are the norm. Second, we show that ourapproach can estimate an approximate embedding
even with only a single context. In particular, we demonstrate that we can separate individual instances of
Trump and trump. Third, we show that our method
can also identify drastic switches in meaning over shortperiods—specifically in our case, for the term Trump
before and after the 2016 election.
We study three substantive cases to show how thetechnique may be put to work. First, we explore partisan differences in Congressional speech—a topic oflong-standing interest in political science (see, e.g.,
Monroe, Colaresi, and Quinn 2008). We show that
immigration is, perhaps unsurprisingly, one of themost differently expressed terms for contemporary
Democrats and Republicans. Our second substantive
case is historical: we compare across polities (andcorpora) to show how elites in the UK and US
expressed empire in the postwar period, how thatusage diverged, and when. Our third case shows how
our approach can be used to measure sentiment. We
build on earlier work (e.g., Osnabrügge, Hobolt, andRodon 2021; Slapin et al. 2018) for the UK House ofCommons, yielding novel insights about the relationship between the UK Prime Minister and his backbenchers on the European Union. We also provide
advice to practitioners on how to use the techniquebased on extensive experiments reported in the Supplementary Materials (SM).
These innovations allow for social scientists to go
beyond general meanings of words to capture situationspecific usage. This is possible without substantial computation and, in contrast to other approaches, requires
only the text immediately around the word of interest.
We proceed as follows: in the next section, we provide some context for what social scientists mean by“context” and link this to the distribution of words
around a focal term. We then introduce the ALC

2algorithm and provide three proofs of concept. Subsequently, we extend ALC to a regression framework andthen present results from three substantive use cases.
We give practical guidance on use and limitations
before concluding.

CONTEXT IN CONTEXT… they are casting their problems on society and who issociety? There is no such thing!
—Margaret Thatcher, interview with
Woman’s Own (1987).

Paraphrased as “there is no such thing as society,”
Thatcher’s quote has produced lively debate in the
study and practice of UK politics. Critics—especiallyfrom the left—argued that this was primarily an
endorsement of individual selfishness and greed. But
more sympathetic accounts have argued that the quotemust be seen in its full context to be understood. The
implication is that reading the line in its original surroundings changes the meaning: rather than embracing
egotism, it emphasizes the importance of citizens’ obligations to each other above and beyond what the state
requires.
Beyond this specific example, the measurement andmodeling of context is obviously a general problem. In
a basic sense, context is vital: we literally cannot understand what is meant by a speaker or author without
it. This is partly due to polysemy—the word “society”
might mean many different things. But the issue isbroader than this and is at the core of human communication. Unsurprisingly then, the study of context has
been a long-standing endeavor in social science. Its
centrality has been emphasized in the history of ideas(Skinner 1969) through the lens of “speech acts”(Austin 1962), describing cultural practices via “thickdescription” (Geertz 1973), understanding “politicalculture” (Verba and Almond 1963), and the psychology of decision making (Tversky and Kahneman 1981).

Approaches to Studying Context
For the goal of describing context in observational data,social science has turned to text approaches—with
topic models being popular (see Grimmer 2010; Quinn
et al. 2010; Roberts, Stewart, and Airoldi 2016). Topic
models provide a way to understand the allocation of
attention across groupings of words.
Although such models have a built-in notion ofpolysemy (a single word can be allocated to different
topics), they are rarely used as a mechanism for studying how individual words are used to convey different
ideas (Grimmer and Stewart 2013). And though topic
approaches do exist that allow for systematic variationin the use of a word across topics by different pieces ofobserved metadata (Roberts, Stewart, and Airoldi2016), they are computationally intensive (especiallyrelative to the approaches we present below). The
common unit of analysis for topic models is thehttps://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Embedding Regression: Models for Context-Specific Description and Inference

document. This has implications for the way that thesemodels capture the logic of the “distributional
hypothesis”—the idea that, in the sense of Firth(1957, 11), “You shall know a word by the company it
keeps”—in other words, that one can understand aparticular version of the “meaning” of a term fromthe way it co-occurs with other terms. Specifically, inthe case of topic models, the entire document is thecontext. From this we learn the relationships (thethemes) between words and the documents in which
they appear.
But in the questions we discuss here, the interest is in
the contextual use of a specific word. To study this,social scientists have turned to word embeddings (e.g.,
Rheault and Cochrane 2020; Rodman 2020). For example, Caliskan, Bryson, and Narayanan (2017) and Garg
et al. (2018) have explored relationships between
words captured by embeddings to describe problematic
gender and ethnic stereotypes in society at large. Word
embeddings predict a focal word as a function of theother words that appear within a small window of that
focal word in the corpus (or the reverse, predict theneighboring words from the focal word). In so doing,
they capture the insight of the distributional hypothesis
in a very literal way: the context of a term are the tokens
that appear near it in text, on average. In practice, this is
all operationalized via a matrix of co-occurrences ofwords that respect the relevant window size. In thelimit, where we imagine the relevant window is theentire document, one can produce a topic model fromthe co-occurrence matrix directly. Thus as the context
window in the embedding model approaches the lengthof the document, the embeddings will increasingly look
like the word representations in a topic model.
Whether, and in what way, embedding models
based on the distributional hypothesis capture“meaning” is more controversial. Here we take a narrow, “structuralist” (in the sense of Harris 1954) view.
For this paper, meaning is in terms of description and isempirical. That is, it arises from word co-occurrences inthe data, alone: we will neither construct nor assume agiven theoretical model of language or cognition. And,in contrast to other scholars (e.g., Miller and Charles
1991), we will make no claims that the distributions per
se have causal effects on human understandings ofterms. Thus, when we speak of the meaning of a focal
word being different across groups, we are talking in athin sense about the distribution of other words within afixed window size of that focal word being different.
Though we will offer guidance, substantive interpretation of these differences for a given purpose is ultimately up to the researcher. That is, as always with suchtext measurement strategies, subject-expert validation
is important.
For a variety of use cases, social scientists want tomake systematic inferences about embeddings—which
requires statements about uncertainty. Suppose wewish to compare the context of “society” as conveyed
by British Prime Ministers with that of US Presidents.
Do they differ in a statistically significant way? To
judge this, we need some notion of a null hypothesis,
some understanding of the variance of our estimates,

and a test statistic. Although there have been efforts tocompare embeddings across groups (Rudolph et al.
2017) and to give frameworks for such conditional
relationships (Han et al. 2018), these are nontrivial to
implement. Perhaps more problematically for mostsocial science cases is that they rely on underlying
embedding models that struggle to produce “good”
representations—that make sense and correctly capture how that word is actually used—when we have few
instances of a term of interest. This matters because weare typically far short of the word numbers that standard models require for optimal performance andterms (like “society”) may be used in ways that areidiosyncratic to a particular document or author.
In the next section, we will explain how we build on
earlier insights from ALC embeddings (Khodak et al.
2018) to solve these problems in a fast, simple, andsample-efficient “regression” framework. Before doing
so, we note three substantive use cases that both motivate the methodological work we do and show its
power as a tool for social scientists. The exercise in allcases is linguistic discovery insofar as our priors are notespecially sharp and the primary value is in stimulating
more productive engagement with the text. Nonetheless, in using the specific approach we outline in thispaper, we will be able to make inferences with attendant statements about uncertainty. In that sense, ourexamples are intended to be illuminating for other
scholars comparing corpora or comparing authors
within a corpus.
Use Case I: Partisan Differences in Word Usage

A common problem in Americanist political science isto estimate partisan differences in the usage of a given
term. Put literally, do Republicans and Democratsmean something different when they use otherwise
identical words like immigration and marriage?
Although there have been efforts to understand differential word rate of use within topics pertaining to theseterms (e.g., Monroe, Colaresi, and Quinn 2008), there
has been relatively little work on whether the samewords appear in different contexts. Below, we use theCongressional Record (Sessions 111–114) as our corpus
for this study (Gentzkow, Shapiro, and Taddy 2018).
This requires that we compare embeddings as a function of party (and other covariates).
Use Case II: Changing UK–US Understandings of“Empire”

The United Kingdom’s relative decline as a GreatPower during the postwar period has been well documented (e.g., Hennessy 1992). One way that we mightinvestigate the timing of US dominance (over the UK,
at least) is to study the changing understanding of theterm Empire in both places. That is, beyond any
attitudinal shift, did American and British policy
makers alter the way they used empire as the century
wore on? If they did, when did this occur? And did theelites of these countries converge or diverge in terms of
their associations of the term? To answer these3

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

questions, we will statistically compare the embedding
for the term Empire for the UK House of Commons(via Hansard) versus the US Congress (via the Congressional Record) from 1935–2010.
Use Case III: Brexit Sentiment from the Backbenches

The UK’s decision to leave the European Union(EU) following the 2016 referendum was momentous(Ford and Goodwin 2017). Although the vote itself was
up to citizens, the build-up to the plebiscite was a matterfor elites; specifically, it was a consequence of theinternal machinations of the parliamentary Conservative Party that forced the hand of their leader, Prime
Minster David Cameron (Hobolt 2016). A natural
question concerns the attitudes of that party in theHouse of Commons toward the EU, both over timeand relative to other issue areas (such as education andhealth policy). To assess that, we will use an embedding
approach to sentiment estimation for single instances ofterms that builds on recent work on emotion in parliament (Osnabrügge, Hobolt, and Rodon 2021). This will
also allow us to contribute to the literature on Memberof Parliament (MP) position taking via speech (see,e.g., Slapin et al. 2018).

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

USING ALC EMBEDDINGS TO MEASURE
MEANINGOur methodological goal is a regression framework forembeddings. By “regression” we mean two related
ideas. Narrowly, we mean that we want to be able toapproximate a conditional expectation function, typically
written E½YjX  where, as usual, Y is our outcome, X is aparticular covariate, and E is the expectations operator.
We want to make statements about how embeddings(our Y) differ as covariates (our X) change. More
broadly, we use “regression” to mean machinery fortesting hypotheses about whether the groups actuallydiffer in a systematic way. And by extension, we wantthat machinery to provide tools for making downstream
comments about how those embeddings differ. In allcases, this will require three related operations:
1. an efficient and transparent way to embed words
such that we can produce high-quality representations even when a given word is rare;
2. given (1), a demonstration that in real problems, asingle instance of a word’s use is enough to produce
a good embedding. This allows us to set up the
hypothesis-testing problem as a multivariate regression and is the subject of the next section;
3. given (1) and (2), a method for making claims about
the statistical significance of differences in embeddings, based on covariate profiles. We tackle that
below.
Ideally, our framework will deliver good representations of meaning even in cases where we have very few
incidences of the words in question. À la carte

4

embeddings (Khodak et al. 2018) promise exactly this.
We now give some background and intuition on thattechnique. We then replicate Rodman (2020)—a recent
study introducing time-dependent word embeddings
for political science—to demonstrate ALC’s efficiency
and quality.

Word Embeddings Measure Meaning throughWord Co-Occurence
Word embeddings techniques give every word a distributed representation—that is, a vector. The length ordimension (D) of this vector is—by convention—
between 100 and 500. When the inner product between
two different words (two different vectors) is high, weinfer that they are likely to co-occur in similar contexts.
The distributional hypothesis then allows us to inferthat those two words are similar in meaning. Althoughsuch techniques are not new conceptually (e.g., Hinton1986), methodological advances in the last decade(Mikolov et al. 2013; Pennington, Socher, and Manning2014) allow them to be estimated much more quickly.
More substantively, word embeddings have beenshown to be useful, both as inputs to supervised learning problems and for understanding language directly.
For example, embedding representations can be used
to solve analogy reasoning tasks, implying the vectors
do indeed capture relational meaning between words(e.g., Arora et al. 2018; Mikolov et al. 2013).
Understanding exactly why word embeddings work
is nontrivial. In any case, there is now a large literature
proposing variants of the original techniques (e.g.,
Faruqui et al. 2015; Lauretig 2019). A few of theseare geared specifically to social science applications
where the general interest is in measuring changes inmeanings, especially via “nearest neighbors” of specificwords.
Although the learned embeddings provide a roughsense of what a word means, it is difficult to use them toanswer questions of the sort we posed above. Consider
our interest in how Republicans and Democrats use thesame word (e.g., immigration) differently. If we train
a set of word embeddings on the entire Congressional
Record we only have a single meaning of the word. We
could instead train a separate set of embeddings—one
for Republicans and one for Democrats—and thenrealign them. This is an extra computational step andmay not be feasible in other use cases where thevocabularies do not have much overlap. We now discuss a way to proceed that is considerably easier.

A Random Walk Theoretical Framework and
ALC Embeddings
The core of our approach is ALC embeddings. The
theory behind that approach is given by Arora et al.
(2016) and Arora et al. (2018). Those papers conceiveof documents being a “random walk” in a discourse
space, where words are more likely to follow otherwords if they are closer to them in an embedding space.
Crucially for ALC, Arora et al. (2018) also proves that
under this model, a particular relationship will follow

Embedding Regression: Models for Context-Specific Description and Inference

for the embedding of a word and the embeddings of thewords that appear in the contexts around it.
To fix ideas, consider the following toy example. Our
corpus is the memoirs of a politician, and we observe
two entries, both mentioning the word “bill”:
1. The debate lasted hours, but finally we [voted on theand it passed] with a large majority.
to be
2. At the restaurant we ran up [a huge winepaid] by our host.
As one can gather from the context—here, the threewords either side of the instance of “bill” in square
brackets—the politician is using the term in two different (but grammatically correct) ways.
The main result from Arora et al. (2018) shows thefollowing: if the random walk model holds, the researcher
can obtain an embedding for word w (e.g., “bill”) by
taking the average of the embeddings of the words
around w (uw Þ and multiplying them by a particularsquare matrix A . That A serves to downweight thecontributions of very common (but uninformative) words
when averaging. Put otherwise, if we can take averages ofsome vectors of words that surround w (based on somepreexisting set of embeddings vw) and if we can find a way
to obtain A (which we will see is also straightforward), we
can provide new embeddings for even very rare words.
And we can do this almost instantaneously.
Returning to our toy example, consider the first, legislative, use of “bill” and the words around it. Suppose wehave embedding vectors for those words from someother larger corpus, like Wikipedia. To keep things
compact, we will suppose those embeddings are all of
three dimensions (such that D = 3), and take the following values:

2

−1:22

32

1:83

32

−0:06

3

2

1:81

32

−1:50

32

−0:12

3

4 1:33 54 0:56 54 −0:73 5 bill 4 1:86 54 −1:65 54 1:63 5:
0:53

−0:81

0:82

voted

on

the

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press|ﬄﬄﬄﬄ{zﬄﬄﬄﬄ}|ﬄﬄﬄﬄ{zﬄﬄﬄﬄ}|ﬄﬄﬄﬄ{zﬄﬄﬄﬄ}

1:57

0:48

−0:17

and

it

passed|ﬄﬄﬄ{zﬄﬄﬄ}|ﬄﬄﬄﬄ{zﬄﬄﬄﬄ}|ﬄﬄﬄﬄ{zﬄﬄﬄﬄ}

Obtaining uw for “bill: simply requires averaging these
vectors and thus
23
0:12
ubill1 = 4 0:50 5,
0:40

senses. Unfortunately, they will not—this is shown
empirically in Khodak et al. (2018) and in our
Trump/trump example below. As implied above, theintuition is that simply averaging embeddings overexaggerates common components associated with frequent (e.g., “stop”) words. So we will need the A
matrix too: it down-weights these directions so they
don’t overwhelm the induced embedding.
Khodak et al. (2018) show how to put this logic into
practice. The idea is that a large corpus (generally the
corpus the embeddings were originally trained on, suchas Wikipedia) can be used to estimate the transformation matrix A. This is a one time cost after which each
new word embedding can be computed à la carte (thusthe name), rather than needing to retrain an entire
corpus just to get the embedding for a single word. Asa practical matter, the estimator for A can be learned
efficiently with a lightly modified linear regression
model that reweights the words by a nondecreasing
function αðÞ of the total instances of each word (nw) in
the corpus. This reweighting addresses the fact thatwords that appear more frequently have embeddingsthat are measured with greater certainty. Thus we learn
the transformation matrix as,
b = argmin
A
A

XW
w=1

αðnw Þ∥vw −Auw ∥22 :(1)

The natural log is a simple choice for αðÞ, and works
b we can introduce new embeddings forwell. Given A,
any word by averaging the existing embeddings for allwords in its context to create uw and then applying theb w . The transformabw = Au
transformation such that v
tion matrix is not particularly hard to learn (it is a linear
regression problem), and each subsequent induced
word embedding is a single matrix multiply.
Returning to our toy example, suppose that web from a large corpus like Hansard or theestimate A
Congressional Record or wherever we obtained theembeddings for the words that surround “bill.” Suppose that we estimate
2

0:81
b = 4 2:02
A
3:14

3:96
4:81
3:81

3
2:86
1:93 5:
1:13

Taking inner products, we have2with the subscript denoting the first use case. We can dothe same for the second case—the restaurant sense of“bill”—from the vectors of a, huge, wine, to, be,and paid. We obtain2
3
0:35
ubill2 = 4 −0:38 5,−0:24
which differs from the average for the first meaning. A
reasonable instinct is that these two vectors should beenough to give us an embedding for “bill” in the two

3
2
3
3:22
−1:91
b  ubill = 4 3:42 5 and vb  ubill = 4−1:58 5:
bbill2 = A
bbill1 = A
v
2
1
2:73
−0:62

These two transformed embeddings vectors are moredifferent than they were—a result of down-weighting
the commonly appearing words around them—but thatis not the point per se. Rather, we expect them to be
informative about the word sense by, for example,
comparing them to other (preestimated) embeddings
in terms of distance. Thus we might find that the nearest
bbill1 areneighbors of v

5

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart
2

3
3:11
legislation = 4 2:52 5
3:38

2

and

3
2:15
amendment = 4 2:47 5,
3:42

bbill2 arewhereas the nearest neighbors of v
3
−1:92
dollars = 4 −1:54 5 and−0:60

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

2

2

3
−1:95
cost = 4 −1:61 5:−0:63

This makes sense, given how we would typically readthe politician’s lines above. The key here is that the
ALC method allowed us to infer the meaning of words
that occurred rarely in a small corpus (the memoirs)
without having to build embeddings for those rarewords in that small corpus: we could “borrow’” andtransform the embeddings from another source. Wellbeyond this toy example, Khodak et al. (2018) findsb a large corpus recovers
empirically that the learned Ain
the original word vectors with high accuracy (greaterthan 0.90 cosine similarity). They also demonstrate that
this strategy achieves state-of-the-art and near state-ofthe-art performance on a wide variety of natural language processing tasks (e.g., learning the embedding ofa word using only its definition, learning meaningful
n-grams, classification tasks, etc.) at a fraction of thecomputational cost of the alternatives.
The ALC framework has three major advantages forour setting: transparency, computational ease, and efficiency. First, compared with many other embedding
strategies for calculating conditional embeddings (e.g.,words over time) the information used in ALC is transparent. The embeddings are derived directly from theadditive information of the words in the context window
around the focal word; there is no additional smoothingor complex interactions across different words. Furthermore, the embedding space itself does not change, itremains fixed to the space defined by the pretrained
embeddings. Second, this same transparency leads tocomputational ease. The transformation matrix A only
has to be estimated once, and then each subsequent
induction of a new word is a single matrix multiply andthus effectively instantaneous. Later we will be able toexploit this speed to allow bootstrapping and permutation procedures that would be unthinkable if there wasan expensive model fitting procedure for each word.
Finally, ALC is efficient in the use of information. Oncethe transformation matrix is estimated, it is only necessary that uw converges—in other words, we only need toestimate a D-dimensional mean from a set of samples. In
the case of a six-word symmetric context window, there
are 12 words total within the context window; thus, foreach instance of the focal word we have a sample of size
12 from which to estimate the mean.
Although Khodak et al. (2018) focused on using the
ALC framework to induce embeddings for rare words
and phrases, we will apply this technique to embed words
used in different partitions of a single corpus or tocompare across corpora. This allows us to capture differences in embeddings over time or by speaker, evenwhen we have only a few instances within each sample.

6

Importantly, unlike other methods, we don’t need an
entirely new corpus to learn embeddings for select focal
words; we can select particular words and calculate(only) their embeddings using only the contexts aroundthose particular words.1 We now demonstrate this powerof ALC by replicating Rodman (2020).2

Proof of Concept for ALC in Small Political
Science Corpora: Reanalyzing Rodman (2020)
The task in Rodman (2020) is to understand changes in
the meaning of equality over the period 1855–2016
in a corpus consisting of the headlines and other summaries of news articles.3 As a gold standard, a subset ofthe articles is hand-coded into 15 topic word categories—of which five are ultimately used in the analysis—andthe remaining articles are coded using a supervised
topic model with the hand-coded data as input. Four
embeddings techniques are used to approximate trends
in coverage of those categories, via the (cosine) distance between the embedding for the word equality
and the embeddings for the category labels. This ischallenging because the corpus is small—the first
25-year slice of data has only 71 documents—and inalmost 30% of the word-slice combinations there arefewer than 10 observations.4
Rodman (2020) tests four different methods by comparing results to the gold standard; ultimately, thechronologically trained model (Kim et al. 2014) is thebest performer. In each era (of 25 years), the model is
fit several times on a bootstrap resampled collection of
documents and then averaged over the resulting solutions (Antoniak and Mimno 2018). Importantly, themodel in period t is initialized with period t–1 embeddings, whereas the first period is initialized with vectors
trained on the full corpus. Even for a relatively small
corpus, this process is computationally expensive, andour replication took about five hours of compute timeon an eight-core machine.
The ALC approach to the problem is simple. For
each period we use ALC to induce a period-specificembedding for equality as well as each of the fivecategory words: gender, treaty, german, race,
and african_american. We use GloVe pretrained
embeddings and the corresponding transformation
matrix estimated by Khodak et al. (2018)—in otherwords, we make use of no corpus-specific informationin the initial embeddings and require as inputs only thecontext window around each category word. Following
Rodman, we compute the cosine similarity between
equality and each of the five category words, for1

For context, there are many approaches in computer science
including anchoring words (Yin, Sachidananda, and Prabhakar2018) and vector space alignment (Hamilton, Leskovec, and Jurafsky2016).
2
Many papers in computer science have studied semantic change (seeKutuzov et al. 2018 for a survey).
3
For replication code and data see Rodriguez, Spirling, and Stewart(2022).
4
We provide more information on the sample constraints in Supplementary Materials, Part A.

Embedding Regression: Models for Context-Specific Description and Inference

FIGURE 1. Replication of Figure 3 in Rodman (2020) Adding ALC Results
ALC

ChronoGS

Race

African American

International Relations

Germany

2

−2

2
0
−2
Gender

1855−
1879
1880−
1904
1905−
1929
1930−
1954
1955−
1979
1980−
2004
2005−
2016

z score normalized model output

0

2
0

1855−
1879
1880−
1904
1905−
1929
1930−
1954
1955−
1979
1980−
2004
2005−
2016

−2

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: ALC = ALC model, CHR = chronological model, and GS = gold standard.

each period. We then standardize (make into z-scores)
those similarities. The entire process is transparent andtakes only a few milliseconds (the embeddings themselves involve six matrix multiplies).
How does ALC do? Figure 1 is the equivalent of
Figure 3 in Rodman (2020). It displays the normalizedcosine similarities for the chronological model (CHR,
taken from Rodman 2020) and ALC, along with thegold standard (GS). We observe that ALC tracks
approximately as well as does Rodman’s chronologicalmodel on its own terms. Where ALC clearly does betteris on each model’s nearest neighbors (Tables 1 and 2): it
produces more semantically interpretable and conceptually precise nearest neighbors than the chronologicalmodel.
We emphasize that in the 1855 corpus, four of the fivecategory words (all except african_american) are
estimated using five or fewer instances. Whereas thechronological model is sharing information acrossperiods, ALC is treating each slice separately, meaningthat our analysis could be conducted effectively witheven fewer periods.
Collectively, these results suggest that ALC is competitive with the current state of the art within the kindof small corpora that arise in social science settings. We
now turn to providing a hypothesis-testing frameworkthat will allow us to answer the types of questions we
introduced above.

TESTING HYPOTHESES ABOUTEMBEDDINGS
Ultimately we want to speak of the way that embeddings differ systematically across levels of covariates.
To do this, we will set up a regression-like framework,where each “observation” is the embedding of a single
word. À la carte embeddings will assist us, but first weshow that it can learn meaningful embeddings from one
example use.

À la Carte on Text Can Distinguish Word
Meanings from One Example UseAbove we explained that ALC averaged pretrained
embeddings and then applied a linear transformation.
This new embedding vector has, say, 300 dimensions,and we might reasonably be concerned that it is too
noisy to be useful. To evaluate this, we need a groundtruth. So we study a recent New York Times corpus;
based on lead paragraphs, we show that we can reliably

7https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

african_american

gendertreatygermanrace

equality

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

equality
theandofwhitesequality
suffragefairness
emancipation
guaranteeingwill
performing
give
blackwellamericanlegislatures
suffrageconstitutions
missouriequalitybritainextraditioninterpolation
ministerrouherequality
extradition
toleration
guaranteeing
speechreichberlinarms
hitlervonvisitfranceeugeniabilateral
relationsenfranchisement
marriagenewmarket
louiseneedenfranchisement
equality
abrogation
discriminations
coquetry

of
the
and
in
to

enacment
abolition
slavery
amendment
abrogation

TABLE 2. Nearest Neighbors for the 2005 Corpus
african_american

gendertreatygermanrace

equality

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

CHR

ALC

crandall’s
costs
unravelingtreatscongresswoman

fidel
equality
cubansnonwhites
lesbians

equality
the
for
of
and

equalityinequalities
inequityinequality
lesbians

narrow
designed
missedassuretrade

equality
affirms
reaffirms
affirming
upholds

maintainshingeholstein’s
equality’skiel

universities
colleges
campusesstriving
schools

universe
1950s
warriors
positspurdy’sequalitysegregation
inequalities
discrimination
inequities

the
for
of
and
to

gendergays
lesbianstransgender
lgbt

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

8

TABLE 1. Nearest Neighbors for the 1855 Corpus

Embedding Regression: Models for Context-Specific Description and Inference

FIGURE 2. Identification of Distinct Clusters
Trump

misclassified

FIGURE 3. Cluster Homogeneity

trump
0.6
Cluster Homogeneity

10

PC2

5

0

0.4

0.2

0.0
lsa

−5

embeddings

bert

alc

Note: Cluster homogeneity (in terms of Trump vs. trump) of
k-means with two clusters of individual term instances embedded
using different methods.
−10
−10

−5

0

5

PC1

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: Each observation represents a single realization of a target
word, either of trump or Trump. Misclassified instances refer toinstances of either target word that were assigned the majority
cluster of the opposite target word.

distinguish Trump the person (2017–2020) from othersense of trump as a verb or noun (1990–2020).5
For each sense of the word (based on capitalization),
we take a random sample with replacement of 400 realizations—the number of instances of trump in our
corpus—from our New York Times corpus and embedthem using ALC. We apply k-means clustering with two
clusters to the set of embedded instances and evaluate
whether the clusters partition the two senses. If ALC
works, we should obtain two separate clouds of points
that are internally consistent (in terms of the word
sense). This is approximately what we see. Figure 2
provides a visualization of the 300-dimensional space
projected to two dimensions with Principal Components Analysis (PCA) and identifying the two clusters
by their dominant word sense. We explicitly mark misclassifications with an x.
To provide a quantitative measure of performance
we compute the average cluster homogeneity: thedegree to which each cluster contains only membersof a given class. This value ranges between 0—bothclusters have equal numbers of both context types—
and 1—each cluster consists entirely of a single context
type. By way of comparison, we do the same exercise
using other popular methods of computing word vectors for each target realization including latent semantic
analysis (LSA), simple averaging of the corresponding5

We used the New York Time‘s developer API to build our corpus.
See https://developer.nytimes.com/docs/articlesearch-product/1/overview.

pretrained embeddings (ALC without transformationby AÞ, and RoBERTa contextual embeddings (Liuet al. 2019).6,7 To quantify uncertainty in our metric,
we use block bootstrapping—resampling individual
instances of the focal word.8 Figure 3 summarizes our
results.
Latent sematic analysis does not fare well in this task;
ALC, on the other hand, performs close to on par withtransformer-based RoBERTa embeddings.9 Simple
averaging of embeddings also performs surprisingly well,coming out on top in this comparison. Does this mean thelinear transformation that distinguishes ALC from simple averaging is redundant? To evaluate this, we look atnearest neighbors using both methods. Table 3 displaysthese results. We observe that simple averaging ofembeddings produces mainly stopwords as nearest
neighbors. The ALC method, on the other hand, outputs
nearest neighbors aligned with the meaning of each term,
Trump is associated with president Trump, whereas
trump is largely associated with its two related othermeanings: a suit in trick-taking games and defeating
someone. This serves to highlight the importance of thelinear transformation A in the ALC method.

6

For LSA we use two dimensions and tf-idf weighting. We foundthese settings produced the best results.
7
RoBERTa is a substantially more complicated embedding methodthat produces contextually specific embeddings and uses word orderinformation.
8Note here that we are treating the A matrix as fixed, and thus we are
not incorporating uncertainty in those estimates. In experiments (seeSupplementary Materials, Part F) we found this uncertainty to beminor and a second-order concern for our applications.
9This may be a result of RoBERTa’s optimization for sentence
embeddings as opposed to embeddings for an individual word.
Nonetheless, it is surprising given that transformer-based models
lead almost every natural language process benchmark task. Evenat comparable performance though, there would be reason not to use
RoBERTa models simply based on computational cost and comparative complexity.

9

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

TABLE 3.

Top 10 Nearest Neighbors Using Simple Averaging of Embeddings and ALC
Trump

trump

Embeddings

ALC

Embeddings

ALC

but
that
the
even
this
because
would
not
which
same

president
impeachingassailingpresident-electimpeachassailed
re-electalludedclintonappointee

but
this
that
even
only
because
so
same
it
the

declarertrumpscolloquies
four-point
upend
sufficeindomitability
topicality
misstep
reprovehttps://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Although this example is a relatively straightforward
case of polysemy, we also know that the meaning of
Trump, the surname, underwent a significant transformation once Donald J. Trump was elected president of
the United States in November 2016. This is a substantially harder case because the person being referred tois still the same, even though the contexts it is employed
in—and thus in the sense of the distributional hypothesis, the meaning—has shifted. But as we show in
Supplementary Materials B, ALC has no problem withthis case either, returning excellent cluster homogeneity and nearest neighbors.
The good news for the Trump examples is that ALC
can produce reasonable embeddings even from single
instances. Next we demonstrate that each of these
instances can be treated as an observation in ahypothesis-testing framework. Before doing so,
although readers may be satisfied about the performance of ALC in small samples, they may wonder
about its performance in large samples. That is, whether
it converges to the inferences one would make from a“full” corpus model as the number of instances
increases; the answer is “yes,” and we provide moredetails in Supplementary Materials C.

À la Carte on Text Embedding Regression
Model: conText
Recall the original statement of the relationshipbetween the embedding of a focal word and theembeddings of the words within its context:
vw = Auw = AEc ½uwc , where the expectation is takenover the contexts, c. Here we note that because thematrix A is constant we can easily swap it into the
expectation and then calculate the resulting expectation conditional on some covariate X : E½Auw jX  . In
particular, this can be done implicitly through a multivariate regression procedure. In the case of word
meanings in discrete subgroups, this is exactly the sameas the use of ALC applied above.
To illustrate our set up, suppose that each vwi is theembedding of a particular instance of a given word insome particular context, like Trump. Each is of somedimension, Dand thus each observation in this setting is
a 1  D embedding vector. We can stack these to
10

produce an outcome variable Y, which is of dimensionsn (the number of instances of a given word) by D. The
usual multivariate matrix equation is thenY = |{z}Xβ þ |{z}
E ,
|{z}
|{z}
nD

npþ1pþ1D

(2)

nD

where X is a matrix of p covariates and includes aconstant term, whereas β is a set of p coefficients and anintercept (all of dimension D). Then E is an error term.
To keep matters simple, suppose that there is aconstant and then one binary covariate indicatinggroup membership (in the group, or not). Then, thecoefficient β0 (the first row of the matrix β) is equivalent
to averaging over all instances of the target word
belonging to those not in the group. Meanwhile, β0 þ
β1 (the second row of β) is equivalent to averaging over
all instances of the target word that belong to the group(i.e., for which the covariate takes the value 1, as
opposed to zero). In the more general case of continuous covariates, this provides a model-based estimateof the embedding among all instances at a given level ofthe covariate space.
The main outputs from this à la carte on text
(conText) embedding “regression” model are• the coefficients themselves, β0 and β1 . These can beused to calculate the estimated embeddings for theword in question. We can take the cosine distance
between these implied embeddings and the (pretrained) embeddings of other words to obtain thenearest neighbors for the two groups.
• the (Euclidean) norms of the coefficients. These will
now be scalars (distances) rather than the vectors of
the original coefficients. In the categorical covariatecase, these tell us how different one group is toanother in a relative sense. Although the magnitudeof this difference is not directly interpretable, we cannonetheless comment on whether it is statisticallysignificantly different from zero. To do this, we use
a variant of covariate assignment shuffling suggested
by Gentzkow, Shapiro, and Taddy (2019). In particular, we randomly shuffle the entries of the Y column
and run the regression many (here 100) times. Eachtime, we record the norms of the coefficients.

Embedding Regression: Models for Context-Specific Description and Inference

We then compute the proportion of those values thatare larger than the observed norms (i.e., with the truegroup assignments). This is the empirical p-value.
Note that, if desired, one can obtain the estimatedsampling distribution (and thus standard errors) of the(normed) coefficients via nonparametric bootstrap. This
allows for comments on the relative size of differences in
embeddings across and within groups as defined by theircovariates. We now show how the conText model may
be used in a real estimation problem.

Our Framework in Action: Pre–Post Election
Hypothesis Testing
We can compare the change in the usage of the word
Trump to the change in the usage of the word Clinton
after the 2016 election. Given Trump won the election
and subsequently became President—a major breakwith respect to his real-estate/celebrity past—we expect
a statistically significant change for Trump relative to
any changes in the usage of Clinton.
We proceed as follows: for each target word-period
combination—Clinton and Trump, preelection(2011–2014) and postelection (2017-2020)—we embedeach individual instance of the focal word from our
New York Times corpus of leading article paragraphs,
and estimate the following regression:

dummy variable equal 1 for 2017–2020 instances of
Trump or Clinton. As before, this is simply aregression-based estimator for the individual subgroups. We will use permutation for hypothesis testing.
Figure 4 plots the norm of the b
β s along with theirbootstrapped 95% CIs. To reiterate, norming means
the coefficient vectors become scalars. The significant
positive value on the Trump  Post_Election coefficient indicates the expected additional shift in theusage of Trump postelection over and above the shift
in the usage of Clinton.
Although this news is encouraging, readers may
wonder how the conText regression model performs
relative to a “natural” alternative—specifically, a fullembeddings model fit to each use of the term bycovariate value(s). This would require the entire corpus (rather than just the instances of Trump andClinton) and would be computationally slow, but
perhaps it would yield more accurate inferences. As
we demonstrate in Supplementary Materials D, inferences are similar and our approach is more stable byvirtue of holding constant embeddings for all nonfocal
words.

RESULTS
We now turn to substantive use cases, beginning withpartisan differences in the United States.

Y ¼ β0 þ β1 Trump þ β2 Post_Election þ β3 Trump
Post_Election þ E,

(3)

where Trump is an indicator variable equal 1 for Trump
instances, 0 otherwise. Likewise Post_Election is a

Partisanship, Ideology and GenderDifferences
We want to evaluate partisan and gender differences in
the usage of a given term in Congress Sessions 111–114

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

FIGURE 4. Relative Semantic Shift from “Trump”

Trump x
Post_Election

***

***

Post_Election

***

Trump

0.0

0.2

0.4

0.6

^

Norm of βsNote: Values are the norm of b
β and bootstrap confidence intervals. See SM Section J for full regression output. *** = statistically significant at
0.01 level.

11

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

FIGURE 5. Differences in Word Meaning by Gender and Party
Republican

Male

^

Norm of β s

0.15

0.10

0.05

also

but

and

abortionmarriage immigration

Note: Generally, different genders in the same party have more similar understanding of a term than the same gender across parties. See
SM Section J for full regression output. All coefficients are statistically significant at 0.01 level.

(Obama years). Our focus is a set of target words
known to be politically charged: abortion, immigration, and marriage. We also include three nonpartisan stopwords—and, the, and but—in ourtarget set as comparison.
We estimate the following multivariate multiple
regression model for each of our words:

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Y = β0 þ β1 Republican þ β2 Male þ E:

(4)

The dependent variable is an ALC embedding of eachindividual realization in the corpus. For the right-handside, we use indicator variables (Republican or otherwise; Male or otherwise). We use permutation toapproximate the null and bootstrapping to quantify
the sampling variance.
Note again that magnitudes have no natural absoluteinterpretation, but they can be compared relatively: that
is, a larger coefficient on X i relative to X j implies thedifference in embeddings for the groups defined by i islarger than the difference in the groups as defined by j.
Our actual results are displayed in Figure 5. The “Male”
coefficient is the average difference across the genderclasses, controlling for party. The “Republican” coefficient is the average difference across the parties,
controlling for gender.
As expected, the differences across parties andacross genders, is much larger for the more politicalterms—relative to function words. But, in addition,embeddings differ more by party than they do bygender. That is, on average, men and women within aparty have more similar understandings of the terms
than men and women across parties.
The “most partisan” target in our set is immigration. Table 4 shows the top 10 nearest neighbors for

12each party. One reading of these nearest neighbors isthat Democrats were pushing for reform of existinglaws, whereas Republicans were mainly arguing for
enforcement. We can corroborate this via the top
nearest contexts—that is, the individual contexts ofimmigration—embedded using ALC—that are closest to each party’s ALC embedding of the term (seeTable 5). This suggests some validity of our generalapproach.
Our approach is not limited to binary covariates.
To illustrate, we regress the target word immigration on the first dimension of the NOMINATE
score10—understood to capture the liberal–conservative spectrum on economic matters (Poole 2005). Thisapproximates a whole sequence of separate embeddings for each speaker, approximated using a line in
the NOMINATE space. We estimate the following
regression:
Y = β0 þ β1 NOMINATE þ E:

(5)

We next predict an ALC embedding for immigration at each percentile of the NOMINATE score andcompute its cosine similarity with a small set of handpicked features. Figure 6 plots these results. Consistent with our results above, we observe how the predicted ALC embedding for immigration is closer toenforce and illegals at higher values of the
NOMINATE score. It is closer to reform andbipartisan at lower values. The feature amend on
the other hand, shows similar values across the fullrange.
10

Downloaded from https://voteview.com/data.

Embedding Regression: Models for Context-Specific Description and Inference

TABLE 4. Top 10 Nearest Neighbors for theTarget Term “Immigration”
Democrats

Republicans

enact, overhauling, reform, legislation,
enacting, overhaul, reforming,
revamp, entitlement, bipartisan
enforce, laws, enact, enacting, legislate,legislations, enforcing, regularize,
immigration, legislation

TABLE 5. Subset of Top Nearest Contexts For
The Target Term “Immigration”
Democrats
this congress to take on comprehensive immigration
reform and fix our broken immigration
should get to work on comprehensive immigration
reform the immigration system we have
Republicans
administration wants to ignore our nation’s
immigration laws and immigration process the
problembroken is the enforcement of our immigration laws andwe have seen that

The Meaning of “Empire”
Recall that our plan for the second case study was tocompare the embedding of Empire in the UK and US
context for the period 1935–2010. In the estimation we
use the top (most frequent) 5,000 tokens of the combined corpora and we estimate a 300-dimensional
GloVe model and corresponding A matrix specific tothe corpus. The multivariate regression analogy ishttps://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Y = β0 þ β1 CongressionalRecord þ E,

(6)estimated for every year of the period. Interest focuseson the (normed) value of β1: when this rises, the use of
Empire is becoming less similar across the corpora(Congress is becoming more distinctive). The timeseries of the β1s is given in Figure 7. The basic summaryis that, sometime around 1949–50, there was a onceand-for-all increase in the distance between US and
UK understandings of Empire. We confirmed thiswith a structural break test (Bai and Perron 1998).
To understand the substance of the change, consider
Figure 8. We report the “most American” and “mostBritish” (with reference to the parliaments) terms fromthe period on either side of the split in the series.
Specifically, we calculate the cosine similarity between
the ALC embedding for Empire and each nearest
neighbor in the UK and US corpus. The x-axis is theratio of these similarities: when it is large (farther to theright), the word is relatively closer to the US understanding of Empire than to the UK one. An asterisk by
the term implies that ratio’s deviation from 1 isstatistically significantly larger than its permuted value,p < 0.01.
The main observation is that in the preperiod, British
and American legislators talk about Empire primarily
in connection with the old European powers—for
example, Britain and France. In contrast, the vocabularies are radically different in the postbreak period.
Whereas the UK parliament continues to talk of the“British” empire (and its travails in “India” and“Rhodesia”), the US focus has switched. For the Americans, understandings of empire are specifically withrespect to Soviet imperial ambitions, and we see this in
the most distinct nearest neighbors “invasion,”“Soviet,” and “communists,” with explicit references
to eastern European nations like “Lithuania.”

Brexit Sentiment from the Backbenches
Our goal is to estimate the sentiment of the Conservative party toward the EU in the House of Commons.
First, the underlying debate text and metadata is from
Osnabrügge, Hobolt, and Rodon (2021), covering theperiod 2001–2019. We are interested in both major
parties of government, Labour and Conservatives.
We divide those parties’ MPs by role: Cabinet(or Shadow Cabinet in opposition) members of thegovernment party are “cabinet,” and all others are“backbenchers,” by definition. We compare policy sentiment in three areas: education (where our term ofinterest is “education”), health (“nhs”), and the EU(“eu”).
In what follows, each observation for us is a representation of the sentiment of a party-rank-month triple
toward a given term. For instance, (the average)
Conservative-backbencher-July 2015 sentiment toward“health.” We describe our approach in SM E; in essence
we measure the inner product between the term ofinterest to the aggregate embeddings of the (positive
and negative) words from a sentiment dictionary(Warriner, Kuperman, and Brysbaert 2013). We thenrescale within party and policy area, obtaining Figure 9.
There, each column is a policy area: education, health,and then the EU. The rows represent the Conservatives
at the top and Labour at the bottom, with the correlation
between Tory backbenchers and cabinet in the middle.
We see an obvious “government versus opposition”
Westminster dynamic: when Labour is in power (so,from the start of the data to 2010), Labour leaders andbackbenchers are generally enthusiastic about government policy. That is, their valence is mostly positive,
which makes sense given almost total government
agenda control (i.e., the policy being discussed is government policy). The Conservatives are the converse:
both elites and backbenchers have negative valence forgovernment policy when in opposition but are much
more enthusiastic when in government. This is true foreducation, and health to a lesser extent. So far, so
expected.
But the subject of the EU (the “eu” column) isdifferent (top right cell). We see that even after theConservatives come to power (marked by the dotted13

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

Median Republican

Median Democrat

illegals

0.4

0.5

enforceamend
0.3

reform0.2

Cosine similarity between
predicted ALC embedding and feature

0.6

FIGURE 6. Cosine Similarity (LOESS Smoothed) between Various Words and “Immigration” at Each
Percentile of NOMINATE Scores

95%

90%

85%

80%

75%

70%

65%

60%

55%

50%

45%

40%

35%

30%

25%

20%

15%

10%

5%

bipartisan

Percentile of DW−NOMINATE(higher values, more Conservative)

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: We mark the median Democrat and median Republican to help calibrate the scale. See SM Section J for full regression output.

black line in 2010), backbench opinion on government
policy toward Europe is negative. In contrast, the Toryleadership are positive about their own policy on thissubject. Only after the Conservatives introduce referendum legislation (the broken vertical line in 2015)
upon winning the General Election do the backbenchers begin to trend positive toward government
policy. The middle row makes this more explicit: thecorrelation between Tory leadership and backbenchsentiment is generally positive or close to zero for
education and health but negative for the EU—that is,moving in opposite directions. Our finding here is that
Cameron never convinced the average Conservative
backbencher that his EU policy was something about
which they should feel positive.
A more traditional approach would be to count the
number of occurrences of terms in the sentiment dictionary and assign each speech a net valence score.
Figure 10 displays that result. Patterns are harder toread. More importantly, only 56% of the terms in the
dictionary occur in the speeches and a full 69% ofspeeches had no overlap with the set of dictionary
terms—and thus receive a score of 0. This contrasts
with the 99% of terms in the dictionary appearing in the
pretrained embeddings, allowing for all speeches to bescored. This is due to the continuity of the embedding
space.

14

ADVICE TO PRACTITIONERS:EXPERIMENTS, LIMITATIONS, CHALLENGES
Our approach requires no active tuning of parameters,
but that does not mean that there are no choices tomake. For example, the end user can opt for different
context window sizes (literally, the number of words oneither side of the target word), as well as different
preprocessing regimes. To guide practice, we now summarize experiments we did on real texts. Below, we use“pretrained” to refer to embeddings that have been fit
to some large (typically on-line) data collection like
Wikipedia. We use “locally fit” to mean embeddings
produced from—that is, vectors learned from—thetexts one is studying (e.g., Congressional debates).
We note that Rodriguez and Spirling (2022) provide
extensive results on this comparison for current models;thus, here we are mostly extending those enquiries toour specific approach. Our full write up can be seen in
Supplementary Materials F–H. The following are themost important results.
First, we conducted a series of supervised tasks,
where the goal is to separate the uses of trump versus
Trump per our example above. We found that removing stopwords and using bigger context windows resultsin marginally better performance. That is, if theresearcher’s goal is to differentiate two separate uses

Embedding Regression: Models for Context-Specific Description and Inference

0.8^
Norm of β

0.7

Structural break

FIGURE 7. Norm of the British and American Difference in Understanding of “Empire,” 1935–2010

0.6

0.5

1935−1936
1937−1938
1939−1940
1941−1942
1943−1944
1945−1946
1947−1948
1949−1950
1951−1952
1953−1954
1955−1956
1957−1958
1959−1960
1961−1962
1963−1964
1965−1966
1967−1968
1969−1970
1971−1972
1973−1974
1975−1976
1977−1978
1979−1980
1981−1982
1983−1984
1985−1986
1987−1988
1989−1990
1991−1992
1993−1994
1995−1996
1997−1998
1999−2000
2001−2002
2003−2004
2005−2006
2007−20082009−20100.4

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: Larger values imply the uses are more different. The dashed lines show the bootstrapped 95% CIs. See SM Section J for full
regression output.

of a term (or something related, such as classifying
documents), more data—that is, larger contexts, lessnoise—make sense. To be candid though, we do notthink such a task—where the goal is a version of
accuracy—is a particularly common one in politicalscience.
We contend a more common task is seeking highquality embeddings per se. That is, vector representations of terms that correctly capture the “true” embedding (low bias) and are simultaneously consistentacross similar specifications (low variance, in terms ofmodel choices). We give many more details in the SM,
but the basic idea here is to fit locally trained embeddings—with context window size 2, 6, and 12—to theCongressional Record corpus (Sessions 107–114). We
then treat those embeddings as targets to be recoveredfrom various ALC-based models that follow, withcloser approximations being deemed better. As an
additional “ground truth” model, we use Stanford
GloVe pretrained embeddings (window size 6, 300
dimensions). We narrow our comparisons to a set of“political” terms as given by Rodriguez and Spirling(2022). We have five lessons from our experiments:
1. Pretraining and windows: Given a large corpus,
local training of a full embeddings model andcorresponding A matrix makes sense. Our suggested
approach can then be used to cheaply and flexibly
study differences across groups. Barring that, using
pretrained embeddings trained on large online corpora (e.g., Stanford GloVe) provides a very reasonable approximation that can be further improved byestimating an A matrix specific to the local corpus. But
again, if data are scarce, using an A matrix trained on
the original online corpus (e.g., Khodak et al.’s 2018 A
in the case of GloVe) leads to very reasonable results.
In terms of context window size, avoid small windows(of size < 5). Windows of size 6 and 12 perform very
similarly to each other and acceptably well in an
absolute sense.
2. Preprocessing: Removing stopwords from contexts
used in estimating ALC embeddings makes very
little difference to any type of performance. In
general, apply the same preprocessing to the ALC
contexts as was applied at the stage of estimating theembeddings and A matrix—for example, if stopwords were not removed, then do not remove stopwords. Stemming/lemmatization does not change
results much in practice.
3. Similarity metrics: The conventional cosine similarity provides interpretable neighbors, but the innerproduct often delivers very similar results.

15

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

FIGURE 8. UK and US Discussions of “Empire” Diverged after 1950

japanese*japan*

italy*

german*germany*

spain*

germans*
italian*france*french*british*britain*Americanindia
europerailwaysafrican*australia*

sharedempirekingdom

continent*colonies*
overseas*
countries*

africa*rhodesia*

territories*

commonwealth*colonial*

BritishCosine similarity ratio (American/British)(a) Pre-1950

sharedAmericancommunists*

soviet*chinese*lithuania*
cuban*
jews*
communist*
communism*moscow*aggression*
russian*revolution*
russia*poland*polish*struggle*

republic*
greek*

socialist
italian*german*centuries*

colonies*
colonial*
territories*
rhodesia*
continent*
africa*india*
europe*african*french*railways*
kingdom*commonwealth*britain*british*australia*

empire*BritishCosine similarity ratio (American/British)
https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press(b) Post-1950
Note: Most US and UK nearest neighbors pre and post estimated breakpoint. * = statistically significant at 0.01 level.

4. Uncertainty: Uncertainty in the calculation of the A
matrix is minimal and unlikely to be consequential
for topline results.
5. Changing contexts over time: Potential changes tocontexts of targets is a second-order concern, at least
for texts from the past 100 years or so.
Before concluding, we note that as with almost all
descriptive techniques, the ultimate substantive interpretation of the findings is left with the researcher tovalidate. It is hard to give general advice on how thismight be done, so we refer readers to two approaches.
First, one can try to triangulate using various types ofvalidity: semantic, convergent construct, predictive,
and so on (see Quinn et al. 2010 for discussion). Second,16

crowdsourced validation methods may be appropriate(see Rodriguez and Spirling 2022; Ying, Montgomery,and Stewart 2021).
Finally, we alert readers to the fact that all of ouranalyses can be implemented using the conText software
package in R (see Supplementary Materials I andhttps://github.com/prodriguezsosa/conText).

CONCLUSION“Contextomy”—the art of quoting out of context toensure that a speaker is misrepresented—has a longand troubling history in politics (McGlone 2005).
It works because judicious removal of surrounding text

Embedding Regression: Models for Context-Specific Description and Inference

FIGURE 9. Conservative Backbenchers Were Unsatisfied with Their Own Government’s EU Policy
Prior to the Referendum

eu
Referendum bill

Valence = 0

nhs

Correlation

Correlation = 0

Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019
Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019
Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019

Valence(Labour)Correlation(Conservative)

Valence(Conservative)

education

CabinetElection

Backbenchers

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: Each column of the plot is a policy area (with the seed word used to calculate sentiment). Those areas are education (education),
health (nhs), and the EU (eu). Note the middle-right plot: rank-and-file Conservative MP sentiment on EU policy is negatively correlated withthe leadership’s sentiment.

can so quickly alter how audiences perceive a central
message. Understanding how context affects meaningis thus of profound interest in our polarized times. But it
is difficult—to measure and model. This is especiallytrue in politics, where our corpora may be small and ourterm counts low. Yet we simultaneously want statisticalmachinery that allows us to speak of statistically significant effects of covariates. This paper begins to addressthese problems.
Specifically, we proposed a flexible approach to
study differences in semantics between groups and overtime using high-quality pretrained embeddings: theconText embedding regression model. It has advantages over previous efforts, and it can reveal new things
about politics. We explained how controversial terms
divide parties not simply in the way they are attached to
topics of debate but in their very meaning. Similarly, we
showed that understandings of terms like “empire” arenot fixed, even in the relatively short term, and instead
develop in line with interests in international relations.
We showed that our approach can be used to measure
sentiment toward policy. It is not hard to imagine otherapplications. For example, there is evidence that votersprefer broad-based appeals (Hersh and Schaffner2013), but these are only possible in cases where meanings are sufficiently similar within groups. Our technique could be used to explore this tension. Similarly,
what is deemed the “correct” interpretation of treaties(e.g., Simmons 2010) or constitutions matters. Our
methods could help structure studies of these changes.
We built our framework on the ALC embedding
strategy. But our general approach is not inextricably
connected to this particular method for estimating
contextually specific meanings. We used it because it
is transparent, efficient, and computationally simple.
We introduced a regression framework for understanding word meanings using individual instance embeddings as observations. This may be easily extended tomore complex functional forms.
There are many potential directions for the framework; we highlight two. First, ALC assumes that themeaning of nonfocal words is essentially constant. This
first-order approximation could be extended withsecond-order information—which words co-occur withwords that co-occur with the focal words—but it is unclear
how much meaning would have to change across groups
for this to matter. Second, we are estimating means inhigh dimensions using only a few data points. This is17

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

FIGURE 10. Replication of Figure 9 Using a Dictionary Approach

nhs

eu

Election

Valence = 0

Conservative

Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019
Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019
Jun 2001
Jul 2002
Sep 2003
Oct 2004
Dec 2005
Feb 2007
Apr 2008
Jun 2009
Sep 2010
Sep 2011
Oct 2012
Nov 2013
Dec 2014
Feb 2016
Mar 2017
May 2018
Jun 2019

LabourValence(scaled within party and target word)education

CabinetReferendum bill

Backbenchers

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

Note: The sentiment patterns are less obvious.

always difficult (see Gentzkow, Shapiro, and Taddy
2019), and our estimates of the norms have a finitesample bias for rare words. Thus care is needed whencomparing words or groups with substantially different
amounts of available data. Future work could consider
the role of term frequency in these measures of meaning.
As social scientists develop further methods to study
these problems, this will sharpen questions which will inturn spur better methods. But to reiterate, technicalmachinery cannot, of itself, answer substantive questions. That is, claims about meaning must be validated,
and the way that differences in measured quantities areinterpreted will always be subject to debate. We hopethat the conText model that we have laid out here canprovide a useful foundation for future work.

American Political Science Review Dataverse: https://doi.org/10.7910/DVN/NKETXF.

ACKNOWLEDGMENTS
We thank audience members at the Midwest Political
Science Association Annual Meeting (2021), the Political Methodology society meeting, the American Political Science Association Annual Meeting (2021),
Vanderbilt University’s Data Science Institute, Princeton University, and the University of Wisconsin(Madison). We are grateful to Clark Bernier, Saloni
Bhogale, Max Goplerud, Justin Grimmer, Alex Kindel,
Hauke Licht, John Londregan, Walter Mebane, andMolly Roberts for comments. We also thank the editor
and four excellent anonymous reviewers for their careful engagement with our work.

SUPPLEMENTARY MATERIALS
To view supplementary material for this article, please
visit http://doi.org/10.1017/S0003055422001228.

DATA AVAILABILITY STATEMENT
Research documentation and data that support thefindings of this study are openly available at the18

FUNDING STATEMENTWork done at Princeton University is in part supportedby The Eunice Kennedy Shriver National Institute ofChild Health & Human Development of the National
Institutes of Health under Award Number
P2CHD047879.

Embedding Regression: Models for Context-Specific Description and Inference

CONFLICT OF INTEREST
The authors declare no ethical issues or conflicts ofinterest in this research.

ETHICAL STANDARDSThe authors affirm this research did not involve humansubjects.

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

REFERENCES
Antoniak, Maria, and David Mimno. 2018. “Evaluating the Stabilityof Embedding-Based Word Similarities.” Transactions of the
Association for Computational Linguistics 6:107–19.
Arora, Sanjeev, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej
Risteski. 2016. “A Latent Variable Model Approach to Pmi-Based
Word Embeddings.” Transactions of the Association for
Computational Linguistics 4:385–99.
Arora, Sanjeev, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej
Risteski. 2018. “Linear Algebraic Structure of Word Senses, with
Applications to Polysemy.” Transactions of the Association for
Computational Linguistics 6:483–95.
Aslett, Kevin, Nora Webb Williams, Andreu Casas, Wesley
Zuidema, and John Wilkerson. 2022. “What Was the Problem in
Parkland? Using Social Media to Measure the Effectiveness of
Issue Frames.” Policy Studies Journal 50 (1): 266–89.
Austin, John Langshaw. 1962. How to Do Things with Words.
Oxford: Oxford University Press.
Bai, Jushan, and Pierre Perron. 1998. “Estimating and Testing Linear
Models with Multiple Structural Changes.” Econometrica 66 (1):47–78.
Caliskan, Aylin, Joanna J. Bryson, and Arvind Narayanan. 2017.
“Semantics Derived Automatically from Language Corpora
Contain Human-Like Biases.” Science 356 (6334): 183–86.
Faruqui, Manaal, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer,
Eduard Hovy, and Noah A. Smith. 2015. “Retrofitting Word
Vectors to Semantic Lexicons.” In Proceedings of the 2015
Conference of the North American Chapter of The Association for
Computational Linguistics: Human Language Technologies, eds.
Rada Mihalcea, Joyce Chai, and Anoop Sarkar, 1606–15.
Cedarville, OH: Association for Computational Linguistics.
Firth, John Rupert. 1957. Studies in Linguistic Analysis. Hoboken,
NJ: Wiley-Blackwell.
Ford, Robert, and Matthew Goodwin. 2017. “Britain after Brexit: A
Nation Divided.” Journal of Democracy 28 (1): 17–30.
Garg, Nikhil, Londa Schiebinger, Dan Jurafsky, and James Zou.
2018. “Word Embeddings Quantify 100 Years of Gender andEthnic Stereotypes.” Proceedings of the National Academy of
Sciences 115 (16): E3635–44.
Geertz, Clifford. 1973. “Thick Description: Toward an Interpretive
Theory of Culture.” Turning Points in Qualitative Research: Tying
Knots in a Handkerchief 3:143–68.
Gentzkow, Matthew, Jesse M. Shapiro, and Matt Taddy. 2019.
“Measuring Group Differences in High-Dimensional Choices:
Method and Application to Congressional Speech.” Econometrica
87 (4): 1307–40.
Gentzkow, Matthew, J. M. Shapiro, and Matt Taddy. 2018.
Congressional Record for the 43rd–114th Congresses: Parsed
Speeches and Phrase Counts [computer file]. Palo Alto, CA:
Stanford Libraries [distributor], 2018-01-16. https://data.stanford.edu/congress_text.
Grimmer, Justin. 2010. “A Bayesian Hierarchical Topic Model for
Political Texts: Measuring Expressed Agendas in Senate Press
Releases.” Political Analysis 18 (1): 1–35.
Grimmer, Justin, and Brandon M. Stewart. 2013. “Text as Data: The
Promise and Pitfalls of Automatic Content Analysis Methods for
Political Texts.” Political Analysis 21 (3): 267–97.

Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016.
“Diachronic Word Embeddings Reveal Statistical Laws of
Semantic Change.” In Proceedings of the 54th Annual Meeting of
The Association for Computational Linguistics (Volume 1: Long
Papers), eds. Katrin Erk and Noah A. Smith, 1489–501. Cedarville,
OH: Association for Computational Linguistics.
Han, Rujun, Michael Gill, Arthur Spirling, and Kyunghyun Cho.
2018. “Conditional Word Embedding and Hypothesis Testing via
Bayes-by-Backprop.” In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, eds. Ellen
Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii,
4890–95. Cedarville, OH: Association for Computational
Linguistics.
Harris, Zellig S. 1954. “Distributional Structure.” Word 10 (2–3):
146–62.
Hennessy, Peter. 1992. Never Again: Britain 1945-1951. London:
Penguin UK.
Hersh, Eitan D., and Brian F. Schaffner. 2013. “Targeted Campaign
Appeals and the Value of Ambiguity.” The Journal of Politics
75 (2): 520–34.
Hinton, Geoffrey E. 1986. “Learning Distributed Representations ofConcepts.” In Proceedings of The Eighth Annual Conference of the
Cognitive Science Society, Vol. 1, 1–12. Amherst, MA: Cognitive
Science Society.
Hobolt, Sara B. 2016. “The Brexit Vote: A Divided Nation, a Divided
Continent.” Journal of European Public Policy 23 (9): 1259–77.
Hopkins, Daniel J. 2018. “The Exaggerated Life of Death Panels?
The Limited but Real Influence of Elite Rhetoric in the 2009–2010
Health Care Debate.” Political Behavior 40 (3): 681–709.
Khodak, Mikhail, Nikunj Saunshi, Yingyu Liang, Tengyu Ma,
Brandon M. Stewart, and Sanjeev Arora. 2018. “A La Carte
Embedding: Cheap but Effective Induction of Semantic Feature
Vectors.” In Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics, eds. Iryna Gurevych
and Yusuke Miya, 12–22. Cedarville, OH: Association for
Computational Linguistics.
Kim, Yoon, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and SlavPetrov. 2014. “Temporal Analysis of Language through Neural
Language Models.” In Proceedings of the ACL 2014 Workshop on
Language Technologies and Computational Social Science, eds.
Cristian Danescu-Niculescu-Mizil, Jacob Eisenstein, Kathleen
McKeown, and Noah A. Smith, 61–5. Cedarville, OH: Association
for Computational Linguistics.
Kozlowski, Austin C., Matt Taddy, and James A. Evans. 2019. “The
Geometry of Culture: Analyzing the Meanings of Class throughWord Embeddings.” American Sociological Review 84 (5): 905–49.
Kulkarni, Vivek, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena.
2015. “Statistically Significant Detection of Linguistic Change.” In
Proceedings of the 24th International Conference on World Wide
Web, gen. chairs. Aldo Gangemi, Stefano Leonardi, andAlessandro Panconesi, 625–35. Geneva: International World Wide
Web Conferences Steering Committee and Republic and Cantonof Geneva, Switzerland.
Kutuzov, Andrey, Lilja Øvrelid, Terrence Szymanski, and Erik
Velldal. 2018. “Diachronic Word Embeddings and Semantic Shifts:
A Survey.” In Proceedings of the 27th International Conference On
Computational Linguistics, eds. Emily M. Bender, Leon
Derczynski, and Pierre Isabelle, 1384–97. Cedarville, OH:
Association for Computational Linguistics.
Lauretig, Adam. 2019. “Identification, Interpretability, and Bayesian
Word Embeddings.” In Proceedings of the Third Workshop on
Natural Language Processing and Computational Social Science,eds. Svitlana Volkov, David Jurgens, Dirk Hovy, David Bamman,
and Oren Tsur, 7–17. Cedarville, OH: Association for
Computational Linguistics.
Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi,
Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, andVeselin Stoyanov. 2019. “Roberta: A Robustly Optimized BERT
Pretraining Approach.” Preprint, submitted on July 26, 2019.
https://arxiv.org/abs/1907.11692.
Mcglone, Matthew S. 2005. “Contextomy: The Art of Quoting out ofContext.” Media, Culture & Society 27 (4): 511–22.
Mikolov, Tomás, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff
Dean. 2013. “Distributed Representations of Words and Phrases

19

Pedro L. Rodriguez, Arthur Spirling, and Brandon M. Stewart

https://doi.org/10.1017/S0003055422001228 Published online by Cambridge University Press

and Their Compositionality.” In Advances in Neural Information
Processing Systems 26, eds. C. J. C. Burges, L. Bottou, M. Welling,
Z. Ghahramani, and K. Q. Weinberger, 3111–19. Red Hook NY:
Curran Associates, Inc.
Miller, George A., and Walter G. Charles. 1991. “Contextual
Correlates of Semantic Similarity.” Language and Cognitive
Processes 6 (1):1–28.
Monroe, Burt L., Michael P. Colaresi, and Kevin M. Quinn. 2008.
“Fightin’Words: Lexical Feature Selection and Evaluation for
Identifying the Content of Political Conflict.” Political Analysis 16(4): 372–403.
Osnabrügge, Moritz, Sara B. Hobolt, and Toni Rodon. 2021. “Playing
to the Gallery: Emotive Rhetoric in Parliaments.” American
Political Science Review 115 (3): 885–99.
Park, Baekkwan, Kevin Greene, and Michael Colaresi. 2020.
“Human Rights Are (Increasingly) Plural: Learning the Changing
Taxonomy of Human Rights from Large-Scale Text Reveals
Information Effects.” American Political Science Review 114 (3):888–910.
Pennington, Jeffrey, Richard Socher, and Christopher D. Manning.
2014. “Glove: Global Vectors for Word Representation.” In
Proceedings of the 2014 Conference on Empirical Methods in
Natural Language Processing, eds. Lucia Specia and Xavier
Carreras, 1532–43. Cedarville, OH: Association for Computational
Linguistics.
Poole, Keith T. 2005. Spatial Models of Parliamentary Voting.
Cambridge: Cambridge University Press.
Quinn, Kevin M., Burt L. Monroe, Michael Colaresi, Michael H.
Crespin, and Dragomir R. Radev. 2010. “How to Analyze Political
Attention with Minimal Assumptions and Costs.” American
Journal of Political Science 54 (1): 209–28.
Rheault, Ludovic, and Christopher Cochrane. 2020. “Word
Embeddings for the Analysis of Ideological Placement in
Parliamentary Corpora.” Political Analysis 28 (1): 112–33.
Roberts, Margaret E., Brandon M. Stewart, and Edoardo M Airoldi.
2016. “A Model of Text for Experimentation in the Social
Sciences.” Journal of the American Statistical Association
111 (515): 988–1003.
Rodman, Emma. 2020. “A Timely Intervention: Tracking theChanging Meanings of Political Concepts with Word Vectors.”
Political Analysis 28 (1): 87–111.

20

Rodriguez, Pedro L., Arthur Spirling, and Brandon M. Stewart. 2022.
“Replication Data for: Embedding Regression: Models forContext-Specific Description and Inference.” Harvard Dataverse.
Dataset. https://doi.org/10.7910/DVN/NKETXF.
Rodriguez, Pedro L., and Arthur Spirling. 2022. “Word Embeddings:
What Works, What Doesn’t, and How to Tell the Difference forApplied Research.” The Journal of Politics 84 (1): 101–15.
Rudolph, Maja, Francisco Ruiz, Susan Athey, and David Blei. 2017.
“Structured Embedding Models for Grouped Data.” In Advances in
Neural Information Processing Systems, Vol. 30, eds. I. Guyon, U.
Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan,and R. Garnett. Red Hook, NY: Curran Associates, Inc.
Simmons, Beth. 2010. “Treaty Compliance and Violation.” Annual
Review of Political Science 13:273–96.
Skinner, Quentin. 1969. “Meaning and Understanding in the Historyof Ideas.” History and Theory 8 (1): 3–53.
Slapin, Jonathan B., Justin H. Kirkland, Joseph A. Lazzaro,
Patrick A. Leslie, and Tom O’Grady. 2018. “Ideology,
Grandstanding, and Strategic Party Disloyalty in the British
Parliament.” American Political Science Review 112 (1): 15–30.
Tversky, Amos, and Daniel Kahneman. 1981. “The Framing ofDecisions and the Psychology of Choice.” Science 211 (4481): 453–58.
Verba, Sidney, and Gabriel Almond. 1963. The Civic Culture:
Political Attitudes and Democracy in Five Nations. Princeton, NJ:
Princeton University Press.
Warriner, Amy Beth, Victor Kuperman, and Marc Brysbaert. 2013.
“Norms of Valence, Arousal, and Dominance for 13,915 English
Lemmas.” Behavior Research Methods 45 (4): 1191–207.
Wu, Patrick Y., Walter R. Mebane, Jr., Logan Woods, Joseph
Klaver, and Preston Due. 2019. “Partisan Associations of Twitter
Users Based on Their Self-Descriptions and Word Embeddings.”(Mimeo) New York: New York University.
Yin, Zi, Vin Sachidananda, and Balaji Prabhakar. 2018. “The Global
Anchor Method for Quantifying Linguistic Shifts and Domain
Adaptation.” In Advances in Neural Information Processing
Systems, Vol. 31, eds. S. Bengio, H. Wallach, H. Larochelle, K.
Grauman, N. Cesa-Bianchi, and R. Garnett, 9434–45. Red Hook,
NY: Curran Associates, Inc.
Ying, Luwei, Jacob M. Montgomery, and Brandon M. Stewart. 2021.
“Topics, Concepts, and Measurement: A Crowdsourced Procedure
for Validating Topics as Measures.” Political Analysis 30 (4): 570–89.