SlowGuess/ACL_2020_OCR huggingface dataset:
Keyon Vafa, Suresh Naidu, David M. Blei
{keyon.vafa,sn2430,david.blei}@columbia.edu
Ideal point models analyze lawmakers' votes to quantify their political positions, or ideal points. But votes are not the only way to express a political position. Lawmakers also give speeches, release press statements, and post tweets. In this paper, we introduce the text-based ideal point model (TBIP), an unsupervised probabilistic topic model that analyzes texts to quantify the political positions of its authors. We demonstrate the TBIP with two types of politicized text data: U.S. Senate speeches and senator tweets. Though the model does not analyze their votes or political affiliations, the TBIP separates lawmakers by party, learns interpretable politicized topics, and infers ideal points close to the classical vote-based ideal points. One benefit of analyzing texts, as opposed to votes, is that the TBIP can estimate ideal points of anyone who authors political texts, including non-voting actors. To this end, we use it to study tweets from the 2020 Democratic presidential candidates. Using only the texts of their tweets, it identifies them along an interpretable progressive-to-moderate spectrum.
Ideal point models are widely used to help characterize modern democracies, analyzing lawmakers' votes to estimate their positions on a political spectrum (Poole and Rosenthal, 1985). But votes aren't the only way that lawmakers express political preferences—press releases, tweets, and speeches all help convey their positions. Like votes, these signals are recorded and easily collected.
This paper develops the text-based ideal point model (TBIP), a probabilistic topic model for analyzing unstructured political texts to quantify the political preferences of their authors. While classical ideal point models analyze how different people vote on a shared set of bills, the TBIP analyzes how different authors write about a shared set of latent
topics. The TBIP is inspired by the idea of political framing: the specific words and phrases used when discussing a topic can convey political messages (Entman, 1993). Given a corpus of political texts, the TBIP estimates the latent topics under discussion, the latent political positions of the authors of texts, and how per-topic word choice changes as a function of the political position of the author.
A key feature of the TBIP is that it is unsupervised. It can be applied to any political text, regardless of whether the authors belong to known political parties. It can also be used to analyze non-voting actors, such as political candidates.
Figure 1 shows a TBIP analysis of the speeches of the 114th U.S. Senate. The model lays the senators out on the real line and accurately separates them by party. (It does not use party labels in its analysis.) Based only on speeches, it has found an interpretable spectrum—Senator Bernie Sanders is liberal, Senator Mitch McConnell is conservative, and Senator Susan Collins is moderate. For comparison, Figure 2 also shows ideal points estimated from the voting record of the same senators; their language and their votes are closely correlated.
The TBIP also finds latent topics, each one a vocabulary-length vector of intensities, that describe the issues discussed in the speeches. For each topic, the TBIP involves both a neutral vector of intensities and a vector of ideological adjustments that describe how the intensities change as a function of the political position of the author. Illustrated in Table 1 are discovered topics about immigration, health care, and gun control. In the gun control topic, the neutral intensities focus on words like "gun" and "firearms." As the author's ideal point becomes more negative, terms like "gun violence" and "background checks" increase in intensity. As the author's ideal point becomes more positive, terms like "constitutional rights" increase.
The TBIP is a bag-of-words model that combines ideas from ideal point models and Poisson factor
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5345-5357 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1. The text-based ideal point model (TBIP) separates senators by political party using only speeches. The algorithm does not have access to party information, but senators are coded by their political party for clarity (Democrats in blue circles, Republicans in red x's). The speeches are from the 114th U.S. Senate.
ization topic models (Canny, 2004; Gopalan et al., 2015). The latent variables are the ideal points of the authors, the topics discussed in the corpus, and how those topics change as a function of ideal point. To approximate the posterior, we use an efficient black box variational inference algorithm with stochastic optimization. It scales to large corpora.
We develop the details of the TBIP and its variational inference algorithm. We study its performance on three sessions of U.S. Senate speeches, and we compare the TBIP to other methods for scaling political texts (Slapin and Proksch, 2008; Lauderdale and Herzog, 2016a). The TBIP performs best, recovering ideal points closest to the vote-based ideal points. We also study its performance on tweets by U.S. senators, again finding that it closely recovers their vote-based ideal points. (In both speeches and tweets, the differences from vote-based ideal points are also qualitatively interesting.) Finally, we study the TBIP on tweets by the 2020 Democratic candidates for President, for which there are no votes for comparison. It lays out the candidates along an interpretable progressive-to-moderate spectrum.
2 The text-based ideal point model
We develop the text-based ideal point model (TBIP), a probabilistic model that infers political ideology from political texts. We first review Bayesian ideal points and Poisson factorization topic models, two probabilistic models on which the TBIP is built.
2.1 Background: Bayesian ideal points
Ideal points quantify a lawmakers's political preferences based on their roll-call votes (Poole and Rosenthal, 1985; Jackman, 2001; Clinton et al., 2004). Consider a group of lawmakers voting "yea"
or "nay" on a shared set of bills. Denote the vote of lawmaker  $i$  on bill  $j$  by the binary variable  $v_{ij}$ . The Bayesian ideal point model posits scalar perlawmaker latent variables  $x_{i}$  and scalar per-bill latent variables  $(\alpha_{j},\eta_{j})$ . It assumes the votes come from a factor model,
$$
x _ {i} \sim \mathcal {N} (0, 1)
$$
$$
\alpha_ {j}, \eta_ {j} \sim \mathcal {N} (0, 1)
$$
$$
v _ {i j} \sim \operatorname {B e r n} \left(\sigma \left(\alpha_ {j} + x _ {i} \eta_ {j}\right)\right). \tag {1}
$$
where  $\sigma (t) = \frac{1}{1 + e^{-t}}$
The latent variable  $x_{i}$  is called the lawmaker's ideal point; the latent variable  $\eta_{j}$  is the bill's polarity. When  $x_{i}$  and  $\eta_{j}$  have the same sign, lawmaker  $i$  is more likely to vote for bill  $j$ ; when they have opposite sign, the lawmaker is more likely to vote against it. The per-bill intercept term  $\alpha_{j}$  is called the popularity. It captures that some bills are uncontroversial, where all lawmakers are likely to vote for them (or against them) regardless of their ideology.
Using data of lawmakers voting on bills, political scientists approximate the posterior of the Bayesian ideal point model with an approximate inference method such as Markov Chain Monte Carlo (MCMC) (Jackman, 2001; Clinton et al., 2004) or expectation-maximization (EM) (Imai et al., 2016). Empirically, the posterior ideal points of the lawmakers accurately separate political parties and capture the spectrum of political preferences in American politics (Poole and Rosenthal, 2000).
2.2 Background: Poisson factorization
Poisson factorization is a class of non-negative matrix factorization methods often employed as a topic model for bag-of-words text data (Canny, 2004; Cemgil, 2009; Gopalan et al., 2014).
Poisson factorization factorizes a matrix of document/word counts into two positive matrices: a matrix  $\pmb{\theta}$  that contains per-document topic intensities, and a matrix  $\pmb{\beta}$  that contains the topics. Denote the count of word  $v$  in document  $d$  by  $y_{dv}$ . Poisson factorization posits the following probabilistic model over word counts, where  $a$  and  $b$  are hyperparameters:
$$
\theta_ {d k} \sim \operatorname {G a m m a} (a, b)
$$
$$
\beta_ {k v} \sim \operatorname {G a m m a} (a, b)
$$
$$
y _ {d v} \sim \operatorname {P o i s} \left(\sum_ {k} \theta_ {d k} \beta_ {k v}\right). \tag {2}
$$
Given a matrix  $y$ , practitioners approximate the posterior factorization with variational inference (Gopalan et al., 2015) or MCMC (Cemgil, 2009). Note that Poisson factorization can be interpreted as a Bayesian variant of nonnegative matrix factorization, with the so-called "KL loss function" (Lee and Seung, 1999).
When the shape parameter  $a$  is less than 1, the latent vectors  $\theta_{d}$  and  $\beta_{k}$  tend to be sparse. Consequently, the marginal likelihood of each count places a high mass around zero and has heavy tails (Ranganath et al., 2015). The posterior components are interpretable as topics (Gopalan et al., 2015).
2.3 The text-based ideal point model
The text-based ideal point model (TBIP) is a probabilistic model that is designed to infer political preferences from political texts.
There are important differences between a dataset of votes and a corpus of authored political language. A vote is one of two choices, "yea" or "nay." But political language is high dimensional—a lawmaker's speech involves a vocabulary of thousands. A vote sends a clear signal about a lawmaker's opinion about a bill. But political speech is noisy—the use of a word might be irrelevant to ideology, provide only a weak signal about ideology, or change signal depending on context. Finally, votes are organized in a matrix, where each one is unambiguously attached to a specific bill and nearly all lawmakers vote on all bills. But political language is unstructured and sparse. A corpus of political language can discuss any number of issues—with speeches possibly involving several issues—and the issues are unlabeled and possibly unknown in advance.
The TBIP is based on the concept of political framing. Framing is the idea that a communicator will emphasize certain aspects of a message – implicitly or explicitly – to promote a perspective
or agenda (Entman, 1993; Chong and Druckman, 2007). In politics, an author's word choice for a particular issue is affected by the ideological message she is trying to convey. A conservative discussing abortion is more likely to use terms such as "life" and "unborn," while a liberal discussing abortion is more likely to use terms like "choice" and "body." In this example, a conservative is framing the issue in terms of morality, while a liberal is framing the issue in terms of personal liberty.
The TBIP casts political framing in a probabilistic model of language. While the classical ideal point model infers ideology from the differences in votes on a shared set of bills, the TBIP infers ideology from the differences in word choice on a shared set of topics.
The TBIP is a probabilistic model that builds on Poisson factorization. The observed data are word counts and authors:  $y_{dv}$  is the word count for term  $v$  in document  $d$ , and  $a_d$  is the author of the document. Some of the latent variables in the TBIP are inherited from Poisson factorization: the non-negative  $K$ -vector of per-document topic intensities is  $\theta_d$  and the topics themselves are non-negative  $V$ -vectors  $\beta_k$ , where  $K$  is the number of topics and  $V$  is the vocabulary size. We refer to  $\beta$  as the neutral topics. Two additional latent variables capture the politics: the ideal point of an author  $s$  is a real-valued scalar  $x_s$ , and the ideological topic is a real-valued  $V$ -vector  $\eta_k$ .
The TBIP uses its latent variables in a generative model of authored political text, where the ideological topic adjusts the neutral topic—and thus the word choice—as a function of the ideal point of the author. Place sparse Gamma priors on  $\theta$  and  $\pmb{\beta}$ , and normal priors on  $\eta$  and  $x$ , so for all documents  $d$ , words  $v$ , topics  $k$ , and authors  $s$ ,
$$
\theta_ {d k} \sim \operatorname {G a m m a} (a, b) \quad \eta_ {k v} \sim \mathcal {N} (0, 1)
$$
$$
\beta_ {k v} \sim \operatorname {G a m m a} (a, b) \qquad x _ {s} \sim \mathcal {N} (0, 1).
$$
These latent variables interact to draw the count of term  $v$  in document  $d$ ,
$$
y _ {d v} \sim \operatorname {P o i s} \left(\sum_ {k} \theta_ {d k} \beta_ {k v} \exp \left\{x _ {a _ {d}} \eta_ {k v} \right\}\right). \tag {3}
$$
For a topic  $k$  and term  $v$ , a non-zero  $\eta_{kv}$  will increase the Poisson rate of the word count if it shares the same sign as the ideal point of the author  $x_{a_d}$ , and decrease the Poisson rate if they are of opposite signs. Consider a topic about gun control and suppose  $\eta_{kv} > 0$  for the term "constitution." An author with an ideal point  $x_s > 0$ , say a conservative
Ideology
Top Words
Liberal	dreamers, dream, undocumented, daca, comprehensive immigration reform, deport, young, deportation
Neutral	immigration, united states, homeland security, department, executive, presidents, law, country
Conservative	laws, homeland security, law, department, amnesty, referred, enforce, injunction
Liberal	affordable care act, seniors, medicare, medicaid, sick, prescription drugs, health insurance
Neutral	health care, obamacare, affordable care act, health insurance, insurance, americans, coverage, percent
Conservative	health care law, obamacare, obama, democrats, obamacares, deductibles, broken promises
Liberal	gun violence, gun, guns, killed, hands, loophole, background checks, close
Neutral	gun, guns, second, orlando, question, firearms, shooting, background checks
Conservative	second, constitutional rights, rights, due process, gun control, mental health, list, mental illness
Table 1. The TBIP learns topics from Senate speeches that vary as a function of the senator's political positions. The neutral topics are for an ideal point of 0; the ideological topics fix ideal points at  $-1$  and  $+1$ . We interpret one extreme as liberal and the other as conservative. Data is from the 114th U.S. Senate.
author, will be more likely to use the term "constitution" when discussing gun control; an author with an ideal point  $x_{s} < 0$ , a liberal author, will be less likely to use the term. Suppose  $\eta_{kv} < 0$  for the term "violence." Now the liberal author will be more likely than the conservative to use this term. Finally suppose  $\eta_{kv} = 0$  for the term "gun." This term will be equally likely to be used by the authors, regardless of their ideal points.
To build more intuition, examine the elements of the sum in the Poisson rate of Equation (3) and rewrite slightly to  $\theta_{dk}\exp (\log \beta_{kv} + x_{a_d}\eta_{kv})$ . Each of these elements mimics the classical ideal point model in Equation (1), where  $\eta_{kv}$  now measures the "polarity" of term  $v$  in topic  $k$  and  $\log \beta_{kv}$  is the intercept or "popularity." When  $\eta_{kv}$  and  $x_{a_d}$  have the same sign, term  $v$  is more likely to be used when discussing topic  $k$ . If  $\eta_{kv}$  is near zero, then the term is not politicized, and its count comes from a Poisson factorization. For each document  $d$ , the elements of the sum that contribute to the overall rate are those for which  $\theta_{dk}$  is positive; that is, those for the topics that are being discussed in the document.
The posterior distribution of the latent variables provides estimates of the ideal points, neutral topics, and ideological topics. For example, we estimate this posterior distribution using a dataset of senator speeches from the 114th United States Senate session. The fitted ideal points in Figure 1 show that the TBIP largely separates lawmakers by political party, despite not having access to these labels or votes. Table 1 depicts neutral topics (fixing the fitted  $\hat{\eta}_{kv}$  to be 0) and the corresponding ideological topics by varying the sign of  $\hat{\eta}_{kv}$ . The topic for immigration shows that a liberal framing emphasizes "Dreamers" and "DACA", while the conservative frame emphasizes "laws" and "homeland security."
We provide more details and empirical studies in Section 5.
Most ideal point models focus on legislative roll-call votes. These are typically latent-space factor models (Poole and Rosenthal, 1985; McCarty et al., 1997; Poole and Rosenthal, 2000), which relate closely to item-response models (Bock and Aitkin, 1981; Bailey, 2001). Researchers have also developed Bayesian analogues (Jackman, 2001; Clinton et al., 2004) and extensions to time series, particularly for analyzing the Supreme Court (Martin and Quinn, 2002).
Some recent models combine text with votes or party information to estimate ideal points of legislators. Gerrish and Blei (2011) analyze votes and the text of bills to learn ideological language. Gerrish and Blei (2012) and Lauderdale and Clark (2014) use text and vote data to learn ideal points adjusted for topic. The models in Nguyen et al. (2015) and Kim et al. (2018) analyze votes and floor speeches together. With labeled political party affiliations, machine learning methods can also help map language to party membership. Iyyer et al. (2014) use neural networks to learn partisan phrases, while the models in Tsur et al. (2015) and Gentzkow et al. (2019) use political party labels to analyze differences in speech patterns. Since the TBIP does not use votes or party information, it is applicable to all political texts, even when votes and party labels are not present. Moreover, party labels can be restrictive because they force hard membership in one of two groups (in American politics). The TBIP can infer how topics change smoothly across the political spectrum, rather than simply learning topics for each political party.
Annotated text data has also been used to pre
dict ideological positions. Wordscores (Laver et al., 2003; Lowe, 2008) uses texts that are hand-labeled by political position to measure the conveyed positions of unlabeled texts; it has been used to measure the political landscape of Ireland (Benoit and Laver, 2003; Herzog and Benoit, 2015). Ho et al. (2008) analyze hand-labeled editorials to estimate ideal points for newspapers. The ideological topics learned by the TBIP are also related to political frames (Entman, 1993; Chong and Druckman, 2007). Historically, these frames have either been hand-labeled by annotators (Baumgartner et al., 2008; Card et al., 2015) or used annotated data for supervised prediction (Johnson et al., 2017; Baumer et al., 2015). In contrast to these methods, the TBIP is completely unsupervised. It learns ideological topics that do not need to conform to pre-defined frames. Moreover, it does not depend on the subjectivity of coders.
WORDFISH (Slapin and Proksch, 2008) is a model of authored political texts about a single issue, similar to a single-topic version of TBIP. WORDFISH has been applied to party manifestos (Proksch and Slapin, 2009; Lo et al., 2016) and single-issue dialogue (Schwarz et al., 2017). WORDSHOAL (Lauderdale and Herzog, 2016a) extends WORDFISH to multiple issues by analyzing a collection of labeled texts, such as Senate speeches labeled by debate topic. WORDSHOAL fits separate WORDFISH models to the texts about each label, and combines the fitted models in a one-dimensional factor analysis to produce ideal points. In contrast to these models, the TBIP does not require a grouping of the texts into single issues. It naturally accommodates unstructured texts, such as tweets, and learns both ideal points for the authors and ideology-adjusted topics for the (latent) issues under discussion. Furthermore, by relying on stochastic optimization, the TBIP algorithm scales to large data sets. In Section 5 we empirically study how the TBIP ideal points compare to both of these models.
4 Inference
The TBIP involves several types of latent variables: neutral topics  $\beta_{k}$ , ideological topics  $\eta_{k}$ , topic intensities  $\theta_{d}$ , and ideal points  $x_{s}$ . Conditional on the text, we perform inference of the latent variables through the posterior distribution  $p(\pmb{\theta},\pmb{\beta},\pmb{\eta},\pmb{x}|\pmb{y})$ . But calculating this distribution is intractable. We rely on approximate inference.
We use mean-field variational inference to fit an
approximate posterior distribution (Jordan et al., 1999; Wainwright et al., 2008; Blei et al., 2017). Variational inference frames the inference problem as an optimization problem. Set  $q_{\phi}(\pmb{\theta}, \pmb{\beta}, \pmb{\eta}, \pmb{x})$  to be a variational family of approximate posterior distributions, indexed by variational parameters  $\phi$ . Variational inference aims to find the setting of  $\phi$  that minimizes the KL divergence between  $q_{\phi}$  and the posterior.
Minimizing this KL divergence is equivalent to maximizing the evidence lower bound (ELBO),
$$
\begin{array}{l} \mathbb {E} _ {q _ {\phi}} [ \log p (\boldsymbol {\theta}, \boldsymbol {\beta}, \boldsymbol {\eta}, \boldsymbol {x}) + \log p (\boldsymbol {y} | \boldsymbol {\theta}, \boldsymbol {\beta}, \boldsymbol {\eta}, \boldsymbol {x}) \\ \left. - \log q _ {\phi} (\boldsymbol {\theta}, \boldsymbol {\beta}, \boldsymbol {\eta}, \boldsymbol {x}) \right]. \\ \end{array}
$$
The ELBO sums the expectation of the log joint (here broken up into the log prior and log likelihood) and the entropy of the variational distribution.
To approximate the TBIP posterior we set the variational family to be the mean-field family. The mean-field family factorizes over the latent variables, where  $d$  indexes documents,  $k$  indexes topics, and  $s$  indexes authors:
$$
q _ {\phi} (\boldsymbol {\theta}, \boldsymbol {\beta}, \boldsymbol {\eta}, \boldsymbol {x}) = \prod_ {d, k, s} q \left(\theta_ {d}\right) q \left(\beta_ {k}\right) q \left(\eta_ {k}\right) q \left(x _ {s}\right).
$$
We use lognormal factors for the positive variables and Gaussian factors for the real variables,
$$
q (\theta_ {d}) = \mathrm {L o g N o r m a l} _ {K} (\mu_ {\theta_ {d}}, I \sigma_ {\theta_ {d}} ^ {2})
$$
$$
q (\beta_ {k}) = \mathrm {L o g N o r m a l} _ {V} (\mu_ {\beta_ {k}}, I \sigma_ {\beta_ {k}} ^ {2})
$$
$$
q (\eta_ {k}) = \mathcal {N} _ {V} (\mu_ {\eta_ {k}}, I \sigma_ {\eta_ {k}} ^ {2})
$$
$$
q (x _ {s}) = \mathcal {N} (\mu_ {x _ {s}}, \sigma_ {x _ {s}} ^ {2}).
$$
Our goal is to optimize the ELBO with respect to  $\phi = \{\pmb {\mu}_{\theta},\pmb{\sigma}_{\theta}^{2},\pmb{\mu}_{\beta},\pmb{\sigma}_{\beta}^{2},\pmb{\mu}_{\eta},\pmb{\sigma}_{\eta}^{2},\pmb{\mu}_{x},\pmb{\sigma}_{x}^{2}\}$
We use stochastic gradient ascent. We form noisy gradients with Monte Carlo and the "reparameterization trick" (Kingma and Welling, 2014; Rezende et al., 2014), as well as with data subsampling (Hoffman et al., 2013). To set the step size, we use Adam (Kingma and Ba, 2015).
We initialize the neutral topics and topic intensities with a pre-trained model. Specifically, we pre-train a Poisson factorization topic model using the algorithm in Gopalan et al. (2015). The TBIP algorithm uses the resulting factorization to initialize the variational parameters for  $\theta_d$  and  $\beta_k$ . The full procedure is described in Appendix A.
For the corpus of Senate speeches described in Section 2, training takes 5 hours on a single
Figure 2. The ideal points learned by the TBIP for senator speeches and tweets are highly correlated with the classical vote ideal points. Senators are coded by their political party (Democrats in blue circles, Republicans in red x's). Although the algorithm does not have access to these labels, the TBIP almost completely separates parties.
NVIDIA Titan V GPU. We have released open source software for Tensorflow and PyTorch. $^{1}$
5 Empirical studies