SlowGuess/ACL_2020_OCR huggingface dataset:
We study the text-based ideal point model (TBIP) on several datasets of political texts. We first use the TBIP to analyze speeches and tweets (separately) from U.S. senators. For both types of texts, the TBIP ideal points, which are estimated from text, are close to the classical ideal points, which are estimated from votes. We also compare the TBIP to existing methods for scaling political texts (Slapin and Proksch, 2008; Lauderdale and Herzog, 2016a). The TBIP performs better, finding ideal points closer to the vote-based ideal points. Finally, we use the TBIP to analyze a group that does not vote: 2020 Democratic presidential candidates. Using only tweets, it estimates ideal points for the candidates on an interpretable progressive-to-moderate spectrum.
5.1 The TBIP on U.S. Senate speeches
We analyze Senate speeches provided by Gentzkow et al. (2018), focusing on the 114th session of Congress (2015-2017). We compare ideal points found by the TBIP to the vote-based ideal point model from Equation (1). (Appendix B provides details about the comparison.) We use approximate posterior means, learned with variational inference, to estimate the latent variables. The estimated ideal points are  $\hat{\pmb{x}}$ ; the estimated neutral topics are  $\hat{\pmb{\beta}}$ ; the estimated ideological topics are  $\hat{\pmb{\eta}}$ .
Figure 2 compares the TBIP ideal points on
speeches to the vote-based ideal points. $^{2}$  Both models largely separate Democrats and Republicans. In the TBIP estimates, progressive senator Bernie Sanders (I-VT) is on one extreme, and Mitch McConnell (R-KY) is on the other. Susan Collins (R-ME), a Republican senator often described as moderate, is near the middle. The correlation between the TBIP ideal points and vote ideal points is high, 0.88. Using only the text of the speeches, the TBIP captures meaningful information about political preferences, separating the political parties and organizing the lawmakers on a meaningful political spectrum.
We next study the topics. For selected topics, Table 1 shows neutral terms and ideological terms. To visualize the neutral topics, we list the top words based on  $\hat{\beta}_k$ . To visualize the ideological topics, we calculate term intensities for two poles of the political spectrum,  $x_s = -1$  and  $x_s = +1$ . For a fixed  $k$ , the ideological topics thus order the words by  $\mathbb{E}[\beta_{kv}\exp (-\eta_{kv})]$  and  $\mathbb{E}[\beta_{kv}\exp (\eta_{kv})]$ .
Based on the separation of political parties in Figure 1, we interpret negative ideal points as liberal and positive ideal points as conservative. Table 1 shows that when discussing immigration, a senator with a neutral ideal point uses terms like "immigration" and "United States." As the author moves left, she will use terms like "Dreamers" and "DACA." As she moves right, she will emphasize terms like "laws" and "homeland security." The TBIP also captures that those on the left refer to health care legislation as the Affordable Care Act, while those on the right call it ObamaCare. Additionally, a liberal
2 Throughout our analysis, we appropriately rotate and standardize ideal points so they are visually comparable.
http://github.com/keyonvafa/tbip
Speeches 111	Speeches 112	Speeches 113	Tweets 114
Corr.	SRC	Corr.	SRC	Corr.	SRC	Corr.	SRC
WORDFISH	0.47	0.45	0.52	0.53	0.69	0.64	0.87	0.80
WORDSHOAL	0.61	0.64	0.60	0.56	0.45	0.44	—	—
TBIP	0.79	0.73	0.86	0.85	0.87	0.84	0.94	0.84
Table 2. The TBIP learns ideal points most similar to the classical vote ideal points for U.S. senator speeches and tweets. It learns closer ideal points than WORDFISH and WORDSHOAL in terms of both correlation (Corr.) and Spearman's rank correlation (SRC). The numbers in the column titles refer to the Senate session of the corpus. WORDSHOAL cannot be applied to tweets because there are no debate labels.
senator discussing guns brings attention to gun control: "gun violence" and "background checks" are among the largest intensity terms. Meanwhile, conservative senators are likely to invoke gun rights, emphasizing "constitutional rights."
Comparison to Wordfish and Wordshoal. We next treat the vote-based ideal points as "ground-truth" labels and compare the TBIP ideal points to those found by WORDFISH and WORDSHOAL. WORDSHOAL requires debate labels, so we use the labeled Senate speech data provided by Lauderdale and Herzog (2016b) on the 111th-113th Senates to train each method. Because we are interested in comparing models, we use the same variational inference procedure to train all methods. See Appendix B for more details.
We use two metrics to compare text-based ideal points to vote-based ideal points: the correlation between ideal points and Spearman's rank correlation between their orderings of the senators. With both metrics, when compared to vote ideal points from Equation (1), the TBIP outperforms WORDFISH and WORDSHOAL; see Table 2. Comparing to another vote-based method, DW-NOMINATE (Poole, 2005), produces similar results; see Appendix C.
5.2 The TBIP on U.S. Senate tweets
We use the TBIP to analyze tweets from U.S. senators during the 114th Senate session, using a corpus provided by VoxGovFEDERAL (2020). Tweet-based ideal points almost completely separate Democrats and Republicans; see Figure 2. Again, Bernie Sanders (I-VT) is the most extreme Democrat, and Mitch McConnell (R-KY) is one of the most extreme Republicans. Susan Collins (R-ME) remains near the middle; she is among the most moderate senators in vote-based, speech-based, and tweet-based models. The correlation between vote-based ideal points and tweet-based
ideal points is 0.94.
We also use senator tweets to compare the TBIP to WORDFISH (we cannot apply WORDSHOAL because tweets do not have debate labels). Again, the TBIP learns closer ideal points to the classical vote ideal points; see Table 2.
5.3 Using the TBIP as a descriptive tool
As a descriptive tool, the TBIP provides hints about the different ways senators use speeches or tweets to convey political messages. We use a likelihood ratio to help identify the texts that influenced the TBIP ideal point. Consider the log likelihood of a document using a fixed ideal point  $\tilde{x}$  and fitted values for the other latent variables,
$$
\ell_ {d} (\tilde {x}) = \sum_ {v} \log p (y _ {d v} | \hat {\boldsymbol {\theta}}, \hat {\boldsymbol {\beta}}, \hat {\boldsymbol {\eta}}, \tilde {x}).
$$
Ratios based on this likelihood can help point to why the TBIP places a lawmaker as extreme or moderate. For a document  $d$ , if  $\ell_d(\hat{x}_{a_d}) - \ell_d(0)$  is high then that document was (statistically) influential in making  $\hat{x}_{a_d}$  more extreme. If  $\ell_d(\hat{x}_{a_d}) - \ell_d(\max_s(\hat{x}_s))$  or  $\ell_d(\hat{x}_{a_d}) - \ell_d(\min_s(\hat{x}_s))$  is high then that document was influential in making  $\hat{x}_{a_d}$  less extreme. We emphasize this diagnostic does not convey any causal information, but rather helps understand the relationship between the data and the TBIP inferences.
Bernie Sanders (I-VT). Bernie Sanders is an Independent senator who caucuses with the Democratic party; we refer to him as a Democrat. Among Democrats, his ideal point changes the most between one estimated from speeches and one estimated from votes. Although his vote-based ideal point is the 17th most liberal, the TBIP ideal point based on Senate speeches is the most extreme.
We use the likelihood ratio to understand this difference in his vote-based and speech-based ideal
Figure 3. Based on tweets, the TBIP places 2020 Democratic presidential candidates along an interpretable progressive-to-moderate spectrum.
points. His speeches with the highest likelihood ratio are about income inequality and universal health care, which are both progressive issues. The following is an excerpt from one such speech:
"The United States is the only major country on Earth that does not guarantee health care to all of our people... At a time when the rich are getting richer and the middle class is getting poorer, the Republicans take from the middle class and working families to give more to the rich and large corporations."
Sanders is considered one of the most liberal senators; his extreme speech ideal point is sensible.
That Sanders' vote-based ideal point is not more extreme appears to be a limitation of the vote-based method. Applying the likelihood ratio to votes helps illustrate the issue. (Here a bill takes the place of a document.) The ratio identifies H.R. 2048 as influential. This bill is a rollback of the Patriot Act that Sanders voted against because it did not go far enough to reduce federal surveillance capabilities (RealClearPolitics, 2015). In voting "nay", he was joined by one Democrat and 30 Republicans, almost all of whom voted against the bill because they did not want surveillance capabilities curtailed at all. Vote-based ideal points, which only model binary values, cannot capture this nuance in his opinion. As a result, Sanders' vote-based ideal point is pulled to the right.
Deb Fischer (R-NE). Turning to tweets, Deb Fischer's tweet-based ideal point is more liberal than her vote-based ideal point; her vote ideal point is the 11th most extreme among senators, while her tweet ideal point is the 43rd most extreme. The likelihood ratio identifies the following tweets as responsible for this moderation:
"I want to empower women to be their own best advocates, secure that they have the tools to negotiate the wages they deserve. #EqualPay"
"FACT: 1963 Equal Pay Act enables women to sue for wage discrimination. #GetitRight #EqualPayDay"
The TBIP associates terms about equal pay and women's rights with liberals. A senator with the most liberal ideal point would be expected to use the phrase "EqualPay" 20 times as much as a senator with the most conservative ideal point and "women" 9 times as much, using the topics in Fischer's first tweet above. Fischer's focus on equal pay for women moderates her tweet ideal point.
Jeff Sessions (R-AL). The likelihood ratio can also point to model limitations. Jeff Sessions is a conservative voter, but the TBIP identifies his speeches as moderate. One of the most influential speeches for his moderate text ideal point, as identified by the likelihood ratio, criticizes Deferred Actions for Childhood Arrivals (DACA), an immigration policy established by President Obama that introduced employment opportunities for undocumented individuals who arrived as children:
"The President of the United States is giving work authorizations to more than 4 million people, and for the most part they are adults. Almost all of them are adults. Even the so-called DACA proportion, many of them are in their thirties. So this is an adult job legalization program."
This is a conservative stance against DACA. So why does the TBIP identify it as moderate? As depicted in Table 1, liberals bring up "DACA" when discussing immigration, while conservatives emphasize "laws" and "homeland security." The fitted
5352
Ideology
Top Words
Progressive Neutral Moderate	class, billionaire, billionaires, walnut, wall street, corporate, executives, government economy, pay, trump, business, tax, corporations, americans, billion trade war, trump, jobs, farmers, economy, economic, tariffs, businesses, promises, job
Progressive Neutral Moderate	#medicareforall, insurance companies, profit, health care, earth, medical debt, health care system, profits health care, plan, medicare, americans, care, access, housing, millions healthcare, universal healthcare, public option, plan, universal coverage, universal health care, away, choice
Progressive Neutral Moderate	green new deal, fossil fuel industry, fossil fuel, planet, pass, #greennewdeal, climate crisis, middle ground climate change, climate, climate crisis, plan, planet, crisis, challenges, world solutions, technology, carbon tax, climate change, challenges, climate, negative, durable
Table 3. The TBIP learns topics from 2020 Democratic presidential candidate tweets that vary as a function of the candidate's political positions. The neutral topics are for an ideal point of 0; the ideological topics fix ideal points at  $-1$  and  $+1$ . We interpret one extreme as progressive and the other as moderate.
expected count of "DACA" using the most liberal ideal point for the topics in the above speech is 1.04, in contrast to 0.04 for the most conservative ideal point. Since conservatives do not focus on DACA, Sessions even bringing up the program sways his ideal point toward the center. Although Sessions refers to DACA disapprovingly, the bag-of-words model cannot capture this negative sentiment.
5.4 2020 Democratic candidates
We also analyze tweets from Democratic presidential candidates for the 2020 election. Since all of the candidates running for President do not vote on a shared set of issues, their ideal points cannot be estimated using vote-based methods.
Figure 3 shows tweet-based ideal points for the 2020 Democratic candidates. Elizabeth Warren and Bernie Sanders, who are often considered progressive, are on one extreme. Steve Bullock and John Delaney, often considered moderate, are on the other. The selected topics in Table 3 showcase this spectrum. Candidates with progressive ideal points focus on: billionaires and Wall Street when discussing the economy, Medicare for All when discussing health care, and the Green New Deal when discussing climate change. On the other extreme, candidates with moderate ideal points focus on: trade wars and farmers when discussing the economy, universal plans for health care, and technological solutions to climate change.
We developed the text-based ideal point model (TBIP), an ideal point model that analyzes texts to quantify the political positions of their authors. It estimates the latent topics of the texts, the ideal points of their authors, and how each author's political position affects her choice of words within each topic.
We used the TBIP to analyze U.S. Senate speeches and tweets. Without analyzing the votes themselves, the TBIP separates lawmakers by party, learns interpretable politicized topics, and infers ideal points close to the classical vote-based ideal points. Moreover, the TBIP can estimate ideal points of anyone who authors political texts, including non-voting actors. When used to study tweets from 2020 Democratic presidential candidates, the TBIP identifies them along a progressive-to-moderate spectrum.
Acknowledgments This work is funded by ONR N00014-17-1-2131, ONR N00014-15-1-2209, NIH 1U01MH115727-01, NSF CCF-1740833, DARPA SD2 FA8750-18-C-0130, Amazon, NVIDIA, and the Simons Foundation. Keyon Vafa is supported by NSF grant DGE-1644869. We thank Voxgov for providing us with senator tweet data. We also thank Mark Arildsen, Naoki Egami, Aaron Schein, and anonymous reviewers for helpful comments and feedback.
Michael Bailey. 2001. Ideal point estimation with a small number of votes: A random-effects approach. Political Analysis, 9(3):192-210.
Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proceedings of ACL.
Frank R Baumgartner, Suzanne L De Boef, and Amber E Boydstun. 2008. The decline of the death penalty and the discovery of innocence. Cambridge University Press.
Kenneth Benoit and Michael Laver. 2003. Estimating Irish party policy positions using computer wordscoring: The 2002 election. *Irish Political Studies*, 18(1):97-107.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 2017. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877.
R Darrell Bock and Murray Aitkin. 1981. Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4):443-459.
John Canny. 2004. GaP: A factor model for discrete data. In ACM SIGIR Conference on Research and Development in Information Retrieval.
Dallas Card, Amber Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of ACL.
Ali Taylan Cemgil. 2009. Bayesian inference for nonnegative matrix factorisation models. Computational Intelligence and Neuroscience, 2009.
Dennis Chong and James N Druckman. 2007. Framing theory. Annual Review of Political Science, 10:103-126.
Joshua Clinton, Simon Jackman, and Douglas Rivers. 2004. The statistical analysis of roll call data. American Political Science Review, 98(2):355-370.
Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51-58.
Matthew Gentzkow, Jesse M Shapiro, and Matt Taddy. 2018. Congressional record for the 43rd-114th Congresses: Parsing speeches and phrase counts. Stanford Libraries.
Matthew Gentzkow, Jesse M Shapiro, and Matt Taddy. 2019. Measuring group differences in high-dimensional choices: Method and application to congressional speech. *Econometrica*, 87(4):1307-1340.
Sean Gerrish and David M Blei. 2011. Predicting legislative roll calls from text. In Proceedings of ICML.
Sean Gerrish and David M Blei. 2012. How they vote: Issue-adjusted models of legislative behavior. In Proceedings of NeurIPS.
Prem Gopalan, Jake M Hofman, and David M Blei. 2015. Scalable recommendation with Poisson factorization. In Proceedings of UAI.
Prem K Gopalan, Laurent Charlin, and David M Blei. 2014. Content-based recommendations with Poisson factorization. In Proceedings of NeurIPS.
Alexander Herzog and Kenneth Benoit. 2015. The most unkindest cuts: Speaker selection and expressed government dissent during economic crisis. *The Journal of Politics*, 77(4):1157–1175.
Daniel E Ho, Kevin M Quinn, et al. 2008. Measuring explicit political positions of media. Quarterly Journal of Political Science, 3(4):353-377.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303-1347.
Kosuke Imai, James Lo, and Jonathan Olmsted. 2016. Fast estimation of ideal points with massive data. American Political Science Review, 110(4):631-656.
Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of ACL.
Simon Jackman. 2001. Multidimensional analysis of roll call data via Bayesian simulation: Identification, estimation, inference, and model checking. Political Analysis, 9(3):227-241.
Kristen Johnson, Di Jin, and Dan Goldwasser. 2017. Leveraging behavioral and social information for weakly supervised collective classification of political discourse on Twitter. In Proceedings of ACL.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233.
In Song Kim, John Londregan, and Marc Ratkovic. 2018. Estimating spatial preferences from votes and text. Political Analysis, 26(2):210-229.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational Bayes. In Proceedings of ICLR.
Benjamin E Lauderdale and Tom S Clark. 2014. Scaling politically meaningful dimensions using texts and votes. American Journal of Political Science, 58(3):754-771.
Benjamin E Lauderdale and Alexander Herzog. 2016a. Measuring political positions from legislative speech. Political Analysis, 24(3):374-394.
Benjamin E Lauderdale and Alexander Herzog. 2016b. Replication data for: Measuring political positions from legislative speech. Harvard Dataverse.
Michael Laver, Kenneth Benoit, and John Garry. 2003. Extracting policy positions from political texts using words as data. American Political Science Review, 97(2):311-331.
Daniel D Lee and H Sebastian Seung. 1999. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788.
Jeffrey B. Lewis, Keith Poole, Howard Rosenthal, Adam Boche, Aaron Rudkin, and Luke Sonnet. 2020. Voteview: Congressional roll-call votes database.
James Lo, Sven-Oliver Proksch, and Jonathan B Slapin. 2016. Ideological clarity in multiparty competition: A new measure and test using election manifestos. British Journal of Political Science, 46(3):591-610.
Will Lowe. 2008. Understanding wordscores. Political Analysis, 16(4):356-371.
Andrew D Martin and Kevin M Quinn. 2002. Dynamic ideal point estimation via Markov Chain Monte Carlo for the US Supreme Court, 1953-1999. Political Analysis, 10(2):134-153.
Nolan M McCarty, Keith T Poole, and Howard Rosenthal. 1997. Income redistribution and the realignment of American politics. American Enterprise Institute Press.
Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, and Kristina Miler. 2015. Tea Party in the house: A hierarchical ideal point topic model and its application to Republican legislators in the 112th Congress. In Proceedings of ACL.
Keith T Poole. 2005. Spatial models of parliamentary voting. Cambridge University Press.
Keith T Poole and Howard Rosenthal. 1985. A spatial model for legislative roll call analysis. American Journal of Political Science, pages 357-384.
Keith T Poole and Howard Rosenthal. 2000. Congress: A political-economic history of roll call voting. Oxford University Press on Demand.
Sven-Oliver Proksch and Jonathan B Slapin. 2009. How to avoid pitfalls in statistical analysis of political texts: The case of Germany. German Politics, 18(3):323-344.
Rajesh Ranganath, Linpeng Tang, Laurent Charlin, and David M Blei. 2015. Deep exponential families. In Proceedings of AISTATS.
RealClearPolitics. 2015. Bernie Sanders on USA Freedom Act: "I may well be voting for it," does not go far enough. Online; posted 31-May-2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of ICML.
Daniel Schwarz, Denise Traber, and Kenneth Benoit. 2017. Estimating intra-party preferences: Comparing speeches to votes. Political Science Research and Methods, 5(2):379-396.
Jonathan B Slapin and Sven-Oliver Proksch. 2008. A scaling model for estimating time-series party positions from texts. American Journal of Political Science, 52(3):705-722.
Oren Tsur, Dan Calacci, and David Lazer. 2015. A frame of mind: Using statistical models for detection of framing and agenda setting campaigns. In Proceedings of ACL.
VoxGovFEDERAL. 2020. U.S. senators tweets from the 114th Congress.
Martin J Wainwright, Michael I Jordan, et al. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305.
We present the full procedure for training the text-based ideal point model (TBIP) in Algorithm 1. We make a final modification to the model in Equation (3). If some political authors are more verbose than others (i.e. use more words per document), the learned ideal points may reflect morbidity rather than a political preference. Thus, we multiply the expected word count by a term that captures the author's morbidity compared to all authors. Specifically, if  $n_s$  is the average word count over documents for author  $s$ , we set a weight:
$$
w _ {s} = \frac {n _ {s}}{\frac {1}{S} \sum_ {s ^ {\prime}} n _ {s ^ {\prime}}}, \tag {4}
$$
for  $S$  the number of authors. We then multiply the rate in Equation (3) by  $w_{a_d}$ . Empirically, we find this modification does not make much of a difference for the correlation results, but it helps us interpret the ideal points for the qualitative analysis.
B Data and inference settings
Senator speeches We remove senators who made less than 24 speeches. To lessen non-ideological correlations in the speaking patterns of senators from the same state, we remove cities and states in addition to stopwords and procedural terms. We include all unigrams, bigrams, and trigrams that appear in at least  $0.1\%$  of documents and at most  $30\%$ . To ensure that the inferences are not influenced by procedural terms used by a small number of senators with special appointments, we only include phrases that are spoken by 10 or more senators. This preprocessing leaves us with 19,009 documents from 99 senators, along with 14,503 terms in the vocabulary.