SlowGuess/NAACL_2022_OCR huggingface dataset:
Barea Sinno $^{1*}$  Bernardo Oviedo $^{2*}$  Katherine Atwell $^{3*}$  Malihe Alikhani $^{3}$  Junyi Jessy Li $^{4}$
<sup>1</sup> Political Science, Rutgers University
$^{2}$  Computer Science,  $^{4}$  Linguistics, The University of Texas at Austin
<sup>3</sup> Computer Science, University of Pittsburgh
barea.sinno@gmail.com, bernyoviedo@utexas.edu, kaa139@pitt.edu
malihe@pitt.edu, jessy@utexas.edu
Analyzing ideology and polarization is of critical importance in advancing our grasp of modern politics. Recent research has made great strides towards understanding the ideological bias (i.e., stance) of news media along the left-right spectrum. In this work, we instead take a novel and more nuanced approach for the study of ideology based on its left or right positions on the issue being discussed. Aligned with the theoretical accounts in political science, we treat ideology as a multi-dimensional construct, and introduce the first diachronic dataset of news articles whose ideological positions are annotated by trained political scientists and linguists at the paragraph level. We showcase that, by controlling for the author's stance, our method allows for the quantitative and temporal measurement and analysis of polarization as a multidimensional ideological distance. We further present baseline models for ideology prediction, outlining a challenging task distinct from stance detection.
Political ideology rests on a set of beliefs about the proper order of a society and ways to achieve this order (Jost et al., 2009; Adorno et al., 2019; Campbell et al., 1980). In Western politics, these worldviews translate into a multi-dimensional construct that includes: equal opportunity as opposed to economic individualism; general respect for tradition, hierarchy and stability as opposed to advocating for social change; and a belief in the un/fairness and in/efficiency of markets (Jost et al., 2009).
The divergence in ideology, i.e., polarization, is the undercurrent of propaganda and misinformation (Vicario et al., 2019; Bessi et al., 2016; Stanley, 2015). It can congest essential democratic functions with an increase in the divergence of political ideologies. Defined as a growing ideo
Two dimensions:
trade and economic liberalism	The U.S. aim is to create a monetary system with enough flexibility to prevent bar-gain-hungry money from rolling around the world like loose ballast on a ship disrupting normal trade and currency flows.
Nixon goals: dollar, trade stability. This must be accompanied, Washington says, by reduction of [trade] barriers ...
One dimension:
trade protectionism	The controls program, which Mr. Nixon inaugurated Aug. 15, 1971, has helped to reduce inflation to about 3 percent yearly, and to boost annual U.S. economic growth to more than 7 percent...
Table 1: Excerpts from news article #730567 in COHA (Davies, 2012). The first paragraph advocates for liberalism and the reduction of trade barriers. It also has a domestic economic dimension. The second paragraph, on the contrary, advocates for protectionism and a domestic controls program.
logical distance between groups, polarization has waxed and waned since the advent of the American Republic (Pierson and Schickler, 2020).<sup>1</sup> Two eras—post-1896 and -1990s—have witnessed deleterious degrees of polarization (Jenkins et al., 2004; Jensen et al., 2012). More recently, COVID-19, the murder of George Floyd, and the Capitol riots have exposed ideological divergences in opinion in the US through news media and social media. With the hope of advancing our grasp of modern politics, we study ideology and polarization through the lens of computational linguistics by presenting a carefully annotated corpus and examining the efficacy of a set of computational and statistical analyses.
In contrast to studying the bias or the stance of the author of the text via linguistic framing (Kulkarni et al., 2018; Kiesel et al., 2019; Baly et al., 2019, 2020; Chen et al., 2020; Stefanov et al., 2020), we
<sup>1</sup>We distinguish ourselves from work that considers other types of polarization, e.g., as a measure of emotional distance (Iyengar et al., 2019) or distance between political parties (Lauka et al., 2018).
* Equal contribution ordered by first name.
Human Language Technologies, pages 231 - 243
study the little explored angle that is nonetheless critical in political science research: ideology of the issue (e.g., policy or concept) under discussion. That is, in lieu of examining the author's stance, we focus on addressing the at-issue content of the text and the ideology that it represents in the implicit social context. The nuanced co-existence of stance and ideology can be illustrated in the following excerpt:
"Republicans and Joe Biden are making a huge mistake by focusing on cost. The implication is that government-run health care would be a good thing-a wonderful thing!-- if only we could afford it." (The Federalist, 9/27/2019)
The author is attacking a liberal social and economic policy; therefore, the ideology being discussed is liberal on two dimensions—social and economic, while the author's stance is conservative. Moreover, our novel approach acknowledges that ideology can also vary within one article. In Table 1, we show an example in which one part of an article advocates for trade liberalism, while another advocates for protectionism.
Together, author stance and ideology inform us not only that there is bias in the media, but also which beliefs are being supported and/or attacked. A full analysis of polarization (that reflects a growing distance of political ideology over time) can then be derived if diachronic data for both author stance and ideology were available. However, while there has been data for the former (with articles from recent years only) (Kiesel et al., 2019), to date, there has been no temporal data on the latter.
In this paper, we present a multi-dimensional framework, and an annotated, diachronic, stance-neutral corpus, for the analysis of ideology in text. This allows us to study polarization as a state of ideological groups with divergent positions on a political issue as well as polarization as a process whose magnitude grows over time (DiMaggio et al., 1996). We use proclaimed center, center-left and center-right media outlets who claim to be objective in order to focus exclusively and more objectively on the ideology of the issue being discussed, without the subjectivity of author stance annotation. We study ideology within every paragraph<sup>2</sup> of an article and aim to answer the following question: which ideological dimension is present and to which ideological position does it correspond to on the liberal-conservative spectrum.
Our extensive annotation manual is developed by a political scientist, and the data then annotated by three linguists after an elaborate training phase (Section 3). After 150 hours of annotation, we present a dataset of 721 fully adjudicated annotated paragraphs, from 175 news articles and covering an average of 7.86 articles per year (excerpts shown in Tables 1, 2, and 3). These articles originate from 5 news outlets related the US Federal Budget from 1947-1975 covering the center-left, center, center-right spectrum: Chicago Tribune (CT), Christian Science Monitor (CSM), the New York Times (NYT), Time Magazine (TM), and the Wall Street Journal (WSJ).
With this data, we reveal lexical insights on the language of ideology across the left-right spectrum and across dimensions. We observe that linguistic use even at word level can reveal the ideology behind liberal and conservative policies (Section 4). Our framework also enables fine-grained, quantitative analysis of polarization, which we demonstrate in Section 5. This type of analysis, if scaled up using accurate models for ideology prediction, has the potential to reveal impactful insights into the political context of our society as a whole.
Finally, we present baselines for the automatic identification of multi-dimensional ideology at the paragraph level (Section 6). We show that this is a challenging task with our best baseline yielding an F measure of 0.55; exploring pretraining with existing data in news ideology/bias identification, we found that this task is distinct from, although correlated with, labels automatically derived from news outlets. We contribute our data and code at https://github.com/bernovie/political-polarization.
Many political scientists and political psychologists argue for the use of at least a bidimensional ideology for domestic politics that distinguishes between economic and social preferences (Carsey and Layman, 2006; Carmines et al., 2012; Feldman and Johnston, 2014).<sup>3</sup> We start with these
3It is important to distinguish between ideology and several other concepts. (1) Partisanship (party identity) (Campbell et al., 1980): a partisan person changes their ideology when their party changes its ideology, whereas an ideological person changes their party when their party changes its ideology. Partisanship is easily conflated with party ID using a unidimensional conceptualization of ideology, but not with a multi-dimensional one. (2) Moral foundations: Haidt et al. (2009) gave an evolutionary explanation of how human
2We use automatically segmented paragraphs since the raw texts were not paragraph-segmented.
two dimensions while adding a third dimension, "Foreign", when the article tackles foreign issues.
Specifically, our annotation task entails examining a news article and annotating each dimension (detailed below) along three levels—liberal, conservative, neutral—for each paragraph. The neutral level for every dimension is reserved for paragraphs related to a specific dimension but either (a) contain both conservative and liberal elements that annotators were unable to ascertain an ideological dimension with confidence, or (b) do not portray any ideology. We additionally provide an irrelevant option if a dimension does not apply to the paragraph. The three dimensions are:
Social: While the (1) socially conservative aspect of this dimension is defined as respect for tradition, fear of threat and uncertainty, need for order and structure, concerns for personal and national security, and preference for conformity, its (2) socially liberal counterpart has been associated with a belief in the separation of church and state, tolerance for uncertainty and change (Jost et al., 2009).
Economic: Similarly, while the (3) economically conservative aspect of this dimension refers to motivations to achieve social rewards, power, and prestige such as deregulation of the economy, lower taxes and privatization (i.e., being against deficit) spending and advocating for a balanced budget, its (4) economically liberal counterpart refers to motivation for social justice and equality such as issues related to higher taxes on rich individuals and businesses and more redistribution.
Foreign: After piloting the bidimensional approach on 300 articles, we find that using only 2 dimensions conflates two important aspects of ideology related to domestic economy and foreign trade. Tariffs, import quotas, and other nontariff-based barriers to trade that are aimed at improving employment and the competitiveness of the US on the international market did not map well onto the bidimensional framework. After consulting several senior political scientists, we adopted a third dimension that dealt with the markets as well as the relations of the US with the rest of the world. While
the (5) globalist counterpart of this dimension accounts for free-trade, diplomacy, immigration and treaties such as the non-proliferation of arms, its (6) interventionist aspect is nationalist in its support for excise tax on imports to protect American jobs and economic subsidies and anti-immigration.
With the annotated data, we demonstrate quantitative measures of polarization (Section 5) and introduce the modeling task (Section 6) of automatically identifying the ideology of the policy positions being discussed.
3 Data collection and annotation
Raw data Since polarization is a process that needs to be analyzed over time (DiMaggio et al., 1996), our annotated articles are sampled from a diachronic corpus of 1,749 news articles across nearly 3 decades (from 1947 till 1974). Articles in this corpus are from political news articles of Desai et al. (2019) from the Corpus of Historical American English (COHA, Davies (2012)) covering years 1922-1986. These 1,749 articles are extracted such that: (1) they cover broad and politically relevant topics (ranging from education and health to economy) but still share discussions related to the federal budget to make our annotations tractable<sup>4</sup>; (2) balanced in the number of articles across 5 news outlets with center-left, central, and center-right ideology (c.f. Section 5): Chicago Tribune (CT), Wall Street Journal (WSJ), Christian Science Monitor (CSM), the New York Times (NYT), and Time Magazine (TM). A detailed description of our curation process is in Appendix A.
The raw texts were not segmented into paragraphs, thus we used Topic Tiling (Riedl and Biemann, 2012) for automatic segmentation. Topic Tiling finds segment boundaries again using LDA and, thus, identifies major subtopic changes within the same article. The segmentation resulted in articles with 1 to 6 paragraphs. The average number of paragraphs per article was 4.
Annotation process Our team (including a political science graduate student) developed an annotation protocol for expert annotators using definitions in Section 2. The annotation process is independently reviewed by four political science professors from two universities in the US who are not authors
<sup>4</sup>Because federal budget stories touch on all aspects of the federal activity, this topic appeals to both liberal and conservative media and thus can provide a good testing ground to showcase our proposed ideological annotation method.
morals, values and traits such as freedom, safety, harm, care, reciprocity, in-group loyalty, authority, equality are formed. Since, some scholars have used these traits to predict ideology whereas others have attempted to understand what traits unite people with the same ideology. (3) Framing: frames are used in many ways in political science. They can refer to different ways scholars describe the same information or when scholars talk about different aspects of a single problem (Chong and Druckman, 2007).
Two dimensions: socially and economically liberal	... Secretary of Defense Robert S. [Mc-Namara] threw his full support today behind the Administration's drive against poverty. ...Mr. McNamara said : “It is the youth that we can expect to be the most immediate beneficiaries of the war on poverty.” He said he was endorsing the “entire program” both as a citizen and as a member of the Cabinet. His endorsement came as his fellow Republicans in Congress continued to hammer away at parts of the Administration’s antipoverty program. ...
Two dimensions: socially and economically conservative	The antipoverty program, the Republicans insisted, would undercut the authority of the Cabinet members by making Sargent Shriver a “poverty czar.” “I don’t see how you can lie down and be a doormat for this kind of operation.”…
of this paper; the research area of two of them is ideology and polarization in the US. We will release our full annotation interface, protocol, and procedure along with the data upon publication.
We sampled on average 7.86 articles per year for annotation, for a total of 721 paragraphs across 175 articles. We divided the annotation task into two batches of 45 and 130 articles, the smaller batch was for training purposes.
In addition to the political science graduate student, we recruited three annotators, all of whom are recent Linguistics graduates in the US. The training sessions consisted of one general meeting (all annotators met) and six different one-on-one meetings (each annotator met with another annotator once). During initial training, the annotators were asked to highlight sentences based on which the annotation was performed.
After the annotations of this batch were finalized, the annotators met with the political science student to create ground truth labels in cases of disagreement. Then, the three annotators received the second batch and each article was annotated by 2 annotators. This annotation was composed of two stages to account for possible subjectivity. In stage 1, each annotator worked on a batch that
Table 2: Excerpts from article #723847 in COHA. Because the first paragraph calls for minimizing income inequality, it is socially liberal; and because advocating for such a program call for an budgetary expenditure, it is also has an economic liberal dimension. The second paragraph advocates for the exact opposites of the positions in the first paragraph. Therefore, it is socially and economically conservative. Sentences most relevant to these labels are highlighted.
Zero dimension	... “The committee is holding public hearings on President Eisenhower's Economic Report, which he sent to Congress last week. The Secretary's [Humphrey] appearance before the group provided an opportunity for political exchanges.
One dimension: economically liberal	Senators Paul H. Douglas of Illinois, J. W. Fulbright of Representative Wright Patman of Texas, all Democrats, were active in questioning Mr. Humphrey. The Democrats asserted that the Administration's tax reduction program was loaded in favor of business enterprises and shareholders in industry and against the taxpayer in the lowest income brackets....
One dimension: economically neutral	Senator Fulbright ... declare[d] that the problem was to expand consumption rather than production. ... “Production is the goose that lays the golden egg,” Mr. Humphrey replied. “Payrolls make consumers.”
Table 3: Excerpts from article #716033 in COHA. The first paragraph is void of ideology. In the second paragraph the topic is anti tax reduction on businesses, thus it is economically liberal. The third paragraph is simultaneously economically conservative and liberal because one speaker is advocating for decreasing tax on businesses and asserting that production gives an advantage to businesses, the other is advocating for decreasing tax on the poor because they need the income and asserting that healthy businesses are the ones who pay salaries for the low income bracket worker.
overlapped with only one other annotator. In stage 2, the two annotators examine paragraphs that they disagree, and met with the third annotator acting as consolidator to adjudicate. Tables 1. 3 and 2 are examples of adjudicated annotation in the data.
Agreement To assess the inter-annotator agreement of stage 1, we report Krippendorf's  $\alpha$  (Hayes and Krippendorff, 2007) for each dimension for the 135 articles after training and before any discussion/adjudication: economic (0.44), foreign (0.68), social (0.39). The agreements among annotators for the economic and foreign dimensions are moderate & substantial (Artstein and Poesio, 2008), respectively; for social, the 'fair' agreement was noticed during annotation, and additional discussion for each paragraph was then held. Afterwards, 25 more articles were independently annotated and assessed with an  $\alpha$  of 0.53. Although the agreements were not perfect and reflected a degree of subjectivity in this task, all dimensional labels were adjudicated after discussions between annotators. In total, creating this dataset cost  $\sim 150$  hours of manual multi-dimensional labeling.
#Docs	Econ.	Soc.	Fgn.	Total
CSM	37	115	63	82	260
CT	14	48	33	16	97
NYT	60	219	114	130	463
TM	52	134	60	89	283
WSJ	12	42	21	21	84
Total	175	558	291	338	1187
Table 4: Dimensional label counts across all 721 paragraphs in the adjudicated data (there can be multiple dimensions per paragraph).
Qualitative analysis of text highlights For the 25 articles used in training, all annotators highlighted the sentences that are relevant to each dimension they annotated. This helped annotators to focus on the sentences that drove their decision, and provided insights to the language of ideology, which we discuss here. On average,  $21\% - 54\%$  of the sentences in a paragraph were highlighted.
We found entities such as "President" and "Congress" were the most prevalent in the highlights, and they tackled social and economic issues combined. This is not surprising as it suggests that when the media quotes or discusses the "President" and "Congress", they do so with reference to more complex policy issues. In contrast, individual congresspeople tackled mostly economic or social issues. This also is not surprising as it suggests that individual congresspeople are more concerned with specific issues. Interestingly, "House" and "Senate" almost always figured more in social issues. This suggest that when news media speaks about a specific chamber, they do so associating this chamber with social issues. Finally, party affiliation was infrequent and was mostly associate with social issues.
4 Ideology analyses
The number of paragraphs per dimension in total is: Economics (558), Social (291), Forign (338), across the 175 articles. In Table 4 we tabulate this for each of the news outlets. Figure 1 shows the dimensional label distributions per outlet for each dimension. Expectedly, the dimensional labels often diverge from proclaimed ideology of the news outlet.
We also analyze the percentage of articles that contain at least one pair of paragraph labels that lean in different directions; for instance, a paragraph with a label of globalist (i.e., liberal) in the foreign dimension and another paragraph with a
Figure 1: Dimensional label distribution per outlet.
Figure 2: Co-occurrence matrices on the paragraph level (left) and article level (right)
label of conservative or neutral in the fiscal dimension. The percentage of such articles is  $78.3\%$ . Out of these articles, we examine the average proportion of neutral, liberal, and conservative paragraph labels, and find neutral labels have the highest share  $(43.27\%)$ , followed by liberal  $(33.20\%)$  and conservative  $(23.53\%)$ . In Figure 2 (right), we visualize the percentage of articles where two dimensional labels co-occur within the same article. The figure indicates that ideology varies frequently within an article, showing that a single article-level label will not be fine-grained enough to capture variances within an article.
In Figure 2 (left) we also show paragraph-level label co-occurrence. Unlike the article-level, the co-occurrences are less frequent and we are more likely to observe co-occurrences along the same side of ideology. Still, we see interesting nuances; for example, on both the paragraph and the article level, the economic dimension is often neutral, and this tends to co-occur with both liberal and conservative positions in other dimensions.
Lexical analysis To understand ways ideology is reflected in text, we also look into the top vocabulary that associates with conservative or liberal ideology. To do so, we train a logistic regression model for each dimension to predict whether a paragraph is labeled conservative or liberal on that dimension, using unigram frequency as features. In Table 5 we show the top most left-leaning (L)
Economic	C: mr (5.2), tax (5.0), truman (3.7), business (3.3), billion (3.2)
L: school (-4.3), education (-3.3), commission (-3.2), senator (-3.0), plan (-2.5)
Social	C: defense (7.5), tax (4.6), air (4.4), billion (3.9), missile (3.8)
L: federal (-3.6), wage (-3.6), would (-3.5), policy (-3.2), labor (-2.9)
Foreign	C: defense (6.9), force (5.3), north (5.2), air (5.0), vietnam (4.6)
L: aid (-9.3), economic (-5.5), foreign (-5.3), germany (-4.3), make (-4.1)
Table 5: Words with the most positive and negative weights from a logistic regression model trained to predict liberal/conservative ideology for each dimension.