SlowGuess/NAACL_2022_OCR huggingface dataset:
or right-leaning (R) vocabulary with their weights. The table intimately reproduces our annotation of ideology. For example, words like federal and Senator allude to the fact that the topic is at the federal level. The importance of education and labor to liberals is also evident in the economic and social dimensions in words like school, education, and wage. The importance of the topic of taxation and defense is evident in conservative ideology in words such as tax, business, missile, and force.
5 Polarization
In this section, we demonstrate how our framework can be used to analyze ideological polarization, quantitatively. To say that two groups are polarized is to say that they are moving to opposite ends of an issue on the multi-dimensional ideological spectrum while, at the same time, their respective political views on ideological issues converge within a group, i.e. socially liberals become also economically liberal (Fiorina and Abrams, 2008). In political science when ideology is multidimensional, polarization is often quantified by considering three measures that capture complementary aspects (Lelkes, 2016): (1) sorting (Abramowitz and Saunders, 1998) (the extent to which the annotated ideology deviates from an outlet's proclaimed ideological bias); (2) issue constraint (Baldassarri and Gelman, 2008) (a correlational analysis between pairs of ideological dimensions); (3) ideological divergence (Fiorina and Abrams, 2008) (the magnitude of the distance between two groups along a single dimension). Together these measures describe changes in the ideological environment over time: a concurrent increase in all three measures indicates polarization in media.
Figure 3: The evolution of the sorting measure, aggregating conservative/neutral/liberal outlets. Moving further away from the zero means articles deviate more from the proclaimed ideology of their outlets.
Limitations: We use only the fully adjudicated data and refrain from using model predictions, since our baseline experiments in Section 6 show that predicting ideology is challenging. Hence, the analysis are demonstrations of what our framework enables, which we discuss at the end of this section, and the conclusions are drawn for our annotated articles only. We group our data in four-year periods to reduce sparsity.
Measure 1: Sorting We adapt the sorting principle of Abramowitz and Saunders (1998) to our data and investigate the difference between the proclaimed ideological bias of a news outlet and the ideology of annotated articles from the outlet. To obtain the bias  $B_{j}$  of a news outlet  $j$ , we average the ratings of each news outlet across common sites that rates media bias (Adfontes, Allsides, and MBFC), yielding: CSM (-0.07), CT (0.15), NYT (-0.36), TM (-0.4), WSJ (0.32) (c.f. Table 8 in Appendix B for ratings from each site).
To obtain the overall ideology  $I_{i}^{(j)}$  of article  $i$  from outlet  $j$ , we take the average of liberal (-1), neutral (0), and conservative (1) labels across its paragraphs in all three dimensions. Thus, for each 4-year time period with  $m$  articles for outlet  $j$ , the sorting measure would be the absolute distance of article vs. outlet ideology  $|\mathrm{avg}_{i=1}^{m}(I_{i}^{(j)}) - B_{j}| / B_{j}$ .
In Figure 3, we plot the sorting measure, averaged across news outlets of the same proclaimed ideological bias. The figure shows that in our sample of articles, the left-leaning news outlets were closest to their proclaimed ideological bias measure over time, whereas the neutral outlets were more liberal before 1957 and after 1964. The right-leaning outlets were more conservative at that time than their proclaimed ideological bias.
Measure 2: Issue constraint This measure refers to the tightness between ideological dimen
Figure 4: The evolution of the issue constraint measure, stratified by pairs of dimensions. Higher values mean some dimensions correlate more strongly than others. Due to the lack of articles that simultaneously contains social & economic dimensions (1st graph), and economic & foreign dimensions (3rd graph) from conservative outlets their respective blue lines start in 1958.
sions over time (Baldassarri and Gelman, 2008) so as to assess, for example, if socially liberal dimensions are more and more associated with economic liberal dimensions for the news outlets. Concretely, for each article we derive its ideology along a single dimension as the average of paragraph annotations along that dimension. We, then, calculate the Pearson correlation between the article ideology of each pair of dimensions, over all articles from one outlet in the same period.
Results in Figure 4, again averaged across news outlets of the same ideological bias, show that for right-leaning media, the correlation between any two dimensions in the annotated data are largely positive (e.g., economically conservative were also socially conservative) until 1967 or 1970. However, for proclaimed left-leaning and neutral outlets, the correlations fluctuates especially when considering the foreign dimension.
Measure 3: Ideological divergence This measures the distance between two ideological groups on a single dimension (Fiorina and Abrams, 2008). We follow Lelkes (2016) and calculate the bimodality coefficient (Freeman and Dale, 2013; Pfister et al., 2013) per dimension over articles from all news outlets over the same time period. The bimodality coefficient ranges from 0 (unimodal, thus not at all polarized) to 1 (bimodal, thus completely
Figure 5: The evolution of the ideological divergence measure stratified by dimension. The dotted line refers to the bimodality threshold (Lelkes, 2016). Higher values mean the ideology of an article along that one dimension is bimodal.
polarized).
Figure 5 shows the evolution of the ideological divergence measure of every dimension. A bimodality measure assesses whether this divergence attained the threshold for the cumulative distribution to be considered bimodal. Ideological distance, as a result, refers to the three bimodal coefficients. We note, for example, that the foreign dimension crossed this threshold between 1956 and 1968. This means that proclaimed left-leaning and right-leaning outlets grew further apart on foreign issues during this time period.
Discussion Taken together, the graphs indicate that the years between 1957 and 1967 are the most noteworthy. During this period, from our sample of articles, we see that polarization was only present in conservative news media because it (1) sorted, as it was significantly more conservative than its composite bias measure, (2) constrained its issues, as evidence by high positive correlation values, and (3) became increasingly bimodal, as the ideological distance between their positions and those of their liberal counterpart on foreign issues increased over time. While this conclusion applies to only the set of articles in our dataset, the above analysis illustrates that our framework enables nuanced, quantitative analyses into polarization. We leave for future work, potentially equipped with strong models for ideology prediction, to analyze the data at scale.
We present political ideology detection experiments as classification tasks per-dimension on the paragraph level.
We performed an 80/10/10 split to create the train, development, and test sets. The development
and test sets contain articles uniformly distributed from our time period (1947 to 1974) such that no particular decade is predominant. To ensure the integrity of the modeling task, all paragraphs belonging to the same article are present in a single split. The number of examples in the splits for each dimension for the adjudicated data are as follows: for the economic dimension, we had 450 training, 50 development, and 58 test examples. For the social dimension, we had 253 for training, 13 for development, and 25 for testing. For the foreign dimension, we had 266 for training, 33 for development, and 39 for testing.
Recurrent neural networks We trained a 2-layer bidirectional LSTM (Hochreiter and Schmidhuber, 1997), with sequence length and hidden size of 256, and 100D GloVe embeddings (Pennington et al., 2014).
Pre-trained language models We used BERT-base (Devlin et al., 2019) from HuggingFace (Wolf et al., 2020) and trained two versions, with and without fine-tuning. In both cases we used a custom classification head consisting of 2 linear layers with a hidden size of 768 and a ReLU between them. To extract the word embeddings we followed Devlin et al. (2019) and used the hidden states from the second to last layer. To obtain the embedding of the whole paragraph<sup>5</sup> we averaged the word embeddings and passed this vector to the classification head.
To find the best hyperparameters we performed a grid search in each dimension. For the economic dimension, the best hyperparameters consisted of a learning rate of 2e-6, 6 epochs of training, a gamma value of 2, no freezing of the layers, a 768 hidden size, and  $10\%$  dropout. For the social dimension, the best hyperparameters were a learning rate of 2e-5, 12 epochs, a gamma of 4, no freezing of the layers, a 768 hidden size, and  $10\%$  dropout. Finally, for the foreign dimension the best hyperparameters consisted of a learning rate of 2e-5, 6 epochs, a gamma of 2, no freezing of the layers, a 768 hidden size, and a  $10\%$  dropout.
Focal loss. To better address the imbalanced label distribution of this task, we incorporated focal loss (Lin et al., 2017), originally proposed for dense object detection. Focal loss can be interpreted as a
Econ	Social	Foreign	Average
Majority	0.30	0.23	0.25	0.26
BiLSTM	0.44	0.37	0.33	0.38
BERT no-ft	0.46	0.31	0.53	0.44
+pre-training	0.42	0.32	0.46	0.40
BERT ft	0.64	0.50	0.52	0.55
+pre-training	0.56	0.47	0.46	0.49
-focal loss	0.61	0.50	0.50	0.54
Table 6: Macro F1 of the models averaged across 10 runs.
dynamically scaled cross-entropy loss, where the scaling factor is inversely proportional to the confidence on the correct prediction. This dynamic scaling, controlled by hyperparameter  $\gamma$ , leads to a higher focus on the examples that have lower confidence on the correct predictions, which in turn leads to better predictions on the minority classes. Since a  $\gamma$  of 0 essentially turns a focal loss into a cross entropy loss, it has less potential to hurt performance than to improve it. We found the best  $\gamma$  values to be 2 or 4 depending on the dimension.
Task-guided pre-training. We also explored supervised pre-training on two adjacent tasks that can give insights to the relationship between tasks. We used distant supervision that labeled the ideological bias of each article according to that of its news outlet from www.allsides.com (Kulkarni et al., 2018). This procedure allowed us to use the unannotated articles.
Table 6 shows the macro F1 for each configuration, averaged across 10 runs with different random initializations. The fine-tuned BERT model, with no task-guided pre-training shows the best performance across all 3 ideology dimensions. It is important to note that all the models do better than randomly guessing, and better than predicting the majority class. This shows that the models are capturing some of the complex underlying phenomena in the data. However, the classification tasks still remain challenging for neural models, leaving plenty of room for improvement in future work.
The BERT ft -focal loss setting ablates the effect of focal loss against a weighted cross entropy loss. with weights inversely proportional to the distribution of the classes in the dimension. This
<sup>6</sup>We also experimented with pre-training on the dataset from Chen et al. (2020). However, because their dataset starts from 2006 (outside of our time domain), this setting performed poorly.
$599\%$  of the paragraphs in the dataset have  $\leq 512$  tokens.
loss helped get a bump in the macro F1 score of around 0.1 for each dimension compared to an unweighted cross entropy loss. However, the focal loss gave further improvements for 2 of the 3 dimensions. Although task-guided pre-training improved the BERT (no fine-tuning) model for 1 of the 3 dimensions, it led to worse performance than BERT (fine-tuned). The improvement on the no fine-tuning setting indicates that there is a potential correlation to be exploited by the ideology of the news outlet, but such labels are not that informative for multi-dimensional prediction. We hope that this dataset provides a testbed for future work to evaluate more distant supervision data/methods.
In contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter (detection of ideological bias in news media) while collapsing ideology to one dimension (Budak et al., 2016; Kulkarni et al., 2018; Kiesel et al., 2019; Baly et al., 2019, 2020; Chen et al., 2020; Ganguly et al., 2020; Stefanov et al., 2020). The proposed computational models classify the partiality of media sources without quantifying their ideology (Elejalde et al., 2018).
Other researchers interested in the computational analysis of the ideology have employed text data to analyze congressional text data at the legislative level (Sim et al., 2013; Gentzkow et al., 2016) and social media text at the electorate level (Saez-Trumper et al., 2013; Barbera, 2015).
In political science, the relationship between (news) media and polarization is also an active area of research. Prior work has studied media ideological bias in terms of coverage (George and Waldfogel, 2006; Valentino et al., 2009). Prior (2013) argues there is no firm evidence of a direct causal relationship between media and polarization and that this relationship depends on preexisting attitudes and political sophistication. On the other hand, Gentzkow et al. (2016) have established that polarization language snippets move from the legislature in the direction of the media whereas (Baumgartner et al., 1997) have shown that the media has an impact on agenda settings of legislatures.
We take the first step in studying multi-dimensional ideology and polarization over time and in news articles relying on the major political science theories and tools of computational linguistics. Our work opens up new opportunities and invites researchers to use this corpus to study the spread of propaganda and misinformation in tandem with ideological shifts and polarization. The presented corpus also provides the opportunity for studying ways that social context determines interpretations in text while distinguishing author stance from content.
This work has several limitations. We only focus on news whereas these dynamics might be different in other forms of communication such as social media posts or online conversations, and the legislature. Further, our corpus is relatively small although carefully annotated by experts. Future work may explore semi-supervised models or active learning techniques for annotating and preparing a larger corpus that may be used in diverse applications.
This research is partially supported by NSF grants IIS-1850153, IIS-2107524, and Good Systems, $^{7}$  a UT Austin Grand Challenge to develop responsible AI technologies. We thank Cutter Dalton, Kathryn Slusarczyk and Ilana Torres for their help with complex data annotation. We thank Zachary Elkins, Richard Lau, Beth Leech, and Katherine McCabe for their feedback of this work and its annotation protocol, and Katrin Erk for her comments. Thanks to Pitt Cyber $^{8}$  for supporting this project. We also acknowledge the Texas Advanced Computing Center (TACC) $^{9}$  at UT Austin and The Center for Research Computing at the University of Pittsburgh for providing the computational resources for many of the results within this paper.
Alan I Abramowitz and Kyle L Saunders. 1998. Ideological realignment in the us electorate. The Journal of Politics, 60(3):634-652.
Theodor Adorno, Else Frenkel-Brenswik, Daniel J
<sup>7</sup>http://goodsystems.utexas.edu
<sup>8</sup>https://www.cyber.pitt.edu
<sup>9</sup>https://www.tacc.utexas.edu
Levinson, and R Nevitt Sanford. 2019. The authoritarian personality. Verso Books.
Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555-596.
Delia Baldassarri and Andrew Gelman. 2008. Partisans without constraint: Political polarization and trends in american public opinion. American Journal of Sociology, 114(2):408-446.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991.
Ramy Baly, Georgi Karadzhov, Abdelhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2109-2116.
Pablo Barbera. 2015. Birds of the same feather tweet together: Bayesian ideal point estimation using twitter data. Political analysis, 23(1):76-91.
Frank R Baumgartner, Bryan D Jones, and Beth L Leech. 1997. Media attention and congressional agendas. Do the media govern, pages 349-363.
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
Alessandro Bessi, Fabio Petroni, Michela Del Vicario, Fabiana Zollo, Aris Anagnostopoulos, Antonio Scala, Guido Caldarelli, and Walter Quattrociocchi. 2016. Homophily and polarization in the age of misinformation. The European Physical Journal Special Topics, 225(10):2047-2059.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? Quantifying media bias through crowdsourced content analysis. *Public Opinion Quarterly*, 80(S1):250-271.
Angus Campbell, Philip E Converse, Warren E Miller, and Donald E Stokes. 1980. The american voter. University of Chicago Press.
Edward G Carmines, Michael J Ensley, and Michael W Wagner. 2012. Who fits the left-right divide? partisan polarization in the american electorate. American Behavioral Scientist, 56(12):1631-1653.
Thomas M Carsey and Geoffrey C Layman. 2006. Changing sides or changing minds? party identification and policy preferences in the american electorate. American Journal of Political Science, 50(2):464-477.
Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing political bias and unfairness in news articles at different levels of granularity. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 149-154.
Dennis Chong and James N Druckman. 2007. Framing theory. Annual Review of Political Science, 10:103-126.
Mark Davies. 2012. Expanding horizons in historical linguistics with the 400-million word corpus of historical american english. Corpora, 7(2):121-157.
Shrey Desai, Barea Sinno, Alex Rosenfeld, and Junyi Jessy Li. 2019. Adaptive ensembling: Unsupervised domain adaptation for political document analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4712-4724.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Paul DiMaggio, John Evans, and Bethany Bryson. 1996. Have american's social attitudes become more polarized? American journal of Sociology, 102(3):690-755.
Erick Elejalde, Leo Ferres, and Eelco Herder. 2018. On the nature of real and perceived bias in the mainstream media. *PloS one*, 13(3):e0193765.
Stanley Feldman and Christopher Johnston. 2014. Understanding the determinants of political ideology: Implications of structural complexity. Political Psychology, 35(3):337-358.
Morris P Fiorina and Samuel J Abrams. 2008. Political polarization in the american public. Annu. Rev. Polit. Sci., 11:563-588.
Jonathan B Freeman and Rick Dale. 2013. Assessing bimodality to detect the presence of a dual cognitive process. Behavior research methods, 45(1):83-97.
Soumen Ganguly, Juhi Kulshrestha, Jisun An, and Haewoon Kwak. 2020. Empirical evaluation of three common assumptions in building political media bias datasets. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 939-943.
Matthew Gentzkow, Jesse M Shapiro, and Matt Taddy. 2016. Measuring polarization in high-dimensional data: Method and application to congressional speech. National Bureau of Economic Research.
Lisa M George and Joel Waldfogel. 2006. The new york times and the market for local newspapers. American Economic Review, 96(1):435-447.
Jonathan Haidt, Jesse Graham, and Craig Joseph. 2009. Above and below left-right: Ideological narratives and moral foundations. *Psychological Inquiry*, 20(2-3):110-119.
Andrew F Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77-89.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780.
Shanto Iyengar, Yphtach Lelkes, Matthew Levendusky, Neil Malhotra, and Sean J Westwood. 2019. The origins and consequences of affective polarization in the united states. Annual Review of Political Science, 22:129-146.
Jeffery A Jenkins, Eric Schickler, and Jamie L Carson. 2004. Constituency cleavages and congressional parties: Measuring homogeneity and polarization, 1857-1913. Social Science History, 28(4):537-573.
Jacob Jensen, Suresh Naidu, Ethan Kaplan, Laurence Wilse-Samson, David Gergen, Michael Zuckerman, and Arthur Spirling. 2012. Political polarization and the dynamics of political language: Evidence from 130 years of partisan speech [with comments and discussion]. Brookings Papers on Economic Activity, pages 1-81.
John T Jost, Christopher M Federico, and Jaime L Napier. 2009. Political ideology: Its structure, functions, and elective affinities. Annual review of psychology, 60:307-337.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. Semeval-2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839.
Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3518-3527.