SlowGuess/NAACL_2022_OCR huggingface dataset:
Yujian Liu $^{1,*}$  Xinliang Frederick Zhang $^{1,*}$  David Wegsman $^{1}$  Nick Beauchamp $^{2}$  Lu Wang $^{1}$
<sup>1</sup>Computer Science and Engineering, University of Michigan, Ann Arbor, MI  
<sup>2</sup>Department of Political Science, Northeastern University, Boston, MA  
<sup>1</sup>{yujuanl,xlfzhang,dwegs,wangluxy}@umich.edu  
<sup>2</sup>n.beauchamp@northeastern.edu
Ideology is at the core of political science research. Yet, there still does not exist general-purpose tools to characterize and predict ideology across different genres of text. To this end, we study Pretrained Language Models using novel ideology-driven pretraining objectives that rely on the comparison of articles on the same story written by media of different ideologies. We further collect a large-scale dataset, consisting of more than 3.6M political news articles, for pretraining. Our model POLITICS outperforms strong baselines and the previous state-of-the-art models on ideology prediction and stance detection tasks. Further analyses show that POLITICS is especially good at understanding long or formally written texts, and is also robust in few-shot learning scenarios.
Ideology is an ubiquitous factor in political science, journalism, and media studies (Mullins, 1972; Freedom, 2006; Martin, 2015). Decades of work has gone into measuring ideology based on voting data (Poole and Rosenthal, 1985; Lewis et al., 2021), survey results (Preotciuc-Pietro et al., 2017; Ansolabehere et al., 2008; Kim and Fording, 1998; Gabel and Huber, 2000), social networks (Barbera et al., 2015), campaign donation records (Bonica, 2013), and textual data (Laver et al., 2003; Diermeier et al., 2012; Gentzkow et al., 2019; Volkens et al., 2021). Each of those approaches has its strengths and weaknesses. For instance, many political figures do not have voting records; surveys are expensive and politicians are often unwilling to disclose ideology. By contrast, political text is abundant, ubiquitous, yet challenging to work with since language is complex in nature, often domain-specific, and generally unlabeled. There thus remains a strong need for general-purpose tools for measuring ideology using text that can be applied across multiple genres.
News Story: Donald Trump tests positive for COVID-19.
Daily Kos (left): It's now clear that Donald Trump lied to the nation about when he received a positive test for COVID-19. ... they're continuing to act as if nothing has changed—and that disregarding science and lying to the public are the only possible strategies.
The Washington Times (right): Trump says he's "doing very well" ... President Trump thanked the nation for supporting him Friday night as he left the White House to be hospitalized for COVID-19. "I want to thank everybody for the tremendous support. ..." Mr. Trump said in a video recorded at the White House.
Breitbart (right): President Donald Trump thanked Americans for their support on Friday as he traveled to Walter Reed Military Hospital for further care after he was diagnosed with coronavirus. "I think I'm doing very well...". Trump said in a video filmed at the White House and posted to social media.
Figure 1: Article snippets by different media on the same news story. Contents that indicate stances and ideological leanings are highlighted in **bold** (for subjective phrases) and in **italics** (for objective events).
Using text as data, computational models for ideology measurement have rapidly expanded and diversified, including classical machine learning methods such as ideal point estimation (Grose-close et al., 1999; Shor and McCarty, 2011), Naive Bayes (Evans et al., 2007), support vector machines (Yu et al., 2008), latent variable models (Barbera et al., 2015), and regression (Peterson and Spirling, 2018); and more recent neural architectures like recurrent neural networks (Iyyer et al., 2014) and Transformers (Baly et al., 2020; Liu et al., 2021). Nonetheless, most of those models leverage datasets with ideology labels drawn from a single domain, and it is unclear if any of them can be generalized to diverse genres of text.
Trained on massive quantities of data, Pretrained Language Models (PLMs) have achieved state-of-the-art performance on many text classification problems, with an additional fine-tuning stage on labeled task-specific samples (Devlin et al., 2019; Liu
* Equal contribution by the first two authors.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1354 - 1374 July 10-15, 2022 ©2022 Association for Computational Linguistics
et al., 2019). Though PLMs suggest the promise of generalizable solutions, their ability to acquire the knowledge needed to detect complex features such as ideology from text across genres remains an open question. PLMs have been shown to capture linguistic structures with a local focus, such as task-specific words, syntactic agreement, and semantic compositionality (Clark et al., 2019; Jawahar et al., 2019). Although word choice is indicative of ideology, ideological leaning and stance are often revealed by which entities and events are selected for presentation (Hackett, 1984; Christie and Martin, 2005; Enke, 2020), with the most notable strand of work in framing theory (Entman, 1993, 2007). One such example is demonstrated in Figure 1, where Daily Kos criticizes Trump's dishonesty while The Washington Times and Breitbart emphasize the good condition of his health.
In this work, we propose to train PLMs for a wide range of ideology-related downstream tasks. We argue that it is critical for PLMs to consider the global context of a given article. For instance, as pointed out by Fan et al. (2019), one way to acquire such context is through comparison of news articles on the same story but reported by media of different ideologies. Given the lack of suitable datasets, we first collect a new large-scale dataset, BIGNews. It contains 3,689,229 English news articles on politics, gathered from 11 United States (US) media outlets covering a broad ideological spectrum. We further downsample and cluster articles in BIGNEWS by different media into groups, each consisting of pieces aligned on the same story. The resultant dataset, BIGNEWSALIGN, contains 1,060,512 stories with aligned articles.
Next we train a new PLM, POLITICS, based on a Pretraining Objective Leveraging Inter-article Triplet-loss using Ideological Content and Story. Concretely, we leverage continued pretraining (Gururanget al., 2020), where we design an ideology objective operating over clusters of same-story articles to compact articles with similar ideology and contrast them with articles of different ideology. The learned representation can better discern the embedded ideological content. We further enhance it with a story objective that ensures the model to focus on meaningful content instead of overly relying on shortcuts, e.g., media boilerplate. Both objectives are used together with our specialized
masked language model objective that focuses on entities and sentiments to train POLITICS.
Our main goal here is to create general-purpose tools for analyzing ideological content for researchers and practitioners in the broad community. Furthermore, when experimenting on 11 ideology prediction and stance detection tasks using 8 datasets of different genres, including a newly collected dataset from AllSides, POLITICS outperforms both a strong SVM baseline and previous PLMs on 8 tasks. Notably, POLITICS is particularly effective on long documents, e.g., achieving  $10\%$  improvements on both ideology prediction and stance detection tasks over RoBERTa (Liu et al., 2019). We further show that our model is more robust in setups with smaller training sets.
Ideology prediction is a critical task for quantitative political science (Mullins, 1972; Freedom, 2006; Martin, 2015; Wilkerson and Casas, 2017). Both classical methods (e.g., Naive Bayes, SVM; Evans et al., 2007; Yu et al., 2008; Sapiro-Gheiler, 2019) and deep learning models (e.g., RNN; Iyyer et al., 2014) have been used to predict ideology on a variety of datasets where ideology labels are available, such as legislative speeches (Laver et al., 2003) and U.S. Supreme Court briefs (Evans et al., 2007). Notably, Liu et al. (2021) pretrains a Transformer-based language generator to minimize the ideological bias in generated text. As generative models are not as effective as masked language models (MLMs) at text classification, our goal differs in that we train MLMs to recognize ideological contents in various domains and tasks.
Stance detection is a useful task for ideology analysis because co-partisans are generally positive towards each other and negative towards counterpartisans (Aref and Neal, 2021). There has been a large body of work on identifying individuals' stances towards specific targets from the given text (Thomas et al., 2006; Walker et al., 2012; Hasan and Ng, 2013). On the methodology side, Mohammad et al. (2016b) and Kucuk and Can (2018) apply statistical models, e.g., SVM, with handcrafted text features. Neural methods have also been widely investigated, including CNN (Wei et al., 2016), LSTM (Augenstein et al., 2016), hierarchical networks (Sun et al., 2018), and representation learning (Darwish et al., 2020).
Recent research focus resides in leveraging
Our data and code can be accessed at https://github.com/launchnlp/POLITICS.
Daily
Kos	HPO	CNN	WaPo	NYT	USA
Today	AP	The Hill	TWT	FOX	Breibhart
Ideology	L	L	L	L	L	C	C	C	R	R	R
# articles	100,828	241,417	64,988	198,529	173,737	170,737	279,312	322,145	243,181	330,166	206,512
# words	738.7	729.9	655.7	803.2	599.4	691.7	572.3	426.3	522.7	773.5	483.5
Table 1: Statistics of BIGNEWSBLN. Media outlets are sorted by ideology from left (L), center (C), to right (R) based on AllSides and Media Bias Chart. HPO: Huffington Post; WaPo: The Washington Post; NYT: The New York Times; TWT: The Washington Times. Additional statistics of raw data size before downsampling and the corresponding publish dates can be found in Table A4.
PLMs for predicting stances, e.g., incorporating extra features (Prakash and Madabushi, 2020). Kawintiranon and Singh (2021) share a similar spirit with our work by upsampling tokens to mask. However, they pre-define a list of tokens customized for the given targets, which is hard to generalize to new targets. We aim to train PLMs relying on general-purpose sentiment lexicons and important entities, to foster model generalizability.
Domain-specific Pretrained Language Models. PLMs, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have obtained state-of-the-art results on many NLP tasks. Inspired by the observation that a continued pretraining phase on in-domain data yields better performance (Gururangan et al., 2020), domain-specific PLMs are introduced (Beltagy et al., 2019; Yang et al., 2020; Huang et al., 2019; Lee et al., 2020). However, they only use the default MLM objective, without considering domain knowledge. In this work, we design ideology-driven pretraining objectives to inject domain knowledge to discern ideologies and related stances.
Focusing on the news domain, PLMs have been primarily used for factuality prediction (Jwa et al., 2019; Zellers et al., 2019; Kaliyar et al., 2021) and topic classification (Liu et al., 2020; Buyukoz et al., 2020; Gupta et al., 2020) by fine-tuning on task-specific datasets. Few work has investigated PLMs for understanding political ideology evinced in texts. One exception is Baly et al. (2020), where they also leverage the triplet loss as the pretraining objective. However, our work is novel in at least three aspects. First, our triplet loss is designed to capture the ideological (dis)similarity among articles on the same story, while the loss used by Baly et al. (2020) operates on articles of the same topic. As a result, their approach can falsely compact representations of very different news contents, e.g., articles on "Japan Economics" and "Indian Troops" both belong to the topic of "Asia". Moreover, our
newly introduced story objective can effectively prevent the model from relying on media-specific language (e.g., "for the New York Times"), while their objective may fail to do so, and thus lacks generalizability to languages used by different media and other ideology-related tasks. Finally, we use BIGNews that contains more than 3M articles, which is more suitable for pretraining large models than the small dataset (35k articles) used by Baly et al. (2020). To the best of our knowledge, we are the first to systematically study and release PLMs for ideology-related study in the US political domain.
3 Pretraining Datasets
3.1 Data Crawling
We collect pretraining datasets from online news articles with diverse ideological leanings and language usage. We select 11 media outlets based on their ideologies (from far-left to far-right) and popularity.<sup>2</sup> We convert their ideologies into three categories: left, center, and right, and crawl all pages published by them between January 2000 and June 2021, from Common Crawl and Internet Archive. We then follow Raffel et al. (2020) for data cleaning, and, additionally, only retain news articles related to US politics. Appendix A describes in detail the steps for removing non-articles pages, duplicates, non-US pages, and boilerplate.
The cleaned data, dubbed BIGNEWS, contains 3,689,229 US political news articles. To mitigate the bias that some media dominate the model training, we downsample the corpus so that each ideology contributes equally. The downsampled corpus, BIGNEWSBLN, contains 2,331,552 news articles, with statistics listed in Table 1. We keep 30K held-out articles as validation set.
2We use https://www.allsides.com and https: //adfontesmedia.com to decide ideology and https: //www.alexa.com/topsites to decide popularity.
3.2 Aligning Articles on the Same Story
We compare how media outlets from different sides report the same story, which intuitively better captures ideological content. To this end, we design an algorithm to align articles in BIGNEWSBLN that cover the same story. We treat each article as an anchor, and find matches from other outlets based on the following similarity score:
$$
\sin \left(p _ {i}, p _ {j}\right) = \alpha * \sin_ {t} \left(p _ {i}, p _ {j}\right) + (1 - \alpha) * \sin_ {e} \left(p _ {i}, p _ {j}\right) \tag {1}
$$
where  $p_i$  and  $p_j$  are two articles,  $\mathrm{sim}_t$  is the cosine similarity between TF-IDF vectors of  $p_i$  and  $p_j$ ,  $\mathrm{sim}_e$  is the weighted Jaccard similarity between the sets of named entities in  $p_i$  and  $p_j$ , and  $\alpha = 0.4$  is a hyperparameter. During alignment, for an article from an outlet to be considered as a match, it must be published within three days before or after the anchor, has the highest similarity score among articles from the same outlet, and the score is at least  $\theta = 0.23$ . Hyperparameters  $\alpha$  and  $\theta$  are searched on the Basil dataset (Fan et al., 2019), which contains manually aligned articles. After deduplicating articles in each story cluster, we obtain BIGNEWSALIGN, containing 1,060,512 clusters with an average of 4.29 articles in each. Appendix B details the alignment algorithm.
4 POLITICS via Continued Pretraining
Here we introduce our continued pretraining methods based on a newly proposed ideology objective that drives representation learning to better discern ideological content by comparing same-story articles (§4.1), which is further augmented by a story objective to better focus on content. They are combined with the masked language model objective, which is tailored to focus on entities and sentiments (§4.2), to produce POLITICS (§4.3).
4.1 Ideology-driven Pretraining Objectives
To promote representation learning that better captures ideological content, we leverage BIGNEWSALIGN with articles grouped by stories to provide story-level background for model training. That is, we use triplet loss (Schroff et al., 2015) that operates over triplets of  $<$  anchor, positive, negative  $>$  to encourage anchor and positive samples to have closer representations while contrasting anchor from negative samples.
Figure 2: Construction of the ideology and story objectives. The middle CNN article is the anchor in this example. Solid black arrow represents positive-pair relation for both objectives; red dashed arrow denotes negative-pair for ideology objective; orange dashed arrow indicates negative-pair for story objective.
Our primary pretraining objective, i.e., ideology objective, uses the triplet loss to teach the model to acquire ideology-informed representations by comparing same-story articles written by media of different ideologies. As shown in Figure 2, given a story cluster, we choose an article published by media on the left or right as the anchor. We then take articles in the same cluster with the same ideology as positive samples, and articles with the opposite ideology as negative ones. The ideology objective is formulated as follows:
$$
\mathcal {L} _ {\mathrm {i d e o}} = \sum_ {t \in \mathcal {T} _ {\mathrm {i d e o}}} \left[ \left\| \mathbf {t} ^ {(\mathbf {a})} - \mathbf {t} ^ {(\mathbf {p})} \right\| _ {2} - \left\| \mathbf {t} ^ {(\mathbf {a})} - \mathbf {t} ^ {(\mathbf {n})} \right\| _ {2} + \delta_ {\mathrm {i d e o}} \right] _ {+} \tag {2}
$$
where  $\mathcal{T}_{\mathrm{ideo}}$  is the set of all ideology triplets,  $\mathbf{t}^{(\mathbf{a})}$ ,  $\mathbf{t}^{(\mathbf{p})}$ , and  $\mathbf{t}^{(\mathbf{n})}$  are the [CLS] representations of anchor, positive, and negative articles in triplet  $t$ ,  $\delta_{\mathrm{ideo}}$  is a hyperparameter, and  $[\cdot]_{+}$  is  $max(\cdot, 0)$ .
Next, we augment the ideology objective with a story objective to allow the model to focus on semantically meaningful content and to prevent the model from focusing on "shortcuts" (such as media-specific languages) to detect ideology. To construct story triplets, we use the same  $\langle$  anchor, positive  $\rangle$  pairs as in the ideology triplet, and then treat articles from the same media outlet but on different stories as negative samples, as depicted in Figure 2. Similarly, our story objective is formulated as follows:
$$
\mathcal {L} _ {\text {s t o r y}} = \sum_ {t \in \mathcal {T} _ {\text {s t o r y}}} \left[ \left\| \mathbf {t} ^ {(\mathbf {a})} - \mathbf {t} ^ {(\mathbf {p})} \right\| _ {2} - \left\| \mathbf {t} ^ {(\mathbf {a})} - \mathbf {t} ^ {(\mathbf {n})} \right\| _ {2} + \delta_ {\text {s t o r y}} \right] _ {+} \tag {3}
$$
where  $\mathcal{T}_{\mathrm{story}}$  contains all story triplets, and  $\delta_{\mathrm{story}}$  is a hyperparameter searched on the validation set.
3Extracted by Stanford CoreNLP (Manning et al., 2014). Our algorithm achieves a mean reciprocal rank of 0.612 on Basil, with detailed evaluation in Appendix B.
4.2 Entity- and Sentiment-aware MLM
Here we present a specialized MLM objective to collaborate with our triplet loss based objectives for better representation learning. Notably, political framing effect is often reflected in which entities are selected for reporting (Gentzkow et al., 2019). Moreover, the occurrence of sentimental content along with the entities also signal stances (Mohammad et al., 2016b). Therefore, we take a masking strategy that upsamples entity tokens (Sun et al., 2019; Guu et al., 2020; Kawintiranon and Singh, 2021) and sentiment words to be masked for the MLM objective, which improves from prior pretraining work that only considers article-level comparison (Baly et al., 2020).
Concretely, we consider named entities with types of PERSON, NORP, ORG, GPE and EVENT. We detect sentiment words using lexicons by Hu and Liu (2004) and Wilson et al. (2005). To allow MLM training to focus on entities and sentiment, we mask them with a  $30\%$  probability, and then randomly mask remaining tokens until  $15\%$  of all tokens are reached, as done in Devlin et al. (2019). Masked tokens are replaced with [MASK], random tokens, and original tokens with a ratio of 8:1:1.
4.3 Overall Pretraining Objective
We combine the aforementioned objectives as our final pretraining objective as follows:
$$
\mathcal {L} = \beta * \mathcal {L} _ {\text {i d e o l o g y}} + \gamma * \mathcal {L} _ {\text {s t o r y}} + (1 - \beta - \gamma) * \mathcal {L} _ {\text {M L M}} \tag {4}
$$
where  $\beta = \gamma = 0.25$ . Using  $\mathcal{L}$ , POLITICS is produced via continued training on RoBERTa (Liu et al., 2019). We do not try to train the model from scratch since BIGNEWSBLN only has  $\sim 10$ GB data, smaller than corpus for RoBERTa ( $\sim 160$ GB). Hyperparameters are listed in Table A5.
Given the importance of ideology prediction and stance detection tasks in political science (Thomas et al., 2006; Wilkerson and Casas, 2017; Chatsiou and Mikhaylov, 2020), we conduct extensive experiments on a wide spectrum of datasets with 11 tasks ( $\S 5.1$ ). We then compare with both classical models and prior PLMs ( $\S 5.2$ ), and among our model variants ( $\S 5.3$ ). We present and discuss results in  $\S 5.5$ , where POLITICS outperform all three baselines on 8 out of 11 tasks. For all models,
Data	Genre	# Train	Len.	Split
Congress Speech (Gentzkow et al., 2018)	speech	7,000	538	rand.
AllSides (newly collected)	news	7,878	863	time
BASIL-article (Fan et al., 2019)	news	450	693	story
BASIL-sentence (Fan et al., 2019)	news	1,197	27	story
Hyperpartisan (Kiesel et al., 2019)	news	425	556	rand.
VAST (Allaway and McKeown, 2020)	cmt	11,545	102	rand.†
YouTube User (Wu and Resnick, 2021)	cmt	1,114	1,213	user
YouTube Cmt (Wu and Resnick, 2021)	cmt	6,832	197	user
SemEval (Mohammad et al., 2016a)	tweet	2,251	17	rand.†
Twitter (Preojuc-Pietro et al., 2017)	tweet	1,079	2,298	user
Table 2: Datasets used for evaluating PLMs vary in text genre, training set size (# Train), length (Len.), and split criterion. Rand. denotes random split. Time split means training on the "past" data and test on the "future". Story split divides articles according to story clusters. User split indicates users in the test are unseen in the training.†: by the original work.
MLM objectives are trained with BIGNewsBLN, and ideology and story objectives are trained on BIGNewsALIGN. Details are in Appendix C.1.
Our tasks are discussed below, with statistics listed in Table 2 and more descriptions in Appendix D.
Ideology prediction tasks for predicting the political leanings are evaluated on the following datasets.
Congress Speech (CongS; Gentzkow et al., 2018) contains speeches from US congressional records, each labeled as liberal or conservative.
- AllSides  $^{6}$  (AllS, new) is a website that assesses political bias and ideology of US media. In this study, we collect articles from AllSides with their ideological leanings on a 5-point scale.
- Hyperpartisan (HP; Kiesel et al., 2019) is a shared task of predicting a binary label for an article as being hyperpartisan or not. We convert it into a 3-way classification task by splitting hyperpartisan news into left and right.
YouTube (YT; Wu and Resnick, 2021) contains discussions on YouTube. cmt. and user refer to predicting left/right at the comment- and user-level, respectively.
- Twitter (TW; Preoiciuc-Pietro et al., 2017) collects a group of Twitter users with self-reported ideologies on a 7-point scale. We merge them into 3-way labels.
Stance detection tasks, which predict a subject's attitude towards a given target from a piece of text, are listed below. All tasks take a 3-way label
<sup>5</sup>We use roberta-base model card from Huggingface.
$^{6}$ https://www.allsides.com.