SlowGuess/NAACL_2021_OCR huggingface dataset:
Julia Mendelsohn
University of Michigan juliame@umich.edu
Ceren Budak
University of Michigan cbudak@umich.edu
David Jurgens
University of Michigan jurgens@umich.edu
The framing of political issues can influence policy and public opinion. Even though the public plays a key role in creating and spreading frames, little is known about how ordinary people on social media frame political issues. By creating a new dataset of immigration-related tweets labeled for multiple framing typologies from political communication theory, we develop supervised models to detect frames. We demonstrate how users' ideology and region impact framing choices, and how a message's framing influences audience responses. We find that the more commonly-used issue-generic frames obscure important ideological and regional patterns that are only revealed by immigration-specific frames. Furthermore, frames oriented towards human interests, culture, and politics are associated with higher user engagement. This large-scale analysis of a complex social and linguistic phenomenon contributes to both NLP and social science research.
Framing selects particular aspects of an issue and makes them salient in communicating a message (Entman, 1993). Framing can impact how people understand issues, attribute responsibility (Iyengar, 1991), and endorse possible solutions, thus having major implications for public opinion and policy decisions (Chong and Druckman, 2007). While past work has studied framing by the news media and the political elite, little is known about how ordinary people frame political issues. Yet, framing by ordinary people can influence others' perspectives and may even shape elites' rhetoric (Russell Neuman et al., 2014). To shed light on this important topic, we focus on one issue—immigration—and develop a new methodology to computationally analyze its framing on Twitter.
Our work highlights unique insights that social media data offers. The massive amount of available
social media content enables us to compare framing strategies across countries and political ideologies. Furthermore, social media provides unique insights into how messages resonate with audiences through interactive signals such as retweets and favorites. By jointly analyzing the production and reception of frames on Twitter, we provide an in-depth analysis of immigration framing by and on the public.
Political communications research has identified numerous typologies of frames, such as issue-generic policy, immigration-specific, and narrative. Each of these frame types can significantly shape the audience's perceptions of an issue (Iyengar, 1991; Chong and Druckman, 2007; Lecheler et al., 2015), but prior NLP work seeking to detect frames in mass media (e.g. Card et al., 2016; Field et al., 2018; Kwak et al., 2020) has largely been limited to a single issue-generic policy typology. Multiple dimensions of framing must be considered in order to better understand the structure of immigration discourse and its effect on public opinion and attitudes. We thus create a novel dataset of immigration-related tweets containing labels for each typology to facilitate more nuanced computational analyses of framing.
This work combines political communication theory with NLP to model multiple framing strategies and analyze how the public on Twitter frames immigration. Our contributions are as follows: (1) We create a novel dataset of immigration-related tweets labeled for issue-generic policy, immigration-specific, and narrative frames. (2) We develop and evaluate multiple methods to detect each type of frame. (3) We illustrate how a message's framing is influenced by its author's ideology and country. (4) We show how a message's framing affects its audience by analyzing favoriting and retweeting behaviors. Finally, our work highlights the need to consider multiple framing typologies and their effects.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2219-2263 June 6-11, 2021. ©2021 Association for Computational Linguistics
2 Framing in the Media
Framing serves four functions: (i) defining problems, (ii) diagnosing causes, (ii) making evaluative judgments, and (iv) suggesting solutions (Entman, 1993). Framing impacts what people notice about an issue, making it a key mechanism by which a text influences its audience.
Framing Typologies We draw upon distinct typologies of frames that can be applied to the issue of immigration: (1) issue-specific, which identify aspects of a particular issue, or (2) issue-generic, which appear across a variety of issues and facilitate cross-issue comparison (de Vreese, 2005).
Issue-generic frames include policy frames that focus on aspects of issues important for policymaking, such as economic consequences or fairness and equality (Boydstun et al., 2013). Other generic frames focus on a text's narrative; news articles use both episodic frames, which highlight specific events or individuals, and thematic frames, which place issues within a broader social context. The use of episodic versus thematic frames can influence the audience's attitudes. For example, episodic frames lead audiences to attribute responsibility for issues such as poverty to individual citizens while thematic frames lead them to hold the government responsible (Iyengar, 1991).
Issue-specific frames for immigration focus on the portrayal of immigrants. Our analysis uses Benson (2013)'s set of issue-specific frames, which represent immigrants as heroes (cultural diversity, integration, good workers), victims (humanitarian, global economy, discrimination), and threats (to jobs, public order, taxpayers, cultural values).
Both issue-specific and generic frames provide unique insights but present advantages and drawbacks. While issue-specific frames analysis are specific and detailed, they are hard to generalize and replicate across studies, which is a key advantage for generic frames (de Vreese, 2005).
Framing effects Studies of framing typically focus on either frame-building or framesetting (Scheufele, 1999; de Vreese, 2005). Frame-building is the process by which external factors, such as a journalist's ideology or economic pressures, influence what frames are used; frame-building studies thus treat framing as the dependent variable. Frame-setting studies treat frames as independent variables that impact how an audience interprets and evaluates issues.
Prior analyses of frame-building in immigration
news highlight region and ideology as particularly important factors. Right-leaning media from conservative regions are more likely to frame immigrants as intruders (van Gorp, 2005), and as threats to the economy and public safety (Fryberg et al., 2012). Framing also differs across countries; while the US press emphasizes public order, discrimination, and humanitarian concerns, the French press more frequently frames immigrants as victims of global inequality (Benson, 2013).
Frame-setting has also been studied in the context of immigration. For example, experimental work has shown that frames eliciting angry or enthusiastic emotions impact participants' opinions on immigration (Lecheler et al., 2015). While past work has analyzed linguistic framing in Twitter immigration discourse (e.g., de Saint Laurent et al., 2020), little is known about how such framing affects users' interactive behaviors such as resharing content, which is a key objective of frame setting.
3 Computational Approaches to Framing
Because many people now generate and consume political content on social media, scholars have increasingly used automated techniques to study framing on social media.
Large-scale research of framing on Twitter has commonly focused on unsupervised approaches. (e.g., Russell Neuman et al., 2014; Meraz and Papcharissi, 2013; de Saint Laurent et al., 2020). Such approaches, including those focused on hashtag analysis, can reveal interesting framing patterns. For instance, Siapera et al. (2018) shows that frame usage varies across events. Similarly, topic models have been used to compare "refugee crisis" media discourses across the European countries (Heidenreich et al., 2019), and to uncover differences in attitudes towards migrants (Hartnett, 2019). Although lexicon analysis and topic models can provide insights about immigration discourse, here, we adopt a supervised approach to ground our work in framing research and to enable robust evaluation.
We draw inspiration from a growing body of NLP research that uses supervised approaches to detect issue-generic policy frames in news articles, a task popularized by the Media Frames Corpus (Card et al., 2015), which contains issue-generic policy frame labels for articles across several issues (Boydstun et al., 2013). Using this corpus, prior work has detected frames with techniques including logistic regression (Card et al.,
Frame Type	Frame	Description
Issue-Generic Policy	Economic	Financial implications of an issue
Capacity & Resources	The availability or lack of time, physical, human, or financial resources
Morality & Ethics	Perspectives compelled by religion or secular sense of ethics or social responsibility
Fairness & Equality	The (in)equality with which laws, punishments, rewards, resources are distributed
Legality, Constitutionality & Jurisdiction	Court cases and existing laws that regulate policies; constitutional interpretation; legal processes such as seeking asylum or obtaining citizenship; jurisdiction
Crime & Punishment	The violation of policies in practice and the consequences of those violations
Security & Defense	Any threat to a person, group, or nation and defenses taken to avoid that threat
Health & Safety	Health and safety outcomes of a policy issue, discussions of health care
Quality of Life	Effects on people's wealth, mobility, daily routines, community life, happiness, etc.
Cultural Identity	Social norms, trends, values, and customs; integration/assimilation efforts
Public Sentiment	General social attitudes, protests, polling, interest groups, public passage of laws
Political Factors & Implications	Focus on politicians, political parties, governing bodies, political campaigns and debates; discussions of elections and voting
Policy Prescription & Evaluation	Discussions of existing or proposed policies and their effectiveness
External Regulation & Reputation	Relations between nations or states/provinces; agreements between governments; perceptions of one nation/state by another
Immigration Specific	Victim: Global Economy	Immigrants are victims of global poverty, underdevelopment and inequality
Victim: Humanitarian	Immigrants experience economic, social, and political suffering and hardships
Victim: War	Focus on war and violent conflict as reason for immigration
Victim: Discrimination	Immigrants are victims of racism, xenophobia, and religion-based discrimination
Hero: Cultural Diversity	Highlights positive aspects of differences that immigrants bring to society
Hero: Integration	Immigrants successfully adapt and fit into their host society
Hero: Worker	Immigrants contribute to economic prosperity and are an important source of labor
Threat: Jobs	Immigrants take nonimmigrants' jobs or lower their wages
Threat: Public Order	Immigrants threaten public safety by being breaking the law or spreading disease
Threat: Fiscal	Immigrants abuse social service programs and are a burden on resources
Threat: National Cohesion	Immigrants' cultural differences are a threat to national unity and social harmony
Narrative	Episodic	Message provides concrete information about on specific people, places, or events
Thematic	Message is more abstract, placing stories in broader political and social contexts
Table 1: List of all issue-generic policy (Boydstun et al., 2013), immigration-specific (Benson, 2013; Hovden and Mjelde, 2019), and narrative (Iyengar, 1991) frames with brief descriptions.
2016), recurrent neural networks (Naderi and Hirst, 2017), lexicon induction (Field et al., 2018), and fine-tuning pretrained language models (Khanehzar et al., 2019; Kwak et al., 2020). Roy and Goldwasser (2020) further extracted subcategories of issue-generic policy frames in newspaper coverage using a weakly-supervised approach. Finally, issue-generic frames have also been computationally studied in other media, including online fora and politicians' tweets (Johnson et al., 2017; Hartmann et al., 2019). We build upon this literature by incorporating additional frame typologies that reflect important dimensions of media discourse with real-world consequences (Iyengar, 1991; Gross, 2008; Eberl et al., 2018). Beyond detecting frames, we computationally analyze frame-building and frame-setting among social media users; though well-studied in traditional news media, little is known about how social media users frame immigration or its effects (Eberl et al., 2018).
Noting that issue-generic policy frames obscure important linguistic differences, several works stud
ied issue-specific frames in news media for issues such as missile defense and gun violence (Morstatter et al., 2018; Liu et al., 2019a). We extend issue-specific frame analyses to immigration by adopting an immigration-specific typology developed by political communication scholars (Benson, 2013).
In contrast to prior NLP work focused on traditional media or political elites (Johnson et al., 2017; Field et al., 2018), we highlight the role that social media publics play in generating and propagating frames. Furthermore, we provide a new computational model of narrative framing (Iyengar, 1991), that together with models for issue-generic policy and issue-specific frames, provides complementary views on the framing of immigration. Finally, our large-scale analysis of frame-setting illustrates the potential for using NLP to understand how a message's framing shapes its audience behavior.
We first collect a large dataset of immigration-related tweets, and then annotate a subset of this
full dataset for multiple types of frames.
Data Collection We extract all English-language tweets in 2018 and 2019 from the Twitter Decahose containing at least one of the following terms: immigration, immigrant(s), emigration, emigrant(s), migration, migrant(s), illegal alien(s), illegals, and undocumented<sup>1</sup>. We focus on content creation and thus exclude retweets from our dataset, though we consider retweeting rates when analyzing the social influence of different frames. We further restrict our dataset to tweets whose authors are identified as being located in the United States (US), United Kingdom (GB), and European Union (EU) by an existing location inference tool (Compton et al., 2014). To compare framing across political ideologies, we obtain ideal point estimates for nearly two-thirds of US-based users with Barbera (2015)'s Bayesian Spatial Following model. Our full dataset contains over 2.66 million tweets,  $86.2\%$  of which are from the United States,  $10.4\%$  from the United Kingdom, and  $3.4\%$  from the European Union.
Data Annotation Tweets are annotated using three frame typologies: (i) issue-generic policy, (ii) immigration-specific, and (iii) narrative frames, where a tweet may use multiple frames simultaneously. We use Boydstun et al. (2013)'s Policy Frames Codebook to formulate our initial guidelines to code for policy frames. We use Benson (2013)'s immigration-specific frames, but follow Hovden and Mjelde (2019) in including an additional category for framing immigrants as victims of war. Finally, we code for narrative frames using definitions from Iyengar (1991). All frames and descriptions can be found in Table 1, with a complete codebook in Supplementary Materials. Because annotation guidelines from prior work focus on elite communications, we first adjusted our codebook to address challenges posed by Twitter content. Changes were made based on feedback from four trained annotators who labeled 360 tweets from 2018, split between the EU, GB, and US.
Even for humans, identifying frames in tweets is a difficult task. Defining the boundaries of what constitutes a message is not trivial. Beyond the text, frames could be identified in hashtags, images, videos, and content from linked pages. Furthermore, tweets are often replies to other users or part of a larger thread. This additional context may
influence an issue's framing. For simplicity, we treat each tweet as a standalone message and label frames based only on the text (including hashtags).
Unlike news stories, where frames are clearly cued, tweets often implicitly allude to frames due to character limitations. For example, a tweet expressing desire to "drive immigrants out" with no additional context may suggest a criminal frame, but criminality is not explicit. To minimize errors, we avoid making assumptions about intended meaning and interpret all messages literally.
Training, development, and test data were annotated using two procedures after four annotators completed four rounds of training. The dataset contains equal numbers of tweets from the EU, UK, and US. Training data was singly annotated and includes 3,600 tweets, while the development and test sets each contain 450 tweets (10% of the full dataset) and were consensus-coded by pairs of trained annotators. We opt for this two-tier approach due to (i) the inherent difficulty of the task $^2$  and (ii) the need to maximize diversity seen in training. During annotator training, pilot studies attained moderate agreement, suggesting that to attain high-reliability, consensus coding with adjudication would be needed (Krippendorff, 2013), which comes at a cost of substantially increased time. Because a large dataset of unique, singly-coded documents is preferable to a small dataset of documents coded by multiple annotators for text classification (Barbera et al., 2021), we decided to increase corpus diversity in the training data by singly-annotating, at the expense of potentially noisier annotation, and to consensus code all evaluation data. On the double annotated data, annotators attained Krippendorff's  $\alpha = 0.45$ . Additional details are provided in Supplementary Material ( $\S$ B, Figures 6 and 7).
Results We observe differences across frame typologies in coverage rates within the annotated data set. While  $84\%$  of tweets are labeled with at least one issue-generic policy frame and  $85\%$  with at least one narrative frame, only  $51\%$  are labeled with at least one issue-specific frame. This difference is due to immigration-specific frames being more narrowly-defined, as they require explicit judgment of immigrants as heroes, victims, or threats. Further details about frame distributions
<sup>1</sup>We obtained this list by starting with the seed terms immigrants, immigration, and illegal aliens. We then added the remaining terms by manually inspecting and filtering nearby words in pretrained GloVe and Word2Vec vector spaces.
For example, in identifying just the primary issue-generic frame of a document, the Media Frames corpus attained an Krippendorff's  $\alpha = \sim 0.6$  (Card et al., 2015, Fig. 4), whereas we ask annotators to identify all frames across three typologies.
Random	LogReg	RoBERTa	FT RoBERTa
0.193	0.296	0.611	0.657
Table 2: F1 scores on the test set for all models, calculated as an (unweighted) average over all frames and initialization seeds. The fine-tuned (FT) RoBERTa model improvements over all models are significant at  $p < 0.001$  (McNemar's test).
Frame Type	Precision	Recall	F1 Score	LRAP
Issue-Generic Policy	0.727	0.721	0.711	0.750
Issue-Specific	0.593	0.531	0.552	0.806
Narrative	0.757	0.887	0.808	0.894
Table 3: Test set performance on each frame typology.
in our annotations can be found in Supplementary Material (§A, Figure 5).
While the precision of issue-specific frames can reveal patterns otherwise obscured by the broader issue-generic frames, this lack of coverage presents two challenges: 1) automated detection is more challenging given this sparsity and 2) analyses of issue-specific frames do not capture a large portion of immigration-related discourse. By incorporating multiple framing strategies, we leverage both the coverage of issue-generic frames and the precision and interpretability of issue-specific frames.
5 Frame Detection
We formulate frame detection as a multilabel classification problem for each of the three typologies, using our dataset to train supervised models.
Experimental Setup Our proposed model is a RoBERTa model (Liu et al., 2019b) trained using binary cross-entropy on the CLS token. We consider both (i) a model trained using the roberta-base parameters and (ii) a second model that has first been fine-tuned on our full set of immigration tweets using masked-language modeling. Fine tuning was performed for 60 epochs. In both models, early stopping is used to avoid overfitting. Models are compared with two baselines: random prediction, and logistic regression with unigram and bigram features. Each model was trained five times with different random seeds and we report bootstrapped mean performance.
Results The fine-tuned RoBERTa model significantly outperforms all baselines (Table 2). RoBERTa has the most substantial gains over logistic regression for low-frequency frames (Supplementary Material §C, Figure 8). These gains for rare frames are essential for analyzing immigra
tion discourse on social media in order to capture diverse perspectives and arguments.
Table 3 shows several evaluation metrics separated by frame type. Precision, recall, and F1 are calculated as unweighted averages over all frames belonging to each category. Overall, issue-generic policy and narrative frames can be detected more effectively than issue-specific frames. This difference reflects that issue-specific frames were sparser in the training data, but also that detecting these frames is inherently more challenging because it requires jointly reasoning about immigration-related topics and how these topics affect immigrants. For example, tweets about immigrants committing crimes and tweets about hate crimes committed against immigrants have distinct issue-specific frames (threat: public order and victim: discrimination), even though these texts can be linguistically quite similar. Given some thematic similarities between typologies, we tested an additional model that jointly predicted frames from all three typologies using the fine-tuned RoBERTa model; however, the resulting model offered worse performance than any single-typology model, suggesting minimal benefits of cross-typology learning. Supplementary Section C contains additional model performance analyses by frame and region.
Figure 1: Average F1 scores on combined dev/test set separated by US authors' ideologies. Issue-generic policy and issue-specific frame detection models have higher performance for conservatives than liberals.
Hegemonic Framing Conservative media's framing of political issues is known to be more consistent, coordinated, and hegemonic than mainstream media, which has been vital to the success of the American conservative movement (Hemmer, 2016; Speakman and Funk, 2020). If the same pattern holds for social media, we would expect automated