SlowGuess/NAACL_2021_OCR huggingface dataset:
Error Type	Description	Example
Plausible interpretation	These instances highlight the challenges of annotation; there are convincing arguments that model's predicted frames can be appropriate labels.	Interestingly, the criteria to which immigrants would be held would not be met by a large number of the ‘British’ people either. Model erroneously predicted Policy
Inferring frames not explicitly cued in text	Model predicts frames that may capture an author's intention but without sufficient evidence from the text	Stop immigration
Model erroneously predicted Threat: Public Order
Missing necessary contextual knowledge	Some frames are directly cued by lexical items (e.g. politicians' names cue Political frame), but model lacks real-world knowledge required to identify these frames	@EricTrump Eric I have been alive longer than your immigrant mother in law and you. I paid more in taxes than you did and your immigrant mother in law combined...
Model missed Political frame
Overgeneralizing highly-correlated features	Many words and phrases do not directly cue frames, but are highly-correlated. The model makes erroneous predictions when such features are used in different contexts (e.g. violence against immigrants, rather than immigrants being violent)	Lunaria's figures from 2018 recorded 12 shootings, two murders and 33 physical assaults against migrants in the first two months since Salvini entered government.
Model missed Victim: Humanitarian frame
Pronoun ambiguity	Coreference resolution is often not possible and annotators avoided making assumptions to resolve ambiguities. For example, “you” can be used to discuss individuals' experiences (episodic) but its impersonal sense can be in broad generalizations (thematic).	It's worse when you have immigrant parents who don’t speak the language cause you have to deal with all the paperwork, be the translator for them whenever they go (...)
its tiring but someone has to
Model predicted Episodic but referent is unclear
Table 4: Types of common errors in frame prediction along with brief descriptions and examples.
frame detection to achieve higher performance on conservative tweets due to more linguistic regularities across messages. Indeed, we find that issue-generic and issue-specific classifiers achieve higher F1 scores on tweets written by conservative authors compared to liberal authors (Figure 1), even though there are fewer conservative tweets in the training data (334 conservative vs 385 liberal tweets). Higher model performance on conservative tweets suggests that, like political and media elites, conservatives on social media are more consistent than liberals in their linguistic framing of immigration. Error Analysis We identify classification errors by qualitatively analyzing a random sample of 200 tweets that misclassified at least one frame. Table 4 shows the most common categories of errors.
6 Frame-Building Analysis
In writing about an issue, individuals are known to select particular frames—a process known as frame-building—based on numerous factors, such as exposure to politicians' rhetoric or their own identity (Scheufele, 1999). Here, we focus on two specific identity attributes affecting frame building: (i) political ideology and (ii) country/region.
The political, social, and historical contexts of an one's nation-state can impact how they frame immigration (Helbling, 2014). Immigration has a long history in the USA relative to Europe, and former European colonial powers (e.g. the UK) have longer immigration histories than other countries (e.g. Norway) (Thorbjørnsrud, 2015; Eberl et al., 2018). Cross-country variation in news framing also arises from differences in immigration policies (Helbling, 2014; Lawlor, 2015), media systems (Thorbjørnsrud, 2015), journals
tic norms (Papacharissi and De Fatima Oliveira, 2008), geographic proximity to immigrant populations or points of entry (Grimm and Andsager, 2011; Fryberg et al., 2012), and immigrants' race/ethnicity (Grimm and Andsager, 2011). At the same time, increased globalization may result in a uniform transnational immigration discourse (Helbling, 2014). Framing variations across countries has implications for government policies and initiatives, particularly in determining what solutions could be applied internationally or tailored to each country (Caviedes, 2015).
Prior studies on the role of ideology in framebuilding have focused on the newspapers or political movements, showing patterns in frames like morality and security by political affiliation in European immigration discourse (Helbling, 2014; Hogan and Haltinner, 2015) or in use of economic frames by American newspapers (Fryberg et al., 2012; Abrajano et al., 2017). However, it remains unclear whether these patterns observed for elite groups can generalize to the effect of individual people's political dispositions.
Experimental Setup We detect frames for all 2.6M immigration-related tweets using the finetuned RoBERTa model with the best-performing seed on development data. Using this labeled data, we estimate the effects of region and ideology by fitting separate mixed-effects logistic regression models to predict the presence or absence of each frame. We treat region (US, UK, and EU) as a categorical variable, with US as the reference level. Ideology is estimated using the method of Barberá (2015), which is based on users' connections to US political elites; as such, we restrict our analysis of ideology to only tweets from the United States.
Figure 2: Logistic regression coefficients of political ideology in predicting each frame. Positive (negative) values correspond to more conservative (liberal) ideology. Only frames associated with ideology after Holm-Bonferroni correction with  $p < 0.01$  are included.
To account for exogenous events that may impact framing, we include nested random effects for year, month, and date. We further control for user characteristics (e.g. the author's follower count, friends count, verified status and number of prior tweets) as well as other tweet characteristics (e.g. tweet length, if a tweet is a reply, and whether the tweet contains hashtags, URLs, or mentions of other users). We apply Holm-Bonferroni corrections on p-values before significance testing to account for multiple hypothesis testing.
Ideology Ideology is strongly predictive of framing strategies in all three categories, as shown in Figure 2. Our results reveal three broad themes.
First, prior work has argued that liberals and conservatives adhere to different moral foundations, with conservatives being more sensitive to in-group/loyalty and authority than liberals, who are more sensitive to care and fairness (Graham et al., 2009). Our results agree with this argument. Liberals are more likely to frame immigration as a fairness and morality issue, and immigrants as victims of discrimination and inhumane policies. More conservative authors, on the other hand, fo
cus on frames with implications for the in-group. They express concerns about 1) immigrants imposing a burden on taxpayers and governmental programs and 2) immigrants being criminals and threats to public safety. We qualitatively observe three distinct, though unsubstantiated, conservative claims contributing to the latter: (i.) Immigrants commit violent crimes (Light and Miller, 2018), (ii.) Undocumented immigrants illegally vote in US elections (Smith, 2017; Udani and Kimball, 2018), and (iii.) Immigrants are criminals simply by virtue of being immigrants (Ewing et al., 2015).
Figure 2 shows a clear ideological stratification for issue-specific frames: liberals favor hero and victim frames, while conservatives favor threat frames. This finding is consistent with prior work on the role perceived threats play in shaping white American attitudes towards immigration (Brader et al., 2008), and the disposition of political conservatism to avoid potential threats (Jost et al., 2003).
Second, while all frame categories show ideological bias, issue-specific frames are the most extreme. Most notably, our analysis shows that focusing solely on issue-generic policy frames would obscure important patterns. For example, the issue-generic cultural identity frame shows a slight liberal bias; yet, related issue-specific frames diverge: hero: cultural diversity is very liberal while threat: national cohesion is very conservative.
Similarly, the issue-generic economic policy frame is slightly favored by more conservative authors, but the related issue-specific frames threat: jobs and hero: worker reveal ideological divides. This finding highlights the importance of using multiple framing typologies to provide a more nuanced analysis of immigration discourse.
Third, more liberal authors tend to use episodic frames, while conservative authors tend to use thematic frames. This difference is consistent with Somaini (2019)'s finding that a local liberal newspaper featured more episodic framing in immigration coverage, but a comparable conservative newspaper featured more thematic framing. Other efforts that examine the relationship between narrative frames and cognitive and emotional responses provide some clues for the observed pattern. For instance, Aarøe (2011) shows that thematic frames are stronger when there are no or weak emotional responses; and that the opposite is true for episodic frames. The divergence of findings could be driven by partisans' differing emotional responses. Our
findings also highlight important consequences for opinion formation. Iyengar (1990) shows that episodic framing diverts attention from societal and political party responsibility; our results suggest that liberal Twitter users are likely to produce (and, due to partisan self-segregation, consume) social media content with such effects.
Figure 3: Effect of author being from the EU (top) or the UK (bottom) relative to the US. Frames with positive  $\beta$  coefficients are associated with authors from the EU (top) and UK (bottom), and frames with negative values are associated with US-based authors. Frames not significantly associated with region after Holm-Bonferroni correction are not included.
Region Immigration framing depends heavily on one's geopolitical entity (US, UK, and EU), as shown in Figure 3. Several notable themes emerge. First, many ideologically-extreme frames in the US, including crime & punishment, security & defense, threat: public order, and threat: fiscal are all significantly more likely to be found in US-based tweets relative to the UK and EU. This pattern suggests that region and ideology, and likely many
other factors, interact in intricate ways to shape how ordinary people frame political issues.
Second, cultural identity is more strongly associated with both the UK and EU than the US. Perhaps immigrants' backgrounds are more marked in European discourse than in US discourse because the UK and EU have longer histories of cultural and ethnic homogeneity (Thorbjørnsrud, 2015). This finding also reflects that Europeans' attitudes towards immigration depend on where immigrants are from and parallels how European newspapers frame immigration differently depending on migrants' countries of origin (Eberl et al., 2018).
Finally, the bottom of Figure 3 shows that users from the UK are more likely to invoke labor-related frames. This prevalence of labor and economic frames has also been found in British traditional media (Caviedes, 2015; Lawlor, 2015), and has been attributed to differences in the labor market. Unlike migrants in the US, Italy, and France, who often work clandestinely in different economic sectors than domestic workers, UK migrants have proper authorization and are thus viewed as competition for British workers because they can work in the same industries (Caviedes, 2015).
7 Audience Response to Frames
Chong and Druckman (2007, p. 116) assert that a "challenge for future work concerns the identification of factors that make a frame strong." Studies of frame-setting—i.e., how a message's framing affects its audience's emotions, beliefs, and opinions—have largely been restricted to small-scale experimental studies because responses to news media framing cannot be directly observed (Eberl et al., 2018). However, Twitter provides insight into the frame-setting process via interactive signals: favorites and retweets. While related, these two actions can have distinct underlying motivations: favoriting often indicates positive alignment between the author and the reader; in contrast, retweeting may also be driven by other motivations, such as the desire to inform or entertain others (boyd et al., 2010). Different audience interactions have been shown to exhibit distinct patterns in political communication on Twitter (Minot et al., 2020). Here, we test how a message's framing impacts both the favorites and retweets that it receives.
Experimental Setup We fit hierarchical linear mixed effects models with favorites and retweets (log-transformed) as the dependent variable on US
Figure 4: Effects of framing on two audience responses: favorites and retweets. The x-axis shows regression coefficients for the presence of each frame in predicting the log-scaled number of responses. Along the y-axis are all issue-generic policy frames (top), immigration-specific frames (middle), and narrative frames (bottom) that are significantly associated with either the number of favorites or retweets.
tweets with detected author ideology. The presence of a frame is treated as a binary fixed effect. We control for all temporal, user-level and tweet-level features as in the prior section, as well as ideology.
Results The framing of immigration has a significant impact on how users engage with the content via retweets and favorites (Figure 4). Many issue-specific frames have a stronger effect on audience responses than either of the other typologies. As recent NLP approaches have adopted issue-generic frames for analysis (e.g., Kwak et al., 2020), the strength of issue-specific frames highlights the importance of expanding computational analyses beyond issue-generic frames, as other frames may have larger consequences for public opinion.
Most frames impact favorites and retweets differently, suggesting that the strength of a frame's effects is tied to the specific engagement behavior. Cultural frames (e.g. hero: integration) and frames oriented around human interest (e.g. morality, victim: discrimination) are particularly associated with more endorsements (favorites), perhaps
due to their increased emotional appeal to readers (Semetko and Valkenburg, 2000).
On the other hand, political factors & implications is most highly associated with increased retweets. As the political frame emphasizes competition and strategy (Boydstun et al., 2013), this result mirrors similar links between the "horse-race" frame in news reports and engagement (Iyengar et al., 2004); users may prefer amplifying political messages via retweeting to help their side win.
Similarly, frames about security and safety (e.g. crime & punishment, victim: humanitarian) are highly associated with more retweets, but not necessarily favorites. While security and safety frames may not lead audience members to endorse such messages, perhaps they are more likely to amplify these messages due to perceived urgency or the desire to persuade others of such concerns.
Finally, Figure 4 shows how a message's narrative framing impacts audience response, even after controlling for all other frames. Both episodic and thematic frames are significantly associated with increased engagement (retweets), but less strongly than issue frames. Having a clear narrative is important for messages to spread, but the underlying mechanisms driving engagement behaviors may differ for episodic and thematic frames; prior work on mainstream media has found that news stories using episodic frames tend to be more emotionally engaging, while thematic frames can be more persuasive (Iyengar, 1991; Gross, 2008).
Users' exposure to political information on social media can have immense consequences. By leveraging multiple theory-informed typologies, our computational analysis of framing enables us to better understand public discourses surrounding immigration. We furthermore show that framing on Twitter affects how audience interactions with messages via favoriting and retweeting behaviors. This work has implications for social media platforms, who may wish to improve users' experiences by enabling them to discover content with a diversity of frames. By exposing users to a wide range of perspectives, this work can help lay foundations for more cooperative and effective online discussions. All code, data, annotation guidelines, and pretrained models are available at https:
//github.com/juliamendelsohn/framing.
Our analysis of frame-building involves inferring political ideology and regional from users with existing tools, so we aggregated this information in our analysis in order to minimize the risk of exposing potentially sensitive personal data about individuals. Our dataset includes tweet IDs along with frame labels, but no additional social information. However, there are also ethical consequences of categorizing people along these social dimensions. We acknowledge that reducing people's social identities to region and ideology obscures the wide range of unobservable and non-quantifiable predispositions and experiences that may impact framing and attitudes towards immigration.
We emphasize that our dataset is not fully representative of all immigration discourse and should not be treated as such. Twitter's demographics are not representative of the global population (Mislove et al., 2011). Furthermore, our dataset only includes tweets with authors from particular Western countries. All tweets were automatically identified by Twitter as being written in English, thus additionally imposing standard language ideologies on the data that we include (Milroy, 2001). Furthermore, language choice itself can be a socially and politically meaningful linguistic cue that may have unique interactions with framing (e.g., Gal, 1978; Shoemark et al., 2017; Stewart et al., 2018; Ndubuisi-Obi et al., 2019).
Although we do not focus on abusive language, our topical content contains frequent instances of racism, Islamophobia, antisemitism, and personal insults. We caution future researchers about potentially traumatic psychological effects of working with this dataset.
We aim to support immigrants, an often marginalized group, by shedding light on their representation on social media. However, there is a risk that malicious agents could exploit our framesetting findings by disseminating harmful content packaged in more popular frames.
We thank Anoop Kotha, Shiqi Sheng, Guoxin Yin, and Hongting Zhu for their contributions to the data annotation effort. We also thank Libby Hemphill and Stuart Soroka for their valuable comments and feedback. This work was supported in part through funding from the Volkswagen Foundation.
Lene Aarøe. 2011. Investigating Frame Strength: The Case of Episodic and Thematic Frames. Political Communication, 28(2):207-226.
Marisa A Abrajano, Zoltan Hajnal, and Hans J.G. Hassell. 2017. Media Framing and Partisan Identity: The Case of Immigration Coverage and White Macropartisanship.
Pablo Barbera. 2015. Birds of the same feather tweet together: Bayesian ideal point estimation using Twitter data. Political Analysis, 23(1):76-91.
Pablo Barbera, Amber E Boydstun, Suzanne Linn, Ryan McMahon, and Jonathan Nagler. 2021. Automated text classification of news articles: A practical guide. Political Analysis, 29(1):19-42.
Rodney Benson. 2013. Shaping Immigration News: A French-American Comparison. Cambridge University Press.
danah boyd, Scott Golder, and Gilad Lotan. 2010. Tweet, tweet, retweet: Conversational aspects of retweeting on Twitter. In 2010 43rd Hawaii International Conference on System Sciences, pages 1-10. IEEE.
Amber E. Boydstun, Justin H Gross, Philip Resnik, and Noah A. Smith. 2013. Identifying Media Frames and Frame Dynamics Within and Across Policy Issues. New Directions in Analyzing Text as Data Workshop, pages 1-13.
Ted Brader, Nicholas A. Valentino, and Elizabeth Suhay. 2008. What triggers public opposition to immigration? Anxiety, group cues, and immigration threat. American Journal of Political Science, 52(4):959-978.
Dallas Card, Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, volume 2, pages 438-444.
Dallas Card, Justin H Gross, Amber E Boydstun, and Noah A Smith. 2016. Analyzing framing through the casts of characters in the news. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1410-1420.
Alexander Caviedes. 2015. An Emerging 'European' News Portrait of Immigration? Journal of Ethnic and Migration Studies, 41(6):897-917.
Dennis Chong and James N. Druckman. 2007. Framing Theory. Annual Review of Political Science, 10(1):103-126.
Ryan Compton, David Jurgens, and David Allen. 2014. Geotagging one hundred million Twitter accounts with total variation minimization. In 2014 IEEE International Conference on Big Data (Big Data).
Constance de Saint Laurent, Vlad Glaveanu, and Claude Chaudet. 2020. Malevolent Creativity and Social Media: Creating Anti-immigration Communities on Twitter. Creativity Research Journal, 32(1):66-80.
Claes H de Vreese. 2005. News framing: Theory and typology. Information Design Journal, 13(1):51-62.
Jakob-Moritz Eberl, Christine E. Meltzer, Tobias Heidenreich, Beatrice Herrero, Nora Theorin, Fabienne Lind, Rosa Berganza, Hajo G. Boomgaarden, Christian Schemer, and Jesper Strömbäck. 2018. The European media discourse on immigration and its effects: a literature review. Annals of the International Communication Association, 42(3):207-223.
Robert M. Entman. 1993. Framing: Toward Clarification of a Fractured Paradigm. Journal of Communication, 43(4):51-58.
Walter A Ewing, Daniel Martinez, and Rubén G Rumbaut. 2015. The criminalization of immigration in the United States. Washington, DC: American Immigration Council Special Report.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in Russian news: A computational analysis of intricate political strategies. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3570-3580.
Stephanie A. Fryberg, Nicole M. Stephens, Rebecca Covarrubias, Hazel Rose Markus, Erin D. Carter, Giselle A. Laiduc, and Ana J. Salido. 2012. How the Media Frames the Immigration Debate: The Critical Role of Location and Politics. *Analyses of Social Issues and Public Policy*, 12(1):96-112.
Susan Gal. 1978. Peasant men can't get wives: Language change and sex roles in a bilingual community. Language in society, 7(1):1-16.
Jesse Graham, Jonathan Haidt, and Brian A Nosek. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029.
Josh Grimm and Julie L. Andsager. 2011. Framing immigration: Geo-ethnic context in california newspapers. *Journalism and Mass Communication Quarterly*, 88(4):771-788.
Kimberly Gross. 2008. Framing persuasive appeals: Episodic and thematic framing, emotional response, and policy opinion. Political Psychology, 29(2):169-192.
Mareike Hartmann, Tallulah Jansen, Isabelle Augenstein, and Anders Søgaard. 2019. Issue framing in online discussion fora. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human