SlowGuess/ACL_2020_OCR huggingface dataset:
DE	AR	TR
Model	F1-Macro	F1-Micro	EM-1	EM-2	EM-A	F1-Macro	F1-Micro	EM-1	EM-2	EM-A	F1-Macro	F1-Micro	EM-1	EM-2	EM-A
Zero-shot
(2.1) Train EN, Test TL	0.48	0.66	0.47	0.31	0.39	0.37	0.39	0.38	0.04	0.24	0.50	0.77	0.76	0.29	0.53
(2.2) Train \( CS_{TL}(EN) \), Test TL	0.53	0.72	0.64	0.39	0.52	0.42	0.46	0.39	0.06	0.26	0.57	0.82	0.86	0.39	0.63
Few-shot (40 TL samples)
(2.3) Train EN, Test TL	0.66	0.75	0.52	0.37	0.44	0.48	0.54	0.41	0.17	0.31	0.77	0.89	0.67	0.73	0.70
(2.4) Train \( CS_{TL}(EN) \), Test TL	0.64	0.76	0.59	0.43	0.51	0.53	0.58	0.35	0.19	0.29	0.84	0.92	0.80	0.73	0.77
Table 2: Comparison of pure-English training and code-switched training in zero-shot and few-shot settings.  $CS$ : Code-Switched.  $EN$ : English.  $TL$ : Target Language (DE, AR, or TR).  $CS_{Y}(X)$ : Code-switch  $X$  with  $Y$ . Underlying models are MultiBERT with ML Focal loss.
Setup	DE	AR	TR
F1-Macro	F1-Micro	EM-1	EM-2	EM-A	F1-Macro	F1-Micro	EM-1	EM-2	EM-A	F1-Macro	F1-Micro	EM-1	EM-2	EM-A
Train: EN → TL. Test: TL
(3.1) MultiBERT	0.59	0.72	0.67	0.33	0.50	0.45	0.49	0.36	0.11	0.26	0.69	0.88	0.82	0.65	0.74
Train: EN. Test: TL → EN
(3.2) MultiBERT	0.65	0.75	0.72	0.42	0.58	0.50	0.54	0.42	0.10	0.29	0.59	0.84	0.71	0.57	0.64
(3.3) EngBERT Uncased	0.63	0.78	0.75	0.44	0.60	0.52	0.55	0.48	0.13	0.34	0.48	0.78	0.73	0.43	0.58
(3.4) EngBERT CaseD	0.53	0.75	0.74	0.41	0.58	0.51	0.54	0.46	0.11	0.32	0.54	0.86	0.75	0.63	0.69
(3.5) Few-shot w/ the best among (3.2), (3.3), (3.4)	0.61	0.79	0.62	0.50	0.56	0.62	0.66	0.48	0.29	0.40	0.70	0.84	0.63	0.57	0.60
Table 3: Exploring the effect of translation between target languages and English (the source) in both directions. We use Google Translate for translation.  $X \rightarrow Y$ :  $X$  translated to  $Y$  using Google Translate.
As input to our models, we follow previous work and rely on news headlines rather than news story content, due to reasons described by Liu et al. (2019). To showcase the gains made on top of a multi-class approach by reformulating the problem as multi-label, we reproduce the method described by Liu et al. (2019) with both English BERT and MultiBERTs (Table 1). In our implementations involving BERT, we use Adam optimizer with a learning rate of 0.02, a maximum sequence length of 128, and we train for ten epochs.
In Table 1, we include experiments that use different configurations of BERT, such as uncased English BERT (EngBERT) and cased Multilingual BERT (MultiBERT) with two different loss functions. Casing decisions were based on previous work (Liu et al., 2019) and recommendations in BERT code repository<sup>2</sup>. As for losses, we experimented with Binary Cross-Entropy and multi-label Focal Loss, as described in Section 4.1.
For evaluation, we follow recent work and report macro and micro-averaged F1-scores (Wu et al., 2019), as well as exact-match (EM) for samples which have single frames (EM-1), two frames (EM-2) and any number of frames (EM-A). In Table 1, we also report Top-2 accuracy, which, for a given sample, computes the top two most confident predictions for each model based on the scores for each frame after the last activation layer, and checks whether those comprise the first frame. We report this metric to demonstrate that by switching from a multi-class model to a multi-label one, we retain
accuracy for the first frame while providing more predictive power with multiple labels.
Note that, to accommodate multiple languages, we favor a multilingual language model. Results in Table 1 show that for our application, there is only an insignificant drop in the predictive power from EngBERT to MultiBERT using multi-label Focal Loss (ML Focal). Moreover, Focal Loss results in higher accuracy in EM-2 while maintaining as high  $F1$ -scores to canonical BCE Loss. Considering the purposes of this paper, as well as the label cardinalities in other language datasets, we favor ML Focal loss for multilingual models.
While being a state-of-the-art machine translation tool, Google Translate is the practitioner's handy translation guide, (Edunov et al., 2018). In Table 3, we explore the effect of the direction of translation to detect frames in German  $(DE)$ , Arabic  $(AR)$  and Turkish  $(TR)$  headlines about US gun violence. Note that in none of the languages is a sufficient size of news framing training data available; thus, to extend framing analysis to multiple languages, cross-lingual transfer learning is needed.
Firstly, we translate GVFC from English, to target language  $TL \in \{DE, AR, TR\}$ , train MultiBERT with ML Focal loss and test on the  $TL$ . Secondly, we use the English training set as is and translate target test sets to English. This latter setup lets us use EngBERT as well. We experiment with both cased and uncased models and observe that uncased performs better in  $DE$  and  $AR$ . Overall, we note that translating test sets to English results in better performance, which is intuitive as the model requires clarity in the language during training. All
8619
models in Tables 3 and 2 use the same loss, and MultiBERT experiments always use the cased version, following the authors' recommendation.
We use 40 target samples of target language, translated to English, and include them in the training set to study few-shot performance. We only train the best performers, primarily based on F1 scores, among (3.2), (3.3) and (3.4), namely the models (3.3), (3.3) and (3.2) for  $DE$ ,  $AR$  and  $TR$ , respectively in (Table 3). For some of the metrics the few-shot performance may drop because the new samples come from a different distribution.
Furthermore, we compare zero-shot and few-shot performances of MultiBERT when trained on original English versus code-switched train sets in Table 2. Both models use the same set of samples; the difference is that in the former, the headlines are in English, whereas in the latter, "important" words are switched with their  $TL$  translations. In a zero-shot setting, code-switched training (2.2) outperforms English training (2.1) significantly for all three languages (F1-macro and F1-micro scores). Considering the few-shot setting, although the improvement gets smaller, the performance of code-switching is on par if not better for all three languages, see (2.3, 2.4). Note that the comparisons we make are primarily based on F1-scores as the model's capability might shift from predicting single-label cases correctly to predicting more multi-labeled cases correctly as well as between common and rare classes. In German, for instance, code-switched few-shot training improves in  $F1$ -scores from zero-shot but remains around the same in terms of  $EM-A$ . The reason for that is because the model predicts multi-label cases ( $EM-2$ ) better by 4 percent points, see (2.2), (2.4) in Table 2.
Notably, considering Tables 2 and 3 together, a simple word-to-word translation for as little as 358 words, improves frame detection performance drastically even to the level of a complete translation of the test set to English. For Turkish, code-switched training beats full translation of the test set into English in a few-shot setting; it results in a comparable performance for German and slightly worse predictions for Arabic. We attribute the overall low performance for Arabic to the relatively small ICR in the annotation process.
To visualize our multi-label model we use the visualization tool by Vig (2019) in Figure 1. In BERT,
(a) Multi-class Model
(b) Multi-label Model
Figure 1: "Wells Fargo gives gun maker a new line of credit, unwayed by nuns' opposition" has Economic Consequences as the first frame and Public Opinion as the second.
every sequence is padded by a special classification token [CLS] from the beginning. Embedding generated for this token is used for classification into 9 classes. Figures 1a and 1b demonstrate the attentions of this token to other tokens in the sequence. Note that the given sample headline has indeed two frames i.e. "Economic Consequences" as the first and "Public Opinion" as the second. However, in a multiclass setup in which the model is configured to produce a single label, it learns to disregard the second frame "Public Opinion" while strongly attending the words "fargo" and "credit" related to the theme of "Economic Consequences". On the contrary, a multi-label model correctly attends all words that are related to both frames i.e. "fargo", "credit", "nuns" and "opposition" and predicts "Economic Consequences" and "Public Opinion" correctly.
Another interesting observation is related to bias induced by translation. In German, the phrase "scharferes Waffenrecht" means "stricter gun regulation". However, Google Translate translates half of the headlines that include the expression as "stricter/sharper gun rights" which makes the model predict "Gun Rights" rather than "Gun Control" as the frame. A discrepancy like this is widely deceptive and jeopardizes the learning, whether it happens in the training or validation set. However, in code-switched training, one has better control over the translation, as one only translates a manageable number of words. We observe that code-switched training escapes this bias through correctly translated keywords "gun" and "laws" to German. Additionally, we find our models catching several annotation errors such as the headline in Turkish
Code-switch Technique	Unique Switched Words	Total Switched Words	F1-Macro	F1-Micro	EM-1	EM-2	EM-A
Zero-Shot (Train EN, Test DE)	0	0	0.48	0.66	0.47	0.31	0.39
Code-switch Omitted Words	387	2121	0.54	0.70	0.53	0.27	0.40
Code-switch nPMI Words	358	7522	0.53	0.72	0.64	0.39	0.52
Code-switch nPMI + Omitted Words	675	8129	0.60	0.70	0.65	0.29	0.47
Table 4: Code-switch analysis for German.
"Obama'dan LGBTI bireylerin gittigi bir kulüpte 49 kişiyi öldüren Orlando saldirrganı hakkinda Türkiye" which translates as "Obama gave a statement about the Orlando shooter who killed 49 in an LGBTI club." is annotated as "Politics". In contrast, the model predicts "Society/Culture" and "Politics", attending to "LGBTI" and "club".
6.1 Code-switching Analysis
In determining the words to code-switch from English to a target language, we mainly considered the metric called nPMI (Section 4.2), which essentially gives the most frequently-used words for each frame. In the English dataset (GVFC), we first list the top 250 words for a given frame based on their nPMI scores and take the union of these across frames, which resulted in a total of 358 case-sensitive words to be dictionary-translated into the target language.
In Table 4, we provide results obtained by using different code-switching methods that use no target language annotations. Note that, since nPMI is a frequency metric, code-switching with nPMI results in this set of words that includes not only frame-indicative words but also a lot of stop words and common words such as "a", "the", "he" or "are". An alternative method, which we called "omitted words" suggests determining important words by omitting a word from the headline and reapplying the trained classifier to the headline with the missing word (similar to Zhong et al. (2019); Ribeiro et al. (2016)). We then compute the drop in the probability as an importance measure for word  $x_{j}$ ,  $Importance(x_{j}) = p(y|x_{1},\ldots ,x_{n}) - p(y|x_{1},\ldots ,x_{j - 1},x_{j + 1},\ldots ,x_{n})$  where  $y$  is a true label. The remaining procedure is similar to nPMI, as we determine the set of important words per frame, 45 of them this time, and combine those which resulted in 387 words. Note that this method results in a set of important words that are more disjointed across frames, which in turn makes the words more frame-specific. No common or stop words made it to the top 45 in any of the frames.
Despite resulting in more sophisticated words, using omitted words to code-switch resulted in
more deficient if not on par scores as compared to nPMI – our primary way of doing code-switching. We argue that the reason for nPMI performing better is the much higher number of total words that get translated to the target language. In Table 4, note that using dictionary translations for only 358 unique words results in a total of 7522 words that are in the target language, which is more than 3.5 times what omitted words method yields. The increased amount of words that end up in the target language helped the MultiBERT classifier distinguish frames in the target language better. Note that in the last line of Table 4, including translations for the omitted words results in inconsistent improvement due to negligible size in the increase of the total words that get translated.
Our experiments show that for code-switching purposes, quantity might override quality which may suggest that for code-switching to be effective in multilingual transfer, translations of simpler words can outperform translations of the domain-and task-specific words, making the resources required to leverage knowledge from the source language to target language even more parsimonious.
6.2 Framing Network Analysis
The network visualization software Netdraw (Borgatti, 2002) was used to visualize the two frame networks depicted in Figure 6.2 based on the predictions generated on U.S. and German news articles from the year 2016 to 2018 by best performing models, i.e., uncased English BERT (Table 1) and code-switched model (Table 2) for English and German respectively. While each node represents a frame, each edge represents the number of times the two corresponding frames co-occurred in the news headline. The more central, the more connected the frame is with other frames. The node size was adjusted to reflect the relative frequency of news coverage of the given frame. That is, a frame with a larger node size more frequently occurs in the news coverage.
Several notable patterns emerge by comparing the frame networks in the U.S. and Germany. It appears that the U.S. media highly politicized the
8621
(a) U.S. Frame Network
Figure 2: Comparison of frame association networks in the U.S. and German news.
(b) German Frame Network
gun violence issue. The frame “politics” is not only the most salient but also the most central, closely connected with several other frames, reflecting, the sensationalism of the U.S. media landscape. The U.S. media tends to link all aspects of social reality to the political fight between the two parties, a pattern not followed in foreign media.
Another important finding is that while the U.S. media broadly framed the gun violence issue from the perspective of mental health, German media rarely mentions this aspect. Rather than blaming individual shooters, the German press paid more attention to U.S. public opinion manifesting as gun violence protests and the U.S. gun regulations. In other words, compared to the U.S.'s news coverage, foreign media tended to attribute the responsibility to the U.S. government.
In the German news coverage, the close association between the frame "society and culture" and "gun rights" is also noteworthy. Frequently linking the U.S.'s unique culture and people's rights to purchase guns in the news presents the U.S. as a "bizarre" place, which may also lead to a negative perception of the country among Germans.
In conclusion, the two frame networks illustrate how an issue can be framed differently in news media of different countries. Considering that the U.S. and Germany are close allies, it would be exciting to examine how countries with tense relations with the U.S. framed gun violence issues. A large-scale comparative framing study would allow a better understanding of the U.S. global image, which we propose as future work, and our multilingual and multi-label tool would make this type of analysis possible. In general, our approach is practical in looking at how media in different countries frame an international issue.
We want to acknowledge two additional properties of a given headline, which neither this nor the pre
vious works in news framing consider (Card et al., 2015; Liu et al., 2019; Field et al., 2018). First is relevance, although rarely, not all headlines that include the specified keywords in Section 3 are actually about U.S. gun violence. Second, an article may be about one particular incident or event related to gun violence, i.e., episodic, or it may focus on the issue of gun violence as an ongoing problem, i.e., thematic. Moreover, some of the episodic articles may not be tendential enough to have a particular frame. Existing works on framing only includes headlines that are both relevant and have frames, whereas, in reality,  $48\%$  of headlines about U.S. gun violence in GVFC do not have a particular frame. Media outlets outside of the U.S. have various rates of tendential articles about gun violence in the U.S. For instance, among the foreign languages we examined, German articles have the highest rate, with  $90\%$  of articles having at least one frame. Among Turkish articles that are "relevant" only  $10\%$  have a frame. In our evaluations, we only considered headlines that are relevant and have at least one frame. While stressing that determining the frame of an article is the most nuanced task in news framing, addressing the challenges mentioned above is still meaningful and constitutes future work.
In this work, we present a novel code-switch model for the task of automatic cross-lingual news frame detection and show that it matches the performance of full translation if not overrides. Moreover, we leverage an existing dataset by making use of multiple labels, create benchmark news framing test sets for three new languages, and employ a variant of Focal Loss to account for class imbalance in the data. In conclusion, while accounting for multiple frames per sample, we demonstrate how a cross-lingual analysis of news framing is informative and insightful in developing a global view surrounding the gun violence problem in the U.S.
Acknowledgment This work is supported in part by the U.S. NSF grant 1838193 and DARPA HR001118S0044 (the LwLL program). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA and the U.S. Government.
8622
Claire Atkinson. 2019. Americans are crazy: Foreign journalists grapple with covering u.s. mass shootings.
Stephen Borgatti. 2002. Netdraw: Graph visualization software. Harvard: Analytic Technologies.
Alexander Buhmann and Diana Ingenhoff. 2015. The 4d model of the country image: An integrative approach from the perspective of communication management. International Communication Gazette, 77(1):102-124.
Dallas Card, Amber Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 438-444.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087.
Ruth DeFoster and Natasha Swalve. 2018. Guns, culture or mental health? framing mass shootings as a public health crisis. Health communication, 33(10):1211-1222.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381.
Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51-58.
Robert M Entman. 2008. Theorizing mediated public diplomacy: The us case. The International Journal of Press/Politics, 13(2):87-102.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in russian news: a computational analysis of intricate political strategies. arXiv preprint arXiv:1808.09386.
Guy J Golan and Itai Himelboim. 2016. Can world system theory predict news flow on twitter? the case of government-sponsored broadcasting. Information, Communication & Society, 19(8):1150-1170.
Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K. Li. 2018. Universal neural machine translation for extremely low resource languages. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 344-354, New Orleans, Louisiana. Association for Computational Linguistics.
Lei Guo and Maxwell McCombs. 2015. The power of information networks: New directions for agenda setting. Routledge, New York and London.
Holli A. Semetko Claes H. De Vreese, Jochen Peter. 2001. Framing politics at the launch of the euro: A cross-national comparative study of frames in the news. Political communication, 18(2):107-122.
Crimson Hexagon. 2018. ForSight social media analytics platform, Last accessed on November 1, 2018.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019. Detecting frames in news headlines and its application to analyzing news framing trends surrounding us gun violence. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 504-514.
Andrew McCallum, Kamal Nigam, et al. 1998. A comparison of event models for naive bayes text classification. In AAAI-98 workshop on learning for text categorization, volume 752, pages 41-48. CiteSeer.
Joseph S Nye Jr. 2004. Soft power: The means to success in world politics. Public affairs.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assetss/researchcovers/languageeunsupervised/language understanding paper. pdf.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
Friederike Schultz, Jan Kleinnijenhuis, Dirk Oegema, Sonja Utz, and Wouter Van Atteveldt. 2012. Strategic framing in the bp crisis: A semantic network analysis of associative frames. *Public Relations Review*, 38(1):97-107.
Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. Journalism Bulletin, 30(4):415-433.
8623