SlowGuess/ACL_2020_OCR huggingface dataset:
Afra Feyza Akyurek<sup>1</sup>, Lei Guo<sup>2,1</sup>, Randa Elanwar<sup>3</sup>, Prakash Ishwar<sup>4,1</sup>, Margrit Betke<sup>1</sup> and Derry T. Wijaya<sup>1</sup>
$^{1}$ Department of Computer Science, Boston University  $^{2}$ College of Communication, Boston University  $^{3}$ Electronics Research Institute, Egypt
$^{4}$ Department of Electrical and Computer Engineering, Boston University {akyurek, guolei, relanwar, pi, betke, wijaya}@bu.edu
News framing refers to the practice in which aspects of specific issues are highlighted in the news to promote a particular interpretation. In NLP, although recent works have studied framing in English news, few have studied how the analysis can be extended to other languages and in a multi-label setting. In this work, we explore multilingual transfer learning to detect multiple frames from just the news headline in a genuinely low-resource context where there are few/no frame annotations in the target language. We propose a novel method that can leverage elementary resources consisting of a dictionary and few annotations to detect frames in the target language. Our method performs comparably or better than translating the entire target language headline to the source language for which we have annotated data. This work opens up an exciting new capability of scaling up frame analysis to many languages, even those without existing translation technologies. Lastly, we apply our method to detect frames on the issue of U.S. gun violence in multiple languages and obtain exciting insights on the relationship between different frames of the same problem across different countries with different languages.
The worldwide image of the United States has dropped precipitously during the past few years (Wike et al., 2018). Among other factors, the increasing number of gun violence incidents appears to affect the U.S. reputation abroad. Whenever a fatal mass shooting happens, it often attracts significant international news attention. While the domestic U.S. news media often links gun violence to individual shooters' mental illness (DeFoster and Swalve, 2018; Liu et al., 2019), foreign media may attribute it to U.S. gun policy and its gun culture e.g., (Atkinson, 2019). This phenomenon is known
as media framing, which is the process of selecting "some aspects of a perceived reality and [making] them more salient in a communicating text, in such a way as to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment recommendation for the item" (Entman, 1993). When foreign media frame the gun violence issue in a way to depict the U.S. as an unsafe and undesired place, it erodes the country's "soft power" (Nye Jr, 2004). Evaluating how different countries frame the U.S. gun violence issue will enrich our understanding of the U.S. soft power in particular and international relations in general. In this work, we develop a multilingual approach to automatically detect frames in news coverage of different languages, thus facilitating the analysis of how different countries with different languages frame a particular issue. Aside from enabling this understanding of foreign public opinion regarding a certain issue or nation, a multilingual approach is essential in media framing analysis, as it is also an understudied problem in many parts of the world.
Given frame-annotated news headlines of a particular topic in a source language (e.g., English), our approach uses word-to-word translation to translate keywords that are indicative of the frames in these headlines to a target language. Then, we fine-tune a state-of-the-art multilingual language model MultiBERT (Devlin et al., 2019) to detect frames on these "code-switched" headlines, combined with a few annotated headlines from the target language. The translated keywords and a few-shot examples act as anchors to adapt MultiBERT to detect frames in the target language. This approach performs comparably if not better than a model trained on the source language and tested on headlines that are translated from the target language to the source. Since our approach requires only simple resources - a dictionary and a few  $(\leq 40)$  annotated examples in the target language
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8614-8624 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
- it is handy for many languages. Moreover, considering the significant improvement gained over the zero-shot transfer, the proposed approach is much more reliable for languages without existing translation technologies or expert annotations.
Due to the subtle nature of framing, it is not uncommon for one news article to involve more than one message. Communication researchers have suggested that the association of different constructs, such as issues and frames in the news, will influence how the audience associate these elements, thus determining how they perceive the world (Guo and McCombs, 2015). The Network Agenda Setting Model suggests that examining the interrelationships between media elements enables researchers to measure media effects in a more nuanced manner. Note that some frames appear more often than others. In this work, we formulate our frame detection model to allow for multi-label frame detection while also addressing the imbalance in the frame distribution by adapting focal loss (Lin et al., 2017) into our multi-label setting. Our multi-label approach allows for the examination of frame co-occurring, or "associative frames" (Schultz et al., 2012), across the news articles. Overall, the contribution of this work are manifold:
(1) We devise a novel code-switch few-shot scheme to train a frame detection model for any language.
(2) We extend the formulation of the frame classification problem and focal loss to a multi-label setting, allowing the model to predict multiple frames for each instance.
(3) We use our multilingual multi-label frame detection model to detect frames in news headlines pertaining to U.S. gun violence issue in multiple countries and languages, and obtain interesting insights on how other countries view the gun violence issue in the U.S. and how frames are related across news articles in different countries with different languages.<sup>1</sup>
Today's international politics not only revolve around military and economic influence but also largely depend on a country's soft power (Nye Jr, 2004). For each nation, constructing a positive
country image to the outside world is crucial to ensure its international competitiveness in this global information society (Buhmann and Ingenhoff, 2015). In this light, more and more governments have realized the importance of public diplomacy, making great efforts to promote their countries' values and perspectives to foreign publics (Entman, 2008; Golan and Himelboim, 2016). However, these efforts are not always successful. Editors of international news media serve as the gatekeepers to decisions which may lead to the framing of a given country contrary to how its government intends. In reporting news about a foreign country, news editors and reporters make conscious or unconscious choices to emphasize specific issues, or emphasize certain aspects of a given topic, which may alter the country's image in the minds of their audience. A multilingual approach is essential to analyze media framing in different parts of the world, which will shed light on foreign public opinion regarding a particular nation.
Communication researchers often rely on manual content analysis to examine media framing in news outlets of different languages (H. De Vreese, 2001). One critique for this type of study is that researchers tend to decide countries for review based on languages spoken in the research team rather than theoretical rationales. This language constraint becomes a more significant challenge in this increasingly globalized media landscape; capturing a holistic picture of international communication would require the analysis of news coverage in a larger number of languages. Arguably, an automatic, multilingual approach of framing analysis would greatly benefit the international communication research community.
In NLP, language models have been effectively fine-tuned or used in downstream tasks such as text classification (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018). Further, the introduction of deep contextual language embedding such ELMO (Peters et al., 2018), which uses bi-directional LSTMs and BERT (Bi-directional Encoder Representations from Transformers) (Devlin et al., 2019), has been another milestone in this line of work. BERT is currently one of the state-of-the-art models in language modeling.
News framing was first brought to the attention of the computational linguistics community by the Media Frames Corpus (Card et al., 2015), which addresses three issues: immigration, tobacco, and
$^{1}$ Code and data are available at https://github.com/feyzaakyurek/newsframing
8615
same-sex marriage. Field et al. (2018) analyzes the framing of the U.S. and agenda-setting in Russian news. Our work is similar to (Field et al., 2018) in terms of using nPMI to find essential words. Furthermore, our work advances previous research by leveraging a multilingual language model, facilitating transfer learning in news framing, and relying on parsimonious resources, that is, 50,000 lexical translations vs.  $\sim 350$  in our case.
The current state-of-the-art model (Liu et al., 2019) for frame detection fine-tunes BERT on frame-annotated English news headlines with the standard multiclass focal loss objective (Lin et al., 2017). Their approach predicts only a single frame, which is insufficient given the multifaceted nature of news framing in which multiple frames often co-occur in the same headline. Indeed, more than a quarter of the Gun Violence Frame Corpus (GVFC) has more than one frame (Liu et al., 2019). In this work, we fine-tune MultiBERT to detect frames in multiple languages' headlines with our multi-label focal loss. Our approach can predict (and be evaluated on) multiple frames for each headline, which is a more complex task while being comparable to their work in terms of the average F1 performance. Similar to their work, we detect frames on news headlines as they provide the most direct clue to the potential influence of the news coverage.
3 Dataset Creation
GVFC is a dataset of news articles from 21 major U.S. news organizations related to U.S. gun violence that contains news headlines and their domain-expert frame annotations (Liu et al., 2019). We extend GVFC to include headlines in other languages by following their process of curating GVFC. We first drew our sample of news articles from German-, Turkish-, and Arabic-speaking news websites, using Crimson Hexagon's ForSight social media analytics platform (Hexagon, 2018), retrieving items that had at least one keyword in their headlines from the following list of words - {"gun", "firearm", "NRA", "2nd amendment", "second amendment", "AR15", "assault weapon", " rifle", "Brady act", "Brady bill", "mass shooting"} - that have been translated into German, Turkish, and Arabic respectively by native speakers of the languages. In curating the multilingual datasets, we used the same set of frames as in GVFC.
We then trained two native speaker coders for each language to apply the GVFC codebook proto
col for identifying frames and then measured their intercoder reliability (ICR) in annotating a sample of 350, 200, and 210 German, Turkish, and Arabic news headlines, respectively. The coders achieve  $92.6\%$ ,  $98.5\%$ ,  $78.1\%$  agreement rates in identifying the first frame and  $78.9\%$ ,  $97.9\%$ ,  $74.3\%$  agreement rates for the second frame for German, Turkish, and Arabic samples. Additionally, Krippendorff's Alpha for the 1st frame and the 2nd frame are 0.89, 0.66; 0.90, 0.74, and 0.69, 0.26 for German, Turkish, and Arabic, respectively.
Once a minimum of  $70\%$  agreement was reached, one coder of each language continued to code more headlines. Annotation resulted in a total of 326, 100, and 388 non-duplicate headlines for German, Turkish, and Arabic. The average number of labels, i.e., label cardinalities, per headline are 1.4, 1.5, and 1.5, for German, Turkish, and Arabic, whereas it's 1.3 in GVFC, which is in English. As we can observe from the agreement rates, the Arabic data has a relatively weaker ICR, while the Turkish data has the best ICR. As high ICR values imply that two coders consistently categorized the content similarly, they signal a high validity of the coded results. In turn, this is reflected in the performance of our model as it performs the worst in Arabic (Section 5). Nonetheless, the quality of our curated data is substantially higher – the average of Krippendorff's alpha is 0.82 – than contemporaries such as MFC (which is only in English) with an average alpha of less than 0.6 (Card et al., 2015).
In this work, we extend the current state-of-the-art model on the GVFC (Liu et al., 2019), which predicts only the first frame, into a multi-label approach and evaluate it across multiple languages. As previous work has showcased that BERT surpasses LSTM and GRU-based architectures, we shift our focus in this work from architecture optimization to scalability of news framing analysis across multiple languages in a multi-label setting.
BERT relies on multiple stacks of the Transformer's encoder blocks (Devlin et al., 2019; Vaswani et al., 2017) to learn vector representations of sentences. A single encoder block is composed of a self-attention layer followed by a fully-connected layer. When a sentence – a sequence of tokens – is fed into the encoder, it passes through an embedding layer, a self-attention layer, and fully-connected layers before being passed to the upper
8616
encoder block. The self-attention layer embodies three matrices called  $W^{Q}$  for the query,  $W^{K}$  for the key, and  $W^{V}$  for the value. Each of these matrices is of size  $vocab\_size \times hidden\_size$ , and thus each token in the vocabulary has its corresponding  $q$ ,  $k$ , and  $v$  vectors. Representations for each token are contextualized; namely, the representation of a token is the weighted average of all representations in the sequence. Therefore, the vector representation for token  $x_{i}$  is given by
$$
\operatorname {v e c} _ {-} \operatorname {r e p} \left(x _ {i}\right) = \sum_ {j \in S} v _ {j} \operatorname {S o f t m a x} \left(q _ {i} \cdot k _ {j} / \sqrt {d}\right)
$$
where  $d$  is the size of the key vectors in  $W^K$
and  $S$  is the set of all tokens in the same sequence as  $x_{i}$ , including  $x_{i}$ .
BERT adds a special token for classification [CLS] at the beginning of each sequence. Then it learns the representation of this token and other tokens in the sequence by training on Wikipedia corpus for two language tasks: next sentence prediction and Masked Language Model (MLM), which was initially inspired by the Cloze task (Taylor, 1953). The contextual representation of the [CLS] token encodes the syntactic and semantic constructs of the sequence, and one can fine-tune BERT for various down-stream tasks.
Fine-tuning BERT performs well on new tasks even with small datasets, which can be attributed to the data-efficient deep attention mechanism (Devlin et al., 2019; Vinyals et al., 2015). The knowledge encoded within the vector representations of the tokens through pre-training also helps the classifier with the language understanding part of the task, reducing the need for a larger dataset.
Finally, a multilingual version of pre-trained BERT, MultiBERT, which is trained on the entire Wikipedia dumps of 104 languages with the largest Wikipedia, has recently been released, making it an excellent candidate for scaling to multiple languages. The multilingual pre-training and the utilization of sub-word tokenization allows MultiBERT to represent sequences from any of these 104 languages (Gu et al., 2018) and enables zero-shot classification on any of the languages (i.e., train on one language and test on another).
In our case, since reproducing the effort put in GVFC, which was created by highly qualified journalism students in other languages, is prohibitive, employing a cross-lingual model such as MultiBERT renders scaling to other words possible.
4.1 Multi-label News Frame Detection
For frame detection purposes, we classify news articles into nine frame categories based on their headlines. Devlin et al. (2019) recommends using the embedding generated for the special token called [CLS], which is padded to the beginning of every sentence. All tokens, including [CLS] are of length  $H = 768$ . The representation for [CLS] is generated by attending every word in the sequence.
We modify BERT by appending to it a fully connected layer which acts as a classifier taking in the embedding generated for [CLS] after 12 layers of encoders and mapping it into  $K = 9$  output neurons. Hence, the only parameters trained from scratch during fine-tuning are those of the classifier layer's,  $W \in \mathbb{R}^{H \times K}$ . Finally, we use Sigmoid activations to obtain nine outputs, each between 0 and 1, which are interpreted as scores for nine classes. During inference, we use the threshold of 0.5 on these scores to binarize the output.
We fine-tune MultiBERT with two different losses: the standard Binary Cross-Entropy loss, and a multi-label variation of the weighted focal loss (Lin et al., 2017). We compute the Binary Cross-Entropy (BCE) loss, also named as Sigmoid Cross-Entropy loss, for a single sample  $\mathbf{x}$  as,
$$
\begin{array}{l} B C E (f) = - \frac {1}{| K |} \sum_ {i = 1} ^ {| K |} \left(y ^ {(i)} \log \left(\hat {y} ^ {(i)}\right) + \right. \\ (1 - y ^ {(i)}) \log (1 - \hat {y} ^ {(i)})) \\ \end{array}
$$
where predictions are given by
$$
\hat {\mathbf {y}} = [ \hat {y} ^ {(1)}, \dots , \hat {y} ^ {(| K |)} ] = \frac {1}{(1 + e x p (- f (\mathbf {x})))}
$$
$\mathbf{y} = [y^{(1)}, \ldots, y^{(|K|)}]$  are the gold binary labels and  $f$  is BERT with classifier.
Considering the high degree of class imbalance in the GVFC dataset, which deteriorates within the multilingual datasets we developed, we adopt a multi-label variation of binary focal loss (Lin et al., 2017). As a reminder, the focal loss for a single sample  $\mathbf{x}$  is defined as,
$$
F L (f) = - \alpha (1 - p) ^ {2} l o g (p)
$$
where  $p = (1 - y)(1 - \hat{y}) + y\hat{y}$  and  $y\in \{0,1\}$  is the true label, also  $\hat{y} = 1 / (1 + \exp (-f(\mathbf{x})))\in \mathbb{R}$  and  $\alpha$  is the balancing factor, which is usually normalized inverse class frequency. Hence, the smaller the class, the higher the  $\alpha$  and vice versa, which balances the importance of each class' examples
8617
while  $f$  is the hypothesis e.g., neural network. In the multi-label case, we alter focal loss formulation such that  $y$  and  $\hat{y}$  become  $\mathbf{y} \in \{0,1\}^{|K|}$  and  $\hat{\mathbf{y}} \in \mathbb{R}^{|K|}$ . Moreover, for  $\alpha$  we propose using
$$
\alpha = \left[ (\alpha_ {1} ^ {(0)}, \alpha_ {1} ^ {(1)}), \ldots , (\alpha_ {k} ^ {(0)}, \alpha_ {k} ^ {(1)}) \right]
$$
where  $\alpha_{k}^{(j)}$  is the normalized inverse frequency of the event  $y^{k} = j$  where  $j\in \{0,1\}$ . In other words, we interpret each class as *two classes*, either 0 or 1, and compute inverse class frequencies for all  $2*|K|$  classes and normalize them such that  $\sum_{k\in K}\sum_{j\in \{0,1\}}\alpha_{k}^{(j)} = 1$ . We observe that this loss matches BCE in  $F1$  scores and prevails it in multi-label accuracy score EM-2 (Exact Match for two frames) by a significant  $11\%$  margin as in Table 1. We use two Binary Relevance approaches based on Naïve Bayes and MultiBERT, respectively, as our baselines. Naïve Bayes is a standard baseline for text classification which leverages Bayes theorem and utilizes word frequencies as features (McCallum et al., 1998). For regularization, we apply add-1 smoothing. The standard configuration for Naïve Bayes is multi-class. One intuitive technique of tailoring Naïve Bayes into a multi-label problem is called Binary Relevance (BR). BR is the method of training  $|K|$  one-vs-rest classifiers independently for each of class  $k\in K$  on the same dataset. As our second baseline, we train nine binary MultiBERTs in a one-vs-rest manner.
4.2 Multilingual Models
GVFC dataset is composed of 1300 relevant samples for the issue of Gun Violence and is only available in English. For cross-lingual transfer, MultiBERT with multi-label Focal loss provides the highest accuracy within English samples that have more than one correct class by a significant  $11\%$  margin,  $62\%$  vs.  $51\%$  in EM-2, while maintaining the same level of F-1 scores as given in Table 1.
Firstly, we explore zero-shot and few-shot performances of our MultiBERT model with Focal loss which is trained on the English dataset as in 2.1 and 2.3 of Table 2. We use German (DE), Arabic (AR), and Turkish (TR) as our target languages to explore the cross-lingual performance of our model to a variety of languages for which we have some validation set but not train set. In our few-shot models, we use extra 40 samples from the target language, i.e., DE, AR, or TR, and use the same training configurations as in the initial training, which we describe in Section 5.
Model (Loss)	F1-Macro	F1-Micro	EM-1	EM-2	Top-2	EM-A
MULTIClass
EngBERT (Liu et al., 2019)	0.77	0.83	0.86	N/A	0.93	0.83
MultiBERT	0.73	0.79	0.82	N/A	0.89	0.79
MULTI-LABEL
BR w/ Naïve Bayes	0.58	0.65	0.58	0.29	0.68	0.51
BR w/ MultiBERT (Binary Focal)	0.74	0.82	0.69	0.58	0.87	0.66
EngBERT (ML Focal)	0.76	0.82	0.71	0.62	0.94	0.69
MultiBERT (ML Focal)	0.76	0.82	0.71	0.62	0.92	0.69
MultiBERT (BCE Loss)	0.76	0.82	0.79	0.51	0.91	0.72
Table 1: English results. Multiclass models consider only the first frame correct and are evaluated accordingly. EM-1, EM-2, EM-A, Top-2: See Section 5. ML: Multi-Label, BR: Binary Relevance.
Furthermore, since the news framing task is fairly a keyword-driven phenomenon (Field et al., 2018), we developed a set of keywords that occur most frequently in a given frame. To this end, we utilize the metric called normalized pointwise mutual information (nPMI) which was suggested by Field et al. (2018). nPMI score for a given frame  $F$  and word  $w$  is  $I(F, w) = \log \frac{P(w|F)}{P(w)}$ . Both  $P(w)$  and  $P(w|F)$  are estimated from the training corpus. We determine the set of important words based on nPMI by selecting the top 250 words for each frame – that also have nPMI greater than zero – resulting in 358 total words. We, then, use word-to-word translation to code-switch (CS) the English training set with the target language ( $TL$ ) for these words. In other words, we replace all utterances of “important” words with it’s  $TL$  dictionary translation. For instance, a sample headline in the training set that was code-switched with German becomes
Florida Schütze ein troubled loner mit Weiß supremacist Bindungen.
which originally was "Florida shooter a troubled loner with white supremacist ties" having both frames "mental illness" and "race/ethnicity". We experiment with using the code-switched data for training in both zero-shot and few-shot, using 40 target language examples. Models based on code-switched training are indicated with  $CS_{TL}$  for target language (TL) in Table 2. Code-switched translation is a way of adapting the model to the target language during training. We observed significant improvements or comparable results both in zero-shot and few-shot settings over the model that was trained on the original English data, as demonstrated in Table 2 for all three languages. Furthermore, we explore the effect of translation direction for the news frame detection task using Google Translate in Table 3.