SlowGuess/NAACL_2021_OCR huggingface dataset:
Dataset We follow prior work on automatic prediction of a single, primary frame of a news article as annotated in the Media Frames Corpus (MFC; Card et al. (2015)). The MFC contains a large number of news articles on five contentious policy issues (immigration, smoking, gun control, death penalty, and same-sex marriage), manually annotated with document- and span-level frames labels from a set of 15 general frames (listed in Table 5 in the Appendix). Articles were selected from 13 major U.S. newspapers, published between 1980 and 2012. Following previous work, we focus on the immigration portion of MFC, which comprises 5,933 annotated articles, as well as an additional 41,286 unlabeled articles. The resulting dataset contains all 15 frames. Table 5 (Appendix) lists the
corresponding frame distribution. We partition the labeled dataset into 10 folds, preserving the overall frame distribution for each fold.
Pre-processing and Semantic Role labeling We apply state-of-art BERT-based SRL model (Shi and Lin, 2019) to obtain SRL spans for each sentence. The off-the-shelf model from AllenNLP is trained on OntoNotes5.0 (close to  $50\%$  news text). While a domain-adapted model may lead to a small performance gain, the off-the-shelf model enhances generalizability and reproducibility. Qualitative examples of detected SRL spans are shown in Table 4, which confirm that SRL predictions are overall accurate.
We extract semantic role spans for predicates, their associated first (ARG0) and second (ARG1) arguments for each sentence in a document. For the unsupervised component, we disregard sentences with no predicate, and sentences missing both ARG0 and ARG1.
Sentence Encoder In all our experiments, we use RoBERTa (Liu et al., 2019b) as our sentence encoder, as previous work (Khanehzar et al., 2019) has shown that it outperforms BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019). We pass each sentence through RoBERTa and retrieve the token-level embeddings. To obtain the sentence embedding, we average the RoBERTa embeddings of all words (Eq. 2). To obtain SRL span embeddings, we average the token embeddings of all words in a predicted span (Eq. 1). Following Gururangan et al. (2020), we pre-train RoBERTa with immigration articles using the masked language model (MLM) objective. Only the labeled data is used for pre-training for fair comparison between FRISS and previous models.
Parameter Settings We set the maximum sequence length to RoBERTa 64 tokens, the maximum number of sentences per document to 32, and the maximum number of predicates per sentence to 10.7 We set the number of dictionary terms  $K = 15$ , i.e., the number of frame classes in the MFC corpus. Each dictionary term is of dimension  $D_w = 768$ , equal to the RoBERTa token embedding dimension. We also fix the dimensions of hidden vector  $\boldsymbol{w}_s$  (Eqn. 15) and  $D_h$  to this value. We set the number of descriptors in Focal Triplet Loss
$796\%$  of the sentences are under 64 tokens;  $95\%$  of the documents have less than 32 sentences, and  $\gg 99\%$  of the sentences have less than 10 predicates.
Model	Acc.	Macro-F1
Card et al. (2016)	56.8	-
Field et al. (2018)	57.3	-
Ji and Smith (2017)	58.4	-
Khanehzar et al. (2019)	65.8	-
RoBERTa-S	66.4 (0.008)	58.1 (0.013)
RoBERTa-S+MLM	67.1 (0.008)	58.8 (0.013)
FRISS (labeled only)	68.8 (0.009)	60.1 (0.011)
FRISS (labeled+unlabeled)	69.7 (0.011)	60.5 (0.015)
Table 1: Primary Frame prediction on the MFC immigration data set with mean (standard deviation) over 10-fold cross-validation. We compare our full model FRISS against recent work, and the supervised RoBERTa component with and without pre-training. The results are statistically significant  $(p < 0.05$ ; paired sample t-test). FRISS (labeled only) vs RoBERTa-S+MLM:  $p = 0.009$ ; FRISS (labeled + all unlabeled) vs FRISS (labeled only):  $p = 0.012$ ; FRISS (labeled + all unlabeled) vs RoBERTa-S+MLM:  $p = 0.003$ .
$t = 8$  and the margin pool  $|M| = t$ . We set the balancing hyper-parameter between the supervised and unsupervised loss  $\alpha = 0.5$ , and  $\lambda = 10^{-3}$ . The dropout rate is set to 0.3.
We perform stochastic gradient descent with mini-batches of 8 documents. We use the Adam optimizer (Kingma and Ba, 2015) with the default parameters, except for the learning rate, which we set to  $2 \times 10^{-5}$  (for the RoBERTa parameters) and  $5 \times 10^{-4}$  (for all other parameters). We use a linear scheduler for learning rate decay. The weight decay is applied to all parameters except for bias and batch normalization. We update the Gumbel softmax temperature with the schedule:  $\tau = \max(0.5, \exp(-5 \times 10^{-4} \times \text{iteration})$ , updating the temperature every 50 iterations. For all our experiments, we run a maximum of 10 epochs, evaluate every 50 iterations, and apply early-stopping if the accuracy does not improve for 20 consecutive evaluations.
In this section, we evaluate the performance of FRISS on primary frame prediction for issue-specific news articles against prior work (Sec 5.1), demonstrate the benefit of adding additional unlabeled data to our semi-supervised model (Sec 5.2), and present a qualitative analysis of our model output corroborating its interpretability (Sec 5.3).
Preliminary Studies Preliminary analyses revealed that human annotators disagree on frame
Model	Acc.	Macro-F1
FRISS	68.83	60.05
- focal	68.73	59.98
- Gumbel	68.51	59.70
- focal, Gumbel	68.47	59.72
p only	68.35	59.50
ARG0 only	68.47	59.68
ARG1 only	68.53	59.71
Table 2: Ablation results for FRISS on primary frame prediction. Top removing Focal Triplet Loss (focal; 3.1.3), and/or Gumbel distribution (Gumbel; 3.1.2). Bottom FRISS trained with only a single view (predicate, ARG0 and ARG1) on primary frame prediction.
labels non-uniformly, suggesting that some pairs of frames are perceived to be more similar than others. This observation motivated the Focal Triplet Loss and Gumbel regularization components of our model. In particular, the following groups of frame labels are confused most frequently {"Policy Prescription and Evaluation", "Public Sentiment", "Political"}, {"Fairness", "Legality"}, {"Crime and Punishment", "Security and Defense"}, and {"Morality", "Quality of Life", "Cultural Identity)}. This observation is also corroborated through the empirical gain through the focal triplet loss (Table 2).
5.1 Experiment 1: Frame Prediction
For the supervised model, we report accuracy, as has been done in previous work, as well as MacroF1, which is oblivious to class sizes, shedding light on performance across all frames. Table 1 compares FRISS against related work. Card et al. (2016) incorporate latent personas learnt with a Bayesian model; Field et al. (2018) derive frame-specific lexicons based on pointwise-mutual information; Ji and Smith (2017) incorporate a supervised discourse classifier, and Khanehzar et al. (2019) train frame classifiers on top of RoBERTa-based document embeddings. RoBERTa-S corresponds to the sentence-embedding based component of FRISS (Fig 1(b); left) without and with (+MLM) unsupervised pre-training. Overall, we can see that all our model variants outperform previous work in terms of both accuracy and macro-F1. Experiments were run 5 times with 10-fold cross-validation. The results in Table 1 are statistically significant  $(p < 0.05$ ; paired sample t-test).
To better understand the impact of our various
Figure 2: FRISS frame prediction performance with different portions of the 41K unlabeled documents.
model components, we performed an ablation study on Focal Triplet Loss, the Gumbel regularization, and the impact of individual views. Table 2 shows that both the focal loss and the Gumbel regularization contribute to model performance. Training FRISS with any single view individually leads to a performance drop, which is most drastic if the two arguments are omitted, suggesting that the model relies on both predicate and argument information, with arguments playing a slightly more important role.
5.2 Experiment 2: Benefit of Unlabeled Data
Our semi-supervised model can leverage news articles without a frame label, in addition to a labeled training set. We investigated the impact of training FRISS with different amounts of additional news articles, taken from the unlabeled immigration portion of the MFC. Figure 2 shows the impact of additional unlabeled data on accuracy and F1: Models with access to more unlabelled data tend to result in higher accuracy and Macro F1 scores. Given the abundance of online news articles, this motivates future work on minimally supervised frame prediction, minimizing the reliance on manual labels and maximizing generalizability to new issues, news outlets or languages.
5.3 Experiment 3: Qualitative Evaluation
In this experiment, we explore the added interpretability contributed by the local latent frame representations. Table 4 contains two MFC documents, highlighted with the most highly associated frame for each identified span for  $p$ ,  $a_0$  or  $a_1$ . We can observe that the frame associations (a) are intuitively meaningful; and (b) provide a detailed account of the predicted primary frame. For both documents the gold primary frame is 'Political', the bottom document is classified correctly, whereas the top document is mis-classified as 'Capacity & Resources'. The detailed span-level predictions
Table 3: Spans inferred as most highly associated with the Capacity & Resources (■), Political (■), Legality (■), and Public Sentiment (■) frames, for each view (ARG0, PRED, ARG1).
help to explain the model prediction, and in fact add support for the the mis-prediction, suggesting that predicting a single primary document frame may be inappropriate. In the bottom document “a letter” serves as both  $\mathrm{a}_{1}$  of “Republicans sent a letter”, where it is predicted as ‘Political’, and as  $\mathrm{a}_{0}$  of the clause “a letter [...] describing the legal challenges”, where it is classified as ‘Legality’, another example of the nuance of our model predictions, which can support further in-depth study of issue-specific framing.
The potential of our model for fine-grained frame analysis is illustrated in Table 4, which shows how each particular SRL span contributes differently towards various frame categories. It adds a finer-grained framing picture, and estimate of the trustworthiness of model predictions. It allows to assess the main actors wrt. a particular frame (within and across articles), as well as the secondary frames in each article. Also, using SRL makes our model independent of human annotation, and more generalizable. Going beyond "highlighting indicative phrases", our model can distinguish their roles (e.g., the "ICE" as an actor vs. participant in a particular frame).
Table 3 shows the semantic role spans, which are most closely related to Capacity & Resources (blue), Political (red), Legality (purple) and Public Sentiment (green) descriptors in the latent space. We can observe that all associated spans are intuitively relevant to the {frame, view}. Furthermore, ARG0 spans tend to correspond to active participants (agents) in the policy process (including politicians and government bodies), whereas ARG1 spans illustrate the affected participants (patients such as foreign workers, applicants), pro
Table 4: Two articles from the MFC, annotated with SRL span-level frame predictions generated by FRISS. The true frame label of both articles is Political (red). Each detected span  $(\mathrm{p},\mathrm{a}_0$  or  $\mathbf{a}_1)$  has been highlighted with its most closely associated frame. Darker shades indicate higher confidence. The top document is mis-classified as "Capacity & Resources" (blue), the bottom document is classified correctly.
cesses (reforms, cases, movements), or concepts under debate (political asylum). In future work, we aim to leverage these representations in scalable, in-depth analyses of issue-specific media framing. A full table illustrating the learnt descriptors for all 15 frames in the MFC and all three views is included in Table 6 in Appendix.
We presented FRISS, an interpretable model of media frame prediction, incorporating notions of emphasis framing (selective highlighting of issue aspects) and story framing (drawing on the events and actors described in an article). Our semi-supervised model predicts article-level frame of news articles, leveraging local predicate and argument level embeddings. We demonstrated its three-fold advantage: first, our model empirically outperforms existing models for frame classification; second, it can effectively leverage additional unlabeled data further improving performance; and, finally, its latent representations add transparency to classifier predictions and provide a nuanced article representation. The analyses provided by our model can support downstream applications such as automatic, yet transparent, highlighting of re
porting patterns across countries or news outlets; or frame-guided summarization which can support both frame-balanced or frame-specific news summaries. In future work, we plan to extend our work to more diverse news outlets and policy issues, and explore richer latent models of article content, including graph representations over all involved events and actors.
We thank the anonymous reviewers for their helpful feedback and suggestions. This article was written with the support from the graduate research scholarship from the Melbourne School of Engineering, University of Melbourne provided to the first author. The original news articles used in this work were obtained from Lexis Nexis under the institutional licence held by the University of Melbourne. This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne, established with the assistance of LIEF Grant LE170100200.
Afra Feyza Akyurek, Lei Guo, Randa Elanwar, Prakash Ishwar, Margrit Betke, and Derry Tanti Wijaya. 2020. Multi-label and multilingual news framing analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8614-8624, Online. Association for Computational Linguistics.
Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2013. Identifying media frames and frame dynamics within and across policy issues.
Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers, pages 438-444. The Association for Computer Linguistics.
Dallas Card, Justin H. Gross, Amber E. Boydstun, and Noah A. Smith. 2016. Analyzing framing through the casts of characters in the news. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1410-1420. The Association for Computational Linguistics.
Dennis Chong and James N Druckman. 2007. Framing theory. Annu. Rev. Polit. Sci., 10:103-126.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Robert M Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of communication, 43(4):51-58.
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in russian news: a computational analysis of intricate political strategies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3570-3580.
Lea Frermann and György Szarvas. 2017. Inducing semantic micro-clusters from deep multi-view representations of novels. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1873-1883. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Kirk Hallahan. 1999. Seven models of framing: Implications for public relations. Journal of public relations research, 11(3):205-242.
Xiaochuang Han, Eunsol Choi, and Chenhao Tan. 2019. No permanent Friends or enemies: Tracking relationships between nations from news. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1660-1676, Minneapolis, Minnesota. Association for Computational Linguistics.
Pere-Lluis Huguet Cabot, Verna Dankers, David Abadi, Agneta Fischer, and Ekaterina Shutova. 2020. The Pragmatics behind Politics: Modelling Metaphor, Framing and Emotion in Political Discourse. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4479-4488, Online. Association for Computational Linguistics.
Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former Friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1534–1544, San Diego, California. Association for Computational Linguistics.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 996-1005.
Shima Khanehzar, Andrew Turpin, and Gosia Mikolajczak. 2019. Modeling political framing across policy issues and contexts. In Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association, pages 61-66, Sydney, Australia. Australasian Language Technology Association.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Siyi Liu, Lei Guo, Kate K. Mays, Margrit Betke, and Derry Tanti Wijaya. 2019a. Detecting frames in news headlines and its application to analyzing news framing trends surrounding U.S. gun violence. In Proceedings of the 23rd Conference on Computational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, pages 504-514. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Jörg Matthes and Matthias Kohring. 2008. The content analysis of media frames: Toward improving reliability and validity. Journal of communication, 58(2):258-279.
Viet An Nguyen. 2015. Guided Probabilistic Topic Models for Agenda-Setting and Framing. Ph.D. thesis.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pages 143-152.
Peng Shi and Jimmy Lin. 2019. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1904.05255.
Oren Tsur, Dan Calacci, and David Lazer. 2015. A frame of mind: Using statistical models for detection of framing and agenda setting campaigns. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1629-1638.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
A Frame Descriptors Extractions
We denote the set of spans that belong to a specific  $\{\mathrm{frame},\mathrm{view}\}$  combination as  $C_z^f$  , for all possible combinations of frame categories and views. We obtain the high confidence set  $\hat{C}_z^f$  for each specific  $\{\mathrm{frame},\mathrm{view}\}$  combination by only preserving the spans whose  $g_{z}^{f} > 0.8$  . To obtain a more general sets of spans, we remove the stopwords and lemmatize the spans. To normalize the
sets of high-confidence spans, we calculate the inverse-document frequency for each span  $C_z^f$  and sort the spans accordingly by the inverse-document frequency to obtain the representative descriptors. Table 6 illustrating the learnt descriptors for all 15 frames in the MFC and all three views.
Frame	Frame description	% IMM
Economic:	costs, benefits, or other financial implications	7.0%
Capacity and Resources:	availability of physical, human or financial re-sources, and capacity of current systems Morality: religious or ethical implications	3.5%
Morality:	religious or ethical implications	1.3%
Fairness and Equality:	balance or distribution of rights, responsibilities, and resources	2.6%
Legality, Constitutionality, Jurisdiction:	rights, freedoms, and authority of individuals, cor- porations, and government	16.1%
Policy Prescription and Evaluation:	discussion of specific policies aimed at addressing problems	8.0%
Crime and Punishment:	effectiveness and implications of laws and their enforcement	13.6%
Security and Defence:	threats to welfare of the individual, community, or nation	4.9%
Health and Safety:	health care, sanitation, public safety	4.0%
Quality of Life:	threats and opportunities for the individual's wealth, happiness, and well-being	7.0%
Cultural Identity:	traditions, customs, or values of a social group in relation to a policy issue	9.%
Public Sentiment:	attitudes and opinions of the general public, in- cluding polling and demographics	4.1%
Political:	considerations related to politics and politicians, including lobbying, elections, and attempts to sway voters	16.3%
External Regulation and Reputation:	international reputation or foreign policy of the U.S.	2.2%
Other:	any coherent group of frames not covered by the above categories	0.2%
Table 5: Framing dimensions from (Boydstun et al., 2013). The final column (\% IMM) denotes the frame prevalence in the Immigration portion of the MFC used in the experiments reported in this paper.
Table 6: Spans inferred as most highly associated with: Capacity & Resources (■), Political (■), Legality (■), Public Sentiment (■), Economic (■), Morality (■), Fairness & Equality (■), Policy Prescription & Evaluation (■), Crime & Punishment (■), Security & Defense (■), Health & Safety (■), Quality of Life (■), Cultural Identity (■), External Regulation & Reputation (■) frames.