SlowGuess/NAACL_2021_OCR huggingface dataset:
Shima Khanehzar† Trevor Cohn† Gosia Mikolajczak* Andrew Turpin† Lea Frermann†
† School of Computing and Information Systems
$^{\star}$ School of Social and Political Sciences
skhanehzar@student.unimelb.edu.au
{tcohn,mmikolajcza,aturpin,lfrermann}@unimelb.edu.au
Understanding how news media frame political issues is important due to its impact on public attitudes, yet hard to automate. Computational approaches have largely focused on classifying the frame of a full news article while framing signals are often subtle and local. Furthermore, automatic news analysis is a sensitive domain, and existing classifiers lack transparency in their predictions. This paper addresses both issues with a novel semi-supervised model, which jointly learns to embed local information about the events and related actors in a news article through an auto-encoding framework, and to leverage this signal for document-level frame classification. Our experiments show that: our model outperforms previous models of frame prediction; we can further improve performance with unlabeled training data leveraging the semi-supervised nature of our model; and the learnt event and actor embeddings intuitively corroborate the document-level predictions, providing a nuanced and interpretable article frame representation.
Journalists often aim to package complex real-world events into comprehensive narratives, following a logical sequence of events involving a limited set of actors. Constrained by word limits, they necessarily select some facts over others, and make certain perspectives more salient. This phenomenon of framing, be it purposeful or unconscious, has been thoroughly studied in the social and political sciences (Chong and Druckman, 2007). More recently, the natural language processing community has taken an interest in automatically predicting the frames of news articles (Card et al., 2016; Field et al., 2018; Akyurek et al., 2020; Khanehzar et al., 2019; Liu et al., 2019a; Huguet Cabot et al., 2020).
Definitions of framing vary widely including: expressing the same semantics in different forms
(equivalence framing); presenting selective facts and aspects (emphasis framing); and using established syntactic and narrative structures to convey information (story framing) (Hallahan, 1999). The model presented in this work builds on the concepts of emphasis framing and story framing, predicting the global (aka. primary) frame of a news article on the basis of the events and participants it features.
Primary frame prediction has attracted substantial interest recently with the most accurate models being supervised classifiers built on top of large pretrained language models (Khanehzar et al., 2019; Huguet Cabot et al., 2020). This work advances prior work in two ways. First, we explicitly incorporate a formalization of story framing into our frame prediction models. By explicitly modeling news stories as latent representations over events and related actors, we obtain interpretable, latent representations lending transparency to our frame prediction models. We argue that transparent machine learning is imperative in a potentially sensitive domain like automatic news analysis, and show that the local, latent labels inferred by our model lend explanatory power to its frame predictions.
Secondly, the latent representations are induced without frame-level supervision, requiring only a pre-trained, off-the-shelf semantic role labeling (SRL) model (Shi and Lin, 2019). This renders our frame prediction models semi-supervised, allowing us to use large unlabeled news corpora.
More technically, we adopt a dictionary learning framework with deep autoencoders through which we learn to map events and their agents and patients<sup>1</sup> independently into their respective structured latent space. Our model thus learns a latent multi-view representation of news stories, with each view contributing evidence to the primary
<sup>1</sup>We experiment with three types of semantic roles: predicates and associated arguments (ARG0 and ARG1), however, our framework is agnostic to the types of semantic roles, and can further incorporate other types of semantic roles or labels.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2154-2166 June 6-11, 2021. ©2021 Association for Computational Linguistics
frame prediction from its own perspective. We incorporate the latent multi-view representation into a transformer-based document-level frame classification model to form a semi-supervised model, in which the latent representations are jointly learnt with the classifier.
We demonstrate empirically that our semi-supervised model outperforms current state-of-the-art models in frame prediction. More importantly, through detailed qualitative analysis, we show how our latent features mapped to events and related actors allow for a nuanced analysis and add interpretability to the model predictions<sup>2</sup>. In summary, our contributions are:
- Based on the concepts of story- and emphasis framing, we develop a novel semi-supervised framework which incorporates local information about core events and actors in news articles into a frame classification model.
- We empirically show that our model, which incorporates the latent multi-view semantic role representations, outperforms existing frame classification models, with only labeled articles. By harnessing large sets of unlabeled indomain data, our model can further improve its performance and achieves new state-of-the-art performance on the frame prediction task.
- Through qualitative analysis, we demonstrate that the latent, multi-view representations aid interpretability of the predicted frames.
A widely accepted definition of frames describes them as a selection of aspects of perceived reality, which are made salient in a communicating context to promote a particular problem definition, causal interpretation, moral evaluation and the treatment recommendation for the described issue (Entman, 1993). While detecting media frames has attracted much attention and spawned a variety of methods, it poses several challenges for automatic prediction due to its vagueness and complexity.
Two common approaches in the study of frames focus either on the detailed issue-specific elements of a frame or, somewhat less nuanced, on generic framing themes prevalent across issues. Within the first approach, Matthes and Kohring (2008) developed a manual coding scheme, relying on Entman's
definition (Entman, 1993). While the scheme assumes that each frame is composed of common elements, categories within those elements are often specific to the particular issue being discussed (e.g., "same sex marriage" or "gun control"), making comparison across different issues, and detecting them automatically difficult. Similarly, earlier studies focusing specifically on unsupervised models to extract frames, usually employed topic modeling (Boydstun et al., 2013; Nguyen, 2015; Tsur et al., 2015) to find the issue-specific frames, limiting across-issue comparisons.
Studies employing generic frames address this shortcoming by proposing common categories applicable to different issues. For example, Boydstun et al. (2013) proposed a list of 15 broad frame categories commonly used when discussing different policy issues, and in different communication contexts. The Media Frames Corpus (MFC; Card et al. (2015)) includes about 12,000 news articles from 13 U.S. newspapers covering five different policy issues, annotated with the dominant frame from Boydstun et al. (2013). Table 5 in the Appendix lists all 15 frame types present in the MFC. The MFC has been previously used for training and testing frame classification models. Card et al. (2016) provide an unsupervised model that clusters articles with similar collections of "personas" (i.e., characterisations of entities) and demonstrate that these personas can help predict the coarse-grained frames annotated in the MFC. While conceptually related to our approach, their work adopts the Bayesian modelling paradigm, and does not leverage the power of deep learning. Ji and Smith (2017) proposed a supervised neural approach incorporating discourse structure. The current best result for predicting the dominant frame of each article in the MFC comes from Khanehzar et al. (2019), who investigated the effectiveness of a variety of pre-trained language models (XLNet, Bert and Roberta).
Recent methods have been expanded to multilingual frame detection. Field et al. (2018) used the MFC to investigate framing in Russian news. They introduced embedding-based methods for projecting frames of one language into another (i.e., English to Russian). Akyurek et al. (2020) studied multilingual transfer learning to detect multiple frames in target languages with few or no annotations. Recently, Huguet Cabot et al. (2020) investigated joint models incorporating metaphor,
Source code of our model is available at https://github.com/shinyemimalef/FRISS
Figure 1: Our FRISS model. (a) Given an input document with a frame label  $y$ , we perform semantic role labeling (SRL) to obtain the predicate (blue), ARG0 (purple) and ARG1 (green) for input sentences  $s_1 \ldots s_n$ . (b) The unsupervised module takes as input semantic role embeddings  $(v_{a_0}, v_p, v_{a_1})$  and a sentence embedding  $(v_{s_i})$ , and learns latent, role-specific embedding matrices  $(\mathbf{F})$  in an auto-encoding framework. The latent representations are incorporated into the overall frame classification module. (c) We predict document-level frames based on transformer-based document embeddings  $(\hat{y}_s)$  and the view-specific latent representations  $(\hat{y}_u)$ , using cross-entropy loss with the true frame label  $y$ .
emotion and political rhetoric within multi-task learning to predict framing of policy issues.
Our modelling approach is inspired by recent advances in learning interpretable latent representations of the participants and relationships in fiction stories. Iyyer et al. (2016) present Relationship Modelling Networks (RMNs), which induce latent descriptors of types of relationships between characters in fiction stories, in an unsupervised way. RMNs combine dictionary learning with deep autoencoders, and are trained to effectively encode text passages as linear combinations over latent descriptors, each of which corresponds to a distinct relationship (not unlike topics in a topic model). Frermann and Szarvas (2017) extend the idea to a multi-view setup, jointly learning multiple dictionaries, which capture properties of individual characters in addition to relationships. We adopt this methodology for modeling news articles through three latent views: capturing their events (predicates), and participants (ARG0, ARG1). We combine the unsupervised autoencoder with a frame classifier into an interpretable, semi-supervised framework for article-level frame prediction.
3 Semi-supervised Interpretable Frame Classification
In this section, we present our Frame classifier, which is Interpretable and Semi-supervised (FRISS). The full model is visualized in Figure 1. Given a corpus of news articles, some of which have a label indicating their primary frame  $y$  (Figure 1(a)), FRISS learns to predict  $\hat{y}$  for each document by combining a supervised classification module (Figure 1(c)) and an unsupervised autoencoding module (Figure 1(b)), which are jointly trained. The unsupervised module (i) can be trained with additional unlabeled training data, which improves performance (Section 5.2); and (ii) learns interpretable latent representations which improve the interpretability of the model (Section 5.3).
Intuitively, FRISS predicts frames based on aggregated sentence representation (supervised module; Section 3.2) as well as aggregated fine-grained latent representations capturing actors and events in the article (unsupervised module; 3.1). The un
3The autoencoder is unsupervised wrt. frame-level information, but it relies on an off-the-shelf semantic role labeler.
supervised module combines an auto-encoding objective with a multi-view dictionary learning framework (Iyyer et al., 2016; Frermann and Szarvas, 2017). We treat predicates, their ARG0 and ARG1 as three separate views, and learn to map each view to an individual latent space representative of their relation to the overall framing objective. Below, we will sometimes refer to views collectively as  $z \in \{p, a_0, a_1\}$ . We finally aggregate the view-level representations and sentence representations to predict a document-level frame. The following sections describe FRISS in technical detail.
3.1 Unsupervised Module
3.1.1 Input
Each input document is sentence-segmented and automatically annotated by an off-the shelf transformer-based semantic role labeling model (Shi and Lin, 2019; Pradhan et al., 2013) to indicate spans over the three semantic roles: predicates, ARG0s and ARG1s.
We compute a contextualized vector representation for each semantic role span  $(s_{\mathrm{p}}, s_{\mathrm{a}_0}, s_{\mathrm{a}_1})$ . We describe the process for obtaining predicate input representations  $v_p$  here for illustration. Contextualized representations for views  $a_0$  ( $v_{a_0}$ ) and  $a_1$  ( $v_{a_1}$ ) are obtained analogously. First, we pass each sentence through a sentence encoder, and obtain the predicate embedding by averaging all contextualized token representations  $v_w$  (of dimension  $D_w$ ) in its span  $s_p$  of length  $|s_p|$ :
$$
\boldsymbol {v} _ {\mathrm {p}} = \frac {1}{| s _ {p} |} \sum_ {w \in s _ {p}} \boldsymbol {v} _ {w}. \tag {1}
$$
We concatenate  $\pmb{v}_{\mathrm{p}}$  with an overall sentence representation  $\pmb{v}_{s}$ , which is computed by averaging all contextualized token embeddings of the sentence  $s$  of length  $|s|$ ,<sup>4</sup>
$$
\boldsymbol {v} _ {s} = \frac {1}{| s |} \sum_ {w \in s} \boldsymbol {v} _ {w} \tag {2}
$$
$$
\tilde {\boldsymbol {v}} _ {\mathrm {p}} = \left[ \boldsymbol {v} _ {\mathrm {p}}; \boldsymbol {v} _ {s} \right], \tag {3}
$$
where  $[\cdot ]$  denotes vector concatenation. If a sentence has more than one predicate, a separate representation is computed for each of them.
3.1.2 Multi-view Frame representations
We combine ideas from auto-encoding (AE) and dictionary learning, as previously used to capture the content of fictitious stories (Iyyer et al.,
2016), and its multi-view extension (Fremann and Szarvas, 2017). We posit a latent space as three view-specific dictionaries (Figure 1 (b)) capturing events (predicates;  $\pmb{F}_{p}$ ), their first (ARG0;  $\pmb{F}_{\mathrm{a_0}}$ ) and second (ARG1;  $\pmb{F}_{\mathrm{a_1}}$ ) arguments, respectively. Given a view-specific input as described above, the autoencoder maps it to a low-dimensional distribution over "dictionary terms" (henceforth descriptors), which are learnt during training. The descriptors are vector-valued latent variables that live in word embedding space, and are hence interpretable through their nearest neighbors (Table 3 shows examples of descriptors inferred by our model). By jointly learning the descriptors with the supervised classification objective, each descriptor will capture coherent information corresponding to a frame label in our supervised data set. We hence set the number of descriptors for each dictionary to  $K = 15$ , the number of frames in our data set. For each view  $z \in \{p, \mathrm{a}_0, \mathrm{a}_1\}$ , we define a dictionary  $\pmb{F}_{z}$  of dimensions  $K \times D_w$ .
More technically, our model follows two steps. First, we encode the input  $\tilde{\pmb{v}}_z$  of a known view  $z$  by passing it through a feed forward layer  $\mathbf{W}_h$  of dimensions  $2D_w\times D_h$ , shared across all the views, followed by a ReLU non-linearity, and then another feed forward layer  $\mathbf{W}_z$  of dimensions  $D_h\times K$ , specific to each view  $z$ . This results in a  $K$ -dimensional vector over the view-specific descriptors,
$$
\boldsymbol {l} _ {z} = \mathbf {W} _ {z} \mathrm {R e L U} (\boldsymbol {W} _ {h} \tilde {\boldsymbol {v}} _ {z}), \tag {4}
$$
Second, we reconstruct the original view embedding  $v_{z}$  as a linear combination of descriptors. While previous work used  $l_{z}$  directly as weight vector, we hypothesize that on our fine-grained semantic role level, only one or a few descriptors will be relevant to any specific span. We enforce this intuition using Gumbel-Softmax differentiable sampling with temperature annealing (Jang et al., 2017). This allows us to gradually constrain the number of relevant descriptors used for reconstruction. We first normalize  $l_{z}$ ,
$$
\boldsymbol {d} _ {z} = \operatorname {S o f t m a x} \left(\boldsymbol {l} _ {z}\right), \tag {5}
$$
and then draw  $\pmb{g}$  from the Gumbel distribution, and add it to our normalized logits  $d_{z}$  scaled by temperature  $\tau$ , which is gradually annealed over the
4We also experimented with representing the sentence as the [CLS] token embedding, but found it to perform worse empirically.
training phase:
$$
\boldsymbol {g} \sim \operatorname {G u m b e l} (0, 1)
$$
$$
\boldsymbol {g} _ {z} = \frac {\exp \left(\log \left(\boldsymbol {d} _ {z}\right) + \frac {\boldsymbol {g}}{\tau}\right)}{\sum_ {f} \exp \left(\log \left(\boldsymbol {d} _ {z}\right) + \frac {\boldsymbol {g}}{\tau}\right)}. \tag {6}
$$
We finally reconstruct the view-specific span embedding as
$$
\hat {\boldsymbol {v}} _ {z} = \boldsymbol {F} _ {z} ^ {T} \boldsymbol {g} _ {z}. \tag {7}
$$
3.1.3 Unsupervised Objective
Contrastive Loss We use the contrastive max-margin objective function following previous works in dictionary learning (Iyyer et al., 2016; Frermann and Szarvas, 2017; Han et al., 2019). We randomly sample a set of negative samples  $(N_{-})$  with the same view as the current input from the mini-batch. The unregularized objective  $J_z^u$  (Eq. 8) is a hinge loss that minimizes the L2 norm between the reconstructed embedding  $\hat{v}_z$  and the true input's view-specific embedding  $\boldsymbol{v}_z$ , while simultaneously maximizing the L2 norm between  $\hat{v}_z$  and negative samples  $\boldsymbol{v}_z^n$ :
$$
\begin{array}{l} J _ {z} ^ {u} (\theta) = \frac {1}{| N _ {-} |} \sum_ {\boldsymbol {v} _ {z} ^ {n} \in N _ {-}} \max  (0, 1 + l _ {2} (\hat {\boldsymbol {v}} _ {z}, \boldsymbol {v} _ {z}) \tag {8} \\ - l _ {2} (\hat {\pmb {v}} _ {z}, \pmb {v} _ {z} ^ {n})), \\ \end{array}
$$
where  $\theta$  represents the model parameters,  $|N_{-}|$  is the number of negative samples, and the margin value is set to 1.
Focal Triplet Loss Preliminary studies (Section 5) suggested that some descriptors (aka frames) are more similar to each other than others. We incorporate this intuition through a novel mechanism to move the descriptors that are least involved in the reconstruction proportionally further away from the most involved descriptor.
Concretely, we select  $t$  descriptors in  $\mathbf{F}_z$  with smallest weights in  $\pmb{g}_{z}$  as additional negative samples. We denote the indices of the selected  $t$  smallest components in  $\pmb{g}_{z}$
$$
I = \left[ i _ {1}, i _ {2}, \dots , i _ {t} \right]. \tag {9}
$$
We use  $\pmb{F}_z^t$  to denote the matrix  $(t \times D_w)$  with only those  $t$  descriptors. We re-normalize the weights of the selected  $t$  descriptors, and denote the renormalized weights vector as  $\pmb{g}_z^t =$
$[g_z^{i_1}, g_z^{i_2}, \ldots, g_z^{i_t}]$ . For each element in  $g_z^t$ , we compute an individual margin based on its magnitude. Intuitively, the smaller the weight is, the larger its required margin from a given total margin budget  $|M|$ ,
$$
m _ {z} ^ {i _ {t}} = | M | * \left(1 - g _ {z} ^ {i _ {t}}\right) ^ {2}. \tag {10}
$$
We compute the standard margin-based hinge loss over the additional negative samples with sample-specific margins:
$$
\begin{array}{l} J _ {z} ^ {t} (\theta) = \frac {1}{| T |} \sum_ {i _ {t} \in I} \max  \left(0, m _ {z} ^ {i _ {t}} + l _ {2} \left(\hat {\boldsymbol {v}} _ {z}, \boldsymbol {v} _ {z}\right) \right. \tag {11} \\ - l _ {2} \left(\hat {\boldsymbol {v}} _ {z}, \boldsymbol {v} _ {z} ^ {i _ {t}}\right)). \\ \end{array}
$$
We sum the focal triplet objective  $J_z^t$  with  $J_z^u$ , and then sum over all specific spans  $s \in S_z$ , while adding an additional orthogonality encouraging regularization term.
$$
J _ {z} (\theta) = \sum_ {s \in S _ {z}} \left(J _ {z} ^ {u} + J _ {z} ^ {t}\right) + \lambda \left\| \mathbf {F} _ {\mathbf {z}} \mathbf {F} _ {\mathbf {z}} ^ {\mathbf {T}} - \mathbf {I} \right\| _ {\mathbf {F}} ^ {2}, \tag {12}
$$
where  $\lambda$  is a hyper-parameter that can be tuned. We finally aggregate the loss from all the views:
$$
J (\theta) = \sum_ {z \in \left\{\mathrm {p}, \mathrm {a} _ {0}, \mathrm {a} _ {1} \right\}} J _ {z} (\theta). \tag {13}
$$
3.2 Supervised Document-level Frame Classification
We incorporate the semantic role level predictions as described above into a document-level frame classifier consisting of two parts, which are jointly learnt with the unsupervised model described above: (i) a classifier based on aggregated span-level representations computed as described in Sec 3.1 (Fig. 1 (c; left); Sec. 3.2.1) and (ii) a classifier based on an aggregated sentence representations (Fig. 1 (c; right); Sec. 3.2.2).
3.2.1 Span-based Classifier
The unsupervised module makes predictions on the semantic role span level, however, our goal is to predict document-level frame labels. We aggregate span-level representations  $d_{z}$  (Eq. 5) by averaging across spans and then views:6
$$
\boldsymbol {w} _ {u} = \frac {1}{Z} \sum_ {z \in \left\{\mathrm {p}, \mathrm {a} _ {0}, \mathrm {a} _ {1} \right\}} \frac {1}{| S _ {z} |} \sum_ {s \in S _ {z}} \boldsymbol {d} _ {z} ^ {s} \tag {14}
$$
$$
\hat {y} _ {u} = \operatorname {S o f t m a x} (\boldsymbol {w} _ {u}),
$$
<sup>6</sup>We empirically found these representations to outperform the sparser  $\pmb{g}_{z}$ .
<sup>5</sup>We empirically found that L2 norm outperforms the dot product, and cosine similarity.
where  $Z$  is the number of the views, and  $S_{z}$  are the set of view-specific spans in the current document. We finally pass the logits through a softmax layer to predict a distribution over frames.
3.2.2 Sentence-based Classifier
We separately predict a document-level frame based on the aggregate sentence level representations computed in Eq. (2). We first pass each sentence embedding through a feed forward layer  $W_{r}$  of dimensions  $D_w \times D_w$ , followed by a ReLU non-linearity, and another feed forward layer  $W_{t}$  to map the resulting representation to  $K$  dimensions. Then average across sentences of the current document  $S_{d}$  and pass the result through a softmax layer,
$$
\begin{array}{l} \pmb {w} _ {s} = \mathrm {R e L U} (\pmb {W} _ {r} \pmb {v} _ {s}) \\ \hat {y} _ {s} = \operatorname {S o f t m a x} \left(\frac {1}{| S _ {d} |} \sum_ {s \in S _ {d}} \boldsymbol {W} _ {t} \boldsymbol {w} _ {s}\right). \tag {15} \\ \end{array}
$$
3.3 Full Loss
We jointly train the supervised and unsupervised model components. The supervised loss  $X(\theta)$  consists of two parts, one for the sentence-based classification and one for the aggregated span-based classification:
$$
X (\theta) = X \left(\hat {y} _ {u}, y\right) + X \left(\hat {y} _ {s}, y\right). \tag {16}
$$
The full loss balances the supervised and unsupervised components with a hyper-parameter  $\alpha$ :
$$
L (\theta) = \alpha \times X (\theta) + (1 - \alpha) \times J (\theta). \tag {17}
$$