SlowGuess/NAACL_2022_OCR huggingface dataset:
Nayeon Lee Yejin Bang Tiezheng Yu Andrea Madotto* Pascale Fung
nyleea@connect.ust.hk, pascale@ece.ust.hk
Media news framing bias can increase political polarization and undermine civil society. The need for automatic mitigation methods is therefore growing. We propose a new task, a neutral summary generation from multiple news articles of the varying political leanings to facilitate balanced and unbiased news reading. In this paper, we first collect a new dataset, illustrate insights about framing bias through a case study, and propose a new effective metric and model (NEUS-TITLE) for the task. Based on our discovery that title provides a good signal for framing bias, we present NEUS-TITLE that learns to neutralize news content in hierarchical order from title to article. Our hierarchical multi-task learning is achieved by formatting our hierarchical data pair (title, article) sequentially with identifier-tokens ("TITLE=>", "ARTICLE=>") and fine-tuning the auto-regressive decoder with the standard negative log-likelihood objective. We then analyze and point out the remaining challenges and future directions. One of the most interesting observations is that neural NLG models can hallucinate not only factually inaccurate or unverifiable content but also politically biased content.
Media framing bias occurs when journalists make skewed decisions regarding which events or information to cover (information bias) and how to cover them (lexical bias) (Entman, 2002; Groeling, 2013). Even if the reporting of the news is based on the same set of underlying issues or facts, the framing of that issue can convey a radically different impression of what actually happened (Gentzkow and Shapiro, 2006). Since the news media plays a crucial role in shaping public opinion toward various important issues (De Vreese, 2004; McCombs
Figure 1: Illustration of the proposed task. We want to generate neutral summarization of news articles from varying of political orientations. Orange highlights indicate phrases that can be considered framing bias.
and Reynolds, 2009; Perse and Lambe, 2016), bias in media reporting can reinforce the problem of political polarization and undermining civil society rights.
Allsides.com (Sides, 2018) mitigates this problem by displaying articles from various media in a single interface along with an expert-written roundup of news articles. This roundup is a neutral summary for readers to grasp a bias-free understanding of an issue before reading individual articles. Although Allsides fights framing bias, scalability still remains a bottleneck due to the time-consuming human labor needed for composing the roundup. Multi-document summarization (MDS) models (Lebanoff et al., 2018; Liu and Lapata, 2019) could be one possible choice for automating the roundup generation as both multi-document summaries and roundups share a similar nature in extracting salient information out of multiple input articles. Yet the ability of MDS models to provide neutral description of a topic issue – a crucial aspect of the roundup – remains unexplored.
* This work was done when the author was studying at The Hong Kong University of Science and Technology.
Human Language Technologies, pages 3131 - 3148
In this work, we fill in this research gap by proposing a task of Neutral multi-news Summarization (NEUS), which aims to generate a framing-bias-free summary from news articles with varying degrees and orientation of political bias (Fig. 1). To begin with, we construct a new dataset by crawling Allsides.com, and investigate how framing bias manifests in the news so as to provide a more profound and more comprehensive analysis of the problem. The first important insight from our analysis is a close association between framing bias and the polarity of the text. Grounded on this basis, we propose a polarity-based framing-bias metric that is simple yet effective in terms of alignment with human perceptions. The second insight is that titles serve as a good indicator of framing bias. Thus, we propose NEUS models that leverage news titles as an additional signal to increase awareness of framing bias.
Our experimental results provide rich ideas for understanding the problem of mitigating framing bias. Primarily, we explore whether existing summarization models can already solve the problem and empirically demonstrate their shortcomings in addressing the stylistic aspect of framing bias. After that, we investigate and discover an interesting relationship between framing bias and hallucination, an important safety-related problem in generation tasks. We empirically show that the hallucinatory generation has the risk of being not only factually inaccurate and/or unverifiable but also politically biased and controversial. To the best of our knowledge, this aspect of hallucination has not been previously discussed. We thus hope to encourage more attention toward hallucinatory framing bias to prevent automatic generations from fueling political bias and polarization.
We conclude by discussing the remaining challenges to provide insights for future work. We hope our work with the proposed NEUS task serves as a good starting point to promote the automatic mitigation of media framing bias.
Media Bias Media bias has been studied extensively in various fields such as social science, economics, and political science. Media bias is known to affect readers' perceptions of news in three main ways: priming, agenda-setting, and
framing<sup>1</sup>(Scheufele, 2000). Framing is a broad term that refers to any factor or technique that affects how individuals perceive certain reality or information (Goffman, 1974; Entman, 1993, 2007; Gentzkow and Shapiro, 2006). In the context of news reports, framing is about how an issue is characterized by journalists and how readers take the information to form their impression (Scheufele and Tewksbury, 2007). Our work specifically focuses on framing "bias" that exists as a form of text in the news. More specifically, we focus on different writing factors such as word choices and the commission of extra information that sway an individual's perception of certain events.
Media Bias Detection In natural language processing (NLP), computational approaches for detecting media bias often consider linguistic cues that induce bias in political text (Recasens et al., 2013; Yano et al., 2010; Lee et al., 2019; Morstatter et al., 2018; Lee et al., 2019; Hamborg et al., 2019b; Lee et al., 2021b; Bang et al., 2021). For instance, Gentzkow and Shapiro count the frequency of slanted words within articles. These methods mainly focus on the stylistic ("how to cover") aspect of framing bias. However, relatively fewer efforts have been made toward the informational ("what to cover") aspect of framing bias (Park et al., 2011; Fan et al., 2019). Majority of literature doing informational detection are focused on more general factual domain (non-political information) in the name of "fact-checking" (Thorne et al., 2018; Lee et al., 2018, 2021a, 2020). However, these methods cannot be directly applied to media bias detection because there does not exist reliable source of gold standard truth to fact-check biased text upon.
Media Bias Mitigation News aggregation, by displaying articles from different news outlets on a particular topic (e.g., Google News, $^2$  Yahoo News $^3$ ), is the most common approach to mitigate media bias (Hamborg et al., 2019a). However, news aggregators require willingness and effort from the readers to be resistant to framing biases and identify the neutral fact from differently framed articles. Other approaches have been proposed to
<sup>1</sup> Priming happens when news reporting tells the reader what context of the event should they evaluate the event in; Agenda-setting is when news reporting tell readers what are the most important problems to think about
$^{2}$ https://news.google.com/
<sup>3</sup>https://news.yahoo.com/
provide additional information (Laban and Hearst, 2017), such as automatic classification of multiple viewpoints (Park et al., 2009), multinational perspectives (Hamborg et al., 2017), and detailed media profiles (Zhang et al., 2019b). However, these methods focus on providing a broader perspective to readers from an enlarged selection of articles, which still puts the burden of mitigating bias on the readers. Instead, we propose to automatically neutralize and summarize partisan articles to produce a neutral article summary.
Multi-document Summarization As a challenging subtask of automatic text summarization, multi-document summarization (MDS) aims to condense a set of documents to a short and informative summary (Lebanoff et al., 2018). Recently, researchers have applied deep neural models for the MDS task thanks to the introduction of large-scale datasets (Liu et al., 2018; Fabbri et al., 2019). With the advent of large pre-trained language models (Lewis et al., 2019; Raffel et al., 2019), researchers have also applied them to improve the MDS models, performance (Jin et al., 2020; Pasunuru et al., 2021). In addition, many works have studied particular subtopics of the MDS task, such as agreement-oriented MDS (Pang et al., 2021), topic-guided MDS (Cui and Hu, 2021) and MDS of medical studies (DeYoung et al., 2021). However, few works have explored generating framing-bias-free summaries from multiple news articles. To add to this direction, we propose the NEUS task and creates a new benchmark.
The main objective of NEUS is to generate a neutral article summary  $A_{neu}$  given multiple news articles  $A_{0\dots N}$  with varying degrees and orientations of political bias. The neutral summary  $A_{neu}$  should (i) retain salient information and (ii) minimize as much framing bias as possible from the input articles.
3.2 ALLSIDES Dataset
Allsides.com provides access to triplets of news, which comprise reports from left, right, and center American publishers on the same event, with an expert-written neutral summary of the articles and its neutral title. The dataset language is English and mainly focuses on U.S. political topics that often re
sult in media bias. The top-3 most frequent topics<sup>4</sup> are 'Elections', 'White House', and 'Politics'.
We crawl the article triplets<sup>5</sup> to serve as the source inputs  $\{A_L, A_R, A_C\}$ , and the neutral article summary to be the target output  $A_{neu}$  for our task. Note that "center" does not necessarily mean completely bias-free (all, 2021) as illustrated in Table 1. Although "center" media outlets are relatively less tied to a particular political ideology, their reports may still contain framing bias because editorial judgement naturally leads to human-induced biases. In addition, we also crawl the title triplets  $\{T_L, T_R, T_C\}$  and the neutral issue title  $T_{neu}$  that are later used in our modeling.
To make the dataset richer, we also crawled other meta-information such as date, topic tags, and media name. In total, we crawled 3,564 triplets (10,692 articles). We use  $2/3$  of the triplets, which is 2,276, to be our training and validation set  $(80:20\text{ratio})$ , and the remaining 1,188 triples as our test set. We will publicly release this dataset for future research use.
4 Analysis of Framing Bias
The literature on media framing bias from the NLP community and social science studies provide the definition and types of framing bias (Goffman, 1974; Entman, 1993; Gentzkow et al., 2015; Fan et al., 2019) — Informational framing bias is the biased selection of information (tangential or speculative information) to sway the minds of readers. Lexical framing bias is a sensational writing style or linguistic attributes that may mislead readers. However, the definition is not enough to understand exactly how framing bias manifests in real examples such as, in our case, the ALLSIDES dataset. We conduct a case-study to obtain concrete insights to guide our design choices for defining the metrics and methodology.
4.1 Case-Study Observations
First, we identify and share the examples of framing bias in accordance with the literature (Table 1).
Informational Bias This bias exists dominantly in the form of "extra information" on top of the salient key information about an issue that changes the overall impression of it. For example, in Table 1, when reporting about the hold put on mil
4The full list is provided in the appendix.
In some cases, Allsides does not provide all three reportings, and such cases are filtered out.
Issue A: Trump Put Hold On Military Aid To Ukraine Days Before Call To Ukrainian President
Left: Trump ordered hold on military aid days before calling Ukrainian president, officials say
Right: Trump administration claims Ukraine aid was stalled over corruption concerns, decrees media ‘frenzy'
Center: Trump Put Hold on Military Aid Ahead of Phone Call With Ukraine’s President
Issue B: Michael Reinoehl appeared to target right-wing demonstrator before fatal shooting in Portland, police say
Left: Suspect in killing of right-wing procter fatally shot during arrest
Right: Portland’s Antifa-supporting gunman appeared to target victim, police say
Center: Suspect in Patriot Prayer Shooting Killed by Police
Issue C: Trump Says the ‘Fake News Media’ Are ‘the true Enemy of the People'
Left: President Trump renews attacks on press as ‘true enemy of the people’ even as CNN receives another suspected bomb
Right: ‘Great Anger’ in America caused by ‘fake news’ — Trump rips media for biased reports'
Center: Trump blames ‘fake news’ for country’s anger : ‘the true enemy of the people’
Table 1: Illustration of differences in framing from Left/Right/Center media with examples from ALL-SIDES dataset. We use titles for the analysis of bias, since they are simpler to compare and are representative of the framing bias that exists in the article.
itary aid to Ukraine (Issue A), the right-leaning media reports the speculative claim that there were "corruption concerns" and tangential information "decries media 'frenzy'" that amplifies the negative impression of the issue. Sometimes, media with different political leanings report additional information to convey a completely different focus on the issue. For Issue C, left-leaning media implies that Trump's statement about fake news has led to "CNN receiving another suspected bomb", whereas right-leaning media implies that the media is at fault by producing "biased reports".
Lexical Bias This bias exists mainly as biased word choices that change the nuance of the information that is being delivered. For example, in Issue B, we can clearly observe that two media outlets change the framing of the issue by using different terms "suspect" and "gunman" to refer to the shooter, and "protester" and "victim" to refer to the person shot. Also, in Issue A, when one media outlet uses "(ordered) hold", another media uses "stalled", which has a more negative connotation.
4.2 Main Insights from Case-Study
Next, we share important insights from the case study observation that guide our metric and model design.
Relative Polarity Polarity is one of the commonly used attributes in identifying and analyzing framing bias (Fan et al., 2019; Recasens et al., 2013). Although informational and lexical bias is conceptually different, both are closely associated with polarity changes of concept, i.e., positive or negative, to induce strongly divergent emotional responses from the readers (Hamborg et al., 2019b).
Thus, polarity can serve as a good indicator of framing bias.
However, we observe that the polarity of text must be utilized with care in the context of framing bias. It is the relative polarity that is meaningful to indicate the framing bias, not the absolute polarity. To elaborate, if the news issue itself is about tragic events such as "Terror Attack in Pakistan" or "Drone Strike That Killed 10 people", then the polarity of neutral reporting will also be negative.
Indicator of Framing We discover that the news title is very representative of the framing bias that exist in the associated articles – this makes sense because the title can be viewed as a succinct overview of the content that follows<sup>6</sup>. For instance, in Table 3 the source input example, the right-leaning media's title, and article are mildly mocking of the “desperate” democrats' failed attempts to take down President Trump. In contrast, the left-leaning media's title and an article show a completely different frame – implying that many investigations are happening and there is “possible obstruction of justice, public corruption, and other abuses of power.”
5 Metric
We use three metrics to evaluate summaries from different dimensions. For framing bias, we propose a polarity-based metric based on the careful design choices detailed in §5.1. For evaluating whether the summaries retain salient information, we adopt commonly used information recall metrics (§5.2). In addition, we use a hallucination metric to evaluate if the generations contain any unfaithful hallucination.
There can be exceptions for the non-main-stream media that use clickbait titles.
cinatory information because the existence of such hallucinatory generations can make the summary fake news (§5.3).
5.1 Framing Bias Metric
5.1.1 Design Consideration
Our framing bias metric is developed upon the insights we obtained from our case study in  $\S 4$ .
First of all, we propose to build our metric based on the fact that framing bias is closely associated with polarity. Both model-based and lexicon-based polarity detection approaches are options for our work, and we leverage the latter for the following reasons: 1) There is increasing demand for interpretability in the field of NLP (Belinkov et al., 2020; Sarker et al., 2019), and the lexicon-based approach is more interpretable (provides token-level human interpretable annotation) compared to black-box neural models. 2) In the context of framing bias, distinguishing the subtle nuance of words between synonyms is crucial (e.g., dead vs. murdered). The lexicon-resource provides such token-level fine-grained scores and annotations, making it useful for our purpose.
Metric calibration is the second design consideration, and is motivated by our insight into the relativity of framing bias. The absolute polarity of the token itself does not necessarily indicate framing bias (i.e., the word "riot" has negative sentiment but does not always indicate bias), so it is essential to measure the relative degree of polarity. Therefore, calibration of the metric in reference to the neutral target is important. Any tokens existing in the neutral target will be ignored in bias measurement for the generated neutral summary. For instance, if "riot" exists in the neutral target, it will not be counted in bias measurement through calibration.
5.1.2 Framing Bias Metric Details
For our metric, we leverage Valence-Arousal-Dominance (VAD) (Mohammad, 2018) dataset which has a large list of lexicons annotated for valence  $(v)$ , arousal  $(a)$  and dominance  $(d)$  scores. Valence, arousal, and dominance represent the direction of polarity (positive, negative), the strength of the polarity (active, passive), and the level of control (powerful, weak), respectively.
Given the neutral summary generated from the model  $\hat{A}_{neu}$ , our metric is calculated using the VAD lexicons in the following way:
1. Filter out all the tokens that appear in neutral target  $A_{neu}$  to obtain set of tokens unique to  $\hat{A}_{neu}$ . This ensures that we are measuring the relative polarity of  $\hat{A}_{neu}$  in reference to the neutral target  $A_{neu}$  - results in calibration effect.
2. Select tokens with either positive valence  $(v > 0.65)$  or negative valence  $(v < 0.35)$  to eliminate neutral words (i.e., stopwords and non-emotion-provoking words) – this step excludes tokens that are unlikely to be associated with framing bias from the metric calculation.
3. Sum the arousal scores for the identified positive and negative tokens from Step 2 - positive arousal score  $(Arousal_{+})$  and negative arousal score  $(Arousal_{-})$ . We intentionally separate the positive and negative scores for finer-grained interpretation. We also have the combined arousal score  $(Arousal_{sum} = Arousal_{+} + Arousal_{-})$  for a coarse view.
4. Repeat for all  $\{A_{neu},\hat{A}_{neu}\}$  pairs in the testset, and calculate the average scores to use as the final metric. We report these scores in our experimental results section (§7).
In essence, our metric approximates the existence of framing bias by quantifying how intensely aroused and sensational the generated summary is in reference to the target neutral reference. We publicly release our metric code for easy use by other researchers<sup>7</sup>.
5.1.3 Human Evaluation
To ensure the quality of our metric, we evaluate the correlation between our framing bias metric and human judgement. We conduct A/B testing<sup>8</sup> where the annotators are given two generated articles about an issue, one with a higher  $Arousal_{sum}$  score and the other with a lower score. Then, annotators are asked to select the more biased article summary. When asking which article is more “biased”, we adopt the question presented by Spinde et al. We also provide examples and the definition of framing bias for a better understanding of the task. We obtain three annotations each for 50 samples and select those with the majority of votes.
A critical challenge of this evaluation is in controlling the potential involvement of the annotators'
<sup>7</sup>https://github.com/HLTCHKUST/framing-bias-metric <sup>8</sup>Please refer the appendix for more detail of the A/B testing
personal political bias. Although it is hard to eliminate such bias completely, we attempt to avoid it by collecting annotations from those indifferent to the issues in the test set. Specifically, given that our test set mainly covers US politics, we restrict the nationality of annotators to non-US nationals who view themselves bias-free towards any US political parties.
After obtaining the human annotations from A/B testing, we also obtain automatic annotation based on the proposed framing bias metric score, where the article with a higher  $Arousal_{sum}$  is chosen to be the more biased generation. The Spearman correlation coefficient between human-based and metric-based annotations is 0.63615 with a p-value  $< 0.001$ , and the agreement percentage  $80\%$ . These values indicate that the association between the two annotations is statistically significant, suggesting that our metric provides a good approximation of the existence of framing bias.
5.2 Salient Info
The generation needs to retain essential/important information while reducing the framing bias. Thus, we also report ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002) between the generated neutral summary,  $\hat{A}_{neu}$ , and human-written summary,  $A_{neu}$ . Note that ROUGE measures the recall (i.e., how often the n-grams in the human reference text appear in the machine-generated text) and BLEU measures the precision (i.e., how often the n-grams in the machine-generated text appear in the human reference text). The higher the BLEU and ROUGE1-R score, the better the essential information converges. In our results, we only report Rouge-1, but Rouge-2 and Rouge-L can be found in the appendix.
5.3 Hallucination Metric
Recent studies have shown that neural sequence models can suffer from the hallucination of additional content not supported by the input (Reiter, 2018; Wiseman et al., 2017; Nie et al., 2019; Maynez et al., 2020; Pagnoni et al., 2021; Ji et al., 2022), consequently adding factual inaccuracy to the generation of NLG models. Although not directly related to the goal of NEUS, we evaluate the hallucination level of the generations in our work. We choose a hallucination metric called FeQA (Durmus et al., 2020) because it is one of the publicly available metrics known to have a high correlation with human faithfulness scores. This
is a question-answering-based metric built on the assumption that the same answers will be derived from hallucination-free generation and the source document when asked the same questions.
6 Models and Experiments<sup>9</sup>