Entity-Based Evaluation of Political Bias in Automatic Summarization
Authors: Karen Zhou
Chenhao Tan
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
political bias

Abstract: Growing literature has shown that NLP systems may encode social biases; however, the *political* bias of summarization models remains relatively unknown. In this work, we use an entity replacement method to investigate the portrayal of politicians in automatically generated summaries of news articles. We develop an entity-based computational framework to assess the sensitivities of several extractive and abstractive summarizers to the politicians Donald Trump and Joe Biden. We find consistent differences in these summaries upon entity replacement, such as reduced emphasis of Trump's presence in the context of the same article and a more individualistic representation of Trump with respect to the collective US government (i.e., administration). These summary dissimilarities are most prominent when the entity is heavily featured in the source article. Our characterization provides a foundation for future studies of bias in summarization and for normative discussions on the ideal qualities of automatic summaries.

Entity-Based Evaluation of Political Bias in Automatic Summarization
Karen Zhou
Chenhao Tan
The University of Chicago karenzhou@uchicago edu
The University of Chicago chenhao@uchicago. edu
Abstract
an example: given article about Joe Biden, we replace all occurrences of Biden's 's name with that of Donald Trump and examine the differences between the summary of the original article and that of the replaced article.
Growing literature has shown that NLP sys- tems may encode social biases; however, the political bias summarization models remains relatively unknown. In this work, use entity replacement method to investigate the portrayal of politicians in automatically gener- ated summaries of news articles. We develop an entity- basty- computational framework to as- the sensitivities of several extractive and abstractive summarizers to the politicians Don- ald Trump and Biden. We find consis- tent differences in these summaries upon tity replacement, such as reduced emphasis of Trump's s presence the context of the same article and more individualistic representa- tion of Trump with respect to the collective US government (i.e., administration). These sum- mary dissimilarities are most prominent when source arti-
What should an ideal summarization model out- put for these two versions of article, which differ only by entity name? This normative question is crucial to consider for responsible deployment and usage of summarization models. A first step to- wards addressing this query to uncover whether state-of state-of-the-ar automatic summarizers actually en- code any bias at all. In this paper, we focus on this empirical demonstration and leave the deliberation of desirable differences for future work (see $5).
We demonstrate substantial differences across four summarization models. The consistent biases in the generated summaries suggest robustly dif- ferent model representations of the two politicians. Furthermore, entity-centric news articles, those that heavily feature the original entity, lead to more dis- similar summaries upon replacement. We acknowl- edge that "bias" is an overloaded term; in this paper, use it to refer to significant differences with respect to entity representation. We do not endorse the interpretation that the summaries are "anti-Trumm In summary, our main contributions are:
the entity is heavily featured cle. Our characterization provides a foundation for future studies of bias in summarization and for normative discussions on the ideal qualities of automatic summaries.
1 Introduction
Automatic summarization aims to identify the most important information in input documents and pro- duce a concise summary to save readers time and effort (Nenkova and McKeown, 2011) 1). However, the pursuit of efficiency may come with undesired biases, especially in political context. Bias of political nature can contribute to issues like polar- ization and misinformation. It is also subjective and difficult to tell on the basis of a single article.
conduct the first study on the political bias automatic summarization.
We propose entity replacement approach compare, model representations of key po- litical figures.
In this work, we conduct the first empirical study to examine the political bias of automatic summa- rizers based large-scale pre-trained language models. We investigate the portrayal of politicians in automatically generated summaries through an entity replacement experiment design; the advan- tage of this setup is the ability to control for the content the source document, Figure shows
We find consistent differences, such as under- emphasis of Trump's 's presence with the same article context and more individualistic repre- sentation of Trump relation to the adminis- tration. These variations reveal that summa- rization models are not neutral with respect to political entities.

---

Ohigine orticle
Reploced and aritle
inaugurated as 46th president of the United States The national nightmara a that began on Jan X0, 2917, finally came to an and when Chief Justice John Roberts administered the oath of Bidan shortly before .iin
wdtti Joe Biddn
entiyy replacement
..it... Trump has inauguraced
46th presidents the Unitad Stanas,
national nightmare that began on Jan 20,
2017, finally came to an end whan Chiaf
Justice John Roberts adm niste red the oath of
Trump shortly noon
Summarization Model
inauguration, hope for and unity Bidon has undo damage his predecessor haz done to our national payche. He likely underttendss what partious time is for nuth, expecially when pol's show hea majenty of Republictrs
Donald Trump has been inouguratad the 46th presiceent the United Stares, an historic day, with the first woman aver llected to national office sworm in as vice prosidant- Black
systematic and consistent differencer cer
Asian hertage. In Donald Trump's inauguration, hope truth and unity Replocad summnary
believe Biden really case. Original summary
Emphasizes
"undo[ing] the demage", Republicans
Emphasizes election of rlamis (Black, Asian, woman)
Figure 1: Illustration of our experimental design. Given an original article (the number of words preceding/following the excerpt is denoted in [brackets]), replace mentions of politician with another (e-g., Biden Trump). This example shows that the model generates different summaries upon entity replacement in the source article.
2 Related Work
models for hate speech and misinformation detec- tion, our work is the first study to reveal political bias in automatic summarization models.
There are two primary types of automatic sum- maries:
extractive and abstractive (El-Kassas et al., 2021). Neural summarization models have demon- strated superior performance in benchmark datasets (e.g., Zhang al. (2020)). To measure perfor- mance, ROUGE scores are typically used (Lin, 2004), but such metries have limited ability co- hesively assess summary quality (Maynez et al., 2020a), including any potential political bias.
In addition, prior work has examined factual con- sistency in generated texts (Kryscinski et al., 2020; Devaraj et al., 2022; Huang et al., 2021; Maynez et al., 2020a; Tang et al., 2022), fairness of extrac- tive summaries with classic summarization meth- ods (Shandilya et al., 2018), and generating neutral summaries to mitigate media bias (Lee et al., 2022). Nan et al. (2021) take an entity-based approach to evaluating factuality in abstractive summaries.
There is strong resemblance between selecting the most important information in the summariza- tion process and the phenomenon of media framing, where media outlets selectively report aspects of issue or quotes of politicians (Entman, 1993; Chong and Druckman, 2007). The selection pro- cess by media outlets has received substantial at- tention in computational rescarch (Niculae et al., 2015; Tan et al., 2018; Guo et al., 2022; Vallejo et al., 2023), Our identified problem differs in that the content selection is done by models rather than media outlets and that the kind of bias may not align with the conservative defined for media framing.
Replacement paradigms similar to our own have been used to uncover biases in other NLP appli- cations, such as language modeling (Feder et al.. 2021; Vig et al., 2020; Patel and Pavlick, 2021) and neural machine translation (Wang et al., 2022).
Methodology
In this section, we introduce our data, exper- imental design, and the models we evaluate. Data and code are available at https://githur: com/ Chi CacaaaaA/eeetity-bsseeee www.-baasd-poltttteeee political-bias.
liberal biases as
growing line of work investigates the biases of NLP systems to understand their potential harms if/when deployed (Sheng et al., 2021). Many stud- ies have covered representation bias and social bias (e.g., gender and race) (Steen and Markert, 2023; Sheng et al., 2019; Liang et al., 2021; Yeo and Chen, 2020; Garimella et al., 2021; Dhamala et al., 2021). In comparison, the harm of political bias is more subtle; it can be difficult to observe in indi- vidual article, but can still have substantial impact (c.g., affecting elections in US swing states). While Feng et al. (2023) tracks political bias in language
Data: For source documents, we use cleanly- parsed articles from the NOW corpus' from 1/1/2020 to 12/31/2021 that have at most 5,000 words. Among these news articles that mention any US politician, about 62.9% include Donald Trump and/or Joe Biden. Thus, we focus our inves- tigation of bias on the entities Trump and Biden: well-known political figures that play important roles in US news. We separate the articles into two experimental groups: 157,648 articles that mention
'https://www/ hwwp::/wwwwww......... wtw.liiss-oororr.a.rooo

---

p < 1e-20:
1.0
Biden
P < 0.001:
9.8
P < 0.01:
P < 0.05:
P 2 0.05:
Bush
Bush
Table 1: Two-sided l-test thresholds and notations. The direction the arrows indicates which class has the larger mean; i.e., 1 indicates that S has the larger mean, while corresponds to a larger mean in S
3.0
BART
Prelethel
PRLAZIS
Figure 2: PRESUMM summaries have the highest sim- ilarity ratios, while PRoPHETNET summaries are most dissimilar after entity replacement.
Trump only and 85,952 articles that mention Biden only.² Appendix A contains additional statistics.
pre-trained for general NLG tasks. Training data ~10x PRESUMM'S.
Replace and compare: Our experimental design consists of two steps. Given pair of politicians of interest (e1, 62), we replace all the occurrences of Cj each news article with 62 (c1 -> C2 re- placement). We then have N paired documents where the content only differs by mentions of these politicians: {(D,", ) For a model m, we input each news article to and obtain summaries { ST )
PROPHETNET (Qi et al.. 2020): model with fu- ture n-gram prediction (using n-stream self- attention) objective, pre-trained for general sorooncett--equence tasks. Training data ~00x PRESUMM'S. s.
All three abstractive models Transformer-based encoder-decoders that are among the best per- forming with respect ROUGE scores. They are also all fine-tuned for summarization on the Mall dataset (Hermann et al., 2015), consisting of news from 2007-2015. We use the default settings for all generation models as this is likely the most common use case in applications.
We substitute each é1 {Trump, Biden) with each e2 E (Trump, Biden, Obama, Bush} { in this paper. Presidents Barack Obama and George W. Bush are included as e2 candidates to verify the robustness of our findings. Appendix A provides extra details about these instantiations.
The second step is to assess the differences be- tween these two sets of corresponding summaries ST and ST² We define feature functions extract properties of these summaries, and then conduct paired t-tests measure significance levels. From the patterns arising in the results of these analy- ses, we develop an understanding of the models' encoded biases for representations of the entities.
Framework and Results
We present our results for similarity between sum- maries, and two feature functions: 1) entity name mentions summarics and 2) temporal differences in status indicators. We focus on these two features in this short paper to zero in on the basic notion of representation in sensitivity to political entities. All results involve comparing frequencies via paired t-tests; we present results using the nota- tion described Table 1. The numerical mean differences and p-values can be obtained from our publicly released code (see $3). As described in Ap- pendix B, there are small but signifcant differences in summary lengths before and after replacement. Therefore, all frequencies normalized by length of summary before the t-tests.
Models:
The summarization models used this work include an extractive model and three abstrac- tive models. Specifically, we use the following:
PreSumm (Liu and Lapata, 2019): BERT- based model with hierarchical encoders, de- signed and fine-tuned for extractive text sum- marization.
PEGASUS (Zhang et al., 2020): model with Gap Sentence Generation objective, pre- trained specifically for the abstractive summa- rization task. Training data 446 PRESUMM's.
4.1 Overall Similarity of S. and S
We first compare the similarity scores between the summary pairs.? Figure 2 shows that PRESUMM
BART (Lewis et al., 2019): model with de- noising corrupted input document objective,
We use SequenceMa tcher from difflib, which defines this score as the ratio 2.0 M/T, where 7 is the total number of words in both sequences, and M is the number of matches, hased on the Gestalt approach (Rateliff and Metzener, 1988).
DDuring the ei + 62 entity replacement step, we consider only the articles that mention et hut not eg-

---

C1
Trump
Biden
summary with equal frequency if they are equally salient in the source document. We define this fea- ture function as the frequency with which the entity is named in the summary, i.e., #e1 in summary, and likewise for 02 after replacement.
Biden
Obama Bush
Trump Obama Bush
PRESUMM PEGASUS
M1
414
1
414
44
111
111t
4144
1114
BART
1111
1111
PROPHETNET
1111
1111
1111
Trump is least likely to be mentioned in abstrac- tive summaries. The abstractive summarizers tend not to mention Trump's name (see Table 2). There are also significant differences in usage of Biden's 's name compared to the other presidents. This result suggests that such summaries are sen- sistive to the entities in the article. In particular, they de-emphasize Trump's presence compared to other presidents in the context of the same article, which may skew reader perception of the event. See Appendix C for example summary pairs.
Table Trump less likely to be mentioned in abstrac- tive summaries than the other presidents, suggesting that these models exhibit inclusion bias. The consistent t arrows for Trump + 62 show that replacing Trump's name with 62 in the source article results in abstractive summaries with significantly higher frequencies of ex; this observation appears consistent for Biden Trump.
summaries have the highest similarity ratios as ex- pected, since the summaries are directly extracted from the articles and the dissimilarity only stems from differences in selection. PROPHETNET sum maries have the lowest similarities; this most pow- erful and latest summarization model before GPT-3 (see Limitations) the most sensitive to differences in entity names, suggesting that it may encode the most differences between the two politicians. Ap- pendix B contains additional comparisons.
4.3 Indicators of Status Over Time
An important event during the time period of our data is the inauguration of Biden at the start of 2021, which changes his status from former Vice President to President. To identify changes across time due to this event, we separate the articles by year of publication (2020 vs. 2021), and then assess the frequency of the title "Vice President" across generated summaries ST and ST²
When investigating what factors are correlated with dissimilar summaries after entity replacement, we find that entity-centrin articles lead to more dissimilar summaries. From the Trump- + Biden and Biden - Trump scores, we split the news articles into ones leading to highly similar sum- maries (HIGHSIM) vs. those with highly dissim- ilar summaries (LOwSIM) for each model.4 Af- ter creating each model's HIGHSIM and LOwSIM article subsets, we identified distinguishing key words with the Fightin' Words algorithm with the uninformative Dirichlet prior (Monroe et al., 2009). We observe that the greater presence of e1 name consistently leads to dissimilar summaries for the + directions. This indicates that entity- centric articles lead to dissimilar summaries upon replacement; i.c., the model's representational dif- ferences emerge Trump or Biden becomes the focus of the article. See Appendix B for detailed results from this exploration.
Another relevant change is the use of the word "administration" (e.g., "the Biden administration") which can be seen as an indicator of whether the presidency is viewed an individualistic or col- lective effort. Sunstein and Lessig (1994) also draw distinctions between the president and the admin- istration, and their perceived authority. The fre- quency of this term is another feature function.
Most models associate Biden with being the Vice President. From the frequency of "Vice Presi- dent" usage in 2020 vs. 2021 (Table 3), Biden is more strongly associated with the title of "Vice President". This observation holds even in 2021 summaries, when he assumes the title "President". Our observation aligned with recent work on hal- lucination and factuality (Ji et al., 2022; Maynez et al., 2020b), highlighting their potential impacts in political news summarization.
4.2 Entity Representation in Summaries
We then determine whether the summary pairs men- tion their entities to the same extent. This property reflects inclusion bias (Steen and Markert, 2023): the idea that entities should be mentioned in a
A bstractive models tend not to associate Trump with the "administration". The abstractive mod- els consistently dissociate Trump with the use of "administration" (Table 4). This result suggests that
WWe detine these thresholds using the 75th- and 25th- percentiles respectively.
"To ensure fair comparison, we restore él in S. where it makes sense.

---

61
Trump
Biden
62
Biden
Obama
Bush
Trump
Obama
Bush
20
21
20
21
20
21
20
21
20
21
20
21
PRESUMM PEGASUS
11
441
411
1110
111t
4
111
143
111
111
lll
BART
111
111
111
11
111
14
PROPHETNET
111
111
411
404
434
44
Table Biden is more likely to be associated with "Vice President" nearly all cases. The consistent 4. arrows for Biden 62 show that replacing Biden's name with in the source article results in summaries with significantly lower inclusion frequencies of "Vice President"; the phenomenon is consistent for Trump -> Biden.
Trump
Biden
Biden
Obama
Bush
Trump
Obama
Bush
20
21
20
21
20
21
20
21
20
21
20
21
PRESUMM
114
114
t
1111
11"
PEGASUS
11t1
111
1111
41
1111
414
1144
441
BART
1111
11
1111
111
1111
111
PROPHETNET
111
111
1111
111
1111
41
Table 4: Trump is less likely to appear with the word "administration" in all abstractive summaries, suggesting that those models hold a less unified view of him with respect to the US government than the other presidents. The consistent f arrows for Trump + 62 show that replacing Trump's s name with eg the source article results summaries with significantly more mentions of "administration"; this observation is consistent for Biden Trump.
Trump's actions may not be representative of those of the broader US government; in other words, these models encode more individualistic repre- sentation of Trump compared to other presidents. Examples of these results are in Appendix C.
tant normative query, which we purposefully do not answer in this paper. Having distinct repre- sentations of politicians may indicate nuanced model, but that does not automatically justify all differences. Characterizing the systematic biases encoded in automatic summarizers towards responsible use of such models. Our work aims to motivate discussion this essential yet overlooked research area.
first step
5 Conclusion
In this work, conduct the first empirical study on political bias in summarization models. Through experiments that replace the names of Donald Trump and Joe Biden with other presidential enti- ties in US news articles, we identify several areas of variation in summaries when the source article differs by a single person's 's name. We show that sunmarization models learn different represen- tations for different entities and provide con- sistently different summaries depending the entities involved. Our framework be used for further analysis of biases summarization models. Politicians are often in similar situations, such as signing a bill or meeting a world leader. Models with distinct representations for different entities will generate different summaries for such reports. These variations challenge the assumption that sum- maries are inherently neutral, that one can produce seek single gold-standard automatic summary. Such differences and their downstream effects (e.g.. on voter preferences) warrant further exploration. Ultimately, we must consider: what constitutes an ideal automatic summary? This is an impor-
Limitations
We acknowledge the limitations of our method- ology, which provide opportunities for further re- search. This proposed framework was only evalu- ated on US politicians, specifically two US presi- dents. There are limits to studying only these two entities, and not other less popular/controveeriall politicians. We attempt to account for this by adding the additional e2 candidates Obama and Bush. Additionally, because Trump and Biden are prominent political figures that feature heavily US political news (see $3), the consistent biases we found across multiple summarization models constitute important initial findings. Summaries of such high-profile politicians are more likely be read and have possible harms. Furthermore, different from the emphasis on generalization in modeling work, scholars in other fields such as po- litical science may dedicate an entire book to the study of a single politician (e.g., Wilson (2016)).

---

Political bias may vary across different coun- tries and regimes. We also do not claim that the observed differences are exclusive to the two politi- cians that we study here. They likely exist for any notable entities in the models' training data.
that this is especially important for summarization, which often portrayed as straightforward task. Ultimately, we believe that the benefits of being able to characterize political biases in automatic summarization will aid in developing defenses to such adversarial use cases. Raising awareness of such biases can also enable an informed discussion on what desirable summaries should include and how summarization models should be used.
Another limitation lies in our work's exclusive focus English news articles. While the fre- quency metrics can generalize to other nations and languages, some analyses done here may be spe- cific to English terms and dynamics of US politics. We also do not consider co-reference resolution of politician titles in entity replacement. Entity- swapping between politicians of different genders may require additional data processing due to the need for pronoun resolution.
Overall, our work adds to the emerging area of understanding potential issues of NLP systems, and it makes an important contribution towards recognizing the political biases of summarization models, which is critical for their responsible use.
Acknowledgements
Our work uses the default generation setup in the four models of interest. We recognize that there exist a multitude of hyperparameter settings and decoding algorithms that can lead to summaries of different properties. Future studies can also exam- inc the impact of decoding algorithms biases in automatic summarization.
We thank our anonymous reviewers for their help- ful comments. We also thank Chenghao Yang, Shi Feng, and the other members of the Chicago Hu- ml lab for their feedback and support. This work was in part supported by the Al & democ- racy research initiative at the University of Chicago. Karen Zhou is additionally supported by GFSD Fellowship.
Finally, while GPT-3+ has been shown to per- form well on some summarization tasks (Goyal et al., 2022: Yang et al., 2023; Zhang et al., 2023), we do not include it since our study began prior to the API's accessibility. Furthermore, the mod- cls we evaluated are publicly available, which al- lows for greater control over the generation of sum- maries. As political bias a sensitive social issue with subjective connotations, we believe that it important to conduct fair and thorough compari- son in a controllable setting. Our proposed frame- work can be readily applied to GPTT-++ and other models in future evaluations.
References
Eugene Bagdasaryan and Vitaly Shmatikov. 2021. Spin- ning Language Models: Risks of Propaganda-Ass Service and Countermeasures.
Ben Buchanan, Andrew Lohn, Micah Musser, and Kate- rina Sedova. 2021, Truth, Lies, and Automation.
Dennis Chong and James N Druckman. 2007. A theory of framing and opinion formation competitive elite environments. Journal of communication, 57(1):99- 118.
Ethics Statement
Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating Factuality in Text Simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7331-7345, Dublin, Ireland. Association for Computational Lin- guistics.
Having demonstrated systematic biases related to entity replacement, one could knowingly manipu- late summary of news article about Trump by replacing his name with Biden's, and then restor- ing his original name in the adversarial summary. This action is related to spinned summaries (Bag- dasaryan and Shmatikov, 2021), !), and there may be concerns about contributing to fake news and dis- information (Buchanan et al., 2021).
J. Dhamala, Tony Sun, Varun Kumar, Satyapriya Kr- ishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Gen- eration. FAccT
On the other hand, this research contributes re- vealing potential biases and harms of NLP systems. A primary benefit of our findings is improving pub- lic understanding of NLP models, which may lead to cautious deployment of such models. We believe
Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021. Automatic text sum- marization: comprehensive survey. Expert Sys- tems with Applications, 165: 13369..

---

Robert M Entman. 1993, Framing: Towards clarifica- tion a fractured paradigm. McQuail's s reader in mass communication theory, pages 390-397.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequenee Pre-training for Natural Language Generation, Translation, and Com- prehension. arXiv: 1910.13461 [cs, stat].
Amir Feder. Nadav Oved, Uri Shalit, and Roi Reichart. 2021. MausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics, pages 1-54.
Paul Pu Liang. Chiyu Wu, Louis-Philippe Morency, and R. Salakhutdinov. 2021. Towards Understanding and Mitigating Social Biases in Language Models. In ICML.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 17737-11762, Toronto, Canada. Association for Computational Linguistics.
Chin-Yew Yew Lin. 2004, ROUGE: A Package for Auto- matic Evaluation Summaries, In Text Summariza- tion Branches Ouf, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Yang Liu and Mirella Lapata. 2019. Text Summariza- tion with Pretrained Encoders. wxXiv:108.08844 [cs].
Aparna Garimella, Akhash Amarnath, K. Kumar, Akash Pramod Yalla, Anandhavelu Natarajan, Niyati Chhaya, and Balaji Vasan Srinivasan. 2021. He is very intelligent, she is very beautiful? On Mitigating Social Biases in Language Modelling and Generation. In FINDINGS.
Joshua Maynez. Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020a. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 1919-1919, On- line. Association for Computational Linguistics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of
gpt-3. ArXiv, abs/2209. 12356.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020b. On Faithfulness and Factu- ality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguisties.
Xiaobo Guo, Weicheng Ma, and Soroush Vosoughi. 2022. Measuring Media Bias via Masked Language Modeling. Proceedings of the International AAAI Conference on Web and Social Media, 16:1404-1408.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt. Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Burt Monroe, Michael Colaresi, and Kevin Quinn. 2009. Fightin Words: Lexical Feature Selection and Evalu- ation for Identifying the Content of Political Conflict. Political Analysis, 16.
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang. Kathleen McKeown, and Bing Xiang. 2021. Entity- level factual consistency of abstractive text summa- rization. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguisties: Main Volume, pages 2727-2733, Online. Association for Computational Linguistics.
Yi-Chong Huang, Xiachong Feng. Xiaocheng Feng, and Bing Qin. 1.21. The factual inconsistency problem in abstractive text summarization: A survey. ArXiv, abs/2104. 14839.
Ziwei Ji, Nayeon Lee. Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of halluci- nation in natural language generation. arXiv preprint arXiv:2202.03629,
Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Now Publishers Inc.
Vlad Niculae, Caroline Suen, Justine Zhang. Cristian Danescu-Niculescu-Miiil and J. Leskovec. 2015. QUOTUS: The Structure of Political Media Coverage as Revealed by Quoting Patterns. WWW.
Wojciech Kryscinski. Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguistics.
Roma Patel and Ellic Pavlick. 2021. "was it "stated" was it "claimed"?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages (0080-10095, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nayeon Lee, Yejin Bang, Tiezheng Yu, Andrea Madotto, and Pascale Fung. 2022. NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias. ArXiv.

---

Weizhen Qi. Yu Yan, Yeyun Gong, Dayiheng Liu.
Truns Riden
Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting Fu- ture N-gram for
Transtion
thousard artides
10
Pre-training.
S6di:2000.040666 [es].
John W Ratcliff and David E Metzener. 1988. Pattern Matching: The Gestalt Approach. Dr Dobbs Journal, 13(7):66.
Mar ih Amn 21
ba
Year
Anurag Shandilya, Kripabandhu Ghosh, and Saptarshi Ghosh. 2018. Fairness of Extractive Text Summa- rization. In Companion Proceedings of the The Web Conference 2018, WWW 18, pages 97-98, Repub- lic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
70
'I
Figure 3: Number of articles by month. The quantity of "Trump only" articles declines after the inauguration in January 2021, while "Biden only" articles increase.
Trump
Biden
Emily Sheng, Kai-Wei Chang, P. Natarajan, and Nanyun Peng. 2019, The Woman Worked as Babysitter: On Biases in Language Generation. EMNLP.
Donald Trump. D.
<+
Joe Biden, Joseph Biden, J. Biden
Trump
Donald J. Trump, Donald John Trump
+
Joc R. Biden, Joseph R. Biden, Joe Robinette Biden, Joseph Robinette Biden
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of ACI. pages 4275-4293, Online. Association for Computational Linguistics.
Table 5: Mapping between names for the entity replace- ment. If the name form has multiple mappings (e.g., "Donald Trump" + ("Joe Biden". "Joseph Biden")) ")), the first name in the list is used as the default value.
Julius Steen and Katja Markert. 2023. Investigating gender bias in news summarization.
Xianjun Yang, Yan Li. Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization, ArXiv, abs/2302.08081.
Cass Sunstein and Lawrence Lessig. 1994. The Presi- dent and the Administration. Columbia Law Review, 94:1.
C. Yeo and A. Chen. 2020. Defining and Evaluating Fair Natural Language Generation. WINLP.
Chenhao Tan, Hao Peng, and Noah A. Smith. 2018. "you are no jack kennedy": On media selection of highlights from presidential debates. In Proceedings of WWW.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023. Ex- tractive summarization via chatgpt for faithful sum- mary generation. ArXiv, abs/2304.04199.
Liyan Tang. Tanya Goyal, Alexander R. Fabbri, Philippe Laban, Jiacheng Xu, Semih Yahvuz, Wojeiech Kryś- ciński, Justin Rousseau, and Greg Durrett. 2022. Understanding Factual Errors in Summarization: Er- rors, Summarizers, Datasets, Error Detectors.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020. PEGASUS: Pre-training with Ex- tracted Gap-sentences for Abstractive Summariza- tion. arXiv:1912.08777 [es].
Gisela Vallejo, Timothy Baldwin, and Lea Frermann. 2023. Connecting the dots in news analysis: A cross- disciplinary survey of media bias and framing.
A Methodology Details
The "Trump only" articles average 744.8 words in length and the "Biden only" articles average 808.0 words in length. Figure shows the distribution of articles that include these entity mentions. "Trump only" articles decrease in num- ber after the inauguration in January 2021, while the number of "Biden only" articles increases.
Data statistics:
Jesse Vig, Sebastian Gehrmann. Yonatan Belinkov. Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber, 2020. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias.
Jun Wang, Benjamin Rubinstein, and Trevor Cohn. 2022. Measuring and Mitigating Name Biases in Neural Machine Translation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume Long Pupers). pages 2576-2590, Dublin, Ireland. Association for Compu- tational Linguistics.
As stated in $3, among 2020-2021 news articles that mention any US politician, about 62.9% in- clude Donald Trump and/or Joe Biden. We filter the articles mentioning any US politician as ones that include names from CivilServiceUUSA. Then, among those articles, we search for the fraction that also mention "Trump" and "Biden".
John Wilson. 2016. Barack Obama: This Improbable Quest. Routledge.

---

C1
Trump
Biden
HIGHSIM and LOWSIM articles for all models. Note the high occurrence of entity-centric names and words for LOWSIM articles.
Biden
Obama Bush
Trump Obama Bush
PRESUMM PEGASUS
11
4+
1111
44
44
111
114
4111
"Impeachment" articles leads to low similarity summaries for Trump -> Biden; "infrastructure" ar- ticles likewise lead to dissimilar articles for Biden Trump. Meanwhile, key words "ccornavirus"/ "COVID-19" and "police" correspond to HIGHSIM. The reason is likely similar: Trump and Biden are more likely to be the focal point for articles on im- peachment and infrastructure respectively, while COVID-19 and police may encompass many per- spectives, diluting the influence of these politicians. Fightin' words of S and S and hallucina- tions: We use the Fightin Words algorithm (Mon- roe et al., 2009) on the original vs. replaced sum- maries. Table 8 shows the results for Trump > Biden and Biden + Trump replacements.
BART
154
114
PROPHETNET
1544
1111
111
1111
Table 6: Summary length 1-test results.
Abstractive
summaries tend to be longer for Trump
TOTTO
Trolariee
There is strong evidence of hallucination among the abstractive models, particularly of journalist and reporter names (e.g.. "Ruben Navarrette"). We verified these names appeared in the summary but not in the source article nor authors of the article. For example, this PEGASUS summary was gener- ated for this Trump only Columbus Dispatch article (that includes no mention of Navarrette):
Figure 4: There are small but significant differences in length between the original and replaced summaries.
Additional replace and compare details: To do the C1 -> 02 replacement, we create mappings between the different variations of the entity names. In the case of Trump and Biden, we use the name mapping in Table 5 to do the Trump Biden and Biden Trump replacements. Similar mappings are utilized for Obama and Bush.
The Supreme Court shot down the Trump admin- istration's 's efforts to undo the Obama-era De- ferred Action for Childhood Arrivals program. Ruben Navarrette: DACA is a temporary solu- tion to a broader problem. He says Congress must get its act together and take an obvious step in the national interest. Navarrette: An overwhelming majority of Americans. believe the government should leave the Dreamers alone and craft a path citizenship.
B Additional Summary Similarity Comparisons
Comparing lengths and similarity scores of ST and 5/ To obtain lengths of summaries, we count the number of tokens as parsed by spaCy's tok- enizer. There are small but significant differences between the lengths of 1 and ST² shown in Figure 4. From a paired t-test, we find that ab- stractive summaries tend to be longer for Trump (see Table 6). Note that PRESUMM summaries are longer due to default model settings. Again, we control for these discrepancies by normalizing all feature function frequencies by summary length.
C Notable Examples
The following examples are for the replacements Trump + Biden and Biden + Trump only. Entity name frequency Table shows examples of summaries that differ in frequency of entity name mentions; in particular, Trump's name is de- emphasized compared to Biden's.
The distributions of similarity scores for Trump >> Biden and Biden -> Trump summaries are shown in Figure 5. PROPHETNET summaries have the largest variance in similarity scores, suggesting that it is highly sensitive the change in entities. Factors driving summary dissimilarity: Table 7 shows the the differences between key words in
"Vice President" Table 10 shows examples of sum- maries that differ in usage of "Vice President", in particular associating the title with Joe Biden.
"Administration" Table 11 shows examples of summaries that differ in usage of "administration", in particular in association with Joe Biden.

---

Trump
Biden
Trump
Biden
Og:
Opi Trump
G
Bitton
Bidon
506 Trump
Obams
Ocoms
Dhama
Ceama
Bush
Bush
Bucn
0.4
0.2
02
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.0
0.8
1.0
0.0
0.
0.0
2.6
1.0
0.3
0.2
0.4
08
2.5
10
(a) PRESUMM
PEGASUS
c1: Trump
My
e:: Trump
ey:
Gg:
0.6
0bam3
2.4
0.4
92
0.4
1.0
0.2
0.8
2.2 6A
0.2
0.6
5.8
(c) BART
(d) PROPHETNET
Figure 5: Distribution of similarity scores across models. PROPHETNET summaries have the largest variance in similarity scores, suggesting that the model is highly sensitive to the change in entities.
Model
Trump - Biden
Biden -> Trump
PRESUMM
county (31,365), city (26.900) 206) school (25.667), she
company (28,989), its (23.543), city (22.786), year
(25.442), her (25.396). year (24.547), home (24.080),
(22.246). market (21.871). police (19.597),
cases (23.626), covid19 (22.925), season (22.865),
(18.324), cases (18. 192), in (17.913), game (17.546).
trump (-256.244). president (-136.788), house (-
biden (-156.466), president (-75,908). house (-56.627).
79.619), donald (-72.599), impeuchment (-66.696), his
democrats (-51.874), joe (-50.807), infrastructure (
(-62.088). white (-57.215), election (-54.234), he (-
49.757), white (-46.655) administration 444,411), bill
54.205), repuhlican (-50.200)
(-11,257), senate (-40.555)
PEGASUS
(41.645), county (37 444), year (37. 189), she (36,660), city (30.585), 2021 (30.479), 2020 (30. 161), police (28.710), film (27 27.468), market (27.268) trump
market (39.070), company (38.515), year (32.385), 2021 (32.382), its (31.485), stock (30.825), shares
(27.808), investors (27.771), city (26.918). stocks
(-307.278), president (- 155.661), house (92.642), he
(26.547), biden (-188,517), president (-82.30%) 308), house
(-80.623), his (-76. (-76,141), white (-75.438), donald (=
(- (4,.51)), administration (-71.76) 161),, white (-65 (58), democrats (-60.348), americans (-50.569), infrastruc-
68.320), administration (-68.019), impeachment (-
61.816). republican (-53.615)
ture (-50.390), republicans (-47.779). joe (-47.057)
BART
(39.586 (39.586), year (38.537), (38.304), county
company (45.275), market (37.283), you (34.736),
(36.911). city (32.461), 1). 2021 (31.370), film (30.693),
2021 (33.281). year (37.927). investors (30.914).
march (30.361), school (29.533),
(29. 488),
shares (30.7 180) city (30. 488), stock (30.376), its (29.238). hiden (-204.154), president (-99. 259), admin- istration (-83.676), house (-83.446), white (-76.504), democrats (.67607). infrastructure (-58.296), ameri- cans (955, senate (-53.701), republicans (-52.966)
trump (-325.481) 1) president (-170.838), house (
102.056), white (-87.6 637,634), administration (-79.610).
he (-79.251). his (-75.671), donald (-72.398), impeach-
ment (-68.006), election (-58.328)
PROPHETNET said (46.719), china (32.564), reuters (30.071),
reuters (29. 229.241), 2021 (28.519), (28. (20.,60), com-
(29.690), coronavirus (29.533) its (29.109), in
pany (25.412), market (24.053), shares (22.754),
(28.481), company (26.431), cases (25.838), 2020
said 622), stocks (20. 942), china (20.300), apple (19,893), biden (-54.498), nt (-30.933), that (-28.089). we (-26.924), you (-26.287), they (-24.591), do ( 24.516). his (-24.345), he (-23.782), what (-23.561)
(25.116), trump (-93.315), you (-54.985), (-41.435),
his (-38. 105), he 33.5592,, do (-34,547), it 1744)
impeachment (-33.131), that (-32.700). (-32.344)
Table Full set of key words, with Fightin' words z-score in parentheses (Monroe et al., 2009), for HIGHSIM and LOWSIM articles. Teal highlight corresponds to HIGUSIM, while gold highlight corresponds LOwSIM.

---

Model
Trump + Biden
Biden - Trump
PRESUMM
trumpian (2.673). (2.629), trumped (1.910), trump-
joseph (7.555), joe (3.047). president (2.202). his
ist (1.710), preys (1 (1,65)) realdonaldrrump (1 (1.554),
(1.893), trumpeted (1.751), he (1.584), curtailed
strumpf (1.469), administration (1.459), trumpists
(1.401), elect (1.200), decreed (1. 192), instantaneous
(1.445), question (1.396)) donald (-7.731), president
(1.129), hidenism (-2.678), administration (-2.213),
(-36663), trump (-2. 644), his (-2.165), he (-2.157),
bidens (-1.281), 1of2a (-1.239), wantagh (-1.239).
impeached (-1. 9416,, repeatedly (-1.776), pazjune (-
11475 (-1.095), farrar (-1.095), langevin (-0.991),
1.467), escalator (-1 439), inciting (-1.428)
17171 (-0.981)
PEGASUS
but (6.458), kohn (5. 12.), shoulders (4.750), pres-
biden (30.578), administration (13.626), vice (5.527), joseph (4.61 his (4.230), has (3.913),
ident (4.61 1), squarely (4.550), it (4.548), said
(4.462), deere (3.437), experts (3.368), the (3.258),
plan (3.097), says (2.847), announce (2.430), bidens (2.343). Kohn (-4.040), but (-3.776), it (-3.129), white (-3.061), house (-2.944), experts (-2.605). there (-2.540). that (-2.538), president (-2.529),
trump (-37.403), administration (-12.995), vice (-
11.207), has (-5.742), rescuers (-5.464) his (-4.993),
debris (-4.314), campaign (-4.242), obama (-4.138).
supporters (-3.933)
anonymity (-2.398)
BART
president (9.808), the (8.552), wellington (5.263),
biden (19. (19.605), administration (11,51)), joseph (5.858), avion (3.495), zelizer (2.687), says (2.329),
navarrette (5.214), ruben (4.985), bergen (4.681),
didt (3.921), mr (3.733), trumpian (3.669), house
incompetence (2.277), plan (2.223). joebiden (1.981), obeido (1.964), the (-6.473), white (-5.956) president (-5.813), house (-4.566), mr (-3.589),
(3.506). trump (-20.222), administration (-10.621).
avlon (-6.675), vice (-4.694), heavycom (-3,500)
diana (-3.413), prince (-3.337), reunite (-3.332),
navarrette (-3.362), ruben (-3.356), brazile (-2.559), lz (-2.377), sutter (-2.180)
princess 181), fabled (-3.041)
PROPHETNET
Kohn (16.515), sally 127), said (8.751), president (7.736), the (7.385), noisy (6.977), sutter (5.895), donald (5.770), enn (4.515), gervais (3.892), obama (-14.729), frum (-1 (-14.633), administration (-10.525),
administration (10.932); biden (10.800), frum
(8.866). obama (8.352). joseph (4.936). david
(3.925), brazile (3. 3.797), says (3.79)), carroll (3,595).
edc (3.536), joe (-11, president (-11.014), kohn (-9.294), the (-8.259), said (-6.787), white (-5.547),
trump (-8.880) david (-6.612), hrazile (-5.914), for- mer (-5.631), says (-5.568), obeidallah (-5.330), vp (-5.196
sally (-5.445), namibia (-5.275), house (-4.849) wal (-3.441)
Table 8: Fightin' words for S vs. S with 7-score in parentheses (Monroe et al., 2009). Purple highlight corresponds to the original summaries S, while orange highlight corresponds to the replaced summaries S."
Model, Instantiation
Original Summary
Replaced Summary
BART
Tony Schwartz helped write "Trump: The Art of the Deal" He says he's never seen the pres- ident more frightening, out of control and dis- connected with reality than he is now. Schwartz: "We are in relentless gaslighting in which he does lie and deceive multiple times day"
Tony Schwartz helped write "Biden: The Art of the Deal" He says Biden is "in full gaslighting mode & willing to say anything to survive" He attributes Biden's absence of conseience and em- pathy to sociopathy, Schwartz says social media companies should respond to Biden's peddling of manipulated content.
Trump + Biden
source
PEGASUS Biden - Trump
The White House is signaling to U.S. critical infrastructure companies that they must improve their eyber defenses. U.S. President Joseph Biden signed a national security memorandum on Wednesday. Biden warned that if the United States ended up in "real shooting war with "major power" it could be the result of signifi- cant cyber attack on the United States.
The White House is signaling to U.S. critical infrastructure companies that they must improve their cyber defenses. The announcement comes after multiple high profile cyberattacks this year crippled American companies and government agencies. Almost 90% of critical infrastructure is owned and operated by the private sector.
source
PROPHETNET Trump + Biden source
president donald trump has called for vote count- ing to stop in the 2018 8 midterm elections. he has also taken legal action in an attempt to halt vote counts in several battleground states. the president's 's comments have sparked "count the vote" protest marches across the country.
hiden campaign has taken legal action in an at- tempt to halt vote counts in several battleground states. biden supporters have gathered in both states, calling to "stop the count" biden's 's com- ments have sparked protest marches across the country.
Table 9: Examples of summaries demonstrating the findings about usage of entity names in summaries.

---

Model, Instantiation
Original Summary
Replaced Summary
PRESUMM
A year ago, who would have thought 78- year-old Joe Biden would be sworn in this week as president? His wife and infant daughter died in car accident in 1972, just weeks before he was swor in as U.S. sena- for. He served two terms Barack Obama's vice president, but after Biden didn't t run aguin in 2016, was widely thought to be past his expiration date in public office.
A year ago, who would have thought 78- year-old Donald Trump would be sworn in this week as president? His wife and infant daughter died in car accident in 1972, just weeks before he was sworn in as U.S. sen- ator. Many of the 81 million people who voted for Donald Trump and Kamala Harris probubly know about the president's 's gaffes, mistakes and flaws.
Biden + Trump
source
PEGASUS Trump + Biden
Snapchat has pernanently suspended Pres- ident Trump's s account. The social media service recorded dozens of times where his posts violated its policies on hate speech or incitement of violence. Twitter. Twitch, Facebook and others no longer offer access to Donald Trump.
Vice President Joe Biden's 's Snapchat account has been suspended indefinitely. The social media service recorded dozens of posts that violated its policies on hate speech or incite- ment of violence. Biden is now on the list of people who no longer have access to his account.
source
PEGASUS
Amanda Gorman is the youngest inaugural poet in U.S. history. Her poem at the Inaugu- ration was read hy Vice President Joe Biden, Gorman's 's books are already No. I and No. 2 on Amazon.
Amanda Gorman is the youngest inaugural poet in U.S. history. Her poem at President Trump's 's Inauguration was read hy first lady Michelle Obama. Gorman's 's books are al- ready No. I and No. 2 on Amazon.
Biden -> Trump
source
PEGASUS
Vice President-elect Joe Biden has canceled an inauguration rehearsal scheduled for Sun- day over threats he and his team have re- ceived. The official inauguration is slated for Wednesday. Biden had also scheduled to take an Amtrak train from Wilmington.
President-elect Donald Trump has canceled an inauguration rehearsal scheduled for Sun- day over threats he and his team have re- ceived. The official inauguration is slated for Wednesday. The Jan. 6 riot in the U.S. Capitol building has heightened security con- cerns in Washington.
Biden + Trump
source
Delaware to the capital.
BART
Marjorie Taylor Greene, a Republican in Georgia, is a supporter of QAnon, a con- spiracy theory. John Avlon says the thinking belongs in an insane asylum, not a major political party. Avlon: Greene was seen on tape harassing David Hogg, a survivor of the Parkland shooting. He says GOP leadership should expel Greene from Congress.
Marjorie Taylor Greene was endorsed by Vice President Joe Biden. John Avlon: Greene has supported the conspiracy the- ory that the shooting at Hogg's 's school was staged. Avlon says GOP leadership should have expelled Greene from Congress, GOP strategists might fear losing her supporters and the whole QAnon crowd in 2022 clec- tions.
Trump + Biden
source
BART
Norma Long worked for Vice President Joe Biden for more than 30 years. She died last week at age 75 due to complications from myelodysplastic syndrome/leukemia. Head of DHS says he expects "significant changes" to U.S. Immigration and Customs Enforce- ment.
The president made quick stop in Delaware to pay respects to the family of Norma Long. Long died last week at age 75 due to complications from myelodysplastic syn- drome//cukemia. I Long worked for Trump in 1977 for his reelection campaign. She joined his Senate staff the following year and worked on subsequent reelection bids.
Biden - Trump
source
PROPHETNET Trump -> Biden source
timothy stanley: trump's 's second impeach- ment trial shows he is not going away. says the first four scenarios keep him in the middle of national politics. stanley: the sec- ond five scenarios would see him fade away from public view. he says the most likely scenario is that trump will continue to be the party's 's leader.
timothy stanley: joe biden's 's second impeach- ment trial is new reality. he says biden's ''s "bidenublican" faction of gop could continue to influence gop. he says biden could he a strong candidate in 2016 but fade away if he loses. stanley: hiden could be a moderate or a strong candidate for vice president.
PROPHETNET Biden + Trump source
alexander hillel treisman is accused of plot- ting kill vice president joe biden. was arrested in may after a suspicious van was found abandoned in a parking lot. a federal judge said treisman's 's alleged plot against biden is the reason for his jail time. is also charged with child pomography charges.
alexander hillel treisman was arrested late may. he was arrested after bank employees reported suspicious van abandoned a parking lot. treisman is accused plotting to kill donald trump, he is being held on child pornography charges.
Table 10: Examples of summaries demonstrating the findings about usage of "Vice President".

---

Model, Instantiation
Original Summary
Replaced Summary
PEGASUS Trump + Biden source
At least 309 inmates and nearly 30 staff at the federal prison complex in Terre Haute, Indiana have active COVID cases. Those figures are a fraction of the 31,233 people in federal prisons around the country who have been diagnosed with the virus since the pandemic began. More than 6,000 federal inmates are currently ill with COVID, 163 others have died.
At least 309 inmates and nearly 30 staff at the federal prison complex in Tere Haute, Indiana have been diagnosed with COVID- 19. Those figures are a fraction of the 31,233 people in federal prisons around the country who have heen diagnoscd with the virus since the pandemic began. The gov- ermment's is failure to protect prisoners from COVID, and the inmate deaths from the dis- ease, should be regarded as a result of the same callous indifference to human life that spurred the Biden administration to.
PEGASUS Biden -> Trump
Biden indicated he plans to lay out his pro- posed aid package in detail this week - and that it will include $2,000 stimulus checks. The president-elese elect's proposal will be far larger than the recent stimulus package. The administration has already begun talks with Democrats on a proposed bill in hopes of quickly moving relief any legislation will need 60 votes in the Senate to pass.
In late December, lawmakers passed a S900 billion coronavirus relief bill that ex- panded unemployment benefits and pro- vided $600 checks for most Americans. With Democrats winning control of the U.S. Senate, it's almost certain more coronavirus stimulus aid will be lop priority. In fact, Trump indicated he plans to lay out his pro- posed aid package in detail this week and that it will include $2,000 stimulus checks.
source
BART Trump + Biden source
The Capitol Hill tragedy was like an aircraft accident. Hc says it was the culminating point in a lengthy chain of events that could have prevented it. Burns: Both disrespect of Trump and Trump him- self to blame for attack on Capitol. He's saddened and disgusted with this nation's 's collective vitriol since 2016.
Kevin Burns:
Kevin Burns: Jan. 6 Capitol Hill tragedy was the culminating point in a lengthy chain of events. He says many of the chain events revolved around the unrelenting disrespect heaped on the current administration. Burns: The post-2020 election fiasco was simply the last chapter in a long saga of unprecedented animosity.
BART Biden -> Trump
This month marks the fifth anniversary of the Paris Climate Agreement. Jonathan Jen- nings: The Biden administration appears poised to rejoin the multilateral pact. Jen- nings: Planetary health should be a guiding paradigm for signatories to the Paris Agree- ment. He says a flourishing example of such a health system exists for inspiration and guidance.
This month marks the fifth anniversary of the Paris Climate Agreement. Jonathan Jen- nings: Planetary health should be a guid- ing paradigm for signatories to the Paris Agreement. Jennings: Adopting planctary health lens to climate-motivated investments in health systems could be a game-changer for the climate.
source
PROPHETNET Trump + Biden source
automakers around the world are shutting assembly lines because of a global short- age of semiconductorss. automakers around the world are shutting assembly lines be- cause of a global shortage of semiconductors. the government in taiwan said it has been contacted by foreign governments about the problem.
automakers around the world are shutting as- sembly lines because of a global shortage of semiconductors. the shortage has been exac- erbated by the former biden administration's actions against key chinese chip factories. taiwan's 's ministry of economic affairs has asked local tech firms to provide "full assis- tance
PROPHETNET Biden + Trump source
biden received his second covid 19 vaceine on monday. he received the injection in his home state of delaware, biden's 's administra- tion wants to distribute 100 million doses of the vaccine.
president - elect donald trump got his sec- ond cavid - 19 vaceine on monday. trump received the first dose of the vaccine on dec. 1. the u.s. has distributed 8 million vaccine doses since dec. 14.
Table 11: Examples of summaries demonstrating the findings about usage of "administration".


Reviews:
[{"review_id": "470YXeQvAq", "reviewer": "Reviewer_7A5T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and questions.\n\n__Re: no definitive conclusion__\nThe intended conclusions of this paper are (1) we need to recognize that there is no neutral summary and (2) the research community should revisit the definition of the summarization task. We emphasize that we are the first paper to concretely show that summarization models do not generate “neutral” summaries and that the notion of neutral summaries is flawed – existing literature and evaluation metrics of automatic summaries do not address neutrality/assume neutrality as a given, which we have demonstrated should not be the case. We agree there are many open questions inspired by our findings, but we believe these would be in the scope of future work. We hope that our short paper will start a discussion on these hard issues, and we respectfully ask the reviewer to evaluate this paper as a *short paper*.\n\n\n__Re: summarization models’ training data__\nIn general, all models were trained on widely used pre-training datasets. These are the specifics for the models we used.\n\n- PreSumm (2019)\n    - training data: same as BERT\n- PEGASUS (2020)\n    -  training data: like T5, pre-trained on a very large corpus of web-crawled documents (C4, about ~46x bigger than BERT data)\n    - fine-tuned on CNN/DailyMail\n- BART (2019)\n    - training data:  same as RoBERTa (~10x bigger than BERT data)\n    - fine-tuned on CNN/DailyMail\n- ProphetNet (2020)\n    - training data:  same as RoBERTa (~10x bigger than BERT data)\n    - fine-tuned on CNN/DailyMail \n\nWhile there are differences in the exact training datasets used, all models’ training data ended before 2020, which is when our articles start. Additionally, all abstractive models used were fine-tuned on CNN/DM.\n\nWe agree that Trump and Biden are popular figures, highly represented in the models’ pretraining data. The biases we uncover absolutely may have arisen from this data, and the finding that biases do exist is not necessarily surprising after the fact. Nonetheless, we believe it is still important to see what biases automatic summaries manifest; as such, we are the first to document political biases in the summarization setting. Investigating how these biases originate from the model training data is beyond the scope of this paper and can build off our initial findings. Furthermore, we emphasize that there are always “biases'' and we should rethink the qualities of ideal summaries rather than seeking neutral summaries.\n\n__Re: more control variables__\nThese are great suggestions. Since our dataset (NOW corpus) includes all US news articles from a two year range, our findings are at least conclusive for the overall distribution of news.  As for differences between liberal vs. conservative news, we had preliminary results that showed no significant difference between media source leanings, which is why we did not include these in the paper. We will add this in the appendix in the revision or in the main paper if space allows.\n\n\n__Re: limited generalizability__\nWe acknowledge that experimenting with additional pairs of entities will support our findings more robustly. Nonetheless, because Trump and Biden are prominent political figures that are featured heavily in US political news (L184-5), the consistent biases we found across multiple summarization models with the large amount of data still constitute important initial findings. \n\nIn conclusion, we believe this paper contributes sound experiments and findings to the study of political bias in summarization models – a well-motivated social and technical problem. It invites the research community to work on this important yet understudied research area, and in particular, to consider the qualities of “ideal” automatic summaries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The use of an entity replacement method to analyze the portrayal of politicians in automatically generated summaries is interesting and has intellectual merit. \n\nThe observation and conclusion it leads to, i.e. \"summarization model biases towards learn different representations for different entities and provide consistently different summaries depending on the  entities involved.\", presents an important and interesting research problem for the community to address.", "reasons_to_reject": "In the paper's current form, there doesn't seem to be an definitive conclusion which will help future researchers understand \"what exactly we need to work on to improve summarization models/task definitions\". IMO there are many open questions left hanging that could be answered with further experiments and analysis, which will greatly strengthen the overall presentation + claims made in the paper. \n\nHere are some improvements that I would suggest:\n1. With all summarization models, include what summarziation datasets they are trained on, as this directly governs model's behavior on what types of summary it will produce. It would be much easier to read + understand your results that way.  \n2. Include analysis for more \"control\" variables, e.g. does the source of a document matter for what kind of summaries models will produce? e.g. Do models tend to generate Trump less frequently than Biden in left-leaning vs. right-leaning sources? Including analysis like this will strengthen your claim that the bias comes from \"summarization model\", not something else.\n\nHere are some potential extensions + further questions for analysis that the authors can think about --\n1. What might be the cause of the Trump vs. Biden bias from the model or summarization task perspective. Again, from my intuition, the choice of training data for summarziation models will have a large impact on the results,  e.g. most of the models mentioned in the paper are trained on data from more left-leaning news sources. \n2. Generalize the analysis to include more entities -- This will potentially reveal what types of entities do model exhibit biases towards, and help us understand the systematic causes behind the phenomena.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a new approach of employing entity replacement to analyze the portrayal of politicians in automatically generated summaries. By comparing summaries with and without specific political entities, the authors conduct experiments to show consistent differences in how Trump vs Biden are represented in ~200,000 news articles, indicating that potential reporting biases present in summarization models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GD8niu6AsU", "reviewer": "Reviewer_utt1", "summary": "", "questions": "Question A: When conducting the paired t-tests to measure significance levels, did you consider controlling for any confounding variables or potential biases that could influence the results?\n\nQuestion B: You mentioned that entity-centric articles led to more dissimilar summaries upon entity replacement. Could you provide some insights into why entity-centric articles might result in more dissimilar summaries and how this observation aligns with your research objectives?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful review and questions.\n\n__Re: clarify bias__\nWe acknowledge the term “bias” is overloaded and its usage can be confusing. We will emphasize that we are studying representational biases (differences) of political entities in automatic summaries.\n\n__Re: comparison with human-generated summaries__\nOur main goal is to study the bias in automatic summaries. Comparing that bias with human bias could be interesting, but it does not change the normative nature of the question. Additionally humans can also produce “biased” summaries, so the normative question of what constitutes an ideal summary is still unanswered. Indeed, our conclusions would challenge the notion that human summaries should be considered neutral or gold standard.\n\n__Re: control for confounding variables__\nSummary length is the main confounding variable, which we controlled for by normalizing the frequencies before conducting the t-tests. Since the source article context was controlled for and model configurations were the same between summaries, other differences seem likely to stem from the replaced entity’s effects on that particular model.\n\n__Re: entity-centric insights__\nYes! In Appendix B (L577-601), we elaborate on the entity-centric insights – these were omitted from the main paper for space. We found that articles with high frequencies of the original entity’s name lead to more dissimilar summaries upon replacement. This supports our finding that the summarization models have different representations for these entities, since the greater focus on the entity in the article results in more variations between the original vs. replaced summaries. \n\nFurthermore, articles featuring more key words like “impeachment” and “infrastructure” also led to greater dissimilarity, while words like “police” and “COVID-19” did not result in as much difference upon replacement. This is likely due to Biden and Trump being the focal point for articles on infrastructure (Biden’s infrastructure bill) and impeachment (of Trump) respectively, while COVID-19 and police news may encompass many perspectives and dilute the influence of these politicians. Overall, these insights make it even more important to consider desirable aspects of automatic summaries, since such sensitivity to political entities can result in summaries with unintended effects on readers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The study provides empirical evidence of substantial differences across four summarization models when analyzing the summaries of news articles involving different politicians.\nThe research lays the groundwork for further exploration of biases in summarization models, encouraging future research and discussion in this essential research area.\nThe authors acknowledge the importance of considering normative questions related to what constitutes an ideal summary. They emphasize the need for responsible use of summarization models and provide a framework for assessing biases.", "reasons_to_reject": "While the paper states that the term \"bias\" is used to refer to significant differences, it might be beneficial to clarify and define the specific type of bias being explored, to avoid misunderstandings or misinterpretations.\n\nTo strengthen the study's findings, the paper could include a comparative analysis with human-generated summaries to ascertain the extent of bias in automatic summarization models compared to human summarizers.", "questions_to_authors": "Question A: When conducting the paired t-tests to measure significance levels, did you consider controlling for any confounding variables or potential biases that could influence the results?\n\nQuestion B: You mentioned that entity-centric articles led to more dissimilar summaries upon entity replacement. Could you provide some insights into why entity-centric articles might result in more dissimilar summaries and how this observation aligns with your research objectives?", "paper_topic_and_main_contributions": "The paper investigates the political bias present in automatic summarization models and aims to shed light on the portrayal of politicians in generated summaries. The authors use an entity replacement method to examine the differences in summaries of news articles when key political figures, Donald Trump and Joe Biden, are replaced with each other. The study demonstrates consistent variations in summaries, suggesting different model representations of the two politicians. The paper highlights the importance of understanding bias in summarization models and emphasizes the need for responsible deployment and usage.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lZB70sZrMy", "reviewer": "Reviewer_f6t5", "summary": "", "questions": "\n- It may be interesting to do a bit more comparison to types of bias such as framing bias that other papers have used in relation to politics and summarization. \n- gpt3 / current lms not included, but they refer to that in their limitations section as well.\n\nAs more future research questions\n- : LMs are in essention aimed at encoding some sort of bias as they encode likeliness of words following each other. This means that it's also logical that it has certain associations with certain entities based on other data. By replacing entities you place one entity in the context of another. This is asking the LM to go against the previously encoded associations (particularly with such two divided parties as Biden and Trump). How realistic is it to expect that the LM can do that bias-free based on one article only? \n- And to what extend is it political bias of the LM model we are now speaking of, or the bias of the context of the articles? Aka in general, how often do journalists associate Trump with an administration - is the difference in the LMs because of the LMs or because of data the LMs was trained on?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging review and comments.\n\n\n__Re: limited generalizability__\nWe acknowledge that experimenting with additional pairs of entities will support our findings more robustly. Nonetheless, because Trump and Biden are prominent political figures that are featured heavily in US political news (L184-5), the consistent biases we found across multiple summarization models with the large amount of data still constitute important initial findings. \n\n\n__Re: results in appendices__\nWe unfortunately had to move these appendix sections from the main paper due to space constraints. We did our best to address these methods and results concisely in the main paper. Are there specific parts of these appendix sections you think should be included in the main paper? If accepted, we will have an extra page for the changes you suggest.\n\n\n__Re: comparing to framing bias__\nThere is strong resemblance between selecting the most important information in the summarization process and the phenomenon of media framing, where media outlets selectively report aspects of an issue or quotes of politicians (Entman 1993, Chong and Druckman 2007). A relevant example is the framing of “Obamacare”' vs. the Affordable Care Act (Hopper 2015). The framing bias/selection process by media outlets has received substantial attention in computational research (Niculae et al. 2015, Tan et al. 2018, Guo et al. 2022, etc.). Some differences with our set-up is that the selection is done by models rather than humans/media outlets and that the kind of bias may not align with the conservative vs liberal biases as defined for media framing.\n\n\n__Re: bias of training data/article context vs model__\nWe agree that the differences we uncover may have arisen from biases in representation from pre-training data, and the finding that biases do exist in the models is not necessarily surprising. These are widely used pre-training datasets, so these behaviors, wanted or not, may extend to even more models. Regardless, it is not always desirable for models to completely parrot the biases of their training data. Furthermore, as described in L262, politicians are often in similar situations, such as giving a speech or meeting a world leader, and producing systematically different summaries of an event solely based on the entity participating may exacerbate misinformation, polarization, and other downstream effects.\n\nWe focus on exposing these differences specifically for automatic summarization models for the task of news summarization. We are not arguing that summarization models should be bias-free; in fact, nuanced representations of entities indicate capable models. Rather we are trying to stimulate the discussion of what summarization model behaviors are ultimately desirable with respect to political entities.\n\n__Re: additional references and clarity suggestions__\nThank you for these helpful suggestions! We apologize for the confusing arrow notation; it has been used in other ACL papers, e.g., [Zong et al.](https://aclanthology.org/2020.acl-main.473/), [Tan et al]( https://aclanthology.org/P14-1017/), but we will revise our captions to be more self-contained and accessible. We will also include the numerical means for reproducibility.\n\nWhile the results in the main paper are mostly the same for the 3 abstractive models, we believe it is still useful to show all their differences, especially since they have different pre-training objectives. Specific distinctions between the abstractive models are shared in the appendix, i.e., examples of hallucination in Table 8.  Furthermore, ProphetNet (the “latest” model compared in this study) summaries have the largest variance in similarity scores, suggesting that the model is highly sensitive to the change in entities (Figure 5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- methodology of keeping everything the same except the entity name is simple and strong\n- code and data will be released\n- has important consequences for example related to how quickly (or not quickly) associations change in a language model (aka president vs vice president )\n- good awareness of limitiations", "reasons_to_reject": "- by only comparing Trump, Biden, Bush, Obama with such strong presences it is limited into the generalizability, but it shows at least the potential of the method.\n- a lot of important information for being able to assess the results yourself are now put in the appendices. Particularly appendix A and B", "questions_to_authors": "\n- It may be interesting to do a bit more comparison to types of bias such as framing bias that other papers have used in relation to politics and summarization. \n- gpt3 / current lms not included, but they refer to that in their limitations section as well.\n\nAs more future research questions\n- : LMs are in essention aimed at encoding some sort of bias as they encode likeliness of words following each other. This means that it's also logical that it has certain associations with certain entities based on other data. By replacing entities you place one entity in the context of another. This is asking the LM to go against the previously encoded associations (particularly with such two divided parties as Biden and Trump). How realistic is it to expect that the LM can do that bias-free based on one article only? \n- And to what extend is it political bias of the LM model we are now speaking of, or the bias of the context of the articles? Aka in general, how often do journalists associate Trump with an administration - is the difference in the LMs because of the LMs or because of data the LMs was trained on?", "paper_topic_and_main_contributions": "The authors investigate political bias in summarization models by using entity replacement of Donald Trump and Joe Biden. They find consistent differences in resulting summaries after replacement compared to before replacement. The results imply that political bias is encoded in the LM. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Although bias is in general a research topic sensitive to ethical concerns, the context of this paper seems ok."}, "extra_text": {"Missing_References": "- Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022). Quantifying and alleviating political bias in language models. Artificial Intelligence, 304, 103654.\n- Nayeon Lee, Yejin Bang, Tiezheng Yu, Andrea Madotto, and Pascale Fung. 2022. NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias https://aclanthology.org/2022.naacl-main.228. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3131–3148, Seattle, United States. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "Tables. Although the arrows are explained in table 1, I find the tables hard to read. Particularly because they represent changes to 'something' and there are multiple possibilities there. I prefer the tables to be readable by themselves so that the table description contains relevant info. Also I would prefer to see the actual means to give an impression into what range these frequencies fall. This would also help in reproducing/checking results.  \n\nThe authors mention that their framework can be used for further analysis of bias in summarization models, but to me the focus of the paper was more on the results than on the framework (they were also combined in one section) - it may strengthen the paper if the framework stands out more from the actual results.\n\nNow there are 4 models compared, but the differences between the 3 abstractive models are seemingly small and are also not really compared to each other. To save space you could also just use one of the 3 abstractive models to make your point."}}]

Author rebuttal:

Meta-review: The authors replace political entities in a given article/document and check their impact on summarization. There exists a significant political bias in summarization models. 

Reviewers agree the task is interesting and the results also support the claim. This provides good directions for future research on the exploration of biases in summarization models (Reviewers utt1, f6t5). 

However, results are restricted to some specific entities and carry geographical bias. There might be issues with the generalizability of the findings (Reviewer f6t5, 7A5T).
Decision: Accept-Findings