SlowGuess/NAACL_2022_OCR huggingface dataset:
Souneil Park, Kyung-Soon Lee, and Junehwa Song. 2011. Contrasting opposing views of news articles on contentious issues. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 340-349.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021. Efficiently summarizing text and graph encodings of multi-document clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4768-4779.
Elizabeth M Perse and Jennifer Lambe. 2016. Media effects and society. Routledge.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic models for analyzing and detecting biased language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1650-1659.
Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 44(3):393-401.
Abeed Sarker, Ari Z Klein, Janet Mee, Polina Harik, and Graciela Gonzalez-Hernandez. 2019. An interpretable natural language processing system for written medical examination assessment. Journal of biomedical informatics, 98:103268.
Dietram A Scheufele. 2000. Agenda-setting, priming, and framing revisited: Another look at cognitive effects of political communication. Mass communication & society, 3(2-3):297-316.
Dietram A Scheufele and David Tewksbury. 2007. Framing, agenda setting, and priming: The evolution of three media effects models. Journal of communication, 57(1):9-20.
All Sides. 2018. Media bias ratings. Allsides.com.
Timo Spinde, Christina Kreuter, Wolfgang Gaissmaier, Felix Hamborg, Bela Gipp, and Helge Giese. 2021. Do you think it's biased? how to ask for the perception of media bias. In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 61-69. IEEE.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 152-158.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeno, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Remy Baly, et al. 2019b. Tanbih: Get to know what you are reading. EMNLP-IJCNLP 2019, page 223.
A Topics covered in ALLSIDESdataset
The ALLSIDESdataset language is English and mainly focuses on U.S. political topics that often result in media bias. The top-5 most frequent topics are 'Elections', 'White House', 'Politics', 'Coronavirus', 'Immigration'.
The full list is as follow (in a descending order of frequency): [‘Elections’, ‘White House’, ‘Politics’, ‘Coronavirus’, ‘Immigration’, ‘Violence in America’, ‘Economy and Jobs’, ‘Supreme Court’, ‘Middle East’, ‘US House’, ‘Healthcare’, ‘World’, ‘US Senate’, ‘National Security’, ‘Gun Control and Gun Rights’, ‘Media Bias’, ‘Federal Budget’, ‘Terrorism’, ‘US Congress’, ‘Foreign Policy’, ‘Criminal Justice’, ‘Justice Department’, ‘Trade’, ‘Impeachment’, ‘Donald Trump’, ‘North Korea’, ‘Russia’, ‘Education’, ‘Environment’, ‘Free Speech’, ‘FBI’, nan, ‘Abortion’, ‘General News’, ‘Disaster’, ‘US Military’, ‘Technology’, ‘LGBT Rights’, ‘Sexual Misconduct’, ‘Voting Rights and Voter Fraud’,
Joe Biden, 'Race and Racism', 'Economic Policy', 'Justice', 'Holidays', 'Taxes', 'China', 'Polarization', 'Democratic Party', 'Religion and Faith', 'Sports', 'Homeland Security', 'Culture', 'Cybersecurity', 'National Defense', 'Public Health', 'Civil Rights', 'Europe', 'Great Britain', 'Banking and Finance', 'Republican Party', 'NSA', 'Business', 'State Department', 'Facts and Fact Checking', 'Media Industry', 'Labor', 'Veterans Affairs', 'Campaign Finance', 'Life During COVID-19', 'Transportation', 'Marijuana Legalization', 'Agriculture', 'Arts and Entertainment', 'Fake News', 'Campaign Rhetoric', 'Nuclear Weapons', 'Israel', 'Asia', 'CIA', 'Role of Government', 'George Floyd Protests', "Women's Issues", 'Safety and Sanity During COVID-19', 'Animal Welfare', 'Treasury', 'Science', 'Climate Change', 'Domestic Policy', 'Energy', 'Housing and Homelessness', 'Bridging Divides', 'Mexico', 'Inequality', 'COVID-19 Misinformation', 'ISIS', 'Palestine', 'Bernie Sanders', 'Tulsi Gabbard', 'Sustainability', 'Family and Marriage', 'Pete Buttigieg', 'Welfare', 'Opioid Crisis', 'Amy Klobuchar', 'Food', 'EPA', 'South Korea', 'Alaska: US Senate 2014', 'Social Security', 'US Constitution', 'Tom Steyer', 'Andrew Yang', 'Africa']
B Additional Salient Information Score Results
We report additional Salient information F1 (Table 5) and Recall (Table 6) scores for ROUGE1, ROUGE2 and ROUGE-L.
ROUGE1 F1	ROUGE2 F1	ROUGE L F1
LEXRANK	33.60%	13.60%	29.77%
BARTCNN	33.76%	13.67%	30.57%
PEGASUSMULTI	30.03%	10.28%	26.70%
BARTMULTI	23.01%	6.84%	20.55%
NEUSFT	36.76%	16.27%	32.86%
NEUS-TITLE	35.49%	15.69%	32.05%
C Details for Human Evaluation (A/B testing)
We first presented the participants with the definition of framing bias from our paper, and also
Table 5: Additional Salient Info Scores. F1 scores for ROUGE1, ROUGE2 and ROUGEL for ALLSIDES test-set. For the scores, the higher number is the better.
ROUGE1
RECALL	ROUGE2
RECALL	ROUGEL
RECALL
LEXRANK	39.08%	17.66%	34.69%
BARTCNN	35.63%	15.32%	32.22%
PEGASUSMULTI	44.42%	16.99%	39.45%
BARTMULTI	35.76%	12.48%	32.08%
NEUSFT	35.11%	15.74%	31.43%
NEUS-TITLE	36.07%	16.47%	32.63%
Table 6: Additional Salient Info Scores. Recall scores for ROUGE1, ROUGE2 and ROUGEL for ALL-SIDES testset. For the scores, the higher number is the better.
showed examples in Table 1 to ensure they understand what framing bias is. Then we asked the following question: "Which one of the articles do you believe to be more biased toward one side or the other side in the reporting of news?" This is modified to serve as a question for AB testing based on "To what extent do you believe that the article is biased toward one side or the other side in the reporting of news?" The original question is one of the 21 questions which are suitable and reliable for measuring the perception of media bias, designed by Spinde et al. (2021).
The participants (research graudate students) have different nationalities including Canada, China, Indonesia, Iran, Italy, Japan, Poland and South Korea (ordered in an alphabetical order). All of participants answered to be not having political leaning towards U.S. politics. All participants are fully explained on the usage of collected data in this particular work and agreed on it.
All our experimental codes are based on the HuggingFace (Wolf et al., 2020). We used the following hyperparameters during training and across models: 10 epoch size,  $3e - 5$  learning rate, and a batch size of 16. We did not do hyper-parameters tuning since our objective is to provide various baselines and analysis. Training run-time for all of our experiments are fast  $(< 6\mathrm{hr})$ . We ran all experiments with one NVIDIA 2080Ti GPU with 16 GB of memory. The experiment was a single-run.
E Generation Examples from Different Models
To help better understand performances of each models, we provide more examples of generation
from all baseline models and our proposed models along with the target neutral article summary. The examples can be found in Table 7, 8, 9.
F Illustration of hallucinatory framing bias
In Table 10, we provide more examples of hallucinatory framing bias that was discussed in analysis.
Target: The House Ways and Means Committee has requested that the IRS turn over the last six years of tax returns from President Trump.
LExRANK: Representative Richard Neal invoked law that allows House Ways and Means Committee chair to obtain any person's tax information. The request by House Democrats for six years of Donald Trump's tax returns opens a serious new political, legal, constitutional and personal front in the party's battle to use its new majority to check a defiant President.
BARTCNN: Rep. Richard Neal invoked law that allows House Ways and Means Committee chair to obtain any person's tax information. The request by House Democrats for six years of Donald Trump's tax returns opens a serious new political, legal, constitutional and personal front in the party's battle to use its new majority to check a defiant President.
PEGASUSMULTI: The battle over President Trump's tax returns is officially on. Rep. Richard Neal, chairman of the House Ways and Means Committee, on Wednesday formally asked the IRS for six years of Trump's personal and business tax returns, as well as the returns of some of his companies, reports the Washington Post. The panel has the power to subpoena someone's tax information, and Neal has signaled that he plans to do just that. "The American people have a right to know who the president of the United States is, and we have a responsibility to do our part to ensure that the tax laws are administered in an efficient and thorough manner," he said in a statement, per the Hill. The White House has signaled that it will fight the request, with press secretary Sarah Huckabee Sanders calling it "an abuse of the tax-writing committee's authority." The letter to the IRS was signed by Neal, a Massachusetts Democrat, and was obtained by the Wall Street Journal. The Journal notes that Neal has the authority under a law passed in 1969 that allows the House Ways and Means Committee to obtain tax information from anyone.
BARTMULTI: the head of the powerful house tax-writing committee has asked the government for six years of president trump's tax returns, a move that is expected to lead to a long court battle with the white house, the washington post reports. in a letter to the irs on wednesday, rep. richard h. Neal, chairman of the house ways and means committee, invoked a law that allows any person to obtain any person's personal tax information, the wall street journal reports. according to the hill, the move is a "serious new political, legal, constitutional, and personal front in the party's battle to use its new majority to check a defiant president." it is a bid to solve one of the most tantalizing and enduring mysteries of the trump campaign and his presidency — what is it in the president's unseen financial and business life that he doesn't want americans to see?
NEUSFT: House Ways and Means Committee Chair Richard Neal (D-MA) has formally requested President Trump's tax returns from the IRS.
NEUS-TITLE: TITLE  $\coloneqq$  > House Committee Seeks Trump Tax Returns. ARTICLE  $\coloneqq$  > The House Ways and Means Committee formally requested President Trump's tax returns on Wednesday, kicking off what could be a protracted legal fight between Congress and the Trump administration.
Table 7: Neutral Summary Generation Examples from baseline models and NEUS-TITLE.
Target: The State Department has blocked Gordon Sondland, the U.S. Ambassador to the European Union, from testifying at a scheduled deposition on the Ukraine controversy. Sondland is a key figure as he had texted with Bill Taylor, a top U.S. diplomat in Ukraine, arguing over allegations of a quid pro quo. This move by the State Department has triggered a new clash with House Democrats as part of their impeachment inquiry, and President Trump has dismissed the deposition as a "a kangaroo court" as he defended the State Department decision.
LExRANK: The Trump administration on Tuesday blocked a planned deposition from Gordon Sondland, the U.S. ambassador to the European Union and a central figure in the Democrats' impeachment inquiry, according to a statement by his lawyer. Democrats swiftly called the administration's move an act of obstruction and accused President Trump of hiding information about his efforts to pressure the Ukrainian government to investigate his political rivals.
BARTCNN: The State Department instructed Ambassador to the European Union Gordon Sondland not to appear for the deposition, according to his counsel. Democrats swiftly called the administration's move an act of obstruction and accused President Trump of hiding information about his efforts to pressure the Ukrainian government to investigate his political rivals.
PEGASUSMULTI: The Trump administration has blocked US ambassador to the European Union Gordon Sondland from testifying before a House panel investigating possible obstruction of justice by the president, the New York Times reports. Sondland is a key witness in the probe into whether Trump obstructed justice when he fired FBI chief James Comey. Sondland had been scheduled to testify in front of the House Judiciary Committee on Tuesday, but the State Department ordered him not to appear, according to the Washington Post. Sondland is a former British ambassador to the US. In a Tuesday tweet, President Trump said he would love to send Sondland to testify, "but unfortunately he would be testifying before a totally compromised kangaroo court, where Republican's rights have been taken away, and true facts are not allowed out for the public to see." He added: "I believe you are incorrect about President Trump's intentions."
BARTMULTI: a key witness in the house impeachment inquiry was ordered not to appear at a scheduled deposition on Tuesday, ramping up tensions between congressional democrats and the trump administration. the state department instructed ambassador to the eu and key figure in the impeachment inquiry, Gordon sondland, not to show up for the deposition, according to his counsel. democrats called the move an act of obstruction and accused president trump of hiding information about his efforts to pressure the ukraine government to investigate his political rivals, the new york times reports. " this is a clear attempt to block testimony from key witnesses in the ongoing impeachment investigation, " said house minority leader nancy pelosi, per the washington post. " it's an impeachable act. " in a tweet tuesday night, trump said he " would love to send ambassador [ sundland ], a really good man and great american, to testify, but unfortunately he would be testifying before a totally compromised kangaroo court, where republican ' s rights have been taken away, and true facts are not allowed out for the public to see. "
NEUSFT: The Trump administration blocked a planned deposition from Gordon Sondland, the U.S. ambassador to the European Union and a central figure in the Democrats' impeachment inquiry, according to a statement from his lawyer. President Trump responded via Twitter, saying that he "would love to send Ambassador Sonderland, a really good man and great American, to testify, but unfortunately he would be testifying before a totally compromised kangaroo court, where Republican's rights have been taken away, and true facts are not allowed out for the public to see."
NEUS-TITLE: TITLE  $\coloneqq$  > State Dept Orders Ambassador Sondland Not to Testify Before House. ARTICLE  $\coloneqq$  > The State Department has ordered Ambassador Gordon Stondland not to testify before the House of Representatives in the impeachment inquiry of President Donald Trump. Democrats have accused the Trump administration of blocking statements from witnesses.
Table 8: Continued from Previous Page: Neutral Summary Generation Examples from baseline models and NEUS-TITLE.
Target: Ukrainian police have opened an investigation into whether or not U.S. Ambassador Marie Yovanovitch came under surveillance before she was recalled from her post in Ukraine last May. Democrats have released documents that show Lev Parnas, an associate of Rudy Giuliani, communicating about Yovanovitch's removal.
LEXRANK: Ukraine's government announced Thursday that police are investigating whether ousted U.S. ambassador Marie Yovanovitch was subject to illegal surveillance, in response to new documents released ahead of President Trump's impeachment trial. Those documents, released by Democratic lawmakers, showed Lev Parnas – an associate of Trump lawyer Rudy Giuliani – communicating about the removal of Marie Yovanovitch as the ambassador to Ukraine.
BARTCNN: Police in Ukraine have opened a criminal investigation into whether allies of President Trump had the U.S. ambassador to the country under surveillance. Newly released documents suggest that Ms. Yovanovitch was being watched in Kyiv. The Ministry of Internal Affairs of Ukraine says it's asking the FBI to help its inquiry.
PEGASUSMULTI: The Ukraine has opened a criminal investigation into allegations that President Trump's allies had the US ambassador to the country under surveillance while she was stationed in the country, the New York Times reports. According to the Wall Street Journal, the allegations surfaced in newly released documents released by Democrats ahead of Trump's impeachment trial. The documents show that Rudy Giuliani's associate, Lev Parnas, communicated about the removal of Marie Yovanovitch as ambassador to Ukraine with Yovanovitch's former boss at the State Department, Sergey Kislyak. Yovanovitch was removed from her post in May of last year. Ukraine's Ministry of Internal Affairs says it's asking the FBI to help with the investigation. "Ukraine cannot ignore such illegal activities on the territory of its own state," the ministry says in a statement
BARTMULTI: new documents released ahead of president trump's impeachment trial suggest that the us ambassador to the country was under surveillance while she was stationed in the country, the new york times reports. according to the wall street journal, the allegations were made in a text message between a us politician and an associate of trump lawyer rudy giuliani. in the messages, the politician says he'd like to see former ambassador to ukraine and current ambassador to russia, former deputy foreign minister oleksandr turchynov, removed from his post. " i would like to remove him from his position," the politician wrote in the message, which was sent in april 2017. the messages were sent in response to newly released documents from the house intelligence committee. those documents show that former trump lawyer paul ryan had been trying to get rid of yovanovitch, who was ambassador at the time, for more than a year, the journal reports. the messages also show that ryan's chief of staff, george w. bush, had been in touch with ryan about removing her from her post.
NEUSFT: Ukraine's national police are investigating allegations that U.S. Ambassador Marie Yovanovitch was subjected to illegal surveillance while she was stationed in Kyiv.
NEUS-TITLE: TITLE  $\Rightarrow$  Ukraine Opens Investigation Into Claims U.S. Ambassador Yovanovitch Was Under Surveillance. ARTICLE  $\Rightarrow$  Ukraine's national police are investigating allegations that former U. S. ambassador Marie Yovanovich was subjected to surveillance while she was stationed in Kyiv, following the release of documents by Democratic lawmakers.
Table 9: Continued from Previous Page: Neutral Summary Generation Examples from baseline models and NEUS-TITLE.
Table 10: Examples of hallucinatory framing bias from MDS models and the corresponding the source input.