How (Un)Fair is Text Summarization?
Authors: Hannah Brown
Reza Shokri
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural language processing
Summarization
Fairness

Abstract: Creating a good summary requires carefully choosing details from the original text to accurately represent it in a limited space. If a summary contains biased information about a group, it risks passing this bias off to readers as fact. These risks increase if we consider not just one biased summary, but rather a biased summarization algorithm. Despite this, little work has measured whether these summarizers demonstrate biased performance. Rather, most work in summarization focuses on improving performance, ignoring questions of bias. In this paper we demonstrate that automatic summarizers both amplify and introduce bias towards information about under-represented groups. Additionally, we show that summarizers are highly sensitive to document structure, making the summaries they generate unstable under changes that are semantically meaningless to humans, which poses a further fairness risk. Given these results, and the large scale potential for harm presented by biased summarization, we recommend that bias analysis be performed and reported on summarizers to ensure that new automatic summarization methods do not introduce bias to the summaries they generate.

Under review as a conference paper at ICLR 2023
How (UN)FAIR IS TEXT SUMMARIZATION?
Anonymous authors
Paper under double-bling blind review
ABSTRACT
Creating a good summary requires carefully choosing details from the original text to accurately represent it in a limited space. If a summary contains biased information about a group, it risks passing this bias off to readers as fact. These risks increase if we consider not just one biased summary, but rather a biased summarization algorithm. Despite this, no work has measured whether these sum- marizers demonstrate biased performance. Rather, most work in summarization focuses on improving performance, ignoring questions of bias. In this paper, we demonstrate that automatic summarizers both amplify and introduce bias towards information about under-represented groups. Additionally, we show that summa- rizers are highly sensitive to document structure, making the summaries they gen- erate unstable under changes that are semantically meaningless to humans, which poses further faimesss risk. Given these results, and the large-scale potential for harm presented by biased summarization, we recommend that bias analysis be per- formed and reported summarizers ensure that new automatic summarization methods do not introduce bias to the summaries they generate.
INTRODUCTION
In any piece of text, bias against a group may be expressed. This bias may be explicit or implicit and can be displayed either in what information is included (e.g.. including information that is exclusively negative about one group and exclusively positive about another), where in the article it comes from (e.g., only selecting sentences from the start of articles), or how it is written (c.g., saying "a man thought to be involved crime died last night after an officer involved shooting" vs. "a police officer shot and killed an unarmed man in his home last night").
The presence of any bias in a longer text may be made worse by summarizing it. A summary can be seen as presentation of the most salient points of larger piece of text, where the definition of "salient information" will vary according to various ideologies person holds. Due to this subjec- tivity and the space constraints summary imposes, there is heightened potential for summaries to contain bias. Readers, however, expect that summaries faithfully represent articles. Therefore, bias in the text of a summary is likely to go unquestioned. If summary presents information in way biased against a group, readers are likely to believe that the article exhibited this same bias, as checking the truth of these assumptions requires a high amount of effort. This poses several risks. First, an echo chamber effect, where the bias in generated summaries agrees with biases the reader already has. The opposite is also risk. An article may present more or less the same amounts of in- formation about multiple groups whereas its summary includes more information about one group, leading readers to believe the most important information in the article is about only one group.
As writing summaries manually carries a large cost, automatic summarization is an appealing solu- tion. However, where one biased summary a problem, biased sunmarization algorithm capable of summarizing thousands of articles in the time it takes human to generate one, is a disaster. In recent years, automatic summarization has increased availability, both for personal and commer- cial use. Summarization algorithms have suggested for use on news articles, medical notes, business documents, legal texts, personal documents") and conversation transcript Despite the sensitivity of these applications, to the best of our knowledge, work has measured the bias towards groups of summaries generated by common summarization algorithms.

---

Under review as a conference paper at ICLR 2023
In this paper, we present the first empirical analysis to measure the bias of automatic summarizers, for seven different techniques (ranging from the ones based on information retrieval methods to the ones using large language models; and for high impact techniques provided by Microsoft and Ope- nAT). We design a method to quantitatively measure to what extent article structure influences bias in the generated summaries, as well as the inclusion of information about different gender, racial/ethic, and religious groups We also study the causes of bias by varying the influential factors, including the summarization parameters, and distribution of input documents. We show that summarizers:
1. Can suppress information about minority group in the original text.
2. Prioritize information about certain groups over others, regardless of the amount of information about them in the original text.
3. Amplify patterns of bias already shown in human summaries in where information is selected from in the original text.
4. Demonstrate sensitivity to the structure of the articles they summarize, and are fragile to changes in those articles that are meaningless to humans.
These findings indicate that it is not safe use automatic summarization at scale in situations where the text they generate could influence large numbers of readers. Doing so risks misinform- ing and increasing bias held by readers. We conclude that assessments of bias in summarization algorithms should be performed, and this bias should be reduced before models are released.
2 WHAT IS BIAS IN A SUMMARIZER?
Let us start by defining what biased summary is. We consider biased summary to be be any summary that misrepresents a group some way, relative to the original tex whether this the amount or kind of information expressed. We further divide our definition of bias into two subcategories based on their cause: content bias and structure bias. These definitions draw on existing taxonomies of representational harms in NLP (Hovy & Spruit 2016 Barocas al. 2017). but are tailored to summarization.
We define content bias as bias towards the mention of group in text. In this type of bias, if group A is mentioned in the article, the summarizer will generate a biased summary. Changing the structure of the text will have no effect on how biased the summary is. We further divide content bias into five subcategories: under-representation, inclusion/exclusion, inaccuracy, sentiment bias, and framing bias, and explain each in Table
On the other hand, we define structure bias as a bias as a result the structure of the text being sum- marized. In contrast content bias, structure bias is invariant the presence of information about different groups, but will change when document structure is modified. We define three structure bias subcategories: position bias, sentiment bias, and style bias, further explained in Table
There are situations where the line between group and structure bias is not so clear cut. For example, articles about group A are written in a way that summarizer has a structure bias against, but articles about group B are written in way that does not illicit this bias, the summarizer will show a consistent bias against group A. Though this is a result a structure bias, is also an example of content bias because the article structure and the group being discussed are Repandeent.egarrlees of the cause, biased summaries have the potential for real, negative effects on marginalized groups.
3 OUR METHODS FOR MEASURING BIAS
Each of the types of bias defined in Table T represents a real problem in automatic summarization. However, we are not able to explore all of these biases in this paper, and some (i.e. framing bias) would be best explored using manual linguistic analysis than computational techniques. Instead, we aim to explore the content and structure bias of summarizers along three axes: underrepresentation,
We recognize that it could be argued that perfectly unbiased summarizer would not misrepresent groups all, even if the original text does, this is strong constraint to place summarization algorithm that does have concept of social bias.
2

---

Under review as a conference paper at ICLR 2023
Table 1: Potential ways in which summarizer could show content and structure bias. Bias name appears on the left, bias category (content or structure) in the middle, and a description of the bias on the right. We consider all measures relative the original article. We include an expanded table with examples of biased summaries in Appendix
Name
Category
Description
Underrepresentationn
Content
Including less about one group than another in the summary. In/excluding a detail only when specific group is involved. Including information in a way that inaccurately reports infor- mation about specific group (includes hallucination for ab- stractive models).
Inclusion/exclusion
Content
Inaccuracy
Content
Sentiment bias
Content
Using more positive sentiment for group another. Using different sentence structure frame a situation involving one group vs another.
Framing bias
Content
Position bias Sentiment Bias
Structure
Selecting text from specific position in the article Preferring information of positive or negative sentiment Including sentences written in specific style. For English this could include active/passive voice, first/third person, etc.
Structure
Style Bias
Structure
position, and style. To this end, we design three experiments to measure these three different types of bias described in more detail in the following subsections.
Data and Models For our content bias analysis, we use synthetic article corpus, the generation of which is explained in Appendix [ For all other experiments, we use the CNN/DailyMaii corpus (See) al. 2017, Hermmnn af., 2015). This corpus consists of 300k articles, written between 2007 and 2015, from CNN and the DailyMail. The corpus also contains reference summaries, generated from the bullet point highlights each article. To generate summaries of these articles for our experiments, we use a four extractive and three abstractive summarizers described in Table
Table 2: Summarizers used in the experiments for this paper. choose a mixture extractive summarizers, which generate summaries by picking relevant sentences from the article, and abstrac- tive summarizers, which generate their own text. Azure and GPT-3 provide only API access, while all other models downloaded from pre-trained checkpoints. All models except TextRank, which fully unsupervised, use machine learning.
Summarizer
Type
TextRank Mihhalcea & Tarau PreSumm (Liu & Lapata
2004
Extractive
2019)
Extractive
MatchSum ZZhong et al.
2020
Extractive
Azure
Extractive
BART Lewis et al. 2019)
Abstractive
PEGASUS (Zhang et al.
2020
Abstractive
GPT-3 Brown et al. 2020)
Abstractive
3.1 CONTENT BIAS ANALYSIS
Given articles about different (combinations of) groups, do summarizers tend to include more infor- mation about some groups than others? For example, given articles that are half about men and half' about women, their summaries should contain the same ratio of information. However, summa- rizer with a bias against women may amplify information about men and supress information about women in its summaries. To explore this problem, compare the amount of group information for each group in the original text to that in the summary. If information about a group is consistently reduced more increased less than other groups, this indicates content bias in the summarizer.
3

---

Under review as a conference paper at ICLR 2023
In our analysis, we consider nine groups from three categories: and women for gender; Black, White, Hispanic, and Asian for race/ethnicity; and Islam, Christianity, and Judaism for religion. To fully explore the effect of different combinations of groups, we use GPT-2 (Radford et al. 2019) to generate a synthetic corpus of articles designed to contain different ratios of information for group pairs from each category. We describe our methods for measuring the amount of group information in text in the following subsections. Full methods for generating our synthetic corpus can be found in Appendix B
Group Association Scores In order measure underrepresentation bias, must first be able to measure how much information about a group is present. While this could be done using a keyword counting approach by creating lists of words corresponding to each group and quantifying the group information as the number of words from each list that appear in a text, this is likely to miss information it is not feasible to specify every word related to group. Instead, inspired by work measuring bias in static embeddings (Caliskan et al., 2017 Bolukbasi et al. 2016 Garg et al., 2018), we measure the group membership of each word in a text by generating its embedding using a word2vec model (Mikolov et al., (2013) and measuring its average cosine similarity to group word lists from Garg et al. (2018); Bertrand Mullainathan (2004); Manzini al. (2019).
After we have these scores, we aggregate the scores to obtain text-wide measure of association with each group. We refer to this as a group association score, or A, and it is calculated as follows:
A(T,g) [
E
S(t,g)
(t,))(,
Where T the text, g the group, S(1.g) the average cosine similarity between token / and group 9. and a; is the cosine similarity threshold for group i above which we consider words to have a strong similarity to the group 4 As it is highly unlikely that words belonging to some POS will have true relation to group, we use differrnt POS filters to calculate these association scores depending on the group category. For gender, we calculate the scores only for pronouns and nouns and for race/ethnicity and religion we use nouns and adjectives.
For each group, we calculate the representation score, R(T,9 .)) as relative proximity to the group with the maximum difference association score in the text:
A(T. g)
R(T .) (axx(A(T, v))
This means that for a group with score 0.4 over a set of texts, we would expect to see 40% the infor- mation about that group vs. the group each article is about. Unbiased summarizers should produce summaries with roughly the same proximity scores for each group as the article. If these scores de- crease for group after summarization, this indicates an underrepreseentation bias toward that group. Likewise, if these scores increase for a group, this indicates a tendency to overrepresent this group. For summary that does not show underrepresentation bias, we expect that representation scores for each group in the summary to be the same as those in the original article.
3.2 POSITION BIAS ANALYSIS
If information given at the end of an article is moved to the beginning, more likely to appear in the summary? As it is impossible to write about two different things in the same space, if there are multiple groups mentioned in an article, one must appear first. If summarizers show a bias for picking earlier or later information, this may also pose a fairness problem towards different groups.
It has been found that reference summaries and some automatically generated summaries for news articles show bias for information that appears near the beginning of the articles Jung et al. (2019). However, it is not known if this pattern holds for more recent summarization algorithms,, nor if the amount of position bias between human and automatic summarization comparable. Our goal to measure the amount of position bias in reference summaries to that shown by each summarizer
FFor description of how calculate these thresholds, see the Appendix C33
4

---

Under review as a conference paper at ICLR 2023
to see if this existing bias is amplified through the use of automatic summarization. To do this, we compare the distribution of sentence locations between automatically generated summaries and reference summaries to measure any differences.
Because we examine abstractive and extractive summaries, we cannot measure where each summary sentence is from through exact matching. Instead, create TF-IDF vectors of each sentence in the article and summary and compute the cosine similarity between each, marking article sentences with high similarities being present in the summary.
3.3 PHRASING BIAS ANALYSIS
Given paraphrased version of article, do summaries still include the same information in their summaries? We assume that summary should include the same information regardless of how it is phrased the original text. However, do know if this is the case for automatic summariza- tion. A summarizer that is sensitive to the phrasing of sentences may generate drastically different summaries of an article and a paraphrased version of that article. Fragility of this kind is highly undesirable. For one thing, it is highly unintuitive to human authors. If one article contains the exact same information as another, we expect that their summaries should also contain the same informa- tion, regardless of article phrasing. But additionally, sensitivity to phrasing could lead to amplified content bias. Imagine an author who consistently uses a certain phrase sentences about one group and a summarizer that never chooses sentences with this phrasing. This summarizer would then unfairly exclude information about this group due to its sensitivity to phrasing.
We perform two paraphrasing experiments measure the sensitivity of each summarizer sentence phrasing: one targeted version and one random version. the targeted version, we paraphrase the article sentences that originally appear in the generated summary. In the untargeted version, paraphrase a random 10% of sentences in the article. We compare the sentencees that appear in the original summary to those that appear in the newly generated summaries. If the summarizers are sensitive to sentence phrasing, we should observe lots of changes in the sentences selected for the original summary and the summary after paraphrasing.
4 RESULTS
The results of our experiments show that automatic summarization is sensitive to the structure of articles, as well as, to some degree, the groups discussed in the articles. Additionally, when com- pared to the reference summaries and the original articles, machine learning based summarization algorithms show a tendency of exacerbating the position bias shown by the reference summaries. Due to space constraints, report only results for gender. However, results hold for other groups as well, and are presented in the appendix.
4.1 CONTENT BIAS ANALYSIS
We summarize articles comprised of half sentences from articles about men and half about women. We find that all machine-learning based algorithms show a large sensitivity the ordering of the information in the text, suggesting a position bias, which we explore further in Section 4.2 As shown in Fig. [ almost all machine-learning based summarizers show strong preference for sen- tences about the group that appears first in the article. Even though there are an equal number of sentences in each article about men and women, these summarizers show a tendency to amplify formation that appears earlier in the article and surpress information that comes later. GPT-3 is the exception, showing an opposite, but much stronger tendency. It prioritizes information that appears Mibeinye hálpnn tippattiieepppttiitiiiii it iniiiin den httpa/anwccaaamiiiiioooooorrrrii aag ianliclns dlggin Withh www.a//mnnoottttootatte.aea. machine learning based models except GPT-3 consistently maintain or increase the men's 's represen- tation scores while decreasing those of women, indicating a preference for information about men. However, when this order is reversed, this pattern follows suit Fig. 2b Despite the small ratio of sentences about women present in the summaries, is still amplified occurs in the preferred location in the article.
mahhwith 10% about women, all
5

---

Under review as a conference paper at ICLR 2023
1
Representation score
0.8
0.6
I Women
D Men
0.4
0.2
0
textrank presumm
matchsum
azure
bart
pegasus
gp13
textrank
presumm
matchsum
azure
bart
pegasus
gpt3
(a) 50% of sentences from articles about men fol- lowed by 50% from articles about women
(b) 50% sentences from articles about women fol- lowed by 50% from articles about men
Figure I: Representation scores for men and women in summaries of balanced multi-group articles. Dashed lines are baseline representation scores of original articles, while bars scores of summaries. Error bars represent 95% confidence intervals.
Representation score
0.8
0.6
Women
Men
0.4
0.2
0
textrank
presumm
matchsum
azure
bart
pegasus
gpt3
textrank
presumm
matehsum
azure
bart
pegasus
gpt3
(a) 90% sentences from articles about fol- lowed by from articles women
(b) 10% sentences from articles about women fol- lowed by 90% from articles about men
Figure 2: Representation scores for men and women in summaries of imbalanced multi-group ticles. Dashed lines are baseline representation scores of original articles, while bars scores of summaries. Error bars represent 95% confidence intervals.
To see if all of the bias is due to location, we summarize articles about only men and women and measure if ampliffcation/suprssiin still occurs. As shown in Fig. 3b we observe that for articles about men, no significant changes occur for either group. In contrast, in summaries of articles about women, the extractive summarizers maintain this pattern, but the abstractive summarizers show tendency to underrepresent women and overrepresent men their summaries. As seen Fig. Ba this is particularly exacerbated when summary length is increased, with BART and GPT-3 making significant (p < 0.05) changes to the representation scores men and women in summaries 50% the length of the original article.

---

Under review as a conference paper at ICLR 2023
Representation Score
Women
0.5
Men
textrank
azure
bart
pegasus
gpl3
textrank
azure
bart
pegasus
gpt3
(a) Articles about women
(b) Articles about men
Figure 3: Representation scores for men and women in summaries 50% the length of single-group articles, Dashed lines are baseline representation scores of original articles, while bars scores of summaries. Error bars represent 95% confidence intervals. Here, because we cannot modulate the length of generated summaries, MatchSum and Presumm are not included.
502
030
030
(a) CNN/DM reference summaries
(b) Azure summaries
(c) GPT-3 summaries
Figure 4: Percent of sentences chosen from each position in original articles from the CNN/DailyMail corpus by the reference summaries and by Azure and GPT-3.
Overall, these results show a pattern unpredictability and fragility in machine learning based sum- marization algorithms, which seem to depend more on the ordering than the presence of information. While some ML based summarizers are more susceptible to this than others (c.g., GPT-3), all exhibit similar tendencies.
4.2 POSITION BIAS ANALYSIS
The pattern observed in Section 4.1 for summaries synthetic corpus also holds for sum- maries of real articles from CNN/DailyMail. All summarizers except GPT-3 show preference for sentences that appear close the start of articles. This preference exists in the reference summaries as well, however, it is clearly amplified by MatchSum, PreSumm, and Azure, as shown in Fig. 4t (all machine learning based abstractive models), which select sentences from the start of articles more often and almost never select sentences from the end of articles. In contrast, GPT-3 selects sentences from the start of articles less often, and selects sentences from the end at significantly higher rate than the reference summaries Fig. 4c These inconsistencies show marked differences between human and automatic text summarization, and, in combination with the content bias results shown in Section 4.1, suggest that machine learning based summarizers may be learning more about detere A AYY And less about what information to select.
Our paraphrasing analysis shows that summarizers are sensitive to the structure of the articles they summarize. Specifically, however, they are sensitive to changes in the structure of sentences that
Consullt Appendix for all figures

---

Under review as a conference paper at ICLR 2023
would have appeared in the original summary. As shown in Fig. pa when the sentences chosen for paraphrasing are those that appear in the summary of the un-paraphrased article, the likelihood that the summary will remain the same is quite low. However, when similar amount of random sentences are paraphrased uses the same sentences the un-paraphrased version, as ibilluttisssss........ This demonstrates that summarizers are highly sensitive to the paraphrasing of sentences that they rank as highly important, showing a structure bias.
the untargeted version, often the summary of the paraphrased article
TextRank
MatchSum
PreSumm Ext
Azure
PEGASUS
BART
TextRank
MatchSum
PreSumn Ext
Azure
PEGASUS
BART
Overlap
0.5
0.5
0
0
(a) Targeted paraphrasing
(b) Untargeted paraphrasing
Figure 5: The proportion of overlap in sentences chosen for summaries before and after paraphrasing for each of the paraphrased versions of each article. While all are relatively unstable the targeted version (Fig. 5a in the untargeted version (Fig. 5b), the percent overlap is noticably higher.
LIMITATIONS
Synthetic Data Using news articles study the content bias of summarizers comes with a number of challenges. First, news articles are not always about specific groups. Additionally, some com- binations groups show up very rarely, making it impossible to get data about them. Ideally, we would modify the amount of information present for each group measure how much summarizers prioritize some groups over others. Using synthetic data gives us the flexibility to limit the groups present in article combine different amounts of information about specific groups measure the impact of both group and amount of information on generated summaries.
Types of Bias We are limited exploring types of bias that can be reliably detected using com- putational methods. This ignores many more subtle types of bias, such as misrepresentation, which are no less important for not being included in our analysis. This is not intended as an exhaustive study of the types of bias that can occur in automatic summaries. Rather, it is an initial exploration which we hope will inspire others to consider the bias summarization systems may introduce. Other types of bias, such framing bias would be best explored using manual linguistic analysis, and this a line of future research we strongly encourage.
RELATED WORK
To best understand our work, is important to have an understanding of itself, as well as linguistic and NLP methods for measuring bias, and prior work analyzing various types of bias present in automatic summarization.
automatic summarization
Summarization Methods There are two methods of automatic summarization: extractive and ab- stractive. In extractive summarization, articles" sentences are ranked according some algorithm and the top are chosen as the summary of the text. We study the extractive models TextRank (Mi- halcea & Tarau 2004). which is based the PageRank algorithm, along with PreSumm (Liu & Lapata (2019) and MatchSum (Zhong et al. 2020), which use embeddings and neural networks to score and select sentences for summary.
8

---

Under review as a conference paper at ICLR 2023
In abstractive summarization, the summarization model generates every word of the summary itself rather than taking from the article directly (though some models, notably Pointer-generator (See) et al. 2017). do have mechanisms to directly copy). As these models generative, they generally rely on neural nets using typical encoder-decoder architecture. Examples of models that have been used for abstractive summarization include PreSummAAs (Liu & Lapata 2019), the abstractive ver- sion of PreSumm, the language model BART (Lewis et al., (019)), and PEGASUS (Zhang et al.) 2020), a language model designed by Google with summarization in mind. All of these models have obtained SOTA performance at some time or another for summarization on the CNN/DailyMail (See)
al. 2017 Hermann al.. 2015) dataset.
Linguistic and NLP measures for bias There have been various studies and measures of bias in NLP, notably in word, both quantifying the bias in embeddings (Bolukbasi et al. 2016 Caliskan et al. 2017) and using embeddings to measure the bias in human text (Garg et al. 2018 Closely related to our work, Dacon & Liu (202T) measure gender bias in news article abstracts using a variety of computational techniques, finding that they are overwhelmingly biased against women.
In linguistics, studies of bias generally use a combination of lexical analysis and discourse analy- sis to measure how members of different groups talk, how they talked about, and often what stereotypes exist around both "Mendoza-Dentoon 2014 Baker] 2012 Fairclough 2013). Automated analysis restricts ability to techniques that rely on more qualitative analysis, but we draw inspiration from them in our definitions of bias and our experiments.
Properties of Automatic Summarization Dash et al. (219) explore related topic to this paper, measuring faimess in automatically generated tweet collections by user demographic, finding that existing extractive methods exhibit bias toward tweets written by different groups, dependent on the corpus used. Keswani & Celis (2021) perform a similar line of work, exploring dialect diversity in Twitter summaries and finding that summarizers prefer tweets in Standard American English (SAB) over other dialects.
Jung et al. (2019) find that, for news articles, human summaries tend to select information from near the start of articles. Additionally, they show that various summarization algorithms place great im- portance on the position of information. Lee et al. (2022) measure and attempt to mitigate political framing bias in summaries of news articles by encouraging the generation of summaries that syn- thesize political views from a variety of sources. This relates to work from Sharevski et al. (2021) who show it's possible adversarially bias summarizers toward writing in a particular political writing style. Finally, Jørgensen & Søgaard (2021) explore possible bias in summary evaluation by conducting pilot study exploring how user demographic relates to perception of summaries. Their results show that generated summaries are most strongly preferred by older White men, while other demographics rank them more poorly.
CONCLUSION
In this paper have shown that machine learning based summarization algorithms demon- strate bias toward under-represented groups and are highly sensitive to article structure. What groups are mentioned, where information appears in the article, and how important sentences are phrased all have large effects on the generated summaries.
All of these exhibited biases have potential cause harm. Readers rely on summaries to accurately represent longer text, and if summarizer systematically introduces position, sentiment, or content bias this risks both misrrepresnnting the original text and biasing the reader. Although this bias is not intentionally designed into these algorithms, it should be tested and corrected for in future summarization algorithms. Especially as automatic summarization is marketed more and more as both a commercial and personal-use tool, it is important to measure and correct the bias in these systems before deploying them.
While summarizer may obtain high ROUGE score, this does not necessarily mean that the summaries produced will fairly reflect the articles they represent. They may be biased in how they refer to certain groups, in the sentiment they use, in terms of the article structure they favor. We recommend that more bias evaluation methods, like the ones used in this paper, be used when evaluating new summarizers to avoid creating unintentionally biased summarizers in the future.

---

Under review as a conference paper at ICLR 2023
8 ETHICAL CONSIDERATIONS
Any study of bias inherently carries ethical considerations. A summary that amplifies bias already present in the original article or generates new bias has potential to cause harm. Especially as automatic summarization is marketed more and more as both a commercial and personal-use tool, it important to explore the problems this could create. Sharevski et al. (2021) have shown that is possible to bias summary to specific political ideology, and work has shown that even without modification, different summarizers treat groups differently and are sensitive to changes in article format. This raises concerns about summarization that should be kept in mind when designing and evaluating models in the future.
In our analysis consider only one corpus, which contains writing with relatively uniform writing style, and is written in Standard American English. This does not allow us to explore bias in language style dialect, which is another possible axis of bias in summarizers.
Our motivation for using embedding similarity scores to measure group membership is to allow for flexibility that wordlists cannot provide. As word embeddings have been shown encode factual information, using these scores is a good way to achieve this. However, these scores have also been shown to encode human biases and stereotypes, which we found be true in our analysis as well. For example "terror" was the most associated word with the category "Islam" in some examples. While some cases articles may actually be using these words with stereotypical associations to refer to groups, this is not always the case, and could introduce some amount of error. Based on our observations, this risk small, but does represent a possible limitation.
REPRODUCIBILITY
We provide all code used perform analysis, along with the GPT-2 generated corpus. Full details for both how we generated this synthetic corpus, and how we performed our analyses are provided in our appendix. All other data and models publicly available and we do not modify them. Though some of the models we use exhibit randomness in the summaries they generate, we have run sufficient experiments that we confident results could be reproduced.
REFERENCES
Paul Baker. Acceptable bias? using corpus linguistics methods with critical discourse analysis. Critical Discourse Studies, (3):24772566 2012. doi: 10. 011110/744559442222666668
URL
https://ddi.org//33 1080 TTt...10100000011111 201.2001.00 688297
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. The problem with bias: Alloca- tive versus representational harms in machine learning. In 9th Annual conference of the special interest group for computing. information and society, 2017.
Marianne Bertrand and Sendhil Mullainathan. Are emily and greg more employable than lakisha and jamal? field experiment labor market discrinination. American economic review, 94(4): 991-1013, 2004.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media, Inc.", 2009.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1 2020.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334): 2017.
10

---

Under review as a conference paper at ICLR 2023
Jamell Dacon and Haochen Liu. Does Gender Matter in the News? Detecting and Examining Gender Bias in News Articles. In Companion Proceedings of the Web Conference 2021, pp. 385- 392. Ljubljana Slovenia, April 2021. ACM. ISBN 978 -3---444-444... doi: 10. 10.1145/344244.. 3452325. URL httttttss:///c acm. org. 0000:///..114/4444444444 4444444444422222 3452325
10. 11
Abhisek Dash, Anurag Shandilya, Arindam Biswas, Kripabandhu Ghosh, Saptarshi Ghosh, and
Abhijnan Chakraborty. Summarizing User-generated Textual Content: Motivation and Methods
for Fairness in Algorithmic Summaries. arXiv: 0811191470 [cs], September 2019. URL hthhtp:
//arxiv.org/aai http//aab//111..91111
arXiv: 1810.09147.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai- Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM Conference on Faimess, Accountability, and Transparency, pp. 862-872, 2021.
Norman Fairelough. Critical discourse analysis. In The Routledge handbook of discourse analysis, pp. 9-20. Routledge, 2013.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115 (16):E3635-E366 E3644, 2018.
Ian Haney-López. Dog whistle politics: How coded racial appeals have reinvented racism and wrecked the middle class. Oxford University Press, 2014.
Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and compre- nips. hend. In NIPS, pp. 1693-1701, 2015. URL http::// www..cccpppppppeecec cc/paper/ 1845-teaching machines nees to read and comprehend
Dirk Hovy and Shannon L. Spruit. The social impact of natural language processing, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 591-598, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.186553v1/P11-22999
URL http::::s///aaaaahhhho /ttp:///....ttttts/////.a.aaaaaa hta.tt/ss/....aaanoooooooo./////
2096
Tachee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. Earlier Isn't Always Better: Sub- aspect Analysis on Corpus and System Biases in Summarization. arXiv: / 1723 [cs], August 2019. URL httttttt://aaat
wwt/eerg///aa..eeee.eee awt/////////// 1908. 19723
arXiv: 1108.
Anna Jørgensen and Anders Søgaard. Evaluation of Summarization Systems across Gender, Age, and Race. a:X1v:2110.04884 [cs], October 2021. URL http://aaravv.org/abb//22
04384 arXiv: 2110.04384.
Vijay Keswani and L.. Elisa Celis. Dialect Diversity in Text Summarization on Twitter. In Pro- ceedings of the Web Conference 2021, WWW '21, pp. 3802-3814, New York, NY, USA, April 2021. Association for Computing Machinery. ISBN --78----550---12--7 3450108. URL https: hwt. htt.://oo.ooroo///////o////////a 111.11....1111.1 0055/3442338111 1ttpg//00...115542221133333
doi: 10.1145/3442381.
Nayeon Lee, Yejin Bang, Tiezheng Yu, Andrea Madotto, and Pascale Fung. Neus: Neutral multi- news summarization for mitigating framing bias. arXiv preprint arXiv:2204.04990,, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-to-Sequencc Pre- training for Natural Language Generation, Translation, and Comprehension. arXiv: 19/0.1341 [cs, stat], October 2019. URL httt:://axxivvoorg/a//////11 13461 1910.13461.
arXiv:
Yang Liu and Mirella Lapata. Text Summarization with Pretrained Encoders. arXiv: /908:/90..00 08345 [cs]. September 2019. htt.tttpp//aii.... www./bsg.......o...o 1998. 08345 arXiv:
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. Black is to criminal casian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv: 1904. 04047, 2019.
11

---

Under review as a conference paper at ICLR 2023
Norma Mendoza-Denton. Homegirls: Language and cultural practice among Latina youth gangs. John Wiley & Sons, 2014.
Rada Mihalcea and Paul Tarau. TextRank: Bringing Order into Text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404-411, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv: 2013.
Slav Petrov, Dipanjan Das, and Ryan McDonald. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation Istanbul, Turkey, May 2012. European Language Resources Associ-
(LREC' 12), pp.
ation (ELRA). URL http:://tp/tt 1rec- conf http:://www..e.....ooooooo////e/ ort.e//..o.....oooooceeeeeeee/ea
274_ S74_Paper.pple
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18665/111117110999 URL https:
7/ //cclantholooy..rr//P1111
Filipo Sharevski, Peter Jachim, and Emma Pieroni. Regulation TL;DR: Adversarial Text Summa- rization of Federal Register Articles. In Proceedings of the 3rd Workshop on Cyber-Securing Arms Race, CYSARM '21, pp. 37-42. New York, NY, USA, November 2021. Association for Computing Machinery. ISBN 978- 1 1-4503-866116. doi: 10.1145/34443344448611 URL
pettps:/ddi..org orr/ 1145 /43554447477... 3486917
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: Sfate-of-the-att natural language processing. arXiv preprint 1r11v:1910.0377,, 2019.
Jingging Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. arXiv:1912.08777 [cs], July 2020. URL
http:://rxiv.oor//aa
08777 arXiv: 1912.08777.
Ming Zhong. Pengfei Liu, Yiran Chen, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. Extractive Summarization as Text Matching. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6197-6208, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/20220//1220....// acl- main.552. URL https: atcps::/aclanthology.... org/ apl- 552
A TYPES AND EXAMPLES OF CONTENT BIAS
Table Potential ways in which summarizer could be biased with respect group. Highlighted original text is text not selected that highlights the bias.
Category
Description
Original Text
Summary
Continued next page
12

---

Under review as a conference paper at ICLR 2023
Table Potential ways in which summarizer could be biased with respect group. Highlighted original text is text not selected that highlights the bias. (Continued)
Over-representatioon
Including more about one group than another relative to the ratio in the original text
The women's team scored very high in the competition. They practiced for many months to get here and it paid off. Their scores set new record. The men's team did pretty well too. They heat few other teams.
The women's team scored very high in the competition. The men't 's team did pretty well too. They beat few other teams.
Inclusion/cxclusion
In/excluding detail only when a specific group is involved
Both the women's and men's figure skating teams are headed the Beijing olympics. They will compete throughout February. Both are expected to win lots of medals. We've gotten a their
Both the women's and men's figure skating teams are headed to the Beijing olympics. Both expected to win lots of medals. The women's costumes were very pretty.
chance to costumes early. The women's costumes are very pretty. The men's s costumes are also very good.
Inaccuracy
Including information in way that inac- curately reports infor- mation hallucinat- ing inaccurate infor- mation (for abstractive models) about spe- cific group
What to do some- one's having seizure: Move them to safe position. Roll them to their side.
What to if some- one's 's having seizure: Hold them down. Put something in their mouth.
It's also important to know what not do to do. Here's list of things not to do if someone's having seizure: Hold them down. Put some- thing in their mouth.
Continued on next page
13

---

Under review as a conference paper at ICLR 2023
Table Potential ways in which summarizer could be biased with respect group. Highlighted original text is text not selected that highlights the bias. (Continued)
Sentiment bias
Using more positive sentiment sentences for one group than another
Two common pet choices are dogs and cats. One may be better than the other depending lifestyle. Dogs are very loving and make great pets, Dogs re- quire lot of attention and care and need be walked every day. Cats can be very affectionate and are great pets. Cleaning up after cats is also a lot of work.
Two common choices are dogs and cats. Dogs require lot of attention and care and need to be walked every day. Cats can be very affectionate and are great pets.
your
Framing bias
Using different sen- tence structure to frame a situation when group is involved vs another
police officer fatally shot man at 8pm last night. An officer's ss gun discharged, killing the man and injuring another last night.
An officer's gun dis- charged, killing the man and injuring an- other last night.
14

---

Under review as a conference paper at ICLR 2023
B SYNTHETIC CORPUS GENERATION
We generate a synthetic corpus of articles using GPT-2 (Radford et al., 2019) and prompts from the We refer these articles as single-group arti-
BOLD dataset (Dhamala et al. 2021) for each group O cles, as they are prompted to generate articles about each group. We also create multi- articles by sampling different amounts of contiguous sentences from single-group articles and combining them to form a new article with information about tWO groups. We generate multi-group articles for each combination of two groups within each of our three group categories, constructing one version with the first group at the beginning of the article, and another with this group at the end of the article (using the same sentences). This allows to explore the effect of different group combinations and amounts of information while controlling for the importance of the order information appears in.
We generate our text using the version of GPT-2 available on the HuggingFace Hub (Wolf et al. 2019 as of September 2022 using the following parameters:
min_lengthh=00
max max.lengthh 0.2
top_k=40
top.p=0.99 95
no.repeat.ngrams
Additionally, for each group, we set bad_words _ids to the union of all other groups' keyword lists, which we find helps keep articles "on track" about the group they' 're supposed be about. Without this, for example, many articles about women talk mostly about men and vice versa.
C CONTENT BIAS ANALYSIS DETAILS
C.I
WORD LISTS
We use lists of representative keywords for each of the nine groups we study. These lists used for both filtering articles and the group analysis we perform using word embeddings. The lists for the male, female, Hispanic, Asian, Islam, and Christianity groups are taken from Garg al. (2018) with some minor changes made to better suit the corpus. For example, we remove the word "cross" from the Christian word list as it is often used outside the context of Christianity in our corpus. We use the same lists as Caliskan et al. (201)) 7) for the White and Black group lists (originally from Bertrand Mullainathan (2004)), which uses first names rather than last names as done for the other racial groups. This is because, as Garg al. (2018) describe, there is very large overlap in the typical last names of white and Black people in the US owing large part to the country's history of slavery. For the Jewish group, we use the list used by Manzini et al. 2019).
C.2 GROUP SIMILARITY SCORES
We generate embeddings using a word2vec model (Mikolov et al., 2013) pre-trained on the Google News corpus. Before measuring similarity, we tokenize and POS tag each document with NLTK (Bird et al. 2009) and the universal tagset (Petrov et al. 2012) (this allows us to filter words by POS, which explain further in Section 3.1.. Then, we measure the group similarity for each token embedding, t in the text as follows:
s,(t) = G.
Σ s(t,g)
gEG,
Where G, is ith set of group keywords, and s(1, g) is the cosine similarity between the two token embeddings.
We collect these scores for each word in both the original article and its summary and aggregate them into text association scores, described below.
For detailed information on generation parameters, see the appendix.
15

---

Under review as a conference paper at ICLR 2023
We observed that, though these scores they can be quite noisy for words with low scores. This noise comes as a result of many words in each article having baseline similarity to people (something all the groups have in common) and results in the majority of words in an article receiving quite low scores, while the minority, most indicative of group presence in the text, receive very high scores. Therefore focus only the high scoring words to determine group membership and discard lower scores as uninformative.
good indicator of the groups present in article,
C.3 THRESHOLDS
The thresholds used determine high scoring words were calculated using similarities of each group list to the other group lists. For each group, we take the median similarity of the most similar alternate group to be the threshold above which we consider words important. practice, we find that this reduces the number of words that really belong to group that given falsely high association scores for other groups.
C.4 ARTICLE FILTERING
As an additional filter, we restrict our analysis articles that contain at least one word from least one of our group keyword sets. This is to ensure that at least one group is truly present in the article, and does not significantly reduce the number of articles we are examining (the original CNN DM dataset is 311 1199 articles, and with this filter, we analyze 296,460 articles). We recognize that bias could still be present a group is not explicitly mentioned; our group keywords are not all encom- passing and there are many implicit ways to signal group information (c.g.. doggwhistling Haney- López (2014)). However, manual linguistic analysis far more suited for exploring this type of bias, and we find that this filtering makes analysis more reliable.
D PARAPHRASING ANALYSIS DETAILS
We use the pre-trained PEGASUS paraphrased from HuggingFace (Wolf al. 2019) for these experiments and run the analysis on the test split of CNNiDailyMail (11k articles). We create five versions of each paraphrased article in order to measure the effect of different paraphrases which sentences are selected for an article.
E ADDITIONAL LOCATION ANALYSIS PLOTS
As discussed in the main paper, we find that Presumm, MatchSum, and Azure show a strong ten- dency select sentences that occur early in articles-stroneee than that demonstrated by the refer- ence summaries. In contrast, we find that GPT-3 prefers information that comes at either the start or the end, with much stronger preference for information the end than shown by reference summaries.
Note that Presumm and MatchSum have relatively short length limits of 512 tokens, which means they are unable to select sentences from the end of long articles. This impacts these figures, and is the reason for the dropoff frequency between 0.4 and 0.6 in their plots. However, if this were the only effect, we would expect that there would be more sentences selected between 0.1 and 0.4 for both summarizers, and not the sharp spike in sentences taken from the first 2% of the article, as we with MatchSum and Presumm.
http:://wuugiiigfaacccccuu/n//// paraphrase
16

---

Under review as a conference paper at ICLR 2023
0.08
007
0110
0.05
Probabiltt
0.75
Pobabiity
0.05
0.000
0.04
0.02
0.50
0.0
0.05
0.00
0.000
04
0.
04
05
08
Articl Quantile
Article Quantile
(a) TextRank
(b) MatchSum
0.12
0.00
0.05
Pobaabity
0.08
Pobabilty
0.06
0.04
0.07
0.
0.
0.4
06
0.
1.0
0.0
02
04
06
0.
10
Artide Quantile
Article Quantile
(c) Presumm
(d) BART
0.07
0.06
0.10
Robabiity
005
0.08
0.04
0.03
0.04
00
0.2
0.4
05
1.0
0.0
0.6
0.8
10
Article Quantile
Article Quantile
PEGASUS
(f) Azurc
0.06
0.07
0.05
0.06
Probabiity
Pobabilyy
0.04
0.03
0.00
0.02
0.02
0.00
0.
Article Quantile
Article Quantile
(g) GPT-3
(h) Reference Summaries
17

---

Under review as a conference paper at ICLR 2023
E.1
GENDER
E.1.1
RECURSIVE SUMMARIZATION
Representation Score
Women
0.5
Men
2
3
4
5
7
8
9
10
Step
(a) Articles about men
Representation Score
Women
0.5
Men
0
3
4
7
8
10
Step
(b) Articles about women
Figure 7: Ratio of information present about each group relative the highest group information present in the article. A score of for a group and summarizer means that a reader could expect to that group with the most information in every summary we generated using that summarizer. A score of 0.4 means the reader could expect to see 40% of the information about that group vs. the group with the most information in each summary.
Here we observe similar patterns to that seen in single group summarization. Abstractive summa- rizers show lendency first gain, then lose information about men for articles about women, and lose and then gain information about women. However, for articles about men, the amount of information about both groups stays relatively constant across steps of summarization.
E.2 RELIGION
For single group articles about different religions, observe that while articles about Islam gen- erally preserve the baseline amount of information from the article, summaries of articles about Judaism and Christianity tend lose information about their respective groups (shown in Fig. ( However, unlike in the gender case, the representation scores of other groups do not tend to increase, suggesting that the overall information about religion in these summaries being decreased, rather than the emphasis being shifted to another religion.
Similar patterns what see for gender also hold for religion. multi-group articles about Judaism and Christianity, regardless of which group has more information in the original article, the prioritized group in the summary has more do with which information presented first. As shown in Figs. and 10 all summarizers but GPT-3 show preference for text that appears earlier the article.
18

---

Under review as a conference paper at ICLR 2023
Representation Score
0.8
0.6
0.4
0.2
textrank
azure
bart
pegasus
2p13
textrank
azure
bart
pegasus
3p13
(a) Judaism
(b) Islam
Judaism
Islam
Christianity
textrank
azure
bart
pegasus
EPL3
(c) Christianity
Figure Representation scores for summaries 50% the length of articles about single groups. Dashed lines are baseline representation scores of original articles, while bars scores of summaries. Error bars are the 95% confidence interval for each score.
19

---

Under review as a conference paper at ICLR 2023
Representation Score
0.8
0.6
Judaism
Islam
0.4
Christianity
0.2
textrank
presumm
matchsum
bart
pegasus
8p13
textrank
presumm
matchsum
bart
pegasus
BpL3
(a) Sentences about Islam first
(b) Sentences about Judaism first
Figure 9: Representation scores for summaries 10% the length of articles. Articles comprised of 50% sentences about Judaism and Islam. Dashed lines are baseline representation scores of original articles, while bars scores of summaries. Error bars the 95% confidence interval for each score.
20

---

Under review as a conference paper at ICLR 2023
Representation Score
0.8
0.6
Judaism
Islam
0.4
0
Christianity
0.2
textrank
presumm
matchsum
bart
pegasus
gpt3
textrank
presumm
matchsum
bart
pegasus
gp13
(a) 90% sentenees from articles about Islam, then
10% sentences from articles about Judaism, then 90% from articles about Islam
10% from articles about Judaism
Representation Score
0.8
0.6
D Judaism
Islam
0.4
Christianity
0.2
textrank
presumm
matchsum
bart
pegasus
8p13
textrank
presumm
matchsum
bart
pegasus
8pt3
(c) 90% sentences from articles about Judaism, then 10% from articles about Islam
(d) 10% sentences from articles about Islam, then
90% from articles about Judaism
Figure 10: Representation scores for summaries 10% the length of unbalanced articles about Ju- daism and Islam. Dashed lines are baseline representation scores of original articles, while bars scores of summaries, Error bars are the 95% confidence interval for each score.
21

---

Under review as a conference paper at ICLR 2023
E.3 RACE
For race, the text association scores have several problems. While they accurately indicate the presence of information for both the Hispanic and Asian groups, for the Black and White groups, find that the scores are not good indicators of what information is there. Black scores are consistently high for articles about all groups, and especially correlate with White scores. This same problem has been observed in other work that uses word lists based on names for race (Garg et al. 2018), and is not surprising. However, it does mean that we cannot report definitive results for race using our scores.
22


Reviews:
[{"review_id": "GgxAmmgVt2", "reviewer": "Reviewer_4JiR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at the problem of bias in automatic summarization algorithms. The authors design a method to quantitatively measure the extent to which an article's attributes, such as structure, paraphrasing, and article content influence _bias_ in generated summaries. They also study the causes of _bias_ by varying the influential factors, including the summarization parameters and distribution of input documents. The findings indicate that machine learning based summarization algorithms can introduce position or content bias, which can lead to misrepresentation of the original text and/or biasing the reader. The authors recommend that more bias evaluation methods be used when evaluating new summarizers to avoid creating unintentionally biased summarizers in the future.", "summary_of_the_review": "The paper is well motivated, but the exposition is confusing and several details are missing. The authors rely on the premise that a certain way of choosing a summary is wrong, but they do not show what the alternative would be practically and whether that is indeed more preferable to an end user. The paper appears rushed, with some claims presented as facts without any supporting data or citations. It is also difficult to tell how one could replicate the experimental setup and reproduce the results given the sparse details.", "strength_and_weaknesses": "## Strengths:\n- Paper is well motivated. The scope of text summarization methods in practice points to a need to pay much more attention to this question.\n- Some of the results offer interesting insights. Such as results in Figure 2, as it shows model reliance on what comes early in an article. But there are also weaknesses in this analysis as I share below.\n\n## Weaknesses:\n- The exposition is confusing, several details are missing and assumptions are made without declaration. There is no qualitative analysis to help the reader better follow how the authors are changing structure or content, or what a synthetic summary looks like, or what does it mean to be an article about _men only_ but still have sentences about _women_ in them (Figure 3) or whether the authors' several perturbation strategies aren't altering valuable information?\n- The paper relies on the premise that a certain way of choosing a summary (say weighing what comes earlier in an article more), given an article is wrong. But it does not show what the alternative would be and whether that alternative is indeed more preferable to an end user. It is hard to say that given two different texts to an end user (since these are different as they've been reordered), they would somehow still write the same summary for both articles. There is also no end user analysis showing that these _biased_ summaries from a model that rely more on what comes earlier in the article are offering less value to the end users than what a counterfactual _unbiased_ summary would offer. Experiments in Figure 2 come close to this point but it does not come clearly. Summarization is not an objective task, so the gold standard is to show resulting summaries to humans and see whether they prefer one model or the other. But that's hard as the authors don't characterize what such an _unbiased_ summary would be?\n- The paper appears rushed. Some examples: Figure 7 has no legend, Table 3's caption says some text is in red and other in blue, but the only color there is yellow. On Page 3, its said that the generation details for the synthetic corpus are in Section 5, on Page 4 its said that they are in Appendix B, but it seems they are in neither. Appendix B comes close to sharing some information but after reading it, I could still not tell how someone could replicate this synthetic data generation because details are sparse. Also, a footnote in this paragraph says that more details are present in the Appendix?\n- Several claims have been presented as facts but no citations or data are offered to support them. Some examples: \n  - Introduction: \"Readers, however, expect that summaries faithfully represent articles.\" First, what do you mean by faithful in this context? Second, if this is the case, please add at least one behavioral science citation to this. \n  - Footnote 3: What notion of fairness does this relate to? \n  - Section 3.3: \"While we could expect a human to generate a summary...\" What is the evidence to support this expectation? Is there any data, any pilot study you did, any previous work that looks at this?\n  - Section 5: \"We conduct manual analysis...\" No details are shared on this analysis and no results are presented.\n- I have read the experiments and results section several times, yet I can't tell whether I would be able to replicate the setup and reproduce these results given how sparse the details are.\n  - There are error bars in the plots. What do these represent?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GF77iASXDcC", "reviewer": "Reviewer_KeGL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes the typeAs of biases that can appear in automatic\ntext summarization and presents an initial study of such biases using\ntwo corpuses: a synthetically generated one and the CNN/Daily mail\ncorpus.\n\nThe paper selects a few types of biases that can be analyzed\nautomatically and proposes metrics (or proxies) for assessing biases,\nsuch as content bias wrt representation and structure biases.\n\nAcross several models analyzed (both extractive and abstractive),\nthere seem to be a general tendency to minimize the representation of\na group over the other. Similarly, most summarizers (with the\nexception of GPT-3) seem to prefer sentences appearing at the\nbeginning of the text over sentences at the end of the text.", "summary_of_the_review": "The paper is a first to analyze bias in automatic text summaries and\nit would improve in quality if a human validation of metrics used and\nresults obtained would be included. I strongly encourate the authors\nto include such a validation.", "strength_and_weaknesses": "Strengths:\n- first study that I am aware of that looks at bias in text summarization\n- discussion on types of biases possible in text summarization\n- an intial study of bias in text summarizers\n\nWeaknesses:\n- I'm not convinced whether the proxies/metrics for measuring content bias are accurate\n- Lack of some human validation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9e14O4Ur94J", "reviewer": "Reviewer_tjG5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the presence bias introduced by text summarization models. The authors propose to measure bias in two dimensions: content and structure. They define content bias as tendency to mention a specific group (e.g., gender, religion, etc) in a text, whereas structural bias refers to bias as a result of structural features of the text (position, sentiment, and style). In this paper, they limit the scope of investigation to one type of content bias (underrepresentation) and two types of structure bias (position and style). To measure content bias, a representation score R(T, g) is proposed to gauge the proximity of a group g to a text T. For position bias, the content of model-generated summaries and reference summaries is compared according to its position in the input documents. Finally, for style bias, the authors measure the impact of paraphrasing sentences in the original document, under the assumption that the same content should be selected regardless of the style of writing. Those criteria were applied to a synthetic dataset generated by GPT-2 and the CNN/DM news summarization dataset. Based on the results, the authors conclude that summarization models exhibit preference for certain groups over others, amplify patterns of bias, and demonstrate sensitivity to the structure of articles.", "summary_of_the_review": "The claims that summarization models exhibit content bias are weakly supported by the experiments. In terms of structural bias, this paper finds that summarization models trained on news articles have lead bias, which is a well-known fact in the literature. For the lack of novelty and solid evidence for the claims, the recommendation is for the paper rejection.", "strength_and_weaknesses": "Strengths:\n1. The paper addresses an important and underexplored topic in the automatic summarization.\n2. The experiments show that under similar conditions of positioning in the source documents, summarization systems demonstrate preference for groups, for instance, by amplifying content related to men (versus women).\n\nWeaknesses:\n1. The content bias analysis is based on a synthetic dataset generated by GPT-2, so that the summary inference was performed on out-of-domain data. If summarization models are sensitive to paraphrasing and structural features, then synthetic documents may also exhibit artifacts that influence the results. It would be important to perform a similar analysis on selected documents from CNN/DM. Alternatively, a human evaluation experiment using the synthetic data could provide more evidence that the bias is really introduced by the summarization models.\n2. The position bias analysis, the authors state that \"clearly amplified by MatchSum, PreSumm, and Azure, as shown in Fig. 4b.\" However, it is not clear if the observed pattern is caused by model preference or simply by truncation of inputs. If we observe the figure in Appendix E, the distribution for PEGASUS and BART are more similar to the reference summaries distribution. In contrast, MatchSum, Presumm, and Azure decrease the frequency sharply between 0.4 and 0.6. Interestingly, Presumm and MatchSum use the same backbone model with a maximum input length of 512 tokens, whereas PEGASUS and BART support inputs of 1024 tokens, which might explain their capacity to select more content from the later positions of the articles. For reference, CNN/DM input documents have 766 words on average (\"Extractive Summarization as Text Matching\", Zhong et al., 2020). Do the quantile calculation in the figures take into account this truncation effect?\n3. No details or citation is provided for the paraphrasing model. Just the link for its checkpoint at https://huggingface.co/tuner007/pegasus_paraphrase.\n4. Authors mention that they \"conduct manual analysis on a randomly selected subset of the article\" but no further details are provided.\n5. Experiments are based only on CNN/DM, which is well known for its lead bias for important content. Experiments on additional datasets could make the claims stronger.\n\nMinor comments:\n1. Typo in page 6: \"Overall, t hese results show a pattern of unpredictability...\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "23hKi56wqza", "reviewer": "Reviewer_45gx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper draws attention to the problem of Bias Analysis in text summarization tasks which has not been addressed before. It defines the different types of biases present in summarisation task. It performs Bias Analysis using both abstractive and extractive automatic summarising models. The experiments suggest biasness depicted by these models", "summary_of_the_review": "The paper lacks comprehensive evaluations using different embeddings, models and datasets. Further, the paper can be hard to comprehend due to lack of explanations of figures and curated examples.\n\n============After rebuttal============================\n\nThe authors acknowledge the limitations pointed out, however the paper is very rushed and incomplete in terms of experimentation and their results. Adding the suggested results would definitely improve the quality of the paper, therefore I encourage the authors to do. However, it is still a reject from me with no changes in the previous score.\n", "strength_and_weaknesses": "Strengths\nIt addresses a significant problem largely ignored by the Automatic Text Summarisation Community of Bias Analysis. \nIt points out potential risks due to the ignorance of Bias analysis in Text Summarisation.\nDefines different types of biases in Automatic Text Summarisation.\nGood initial exploration to push the research in this direction.\nWeaknesses\nThe paper does not compare the results of automatic summarizers with human curated summarization.\nOnly used single embedding word2vec (word embeddings can themselves be baised) and only considered one dataset CNN/Dailymail, without explaining any rationale behind the selection. To add value, authors should have explored other text summarization datasets with different writing styles to really bring out the different style biases rather than just paraphrasing.\nThe authors only considered a few models, ignoring sota models on CNN/Dailymail dataset. It may have been possible that the SOTA model behaves differently.\nThe paper lacks any association/description between the metrics used for bias in this paper v/s different text summarisation metrics. It would have been nice to consider why and how these metrics are different to give a better understanding.\nPaper lacks any results about biases based on religion and race even though they claim to have done so in the initial section of the paper. Only one incomplete figure about religion is present which is hard to comprehend as it lacks any legend.\nThe results and figures need to be explained further. Explaining the ideal score for an automatic summarizer in the figures would have given a better idea. I did not really understand the dashed lines in the figure given as Baselines. An explanation with an example will go a long way in making things much more clear and bringing out the results effectively.\nFirst it claims no work has been done on bias-ness in text summarization then it points out a previous work on structure biasness (Jung et al, 2019).\nIn the paraphrasing of articles, the authors do not show how good the paraphrased sentences are. This raises the concern that if the paraphrased version of sentences itself were curated in a manner that they did not include proper information. A human evaluation of a few paraphrased sentences would have cleared this issue. Also there are no examples from the synthetic dataset curated by the authors.\nIn the Fig.3, two of the extractive summarisers are not mentioned namely presumm and matchsum\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject