Bias in News Summarization: Measures, Pitfalls and Corpora (2024)

Abstract: Summarization is an important application of large language models (LLMs). Most previous evaluation of summarization models has focused on their content selection, faithfulness, grammaticality and coherence. However, it is well known that LLMs can reproduce and reinforce harmful social biases. This raises the question: Do biases affect model outputs in a constrained setting like summarization? To help answer this question, we first motivate and introduce a number of definitions for biased behaviours in summarization models, along with practical operationalizations. Since we find that biases inherent to input documents can confound bias analysis in summaries, we propose a method to generate input documents with carefully controlled demographic attributes. This allows us to study summarizer behavior in a controlled setting, while still working with realistic input documents. We measure gender bias in English summaries generated by both purpose-built summarization models and general purpose chat models as a case study. We find content selection in single document summarization to be largely unaffected by gender bias, while hallucinations exhibit evidence of bias. To demonstrate the generality of our approach, we additionally investigate racial bias, including intersectional settings.

heading: 1 Introduction
text: Pretrained large language models (LLMs) have increasingly found application across a wide variety of tasks, including summarization (Lewis et al., 2020; Zhang et al., 2020; Goyal et al., 2022). While such models often evaluate favourably especially in human judgement for content (Goyal et al., 2022), it is also well known that pretrained language models can often carry undesirable social biases (Dinan et al., 2020; Liang et al., 2022; Bommasani et al., 2021). This raises the prospect of considerable harm being caused by their practical application.
However, often these biases are studied in settings where model inputs are specifically crafted to reveal social biases (Rudinger et al., 2018; Sheng et al., 2019; Parrish et al., 2022). Biases are also often observed in relatively unconstrained settings, such as dialog (Dinan et al., 2020) or the generation of personas (Cheng et al., 2023). While insights won in this way are highly valuable in understanding the potential negative impacts of LLMs, it is not always clear how they map to other applications.
Summarization in particular is a highly conditional task. While there are many ways to summarize a document, the input document limits the entities and facts a model can work with. This might, intuitively, reduce how many new biases a model can introduce, as long as it is faithful.
This leads us to ask: How can we study bias in text summarization? and To which extent do current models exhibit biases when applied to text summarization? We focus on gender bias in English since it is a well-known issue in LLMs (Zhao et al., 2018; Dinan et al., 2020; Saunders and Byrne, 2020; Bartl et al., 2020; Honnavalli et al., 2022, among others) and has grammatical indicators in English, making it an useful phenomenon to develop fundamental methodology for studying bias in text summarization. We run our experiments in a single document news summarization since it is a popular task (See et al., 2017; Narayan et al., 2018; Lewis et al., 2020; Zhang et al., 2020) which is performed well by current models (Goyal et al., 2022) and also a likely application of summarization models. To show the generality of our approach, we conduct additional experiments on racial bias.
While an ideal evaluation would be conducted on naturally occurring data, we find that it is difficult to disentangle biases that are present in the summaries from biases that are already in the input documents. We thus propose a procedure that exploits high-quality linguistic annotations to generate mutations of real-world news documents with
5962
controlled distribution of demographic attributes. We make the following contributions:1
1. We propose and motivate a number of definitions for bias in text summarization and include novel measures to assess them.
2. We highlight the importance of disentangling input driven and summarizer driven biases.
3. We conduct practical gender bias evaluation of both purpose-built summarization models and general purpose chat models for English.
4. We demonstrate that our measures can be used to study other biases by also evaluating racial bias in these models, including intersectional scenarios with gender.
We find that all models score very low on bias in their content selection functions. That is, we find no evidence that the gender of an entity influences the salience of that entity within the summarizers’ content models. Where gender bias occurs, it is often linked to hallucinations. For racial bias we find largely comparable results. Figure 1 shows a schematic overview of our approach, along with an example of a gender-biased hallucination.

heading: 2 Bias in Text Summarization
text: Bias in NLP is an overloaded term, which is not always used with a clear definition (Blodgett et al., 2020). Before we continue, we thus need to establish our expectations for an unbiased summarizer.
One approach chosen, for example, by Liang et al. (2022) is to require that all demographic groups receive equal representation in the generated summaries, following an equality of outcome
1All code is available at https://github.com/ julmaxi/summary_bias.
paradigm (Hardt et al., 2016). While a valid perspective, it requires models to actively counteract biases that might be present in the input documents. This is at odds with faithfully representing their content and would thus likely reduce summarizer utility. We instead expect summarizers to be faithful to the input but to not amplify their bias. We define three forms of bias under this setting and discuss their harms (Barocas et al., 2017): inclusion bias, hallucination bias and representation bias.
Inclusion bias captures the idea that the (apparent) membership of an entity in some demographic group should not influence how likely that entity is to be mentioned in a summary. If we frame content inclusion in terms of a classification problem over the content units in a document, this corresponds to demanding equality of opportunity (Hardt et al., 2016), as opposed to equality of outcome. For example, if both a male- and a female-coded entity are mentioned with otherwise similar salience in a document, the resulting summary should not be more likely to mention the male-coded entity than the female-coded entity, or vice versa. Inclusion bias is thus a property of the summarizer’s content selection mechanism. Inclusion bias poses a form of allocative harm (Barocas et al., 2017) since it reduces visibility of members of certain groups if, for example, news is consumed through the filter of automatic summarization.
Abstractive summarization systems suffer from hallucinations (Kryscinski et al., 2020; Cao et al., 2022), that is summary content unsupported by the input. If one demographic group is more likely to feature in them, this would lead to an overrepresentation of this group and entail harms similar to inclusion bias. We call this hallucination bias.
The above measures can not capture all kinds of possible bias. As an additional canary, we fi-
nally also measure Representation Bias, which intuitively measures any kind of summary deviation based on which groups are mentioned in the input. A system exhibits representation bias if it produces different summaries for similar content that relates to different groups. This includes content only included for some groups, entities having different salience in the summary, and differences in summary quality. By definition, the presence of any other biases, except hallucination bias, requires the presence of representation bias, but it does not necessarily entail any harms itself. In English texts, for example, we would expect some level of gender representation bias for grammatical reasons.
We want to emphasise that we do not claim that our definitions are universal. They specifically assume we want a summarizer that faithfully reflects the input, regardless of any potential biases therein.

heading: 3 Bias Measures
text: We operationalize our bias measures for a set of demographic groups G. Note that, while in our experiments we only instantiate G as a pair of two groups, all measures generalize to multiple groups.

heading: 3.1 Inclusion: Word Lists
text: A common way to measure bias in text generation is via word lists. For example, Liang et al. (2022) use word lists to evaluate gender bias in LLMs for a variety of tasks, including summarization on CNN/DM (Hermann et al., 2015) and XSum (Narayan et al., 2018). We assume word lists Wg that identify mentions of each relevant demographic group g ∈ G, and refer to these words as identifiers in the remainder of this work.2
We then compute the frequency of identifiers in Wg in the set of summaries S: cnt(Wg, S), deriving an empirical distribution over group identifier frequency Pobs(g) =
cnt(Wg ,S)∑ g′∈G cnt(Wg′ ,S) , g ∈ G. The bias measurement is the total variation distance between Pobs and a reference distribution Pref.
As Liang et al. concentrate on equality of outcome (see Section 2), they set Pref as uniform. We instead take Pref as the input distribution computed on the source documents to measure inclusion bias.

heading: 3.2 Inclusion: Entity Inclusion Bias
text: While word lists are a convenient tool for measuring bias when we know little about the target
2We use the same male and female lists as Liang et al. (see Appendix A) but the definition is not list-dependent.
domain, the lists must be curated manually, which limits the phenomena they can capture. In summarization, we expect that the inclusion and exclusion of entities3 may often be a useful proxy for determining bias. As stated in Section 2, the content selection function of a system without inclusion bias should not be influenced by the group membership of entities in the input. More formally:
∀gi, gj ∈ G : p(e ∈ S|g(e) = gi, e ∈ D) = p(e ∈ S|g(e) = gj , e ∈ D) (1)
where e ∈ D, e ∈ S indicates that an entity e is mentioned in the source document and summary, respectively and g(e) = gi indicates that entity e is marked as a member of a demographic group gi.
We quantify this as the maximum odds ratio between the inclusion probability of two demographic groups. This allows us to compare summarizers with different overall entity density in their summaries. Let pgi = p(e ∈ S|g(e) = gi, e ∈ D). The inclusion bias score then is
max gi,gj∈G
pgi 1−pgi pgj 1−pgj − 1 (2)
where an unbiased system receives a score of 0.

heading: 3.3 Hallucination: Entity Hallucination Bias
text: We operationalize hallucination bias by demanding that the probability of a hallucinated entity belonging to a particular demographic group is the same for all groups:
∀gi, gj ∈ G : p(g(e) = gi|e ̸∈ D, e ∈ S) = p(g(e) = gj |e ̸∈ D, e ∈ S) (3)
We measure the total variation distance between p(g(e)|e ̸∈ D, e ∈ S) and the uniform distribution, since hallucinations introduce new entities, as opposed to reproducing input entities.

heading: 3.4 Representation: Distinguishability
text: Representation bias demands indistinguishability of summaries generated for similar inputs that discuss different demographic groups. We operationalize it by creating a classifier to identify which group is discussed in the input from the summary.
Let S be a set of summaries generated from inputs where each input primarily discusses one of
3We use entity exclusively with reference to persons
the demographic groups of interest and where content is independent of the group mentioned in the input. Let ui = 1|Sg(si)|−1 ∑ sj∈Sg(si)\{si}
sim(si, sj) be the average similarity between a summary si and all summaries Sg(si) that have been generated for inputs with the same demographic group that is predominant in si. Similarly, let ūi be the same for the set of summaries generated for different demographic groups. We say si is distinguishable if ui > ūi and compute the distinguishability score as the zero-centered accuracy score of this classifier:
2
|S|
|S|∑
i
1(ui > ūi)− 1 (4)
The metric is parameterized by a similarity function. We use cosine similarity with two representations: A bag of words based representation, and a dense representation derived from Sentence BERT4 (Reimers and Gurevych, 2019). To avoid distinguishability via simple grammatical cues and names, we replace all pronouns with a gender neutral variant (they/them etc.) and names with the markers FIRST_NAME/ LAST_NAME.

heading: 4 Input Documents are Already Biased
text: All proposed measures, except hallucination bias, require us to isolate the effect of a particular demographic group in the input. However, with real
4We use the all-MiniLM-L6-v2 model.
world data it is difficult to disentangle input driven biases from biases introduced by the summarizer.
To demonstrate, we investigate the frequency of gender identifiers from our inclusion score word lists Wg on CNN and XSum inputs. We find that 62%/74% of identifiers are male for CNN/XSum, i.e. men are mentioned at a much higher rate.
While this simple frequency issue would be mitigated by our formulation of inclusion bias that takes the input distribution into account (see Section 3.1), we find that the underlying issue goes beyond just mention frequency. To demonstrate this, we split the data into two sets, one, where the frequency of female identifiers is higher, and one, where the frequency of male identifiers is higher. We then apply the Fightin’ words method (Monroe et al., 2017) with an uninformative Dirichlet prior (α = 0.01) to identify words that have a significantly different frequency between male and female texts.5 Since all identifiers have paired male/female variants, we replace these pairs with special markers. This allows us to compare the frequency of the male/female variants (e.g. “mother” being more frequent in documents tagged female than “father” in documents tagged male).6 We show results in Table 1.
Ignoring the titles (Mr./Mrs./Ms.), we see that a number of words have highly significant z-scores (z ≫ 1.96). Specifically, in both corpora the male documents are much more likely to mention sportsrelated words7, while documents with more female identifiers have much higher occurrence of words related to family like husband, children etc.
We demonstrate the consequences of biased input by examining word inclusion bias of clearly biased and unbiased summarizers. We consider two content-agnostic baselines: Random selects three random sentences. Lead selects the first three. We also study two content-aware summarizers. For this we heuristically classify every article as either mentioning more family or more sport based keywords or neither (unknown). Details can be found in Appendix B. Topic randomly samples one, three or six sentences when the article is classified as family, unknown, or sport respectively. Sexist selects three sentences to maximize the frequency of
5This corresponds to computing the log-odds ratio of token frequencies with a small smoothing factor and then dividing by their standard deviation to receive a z-score.
6We ignore the pronouns him/her/his/hers in this context due to the POS ambiguity of “her”.
7This includes the parentheses, which are frequently used in sport reporting, e.g. for results.
male identifiers for sport and of female identifiers for family articles, acting randomly otherwise. The latter is clearly the most biased, while neither Random nor Lead can, by construction, amplify bias. Any bias in Topic is a correlation of topics with gender in the input, not due to the algorithm.
We evaluate with word list inclusion bias, since we neither have reliable entity annotation for the CNN/DM or XSum corpora, nor, as our analysis shows, an independent distribution of content and gender as required for distinguishability. Results in Table 2 highlight that: a) Without correction for the input distribution, Random, Lead and Topic appear highly biased, while Sexist the least biased. The latter is a consequence of it barely decreasing female representation in sport documents, where representation is already low in the input, but boosting it in summaries for family related articles. b) Even our proposed correction, Topic scores higher on bias than Sexist, which clearly does not represent the bias of the underlying algorithms.

heading: 5 Gender Bias Experiments

heading: 5.1 Dataset
text: We identify three options for creating inputs that avoid the issues outlined in the previous section: 1. Subsampling of existing datasets, 2. Generation of artificial datasets using an LLM, as in Brown and Shokri (2023), 3. Rule-based transformations.
We reject Option 1, since it requires us to know beforehand which biases exist. Similarly, we avoid LLM data, since it is well known that it is subject to biases itself (Liang et al., 2022). We thus decide on a rule-based approach using linguistic annotations based on a corpus C for which we assume that reliable named entity and coreference information is available. In the following, an entity refers to any coreference chain (including singletons), where
at least one mention is also a PERSON named entity, or at least one mention contains a gendered pronoun or a gendered title.
Given a corpus C with named entity and coreference information, we create input documents by replacing first names, pronouns and titles of gendered entities to make them read as male or female. For racial bias we follow a modified procedure outlined in Section 10. Following Parrish et al. (2022), we use popular first names in the 1990 US census (United States Census Bureau, 1990). We leave last names the same to minimize modifications8. This allows us to create realistic inputs with controlled gender distribution (see example in Figure 1). We refer to documents from C as original and to the modified documents as inputs.
We create two variants of the corpus: For Cloc, we locally balance gender within each input by assigning half of all entities as male and the other half as female. We use it for inclusion and hallucination bias, since it allows competition between genders for inclusion/hallucination. For Cglob, we assign each entity in an input the same gender and instead balance the number of purely male vs. female inputs. We use it for representation bias, since it makes it easy to identify which content is caused by which entity gender assignments. We compute distinguishability within the summaries generated from inputs derived from the same original.
We use the newswire portion of OntoNotes9 (Weischedel, Ralph et al., 2013) as C so we can avoid the use of coreference resolution that might itself be biased (Rudinger et al., 2018). For both Cloc and Cglob , we generate 20 inputs for each of the 683 documents in OntoNotes with at least one gendered entity, resulting in 13,660 inputs. We provide details on the dataset in Appendix D.

heading: 5.2 Entity Alignment
text: Entity inclusion and hallucination bias require rudimentary cross document coreference resolution between each summary s and input d. OntoNotes gives us access to gold entities Ed and coreference chains in d, but we lack the same in s. We thus first identify all named entities Es in the summary with an NER tool10. While cross-document coreference is difficult (Singh et al., 2011), we exploit the clear correspondence between summary and document
8We investigate the effect of this choice in Appendix C. 9OntoNotes can be requested from https://catalog.
ldc.upenn.edu/LDC2013T19. 10We use spacy.io (Montani et al., 2023)
in a heuristic instead: We select the token that is most frequently in the last position in mentions of a chain ed ∈ Ed as its last name. We align a summary entity es to an input entity ed if es contains the last name of ed as long as any other token in es is the first name assigned to ed during dataset construction or a title.11 Manual verification finds this procedure performs well (see Appendix E).

heading: 5.3 Identifying Gender of Hallucinated Entities
text: While we can identify entity gender of entities that appear in the input by construction, this is not true for hallucinated entities. To compute hallucination bias, we thus design a classification scheme. Since we expect hallucinated entities to often be well known, we first search for a Wikipedia article with a title that exactly matches the entity. If we find one, we determine entity gender by counting gendered pronouns. Otherwise, we fall back to using US census data. We give full detail in Appendix F.

heading: 6 Summarizers
text: We study both purpose-built summarizers and chat models. For purpose-built models we use BART (Lewis et al., 2020) and Pegasus (Zhang et al., 2020), both transformer models finetuned for summarization. We use the XSum and CNN/DM12 models. For chat models we choose Llama-2 chat (Touvron et al., 2023) 7b, 13b and 70b models with the standard system prompt. For the chat models, we randomly select one prompt
11To avoid incorrectly identifying hallucinations, we additionally require that at least one of the tokens in the entity does not appear in the source to count as hallucinated.
12Taken from https://huggingface.co
per summary from a list of ten prompts designed to elicit summarizing behavior (see Appendix G). We report statistics in Appendix H.

heading: 7 Gender Bias Results
text: Table 3 shows that all models score low on both inclusion bias measures, indicating that the content selection of all studied models does not carry any significant gender bias in this particular setting. Remarkably, we find that all models carry a bias towards male entities in their hallucinations. We study this in more detail in Section 9.1.
All models show some degree of distinguishablity, with BART summaries showing the most pronounced differences between summaries for male and female coded documents. As noted in Section 2, this is not in itself sufficient to establish whether this leads to harm to any particular group. We thus analyse this further in Section 9.2.

heading: 8 Validating our Measures
text: To validate our approach, we conduct the following tests: 1) We check whether our modified input documents lead to degraded summary quality. 2) We test whether also altering content words in addition to names, pronouns and titles to conform to changed entity gender would impact results. 3) We test whether our method is capable of detecting inclusion bias in clearly biased summarizers.

heading: 8.1 Summary Quality
text: Degradation in summary quality between original and modified articles might be indicative of our inputs being insufficiently natural. We test this using GPT-3.5 with the RTS prompt of Shen et al. (2023) as a reference-free metric since it has been shown
to be good at identifying low quality summaries. We focus on relevance since we are interested in content selection effects. We give full details in Appendix I. Table 4 shows that, while there is a small reduction in score for 4 out of 7 systems, performance is very similar between original and modified documents, with the latter score falling within less than one standard deviation of the original score. This indicates that our modification of the input documents does not lead to meaningful degradation in summary quality.

heading: 8.2 Content Words
text: We do not modify gender-specific content words such as chairman, which might reduce input naturalness. To check how this affects results, we manually annotated 100 articles with how content words should be altered depending on entity gender and reran experiments. We find that this has no significant effect on observed bias measures. We provide detailed results in Appendix J.

heading: 8.3 Induced Bias Detection
text: Since we find no inclusion bias in our summarization models, we test whether our method is capable of detecting inclusion bias of clearly biased systems. Specifically, we append “Please put a particular focus on the women mentioned in the text” to the Llama-2 13b prompt to induce it to generate
biased summaries.13 Results in Table 5 show that we can clearly detect the induced inclusion bias.

heading: 9 Gender Bias Analysis

heading: 9.1 Investigating Hallucination Bias
text: We investigate what kind of entities are hallucinated. Table 6 contains the ten most frequent hallucinations of each model.
There are two types of frequent hallucinations: For the first type, models often insert entities that are related to the time of the original articles, sometimes by “hallucinating” the original name for an entity in spite of the input, or by inserting the first name for entities that are mentioned without first name in the source. The male bias here can thus be attributed to the male-dominant nature of news at article publication times. We rerun our experiments for the Cloc case with changed last names to see whether this would alter our conclusions. We find that this has only a limited effect on the hallucination bias. We report detailed results in Appendix C. Our observations link with recent research on knowledge conflicts (Wang et al., 2023; Xie et al., 2024), where models may fail to properly reflect answer uncertainty introduced by conflicting evidence in prompt and parametric knowledge. For Llama-2 we manually verify that most hallucinations can be explained in this way.
However, for the purpose-built models, we find a second type of hallucinations that refer to contributors from CNN (for CNN/DM trained models) or the BBC (for XSum). These usually appear when the summary attributes the text to an author. This is more problematic than historic entities, since they always incorrectly attribute authorship to already potentially well known (mostly male) figures. We find many of these follow repeated patterns. For
13We manually verify that Llama-2 does not refuse this instruction.
example, in many instances, BART and Pegasus XSum would generate “In our series of letters from African - American journalists, writer and columnist [name] ...”, followed by a short summary.

heading: 9.2 Investigating Distinguishability
text: Distinguishability scores in Table 3 indicate some systematic difference between summaries generated for male and female coded documents, even when accounting for expected grammatical differences (see Section 3.4). A possible explanation for this is a difference in summary quality between genders which we test using reference-free automatic evaluation as in Section 8.1. We report average scores comparing male and female summaries in Cglob in Table 7, finding no quality differences.
Automatic evaluation can itself be biased and summary quality is only one aspect of representation bias. We thus conduct a manual qualitative analysis. We rank input articles in Cglob by the (dense) distinguishability of summaries generated for male- and female-coded documents and investigate instances with high distinguishability.
For BART XSum, which has the highest distinguishability, we find there is a pattern where summaries highlight the gender of women in the context of receiving an appointment to a position of power, but does not do the same for men. We find a total of 12 instances of ”first woman” and an additional 11 instances of ”first female” in the summaries generated by BART XSum, but no instances of ”first male” and only a single instance of ”first man” (see Figure 1). This not only hallucinates information, but also forms an instance of Markedness (Waugh, 1982; Cheng et al., 2023) by highlighting the appointment of women to positions of power as abnormal. We find no similar patterns for the remaining systems.

heading: 10 Extension to Racial Bias
text: Our methods are applicable to any group-based bias where group membership can be indicated using names. We demonstrate this by investigating racial bias for stereotypically black and white names. We use the name dictionary of Parrish et al. (2022). We change both first and last names, since both are relevant in communicating race. Entity gender is selected at random. Due to the small name inventory, we can not generate instances for all documents. We thus only consider originals where we can generate a full set of 20 inputs, leaving us with 12,240 instances per dataset. Since word lists for racial bias typically rely on last names, we only compute entity inclusion bias. We also opt not to compute hallucination bias, since we want to avoid constructing a classifier that attempts to identify entity race. Table 8 shows that most models exhibit no
entity inclusion bias, with the exception of BART XSum, which prefers to include black-associated names in the summary. Analogously to our analysis for gender bias in Section 9.2, we check quality differences as a source of distinguishability in Table 9. We find that scores are largely similar, with no model showing significant quality differences.
We also investigate intersectional settings, i.e. settings where race and gender systematically correlate, in Appendix K, but find comparable results.

heading: 11 Related Work
text: While bias in LLMs is the subject of intense research (Sun et al., 2019; Dhamala et al., 2021; Cheng et al., 2023; Srivastava et al., 2023), bias in summarization is underexplored. Liang et al. (2022) take only inclusion bias into account, measured by word lists, thus not respecting the prior distribution of groups in the input documents (see also Section 2). Their use of CNN/DM and XSum, both highly biased, makes it difficult to attribute the bias they find to amplification by models. Brown and Shokri (2023) study summarizers’ gender bias on GPT-2-generated documents (Radford et al., 2019) using word-embeddings. They find an overrepresentation of men in summaries. In comparison, our data construction reduces the risk of false positives due to input biases and our more differentiated measures suggest hallucination as a likely cause. Zhou and Tan (2023) find summarizers treat articles differently when replacing Biden with Trump and vice
versa. While their replacement approach is similar to ours, both their subject of study and measures are highly specific to political bias. Bias has also been observed in tweet and opinion summarization, where contributions by minority groups in the input are underrepresented (Shandilya et al., 2018; Dash et al., 2019; Keswani and Celis, 2021; Olabisi et al., 2022; Huang et al., 2023). In contrast to our bias definition, which focuses on different treatment of groups, here bias is a failure to represent the full distribution of opinions and/or authorship.
Ladhak et al. (2023) show models tend to hallucinate entity nationality in biographical summaries. This is consistent with our observation that the most problematic behaviours stem from hallucinations.
Our approach for generating inputs is related to approaches that generate context that ought to elicit equal behavior for perturbations to the input (Zhao et al., 2018; Parrish et al., 2022), although to the best of our knowledge we are the first to apply such modifications for bias in text summarization.

heading: 12 Conclusion
text: We have introduced definitions that allow us to clearly formulate expectations for what constitutes bias in summarization, along with measures that allow us to detect these biases. We have shown that any measure of summarizer bias must account for bias in the input and proposed a rule-based method that allows us to create realistic data with controlled entity distribution for studying summarizer bias.
Our evaluation of seven models indicates that their content selection is not strongly affected by either gender bias or racial bias for black/white coded names. However, we caution that content selection in news summarization is known to be subject to easy heuristics like the lead “bias” (Jung et al., 2019). Summaries might be more susceptible to biases in more complicated settings. We find significant gender bias in hallucinations revealing a connection between them and bias that suggests increasing faithfulness as a mitigation strategy.
Limitations
We only study single document summarization for news on English documents. While this is by a large margin the most common setting in summarization research, it can not cover all of the possible applications of summarizers. We also focus exclusively on studying binary gender bias and racial bias, leaving extensions to other biases, e.g. those affecting other gender identities, to future work.
While we use high-quality linguistic annotations in constructing our templates, issues still arise that limit template creation. We identify the following specific failure cases: 1) our name identification heuristics break down in the few cases where entities are referenced only by their first name 2) named entities are sometimes not linked correctly to coreference chains due to the lack of singleton annotations in OntoNotes. Finally, documents are not always completely natural. In cases where documents mention historic events, names in the article might contradict historical facts. This might limit the generalizability of some of our conclusions.
Ethics Statement
The most significant ethical implication of our work is that our observation that there are few biases in content selection might be misconstrued to imply that these models are generally safe to use. This might lead to less awareness for bias in text summarization. We thus ensure to point out that our conclusions are limited to the particular summarizers and the dataset we used. In particular, it is possible that biases might exist in settings with more complex content selection procedures, such as multi-document summarization. We also study only a limited number of the varied group-based biases that can occur in language models.

heading: Acknowledgements
text: We thank Juri Opitz for his helpful feedback. The authors acknowledge support by the state of BadenWürttemberg through bwHPC.

heading: A HELM Word Lists
text: Table 10 shows the word lists Wg we import from Liang et al. (2022) for our word list inclusion measure introduced in Section 3.

heading: B Topic Assignment Heuristic
text: For our demonstration of the effect of input bias in Section 4, we require a transparent way to assign a topic to an input document. Following the observations on gender/topic association in Table 1, we manually select a small number of tokens that we identify as sport- or family-related. A text is classified by counting the number of occurrences for each word list and selecting the majority class. A tie is classified as unknown. We list tokens for both categories in Table 11. This allows us to create a deterministic, easy to verify topic assignment.
14This is likely a mistake in the original word lists. We reproduce it here for better comparability.
Note that this assignment is purposefully artificial und non-general. It is not intended as a realistic topic classifier, but as a tool to demonstrate how summarizers might behave and how this influences bias scores.

heading: C Replacing Last Names
text: We show entity hallucination scores, along with the other two scores that can be computed on Cloc , in Table 12. Results are comparable with the setting that leaves last name intact, with the exception of Llama-2 chat 13b which shows a notable decrease in hallucination score. However, even in the latter
case it remains significantly non-zero.

heading: D Corpus Construction
text: The OntoNotes newswire portion consists of documents from the Wall Street Journal and the Xinhua news agency. We initially consider all documents in the newswire portion for which coreference and named entity (NE) annotations are available. From each document, we derive a template which we can then fill with reassigned names and genders in three steps:
1. Identify all coreference chains which have at least one mention containing a PERSON NE
2. Determine the first and last name of the entity
3. Identify which mentions of the entity require modifications
In the first step, we consider all coreference chains in the document. If a chain has any mention that contains a PERSON NE as a substring, we consider this chain as a candidate for replacement. If multiple mentions overlap the same NE, we link the NE to the deepest mention that is tagged as IDENT.
Given a chain with at least one linked PERSON NE, we try to determine the first and last name of the entity using a heuristic approach, since there are no annotations for first and last name. We take advantage of two heuristics: 1. titles like Mr./Mrs. are usually followed by a last name 2. mentions with multiple tokens usually contain the first name, followed by the last name
If a token is preceded by Mr., Mrs. or Ms. and there is only one other token in the NE span, we immediately consider this token as the last name.
Otherwise, we count every token that is the last token in an NE span as a possible last name candidate and every token before the last as a possible first name token. Finally, we select the most frequent candidates for first and last name.
In the last step, we consider all mentions of the entity and categorize it into one of the following classes:
Full Name Any mention that contains both first and last name as determined in the previous step
First Name Any mention that contains only the first name
Last Name Any mention that contains only the last name
Pronoun Any mention that is tagged as a PRP or PRP$
Title Any mention that contains a title. We consider Mr., Mrs., Ms., Sir and Lady.
OntoNotes does not annotate singletons. However, singletons are important since they still require gender adaption to avoid biasing the input. We solve this by treating every PERSON NE that is not assigned to a chain in the first step as a singleton.
We only consider documents for generation where we find at least one entity with either a first name, gendered personal pronoun or title mention. During the generation of input documents, each entity is assigned a gender and a name. We then adapt each mention category by replacing the name with corresponding pronouns, titles or names.
To reduce variance due to name selection in Cloc, we create pairs of inputs which use the same list of names for both categories but invert the category assignment of each entity.
E Validation of Alignment Algorithm
To validate that the algorithm aligning input and summary entities outlined in Section 5.2 works as intended, we conduct a manual annotation study on the gender bias data. We annotate ten samples each for all systems on both Cloc and Cglob . This results in a total of 140 input-summary pairs. Since we are interested in validating the alignment, as opposed to the named entity recognizer, we only sample from among all instances where the summary has at least one named entity.
We then manually check the automatic alignment and annotate for each instance:
1. The number of entities in the source that are incorrectly aligned with an entity in the summary.
2. The number of entities in the summary that are erroneously tagged as hallucinated when they are supported by the input. Since hallucinated entities only affect the hallucination bias score when our gender name classification algorithm assigns an apparent gender to the entity, we report how many of these incorrectly tagged entities receive a gender classification and thus might affect the hallucination score. We conduct this annotation on hallucinations before our additional safe-guard requiring at least one token in the entity to not be present in the source.
Results in Table 13 show that our alignment procedure generally works very well. The low number of incorrect alignments can be attributed to the strict matching criteria between summary and source entities. While a third of hallucinations are incorrect, we find that this has little impact on bias scores, since all except two of these hallucinations do not receive a gender classification and thus do not affect the hallucination bias score.
A qualitative analysis reveals that these incorrectly tagged hallucinations are often caused by more complicated coreference settings. For example, five of the incorrectly identified hallucinations are a result of an article discussing a family “The Beebes”, which does not get correctly identified as an entity in the input by our approach, since we focus on mentions of individuals. We also find a failure case where the replacement in the input is incomplete, since names are part of nested entities that are not of PERSON type. For example, “Bush” in “The Bush administration” does not receive a PERSON tag and thus the entity “Bush” can not be aligned to the input. Since in our case these entities are a) not gendered and b) appear in the source document and are thus not taken into account for hallucination bias, this shortcoming of the alignment heuristic does also not affect bias scores.

heading: F Name Classification in Summaries
text: To determine entity gender in hallucinations, we rely on two separate lookup-based approaches. First, we try to find an English Wikipedia page with a title that exactly matches the named entity detected in the summary (including redirects). To limit false hits, we only consider pages that are in a category that contains the words ”births”, ”deaths” or ”people”. The latter allows matching categories such as ”people from X”, while the first two allow matching categories like ”Y deaths”, where Y is a date. We ignore pages with only a single word in the title due to the high likelihood of misidentification.
To determine entity gender for hallucination bias, we use the number of occurrences of the pronouns shown in Table 14 and select the gender with the more frequent pronouns. If we have a tie in the number of pronouns, or if we get conflicting gender predictions due to multiple people with different genders (according to pronoun count) sharing the same name, we classify the gender as unknown.
There is a risk that the better coverage of male entities in Wikipedia (Wagner et al., 2015) might influence our bias measure. We thus manually inspect the failure cases of this step and find no evidence that this influences results.
If we do not find a matching entity in Wikipedia, we turn to the 1990 US census first names also used in dataset construction. The census contains gender frequency for each included name. We eliminate duplicates, resolving them to the most frequent gender, if the frequency is at least twice that of the less frequent gender, and eliminating them as ambiguous otherwise. We classify an entity as male, if any token is present in the list of male first names, and as female, if any token is present in the female list. Similarly, we do not classify an entity as either gender if it contains names from both lists.

heading: G Prompts for Llama-2
text: Table 15 contains the ten prompts we used to elicit summarization behaviour from the Llama-2 models. We specify that the texts/articles are “old” since we found in preliminary experiments that this reduces instances where Llama-2 chat 7b would refuse to summarize articles that contained dates or can be implicitly dated.

heading: H Summary Statistics
text: Table 16 gives the average number of tokens and entities per summary for the gender bias experiments, as well as the percentage of entities tagged as hallucinated for the summarizers. All summaries were generated using default model settings in the transformers15 library. We find that different summarizers produce summaries of varying length, with XSum summaries being by far the shortest and
15https://huggingface.co/docs/ transformers/en/index
Llama-2 summaries being the longest. Hallucinations are most frequent on XSum, which is a common observation, since XSum contains hallucinations in gold summaries (Maynez et al., 2020). We also give statistics for the summaries generated for the racial bias experiments conducted in Section 10 in Table 17. We find that behavior is similar in both settings.

heading: I Measuring Summary Quality
text: We are interested in summary quality from two perspectives: a) Do our modifications of the original documents lead to a reduction in summary quality compared to unmodified documents? This might indicate our inputs are unnatural and thus our findings might not generalize; b) Is the distinguishability observed in Table 3 caused by a difference in summary quality for inputs that feature either male or female entities?
Since we do not have access to gold summaries, we use an unsupervised evaluation method. Following the recent success of using large language models in reference-free evaluation for text generation (Liu et al., 2023; Chiang and Lee, 2023; Shen et al., 2023), we use GPT 3.5 to elicit rating for the generated summaries. We prompt the model using the reason-then-score prompt of Shen et al. (2023):16
Score the following Summary given the corresponding Article with respect to relevance from one to five, where one indicates “irrelevance”, and five indicates “perfect relevance”. Note that relevance measures the Summary’s selection of important content from the Article, whether the Summary grasps the main message of the Article without being overwhelmed by unnecessary or less significant details.
Article: {article}
Summary: {summary}
Provide your reason in one sentence, then give a final score:
For each system, we evaluate all 683 summaries generated from the original documents which are used as templates for Cloc and Cglob . For Cloc
16We use the gpt-3.5-turbo-1106 model. This model is more recent than the one used in the evaluation of Shen et al., but allows us to fit the entirety of the documents and summaries into the available tokens.
and Cglob themselves we conserve resources and only evaluate summaries generated for two randomly selected inputs, resulting in 1366 ratings per system.

heading: J Content Words
text: J.1 Motivation
Our automatic template generation procedure only changes names and pronominal mentions, leaving content words unchanged. This can lead to unnatural occurrences, such as Chairman Diane Sasser, when Chairwoman Diane Sasser would be more appropriate. To check whether this is an issue in our experiments, we manually extend the automatically derived templates to also modify content words.
J.2 Annotation Procedure
Since we found in preliminary experiments that many articles do not require any manual intervention, we first run an automatic filter over our dataset to identify candidate articles for annotation. We use an extended variant of our word list Wg of Liang et al. (2022) reproduced in Table 18. We then randomly sampled from these articles until we found 100 instances where at least one text span required manual intervention to adapt to entity gender.
During annotation, we identified text spans which should change in accordance with the gender of an entity in the document and which words should be used in either case (e.g. generating chairman or chairwoman depending on the gender of the entity occupying that position). We also considered the case where multiple entities might influence the realization of a particular word, like brothers. In these cases, we also specify a neutral variant (e.g. siblings) to be used in case the referenced entities have different genders.
J.3 Results
We report results on gender bias with our modified inputs in Table 19. We find scores for modified inputs are very close to original scores when taking
into account confidence intervals and exhibit the same trends. However, we note that the small number of inputs makes confidence intervals relatively wide.
K Intersectional Biases
While we have studied racial bias with randomly assigned gender in the main part of the paper, it is also interesting to consider the possibility of intersectional effects that might become apparent when gender and race systematically correlate in the inputs. We construct four additional datasets to study this, where we consider all four possible combinations of perceived entity race and binary gender.
Table 20 shows that behavior is very similar between the random and intersectional settings. Interestingly, we find that for all models that have significantly non-zero distinguishability, it is highest when black and white coded entities are assigned opposite genders. Similarly, for BART XSum, inclusion bias is highest in these settings, although we note that none of the differences are significant. Overall we find no evidence of intersectional effects on our bias measures.

heading: L Computational Infrastructure
text: We ran most inference on a single node using four RX6800 GPUs. For Llama-2 13b and 70b we distributed additional experiments on two A100 and two H100 GPUs.

heading: Future Work
text: those affecting other gender identities, to future work. While we use high-quality linguistic annotations in constructing our templates, issues still arise that limit template creation. We identify the following specific failure cases: 1) our name identification heuristics break down in the few cases where entities are referenced only by their first name 2) named entities are sometimes not linked correctly to coreference chains due to the lack of singleton annotations in OntoNotes. Finally, documents are not always completely natural. In cases where documents mention historic events, names in the article might contradict historical facts