RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Data
We use XSUM (Narayan et al., 2018 ) and CNN-DM (Hermann et al., 2015 ) for news, as well as niche domains like PubMed (medicine; Cohan et al. 2018 ) and legal bills (law; Kornilova and Eidelman 2019 ) for comparison. We select articles shorter than 4096 tokens from the test sets to accommodate model token limitations, resulting in approximately 22,000 articles for news, 3,000 for billsum, and 200 for PubMed.
We randomly (i.i.d.) sample 50 articles from each domain.
We provide more data statistics in Appendix A.1
Model Details
We run experiments with GPT-3.5 ( gpt-3.5-turbo-0301 ) and Flan-T5-XL Chung et al. ( 2022 ) .
We use a general prompt similar to prior work (Goyal et al., 2022 ) for generating summaries across domains.
Specifically, the prompt is as follows: "Article: [article]. Summarize the above article."
Annotation Collection
To acquire manual assessments of model-generated summaries, we hire domain experts via Upwork. 2 2 2 Upwork is a contracting platform suited to such work because it allows hiring individuals with specific background; http://upwork.com . We recruit two experts for each domain: linguistics experts for news, attorneys in civil litigation and public policy for the legal domain, and medical doctors (MDs) for the medical domain.

Our evaluation consists of two rounds. In the first round, annotators primarily assess the factual consistency of summaries in relation to the source article. We collect sentence-level annotations, instructing annotators to identify sentences with inconsistencies. The average proportion of such sentences in each domain is a key reported result.
The inter-annotator agreement at the summary level was determined by calculating the fraction of instances where both annotators identified a summary as inconsistent with respect to the source. The agreement values are 0.80, 0.72, and 0.85 for news, billsum, and PubMed, respectively.
We provide more details about annotation, including agreement statistics, in the Appendix A.2

In the second round of annotations, we categorize errors based on typology previously introduced (Tang et al., 2022 ) .
These errors include: (a) Intrinsic errors, which misrepresent source content, and (b) Extrinsic errors, or “hallucinations”, which introduce terms or concepts not in the source. Past research (Cao et al., 2021 ) has shown that hallucinations can align with real-world knowledge and even be beneficial.

To distinguish extrinsic errors further, we sub-categorize them into: Extrinsic nonfactual errors, which are hallucinations inconsistent with world knowledge; and Extrinsic factual errors, where hallucinations align with world knowledge. Additionally, considering that LLMs are trained on data up to specific points in time, we introduce Extrinsic factual outdated errors, which capture hallucinations that are outdated but were once in alignment with world knowledge (e.g., former presidents of countries).
To assess the factual nature of hallucinations, annotators use online resources like Google Search and Wikipedia, in keeping with prior work (Cao et al., 2021 ) .
How prevalent are errors across domains?
Figure 1(a) shows the average proportion of sentences marked as inconsistent (with respect to the corresponding input) in summaries generated by GPT-3.5 Brown et al. ( 2020 ) and Flan-T5 XL Chung et al. ( 2022 ) for three domains: News, medical, and legal.
Perhaps surprisingly, we observe a higher prevalence of
inconsistencies for news articles, as compared to the specialized domains of medicine and law.
While Flan-T5 introduces more errors than GPT-3.5 overall, the trends are analogous.
Error categories across domains
We next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd.
Figure 1(b) reports the distribution of error categories for both models. 3 3 3 Model-specific distributions are in Appendix A.6 There are more extrinsic errors introduced in the news domain compared to the niche domain datasets.
We include “mixed” errors for cases where errors were classified as different types (intrinsic/extrinsic) by annotators.
The news domain has a higher frequency of such cases.
Reviewing these, we find that they include cases where the summary both misinterprets source information and where it introduces new information. We provide examples in Appendix A.5 .

An “other” option is available to annotators, along with a comment box for capturing miscellaneous errors. Annotator comments highlight instances where there is no clear misunderstanding but instead a misleading overall impression, such as the over-generalization of specific information in the summary
How extractive are summaries, and how does this relate to factuality?
We investigate the relationship between extractiveness (i.e., degree of copying) and factual accuracy across domains.
Specifically, we take the proportion of 3-gram sequences in the summary that are also present in the source for each source-summary pair as a proxy measure for extractiveness.

Figure 2 reveals that there is a comparable level of copying across different models and domains.
However, models tend to copy more often when summarizing articles in the PubMed dataset; this could explain the lower frequency of errors in this domain, since extractive summaries are unlikely to “hallucinate” by definition.
We calculated Spearman rank correlations between 3-gram overlaps and factuality scores for article-summary pairs.
The correlations for the news, billsum, and PubMed domains are 0.61, 0.38, and 0.16 respectively.
Domain representation in pretraining corpora and its relation to factuality.
One possible explanation for the higher proportion of factual errors in news datasets compared to specialized domains is that general news has greater representation in the training data.
As a proxy to measure model exposure to articles belonging to these domains we prompt LLMs to generate overviews of articles based on titles only (headlines for news articles, bill titles for billsum, and study titles for PubMed).

We use the template “Generate a comprehensive overview of the following topic: [title]” to generate text for each article title, assessing LLMs’ memorization. We speculate that increased exposure to an article topic in training data should enable LLMs to reproduce more content present in the original article (as seen with popular celebrities/events, for instance). We assess information overlap between the generated text and original article using ROUGE-L recall, favoring it over embedding based metrics because it emphasizes longest common subsequences based on exact word matches, which makes it suitable for measuring memorization.
This is also preferable for content containing specialized terminology like PubMed abstracts and legal articles.

Figure 3 shows that GPT-3.5 and Flan-T5-XL have higher ROUGE-L recall scores for news, suggesting that these models have had more exposure to news topics;
this could explain the increased extrinsic error rate in news summaries.
Furthermore, in Appendix A.7 , we show similar trends using an alternative approach to measure domain representation by directly querying the pretraining corpus with article titles, and using the number of retrieved articles as a proxy for representation.

Are existing automatic systems for factual evaluation reliable across different domains? Prior research has focused on creating automated metrics for evaluating factuality of generated summaries using question answering (Scialom et al., 2021 ; Fabbri et al., 2021 ) , natural language inference (NLI; Laban et al. 2022 ), dependency entailment (Goyal and Durrett, 2020 ) , and classification methods (Kryściński et al., 2019 ) .
The performance of these metrics has been assessed almost exclusively on evaluation benchmarks comprising model-generated summaries annotated for factuality in the news domain (Kryściński et al., 2019 ; Wang et al., 2020 ; Huang et al., 2020 ; Maynez et al., 2020 ; Pagnoni et al., 2021 ; Cao and Wang, 2021 ; Goyal and Durrett, 2021 ; Cao et al., 2022 ) .
The effectiveness of such automated factuality metrics outside of news is underexplored.

To address this, we use our annotated dataset to examine the performance of QAFactEval (Fabbri et al., 2021 ) , QuestEval (Scialom et al., 2021 ) and SummaC variations (Laban et al., 2022 ) across all three domains.
The results in Table 1 reveal that automated metrics struggle when applied to niche domains.
We note that the lower scores observed for PubMed could be due to the scarcity of observed errors in this dataset, which makes it challenging to reliably evaluate its performance.
Conclusions
We analyzed zero-shot summarization abilities of two LLMs, focusing on factuality. Surprisingly, inaccuracies were more likely to be introduced in summaries of news articles compared to legal and biomedical domains.
Specifically, in this domain we observed more extrinsic errors—i.e., hallucinations of content not mentioned in the source—whereas errors in specialized domains were typically related to an apparent “misunderstanding” of concepts in the source.

We hypothesize that the discrepancy could result from a higher proportion of news articles in the model’s pretraining data, supported by preliminary evidence. Additionally, we observed lower Spearman rank correlations between automated metrics and human annotations in specialized domains compared to news articles, highlighting the necessity for manual evaluations or the development of new metrics for diverse benchmarks.
Limitations
This work has a few important limitations.
The main challenge in achieving a comprehensive evaluation is the cost involved in hiring domain experts. For news domain, we hire proofreaders and linguists at an average hourly rate of $30 USD/hr. For billsum, we hire attorneys at $40 USD/hr, and for pubmed, we hire doctors at $50 USD/hr. The total cost of annotating 100 article-summary pairs across the three domains amounts to approximately $3000 USD, making scalability of the annotations challenging.

We evaluated only two (representative) LLMs; it is possible that other models would show different patterns in behaviour.
Another limitation of this work is that we used only a single prompt to generate summaries; although similar to a previously evaluated prompt (Goyal et al., 2022 ) it is unclear how choice of prompt might interact with factuality of outputs across domains.
A.1 Data Statistics
This section presents additional data statistics in Table 2 , including the average number of sentences in both summaries and source articles across various domains, offering context for comparisons.
A.2 Annotation Details
We recruited annotators on the Upwork platform and selected two domain experts for each task.
In the first round, annotators identified sentences in the summary that were inconsistent with the source.
The agreement at the summary level includes all cases where both annotators marked at least one sentence in the summary as inconsistent.
At the sentence level, we calculated agreement as a function of the fraction of instances in which annotators marked the same sentence within a summary as being inconsistent with the source.
We calculate agreement for the error categories by considering the pre-defined error types chosen by each annotator. Notably the datasets, particularly pubmed, has an imbalance due to the dataset’s significant skew in error labels, resulting in a higher expected chance agreement and lower Cohen’s kappa scores. Therefore, we provide the average inter-annotator agreement and Cohen’s kappa scores in the table 3
A.3 Inconsistent summary annotation
In the first annotation round we asked annotators to mark sentences with unsupported information, i.e., any information not explicitly found in the source, and which could not readily be inferred from the source alone. An example is shown in figure 4(a)
A.4 Error category annotation
In the second round of annotation, we asked annotators to categorize errors identified in the first round. The options provided are shown in Figure 4(b) .
We map the options to categories as follows

(a) terms or concepts from the source are misrepresented are mapped to intrinisc errors

(b) The information in the summary is not found in the source but can be verified via an internet search as accurate is mapped to extrinsic (factual) errors

(c) The information in the summary is not found in the source and can be verified via an internet search as being accurate at a previous time but is outdated is mapped to extrinsic(factual, outdated) and

(d) The information in the summary is not found in the source and can not be verified via an internet search is mapped to extrinsic(nonfactual)

3) Other with a mandatory comment.

An example of this round is displayed in Figure 4(b)
A.5 Mixed errors
We highlight some examples of the mixed error category annotations in Figure 5
A.6 Error categories per model
In Figure 6 , we present error category distributions for the Flan-T5 and GPT-3.5 models separately. Specifically, for the Flan-T5 model in the news domain, errors are typically categorized as "mixed" or marked as intrinsic and extrinsic errors, with no instances labeled as "other." For both models, the trend shows that intrinsic errors in specialized domains are equal to or higher than those in the news domain.
A.7 Alternative method for domain representation
As an alternative method for evaluating domain representation and its relation to factuality, we use the C4 dataset to query article titles.
C4 is a large dataset derived from the the Common Crawl web corpus. 4 4 4 https://commoncrawl.org It was used to train the T5 Transformer models (Raffel et al., 2020 ) .
The number of relevant articles found for each title serves as a proxy for article representation in the training data.
We use a C4 search tool to query the C4 dataset. 5 5 5 https://c4-search.apps.allenai.org/ Queries for each article are manually designed using key terms from the article title with the “AND” condition.

Figure 7 demonstrates that queries for news domain retrieved more articles in the C4 dataset compared to Billsum and Pubmed articles.
A.8 Model Details
We use the default decoding parameters to generate text from GPT-3.5 and Flan-T5-XL. We use the Huggingface Transformers library 6 6 6 https://huggingface.co/ to implement Flan-T5-XL.
(a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Figure 2: Proportion of 3-gram overlaps between model generated summaries and articles. We observe the most copying in the case of PubMed (especially under Flan-T5). This likely explains the greater factuality observed in this domain, and may reflect unfamiliarity with the domain (see Figure 3 ).; Proportion of 3-gram overlaps between model generated summaries and articles. We observe the most copying in the case of PubMed (especially under Flan-T5). This likely explains the greater factuality observed in this domain, and may reflect unfamiliarity with the domain (see Figure 3 ).
Figure 3: ROUGE-L recall scores of original articles in comparison with LLM-generated documents to measure domain exposure during pretraining. Models show higher familiarity with news topics, which may lead to the inclusion of unsupported content in summaries.; ROUGE-L recall scores of original articles in comparison with LLM-generated documents to measure domain exposure during pretraining. Models show higher familiarity with news topics, which may lead to the inclusion of unsupported content in summaries.
(a) First round example annotation where the third sentence was marked as inconsistent .; (b) Second round of annotation where the annotator marked the category for the inconsistent sentence; Annotation interface with questions asked and example annotation on both round of annotations
Figure 5: Examples of sentences annotated with different categories in the news dataset by annotators along with
comments provided.; Examples of sentences annotated with different categories in the news dataset by annotators along with
comments provided.
(a) Distribution of error categories across domains for GPT-3.5 model summaries; (b) Distribution of error categories across domains for Flan-T5-XL model summaries; Distribution of error categories across domains per-model
Figure 7: C-4 dataset search results for queries on news, billsum and pubmed articles. The retrieval results show that there is more representation of news articles in the C4 dataset.; C-4 dataset search results for queries on news, billsum and pubmed articles. The retrieval results show that there is more representation of news articles in the C4 dataset.
Performance of automated factuality metrics across domains. We report the spearmanrank correlation between the average proportion of inconsistent sentences and the predicted scores by the automated metrics.
Data statistics of average number of sentences in the source, summary found in the sampled data. We also include the average number of inconsistent sentences found in summaries of respective domains
We present inter-annotator agreement metrics for sentences, categories and summaries across diverse domains. Cohen’s kappa scores are enclosed in parentheses for each level of annotation, often reflecting lower values. This is primarily attributed to substantial skew in error labels within the dataset, resulting in increased expected chance agreement and consequently lower kappa scores.