Introduction
Modern LLMs now offer strong zero-shot summarization performance, and even surpass fine-tuned models according to human assessments Goyal et al. ( 2022 ) .
Indeed, zero-shot summaries are sometimes deemed comparable in quality to reference summaries Zhang et al. ( 2023 ) .
Past evaluative work, however, has focused nearly exclusively on news article summarization, a domain in which there is no shortage of available training data.

But zero-shot summarization is perhaps most appealing in niche domains where acquiring training data with which to fine-tune summarization models is sparse and may be prohibitively expensive to collect.
Recent work Shaib et al. ( 2023 ); Tang et al. ( 2023 ) suggests the promise of zero-shot summarization in such domains.
However, there has not yet been a comprehensive investigation of the factuality of model outputs produced in zero-shot summarization across multiple domains (i.e., beyond news). Here we address this gap, and compare the quality of zero-shot summaries generated in niche domains (law, medicine) to those generated for news articles.

In evaluating these models, we center the consistency and faithfulness of summaries generated by LLMs with respect to the input (source) document.
Inconsistencies within summaries have long posed a challenge (Maynez et al., 2020 ; Pagnoni et al., 2021 ) , motivating approaches intended to mitigate this issue (Zhu et al., 2020 ; Cao and Wang, 2021 ) , and for automated evaluation of factuality (Kryściński et al., 2019 ; Goyal and Durrett, 2020 ; Fabbri et al., 2021 ; Scialom et al., 2021 ; Laban et al., 2022 ; Luo et al., 2023 ) .
Here we systematically assess the factual accuracy of zero-shot summarizers across a diverse set of specialized domains.

Specifically, we look to answer four major questions. (1) What is the prevalence of errors in zero-shot summaries across various domains, and how does this compare to established results on news summarization tasks? (2) Are the types of errors observed in these niche domains different from what has been seen in news article summarization?
(3) What is the relationship between the frequency of domains in training corpora and the likelihood of model hallucinations in these domains?
(4) Are existing automatic systems for factual evaluation reliable across multiple domains?

To answer these questions, we enlist expert annotators to manually evaluate the outputs from two representative zero-shot summarization systems—GPT-3.5 ( gpt-3.5-turbo-0301 ; Brown et al. 2020 ) and Flan-T5-XL (Chung et al., 2022 ) —across standard and niche summarization datasets.
Specifically, we evaluate (zero-shot) summaries of medical and legal documents, as well as news articles for reference.

In general, we find that the proportion of factual inconsistencies in summaries varies considerably across domains, calling into question the community focus on news summarization datasets specifically.
Further, we find evidence that the prevalence of articles in pretraining data from a given domain may correlate with the factuality of summaries of articles from the same.
We speculate that this may be due to the model introducing content implicit in its weights in such cases (whereas it may have less “knowledge” in niche domains), although this would need to be validated in future work.