Responsible AI Considerations in Text Summarization Research: A Review of Current Practices
Authors: Yu Lu Liu
Meng Cao
Su Lin Blodgett
Jackie CK Cheung
Alexandra Olteanu
Adam Trischler
Venue: EMNLP 2023 Findings (2023)
Keywords: responsible AI
automatic summarization

Abstract: AI and NLP publication venues have increasingly encouraged researchers to reflect on possible ethical considerations, adverse impacts, and other responsible AI issues their work might engender. However, for specific NLP tasks our understanding of how prevalent such issues are, or when and why these issues are likely to arise, remains limited. Focusing on text summarization—a common NLP task largely overlooked by the responsible AI community—we examine research and reporting practices in the current literature. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020–2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals. We also discuss current evaluation practices and consider how authors discuss the limitations of both prior work and their own work. Overall, we find that relatively few papers engage with possible stakeholders or contexts of use, which limits their consideration of potential downstream adverse impacts or other responsible AI issues. Based on our findings, we make recommendations on concrete practices and research directions.

Responsible AI Considerations in Text Summarization Research: A
Review of Current Practices
Yu Lu Liu Meng Cao Su Lin Blodgett
Jackie Chi Kit Cheung Alexandra Olteanu
Adam Trischler
Mila Quebec Artificial Intelligence Institute MccGill University
Microsoff Research, Montréal, Canada *Canada CIFAR Al Chair
Tyu.1.liu, meng cao 0ma00.ml.............m. mtgaill.mmgil.... ca jackie. cheungemcgill.
(sulin. alexandra olteanu) [rrean/)miccooofteee @microsoft.com com
adam. adam.trischheregmal.. com
Abstract
What kind of work the communit priortiziin?
Community
Focus
Who ane the practtioner?
What mativates their work?
AI and NLP publication venues have increas- ingly encouraged researchers reflect on pos- sible ethical considerations, adverse impacts, other responsible AT issues their work might engender. However, for specific NLP tasks our understanding of how prevalent such issues are, or when and why these issues are likely to arise, remains limited. Focus- ing on text summarization-a common NLP lask largely overlooked by the responsible Al community-we examine research and report- ing practices in the current literature. We con- duet multi-round qualitative analysis of 333 summarization papers from the ACL Anthol- ogy published between 2020-2022. We fo- how, which, and when responsible AI issues are covered, which relevant stakehold- ers are considered. and mismatches between stated and realized research goals. We also discuss current evaluation practices and con- sider how authors discuss the limitations of both prior work and their own work. Over- all, we find that relatively few papers engage with possible stakeholders contexts of use, which limits their consideration of potential downstream adverse impacts or other responsi- ble Al issues. Based on findings, make recommendations on concrete practices and re- search directions.
What
pradioss?
"good" summaar concoptualzed?
do practtioners limiations their work?
and
What athical pracitioners consiter?
stakehokders practtooerss consider?
Responsible Summarization
Figure Overview questions we examine when an- alyzing practices related to how the contemporary text summarization literature engages with RAI issues.
Generated summaries might, e.g., misgender the people they deseribe; give rise to libelous represen- tations by failing to appropriately qualify claims; mislead users by giving rise to inferences that are ambiguous or unsupported by the source text; rep- resent contested topics unfairly; or be susceptible to adversarial perturbations in the source text.
Recently, there have been growing efforts to incorporate, in Al and NLP research practice, reflections about ethical considerations, adverse impacts, and other responsible Al (RAI) issues that NLP and AT research- related applications
can exacerbate (Boyarskaya et al., 2020; Nanayakkara et al., 2021; Hardmeier et al., 2021). Despite these efforts and the array of risks, litte work has comprehensively examined responsible Al concerns arising from summarization systems. In this work, we investigate research and reporting practices related how, when, and which RAT issues are covered in the contemporary text summarization literature. To examine these practices, we developed, through a multi-round annotation process, a set of annotation guidelines targeting aspects relevant to RAI Following these guidelines, we conducted a detailed, systematic review of 333 summarization papers published between 2020 and 2022 in the ACL Anthology.
1 Introduction
Text summarization an important NLP task where the goal is to generate a shorter version of an input text while preserving its main ideas. Ap- plications involving text summarization range from storyline generation and sentence compression to meeting notes summarization and email commit- ment reminders.
As their capabilities increase, especially with the emergence of large language models, automatic text summarization systems have seen increasing use ose-despite the known risks generating incorrect, biased, or otherwise harmful summaries.
Specifically, we examine how authors discuss limitations of both prior work and their own work.
Paper annotations are available upon request.

---

which RAI issues they consider, the relevant stake- holders they imagine or serve, as well how stated and realized research goals might often differ.
intrinsic qualities of the text such as whether the summaries preserve relevant content or are non-redundant, fluent and coherent (Gkatzia and Mahamood, 2015), but rarely extrinsic criteria like how the summaries are used in downstream applications (Zhou et al., 2022).
We do so to help foreground our choices as a community- how we write, how we frame problems, how we consider social context, and how we broadly think about RAI issues-and to make these choices explicit (rather than implicit) so that we may better understand their implications. Since the NLP community has only recently started to prioritize these issues, taking an early snapshot of emerging practices can provide insight into why the community might be struggling with consider- ing limitations of its work, ethical considerations. adverse impacts, and other related issues.
More recently, a growing emphasis has been placed on ensuring that generated summaries are consistent with the source text, abstractive sys- tems risk generating so oo-called "hallucinations," i.e., text that distorts or is unsupported by the source text (Cao et al., 2020; Dong et al., 2020; Falke al., 2019; Kryscinski et al., 2020; Kumar and Cheung, 2019). Related concerns about factuality, accuracy, and coherency have all been bundled under halluci- nations, obscuring what issues the authors are after and the range of harms or adverse impacts they can bring about. Our work examines how assessment practices might limit the space of responsible Al issues the community considers by possibly obfuscating some issues and foregrounding others.
We find that despite the introduction of impact statements and ethical considcrations sections at both NLP (Benotti and Blackburn, 2022) and Al conferences more generally (Ashurst et al., 2022: Nanayakkara et al., 2021; Boyarskaya et al., 2020), relatively few papers engage with possible stakeholders contexts of use, and fewer still-less than 15% in the set we reviewed-wild with responsible AI issues. We discuss how this limits the space of responsible AI issues of which the community may be aware, as well as the community's ss capacity to speculate effectively on potential issues. We make recommendations for how the community can improve summarization research practices to be more responsible.
Responsible AI and summarization. While a great deal of work on responsible Al issues has emerged for NLP broadly, much less work has addressed summarization specifically. Carenini and Cheung (2008) examine whether a summary reflects the distribution of opinions in source documents. Shandilya et al. (2018) and Dash et al. (2019) consider whether summarization systems fairly represent document authors from different demographic groups, while Shandilya et al. (2020) explore readers' perceptions of fairness in summaries, finding that ROUGE metrics are not well-suited to capturing perceptions of summary fairness. Meanwhile, Keswani and Celis (2021) find that summarization systems produce summaries under-representing already-minoritizer language varieties. In our analysis of the summa- rization literature, we explore to what extent papers acknowledge these existing concerns and aim to un- cover issues not previously raised by existing work.
2 Background & Related Work
Assessing summaries. Text summarization is a longstanding NLP research topic with a growing number of applications (Mani, 2001; Nenkova and McKeown, 2012), particularly with the increased availability of both datasets and neural summariz- ers (Dong, 2018). Text summarization systems often evaluated using string and word matching metries like ROUGE (Lin, 2004) to assess the similarity between references or gold summaries and system generated summaries. These methods often do not correlate well with human judgments, spurring research into developing automatic metrics with higher human correlations (Fabbri et al., 2021). Nevertheless, automatic metrics can obfuscate when systems may may not work; e.g., are the summaries more likely leave out a certain type of content? Do they work equally well for content by different speakers? When human judgments are included, they often examine
Meta-analyses in NLP. We draw inspiration from recent work that analyzes research and reporting practices in NLP. Blodgett et al. (2020) explore how NI.P papers describe "bias," finding that definitions often vague, vary widely, and may not be well- matched accompanying technical approaches, while Benotti and Blackburn (2022) examine eth- ical considerations sections in ACL 2021 papers, finding that relatively few (~15%) include such sections, and that some of these (~~0%% do not

---

meaningfully address either benefits or harms of the research. Blodgett et al. (2021) examine how four benchmark datasets conceptualize and opera- tionalize stereotyping, while Devinney et al. (2022) analyze papers on gender bias in NLP to uncover how gender is thcorized, finding that thcorizations rarely are made explicit engage with gender theo- ries beyond NLP. Via interviews and survey, Zhou al. (2022) examine practitioners' assumptions and practices when evaluating natural language generation systems. Elsewhere, work has analyzed evaluation practices in natural language generation (Gkatzia and Mahamood, 2015; van der Lee et al., 2019; Howcroft et al., 2020, i.a.). We draw on these papers in our own investigation of how papers de- scribe the goals of their work, the approaches they take in evaluating progress towards those goals, and the responsible Al issues they may raise.
3.2 Paper Review & Annotation
3.2.1 Exploratory Review
To scope our literature review and determine the practices we wanted to capture, we started with small set of 8 summarization papers. We wanted variety of papers in terms of publication venues and domain, and we were also interested in papers with an "ethical considerations" section, The selected papers are either published *CL venues (DeYoung et al., 2021; Zhao et al., 2020; Feng et al., 2021; Aralikatte et al., 2021; Zhang et al., 2021b) or HCI and social computing venues (Zhang et al., 2020; Tran et al., 2020; Molenaar et al., 2020), and they cover summarization of medical literature, medical dialogues, legal cases, and emails. We observed differences among these papers, with those published HCI/social comput- ing venues focusing more on how summarization systems are used and on their stakeholders, which, along with our research questions, informed our carly annotation aspects. These aspects included mentioned stakeholders, author affiliation, domain. limitations, and more.
3 Methods
To understand research practices surrounding how, when, and which RAT issues are or should be con- sidered by the text summarization community, we conducted systematic survey of the summariza- tion literature. To do so, we followed several steps: we first i) gathered a collection of recent text sum- marization papers to be examined and annotated ($3.1) and ii) reviewed a small set of text sum- marization papers published various venues to explore relevant practices ($3.2.1). Drawing on this exploratory review, iii) we then developed an an- notation scheme (detailed in $3.3), which we used to annotate our collections of text summarization papers ($3.2.2). Finally, iv) analyze the anno- tations to understand emerging practices ($3.2.1).
3.2.2 Paper Annotation
From this starting point, we developed a common annotation scheme over several iterations (Rounds & 2 below), which was then used to annotate the collection of text summarization papers. Ap- pendix A provides detailed statistics on the process. Round 1: Developing & refining the annotation scheme. Guided by the initial annotation dimen- sions described above, every author open-coded 20 papers such that each paper was coded by authors, totaling 60 papers. We periodically compared our annotations, updated the annotation scheme to resolve confusions and disagreements, and revised our annotations when necessary. For example, we split the initial domain category into intended domain and actual domain to better track differences between the two, which we noticed in many papers. At the end of this round, we arrived at the scheme overviewed in $3.3.
3.1 Paper Collection
We focused papers published between 2020 and 2022 in the ACL Anthology. To do so, first gathered all papers with "summarization" in their title or abstract. After manually removing unre- lated papers (i.e., papers using "summarization" for purposes other than the task of textual sum- marization), we obtain 401 summarization papers. We then manually filter out papers where the main focus was not text summarization (e.g.. natural language generation papers where summarization is one of many evaluated tasks). This resulted in the set of 333 papers that we annotate.
Round 2: Applying & clarifying the annotation scheme. Using the scheme, we coded larger sub- set of 131 papers. While in this round each paper was coded by a single author, we continued to peri- odically discuss ambiguous cases, clarify the anno- tation scheme, and update annotations accordingly. Round 3: Hired annotators. With the guidelines finalized, for the remainder of the papers, we hired
Wo excluded demonstration, shared task description, tuto- rial, and workshop papers.

---

7 graddate students in the field of NLP. We paid them at a rate of 30 CAD per hour, which is roughly equivalent to the wage of teaching assistants at our university. We started by briefing the annotators with a 1.5-hour paid training session on our project goal, how their annotations would be used, and the annotation scheme, illustrated by examples from the first two annotation rounds. We then scheduled 26 two-hour sessions held via rideo-conference, with one author present at all times answer questions and offer clarifications. The annotators could choose which and how many sessions to attend. The annotators were reminded of their right to periodically suspend or quit the an- notation, without any impact on their pay. A total of 142 papers were annotated by hired annotators.
is explicitly specified throughout the paper's introduction (i.c., a failure state an intended domain), or when the paper explicitly intends to be general (i.e., explicitly stating that its contribution is gemeral-purpose, or that it can be used in any domain or application).
Research goals: authors' stated goals. Annotators either copy or summarize the paper's goal, based on the abstract and the introduction of the paper. This provides additional context to the contribu- tions, intended or domain, well as issues with current practices the authors aim address. Affiliation: the authors' affiliation, including whether there is at least one author affiliated with an academic institution, with industry, or other organizations (e.g., government or NGOs).
3.3 Annotation Scheme
3.3.2 Data & Evaluation Practices
To help us reflect on when RAI issues are brought up, how they are framed, and by whom, our annotation scheme covers aspects related to each paper's goals & authors ($3.3.1), evaluation practices ($3.3.2), as well as stakeholders (if any mentioned), limitations (of prior work or current work), and ethical considerations ($3.3.3).
Evaluation practices reflect the space of concerns (including RAI issues) that the community is aware of, and can also give rise to their own RAI issues. We therefore annotate papers according to:
Actual domain: the domain(s) of the data that is actually used in the papers for evaluation or other purposes, with the same codes as the intended do- main. This enables us to examine discrepancies between intended and actual domains.
3.3.1 Paper Authors & Goals
As we aim to examine how practitioners engage with RAI in their work, we need to know who the practitioners are, what their work is, and what motivates their work. These aspects not only con- textualize our survey, but also provide cues about potential usage scenarios, which may determine what harms are likely to occur. Specifically, we consider the following aspects:
Quality criteria: text properties practitioners fo- cus on when evaluating summarization systems. We annotate this aspect to understand what is con- ceptualized as a "good" summary.
3.3.3 Limitations & Ethical Considerations
Lastly, we are also interested both what kind of limitations (of both their work and prior work), ethical considerations, and stakeholders the authors explicitly bring up, as well as limitations that they might have overlooked.
the type(s) of contribution a paper makes the research community, including new dataset, a summarization system (including new models, methods, or techniques), an evaluation metric, an application of automatic text summariza- tion, a comprehensive evaluation of a collection of existing artifacts, or other types of contributions. This allows us to examine, for instance, whether au- thors of papers with certain types of contributions are more likely engage in ethical reflection.
Contributions:
Limitations of prior work: what authors de- scribe as weaknesses of prior work to track what existing issues the authors engage with. To capture this, annotators copy or summarize passages where limitations of prior work are covered.
Limitations of one's work: whether and how the authors discuss the limitations of their own work. Annotators again either copy or summarize relevant passages.
Intended domain: the domain(s) the work is stated to be developed for, including news articles, dialogue, computer code, medical documents, blogs (e.g.. Twitter), opinions (e.g.. customer reviews), scientific articles, wiki (Wikipedia or Wikipedia-like platforms), other domains, or general. The latter code is used when nothing
Other limitations identified by annotators: the notes annotators took about any limitations they noticed while reviewing that were not already men- tioned by the authors.

---

Mentions stakeholders
Contributions
Intended Domain
Swork
System
224
General
173
YES, ONLY hum.
Ansceppted or eetsting uners
22%
Dataset
91
News
45
YES, 35%
Metric(s)
36
Dialogue
NO 247
Other makcholders
7%
Evaluation
73
Opinion
19
Application(s) & Other
34
Medical
17
Discusses own work's limitations
Discusses othical considerations
Author Affiliation
Scientific
10
Academic
299
Code
9
Industry
121
Wiki
9
YES 37%
(collab of above two)
(95)
Blog
Other
32
Other
26
Table I: Overview of corpus of papers we reviewed.
Summary statistics about mentioned stake- holders, and about how often papers cover limitations of one's work and ethical considerations.
Figure 2:
4.1 Community Focus
To help unpack why and how authors might (or might not) approach responsible Al considerations. we first wanted to understand who is conducting the research and how they conceptualize their work- what they are creating, who they are creating it for, and what outcomes they envision-helping us to contextualize current and possible future practices.
whether there para- graph section dedicated to ethical considerations, broader impacts of the work, or similar related top- ics, and if so, what is discussed therein. Annotators copy or summarize the section.
Ethical considerations:
There is a focus on developing new systems, with comparatively less emphasis on evaluation, metries, datasets, or applications. Nearly 70% of the 333 papers contribute systems (including models and methods), while fewer than 30% con- tribute new datasets and around 10% contributed new metries. An even smaller fraction (less than 2%) of papers focus on applications. This emphasis on developing new summarization systems, models, or techniques echoes concerns about the devalua- tion of e.g.. data work which is often framed as "pe- ripheral, rather than central" to Al research (Gero et al., 2023}iin contrast the prestige of doing what is perceived to be more "technical" work such as modelling or system building.
Stakeholders: whether the authors mention any stakeholders and who these stakeholders are, in- cluding human annotators, cxisting or anticipated users of a system, other researchers (in machine learning, AI, NLP, or related fields), or other stake- holders. We track this information because consid- cring stakeholders is critical for envisioning harms and unintended consequences (Boyarskaya al., 2020; Buçinca al., 2023).
3.4 Analyzing Annotations
In addition to the codes we assigned to papers during the annotation, to further characterize partic- ular practices (e.g., how many papers evaluate for factuality?), also used keyword search and mea- sured keyword frequency. To further assess various subsets of papers (e.g., papers that discuss their own limitations), we performed qualitative coding on extracted quotes, revisiting papers as necessary. Appendix B contain more details on this analysis (e.g., keywords used, codes for qualitative coding).
Many systems, metrics and datasets in- tended to be general-purpor not explicitly stating an intended domain means the work is implicitly intended to be "general-purpose" 555% of 224 of papers contributing new systems Similarly, 72% (26 out of 36) of papers contributing met- rics and ~23 (21 out of 91) of those contributing datasets are also intended for general-purpos set- tings. However, these ostensibly general-purpose artifacts are often only tested trained on few domain-specift datasets and scenarios ($4.2).
Assuming that
intend them to be "general-purposs
4 Findings
Our systematic review surfaced insights related to what the text summarization community has been focusing on ($4.1), common evaluation practices the community employs ($4.2), and how the com- munity engages with ethical considerations ($4.3).
The text summarization literature remains driven by academic research, though there is also significant interest from industry. ~90% of the reviewed papers were co-authored by at least one academia-affilee author, with ~32% of these

---

papers being collaborations with industry authors. There are comparatively fewer papers solely writ- ten by authors affiliated with industry or other non- academic organizations. Because such organiza- tions may be more connected to specific deploy- ment settings and users (Zhou et al., 2022), their low representation may represent barrier to en- gaging with the impacts of summarization systems.
Examples include helping workers by summarizing meetings or emails (c.g., Singh al., 2021; Zhang et al., 2022) and helping health professionals by summarizing medical encounters or files (e.g.. Hu et al., 2022; Adams et al., 2021). ~20% of 74 papers referencing usets aim to improve customer experience, which involves transactional relation- ship between those who would deploy the summa- rization system and those who would use output summaries, for example, summarizing product re- views to "make the shopping process more useful and enjoyable for customers" (Oved and Levy, 2021) or summarizing livestreamed content to "fully meet the needs of customers lon livestreaming platforms]" (Cho et al., 2021). A more expansive conception of benefits might help practitioners con- sider more stakeholders, applications, and impacts.
Papers rarely mention stakeholders when imagining intended use contexts. While ~76% of reviewed papers mention stakeholders, fewer than half seem to mention stakeholders other than human annotators, who are typically mentioned in the context of evaluation practices rather than when discussing research goals. Only ~22% of all annotated papers consider anticipated existing users, while ~12% mention other researchers. Conceptualizing a contribution without conceptual- izing stakcholders may mean that the contribution will not meaningfully benefit any particular stakeholders, and may also make it more difficult to reason about limitations or adverse impacts.
4.2 Evaluation Practices
To examine current practices we considered actual domains (i.e., the domain of the data used or col- lected in the paper), researchers' conceptualization of summary quality, and which quality criteria they tend to prioritize.
Papers referencing users are often both explicit about who those users are and specify an in- tended domain. About foree-fourths of the 74 papers we found to explicitly reference users, both describe who these users are and how they would benefit from automatic summarization, c.g., "auto- matic summarizing tool that can generate abstracts for scientific research papers [...] can save much time for researchers and also readers" (To et al., 2021). Many of these papers explicitly mention specific intended domain, with only small frac- tion of them (about 8%) (implicitly or explicitly) intending their work to be seneral-purposee The remaining one-fourth only vaguely mention users, sometimes by specifying what users might want (e.g.. "a user might be looking for an overview summary more detailed one" " (Xu and Lapata, 2020)), without specifying who they might be. This is particularly the case when the authors intend for their work to be general purpose (67%, 12 out of 18). Not having a clear application or domain in mind can, however, make it difficult for authors to imagine users or other stakcholders.
Most papers on general-purpose systems, met- rics, and datasets solely use data from the news domain. We estimate that ~522 of 122 pa- pers contributing seneral-purpose systems only use news data when developing or evaluating systems, methods or models. This is not surprising since the most common summarization datasets are from the news domain (Dernoncourt al., 2018; El-Kassas et al., 2021). For general-purpose metrics (i.e., not developed for only restricted set of applications or domains and meant to be applied broadly), this percentage is ~77% out of 26 papers. Similarly, datasets introduced by papers that do not state intended domain, explicitly aim to be general- purpose, all collect their data from the news domain. These practices could introduce risks, as e.g., sys- tems ostensibly developed to be general-purpon but only trained and evaluated on a restricted set of domains cannot be reliably used in other domains.
While quality criteria concerning information saliency, linguistic properties, and factuality are frequently evaluated, criteria such bias and usefulness are rarely evaluated, if ever. Cri- teria related to information saliency (e.g., "infor- mativeness." "relevance," "redundancy") are men- tioned by ~41% of all reviewed papers. This is followed by criteria related to linguistic proper-
Imagined benefits often only include reducing anticipated users' labor or improving customer experiences. ~54% (40 out of 74) of papers ref- erencing users aim to reduce some type of labor. these instances, the work is meant to automate, speed up, or even replace parts of users' workflow.

---

ties (e.g., "coherence," "fluency." "readability"), mentioned by ~39% of papers, and criteria related to factuality (e.g., "factual consistency," "halluci- nation," "faithfulness"), mentioned by ~288 of papers. Other criteria, such as summary usefulness (c.g., "how useful is the extracted summary to sat- isfy the given goal, in our case, to answer the given query" (Iskender et al., 2020)), and whether the summaries exhibit some bias (e.g-, bias in text sen- timent polarity (Sarkhel et al., 2020)) are rarely if ever mentioned. As a consequence, current practice seldom assesses whether more user-facing goals of summarization (e.g., the actual reduction of labor) are attained. Task-based evaluation, where sum- maries are assessed based on how they help humans perform a particular task (Lloret et al., 2018), is not a foreign concept in automatic summarization (Van Labeke et al., 2013; Zhu and Cimino, 2015; Jimeno- Simeno-Yepes et al., 2013) and could be adopted by the community to better suit certain research goals.
these metrics. Since the reliability of ROUGE has been questioned (Novikova et al., 2017: Bhandari et al., 2020), there is a risk that metric scores do not reflect the true performance of evaluated systems. Human evaluation is widely used, but details on how it is carried out are often missing, Some form of human evaluation seems used in majority of papers, with ~58% of all papers including mentions of human annotators. Yet paper an- notators noted limitations in how these evaluations were carried out for 24% (47 of 194) of papers mentioning human annotators. Some of the issues most salient to our paper annotators included pa- pers lacking detail about who the human evaluators are (noted for 22, ~47% of 47 papers); the text properties or quality criteria human evaluators were asked to rate, such asking annotators to score "importance" and "readability" without providing clear definitions (noted for 11, 23% of 47 papers); and the evaluation process in general, such as whether annotators were shown source doc- uments during evaluation (noted for 9, ~99 of 47 papers). These issues are particularly problematic for reproducibility and research standards. The community could adopt best practices developed for evaluation design, transparency, and analysis in human evaluation of text generation systems (van der Lee et al., 2019; Schoch et al., 2020).
While factuality, information saliency, and linguistic properties are frequently evaluated, these criteria are less commonly conceptualized as part of research goals and limitations. Com- paratively, only ~ 10% of all examined papers ex- plicitly aim to address factuality-related qualities (e.g., better "evaluate faithfulness." "localizing fac- tuality errors" in output summaries, to prevent model "hallucinations"), and 155% of papers note factuality-relltte limitations in prior work (e.g.. "generating summaries that are faithful to the input is an unsolved problem" (Aralikatte et al., 2021)) )).. Similarly, only 8% of examined papers consider information saliency-related criteria as part of re- search goals, while 12% of papers point to these criteria when covering limitations of prior work. 5%
4.3 Limitations and Ethical Considerations
Finally, we examine whether and how the commu- nity has engaged with ethical considerations and limitations of their own work and of existing work.
Most papers do not discuss the limitations of their own work, and rarely include any ethical reflections. We estimate that ~63 of all annotated papers do not include a diseussion about the limitations of their own work, while only ~ 14% of surveyed papers have section on ethical considerations. Pupers proposing datasets are more likely to have an ethical considerations section (~20%, 19 out of 92) than those proposing systems (~100% 23 out of 224). Work without such explicit reflections may not be able to effectively incorporate potential weaknesses ethical concerns into the design and evaluation of their proposed systems, datasets, or metrics.
For linguistic properties, these percentages for research goals, respectively 9% for limitations of prior work. Naming these criteria as desirable, and explicitly targeting them in research, would facilitate the adoption of more careful operational- izations and engagement with the risks they may give rise to.
Evaluation of output summaries still relies heavily on ROUGE or on other similar automatic metrics based on lexical overlap, with ~90% of 224 papers proposing new systems using such metrics. This fraction is ~87 (79 out of 91) for papers contributing datasets and 70% (51 out of 73) for papers providing comprehensive evaluations. Over- all, about 22% of all examined papers only use
When authors conceptualize ethical concerns, they often turn to data-related issues. ~62% of the 45 papers we found to include ethical con- siderations sections cover data issues in these sec-

---

tions. The data-related issues that are foregrounded include: data access and copyright (21 papers)- e.g., specifying that the data is publicly available; data privacy (13 papers)-mostl stakeholders who are either the people producing the data (e.g., pro- fessional writers of a scraped website (Liu et al., 2021)) or the people described by the data (e.g.. users and customer service agents of e-commerce websites where data is collected (Lin al., 2021));
Of these issues, however, only factuality seems to be conceptualized as an ethical concern, with 38% (17 out of 45) of papers with ethical consideration sections mentioning factuality- related concerns. From our assessment, no papers covering ethical concerns seem to relate them to quality criteria such as linguistic properties, or information saliency or coverage.
While factuality is sometimes conceptualized as an ethical issue, few papers reflect on the impact of factual errors. Only 6 of 17 papers (~35%) seem to name factuality as an ethical concern by describing adverse impacts of factuality-related model failures, with 4 naming "misinformation" "bad influence" in the news domain, one "misinformation" in the context of corporate meetings (e.g., which "would negatively affect comprehension and further decision making" (Zhong et al.. 2021)), and one the "risk of misinterpretation of evidence and subsequent [medical] malpractice" (Otmakhova et al., 2022).
and data "bias" (11 papers).
When mentioned, data bias remains poorly de- fined or under-specifted. When discussing pos- sible biases in their data, papers tend to only briefly and generically mention "bias" or a type of "bias" (e.g.. "political bias", "gender bias", "bi-
ased views"). From our assessment, only 3 papers seem to provide more detail beyond these brief mentions (Adams et al., 2021; Cao and Wang, 2022; Zhong al., 2021). Yet, even when bias issues discussed in more depth, what is meant by data bias or the concerns or harms it can give rise to remain vaguely specified. For instance, (Zhong et al., 2021) mention how "meeting datasets rarely contain any explicil gender information, Lyet] anno- tators still tended to use 'he' as pronoun" without further elaboration about e.g., the harmful stereo- types these biases might reproduce whether the viewpoints of certain users might be unequally represented or misattributed in resulting systems' meeting notes summaries. While is encouraging to see data bias identified as a source of concern, there is an opportunity to do so consistently and to provide a clearly articulated conceptualization of what is meant by data bias (Blodgett et al., 2020; Goldfarb- Toldfarb-Tarrant et al., 2023).
The other 11 papers either generically de- scribe factuality factuality-related related model failures (e.g.. "Even though our models yield factually consistent sum- maries [...] they can still generate factually incon- sistent summaries or sometimes hallucinate infor- mation" " (Jiang al., 2021), or describe factuality- related concerns as an "open problem" (Xiao et al., 2022) or as problem with "unacceptable outcome" in "high-impact" domains (such as scientific and medical domains, De Young et al. (2021)) without much elaboration. A few papers (3) also explicitly warn that their models are not ready for deploy- ment due the lack of guarantees for the factual correctness of model outputs.
While papers often discuss limitations related to various quality criteria, these are rarely con- ceptualized ethical concerns. Papers describe a range of issues when reflecting on limitations. ~44 of the 122 papers we found discuss limita- tions talk about factuality-related issues-range -ranging from only brief mentions (e.g., generic references to "factual errors" or "hallucinations"). to more de- tailed descriptions (e.g., "factual errors by mixing up important details [such as] mixing up the victim and suspect of a crime, mixing up locations and dates" (Panthaplackel al., 2022)). Limitations related linguistic properties (c.g., length, word novelty, coherence, fluency) are also sometimes mentioned (20 out of 122), as are issues related to information saliency or coverage (12 out of 122).
Papers describing ethical considerations often do not engage with intended use context. We estimate that fewer than half of the 45 papers-eplicc itly considering ethical issues engage with intended use contexts. For example, only 3 papers explicitly mention the need for human oversight in system deployment, and only of these describe the stake- holders who would be responsible for supervision with one paper noting that "[t]he most natural ap- plication of this technology is not as replacement for human scribe, but as an assistant one. By providing tools that aid a human scribe one can mitigate much of the risk of system,fallure, failures, such as hallucination" (Zhang et al., 2021a). Ethical con- siderations not grounded in use contexts may not be able to realistically anticipate adverse impacts.
explic-

---

When stakeholders are mentioned in ethical considerations, potential harm to them is often overlooked. Discussion of stakeholders is restricted to the compensation of human annotators (13 out of 45 papers), data privacy (13 out of 45 papers), and intended positive impacts on anticipated users (15 out of 45 papers). This may limit the conceptualization and evaluation of benefits and harms. The above requirement for human oversight, for instance, does not consider whether might increase labor instead of reducing it, nor does consider when or which stakcholders are well equippdd to supervise.
highlight two areas that tend be overlooked: i) some model failures are rarely conceptualized as ethical concerns; ii) intended use contexts, includ- ing stakeholders, are rarely involved in ethical con- siderations, which prevents authors from imagining potential harm to said stakeholders.
Recommendations: To better engage with limita- tions and ethical concerns, we recommend to:
1) Reflect explicitly on the conceptualization of their work's intended use context, and of what constitutes good summary that context. What assumptions about system capabilities, stakeholders, or intended domains do choices of quality criteria and accompanying evaluations carry (Zhou et al., 2022)? What are the implica- tions of these assumptions and choices?
5 Discussion and Recommendations
Intended use contexts are often not well- described. We find that many papers do not spec- ify intended domain, and few works mention stakeholders, such as existing or intended users. when imagining intended use contexts. Even when such stakeholders are mentioned, imagined benefits are quite narrow in scope.
2) Engage with prior literature (e.g., Bender, 2019; Weidinger al., 2022) on ethical concerns and real harms to which NLP systems can give rise, such as hate speech, stereotyping, and misinfor- mation. This could help practitioners critically reflect on their own work and more clearly en- gage with issues that have already been recog- nized as ethical concerns, such as "bias" (Blod- gett et al., 2020).
Recommendations: We encourage practitioners to conceptualize their contributions' intended use context by articulating, as much as possible, rele- vant stakeholders, intended domains, and potential benefits and adverse impacts to those stakeholders.
3) Engage with ethical issues the text summa- rization community has already recognized. Through our survey we identified issues, such as the risk of misgendering stakeholders in summa- rization, which have already been pointed out by some members of the community e.g., (Zhong et al., 2021). Engaging with these issues could also help practitioners imagine limitations and ethical concerns of their own work.
Quality criteria such as bias and fairness are rarely considered. We find that the priority in summarization evaluation is often on information saliency, linguistic properties, and factuality.
Recommendations: While these quality criteria are important, other criteria (e.g.. social bias. usability) might be relevant and of interest to stakcholders. We encourage authors to consider these criteria, clearly define them (which may require grounding them in specific contexts), and adopt evaluation practices that meaningfully capture them. To this end, we encourage the development of evaluation instruments (e.g.. benchmarks or human evaluation protocols), especially those tailored for, or adapt- able to, specific use contexts.
Conclusion
We surveyed 333 recent text summarization pa- pers from the ACL Anthology to examine how the summarization community currently concep- tualizes and engages with broader responsible Al issues, and discuss how this might be impacted by existing research practices. While we are heart- ened by some of the practices we observed, such as evaluating issues like factuality, there remains significant opportumity to also foreground other responsible Al concerns. We hope that, by high- lighting current practices and offering actionable guidance, this work will encourage a reflective, collective research and reporting practice sum- marization research and beyond.
There is lack of engagement with limitations and ethical concerns. We find that most papers do not have discussions their own limitations, ethi- cal considerations, and other related issues. When they do, they often focus on data-related concerns. This practice is not wrong by itself, but could be indicative of a narrow range of ethical concerns practitioners might be aware of. We especially

---

7 Limitations
Jules Barbe for his early work on this project. Fi- nally, we thank the anonymous reviewers for their valuable feedback.
Our findings are limited to the papers covered by survey, which come from the ACL Anthol- ogy and are written in English. Works from other sources, such as venues with a different focus (e.g., venues focusing on Al applications) or those hav- ing a different demographic distribution than ACL, might paint different picture. Our findings are also limited to the time period it covers: for exam- ple, between 2020-2022 some venues had not yet introduced the requirement to have "Limitations" (or "Broader Impacts" or "Ethical Considerations") sections. We hope to see how the picture might change in the future as such requirements become more and more standard.
References
Griffin Adams, Emily Alsentzer, Mert Ketenci, Jason Zucker, and Noêmie Elhadad. 2021. What's in a summary? laying the groundwork for advances in hospital-course summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4794-4811, Online, Association for Computational Linguistics.
Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. 2021. Focus at- tention: Promoting faithfulness and diversity in sum- marization. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Nat- ural Language Processing ( (Volume 1: Long Papers), pages 0097-6 Online, Association for Computa- tional Linguistics.
Furthermore, our findings are limited by our pa- per annotation process. The annotation guidelines described in Section 3.3 could overlook attributes which we failed to imagine with our current under- standing of responsible AT issues and the task of automatic summarization. While we carried out the steps detailed in Section 3.2.2 with the goal of ensuring high annotation quality, the annotation process is imperfect; not all annotators are trained in responsible Al issues and they do not perfectly follow the annotation guidelines nor follow them the same way as collective.
Carolyn Ashurst, Emmie Hine, Paul Sedille, and Alexis Carlier. 2022. Ai ethics statements: analysis and lessons learnt from neurips broader impact state- ments. In Proceedings of the 2022 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 2047-2056
Emily M. Bender. 2019. A typology of ethical risks in language technology with eye towards where transparent documentation can help. Presented at The Future of Artificial Intelligence: Language, Ethics, Technology Workshop. httpa://bit.ly 2P9
8 Ethical Considerations
As with any research undertaking, our work may also have unintended outcomes. By only fore- grounding a subset of ethical and other responsible AI concerns our discussion of the findings, we may inadvertently suggest that other issues deserve less consideration. The authors and annotators are also limited by our own conceptualizations of re- sponsible Al issues, and there may be issues we fail to recognize.
Luciana Benotti and Patrick Blackburn. 2022. Ethics consideration sections in natural language process- ing papers. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 4509-4516, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Manik Bhandari, Pranav Narayan Gour, Atabak Ash- faq, and Pengfei Liu. 2020. Metries also disagree in the low scoring range: Revisiting summarization evaluation metries. In Proceedings of the 28th Inter- national Conference on Computational Linguisties. pages 1011-711, Barcelona. Spain (Online). Inter- national Committee on Computational Linguistics.
Acknowledgements
This work is supported by a joint Microsoft Re- search- Mila grant. Yu Lu Liu is also supported by Fonds de Recherche du Québec Nature et Technologies master research scholarship (File #330991). Jackie C.K. Cheung is consulting re- searcher at Microsoft Research Montréal. We thank the hired paper annotators for their contribution: Martin Pömsl, Sabina Elkins, Arjun Vaithilingam Sudhakar, Akshatha Arodi, Andrei Mircea, Kushal Arora, and Cesare Spinoso-Dr Piano, We also thank
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: critical survey of "bias" in NLP. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5454- 5476, Online. Association for Computational Lin- guistics.

---

Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyp- ing Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 1004-1015, Online. As- sociation for Computational Linguistics.
tional Conference on Language Resources and Eval- uation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Hannah Devinney, Jenny Björklund, and Henrik Björk- lund, 2022. Theories of" gender" in nlp bias re- searchtheories of gender in natural language process- ing. In ACM FAccT Conference 2022, Conference on Fairness, Accountability, and Transparency, Hy- brid via Seoul, Soth Korea, June 21- 14,14,22 2022.
Margarita Boyarskaya, Alexandra Olteanu, and Kate Crawford. 2020. Overcoming Failures of Imagina- tion in Al Infused System Development and Deploy- ment. In Proceedings of the Navigating the Broader Impacts of Al Research Workshop.
Jay De Young, Iz Beltagy, Madeleine van Zuylen, Bai- ley Kuehl, and Lucy Wang. 2021. MS*2: Multi- document summarization of medical studies. In Pro- ceedings of the 202/ Conference on Empirical Meth- ods Natural Language Prucessing, pages 7494- 7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, and Saleema Amershi. 2023. Ahal: Facilitating im- pact assessment by generating examples of harms. arXiv preprint ar 2:36::6200032280
Yue Dong. 2018, A survey on neural network- based summarization methods. arXiv preprint 17Xx::8804.04589
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit. Cheung. 2020. Factual error correction for abstrac- tive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251-6255 Online. Association for Computational Linguistics.
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020. Multi-fact correction in abstractive text summariza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages tion for Computational Linguistics.
Online. Associa-
Shuyang Cao and Lu Wang. 2022. HIBRIDS: Atten-
tion with hierarchical biases for structure- long document summarivation. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 786-807, Dublin, Ireland. Association for Computa- tional Linguistics.
Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed, 2021. Automatic text summarization: A comprehensive survey. Ex- pert Systems with Applications, 165:113679)
Alexander R Fabbri, Wojciech Kryšciński,, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics, 9:391-409.
Giuseppe Carenini and Jackie C. K. Cheung. 2008. Ex- tractive vs. NLGG-aaed abstractive summarization of evaluative text: The effect of corpus controversial- ity. In Proceedings of the Fifth International Nat- ural Language Generation Conference, pages 33- 41. Salt Fork, Ohio, USA. Association for Compu- tational Linguistics.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An in- teresting but challenging application for natural lan- guage inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2214-2220, Florence, Italy, Associa- tion for Computational Linguistics.
Sangwoo Cho, Franck Dernoncour, Tim Ganter, Trung Bui, Nedim Lipka, Walter Chang, Hailin Jin, Jonathan Brandt, Hassan Foroosh, and Fei Liu, 2021. StreamHover: Livestream transcript summarization and annotation. In Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Processing, pages 1457-6474, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021. Language model notator: Exploring DialoGPT for dialogue summa- rization. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Nat- ural Language Processing ( (Volume 1: Long Papers), pages
Abhisek Dash, Anurag Shandilya, Arindam Biswas, Kripabandhu Ghosh, Saptarshi Ghosh, and Abhijnan Chakraborty. 2019. Summarizing usee-gene textual content: Motivation and methods for fair- ness in algorithmic summaries. Proc. ACM Hum.- Comput. Interact., 3(CSCW).
Online. Association for Computa- tional Linguistics.
Katy Ilnnk Gero, Payel Das, Pierre Dognin, Inkit Padhi, Prasanna Sattigeri, and Kush R. Varshney. 2023. The incentive gap data work in the era of large models. Nature Machine Intelligence.
Franck Deronncurr, Mohammad Ghassemi, and Wal- ter Chang. 2018 A repository of corpora for sum- marization. In Proceedings of the Eleventh Interna-

---

Dimitra Gkatzia and Saad Mahamood. 2015. snap- shot of NLG evaluation practices 2005 2014. In Proceedings of the 15th European Workshop on Nat- ural Language Generation (ENLG), pages 57-60, Brighton, UK. Association for Computational Lin- guistics.
Vijay Keswani and L. Elisa Celis. 2021. Dialect diver- sity in text summarization on twitter. In Proceed- ings of the Web Conference 2021, WWW "21, page 3802-3814, New York, NY, USA. Association for Computing Machinery.
Kundan Krishna, Jeffrey Bigham, and Zachary C. Lip- ton. 2021. Does pretraining for summarization re- quire knowledge transfer? In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 3178-311 7788-119.11 Punta Cana, Dominican Republic. Association for Computational Linguisties.
Seraphina Goldfarb- Tarrant, Eddic Ungless, Esma Balkir, and Su Lin Blodgett. 2023. This prompt is measuring <mask>: evaluating hias evaluation in language models. In Findings of the Association for Computational Linguistics: ACL 2023. pages 2209- 2225, Toronto, Canada. Association for Computa- tional Linguisties.
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguisties.
Christian Hardmeier, Marta R Costa-jussà, Kellie Web- ster, Will Radford, and Su Lin Blodgett. 2021. How to write a bias statement: Recommendations for submissions to the workshop gender bias in nlp. arXiv preprint arXiv: 104.00306.
Krtin Kumar and Jackie Chi Kit Cheung. 2019. Under- standing the Behaviour of Neural Abstractive Sum- marizers using Contrastive Examples. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Techmologies, Volume (Long and Short Pupers), pages 3949-3954, Min- neapolis, Minnesota. Association for Computational Linguistics.
David M. Howcroft, Anya Belz, Miruna-Adrian Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised def- initions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169-182, Dublin, Ireland. Association for Computa- tional Linguistics.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg. Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th Interna- tional Conference on Natural Language Generation, pages 355-368, Tokyo, Japan. Association for Com- putational Linguistics.
Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung- Hui Chang. 2022. Graph enhanced contrastive learning for radiology findings summa- rization. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 4677-4688, Dublin, Ireland. Association for Computational Linguistics.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out, pages 81.
Neslihan Iskender, Tim Polzehl, and Sebastian Möller. 2020. Towards reliable and robust methodology for crowd-based subjective quality assessment of query-based extractive text summarization. In Pro- ceedings of the 12th Language Resources and Evalu- ation Conference, pages 245-253, Marseille, France. European Language Resources Association.
Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2021. CSDS: A fine aine-graind Chinese dataset for customer service di- alogue summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 4436-4451, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yichen Jiang, Asli Celikyilmaz, Paul Smolensky, Paul Soulos, Sudha Rao. Hamid Palangi, Roland Fernan- dez, Caitlin Smith, Mohit Bansal, and Jianfeng Gao. 2021. Enriching transformers with structured tensor- product representations for abstractive summariza- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4780 4793, Online, Association for Computational Linguistics.
Siyi Liu, Sihao Chen, Xander Uyttendaele, and Dan Roth. 2021. MultiOpEd: A corpus of multi- perspective news editorials. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4345-4361, On- line. Association for Computational Linguisties.
Antonio Jimeno-Yeper Yepes, Laura Plaza, James G. Mork, Alan R. Aronson, and Alberto Díaz. 2013. Mesh in- dexing based on automatically generated summaries. BMC Bioinformatics, 14:208 - 208.
Elena Lloret, Laura Plaza, and Ahmet Aker. 2018. The challenging task of summary evaluation: overview. Language Resources and Evaluation. 12:1011148.

---

Inderject Mani. 2001. Automatic summarization, vol- ume 3. John Benjamins Publishing.
controllable lengths. In Proceedings of the 28th In- ternational Conference on Computational Linguis- tics, pages 1-812, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Sabine Molenaar. Lientje Maas, Verónica Burriel, Fabi- ano Dalpiaz, and Sjaak Brinkkemper. 2020. Medical Dialogue Summarization for Automated Reporting in Healthcare, pages 76-88.
Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. "this is a problem, don't i you agree?" framing and bias in human evaluation for natural language gener- ation. In Proceedings of the Ist Workshop on Evalu- ating NLG Evaluation, pages 10-16, Online (Dublin, Ireland). Association for Computational Linguistics.
James Mullenbach, Yada Pruksachatkun, Sean Adler, Jennifer Seale, Jordan Swartz, Greg McKelvey. Hui Dai, Yi Yang. and David Sontag. 2021. CLIP: A dataset for extracting action items for physicians from hospital discharge notes. In Proceedings of the 59th Annual Meeting of the Association for Compu- national Linguistics and the I 11th International Joint Conference on Natural Language Processing (Vol- une 1: Long Pupers), pages 1365-1378, Online. As- sociation for Computational Linguistics.
Anurag Shandilya, Abhisek Dash, Abhijnan Chakraborty, Kripabandhu Ghosh, and Saptarshi Ghosh. 2020. Fairness for whom? understanding the reader's 's perception of fairess in text summa- rization. In 2020 IEEE International Conference on Big Data (Big Data), pages 3692-3701.
Priyanka Nanayakkara, Jessica Hullman, and Nicholas Diakopoulos. 2021. Unpacking the expressed conse- quences of ai research in broader impact statements. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 795-806.
Anurag Shandilya, Kripabandhu Ghosh, and Saptarshi Ghosh. 2018. Fairness of extractive text summariza- tion. In Companion Proceedings of the The Web Conference 2018, WWW '18, page 97-98, Repub- lic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Ani Nenkova and Kathleen McKeown. 2012. A sur- vey of text summarization techniques. In Mining text data, pages 43-76. Springer.
Muskaan Singh, Tirthankar Ghosal, and Ondrej Bo- jar. 2021. An empirical performance analysis of state-of-ther thrtt--rttttttt summarization models for automatic minuting. In Proceedings of the 35th Pucific Asia Conference on Language, Information and Compu- tation, pages 50 50-60, Shanghai, China. Association for Computational Lingustics.
Jekaterina Novikova, Ondfej Dušek, Amanda Cer- Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Huy Quoc To, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen, and Anh Gia-Tuan Nguyen. 2021. Mono- lingual vs multilingual BERTology for Vietnamese extractive multi-documeet summarization. ceedings of the 35th Pacific Asia Conference on Lan- guage, Information and Computation, pages 692- 699, Shanghai, China. Association for Computa- tional Lingustics.
Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, and Jey Han Lau. 2022. The patient is more dead than alive: exploring the current state of the multi- document summarisation of the biomedical litera- ture. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 5098-5111, Dublin, Ire- land. Association for Computational Linguistics.
Pro-
Vu Tran, Minh Le Nguyen, Satoshi Tojo, and Ken Satoh. 2020. Encoded summarization: summariz- ing documents into continuous vector space for le- gal case retrieval. Artificial Intelligence and Law, 28(4):441-46)
Naday Oved and Ran Levy. 2021. PASS: Perturb-and select summarizer for product reviews. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 351-365, Online. Association for Computational Linguistics.
Nicolas Van Labeke, Denise Whitelock, Debora Field, Stephen Pulman, and John Richardson. 2013. Ope- nessayist: Extractive summarisation and formative assessment of free-text essays.
Sheena Panthaplackel, Adrian Benton, and Mark Dredze. 2022. Updated headline generation: Cre- ating updated summaries for evolving news stories. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pages 6438-6461, Dublin, Ireland. Association for Computational Linguistics.
Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Ken- ton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2022. Taxonomy of risks posed by language models. FAceT '22, page 214-229, New York, NY, USA. Association for Computing Machinery.
Ritesh Sarkhel, Moniba Keymanesh, Arnb Nandi, and Srinivasan Parthasarathy. 2020. Interpretable multi- headed attention for abstractive summarization at

---

Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid-base masked sentence pre-training for multi-document summa- rization. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 2245-5263, Dublin, Ireland. Association for Computational Linguisties.
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé 111. Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Computa- tional Linguisticx: Human Language Technologies, pages 314-324, Seattle, United States. Association for Computational Linguistics.
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fin query focused multi-document summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3632-3645, Online, Association for Computa- tional Linguistics.
Xinxin Zhu and James J Cimino. 2015. Clinicians' evaluation of computer-assisted medication summa- rization of electronic medical records. Comput. Biol. Med., 59:221-231.
Kexun Zhang, Jiaao Chen, and Diyi Yang. 2022. Fo- cus on the action: Learning to highlight and summa- rize jointly for email to-do items summarization. In Findings of the Association for Compulational Lin-
A Statistics on Paper Annotators
Annotators took on average around 23 minutes per paper. Excluding the exploratory review and Round 1, the number of papers coded by each an- notator is:
guistics: ACL 2022, pages 4095-4106, Dublin, Ire- land. Association for Computational Linguistics.
Longxiang Zhang, Renato Negrinho, Arindam Ghosh, Vasudevan Jagannathan, Hamid Reza Hassanzadeh, Thomas Schaaf, and Matthew R. Gormley. 2021a. Leveraging pretrained models for automatic summa- rization of doctor-patient conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3693-3712, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Round 2 totalling 131 papers
Author-Annotator 1: 15
Author-Annotator 2: 20
Author-Annotator 3: 5
Author-Annotator 4: 2
Author-Anootaoor 5: 84
Author-Anmotator 6: 5
Shiyue Zhang, Asli Celikyilmaz, Jianfeng Gao, and
Mohit Bansal. 2021b. EmailSum: Abstractive email
Round 3 totalling 142 papers
thread summarization. In Proceedings of the 59th
Hired Annotator 12
Annual Meeting of the Association for Computa-
Hired Annotator 2: 14
tional Linguistics and the 11th International Joint
Conference on Natural Language Processing (Vol-
Hired Annotator 3: 21
une 1: Long Papers), pages 6895-6909, Online. As-
Hired Annotator 4: 42
sociation for Computational Linguistics.
Hired Annotator 5: 42
Hired Annotator 6: 2
Xiang Zhang, Ping Geng, Tengteng Zhang, Qian Lu,
Peng Gao, and Jing Mei. 2020. Aceso: Pico-
Hired Annotator 7: 9
guided evidence summarization medical litera- ture. IEEE Journal of Biomedical and Health Infor- matics, (4(99)2663-2677)
While inter-annotator agreement (IAA) is often used as proxy for annotation quality, particularly when trying to determine some ground truth, our paper reviews are meant gold standard, and instead we looked for building consensus on ambiguous cases. We aim to surface issues in the current practices and provide rough estimates of how prevalent these issucs might be. To ensure quality, we recruited annotators with expertise in the field, and encourage frequent discussions of ambiguous cases.
Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020. Reducing quantity ballucinations in abstractive sum- marization. In Findings the Association for Com- putational Linguistics: EMNLP 2020, pages 2237- 2249, Online. Association for Computational Lin- guistics.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based -olti-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
B Methodology
Here, we provide additional details about protocols we followed while coding and analyzing the set of papers included in our review.

---

Code
Theme description
Num. Papers
Hannotator pay
considerations related to how the annotators were paid demographic other information related to expertise annotators how data proposed/used can accessed concerns about biases in datasets
#annotator, description #data access
5
#data bias
12
#stakeholder privacy
privacy concerns related to stakcholders' information/data concerns related to computational time and resources concerns related to factual errors, e.g., "hallucinations" describe the intended use domain, or the broader context
#compute_ info
#factuality
17
#intended_use use
gno_deploymeen
explicitly mentions that deployment is too premature, not ready
4
Table 2: Resulting codes and corresponding themes in "ethical considerations" sections.
Code
Theme description
Num. Papers
#novel words
issues related systems being unable to use words that are not present in the input text considerations related to the length of the input text or of the sumuarr the output summaries referencing the wrong gender pronouns/erms
#length
#misgender
#relevance
whether the information contained in the output is relevant, non-redune renrddandant
8
#info recall
whether key information from the source included by the output aspects related to factual consistency, e.g., "hallucinations" properties related to e.g., coherence, fluency, grammar
7
#factuality
29
#readability
12
Haaluree mode
specific circumstances where model/metriclete. perform poorly concerns about generalizability to other domains, languages, etc.
#doubt generalize
#weak methods
known or potential weaknesses with their methodology
#weak experiment
known or potential weaknesses with their experimental design
#complex use
using system or method is complex or requires extensive computational resources
Table 3: Resulting codes and corresponding themes in authors' discussions of the limitations of their own work.
B.1 Community Focus
domain was meant to be. For quality criteria, we examined the words authors frequently use to de- scribe the quality criteria they consider, and per- formed the following keyword searches to estimate how often authors consider these criteria:
To examine research goals, our analysis considered mentioned stakeholders as we were interested in how the authors envision anticipated or existing users to benefit from their work, and how these users described. For this, we first identified the papers that mention users. As we observed that the code other stakeholders was sometimes used denote users, we also manually filtered all papers coded other stakeholders for mentions of potential or existing users. When the description of research goals in these papers did not mention users, we revisited the papers to locate passages elaborating on how users benefit, which we then iteratively coded to identify the themes covered in Section 4.
information coverage: "relevan" (for rele- vant/relevance), "repetition", "informat" (for infornatiov/inoommivennss "redundancy", "salien" (for salient/saliency), and "content coverage"
information presentation: "fluen" (for fluent, flu- ency), "gramma" (for grammar, grammaticality), "readab" (for readable, readability), "coheren" (for coherent, coherence), "length", "novel" (for novel words)
factuality: "factual" (also for factuality), halluci- nat (for hallucinate, hallucination), faithful (also for faithfulness), consisten (for consistency), cor- rect (also for correctness).
To check whether commonly evaluated quality criteria such as factuality, information saliency, and linguistic propertics were also conceptualized as part of research goals, we used the same keywords listed in the next section ($B.2) to estimate the num- ber of papers focusing on these criteria (discussed in $4.2).
To estimate how frequently ROUGE- like auto- matic metrics are used, was tracked it using the tag "ROUGE" during the paper annotations.
B.3 Ethical Considerations
B.2 Evaluation Practices
After inspecting the annotators' summaries pro- vided by the annotators for the ethical consider- ation sections, we discarded papers which we found be mistakenly annotated as having an eth- ical consideration section: i) Krishna al. (2021): the annotation pulled passages from the abstract.
To surface insights about current evaluation prac- tices, we primarily examined aspects related the actual domain, as well as commonly considered quality criteria. For actual domain, we were in- terested in discrepancies with what the intended

---

There's ethical consideration section in the pa- per. ii) Mullenbach al. (2021): the paper has "potential impact" section in the introduction that we believe addresses the paper goal.
The specific codes we obtained after iteratively coding the ethical considerations sections to sur- face themes are listed in Table 2.
B.4 Limitations of one's own work
The specific codes we obtained after iteratively coding the passages about the authors discussions of the limitations of their own work are listed in Table 3.


Reviews:
[{"review_id": "lpVNKZ4cdn", "reviewer": "Reviewer_vqro", "summary": "", "questions": "(A) Did this study include workshops affiliated with main conferences? Shared tasks are a common feature of many workshops, and those are often explicitly built are system building.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and we are grateful that they are recognizing the contributions of our work. We will first address their general concerns, followed by answers to their questions.\n\n# Utility of Our Work\nWe first address the reviewer’s doubt about the work’s utility. We are somewhat confused by the reviewer’s comment (1) about **system performance and use of metrics**. The academic community has cared about metrics beyond ROUGE: prior work -- dating many years back -- has questioned the use of metrics like ROUGE (e.g., Murray et al., 2005) and proposed alternatives (e.g., Hovy et al., 2006). As for industry practitioners, as they need to deal with user facing settings, they often have to go beyond metrics like ROUGE (e.g., Zhou et al., 2022). We would appreciate it if the reviewer can clarify this comment and elaborate on their thoughts about the connection between the performance of current text summarization systems and the utility of our work.\n\nAs for the comment (2) about the **timing**, the reviewer is correct: NLP conferences have only recently started to encourage (and later require) ethical considerations, limitations, and other related sections; and thus these practices are fairly new. We believe it is helpful to take an early snapshot of emerging practices in the community. While the community has only recently started to prioritize these issues, we believe our work helps us understand, early on, why the community might be struggling with considering limitations of its work, ethical issues, etc. \n\n# Annotation Quality\nAbout annotation quality, we did not report agreement measures as our annotation was done via open-coding, meant to surface issues in the current practice and provide estimates of how prevalent these issues might be. We ensured annotation quality via selection of annotators (expertise in the field) and via frequent meetings to discuss ambiguous cases. We also provided details on the annotators and the annotation process in Section 3.2.2.\n\n# Questions\n### On whether workshop papers are included\nWe thank the reviewer for pointing this out to us! We did not include workshops affiliated with the main conference in our review. Not including this information was an oversight on our side. We will clarify this in the paper and add a note about it when discussing the limitations of our work.\n\n### Line 366-380 prevalence of system work\nWe thank the reviewer for raising this concern. The statistics we present are actually not about the impact or the importance of each type of contribution, but rather about their prevalence in the papers we reviewed. To avoid misunderstandings, we will clarify this by rephrasing the sentence at line 366: “Papers contributing new systems are prevalent, whereas comparatively less papers work on evaluation, metrics, datasets, or applications.”\n\nWe actually believe lines 373-380 are in line with the reviewer's view about the devaluation of data work, which we wholeheartedly agree with: \n> “This emphasis on developing new summarization systems, model, or techniques echoes concerns about the devaluation of e.g., data work which is often framed as ‘peripheral, rather than central’ to AI research (Gero et al., 2013). -- in contrast to the prestige of what is perceived to be more ‘technical’ work such as modelling or system building.”\n\nFinally, we thank the reviewer for identifying mistakes relating to style and presentation: we will define RAI before using this abbreviation and will delete “about” on line 65 as we annotated exactly 333 papers. \n\n# **References**\nEduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization evaluation with basic elements. In *Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)*, Genoa, Italy. European Language Resources Association (ELRA)\n\nGabriel Murray, Steve Renals, and Jezation Evaluation\nwith Basic Elements. Carletta. 2006. Evaluating automaticed summaries of meeting recordings. In *Proceedings of the ACL Workshop on Intrinsic and Extrinsic EFifth International Conference on Language Resources and Evaluation (LREC’06)*, Genoa, Italy. European Language Resources Association (ELRA).\n\nKaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. 2022. Deconstructing NLG evaluation: Evaluation practices, assumptions, and their implications. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 314–324, Seattle, United States. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very well written and easy to follow. It contains a high-quality analysis of recent summarization trends in the *ACL community, and highlights some commonalities of the research approaches. It makes some reasonable recommendations with respect to research focus and aligning researchers with their intended users, and finds that researchers are often not aligned.", "reasons_to_reject": "(A) I question the utility of this work at this time - (1) system performance has only recently reached performance levels that we, as an academic community, need to care about metrics beyond ROUGE (and that does not seem to have any impact on what industry does), (2) on any meaningful time-scale, the community has only recently started to prioritize ethics and limitations considerations. I am unaware of any longer-term study of the effectiveness, impact, or other longer term effects of these new section requirements on paper quality, content discussed, or informativeness of that content.\nI don’t see this as a true reason to reject this paper, but an incompletely analyzed aspect that should be considered when making research recommendations (are we already trending in the correct direction?).\n\n(B) There are no measures of annotation quality or agreement. The paper explicitly calls out papers for lacking details on human evaluation (line 545-) but fails to provide any quality measures of the evaluation process results.b\n", "questions_to_authors": "(A) Did this study include workshops affiliated with main conferences? Shared tasks are a common feature of many workshops, and those are often explicitly built are system building.\n", "paper_topic_and_main_contributions": "This paper presents a collection, annotation, and analysis of about 330 summarization papers published 2020-2022 with a focus on responsible AI. It describes an initial scoping annotation by the authors, resulting in an annotation scheme covering a variety of aspects (contributions, analysis, measurements, stated intent, and interactions with potential users). This annotation is used to identify several (sometimes implicitly bad) themes in summarization research, and the paper makes several explicit recommendations for performing summarization research.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Major:\nLine 366-380 “There is a focus on developing new systems, with comparatively…”: Alternatively the higher number of papers may be because developing a new model is simply easier than collecting quality data. Or modeling may have more money or (arbitrarily) news coverage. By not considering alternative explanations, this paper chooses a particular bias in understanding why certain work gets done. The models all use data-if we use citations (a crude measure of importance), are there more data citations, or are there more modeling citations in the field? Is the comparatively small number of metric development papers a sign that metrics are considered important, or more a sign that we (as a field) lack good metrics? My own views directly contradict the supposition in this paper: data work is more enduring and impactful than any particular system, and system work is in some sense disposable when a higher quality system arrives.\n\nNits:\nLine 53: RAI is used before definition.\nLine 65: about 333?\n\n"}}, {"review_id": "7vfXWIeHqe", "reviewer": "Reviewer_ao4U", "summary": "", "questions": "The authors conduct an interesting review on the current state of text summarization. They go through an extensive annotation process, making their results reliable (although I do have remarks on this at a later stage). \n\nHowever, my main gripe with this paper is that because the review is set up so broadly, a main point or goal seems to be lacking. The paper is set up as to focus on responsible AI, which is only a small part of their systematic review. The authors discuss various points related to paper authors & goals, data & evaluation practices, besides limitations & responsible AI. Therefore, it is difficult to find the main narrative in the findings: for instance, how does evaluation correspond with ethics, and stakeholder mentions? Is it a responsible AI issue that papers rarely mention stakeholders (and why is this)? What does it mean for responsible AI that work in summarization is largely driven by academic research? All these links are left implicit throughout the paper, so that it is difficult for readers to take away lessons and apply the findings to their own research. \n\nFor instance, when I read passages such as \"indeed, not having a clear application or domain in mind can make it difficult for authors to imagine users or other stakeholders\" I quite desired an extra step here which explicitly link this to responsible AI, by mentioning something like \"which, in turn, makes it more challenging to describe in what ways their work can be potentially harmful.\"\n\nFurthermore, I am missing an argument from literature that describes why for instance factuality can be an ethical issue, or what kind of harms to stakeholders can occur. Explaining the relevance and prevalence of certain ethical AI issues will also help to provide researchers with starting points to better their responsible AI considerations. For instance, statements such as \"despite the very real harms to which\nNLP systems can give rise (e.g., Bender, 2019; Weidinger et al., 2022)\" warrant further explanation on the very real harms.\n\nI will address this more specifically in a question format, and other questions related to clarity and methodological issues below.\n\n- General (see point made above): To what degree do the different themes from the systematic review relate to responsible AI issues?\n- General, but more specifically 131-134 \"While a great deal of work on responsible AI issues has emerged for NLP broadly, much less work has addressed summarization specifically.\" --> Why does summarization specifically need attention in this regard? How are developments there different from what is happening with NLP/NLG in general?\n- General (but mostly related to §4.3 and §4.4, see point made above) --> Could you provide more information on the relevance and prevalence of certain ethical/responsible AI issues?\n- 197-198: \"e.g., NLG papers where summarization is one of many evaluated tasks\" --> Does this also include papers that are not experimental in nature, such as papers discussing evaluation practices of summarization? This is addressed later on, but it would be good to make explicit that papers do not necessarily have to be experimental in nature here.\n- §3.2.1 --> Can you give details on the number of annotators and details on the annotators who did this exploratory review? Were these all the authors of this paper?\n- §3.2 --> Do you have any notes on the total time it took annotators to go through their batches? Any details on how many papers were coded by multiple annotators? Any IAA metrics?\n- 289 \"or when the paper explicitly intends to be general\" --> What is meant by this? Applicable to multiple domains?\n- §3.3.1 Research goals --> Could no taxonomy be made here?\n- §3.3.1 Research goals --> Could you give a brief explanation why this category is relevant (in the context of responsible AI)?\n- §3.3.1 Quality criteria --> What kind of taxonomy is used here? Is specific distinction made between automatic and human evaluation?\n- 583-588 \" data privacy (13 papers)—most specifying the stakeholders, who are either the people producing the data (e.g., professional writers of a scraped website) or the people described by the data (e.g., users of e-commerce websites where data is collected)\" --> In what way is privacy here related to the stakeholders most commonly?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughtful and detailed comments. We will first respond to the reviewer’s general concerns, then answer the more specific questions, and finally answer those related to style and presentation.\n\n# General Concerns\n## Our Focus on the Task of Summarization\nWe respectfully disagree that summarization is a narrow topic. Not only is there a rich literature on summarization (El-Kassas et al., 2021., Widyassari et al., 2022., etc.), but it is also one of the tracks at EMNLP this year.\n\nThis brings us to answer the reviewer's question: **“Why does summarization specifically need attention in this regard? How are developments there different from what is happening with NLP/NLG in general?”**\n\nAlthough findings from work on RAI in NLP/NLG could apply to summarization, there are task specificities that may be neglected (perhaps for good reasons, as these work aims for a much wider scope) that deserve their own examination. For example, summarizing necessarily involves omitting less relevant information from the source document -- unlike translation, for instance. Whether summarization systems fairly reflect different opinions or contributions from different authors is thus worth investigating (Carenini and Cheung (2008), Shandilya et al. (2018), Dash et al. (2019) which are cited in our related work section).\n\nIn the same way, due to these task specificities, there may be practices and RAI issues that are specific to summarization and are worth studying as they are not addressed in previous work. Other NLP/NLG tasks also have their own specificities and are also worth studying -- we hope that our methodology could be adapted to study, for instance, how practitioners engage with RAI issues in machine translation, or in dialogue generation, etc. \n\n## Background on Responsible AI Issues\nWe thank the reviewer for raising the concern that readers may not be familiar with responsible AI issues. We will add explanations and relevant literature to make it easier for readers to understand the rationale behind the aspects covered by our annotation scheme and the implications of the issues we identify. We will briefly outline them here, thereby also answer the reviewer’s first general question:\n\n- **Paper authors & goals** contextualize our findings. As we aim to examine how practitioners engage with RAI in their work, we need to know who are the practitioners (author affiliation), what is the work (type of contribution), and what are their intentions behind their work (research goals, intended domain). This also provides cues to potential usage scenarios, which may determine what harms are likely to occur. \n- **Evaluation practices** could both limit the space of RAI issues the community is aware of (further elaborated in the response to one of the reviewer’s clarification questions) and lead to RAI issues. For example, evaluating “general-purpose” systems on only news domain data could lead to practitioners being unaware of how said systems behave in other domains of application: do they perform reliably and safely?\n- As for **stakeholders**, the reviewer is right: not considering stakeholders makes it very challenging to think about potential harms (e.g., trying to imagine the harm of summarization systems in the medical domain without thinking about patients). \n\nAnother related point is about the **relevance and prevalence of certain issues** (also presented as the 3rd general question from the reviewer). We thank the reviewer for bringing it to our attention, and we will clarify the link between issues identified in our work and their relevance/prevalence in the broader literature. \n\nFor factuality-related ethical issues, for example, we will provide pointers to literature on the impacts of misinformation (e.g., Loomba et al., 2021 on the impact of misinformation on health) and lack of factuality in general or in domains like news (e.g., Wilner et al., 2022 on the impact of factual errors on media trust). \n\nThe importance of many ethical issues are currently explained in the cited works. For example, Weidinger et al., 2022 presents a taxonomy of risks covering discrimination, harmful stereotypes, privacy, impacts of false information, malicious use, etc. As suggested by the reviewer, we will draw clearer links between the work we cite and the issues we describe in text by providing as much explanation as we can in text. \n\n## Connection between Findings and Conclusion for the Field\nWe thank the reviewer for pointing out this problem to us. We now realize that our organization of the concluding paragraphs (recommendations in Section 4.4, followed by the summary of findings in Conclusion) has caused confusion. We will re-organize these paragraphs and clarify the implications and significance of the results by creating a new “Discussion” section where we will better summarize the findings, offer our recommendations, and better link the two:\n\n**A. Intended use contexts are often not well-described** (currently conveyed in conclusion (i)). We view intended use contexts through 2 lenses:\n- Intended domain of application (i.e., where the work is intended to be used): There are many works that do not have a specific domain of application, implicitly intended to be “general-purpose” (Ln 387).\n- Mentioned stakeholders (i.e., who are the intended users or people impacted by the work): few works mention stakeholders when imagining intended use contexts (Ln 406). Imagined benefits are also quite narrow in scope (Ln 444).\n\n**B. Quality criteria such as bias, fairness, etc., are rarely considered.** (currently conveyed in conclusion (iii)). The priority in evaluation is on information saliency, linguistic properties, and factuality (Ln 486). \n\n**C. We see a lack of engagement with limitations of own work and ethical concerns** (currently partially conveyed in conclusion (i) and (ii)):\n- Most works do not have discussions on their own limitations, ethical considerations, etc. (Ln 562). Work that do often focus on data-related concerns (which is not wrong by itself, but could be indicative of a narrow range of ethical concerns practitioners might be aware of)\n- Model failures (e.g., factual errors) are rarely conceptualized as ethical concerns. (Ln 605, Ln 627)\n- When considering ethical considerations, intended use contexts (including stakeholders) are rarely involved. Potential harm to stakeholders are thus often overlooked (Ln 653, Ln 673)\n\nWith these 3 points presented first, it would be hopefully easier to understand where our recommendations comes from:\n- Our 1st recommendation (try conceptualizing the intended use context of a contribution) is directly connected to point A above. \n- Our second recommendation (being explicit about how a “good” summary is conceptualized) is directly connected to point B above and to our findings about domain mismatch in evaluation (Ln 473), reliance on ROUGE and similar metrics (Ln 533), and underspecified human evaluation (Ln 545). \n- Remaining recommendations about engagement with ethical concerns are all tied to point C above.\n\n# Specific Questions\n### Ln 197-198 papers that are not experimental in nature \nNo, we did not filter collected papers based on whether they are experimental in nature or not. We will follow the reviewer’s suggestion and emphasize this. Here, we aimed to exclude papers where summarization is not the main focus. Using the reviewer’s example, a paper discussing evaluation practices in summarization would be included whereas a paper discussing evaluation practices of NLG (covering other tasks such as machine translation, question generation, etc.) would be filtered out.\n\n### §3.2.1: details on the exploratory review\nThe exploratory review is indeed done by the authors of this paper, a team of 6 people. We unfortunately cannot provide further details on our experience/background due to concerns about anonymity. \n\n### §3.2: details on annotation time, number of papers/annotator, inter-annotator agreement\nOnce the annotation scheme was finalized, annotators took on average around 23 minutes per paper. Excluding the exploratory review and Round 1, the number of papers each annotator coded is as follows:\n\n(Round 2 totalling 131 papers)\n- Author-Annotator 1: 15\n- Author-Annotator 2: 20\n- Author-Annotator 3: 5\n- Author-Annotator 4: 2\n- Author-Annotator 5: 84\n- Author-Annotator 6: 5\n\n(Round 3 totalling 142 papers)\n- Hired Annotator 1: 12\n- Hired Annotator 2: 14\n- Hired Annotator 3: 21\n- Hired Annotator 4: 42\n- Hired Annotator 5: 42\n- Hired Annotator 6: 2\n- Hired Annotator 7: 9\n\nAs for inter-annotator agreement, we understand that it is often used as a proxy for annotation quality, particularly when trying to determine some ground truth. Our annotations are, however, not meant as gold standard; rather, the goal of our annotation process is to surface issues in the current practice and provide estimates of how prevalent these issues might be. We thank the reviewer for drawing our attention to these statistics. We will include them in the appendix.\n\n### Ln 289: papers explicitly intending to be general\nWhen a paper explicitly intends to be general, it explicitly states that its contribution is general-purpose, and that it can be used in any domain or application. We will add a sentence clarifying this.\n\n### §3.3.1: research goals\nWe did not make a taxonomy for research goals because authors describe their goals in many different ways; we sought to capture these descriptions as faithfully/directly as possible, and thus kept this category open-ended. We address the second question on research goals in our response to the reviewer’s general concerns above.\n\n### §3.3.2: quality criteria \nWe did not use a taxonomy when annotating for “quality criteria”, nor did we make a distinction between automatic and human evaluation; like the “research goals” category, this category was kept open-ended, where annotators extracted text properties that papers mention when describing summarization systems’ evaluation.\n\n### Ln 583-588: data privacy and stakeholders\nWe are not sure we understood the reviewer's question, but we will try to answer it by clarifying this passage. We meant here that when data privacy is brought up as an ethical concern, the authors often specify the people who could be impacted: people producing the data and people described by the data (the mentioned stakeholders). We could imagine, for instance, that a dataset that is not properly anonymized could leak private information of these stakeholders, thereby harming them. We hope this clarification answers the reviewer's question. If not, we ask the reviewer to please elaborate and we will answer in our future reply. \n\n# Questions Related to Style and Presentation\nWe thank the reviewer for identifying the “RAI” abbreviation issue and nested parentheses, and we will fix them. \nWe also thank the reviewer for identifying instances where we should have presented exact numbers and percentages. We will do so in our revised version:\n- Ln 385: 54.5% → 122 out of 224 papers contributing new systems.\n- Ln 530-532: \n    - Factuality Related: 32 in goals, 46 in prior work limitations;\n    - Information saliency: 27 in goals, 44 in prior work limitations;\n    - Linguistic properties (e.g., fluency, novel words, length, etc.): 17 in goals, 29 in prior work limitations;\n\nFinally, to clarify **Ln 128-130**, we meant that it is possible that some evaluation practices (e.g., the use of ROUGE, evaluating for factual consistency, etc.) might lead to some RAI issues being considered while others might be overlooked by the community (i.e. “failure of imagination”). To illustrate, if not evaluating for a certain property (e.g., factuality) is a common evaluation practice, then model failures related to said property (e.g., factual errors in output summaries) would likely be overlooked as a potential cause of harm. If only using news domain data is a common practice, then the community might fail to imagine how summarization systems could be used/misused and cause harm in other domains of application. We will reformulate this part to make this point clear to readers.\n\n# References\nWafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021. Automatic text summarization: A comprehensive survey. *Expert Systems with Applications*, 165:113679.\n\nSahil Loomba, Alexandre de Figueiredo, Simon J. Piatek, Kristen de Graaf, and Heidi J. Larson. 2021. Measuring the impact of covid-19 vaccine misinformation on vaccination intent in the uk and usa. *Nature Human Behaviour*, 5(3):337–348.\n\nAdhika Pramita Widyassari, Supriadi Rustad, Guruh Fajar Shidik, Edi Noersasongko, Abdul Syukur, Affandy Affandy, and De Rosal Ignatius Moses Setiadi. 2022. Review of automatic text summarization techniques methods. *Journal of King Saud University - Computer and Information Sciences*, 34(4):1029–1046.\n\nTamar Wilner, Ryan Wallace, Ivan Lacasa-Mas, and\nEmily Goldstein. 2022. The tragedy of errors: Political ideology, perceived journalistic quality, and media trust. *Journalism Practice*, 16(8):1673–1694.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough systematic review, with a well-described annotation process\n- Interesting results for the field", "reasons_to_reject": "- Paper lacks coherence: connection between the results and the conclusion for the field are missing\n- Narrow topic (summarization)\n- Background information on why responsible AI issues are actually such issues is often left implicit.", "questions_to_authors": "The authors conduct an interesting review on the current state of text summarization. They go through an extensive annotation process, making their results reliable (although I do have remarks on this at a later stage). \n\nHowever, my main gripe with this paper is that because the review is set up so broadly, a main point or goal seems to be lacking. The paper is set up as to focus on responsible AI, which is only a small part of their systematic review. The authors discuss various points related to paper authors & goals, data & evaluation practices, besides limitations & responsible AI. Therefore, it is difficult to find the main narrative in the findings: for instance, how does evaluation correspond with ethics, and stakeholder mentions? Is it a responsible AI issue that papers rarely mention stakeholders (and why is this)? What does it mean for responsible AI that work in summarization is largely driven by academic research? All these links are left implicit throughout the paper, so that it is difficult for readers to take away lessons and apply the findings to their own research. \n\nFor instance, when I read passages such as \"indeed, not having a clear application or domain in mind can make it difficult for authors to imagine users or other stakeholders\" I quite desired an extra step here which explicitly link this to responsible AI, by mentioning something like \"which, in turn, makes it more challenging to describe in what ways their work can be potentially harmful.\"\n\nFurthermore, I am missing an argument from literature that describes why for instance factuality can be an ethical issue, or what kind of harms to stakeholders can occur. Explaining the relevance and prevalence of certain ethical AI issues will also help to provide researchers with starting points to better their responsible AI considerations. For instance, statements such as \"despite the very real harms to which\nNLP systems can give rise (e.g., Bender, 2019; Weidinger et al., 2022)\" warrant further explanation on the very real harms.\n\nI will address this more specifically in a question format, and other questions related to clarity and methodological issues below.\n\n- General (see point made above): To what degree do the different themes from the systematic review relate to responsible AI issues?\n- General, but more specifically 131-134 \"While a great deal of work on responsible AI issues has emerged for NLP broadly, much less work has addressed summarization specifically.\" --> Why does summarization specifically need attention in this regard? How are developments there different from what is happening with NLP/NLG in general?\n- General (but mostly related to §4.3 and §4.4, see point made above) --> Could you provide more information on the relevance and prevalence of certain ethical/responsible AI issues?\n- 197-198: \"e.g., NLG papers where summarization is one of many evaluated tasks\" --> Does this also include papers that are not experimental in nature, such as papers discussing evaluation practices of summarization? This is addressed later on, but it would be good to make explicit that papers do not necessarily have to be experimental in nature here.\n- §3.2.1 --> Can you give details on the number of annotators and details on the annotators who did this exploratory review? Were these all the authors of this paper?\n- §3.2 --> Do you have any notes on the total time it took annotators to go through their batches? Any details on how many papers were coded by multiple annotators? Any IAA metrics?\n- 289 \"or when the paper explicitly intends to be general\" --> What is meant by this? Applicable to multiple domains?\n- §3.3.1 Research goals --> Could no taxonomy be made here?\n- §3.3.1 Research goals --> Could you give a brief explanation why this category is relevant (in the context of responsible AI)?\n- §3.3.1 Quality criteria --> What kind of taxonomy is used here? Is specific distinction made between automatic and human evaluation?\n- 583-588 \" data privacy (13 papers)—most specifying the stakeholders, who are either the people producing the data (e.g., professional writers of a scraped website) or the people described by the data (e.g., users of e-commerce websites where data is collected)\" --> In what way is privacy here related to the stakeholders most commonly?", "paper_topic_and_main_contributions": "This paper provides a systematic review on responsible AI practices within the subfield of summarization. The authors start with an extensive annotation procedure where they look more specifically at (1) paper authors & goals, (2) data & evaluation practices, and (3) limitations & responsible AI. The main goal of the paper is to investigate to what degree the implications of summarization work is reported on. The authors find that this is currently not being done thoroughly.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 053 \"RAI\" --> This abbreviation needs to be defined first.\n- 128-130 \"Our work examines how assessment practices might limit the space of responsible AI issues the community considers.\" --> As a reader, I find it hard to understand what is meant with \"assessment practices\" and \"limit the space of responsible AI issues\". Could you reformulate this?\n- 385 \"more than half\" --> Can you give an exact percentage?\n- 530-532 \"Linguistic properties and criteria related to information saliency are, in comparison, less frequently mentioned\" --> Could you specify how often they are mentioned?\n- 570-571 \"(∼20% (19 out of 92) than those proposing systems (∼10% (23 out of 224))\" --> This is somewhat of a personal preference, but try to avoid nested parentheses."}}, {"review_id": "go5c9SWKoh", "reviewer": "Reviewer_8kcV", "summary": "", "questions": "I have the following questions regarding the paper:\n1. Can you provide an explanation of how the three findings mentioned in the Conclusion (Section 5) are related to the findings presented in Section 4? \n2. Have you conducted an analysis of the usability of the methods described in the papers you examined? For instance, did you investigate how easily the results of these methods can be reproduced, or whether any of these methods have adverse effects on the environment? ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and will address their general concerns, followed by answers to their questions.\n\n# General Concerns\n## Analysis Scope\nWe actually looked at both metadata AND content of each annotated paper. Many aspects in our annotation scheme concern the content of the paper: how the paper authors discuss limitations of their own work, the stakeholders they mention (if any), what they discuss within their paper’s ethical consideration section (if there is such a section), etc. Using a paper’s metadata alone, this valuable information is unobtainable.\n\n## Representative Examples\nWe agree that including more representative examples will strengthen our findings. To improve the current version, we will add more quotes to illustrate a certain practice some of the annotated papers are doing (e.g., in lines 613-616, we quoted from Panthaplackel et al., 2022 to illustrate specific descriptions of factuality-related issues). \n\n## Clarity in Findings\nWe thank the reviewer for pointing out this problem to us. We now realize that our organization of the concluding paragraphs (recommendations in Section 4.4, followed by the summary of findings in the Conclusion) has caused confusion. We will re-organize these paragraphs and clarify the implications and significance of the results by creating a new “Discussion” section where we will better summarize the findings, offer our recommendations, and better link the two. \n\n# Questions\n### Q1: link between findings and conclusion\nThe above point leads us to answer the reviewer’s Q1: The three points presented in the conclusion were meant to summarize the findings in Section 4 (except for recommendations in 4.4). The following improved structure might offer more clarity:\n\n**A. Intended use contexts are often not well-described** (currently conveyed in conclusion (i)). We view intended use contexts through 2 lenses:\n- Intended domain of application (i.e., where the work is intended to be used): There are many works that do not have a specific domain of application, implicitly intended to be “general-purpose” (Ln 387).\n- Mentioned stakeholders (i.e., who are the intended users or people impacted by the work): few works mention stakeholders when imagining intended use contexts (Ln 406). Imagined benefits are also quite narrow in scope (Ln 444).\n\n**B. Quality criteria such as bias, fairness, etc., are rarely considered.** (currently conveyed in conclusion (iii)). The priority in evaluation is on information saliency, linguistic properties, and factuality (Ln 486). \n\n**C. We see a lack of engagement with limitations of own work and ethical concerns** (currently partially conveyed in conclusion (i) and (ii)):\n- Most works do not have discussions on their own limitations, ethical considerations, etc. (Ln 562). Work that do often focus on data-related concerns (which is not wrong by itself, but could be indicative of a narrow range of ethical concerns practitioners might be aware of)\n- Model failures (e.g., factual errors) are rarely conceptualized as ethical concerns. (Ln 605, Ln 627)\n- When considering ethical considerations, intended use contexts (including stakeholders) are rarely involved. Potential harm to stakeholders are thus often overlooked (Ln 653, Ln 673)\n\n### Q2: on analyzing the usability, reproducibility, impact on environment, etc. of the methods\nNo, we did not conduct such an analysis. Our study did not examine properties of the methods such as how reproducible they are, or their impact on the environment. Rather, we focus on how authors engage with these aspects, and potential RAI issues that could arise. To illustrate, instead of investigating how much computational resources each annotated paper actually used (i.e, not the focus of the paper), we noted, in Appendix C.2.3, that 4 papers described their use of computational resources when discussing limitations of their own work (i.e, the authors of these papers *engaged* with the usage of computational resources *when imagining limitations of their own work*).\n\nFinally, we thank the reviewer for their suggestion on including visualization. We will consider doing so in our revised version if space permits. We will also fix the missing number on Ln 534: it should read 200 of 224.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are two main reasons to accept the paper:\n\n1. **Creation of Annotated Dataset**: The authors have meticulously developed an annotated dataset containing summarization papers from ACL Anthology spanning the years 2020 to 2022. This dataset includes essential information such as paper goals, authors, evaluation practices, limitations, and ethical considerations. This comprehensive resource could facilitate further analysis in this domain enabling exploration into how text summarization has been influenced by other NLP tasks and how recent technological advancements have impacted the field.\n\n2. **Systematic Summarization and Clear Structure**: The paper presents a systematic summary of all the research findings, supported by statistical evidence. Moreover, the authors have thoughtfully organized the findings, enhancing the overall clarity and coherence of the paper. ", "reasons_to_reject": "There are three primary reasons to reject the paper:\n\n1. **Limited Analysis Scope**: The paper only focuses on analyzing the metadata of the papers, neglecting valuable information present in the contents of these papers. A more comprehensive analysis that incorporates the actual content could have strengthened the presented findings with additional evidence and insights.\n\n2. **Superficial Analysis and Weak Evidence**: The analysis conducted in the paper is deemed superficial by me (I agree that this opinion is subjective), lacking sufficient strong evidence to support the claims made. For instance, at line 571, a claim is stated without concrete examples from the corpus to substantiate it (\"Work without such explicit reflections may not be .....\"). Including representative examples could have provided more robust evidence for the conclusions.\n\n3. **Lack of Clarity in Findings**: The presented findings do not clearly explain the direction of research in text summarization. The paper appears to fall short in effectively conveying the implications and significance of the results.", "questions_to_authors": "I have the following questions regarding the paper:\n1. Can you provide an explanation of how the three findings mentioned in the Conclusion (Section 5) are related to the findings presented in Section 4? \n2. Have you conducted an analysis of the usability of the methods described in the papers you examined? For instance, did you investigate how easily the results of these methods can be reproduced, or whether any of these methods have adverse effects on the environment? ", "paper_topic_and_main_contributions": "This paper provides a meta-analysis of Text Summarization Papers, focusing on their alignment with responsible AI considerations. By examining how well research papers on text summarization address ethical and responsible AI practices, the study sheds light on the field's current state and potential implications for developing more responsible AI models.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 534, please include the missing value for X to represent (X of 224).\n2. To better explain the findings, consider adding visualizations to enhance clarity. Visual aids can help readers better grasp the presented information."}}]

Author rebuttal:

Meta-review: The paper "Responsible AI Considerations in Text Summarization Research: A Review of Current Practices" presents a survey of how ethical issues are dealt with in the context of automatic summarization. 

The main criticism voiced by the reviews refers to the narrow scope of the paper and the lack in connection between aspects of the paper. 
Points in favour of the paper are the details of the survey and the data set created for this paper.
Decision: Accept-Findings