SlowGuess/NAACL_2022_OCR huggingface dataset:
Maartje ter Hoeve  
University of Amsterdam  
m.a.terhoeve@uva.nl
Julia Kiseleva  
Microsoft Research  
julia.kiseleva@microsoft.com
Maarten de Rijke  
University of Amsterdam  
m.derijke@uva.nl
Automatic text summarization has enjoyed great progress over the years and is used in numerous applications, impacting the lives of many. Despite this development, there is little research that meaningfully investigates how the current research focus in automatic summarization aligns with users' needs. To bridge this gap, we propose a survey methodology that can be used to investigate the needs of users of automatically generated summaries. Importantly, these needs are dependent on the target group. Hence, we design our survey in such a way that it can be easily adjusted to investigate different user groups. In this work we focus on university students, who make extensive use of summaries during their studies. We find that the current research directions of the automatic summarization community do not fully align with students' needs. Motivated by our findings, we present ways to mitigate this mismatch in future research on automatic summarization: we propose research directions that impact the design, the development and the evaluation of automatically generated summaries.
The field of automatic text summarization has experienced great progress over the last years, especially since the rise of neural sequence to sequence models (e.g., Cheng and Lapata, 2016; See et al., 2017; Vaswani et al., 2017). The introduction of self-supervised transformer language models like BERT (Devlin et al., 2019) has given the field an additional boost (e.g., Liu et al., 2018; Liu and Lapata, 2019; Lewis et al., 2020; Xu et al., 2020).
The—often implicit—goal of automatic text summarization is to generate a condensed textual version of the input document(s), whilst preserving the main message. This is reflected in today's most common evaluation metrics for the task; they focus on aspects such as informativeness, fluency, succinctness and factuality (e.g., Lin, 2004; Nenkova
and Passonneau, 2004; Paulus et al., 2018; Narayan et al., 2018b; Goodrich et al., 2019; Wang et al., 2020; Xie et al., 2021). The needs of the users of the summaries are often not explicitly addressed, despite their importance in explicit definitions of the goal of automatic summarization (Spärck Jones, 1998; Mani, 2001a). Mani defines this goal as: "to take an information source, extract content from it, and present the most important content to the user in a condensed form and in a manner sensitive to the user's or application's needs."
Different user groups have different needs. Investigating these needs explicitly is critical, given the impact of adequate information transfer (Bennett et al., 2012). We propose a survey methodology to investigate these needs. In designing the survey, we take stock of past work by Spärck Jones (1998) who argues that in order to generate useful summaries, one should take the context of a summary
Human Language Technologies, pages 46 - 75
into account—a statement that has been echoed by others (e.g., Mani, 2001a; Aries et al., 2019). To do this in a structured manner, Sparck Jones introduces three context factor classes: input factors, purpose factors and output factors, which respectively describe the input material, the purpose of the summary, and what the summary should look like. We structure our survey and its implications around these factors. In Figure 1 we give an example of incorporating the context factors in the design of automatic summarization methods.
Our proposed survey can be flexibly adjusted to different user groups. Here we turn our focus to university students as a first stakeholder group. University students are a particularly relevant group to focus on first, as they benefit from using pre-made summaries in a range of study activities (Reder and Anderson, 1980), but the desired characteristics of these pre-made summaries have not been extensively investigated. We use the word pre-made to differentiate such summaries from the ones that users write themselves. Automatically generated summaries fall in the pre-made category, and should thus have the characteristics that users wish for pre-made summaries.
Motivated by our findings, we propose important future research directions that directly impact the design, development, and evaluation of automatically generated summaries. We contribute the following:
C1 We design a survey that can be easily adapted and reused to investigate and understand the needs of the wide variety of users of automatically generated summaries;
C2 We develop a thorough understanding of how automatic summarization can optimally benefit users in the educational domain, which leads us to unravel important and currently underexposed research directions for automatic summarization;
C3 We propose a new, feasible and comprehensive evaluation methodology to explicitly evaluate the usefulness of a generated summary for its intended purpose.
In Section 1 we introduced the context factors as proposed by Spärck Jones (1998). Each context factor class can be divided into more fine-grained subclasses. To ensure the flow of the paper, we list an overview in Appendix A. Below, we explain
and use the context factors and their fine-grained subclasses to structure the related work. As our findings have implications for the evaluation of automatic summarization, we also discuss evaluation methods. Lastly, we discuss the use-cases of automatic summaries in the educational domain.
2.1 Automatic text summarization
Input factors. We start with the fine-grained input factor unit, which describes how many sources are to be summarized at once, and the factor scale, which describes the length of the input data. These factors are related to the difference between single and multi-document summarization (e.g., Chopra et al., 2016; Cheng and Lapata, 2016; Wang et al., 2016; Yasunaga et al., 2017; Nallapati et al., 2017; Narayan et al., 2018b; Liu and Lapata, 2019). Scale plays an important role when material shorter than a single document is summarized, such as sentence summarization (e.g., Rush et al., 2015). Regarding the genre of the input material, most current work focuses on the news domain or Wikipedia (e.g., Sandhaus, 2008; Hermann et al., 2015; Koupaee and Wang, 2018; Liu et al., 2018; Narayan et al., 2018a). A smaller body of work addresses different input genres, such as scientific articles (e.g., Cohan et al., 2018), forum data (e.g., Volske et al., 2017), opinions (e.g., Amplayo and Lapata, 2020) or dialogues (e.g., Liu et al., 2021). These differences are also closely related to the input factor subject type, which describes the difficulty level of the input material. The factor medium refers to the input language. Most automatic summarization work is concerned with English as language input, although there are exceptions, such as Chinese (e.g., Hu et al., 2015) or multilingual input (Ladhak et al., 2020). The last input factor is structure. Especially in recent neural approaches, explicit structure of the input text is often ignored. Exceptions include graph-based approaches, where implicit document structure is used to summarize a document (e.g., Tan et al., 2017; Yasunaga et al., 2017), and summarization of tabular data (e.g., Zhang et al., 2020a) or screenplays (e.g., Papalampidi et al., 2020).
Purpose factors. Although identified as the most important context factor class by Spärck Jones (1998)—and followed by, for example, Mani (2001a)—purpose factors do not receive a substantial amount of attention. There are some exceptions, e.g., query-based summarization (e.g., Nema et al., 2017; Litvak and Vanetik, 2017), question-driven
summarization (e.g., Deng et al., 2020), personalized summarization (e.g., Móro and Bieliková, 2012) and interactive summarization (e.g., Hirsch et al., 2021). They take the situation and the audience into account. The use-cases of the generated summaries are also clearer in these approaches.
Output factors. We start with the output factors style and material. The latter is concerned with the degree of coverage of the summary. Most generated summaries have an informative style and cover most of the input material. There are exceptions, e.g., the XSum dataset (Narayan et al., 2018a) which constructs summaries of a single sentence and is therefore more indicative in terms of style and inevitably less of the input material is covered. Not many summaries have a critical or aggregative style. Aggregative summaries put different source texts in relation to each other, to give a topic overview. Most popular summarization techniques focus on a running format. Work on template-based (e.g., Cao et al., 2018) and faceted (e.g., Meng et al., 2021) summarization follows a more headed (structured) format. Falke and Gurevych (2017) build concept maps and Wu et al. (2020) make knowledge graphs. The difference between abstractive and extractive summarization is likely the best known distinction in output type (e.g., Nallapati et al., 2017; See et al., 2017; Narayan et al., 2018b; Gehrmann et al., 2018; Liu and Lapata, 2019), although it is not entirely clear which output factor best describes the difference.
In Section 5 we use the context factors to identify future research directions, based on the difference between our findings and the related work.
Evaluation methods for automatic summarization can be grouped in intrinsic vs. extrinsic methods (Mani, 2001b). Intrinsic methods evaluate the model itself, e.g., on informativeness or fluency (Paulus et al., 2018; Liu and Lapata, 2019). Extrinsic methods target how a summary performs when used for a task (Dorr et al., 2005; Wang et al., 2020). Extrinsic methods are resource intensive, explaining the popularity of intrinsic methods.
Evaluation methods can also be grouped in automatic vs. human evaluation methods. Different automatic metrics have been proposed, such as Rouge (Lin, 2004) and BERTScore (Zhang et al., 2020b) which respectively evaluate lexical and semantic similarity. Other methods use an au
tomatic question-answering evaluation methodology (Wang et al., 2020; Durmus et al., 2020). Most human evaluation approaches evaluate intrinsic factors such as informativeness, readability and conciseness (DUC, 2003; Nallapati et al., 2017; Paulus et al., 2018; Liu and Lapata, 2019)—factors that are difficult to evaluate automatically. There are also some extrinsic human evaluation methods, where judges are asked to perform a certain task based on the summary (e.g., Narayan et al., 2018b). So far, usefulness<sup>1</sup> has not been evaluated in a feasible and comprehensive manner, whereas it is an important metric to evaluate whether summaries fulfil users' needs. Therefore, we bridge the gap by introducing a feasible and comprehensive evaluation methodology to evaluate usefulness.
2.3 Automatic summarization for education
Summaries play a prominent role in education. Reder and Anderson (1980) find that students who use a pre-made summary score better on a range of study activities than students who do not use such a summary. As the quality of automatically generated summaries increases (e.g., Lewis et al., 2020; Xu et al., 2020), so does the potential to use them in the educational domain, especially given the increasing importance of digital tools and devices for education (Luckin et al., 2012; Hashim, 2018). With these developments in mind, it is critical that educators are aware of the pedagogical implications; they need to understand how to best make use of all new possibilities (Hashim, 2018; Amhag et al., 2019). The outcomes of our survey result in concrete suggestions for developing methods for automatic summarization in the educational domain, whilst taking students' needs into account.
3 Survey Procedure and Participants
Here we detail our survey procedure. For concreteness, we present the details with our intended target group in mind. The context factors form the backbone of our survey and the setup can be easily adjusted to investigate the needs of different target groups. For example, we ask participants about a pre-made summary for a recent study activity, but it is straightforward to adapt this to a different use-case that is more suitable for other user groups.
<sup>1</sup>We follow the definition of the English Oxford Learner's Dictionary (www.oxfordlearnersdictionaries.com/definition/english/) for usefulness: "the fact of being useful or possible to use", where useful is defined as "that can help you to do or achieve what you want".
(b) Study backgrounds.
Figure 2: Participant details.
We recruited participants among students at universities across the Netherlands by contacting ongoing courses and student associations, and by advertisements on internal student websites. As incentive, we offered a ten euro shopping voucher to ten randomly selected participants.
A total of 118 participants started the survey and 82 completed the full survey, resulting in a  $69.5\%$  completion rate. We only include participants who completed the study in our analysis. Participants spent 10 minutes on average on the survey. In the final part of our survey we ask participants to indicate their current level of education and main field of study. The details are given in Figure 2.
3.2 Survey procedure
Figure 3 shows a brief overview of our survey procedure. A detailed account is given in Appendix B. We arrived at the final survey version after a number of pilot runs where we ensured participants understood their task and all questions. We ran the survey with SurveyMonkey (surveymonkey.com). A verbatim copy is included in Appendix C and released under CC BY license.2
Introduction. The survey starts with an introduction where we explain what to expect, how we process the data and that participation is voluntary. After participants agree with this, an explanation of the term pre-made summary follows. As we do not want to bias participants by stating that the summary was automatically generated, we explain that the summary can be made by anyone, e.g., a teacher, a good performing fellow student, the authors of the original material, or a computer. Recall that an automatically generated summary is a pre-made summary. Hence, our survey identifies the characteristics an automatically generated summary should have. We also give examples of
types of pre-made summaries; based on the pilot experiments we noticed that people missed this information. We explicitly state that these are just examples and that participants can come up with any example of a helpful pre-made summary.
Context factors. In the main part of our survey we focus on the context factors. First, we ask participants whether they have made use of a pre-made summary in one of their recent study activities. If so, we ask them to choose the study activity where a summary was most useful. We call this group the Remembered group, as they describe an existing summary from memory. If people indicate that they have not used a pre-made summary in one of their recent study activities, we ask them whether they can imagine a situation where a pre-made summary would have been helpful. If not, we ask them why not and lead them to the final background questions and closing page. If yes, we ask them to keep this imaginary situation in mind for the rest of the survey. We call this group the Imagined group.
Now we ask the Remembered and Imagined groups about the input, purpose and output factors of the summary they have in mind. We ask questions for each of the context factor subclasses that we discussed in Section 2. At this point, the two groups are in different branches of the survey. The difference is mainly linguistically motivated: in the Imagined group we use verbs of probability instead of asking to describe an existing situation. Some questions can only be asked in the Remembered group, e.g., how helpful the summary was.
In the first context factor question we ask what the study material consisted of. We give a number of options, as well as an 'other' checkbox. To avoid position bias, all answer options for multiple choice and multiple response questions in the survey are randomized, with the 'other' checkbox always as the last option. If participants do not choose the 'mainly text' option, we tell them that we focus on textual input in the current study<sup>3</sup> and ask whether they can think of a situation where the input did consist of text. If not, we lead them to the background questions and closing page. If yes, they proceed to the questions that give us a full overview of the input, purpose and output factors of the situation participants have in mind. Finally, we ask the Remembered group to suggest how their described summary could be turned into their ideal
2https://github.com/maartjeth/survey_useful_summarization
3 Different modalities are also important to investigate, but we leave this for future work to ensure clarity of our results.
Figure 3: Overview of the survey procedure.
summary. We then ask both groups for any final remarks about the summary or input material.
Trustworthiness and future features questions. So far we have included the possibility that the summary was machine-generated, but also explicitly included other options so as not to bias participants. At this point we acknowledge that machine-generated summaries could give rise to additional challenges and opportunities. Hence, we include some exploratory questions to get an understanding of the trust users would have in machine-generated summaries and to get ideas for the interpretation of the context factors in exploratory settings.
For the first questions we tell participants to imagine that the summary was made by a computer, but contained all needs identified in the first part of the survey. We then ask them about trust in computer- and human-generated summaries. Next, we ask them to imagine that they could interact with the computer program that made the summary in the form of a digital assistant. We tell them not to feel restricted by the capabilities of today's digital assistants. The verbatim text is given in Appendix C. We ask participants to select the three most and the three least useful features for the digital assistant, similar to ter Hoeve et al. (2020).
For each question we examine the outcomes of all respondents together and of different subgroups (Table 1). For space and clarity reasons, we present the results of all respondents together, unless interesting differences between groups are found. We use the question formulations as used for the Remembered group and abbreviate answer options. Answers to multiple choice and multiple response questions are presented in an aggregated manner and we ensure that none of the open answers can be used to identify individual participants.
4.1 Identifying branches
Of our participants,  $78.0\%$  were led to the Remembered branch and of the remaining  $22.0\%$ ,  $78.2\%$  were led to the Imagined branch. We asked the few remaining participants why they could not think of a case where a pre-made summary could be useful for them. People answered that they would not
1 All respondents together
2 Remembered branch vs Imagined branch
3 Different study fields
4 Different study levels
5 Different levels of how helpful the summary was according to participants, rated on a 5-point Likert scale (note that only the remembered group answered this question)
Table 1: Levels of investigation. We did not find significant differences for each, but add all for completeness.
trust such a summary and that making a summary themselves helped with their study activities.
4.2 Input factors
Figure 4 shows the input factor results. We highlight some here. Textual input is significantly more popular than other input types (Figure 4a), stressing the relevance of automatic text summarization. People described a diverse input for scale and unit (Figure 4b), much more than the classical focus of automatic summarization suggests. Most input had a considerable amount of structure (Figure 4e). Structure is often discarded in automatic summarization, although it can be very informative.
4.3 Purpose factors
Figure 5 shows the purpose factor results. Participants indicated that the summary was helpful or very helpful (Figure 5f), which allows us to draw valid conclusions from the survey. We now highlight some results from the other questions in this category. For the intended audience of the summaries, students selected level (4) and (5) ("a lot (4) or full (5) domain knowledge is expected from the users of the summary") significantly more often than the other options (Figure 5d). Although perhaps unsurprising given our target group, it is an important outcome as this requires a different level of detail than, for example, a brief overview of a news article. People used the summaries for many different use-cases (Figure 5e), whereas current research on automatic summarization mainly focuses on giving an overview of the input. We show the results for the Remembered vs. Imagined splits,
4This is based on people's initial responses and not on the follow up question if they selected another option than 'text'.
Because we do not find significant differences in the overall results when we exclude the few participants who did not find their summary helpful and we do not find many correlations w.r.t. how helpful a summary was and a particular context factor, we include all participants in the analysis, regardless of how helpful they found their summary, for completeness.
(a) Medium: The study material consisted of (MC)
(b) Scale / Unit: What was the length of the study material? (MC)
(c) Genre: What was the genre of the study material? (MC)
(e) Structure: How was the study material structured? (MR)
(d) Subject Type: How would you classify the difficulty level of the study material? (MC)
Figure 4: Results for the input factor questions. Specific input factor in italics. Answer type in brackets: MC = Multiple Choice, MR = Multiple Response. ** indicates significance ( $\chi^2$ ), after Bonferroni correction, with  $p \ll 0.001$ . If two options are flagged with **, these options are not significantly different from each other, yet both have been chosen significantly more often than the other options.
as the Imagined group chose refresh memory and overview more often than the Remembered group (Fisher's exact test,  $p < 0.05$ ). Although not significant after a Bonferroni correction, this can still be insightful for future research directions. Lastly, participants in the Imagined group ticked more boxes than participants in the Remembered group: 3.33 vs. 2.57 per participant on average, stressing the importance of considering many different use-cases for automatically generated summaries.
4.4 Output factors
Figure 6 shows the results for the output factor questions. Textual summaries were significantly more popular than other summary types (Figure 6a), which again stresses the importance of automatic text summarization. Most participants indicated that the summary covered (or should cover) most of the input material (Figure 6c). For the output factor style we find an interesting difference between the Remembered and Imagined group (Figure 6d). Whereas the Remembered group described significantly more often an informative summary, the Imagined group opted significantly more often for a critical or aggregative summary. Most research on automatic summarization focuses on
informative summaries only. For the output factor structure (Figure 6b), people described a substantially richer format of the pre-made summaries than adopted in most research on automatic summarization. Instead of simply a running text, the vast majority of people indicated that the summary contained (or should contain) structural elements such as special formatting, diagrams, headings, etc. Moreover, the Imagined group ticked more answer boxes on average than the Remembered group: 4.17 vs. 3.56 per participant, indicating a desire for structure in the generated summaries, which is supported by the open answer questions.