Venue: NAACL 2022

# What Makes a Good and Useful Summary?

Incorporating Users in Automatic Summarization Research

 Maartje ter Hoeve

University of Amsterdam

m.a.terhoeve@uva.nl &Julia Kiseleva

Microsoft Research

julia.kiseleva@microsoft.com &Maarten de Rijke

University of Amsterdam

m.derijke@uva.nl

###### Abstract

Automatic text summarization has enjoyed great progress over the years and is used in numerous applications, impacting the lives of many. Despite this development, there is little research that meaningfully investigates how the current research focus in automatic summarization aligns with users' needs. To bridge this gap, we propose a survey methodology that can be used to investigate the needs of users of automatically generated summaries. Importantly, these needs are dependent on the target group. Hence, we design our survey in such a way that it can be easily adjusted to investigate different user groups. In this work we focus on university students, who make extensive use of summaries during their studies. We find that the current research directions of the automatic summarization community do not fully align with students' needs. Motivated by our findings, we present ways to mitigate this mismatch in future research on automatic summarization: we propose research directions that impact the design, the development and the evaluation of automatically generated summaries.

## 1 Introduction

The field of automatic text summarization has experienced great progress over the last years, especially since the rise of neural sequence to sequence models (e.g., Cheng and Lapata, 2016; See et al., 2017; Vaswani et al., 2017). The introduction of self-supervised transformer language models like BERT Devlin et al. (2019) has given the field an additional boost (e.g., Liu et al., 2018; Liu and Lapata, 2019; Lewis et al., 2020; Xu et al., 2020).

The--often _implicit_--goal of automatic text summarization is to generate a condensed textual version of the input document(s), whilst preserving the main message. This is reflected in today's most common evaluation metrics for the task; they focus on aspects such as informativeness, fluency, succinctness and factuality (e.g., Lin, 2004; Nenkova and Passonneau, 2004; Paulus et al., 2018; Narayan et al., 2018b; Goodrich et al., 2019; Wang et al., 2020; Xie et al., 2021). The _needs_ of the users of the summaries are often not explicitly addressed, despite their importance in _explicit_ definitions of the goal of automatic summarization Sparck Jones (1998); Mani (2001). Mani defines this goal as: "_to take an information source, extract content from it, and present the most important content to the user in a condensed form and in a manner sensitive to the user's or application's needs._"

Different user groups have different needs. Investigating these needs explicitly is critical, given the impact of adequate information transfer Bennett et al. (2012). We propose a survey methodology to investigate these needs. In designing the survey, we take stock of past work by Sparck Jones (1998) who argues that in order to generate useful summaries, one should take the context of a summary

Figure 1: Example of most current summarization techniques vs. summarization while incorporating the users in the process.



into account--a statement that has been echoed by others (e.g., Mani, 2001; Aries et al., 2019). To do this in a structured manner, Sparck Jones introduces three _context factor_ classes: _input factors_, _purpose factors_ and _output factors_, which respectively describe the input material, the purpose of the summary, and what the summary should look like. We structure our survey and its implications around these factors. In Figure 1 we give an example of incorporating the context factors in the design of automatic summarization methods.

Our proposed survey can be flexibly adjusted to different user groups. Here we turn our focus to university students as a first stakeholder group. University students are a particularly relevant group to focus on first, as they benefit from using pre-made summaries in a range of study activities (Reder and Anderson, 1980), but the desired characteristics of these pre-made summaries have not been extensively investigated. We use the word _pre-made_ to differentiate such summaries from the ones that users write themselves. Automatically generated summaries fall in the pre-made category, and should thus have the characteristics that users wish for pre-made summaries.

Motivated by our findings, we propose important future research directions that directly impact the design, development, and evaluation of automatically generated summaries. We contribute the following:

* We design a survey that can be easily adapted and reused to investigate and understand the needs of the wide variety of users of automatically generated summaries;
* We develop a thorough understanding of how automatic summarization can optimally benefit users in the educational domain, which leads us to unravel important and currently underexposed research directions for automatic summarization;
* We propose a new, feasible and comprehensive evaluation methodology to explicitly evaluate the usefulness of a generated summary for its intended purpose.

## 2 Related work

In Section 1 we introduced the context factors as proposed by Sparck Jones (1998). Each context factor class can be divided into more fine-grained subclasses. To ensure the flow of the paper, we list an overview in Appendix A. Below, we explain and use the context factors and their fine-grained subclasses to structure the related work. As our findings have implications for the evaluation of automatic summarization, we also discuss evaluation methods. Lastly, we discuss the use-cases of automatic summaries in the educational domain.

### Automatic text summarization

**Input factors.** We start with the fine-grained input factor _unit_, which describes how many sources are to be summarized at once, and the factor _scale_, which describes the length of the input data. These factors are related to the difference between single and multi-document summarization (e.g., Chopra et al., 2016; Cheng and Lapata, 2016; Wang et al., 2016; Yasunaga et al., 2017; Nallapati et al., 2017; Narayan et al., 2018; Liu and Lapata, 2019). _Scale_ plays an important role when material shorter than a single document is summarized, such as sentence summarization (e.g., Rush et al., 2015). Regarding the _genre_ of the input material, most current work focuses on the news domain or Wikipedia (e.g., Sandhaus, 2008; Hermann et al., 2015; Koupaee and Wang, 2018; Liu et al., 2018; Narayan et al., 2018). A smaller body of work addresses different input genres, such as scientific articles (e.g., Cohan et al., 2018), forum data (e.g., Volske et al., 2017), opinions (e.g., Amplayo and Lapata, 2020) or dialogues (e.g., Liu et al., 2021). These differences are also closely related to the input factor _subject type_, which describes the difficulty level of the input material. The factor _medium_ refers to the input language. Most automatic summarization work is concerned with English as language input, although there are exceptions, such as Chinese (e.g., Hu et al., 2015) or multilingual input (Ladhak et al., 2020). The last input factor is _structure_. Especially in recent neural approaches, explicit structure of the input text is often ignored. Exceptions include graph-based approaches, where implicit document structure is used to summarize a document (e.g., Tan et al., 2017; Yasunaga et al., 2017), and summarization of tabular data (e.g., Zhang et al., 2020) or screenplays (e.g., Papalampidi et al., 2020).

**Purpose factors.** Although identified as the most important context factor class by Sparck Jones (1998)--and followed by, for example, Mani (2001)--purpose factors do not receive a substantial amount of attention. There are some exceptions, e.g., query-based summarization (e.g., Nema et al., 2017; Litvak and Vanetik, 2017), question-driven 

summarization (e.g., Deng et al., 2020), personalized summarization (e.g., Moro and Bielikova, 2012) and interactive summarization (e.g., Hirsch et al., 2021). They take the _situation_ and the _audience_ into account. The _use_-cases of the generated summaries are also clearer in these approaches.

**Output factors.** We start with the output factors _style_ and _material_. The latter is concerned with the degree of coverage of the summary. Most generated summaries have an _informative_ style and cover most of the input material. There are exceptions, e.g., the XSum dataset Narayan et al. (2018) which constructs summaries of a single sentence and is therefore more _indicative_ in terms of style and inevitably less of the input material is covered. Not many summaries have a _critical_ or _aggregative_ style. Aggregative summaries put different source texts in relation to each other, to give a topic overview. Most popular summarization techniques focus on a _running format_. Work on template-based (e.g., Cao et al., 2018) and faceted (e.g., Meng et al., 2021) summarization follows a more _headed_ (structured) _format_. Falke and Gurevych (2017) build concept maps and Wu et al. (2020) make knowledge graphs. The difference between abstractive and extractive summarization is likely the best known distinction in output type (e.g., Nallapati et al., 2017; See et al., 2017; Narayan et al., 2018; Gehrmann et al., 2018; Liu and Lapata, 2019), although it is not entirely clear which output factor best describes the difference.

In Section 5 we use the context factors to identify future research directions, based on the difference between our findings and the related work.

### Evaluation

Evaluation methods for automatic summarization can be grouped in _intrinsic_ vs. _extrinsic_ methods Mani (2001b). Intrinsic methods evaluate the model itself, e.g., on informativeness or fluency Paulus et al. (2018); Liu and Lapata (2019). Extrinsic methods target how a summary performs when used for a task Dorr et al. (2005); Wang et al. (2020). Extrinsic methods are resource intensive, explaining the popularity of intrinsic methods.

Evaluation methods can also be grouped in _automatic_ vs. _human_ evaluation methods. Different automatic metrics have been proposed, such as Rouge Lin (2004) and BERTScore Zhang et al. (2020) which respectively evaluate lexical and semantic similarity. Other methods use an automatic question-answering evaluation methodology Wang et al. (2020); Durmus et al. (2020). Most human evaluation approaches evaluate intrinsic factors such as informativeness, readability and conciseness DUC (2003); Nallapati et al. (2017); Paulus et al. (2018); Liu and Lapata (2019)--factors that are difficult to evaluate automatically. There are also some extrinsic human evaluation methods, where judges are asked to perform a certain task based on the summary (e.g., Narayan et al., 2018). So far, _usefulness_1 has not been evaluated in a feasible and comprehensive manner, whereas it is an important metric to evaluate whether summaries fulfil users' needs. Therefore, we bridge the gap by introducing a feasible and comprehensive evaluation methodology to evaluate usefulness.

Footnote 1: We follow the definition of the English Oxford Learner’s Dictionary (www.oxfordlearnersdictionaries.com/definition/english/) for usefulness: “_the fact of being useful or possible to use”_, where _useful_ is defined as “_that can help you to do or achieve what you want”.

### Automatic summarization for education

Summaries play a prominent role in education. Reder and Anderson (1980) find that students who use a pre-made summary score better on a range of study activities than students who do not use such a summary. As the quality of automatically generated summaries increases (e.g., Lewis et al., 2020; Xu et al., 2020), so does the potential to use them in the educational domain, especially given the increasing importance of digital tools and devices for education Luckin et al. (2012); Hashim (2018). With these developments in mind, it is critical that educators are aware of the pedagogical implications; they need to understand how to best make use of all new possibilities Hashim (2018); Amhag et al. (2019). The outcomes of our survey result in concrete suggestions for developing methods for automatic summarization in the educational domain, whilst taking students' needs into account.

## 3 Survey Procedure and Participants

Here we detail our survey procedure. For concreteness, we present the details with our intended target group in mind. The context factors form the backbone of our survey and the setup can be easily adjusted to investigate the needs of different target groups. For example, we ask participants about a pre-made summary for a recent study activity, but it is straightforward to adapt this to a different use-case that is more suitable for other user groups 



### Participants

We recruited participants among students at universities across the Netherlands by contacting ongoing courses and student associations, and by advertisements on internal student websites. As incentive, we offered a ten euro shopping voucher to ten randomly selected participants.

A total of \(118\) participants started the survey and \(82\) completed the full survey, resulting in a \(69.5\%\) completion rate. We only include participants who completed the study in our analysis. Participants spent \(10\) minutes on average on the survey. In the final part of our survey we ask participants to indicate their current level of education and main field of study. The details are given in Figure 2.

### Survey procedure

Figure 3 shows a brief overview of our survey procedure. A detailed account is given in Appendix B. We arrived at the final survey version after a number of pilot runs where we ensured participants understood their task and all questions. We ran the survey with SurveyMonkey (surveymonkey.com). A verbatim copy is included in Appendix C and released under CC BY license.2

Footnote 2: [https://github.com/maartjeth/survey_useful_summarization](https://github.com/maartjeth/survey_useful_summarization)

**Introduction.** The survey starts with an introduction where we explain what to expect, how we process the data and that participation is voluntary. After participants agree with this, an explanation of the term _pre-made summary_ follows. As we do not want to bias participants by stating that the summary was automatically generated, we explain that the summary can be made by anyone, e.g., a teacher, a good performing fellow student, the authors of the original material, or a computer. Recall that an automatically generated summary is a pre-made summary. Hence, our survey identifies the characteristics an automatically generated summary should have. We also give examples of types of pre-made summaries; based on the pilot experiments we noticed that people missed this information. We explicitly state that these are just examples and that participants can come up with any example of a helpful pre-made summary.

**Context factors.** In the main part of our survey we focus on the context factors. First, we ask participants whether they have made use of a pre-made summary in one of their recent study activities. If so, we ask them to choose the study activity where a summary was most useful. We call this group the _Remembered group_, as they describe an existing summary from memory. If people indicate that they have not used a pre-made summary in one of their recent study activities, we ask them whether they can imagine a situation where a pre-made summary would have been helpful. If not, we ask them why not and lead them to the final background questions and closing page. If yes, we ask them to keep this imaginary situation in mind for the rest of the survey. We call this group the _Imagined group_.

Now we ask the Remembbered and Imagined groups about the input, purpose and output factors of the summary they have in mind. We ask questions for each of the context factor subclasses that we discussed in Section 2. At this point, the two groups are in different branches of the survey. The difference is mainly linguistically motivated: in the Imagined group we use verbs of probability instead of asking to describe an existing situation. Some questions can only be asked in the Remembered group, e.g., how helpful the summary was.

In the first context factor question we ask what the study material consisted of. We give a number of options, as well as an 'other' checkbox. To avoid position bias, all answer options for multiple choice and multiple response questions in the survey are randomized, with the 'other' checkbox always as the last option. If participants do not choose the 'mainly text' option, we tell them that we focus on textual input in the current study3 and ask whether they can think of a situation where the input did consist of text. If not, we lead them to the background questions and closing page. If yes, they proceed to the questions that give us a full overview of the input, purpose and output factors of the situation participants have in mind. Finally, we ask the Remembered group to suggest how their described summary could be turned into their ideal

Figure 2: Participant details.



summary. We then ask both groups for any final remarks about the summary or input material.

**Trustworthiness and future features questions.** So far we have included the possibility that the summary was machine-generated, but also explicitly included other options so as not to bias participants. At this point we acknowledge that machine-generated summaries could give rise to additional challenges and opportunities. Hence, we include some exploratory questions to get an understanding of the trust users would have in machine-generated summaries and to get ideas for the interpretation of the context factors in exploratory settings.

For the first questions we tell participants to imagine that the summary was made by a computer, but contained all needs identified in the first part of the survey. We then ask them about trust in computer- and human-generated summaries. Next, we ask them to imagine that they could interact with the computer program that made the summary in the form of a digital assistant. We tell them not to feel restricted by the capabilities of today's digital assistants. The verbatim text is given in Appendix C. We ask participants to select the three most and the three least useful features for the digital assistant, similar to ter Hoeve et al. (2020).

## 4 Results

For each question we examine the outcomes of all respondents together and of different subgroups (Table 1). For space and clarity reasons, we present the results of all respondents together, unless interesting differences between groups are found. We use the question formulations as used for the Remembered group and abbreviate answer options. Answers to multiple choice and multiple response questions are presented in an aggregated manner and we ensure that none of the open answers can be used to identify individual participants.

### Identifying branches

Of our participants, \(78.0\%\) were led to the Remembered branch and of the remaining \(22.0\%\), \(78.2\%\) were led to the Imagined branch. We asked the few remaining participants why they could not think of a case where a pre-made summary could be useful for them. People answered that they would not trust such a summary and that making a summary themselves helped with their study activities.

### Input factors

Figure 4 shows the input factor results. We highlight some here. Textual input is significantly more popular than other input types (Figure 3(a)),4 stressing the relevance of automatic text summarization. People described a diverse input for _scale_ and _unit_ (Figure 3(b)), much more than the classical focus of automatic summarization suggests. Most input had a considerable amount of structure (Figure 3(c)). Structure is often discarded in automatic summarization, although it can be very informative.

Footnote 4: This is based on people’s initial responses and not on the follow up question if they selected another option than ‘text’.

### Purpose factors

Figure 5 shows the purpose factor results. Participants indicated that the summary was _helpful_ or _very helpful_ (Figure 4(f)), which allows us to draw valid conclusions from the survey.5 We now highlight some results from the other questions in this category. For the intended _audience_ of the summaries, students selected level (4) and (5) (_"a lot (4) or full (5) domain knowledge is expected from the users of the summary"_) significantly more often than the other options (Figure 4(d)). Although perhaps unsurprising given our target group, it is an important outcome as this requires a different level of detail than, for example, a brief overview of a news article. People used the summaries for many different use-cases (Figure 4(e)), whereas current research on automatic summarization mainly focuses on giving an overview of the input. We show the results for the Remembered vs. Imagined splits,

\begin{table}
\begin{tabular}{l l} \hline \hline
1 & All respondents together \\
2 & Remembered branch vs Imagined branch \\
3 & Different study fields \\
4 & Different study levels \\
5 & Different levels of how helpful the summary \\  & was according to participants, rated on a \\  & \(5\)-point Likert scale (note that only the _remembered_ group answered this question) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Levels of investigation. We did not find significant differences for each, but add all for completeness.

Figure 3: Overview of the survey procedure.

 

as the Imagined group chose _refresh memory_ and _overview_ more often than the Remembered group (Fisher's exact test, \(p<0.05\)). Although not significant after a Bonferroni correction, this can still be insightful for future research directions. Lastly, participants in the Imagined group ticked more boxes than participants in the Remembered group: \(3.33\) vs. \(2.57\) per participant on average, stressing the importance of considering many different use-cases for automatically generated summaries.

### Output factors

Figure 6 shows the results for the output factor questions. Textual summaries were significantly more popular than other summary types (Figure 4(a)), which again stresses the importance of automatic text summarization. Most participants indicated that the summary covered (or should cover) most of the input _material_ (Figure 4(c)). For the output factor _style_ we find an interesting difference between the Remembered and Imagined group (Figure 4(d)). Whereas the Remembered group described significantly more often an _informative_ summary, the Imagined group opted significantly more often for a _critical_ or _aggregative_ summary. Most research on automatic summarization focusses on informative summaries only. For the output factor _structure_ (Figure 4(b)), people described a substantially richer format of the pre-made summaries than adopted in most research on automatic summarization. Instead of simply a running text, the vast majority of people indicated that the summary contained (or should contain) structural elements such as special formatting, diagrams, headings, etc. Moreover, the Imagined group ticked more answer boxes on average than the Remembered group: \(4.17\) vs. \(3.56\) per participant, indicating a desire for structure in the generated summaries, which is supported by the open answer questions.

**Open answer questions.** We asked participants in the Remembered group how the summary could be transformed into their ideal summary and \(86.9\%\) of these participants made suggestions. Many of those include adding additional structural elements to the summary, like figures, tables or structure in the summary text itself. For example, one of the participants wrote: _"An ideal summary is good enough to fully replace the original (often longer) texts contained in articles that need to be read for exams. The main purpose behind this is speed of learning from my experience. More tables, graphs and visual representations of the study material and

Figure 4: Results for the _input factor_ questions. Specific input factor in italics. Answer type in brackets: MC = Multiple Choice, MR = Multiple Response. ** indicates significance (\(\chi^{2}\)), after Bonferroni correction, with \(p\ll 0.001\). If two options are flagged with **, these options are not significantly different from each other, yet both have been chosen significantly more often than the other options.

 

key concepts / links would improve the summary, as I would faster comprehend the study material."_ Another participant wrote: _"- colors and a key for color-coding - different sections, such as definitions on the left maybe and then the rest of the page reflects the structure of the course material with notes on the readings that have many headings and subheadings."_

Another theme is the desire to have more examples in the summary. One participant wrote: _"More examples i think. For me personally i need examples to understand the material. Now i needed to imagine them myself"_.

Some participants wrote that they would like a more personalized summary, for example: _"I'd highlight some things I find difficult. So I'd personalise the summary more."_ Another participant wrote: _"Make it more personalized may be. These notes were by another student. I might have focussed more on some parts and less on others."_

### Trustworthiness and future features

Of all participants, \(48.0\%\) indicated that it would not make a difference to them whether a summary is machine- or human-generated, as long as the quality is as good as a human-generated one. This last point is reflected in which types of summaries participants would trust more. People opted significantly more often for a human-generated one. For the future feature questions, _adding more details to the summary_ and _answering questions based on the content of the summary_ were very popular. We give a full account in Appendix D.

## 5 Implications and Perspectives

### Future research directions

Our findings have important implications for the design and development of future automatic summarization methods. We present these in Table 2, per context factor. Summarizing, the research developments as summarized in Section 2 are encouraging, yet given that automatic summarization methods increasingly mediate people's lives, we argue that more attention should be devoted to its stakeholders, i.e., to the purpose factors. Here we have shown that students, an important stakeholder group, have different expectations of pre-made

Figure 5: Results for the _purpose factor_ questions. Specific purpose factor in italics. Answer type in brackets: MC = Multiple Choice, MR = Multiple Response, LS = Likert Scale. ** indicates significance (\(\chi^{2}\)), after Bonferroni correction, with \(p\ll 0.001\), * with \(p<0.05\). ! indicates noteworthy results where significance was lost after correction for the number of tests. If two options are flagged, these options are not significantly different from each other, yet both were chosen significantly more often than the other options.

 

summaries than what most automatic summarization methods offer. These differences include the type of input material that is to be summarized, but also how these summaries are presented. Presumably, this also holds for other stakeholder groups and thus we hope to see our survey used for different target groups in the future.

* the purpose and output factors need more emphasis. Our findings also impact the evaluation of summarization methods. We discuss this next.

### Usefulness as evaluation methodology

Following Sparck Jones (1998) and Mani (2001a), we argue that a good choice of context factors is crucial in producing useful summaries for users. It is important to explicitly evaluate this. The few existing methods to evaluate usefulness are very resource demanding (e.g., Riccardi et al., 2015) or not comprehensive enough (e.g., DUC, 2003; Dorr et al., 2005). Thus, we propose a feasible and comprehensive method to evaluate usefulness.

For the evaluation methodology, we again use the context factors. Before the design and development of the summarization method the intended purpose factors need to be defined. Especially the

\begin{table}
\begin{tabular}{p{47.7pt} p{260.2pt}} \hline \hline
**Input Factors** & Stronger focus on developing methods that can: \\ • handle a wide variety and a mixture of **different types** of input documents at once; \\ • understand the **relationships** between different input documents; \\ • use the **structure** of the input document(s). \\ \hline
**Purpose Factors** & \\ \hline • Explicitly define a **standpoint** on the purpose factors in each research project; \\ • Include a comprehensive **evaluation** methodology to evaluate usefulness. We propose this in Section 5.2. \\ \hline
**Output Factors** & Stronger focus on developing methods that can: \\ • output different summary **styles**, e.g., informative, aggregative or critical. Especially the last two require a **deeper understanding** of the input material than current models have; \\ • explicitly model and understand **relationships** between different elements in the summary and potentially relate this back to the input document(s). \\ \hline \hline \end{tabular}
\end{table}
Table 2: Implications for future research directions.

Figure 6: Results for the _output factor_ questions. Specific output factor in italics. Answer type in brackets: MC = Multiple Choice, MR = Multiple Response, LS = Likert Scale. ** indicates significance (\(\chi^{2}\) or Fisher’s exact test), after Bonferroni correction, with \(p\ll 0.001\), * with \(p<0.05\).

 

fine-grained factor _use_ is important here. Next, the output factors need to be evaluated on the use factors. For this, we take inspiration from research on simulated work tasks Borlund (2003). Evaluators should be given a specific task to imagine, e.g., _writing a news article_, or _studying for an exam_. This task should be relatable to the evaluators, so that reliable answers can be obtained Borlund (2016). With this task in mind, evaluators should be asked to judge two summaries in a pairwise manner on their usefulness, in the following format: _The [output factor] of which of these two summaries is most useful to you to [use factor]?_ For example: _The style of which of these two summaries is most useful to you to substitute a chapter that you need to learn for your exam preparation?_ It is critical to ensure that judges understand the meaning of each of the evaluation criteria - _style_ and _substitute_ in the example. We provide example questions for each of the use and output factors in Appendix E.

## 6 Conclusion

In this paper we focused on users of automatically generated summaries and argued for a stronger emphasis on their needs in the design, development and evaluation of automatic summarization methods. We led by example and proposed a survey methodology to identify these needs. Our survey is deeply grounded in past work by Sparck Jones (1998) on context factors for automatic summarization and can be re-used to investigate a wide variety of users. In this work we use our survey to investigate the needs of university students, an important target group of automatically generated summaries. We found that the needs identified by our participants are not fully supported by current automatic summarization methods and we proposed future research directions to accommodate these needs. Finally, we proposed an evaluation methodology to evaluate the usefulness of automatically generated summaries.

## 7 Ethical Impact

With this work we hope to take a step in the right direction to make research into automatic summarization more inclusive, by explicitly taking the needs of users of these summaries into account. As stressed throughout the paper, these needs are different per user group and therefore it is critical that a wide variety of user groups will be investigated. There might also be within group differences. For example, in this work we have focussed on students from universities in one country, but students attending universities in other geographical locations and with different cultures might express different needs. It is important to take these considerations into account, to limit the risk of overfitting on a particular user group and potentially harming other user groups.

## Acknowledgements

We thank Jacobijn Sandberg and Ana Lucic for helpful comments and feedback. This research was supported by the Nationale Politie. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

## References

* Amhag et al. (2019) Lisbeth Amhag, Lisa Hellstrom, and Martin Stigmar. 2019. Teacher educators' use of digital tools and needs for digital competence in higher education. _Journal of Digital Learning in Teacher Education_, 35(4):203-220.
* Amplayo and Lapata (2020) Reinald Kim Amplayo and Mirella Lapata. 2020. Unsupervised opinion summarization with noising and denoising. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1934-1945, Online. Association for Computational Linguistics.
* Aries et al. (2019) Abdelkrime Aries, Djamel Eddine Zegour, and Walid-Khaled Hidouci. 2019. Automatic text summarization: What has been done and what has to be done. _CoRR_, abs/1904.00688.
* Bennett et al. (2012) Paul N. Bennett, Ryen W. White, Wei Chu, Susan T. Dumais, Peter Bailey, Fedor Borisuk, and Xiaoyuan Cui. 2012. Modeling the impact of short-and long-term behavior on search personalization. In _The 35th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR '12, Portland, OR, USA, August 12-16, 2012_, pages 185-194. ACM.
* Borlund (2003) Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. _Information Research_, 8(3).
* Borlund (2016) Pia Borlund. 2016. A study of the use of simulated work task situations in interactive information retrieval evaluations: A meta-evaluation. _J. Documentation_, 72(3):394-413.
* Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 

pages 152-161, Melbourne, Australia. Association for Computational Linguistics.
* Cheng and Lapata (2016) Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 484-494, Berlin, Germany. Association for Computational Linguistics.
* Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 93-98, San Diego, California. Association for Computational Linguistics.
* Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
* Deng et al. (2020) Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Multi-hop inference for question-driven summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6734-6744, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dorr et al. (2005) Bonnie Dorr, Christof Monz, Stacy President, Richard Schwartz, and David Zajic. 2005. A methodology for extrinsic evaluation of text summarization: Does ROUGE correlate? In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_, pages 1-8, Ann Arbor, Michigan. Association for Computational Linguistics.
* Duc (2003) DUC. 2003. Duc 2003: Documents, tasks, and measures. [https://duc.nist.gov/duc2003/tasks.html](https://duc.nist.gov/duc2003/tasks.html).
* Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5055-5070, Online. Association for Computational Linguistics.
* Falke and Gurevych (2017) Tobias Falke and Iryna Gurevych. 2017. Bringing structure into summaries: Crowdsourcing a benchmark corpus of concept maps. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_, pages 2951-2961. Association for Computational Linguistics.
* Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
* Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, pages 166-175. ACM.
* Hashim (2018) Harwati Hashim. 2018. Application of technology in the digital era education. _International Journal of Research in Counseling and Education_, 2(1):1-5.
* Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, Montreal, Quebec, Canada_, pages 1693-1701.
* Hirsch et al. (2021) Eran Hirsch, Alon Eirew, Ori Shapira, Avi Caciularu, Arie Cattan, Ori Ernst, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, and Ido Dagan. 2021. iFacetSum: Coreference-based interactive faceted summarization for multi-document exploration. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 283-297, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Hu et al. (2015) Baotiian Hu, Qingcai Chen, and Fangze Zhu. 2015. LCSTS: A large scale Chinese short text summarization dataset. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1967-1972, Lisbon, Portugal. Association for Computational Linguistics.
* Koupaee and Wang (2018) Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. _CoRR_, abs/1810.09305.
* Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summar 

rization. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4034-4048, Online. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemeyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7817-7880, Online. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Litvak and Vanetik (2017) Marina Litvak and Natalia Vanetik. 2017. Query-based summarization using MDL principle. In _Proceedings of the Workshop on Summarization and Summary Evaluation Across Source Types and Genres, Multilma@EACL 2017, Valencia, Spain, April 3, 2017_, pages 22-31. Association for Computational Linguistics.
* Liu et al. (2021) Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojie Wang. 2021. Topic-aware contrastive learning for abstractive dialogue summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 1229-1243, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
* Luckin et al. (2012) Rosemary Luckin, Brett Bligh, Andrew Manches, Shaaron Ainsworth, Charles Crook, and Richard Noss. 2012. _Decoding learning: The proof; promise and potential of digital education_. Nesta.
* Mani (2001a) Indrejete Mani. 2001a. _Automatic summarization_, volume 3. John Benjamins Publishing.
* Mani (2001b) Indrejete Mani. 2001b. Summarization evaluation: An overview. In _Proceedings of the Third Second Workshop Meeting on Evaluation of Chinese & Japanese Text Retrieval and Text Summarization, NTCIR-2, Tokyo, Japan, March 7-9, 2001_. National Institute of Informatics (NII).
* Meng et al. (2021) Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021. Bringing structure into summaries: a faceted summarization dataset for long scientific documents. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 1080-1089, Online. Association for Computational Linguistics.
* Moro and Bielikova (2012) Robert Moro and Maria Bielikova. 2012. Personalized text summarization based on important terms identification. In _23rd International Workshop on Database and Expert Systems Applications, DEXA 2012, Vienna, Austria, September 3-7, 2012_, pages 131-135. IEEE Computer Society.
* Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_, pages 3075-3081. AAAI Press.
* November 4, 2018_, pages 1797-1807. Association for Computational Linguistics.
* Narayan et al. (2018b) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1747-1759, New Orleans, Louisiana. Association for Computational Linguistics.
* Nema et al. (2017) Preksha Nema, Mitesh M. Khabra, Anirban Laha, and Balaram Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1063-1072, Vancouver, Canada. Association for Computational Linguistics.
* Nemka and Passonneau (2004) Anti Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004_, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
* Papalampidi et al. (2020) Pinelopi Papalampidi, Frank Keller, Lea Frermann, and Mirella Lapata. 2020. Screenplay summarization using latent narrative structure. In _Proceedings of the 

58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, pp. 1920-1933. External Links: Link Cited by: SS4.
* May 3, 2018, Conference Track Proceedings, OpenReview.net, Cited by: SS4.
* L. M. Reder and J. R. Anderson (1980)A comparison of texts and their summaries: Memorial consequences. Journal of Verbal Learning and Verbal Behavior19 (2), pp. 121-134. External Links: ISSN 078-1221, Document, Link Cited by: SS4.
* First International Workshop, FETLT 2015, Seville, Spain, November 19-20, 2015, Revised Selected Papers, Vol. 9577 of Lecture Notes in Computer Science, pp. 10-33. External Links: ISBN 978-3-319-5861-4, Link, Document Cited by: SS4.
* A. M. Rush, S. Chopra, and J. Weston (2015)A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 379-389. External Links: Link Cited by: SS4.
* E. Sandhaus (2008)The New York Times annotated corpus. Linguistic Data Consortium, Philadelphia6 (12), pp. e26752. External Links: Link Cited by: SS4.
* A. See, P. J. Liu, and C. D. Manning (2017)Get to the point: summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1073-1083. External Links: Link Cited by: SS4.
* K. Sparck Jones (1998)Automatic summarizing: factors and directions. In Advances in automatic text summarization, pp. 1-12. Note: MIT press Cambridge, Mass, USA External Links: ISBN 978-3-319-5861-4, Link, Document Cited by: SS4.
* J. Tan, X. Wan, and J. Xiao (2017)Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, pp. 1171-1181. External Links: Link Cited by: SS4.
* M. ter Hoeve, R. Sim, E. Nouri, A. Fourney, M. de Rujke, and R. W. White (2020)Conversations with documents: an exploration of document-centered assistance. In CHIIR '20: Conference on Human Information Interaction and Retrieval, Vancouver, BC, Canada, March 14-18, 2020, pp. 43-52. External Links: Link Cited by: SS4.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and H. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008. External Links: Link Cited by: SS4.
* M. Volske, M. Potthast, S. Syed, and B. Stein (2017)TL:DR: mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, Copenhagen, Denmark, pp. 59-63. External Links: Link Cited by: SS4.
* A. Wang, K. Cho, and M. Lewis (2020)Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, Online. External Links: Link Cited by: SS4.
* L. Wang, H. Baghavan, V. Castelli, R. Florian, and C. Cardie (2016)A sentence compression based framework to query-focused multidocument summarization. CoRRabs/1606.07548. External Links: Link Cited by: SS4.
* Z. Wu, R. Koncel-Kedziorski, M. Ostendorf, and Hannaneh Hajishirzi (2020)Extracting summary knowledge graphs from long documents. CoRRabs/2009.09162. External Links: Link Cited by: SS4.
* Y. Xie, F. Sun, Y. Deng, Y. Li, and B. Ding (2021)Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pp. 100-110. External Links: Link Cited by: SS4.
* J. Xu, Z. Gan, Yu. Cheng, and J. Liu (2020)Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, Online. External Links: Link Cited by: SS4.
* M. Yasunaga, R. Zhang, K. Meelu, A. Pareek, K. Srinivasan, and D. Radev (2017)Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, pp. 452-462. External Links: Link Cited by: SS4.
* S. Zhang, Z. Dai, K. Balog, and J. Callan (2020)Summarizing and exploring tabular data in conversational search. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pp. 1537-1540. External Links: Link, Document Cited by: SS4.

 

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.

 



## Appendix A Overview context factors

\begin{tabular}{|p{142.3pt}|p{170.7pt}|p{170.7pt}|} \hline
**Input Factors** & **Purpose Factors** & **Output Factors** \\ _Form_ & _Situation_ & _Material_ \\ _Structure:_ How is the input text structured? E.g., subheadings, rhetorical patterns, etc. & _Tied:_ It is known who will use the summary, for what purpose and when. \\ _Scale:_ How large is the input data that we are summarizing? E.g., a book, a chapter, a single article, etc. & _Floating:_ It is not (exactly) known who will use the summary, for what purpose or when. \\ _Medium:_ What is the input language type? E.g., full text, telegraphese style, etc. This also refers to which natural language is used. _Genre:_ What type of literacy does the input text have? E.g., description, narrative, etc. & _Targetted:_ A lot of domain knowledge is expected from the readers of the summary. _Autagetted:_ No domain knowledge is expected from the readers of the summary. _Autagetted:_ The summary is structured following a certain standardised format with headings and other explicit structure. \\ _Ordinary:_ Everyone could understand this input type. _Specialized:_ You need to speak the jargon to understand this input type. _Restricted_: The input type text to preview a text. & _Use_ & _Style_ \\ _Specialized:_ You need to speak the jargon to understand this input type. _Restricted_: The input type text is only understandable for people familiar with a certain area, for example because it contains local names. & _Retrieving:_ Use the summary to retrieve source text. & _Interformative:_ The summary considers the raw information that is in the source text. \\ _Unit_ & _Substitutes:_ Use the summary to substitute the source text. & _Artifical:_ The summary gives a critical review of the merits of the source text. \\ _Single:_ Only one input source is given. & _Refreshing:_ Use the summary to refresh ones memory of the source text. & _Aggregative:_ Different source texts are put in relation to one another to give an overview of a certain topic. \\ _Multi:_ Multiple input sources are given. & _Prompts:_ Use the summary as action prompt to read the source text. & \\ \end{tabular} Table 3: Overview of different context factors classes defined by Sparck Jones (1998), with descriptions of the factors within these classes.





## Appendix B Survey overview

## Appendix C Verbatim survey overview

Table 4: A complete overview of the survey. This table includes the explanation that participants received, as well as all the questions and the answer options. If a question was the start of a branch, the direction of the branch has been written behind the answer options in italic. (This was never shown to the participants.) Note that the survey was performed in SurveyMonkey.6 The survey had a lay-out as provided by SurveyMonkey, i.e., it consisted of different pages and colors were used to highlight certain important parts in texts.

Footnote 6: [http://surveymonkey.com](http://surveymonkey.com)

Figure 7: Overview survey design.





**Question Nr.** & **Question and Answer Options**

Q1 & **Introduction and Instructions**

Thank you for taking the time to fill out this survey! Before you start, please take the time to read these instructions carefully. If you still have any questions after reading the instructions, please send them to m.a.terhoeve@uva.nl.

We will give away 10 bol.com vouchers of 10 euros each among the participants. If you would like to take part in the raffle, you can leave your email address at the end of this survey.

**Goal of the study**

The goal of this survey is to get insight in how summaries help or can help you when studying.

**What the survey will look like**

In what follows you will get questions that aim to develop an understanding for:

* For which types of study material it is useful to have summaries
* How these summaries can help you with your task
* What these summaries should look like

We expect this survey to take approximately 10 minutes of your time.

Use the next button to go to the next page once you have filled out all the questions on the page. Use the prev button to go back one page.

**About your privacy**

We value your privacy and will process your answers anonymously. The answers of all participants in this survey will be used to gain insight in how pre-made summaries can be helpful for different types of studying activities. The answers will be presented in a research paper about this topic. This will be done either in an aggregated manner, or by citing verbatim examples of the answers. Again, this will all be done anonymously.

**I agree that I have read and understood the instructions. I also understand that my participation in this survey is voluntarily.**

\(\square\)   I agree



**Question Nr.** & **Question and Answer Options**

Q2 & **Important! Some background knowledge you need to know**

Throughout this survey we make use of the term pre-made summary. It is very important that you understand what this means. On this page we explain this term, so please make sure to read this carefully.

**Definition pre-made summary**

One type of summary is one that you make yourself. Another type of summary is one that has been made for you. In this survey, we focus on this latter type and we call them pre-made summaries.

**Who makes these pre-made summaries?**

These pre-made summaries can be made by a person, for example your teacher, your friend, a fellow student or someone at some official organisation, etc. The pre-made summaries can also be made by a computer.

**What kinds of summaries are we talking about?**

There are no restrictions on what these pre-made summaries can look like. On the contrary, that is one of the things we aim to find out with this survey! But, to give some examples, you could think of a written overview of a text book, highlights in text to draw your attention to important bits, blog posts, etc. These are really just examples and don't let them limit your creativity! You can come up with any example of a pre-made summary that is helpful for you.

**Yes, I understand what a pre-made summary is!**

Yes

Please think back to your recent study activities. Examples of study activities can be: studying for an exam, writing a paper, doing homework exercises, etc. Note that these are just examples, any other study activity is fine too.

**Did you use a pre-made summary in any of these study activities?**

Yes _- participants are led to Q6_

No _- participants are led to Q4_

Q4 & **Can you think of one of your recent study activities where a pre-made summary would have been useful for you?**

Yes _- participants are led to Q25_

No _- participants are led to Q5_



**Question Nr.** & **Question and Answer Options**

Q5 & **Why do you think a pre-made summary would not have helped you with any of your recent study activities?**

Open response - _participants are led to Q48_

**Start branch of participants who described an existing summary**

If you have multiple study activities where you used a pre-made summary, please take the one where you found the pre-made summary most useful.

\begin{tabular}{l l} Q6 & **The original study material consisted of** \\  & Mainly text - _participants are led to Q8_ \\  & Mainly figures - _participants are led to Q7_ \\  & Mainly video - _participants are led to Q7_ \\  & Mainly audio - _participants are led to Q7_ \\  & A combination of some or all of the above - _participants are led to Q7_ \\  & I do not know, because I have not seen the study material - _participants are led to Q7_ \\  & Other (please specify) - _participants are led to Q7_ \\ Q7 & **For now we narrow down our survey to study material that is mostly textual. Do you recall any other recent study activity where you made use of a pre-made summary and where the original study material mainly consisted of text?** \\  & Yes - _participants are led to Q8_ \\  & No - _participants are led to Q48_ \\ Q8 & **What was the goal of this study activity?** \\  & Studying for an exam \\  & Writing a paper / essay / report / etc. \\  & Doing homework exercises \\  & Other (please specify) \\ Q9 & **Who made this pre-made summary?** \\  & A teacher or teaching assistant \\  & A fellow student \\  & An official organisation \\  & The authors of the original study material \\  & A computer program \\  & I am not sure, I found it online \\  & Other (please specify) \\ \end{tabular}





**Question Nr.** & **Question and Answer Options**

\begin{tabular}{l l}  & Now some questions will follow about what the study material that was summarized looked like. \\ Q10 & **What was the length of the study material?** \\  & A single article \\  & Multiple articles \\  & A single book chapter \\  & Multiple book chapters from the same book \\  & Multiple book chapters from various books \\  & A combination of the above \\  & I do not know because I have not seen the study material, only the summary \\  & Other (please specify) \\ Q11 & **How was the study material structured? (Multiple answers possible)** \\  & There was no particular structure - e.g. just one large text \\  & The text contained a title or titles \\  & The text contained subheadings \\  & The text consisted of different chapters \\  & The text consisted of different sections and / or paragraphs \\  & I do not know because I have not seen the study material, only the summary \\  & Other (please specify) \\ Q12 & **What was the genre of the study material?** \\  & Mainly educational (such as a text book (chapter)) \\  & Mainly scientific (such as an academic article, publication, report, etc) \\  & Mainly nonfiction writing (such as (auto)biographies, history books, etc) \\  & Mainly fiction writing (such as novels, short fictional stories, etc) \\  & Other (please specify) \\ Q13 & **How would you classify the difficulty level of the study material?** \\  & Ordinary: most people would be able to understand it \\  & Specialized: you need to know the jargon of the field to be able to understand it \\  & Geographically based: you can only understand it if you are familiar with a certain area, for example because it contains local names \\  & Now we will ask some questions about the purpose of the pre-made summary that you used. \\ Q14 & **The summary was made specifically to help me (and potentially fellow students) with my study activity.** \\  & Strongly Disagree & Neither Agree & Strongly I don't \\ disagree & agree nor & agree know \\  & disagree & \\  & & \\  & & \\  & & \\ \end{tabular}



* [16] M. C. Bender, A. K. Sridharan, and J. R. Friedman, "A new approach to the estimation of the average number of independent variables in a random variable," _Journal of Statistical Planning Research_**43**(2):75-90, 2008.
* [17] D. G. Kendall, "The maximum likelihood method for multivariate data sets," _Annals of Statistics_**11**(1):109-124, 1994.

[MISSING_PAGE_POST]

 



**Question Nr.** & **Question and Answer Options**

Q18 & **How was the summary structured? (Multiple answers possible)**

\(\square\) The summary was a running text, without particular structure

\(\square\) The summary consisted of highlights in the original study material, without

particular structure

\(\square\) The summary itself contained special formatting, such as bold or cursive text,

highlights, etc

\(\square\) The summary contained diagrams

\(\square\) The summary contained tables

\(\square\) The summary contained graphs

\(\square\) The summary contained figures

\(\square\) The summary contained headings

\(\square\) The summary consisted of different sections / paragraphs

Other (please specify)

Q19 & **How much of the study material was covered by the summary?**

None of the

study

of the study

state

states
material was

material was

material was

covered

covered

covered

(1)

(2)

(3)

(4)

(5)

\(\square\)

\(\square\)

\(\square\)

Q20 & **What was the style of this summary?**

\(\square\) Informative: the summary simply conveyed the information that was in the original

study material

Indicative: the summary gave an idea of the topic of the study material, but not

more

Critical: the summary gave a critical review of the study material

Aggregative: the summary put different source texts in relation to one another and

by doing this gave an overview of a certain topic

Other (please specify)



**Question Nr.** & **Question and Answer Options**

Q21 & **Overall, how helpful was the pre-made summary for you? Your score can range from (1) Not helpful at all, to (5) Very helpful.** &

* [15] M. Cacciari, G. Ridolfi and A. Salgado, _A new approach to the calculation of the gluon-gluon cross section in hadronic collisions at HERA_, JHEP **04** (2008) 063, doi:10.1007/JHEP04(2008)063, arXiv:0902.2881v1 [hep-ph].
* [16] S. Alioli et al., _The PDF4LHC Working Group Interim Recommendations on the LHC Run II_, Eur. Phys. J. C **76** (2016) 155, doi:10.1140/epjc/s10052-016-3988-x, arXiv:1505.0010v1 [hep-ex].

[MISSING_PAGE_POST]

 



**Question Nr.** & **Question and Answer Options**

Now we will ask some questions about the purpose of the pre-made summary that would have been helpful. Q32 & **For what type of people should the summary ideally be intended? Your score can range from (1) Untargetted, to (5) Targetted.**

Untargetted:

No domain

knowledge is

expected

from the

users of the

summmary.

(1) (2) (3) (4) (5)

\(\Box\)\(\Box\)\(\Box\)\(\Box\)\(\Box\)\(\Box\)

Q33 & **How would this summary help you with your task? (Multiple answers possible)**

The summary would help to retrieve parts of the original study material

I would use the summary to preview the text that I was about to read

I would use the summary as a substitute for the original study material

I would use the summary to refresh my memory of the original study material

I would use the summary as a reminder that I had to read the original study material

The summary would help to get an overview of the original study material

The summary would help to understand the original study material',

Other (please specify)

Now we will ask some questions about what the summary should look like and cover. Q34 & **What would be the ideal type of the summary?**

Lecture notes

Blog post

Highlights of some kind in the original study material

Abstractive piece of text, such as a written overview of a text book, an abstract of a paper, etc.

Short video

A slide show

Other (please specify)



**Question Nr.** & **Question and Answer Options**

Q35 & **What is the ideal structure of the summary? (Multiple answers possible)** \\  &\(\square\) The summary should be a running text, without particular structure \\  &\(\square\) The summary should consist of highlights in the original study material, without particular structure \\  &\(\square\) The summary itself should contain special formatting, such as bold or cursive text, highlights, etc. \\  &\(\square\) The summary should contain diagrams \\  &\(\square\) The summary should contain tables \\  &\(\square\) The summary should contain graphs \\  &\(\square\) The summary should contain figures \\  &\(\square\) The summary should contain headings \\  &\(\square\) The summary should consist of different sections / paragraphs \\  &\(\square\) Other (please specify) \\ Q36 & **How much of the study material should be covered by the summary?** \\  & None of the & Almost none & Some of the & Most of the & All of the \\  & study & study & study \\  & material & material & material \\  & should be & should be & should be & should be \\  & covered & covered & covered \\  & (1) & (2) & (3) & (4) & (5) \\  & \(\square\) & \(\square\) & \(\square\) & \(\square\) \\ Q37 & **What should the style of this summary be?** \\  &\(\square\) Informative: the summary should simply convey the information that was in the \\  & original study material \\  &\(\square\) Indicative: the summary should give an idea of the topic of the study material, but \\  & not more \\  & Critical: the summary should give a critical review of the study material \\  & Aggregative: the summary should put different source texts in relation to one \\  & another and by doing this give an overview of a certain topic \\  & Other (please specify) \\ \end{tabular}





**Question Nr.** & **Question and Answer Options**

Q38 & **Is there anything else you would want us to know about your ideal summary that we have not covered yet?**

Open response Q39 & **Is there anything else you would want us to know about the original study material that we have not covered yet?**

Open response

**Look out questions**

Now, let's assume the pre-made summary was generated by a computer. You can assume that this machine generated summary captures all the needs you have identified in the previous questions. Q40 & **Would it make a difference to you whether the summary was generated by a computer program or by a human?**

Yes - _participants are led to Q41_

No - _participants are led to Q43_

Q41 & **Please explain the difference.**

Open response Q42 & **Which type of summary would you trust more:**

A summary generated by a human, for example a teacher or a good performing fellow student

A summary generated by a computer

No difference

**Please explain your answer.**

Open response Q44 & **Which type of summary would you trust more:**

A summary generated by a human, for example a teacher or a good performing fellow student

A summary generated by a computer

No difference



**Question Nr.** & **Question and Answer Options**

Now imagine that you can interact with the computer program that made the summary, in the form of a digital assistant. Imagine that your digital assistant made an initial summary for you and you can ask questions about it to your digital assistant and the assistant can answer them. Answers can be voice output, but also screen output, e.g. a written summary on the screen. In the next part we would like to investigate how you would interact with the assistant. Please do not feel restricted by the capabilities of today's digital assistants. Q45 & **Please choose the three most useful features for a digital assistant to have in this scenario.** \(\square\): Summarize particular parts of the study material with more detail \(\square\): Summarize particular parts of the study material with less detail \(\square\): Switch between different summary styles (for example highlighting vs a generated small piece of text) \(\square\): Explain why particular pieces ended up in the summary \(\square\): Provide the source of certain parts of the summary on request \(\square\): Search for different related sources based on the content of the summary \(\square\): Answer specific questions based on the content of the summary Q46 & **Please choose the three least useful features for a digital assistant to have in this scenario.** \(\square\): Summarize particular parts of the study material with more detail \(\square\): Summarize particular parts of the study material with less detail \(\square\): Switch between different summary styles (for example highlighting vs a generated small piece of text) \(\square\): Explain why particular pieces ended up in the summary \(\square\): Provide the source of certain parts of the summary on request \(\square\): Search for different related sources based on the content of the summary \(\square\): Answer specific questions based on the content of the summary Q47 & **Can you think of any other features that you would like your digital assistant to have to help you in this scenario?** Open response

**Background questions**

Thank you for filling out this survey so far! We would still like to ask you two final background questions. Q48 & **What is the current level of education you are pursuing?** \(\square\): Bachelor's degree \(\square\): Master's degree \(\square\): MBA \(\square\): Other, please specify



**Question Nr.** & **Question and Answer Options**

Q49 & **What is your main field of study?**

Open response

**Thank you!**

You have come to the end of our survey. Thanks a lot for helping out! We very much appreciate your time. Q50 & **If you would like to participate in the raffle to win a voucher, please fill out your e-mail address below. We will only use this e-mail address to blindly draw 10 names who win a voucher and to contact you if your name has been drawn.**

Open response

Full results trustworthiness and future feature questions

In this section we report the results for the exploratory questions that we asked about the trustworthiness of a summary generated by a machine versus a human, as well as the results for the questions about features for summarization with a digital voice assistant.

We find that participants are divided on the question whether it would make a difference to them whether the summary was generated by a machine or a computer. If we look at all participants together, we find that \(48.0.\%\) of the participants answered that it would make a difference, whereas \(52.0\%\) answered that it would not. However, if we split the participants based on study background, an interesting difference emerges (Figure 7(a)). Participants with a background in STEM indicated significantly more often that it would not make a difference to them, whereas the other groups of students indicated the opposite. Almost all participants who answered that it would make a difference said that they would not trust a computer on being able to find the relevant information, i.e., all seemed to favor the human generated summary. Only one participant advocated for the computer-generated summary as a _"computer is more objective."_ Almost all participants who said it would not matter to them did add the condition that the quality of the generated summary should be as good as if a human had generated it. One person wrote: _"If the summary captures all previously discussed elements it is effectively good for the same purpose. So then it does not matter who generated it."_ This comment exactly captures the motivation of the setup of our survey.

This caution regarding automatically generated summaries is confirmed by the question in which we asked which type of summary participants would trust more - a human-generated one or a machine-generated one. People chose the human-generated summary significantly more often (Figure 7(b)). This also holds for the participants with a STEM background, which aligns with the responses to the open questions we reported earlier - apparently participants do not fully trust that the condition they raised earlier would be satisfied, namely that only if the machine was just as good as the human, it would not matter for them whether the summary was generated by a machine or a human.

The results for the most and least useful features for a digital assistant in a summarization scenario are given in Figure 7(c) and 7(d). Adding more details to the summary and answering questions based on the content of the summary are very popular features, whereas summarizing parts of the input material with less detail is not.

Lastly, we asked participants whether they could think of any other features that they would like their digital assistant to have in the outlined scenario. A number of participants answered that they would like the digital assistant to generate questions based on the summary, so that they could test their own understanding. For example, one participant said: _"Make questions for me (to test me)"_ and another participant had a related comment: _"Maybe the the digital assistant could find old exam questions to link to parts of the summary where the question is related to, so that there is a function to test if you've understood the summary."_ Another line of answers pointed towards giving explicit relations between the input material and summary, for example: _"Show links between subject materials and what their relation is"_ and another person wrote: _"Dynamic linking from summary to original source is a great added value of generating a summary"_.

## Appendix E Examples evaluation questions

Here we give additional examples for the evaluation questions that can be used for our proposed evaluation methodology. The phrase _"a document that is important for your task"_ should be substituted to match the task at hand. For example, in the case of exam preparations, this could be replaced with _"a chapter that you need to learn for your exam preparation"_. Only the questions with the intended purpose factors should be used in the evaluation.

**Purpose factor _Use & Output factor Style_**:

* The _style_ of which of these two summaries is most useful to you to _retrieve_ a document that is important for your task?
* The _style_ of which of these two summaries is most useful to you to _preview_ a document that is important for your task?
* The _style_ of which of these two summaries is most useful to you to _substitute_ a document that is important for your task?
* The _style_ of which of these two summaries is most useful to you to _refresh your memory_ about a document that is important for your task? 

* The _style_ of which of these two summaries is most useful to you to _prompt_ you to read a source text that is important for your task?

**Purpose factor _Use & Output factor Format_**:

* The _format_ of which of these two summaries is most useful to you to _retrieve_ a document that is important for your task?
* The _format_ of which of these two summaries is most useful to you to _preview_ a document that is important for your task?
* The _format_ of which of these two summaries is most useful to you to _substitute_ a document that is important for your task?
* The _format_ of which of these two summaries is most useful to you to _refresh your memory_ about a document that is important for your task?
* The _format_ of which of these two summaries is most useful to you to _refront_ you to read a source text that is important for your task?

**Purpose factor _Use & Output factor Material_**:

* The _coverage_ of which of these two summaries is most useful to you to _retrieve_ a document that is important for your task?
* The _coverage_ of which of these two summaries is most useful to you to _preview_ a document that is important for your task?

* The _coverage_ of which of these two summaries is most useful to you to _prospect_ you to read a source text that is important for your task?

Figure 8: Results for the future feature questions. Answer type in brackets. MC = Multiple Choice, MR = Multiple Response. ** indicates significance (\(\chi^{2}\) or Fisher’s exact test), after Bonferroni correction, with \(p\ll 0.001\).