SlowGuess/NAACL_2022_OCR huggingface dataset:
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Michael Volske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, Copenhagen, Denmark. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Florian, and Claire Cardie. 2016. A sentence compression based framework to query-focused multi-document summarization. CoRR, abs/1606.07548.
Zeqiu Wu, Rik Koncel-Kedziorski, Mari Ostendorf, and Hannaneh Hajishirzi. 2020. Extracting summary knowledge graphs from long documents. CoRR, abs/2009.09162.
Yuexiang Xie, Fei Sun, Yang Deng, Yaliang Li, and Bolin Ding. 2021. Factual consistency evaluation for text summarization via counterfactual estimation. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 100-110. Association for Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Parek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462, Vancouver, Canada. Association for Computational Linguistics.
Shuo Zhang, Zhuyun Dai, Krisztian Balog, and Jamie Callan. 2020a. Summarizing and exploring tabular data in conversational search. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1537-1540. ACM.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. *Bertscore: Evaluating text generation with BERT*. In *8th International Conference on Learning Representations*, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A Overview context factors
Input Factors	Purpose Factors	Output Factors
Form	Situation	Material
Structure: How is the input text structured? E.g., subheadings, rhetorical patterns, etc.	Tied: It is known who will use the summary, for what purpose and when.	Covering: The summary covers all of the important information in the source text.
Scale: How large is the input data that we are summarizing? E.g., a book, a chapter, a single article, etc.	Floating: It is not (exactly) known who will use the summary, for what purpose or when.	Partial: The summary (intentionally) covers only parts of the important information in the source text.
Medium: What is the input language type? E.g., full text, telegraphese style, etc. This also refers to which natural language is used.	Audience	Format
Genre: What type of literacy does the input text have? E.g., description, narrative, etc.	Targetted: A lot of domain knowledge is expected from the readers of the summary.	Running: The summary is formatted as an abstract like text.
Subject Type	Untargetted: No domain knowledge is expected from the readers of the summary.	Headed: The summary is structured following a certain standardised format with headings and other explicit structure.
Ordinary: Everyone could understand this input type.	Use	Style
Specialized: You need to speak the jargon to understand this input type.	Retrieving: Use the summary to retrieve source text.	Informative: The summary conveys the raw information that is in the source text.
Restricted: The input type text is only understandable for people familiar with a certain area, for example because it contains local names.	Previewing: Use the summary to preview a text.	Indicative: The summary just states the topic of the source text, nothing more.
Unit	Substitutes: Use the summary to substitute the source text.	Critical: The summary gives a critical review of the merits of the source text.
Single: Only one input source is given.	Refreshing: Use the summary to refresh ones memory of the source text.	Aggregative: Different source texts are put in relation to one another to give an overview of a certain topic.
Multi: Multiple input sources are given.	Prompts: Use the summary as action prompt to read the source text.
Table 3: Overview of different context factors classes defined by Spärck Jones (1998), with descriptions of the factors within these classes.
B Survey overview
Figure 7: Overview survey design.
C Verbatim survey overview
Table 4: A complete overview of the survey. This table includes the explanation that participants received, as well as all the questions and the answer options. If a question was the start of a branch, the direction of the branch has been written behind the answer options in italic. (This was never shown to the participants.) Note that the survey was performed in SurveyMonkey. The survey had a lay-out as provided by SurveyMonkey, i.e., it consisted of different pages and colors were used to highlight certain important parts in texts.
<sup>6</sup>http://surveymonkey.com
Question Nr. Question and Answer Options
Introduction and Instructions
Thank you for taking the time to fill out this survey! Before you start, please take the time to read these instructions carefully. If you still have any questions after reading the instructions, please send them to m.a.terhoeve@uva.nl.
We will give away 10 bol. com vouchers of 10 euros each among the participants. If you would like to take part in the raffle, you can leave your email address at the end of this survey.
Goal of the study
The goal of this survey is to get insight in how summaries help or can help you when studying.
What the survey will look like
In what follows you will get questions that aim to develop an understanding for:
- For which types of study material it is useful to have summaries
- How these summaries can help you with your task
- What these summaries should look like
We expect this survey to take approximately 10 minutes of your time.
Use the next button to go to the next page once you have filled out all the questions on the page. Use the prev button to go back one page.
About your privacy
We value your privacy and will process your answers anonymously. The answers of all participants in this survey will be used to gain insight in how pre-made summaries can be helpful for different types of studying activities. The answers will be presented in a research paper about this topic. This will be done either in an aggregated manner, or by citing verbatim examples of the answers. Again, this will all be done anonymously.
I agree that I have read and understood the instructions. I also understand that my participation in this survey is voluntarily.
□ I agree
Q2 Important! Some background knowledge you need to know
Throughout this survey we make use of the term pre-made summary. It is very important that you understand what this means. On this page we explain this term, so please make sure to read this carefully.
Definition pre-made summary
One type of summary is one that you make yourself. Another type of summary is one that has been made for you. In this survey, we focus on this latter type and we call them pre-made summaries.
Who makes these pre-made summaries?
These pre-made summaries can be made by a person, for example your teacher, your friend, a fellow student or someone at some official organisation, etc. The pre-made summaries can also be made by a computer.
What kinds of summaries are we talking about?
There are no restrictions on what these pre-made summaries can look like. On the contrary, that is one of the things we aim to find out with this survey! But, to give some examples, you could think of a written overview of a text book, highlights in text to draw your attention to important bits, blog posts, etc. These are really just examples and don't let them limit your creativity! You can come up with any example of a pre-made summary that is helpful for you.
Yes, I understand what a pre-made summary is!
Q3 Please think back to your recent study activities. Examples of study activities can be: studying for an exam, writing a paper, doing homework exercises, etc. Note that these are just examples, any other study activity is fine too.
Did you use a pre-made summary in any of these study activities?
Yes-participants are led to Q6
No-participants are led to Q4
Q4 Can you think of one of your recent study activities where a pre-made summary would have been useful for you?
□ Yes-participants are led to Q25
No-participants are led to Q5
Question Nr.	Question and Answer Options
Q5	Why do you think a pre-made summary would not have helped you with any of your recent study activities?
Open response – participants are led to Q48
Start branch of participants who described an existing summary
Q6	If you have multiple study activities where you used a pre-made summary, please take the one where you found the pre-made summary most useful.
The original study material consisted of
□ Mainly text – participants are led to Q8
□ Mainly figures – participants are led to Q7
□ Mainly video – participants are led to Q7
□ Mainly audio – participants are led to Q7
□ A combination of some or all of the above – participants are led to Q7
□ I do not know, because I have not seen the study material – participants are led to Q7
□ Other (please specify) – participants are led to Q7
Q7	For now we narrow down our survey to study material that is mostly textual. Do you recall any other recent study activity where you made use of a pre-made summary and where the original study material mainly consisted of text?
□ Yes – participants are led to Q8
□ No – participants are led to Q48
Q8	What was the goal of this study activity?
□ Studying for an exam
□ Writing a paper / essay / report / etc.
□ Doing homework exercises
□ Other (please specify)
Q9	Who made this pre-made summary?
□ A teacher or teaching assistant
□ A fellow student
□ An official organisation
□ The authors of the original study material
□ A computer program
□ I am not sure, I found it online
□ Other (please specify)
Q18 How was the summary structured? (Multiple answers possible)
The summary was a running text, without particular structure
The summary consisted of highlights in the original study material, without particular structure
The summary itself contained special formatting, such as bold or cursive text, highlights, etc
The summary contained diagrams
The summary contained tables
The summary contained graphs
The summary contained figures
The summary contained headings
The summary consisted of different sections / paragraphs
□ Other (please specify)
Q19 How much of the study material was covered by the summary?
None of the study material was covered	Almost none of the study material was covered	Some of the study material was covered	Most of the study material was covered	All of the study material was covered
(1)	(2)	(3)	(4)	(5)
□	□	□	□	□
Q20 What was the style of this summary?
□ Informative: the summary simply conveyed the information that was in the original study material
□ Indicative: the summary gave an idea of the topic of the study material, but not more
Critical: the summary gave a critical review of the study material
□ Aggregative: the summary put different source texts in relation to one another and by doing this gave an overview of a certain topic
□ Other (please specify)
Q35 What is the ideal structure of the summary? (Multiple answers possible)
The summary should be a running text, without particular structure
The summary should consist of highlights in the original study material, without particular structure
The summary itself should contain special formatting, such as bold or cursive text, highlights, etc.
The summary should contain diagrams
The summary should contain tables
The summary should contain graphs
The summary should contain figures
The summary should contain headings
The summary should consist of different sections / paragraphs
□ Other (please specify)
Q36 How much of the study material should be covered by the summary?
None of the study material should be covered	Almost none of the study material should be covered	Some of the study material should be covered	Most of the study material should be covered	All of the study material should be covered
(1)	(2)	(3)	(4)	(5)
□	□	□	□	□
Q37 What should the style of this summary be?
□ Informative: the summary should simply convey the information that was in the original study material
□ Indicative: the summary should give an idea of the topic of the study material, but not more
Critical: the summary should give a critical review of the study material
□ Aggregative: the summary should put different source texts in relation to one another and by doing this give an overview of a certain topic
□ Other (please specify)
Question Nr.	Question and Answer Options
Q38	Is there anything else you would want us to know about your ideal summary that we have not covered yet?
Open response
Q39	Is there anything else you would want us to know about the original study material that we have not covered yet?
Open response
Look out questions
Now, let's assume the pre-made summary was generated by a computer. You can assume that this machine generated summary captures all the needs you have identified in the previous questions.
Q40	Would it make a difference to you whether the summary was generated by a computer program or by a human?
□ Yes – participants are led to Q41
□ No – participants are led to Q43
Q41	Please explain the difference.
Open response
Q42	Which type of summary would you trust more:
□ A summary generated by a human, for example a teacher or a good performing fellow student
□ A summary generated by a computer
□ No difference
Q43	Please explain your answer.
Open response
Q44	Which type of summary would you trust more:
□ A summary generated by a human, for example a teacher or a good performing fellow student
□ A summary generated by a computer
□ No difference
Now imagine that you can interact with the computer program that made the summary, in the form of a digital assistant. Imagine that your digital assistant made an initial summary for you and you can ask questions about it to your digital assistant and the assistant can answer them. Answers can be voice output, but also screen output, e.g. a written summary on the screen. In the next part we would like to investigate how you would interact with the assistant. Please do not feel restricted by the capabilities of today's digital assistants.
Q45	Please choose the three most useful features for a digital assistant to have in this scenario.
□ Summarize particular parts of the study material with more detail
□ Summarize particular parts of the study material with less detail
□ Switch between different summary styles (for example highlighting vs a generated small piece of text)
□ Explain why particular pieces ended up in the summary
□ Provide the source of certain parts of the summary on request
□ Search for different related sources based on the content of the summary
□ Answer specific questions based on the content of the summary
Q46	Please choose the three least useful features for a digital assistant to have in this scenario.
□ Summarize particular parts of the study material with more detail
□ Summarize particular parts of the study material with less detail
□ Switch between different summary styles (for example highlighting vs a generated small piece of text)
□ Explain why particular pieces ended up in the summary
□ Provide the source of certain parts of the summary on request
□ Search for different related sources based on the content of the summary
□ Answer specific question based on the content of the summary
Q47	Can you think of any other features that you would like your digital assistant to have to help you in this scenario?
Open response
Background questions
Q48	Thank you for filling out this survey so far! We would still like to ask you two final background questions.
What is the current level of education you are pursuing?
□ Bachelor's degree
□ Master's degree
□ MBA
□ Other, please specify
Question Nr.	Question and Answer Options
Q49	What is your main field of study?
Open response
Thank you!
Q50	You have come to the end of our survey. Thanks a lot for helping out! We very much appreciate your time.
If you would like to participate in the raffle to win a voucher, please fill out your e-mail address below. We will only use this e-mail address to blindly draw 10 names who win a voucher and to contact you if your name has been drawn.
Open response
D Full results trustworthiness and future feature questions
In this section we report the results for the exploratory questions that we asked about the trustworthiness of a summary generated by a machine versus a human, as well as the results for the questions about features for summarization with a digital voice assistant.
We find that participants are divided on the question whether it would make a difference to them whether the summary was generated by a machine or a computer. If we look at all participants together, we find that  $48.0\%$  of the participants answered that it would make a difference, whereas  $52.0\%$  answered that it would not. However, if we split the participants based on study background, an interesting difference emerges (Figure 8a). Participants with a background in STEM indicated significantly more often that it would not make a difference to them, whereas the other groups of students indicated the opposite. Almost all participants who answered that it would make a difference said that they would not trust a computer on being able to find the relevant information, i.e., all seemed to favor the human generated summary. Only one participant advocated for the computer-generated summary as a "computer is more objective." Almost all participants who said it would not matter to them did add the condition that the quality of the generated summary should be as good as if a human had generated it. One person wrote: "If the summary captures all previously discussed elements it is effectively good for the same purpose. So then it does not matter who generated it." This comment exactly captures the motivation of the setup of our survey.
This caution regarding automatically generated summaries is confirmed by the question in which we asked which type of summary participants would trust more – a human-generated one or a machine-generated one. People chose the human-generated summary significantly more often (Figure 8b). This also holds for the participants with a STEM background, which aligns with the responses to the open questions we reported earlier – apparently participants do not fully trust that the condition they raised earlier would be satisfied, namely that only if the machine was just as good as the human, it would not matter for them whether the summary was generated by a machine or a human.
The results for the most and least useful features for a digital assistant in a summarization scenario are given in Figure 8c and 8d. Adding more details to the summary and answering questions based on the content of the summary are very popular features, whereas summarizing parts of the input material with less detail is not.
Lastly, we asked participants whether they could think of any other features that they would like their digital assistant to have in the outlined scenario. A number of participants answered that they would like the digital assistant to generate questions based on the summary, so that they could test their own understanding. For example, one participant said: "Make questions for me (to test me)" and another participant had a related comment: "Maybe the the digital assistant could find old exam questions to link to parts of the summary where the question is related to, so that there is a function to test if you've understood the summary." Another line of answers pointed towards giving explicit relations between the input material and summary, for example: "Show links between subject materials and what their relation is" and another person wrote: "Dynamic linking from summary to original source is a great added value of generating a summary".
E Examples evaluation questions
Here we give additional examples for the evaluation questions that can be used for our proposed evaluation methodology. The phrase "a document that is important for your task" should be substituted to match the task at hand. For example, in the case of exam preparations, this could be replaced with "a chapter that you need to learn for your exam preparation". Only the questions with the intended purpose factors should be used in the evaluation.
Purpose factor Use & Output factor Style:
- The style of which of these two summaries is most useful to you to retrieve a document that is important for your task?
- The style of which of these two summaries is most useful to you to preview a document that is important for your task?
- The style of which of these two summaries is most useful to you to substitute a document that is important for your task?
- The style of which of these two summaries is most useful to you to refresh your memory about a document that is important for your task?
(a) Would it make a difference to you whether the summary was generated by a computer program or by a human? (MC)
(b) Which type of summary would you trust more? (MC)
(d) Please choose the three least useful features for a digital assistant to have in this scenario. (MR)
(c) Please choose the three most useful features for a digital assistant to have in this scenario. (MR)
Figure 8: Results for the future feature questions. Answer type in brackets.  $\mathrm{MC} =$  Multiple Choice,  $\mathrm{MR} =$  Multiple Response. \*\* indicates significance  $(\chi^2$  or Fisher's exact test), after Bonferroni correction, with  $p\ll 0.001$
- The style of which of these two summaries is most useful to you to prompt you to read a source text that is important for your task?
Purpose factor Use & Output factor Format:
- The format of which of these two summaries is most useful to you to retrieve a document that is important for your task?
- The format of which of these two summaries is most useful to you to preview a document that is important for your task?
- The format of which of these two summaries is most useful to you to substitute a document that is important for your task?
- The format of which of these two summaries is most useful to you to refresh your memory about a document that is important for your task?
- The format of which of these two summaries is most useful to you to prompt you to read a source text that is important for your task?
Purpose factor Use & Output factor Material:
- The coverage of which of these two summaries is most useful to you to retrieve a document that is important for your task?
- The coverage of which of these two summaries is most useful to you to preview a document that is important for your task?
- The coverage of which of these two summaries is most useful to you to substitute a document that is important for your task?
- The coverage of which of these two summaries is most useful to you to refresh your memory about a document that is important for your task?
- The coverage of which of these two summaries is most useful to you to prompt you to read a source text that is important for your task?