SlowGuess/NAACL_2022_OCR huggingface dataset:
Ruijia Cheng*  
University of Washington  
rcheng6@uw.edu
Alison Smith-Renner  
Dataminr Inc.  
arenner@dataminr.com
Ke Zhang  
Dataminr Inc.  
kzhang@dataminr.com
Joel R. Tetreault
Dataminr Inc.  
etreault@dataminr.com
Automatic text summarization systems commonly involve humans for preparing data or evaluating model performance, yet, there lacks a systematic understanding of humans' roles, experience, and needs when interacting with or being assisted by AI. From a human-centered perspective, we map the design opportunities and considerations for human-AI interaction in text summarization and broader text generation tasks. We first conducted a systematic literature review of 70 papers, developing a taxonomy of five interactions in AI-assisted text generation and relevant design dimensions. We designed text summarization prototypes for each interaction. We then interviewed 16 users, aided by the prototypes, to understand their expectations, experience, and needs regarding efficiency, control, and trust with AI in text summarization and propose design considerations accordingly.
In this era of rapid information consumption, access to high-quality summaries, such as online news highlights and research paper abstracts, is increasingly important. However, summarization is difficult for humans, demanding high cognitive load and expertise (Hidi and Anderson, 1986). Algorithmic approaches can automate summarization but typically underperform humans and require many high-quality human-written summaries for training (Durrett et al., 2016; Huang et al., 2021). AI systems that involve humans typically constrain their input to data preparation (Lloret et al., 2013) or final evaluation (Khashabi et al., 2021) as the first or last step in the summarization workflow. How can humans work together with AI to produce better summaries?
Other text generation tasks, such as machine translation and creative writing, offer inspiration.
Alejandro Jaimes
Dataminr Inc.  
ajaimes@dataminr.com
Figure 1: Five human-AI interactions in text generation from Study 1, illustrated as summarization tasks. Explanation of the actions and visual elements are in §2.2.
Beyond being data providers and evaluators, humans can generate text with the assistance of AI. For example, humans can revise AI translation (i.e., "post-editing") (Green et al., 2013) or write with AI suggestions (Clark et al., 2018). These interaction techniques may apply to summarization, yet our understanding of the possible human-AI interactions and how to design for them is incomplete.
To that end, we conducted two studies to explore: RQ1 What are the different ways that humans and AI can interact in text generation? RQ2 What are humans' experience and needs with these different types of AI assistance in the context of summarization? For these studies, we took a human-centered approach—focusing on what humans want from AI and how to improve their experience, rather than
* Research work was done while the author was interning at Dataminr Inc.
Human Language Technologies, pages 458 - 482
improving AI models. In Study 1, we conducted a systematic literature review of 70 papers that involve AI-assisted text generation systems and developed a taxonomy of five different human-AI interactions (Figure 1), including distinct human actions, controls, workflows, and interface features. In Study 2, to explore human experience with these five interactions in the specific context of text summarization, we conducted interviews with 16 users using prototype interfaces that we developed and identified varied user needs regarding efficiency, control, and trust, informing design considerations for AI-assisted text summarization and generation systems.
This work contributes: 1) the first known systematic literature review that provides a taxonomy of human-AI interactions for text generation, 2) an interview user study to understand user experience and needs in each interaction, and 3) an outline of the design space for AI-assisted text summarization and broader text generation systems. This research is a formative, initial exploration that first maps the different types of human-AI interaction for text generation and then surfaces user needs and perceptions for each type in the context of human-AI summarization. Such formative work is crucial for understanding what users might want and need from systems without biasing them by existing implementations—future researchers can use these findings to design and evaluate new human-AI text summarization systems.
2 Study 1: Systematic Literature Review of Human-AI Text Generation
For the first study, We performed a systematic literature review of human-AI text generation and developed a taxonomy of existing interaction types, which differ by the level of human control over the output, whether the AI iteratively updates its output based on human interaction, and whether human or AI initiates the interaction workflow. This study identifies and synthesizes the types of interaction explored in prior work. While we do not contribute any entirely new human-AI interactions for text generation, our taxonomy provides grounding for our second study—exploring user needs and experience with these interactions—as well as future research and design in this space. Future AI/HCI researchers can refer to this taxonomy when designing text generation systems or experiments to select the interaction and design elements that best
fit their scenario and to compare findings with prior work toward a more formalized understanding of the space.
We conducted a comprehensive search on academic papers about human-AI interaction in text generation from online indices and relevant workshops, e.g., ACM Digital Library, $^{1}$  arxiv.org, $^{2}$  HCI-NLP. $^{3}$  With the 692 papers queried from these sources, we manually coded the titles and abstracts against our inclusion criteria (e.g., for the goal of text generation, engaging humans beyond data preparation or evaluation) and identified a final set of 70 papers that describe human-AI interaction (or AI assistance) for the humans' goal of generating text (opposed to offline model training or evaluation). See further details on sources, query methods, inclusion criteria, and analysis process in Appendix A.1.
2.2 Findings: Five Human-AI Interactions in Text Generation
From the 70 papers, we identified five human-AI text generation workflows featuring distinct human actions—guiding model output, selecting or rating model output, post-editing, interactive editing, and writing with model assistance (Figure 1). We present the taxonomy that summarizes our findings in Table 1. For each interaction, we describe the human action, the type of human control, whether the model iterates based on human action, who (human or model) initiates the workflow, and interfaces from literature. We explain the interactions and visual elements in Figure 1 as follows:
Guiding Model Output. Humans can give model guidance to generate text (Figure 1, A). Here, humans are in the driver seat of the generation process with power to initiate and control the final output. Humans provide preferences (illustrated as the slider icon in the figure) to the model. The model takes human input and original text (the document icon) and produces text as the final product (the rightmost four-line summary icon), concluding the interaction with no further iteration. The specific guidance that a human can provide is varied and depends on their ML expertise. Model developers can adjust model parameters such as neural attention
1https://dl.acm.org/ 2https://arxiv.org/ 3https://aclanthology.org/2021. hcinlp-1.0/
Human Action	Human Control	Model Iteration	Workflow Initiation	Interfaces
Guiding model output	Generating No	Human	Model adjustment (e.g., attention weights and hyperparameters): (Zhang et al., 2011; Passali et al., 2021); Semantic prompts (e.g., keywords, topic tags, and written prompts): (Pouliquen et al., 2011; Ghazvininejad et al., 2017; Clark et al., 2018; Fan et al., 2018; Zarinbal et al., 2019; He et al., 2020; Osone et al., 2021; Chang et al., 2021; Strobelt et al., 2021); Style specification (e.g., sentiment slider, length of text): (Ghazvininejad et al., 2017; Freiknecht and Effelsberg, 2020); Modification on input (e.g., select/deselect chunks of input text): (Pouliquen et al., 2011; Zhang et al., 2011; Gehrmann et al., 2019)
Selecting or Rating Model Output	Selecting (or rating)	No	Model	Multiple choices: (Zhang et al., 2011; Pouliquen et al., 2011; Kreutzer et al., 2018; Rosa et al., 2020; Stiannon et al., 2020); Rating (e.g., Likert scale, numerical, binary): (Nguyen et al., 2017; Lam et al., 2018, 2019; Zarinbal et al., 2019; Bohn and Ling, 2021; Wu et al., 2021)
Post-editing	Editing	No	Model	Free-form text box: (Denkowski and Lavie, 2012; Yao et al., 2012; Yamaguchi et al., 2013; Turchi et al., 2014; Turner et al., 2015; Chu and Komlodi, 2017; Huang et al., 2020; Moramarco et al., 2021); Suggestion for edits (e.g., chatbot, substitution dropdowns): (Liu et al., 2011; Shi et al., 2013; Weisz et al., 2021; Passali et al., 2021); Scaffold for context (e.g., embedded dictionary): (Sugiyama et al., 2011; Lin, 2011); Productivity support (e.g., editing priority, auto-correction): (Lagarda et al., 2015; Peris and Casacuberta, 2019b; Wang et al., 2020; Zhao et al., 2020; Weisz et al., 2021)
Interactive Editing	Editing	Yes	Model	Prefix-based edits: (González-Rubio et al., 2013; Peris and Casacuberta, 2018, 2019b,a); Editors at arbitrary locations: (González-Rubio et al., 2016; Weng et al., 2019)
Writing with Model Assistance	Generating & Editing	Yes	Human	Auto-completion: (Green et al., 2014; Torregrosa et al., 2014; Gero and Chilton, 2019; Santy et al., 2019; Buschek et al., 2021; Ippolito et al., 2019; Calderwood et al., 2020; Bhat et al., 2021; Clark and Smith, 2021); Substitution dropdowns: (Green et al., 2014; Torregrosa et al., 2014; Santy et al., 2019; Ippolito et al., 2019; Gero and Chilton, 2019; Calderwood et al., 2020; Buschek et al., 2021; Padmakumar and He, 2021); Asynchronous suggestions: (Torregrosa Rivero et al., 2017; Clark et al., 2018)
Table 1: A taxonomy of human-AI interaction for text generation: human action, the type of control that humans have over the final output (Human Control), including generating, editing, or selecting/rating AI-generated text, whether the AI iterates, dynamically providing updated outputs based on human interaction (Model Iteration), who (Human or Model) initiates the workflow (Workflow Initiation), and what interfaces are used (Interfaces).
weights and hyperparameters. Alternatively, lay-users can prompt the model using semantic cues (e.g., keywords, topic tags, or descriptive prompts), or specify the style or details of the text (e.g., sentiment or length). Some interfaces support selecting or deselecting what from the input text should be used by the model.
Selecting or Rating Model Output. Humans can select from or rate generated texts (Figure 1, B). The model initiates such workflows by generating candidates of final output (the 3 four-line summary icons). The human does not directly generate text, instead, they have the control to select from or rate the candidates (the “+/-” icon) to support final output. We focus on the case where human input decides which candidate is chosen as the final product, but such feedback can also be used for online model training (e.g., active learning), represented as the dotted arrow going from the final product to
the model in the figure.
Post-editing. Post-editing (Figure 1, C)—common in machine translation (Green et al., 2013)—starts with text drafted by AI (the four-line summary icon with an incomplete last line), which humans edit (the pen icon). The workflow concludes when the human finishes editing, without any further iteration by the model, although the edited text can be used for future model training (the dotted arrow going from the final product summary back to the model). In many interfaces, users post-edit AI-generated text in text boxes. Some systems include innovative editing paradigms (e.g., chatbots) and other supports, such as drop-down menus for word or sentence substitution and scaffolds for context (e.g., embedded dictionaries) to aid understanding. Further, productivity supports can reduce workload, such as signalling where edits are needed and automatic
error amendment based on users' editing history.
Interactive Editing. Humans can edit text interactively with AI (Figure 1 D). Like post-editing, this dynamic editing interaction is also commonly used in machine translation systems, i.e., "interactive machine translation" (Barrachina et al., 2009). First, the model generates an incomplete draft (the three-line summary icon with a incomplete last line), to which human provides edits. While post-editing would stop at this point, in interactive editing, the model iterates on the human-edited text (the complete three-line summary on the top) to update and generate more text for continued human editing, iteratively and in real-time (the solid line arrow). This iterative human-AI interaction (in the dashed frame) results in the final product. Interactive editing interfaces take many forms. "Prefix-based" edits are specified on a left-to-right phrase by phrase fashion, while the model makes new predictions on the rest of the sentence. In other interfaces, humans edit at arbitrary positions of the AI-generated sentence and the model updates the whole sentence. Some systems offer additional editing support, such as highlighting necessary edits and dropouts for substitution.
Writing with Model Assistance. Finally, humans can write with AI assistance (Figure 1, E). Humans initiate this workflow and have a high-level control: humans begin writing while the model provides suggestions and can revise their writing based on the suggestions or ignore them. This iterative human-AI interaction (in the dashed frame) generates the final text. In this process, the model iterates, providing additional suggestions based on humans' writing. Assisted writing interfaces include real-time auto-completion, which can happen at the word, phrase, or sentence level. Some systems offer alternative suggestions in dropouts from which humans choose. Others provide asynchronous suggestions, presented after users have finished writing to reduce distraction.
3 Study 2: Interview Study on AI-assisted Text Summarization
We present a user study in which we evaluated interactions in AI-assisted text summarization with the context of text summarization through interviews aided by prototype interfaces. Our goal is not to prescribe which is "best" but to achieve a qualitative understanding of user needs with each
interaction to inform future research and design.
3.1 Prototype Design
We first developed prototype interfaces to represent the five interactions identified in Study 1. We used these prototypes in our user interviews to elicit needs, expectations, and experience around AI-assisted text summarization.
While some prototypes for these interactions exist in the literature for broader text generation tasks, many include additional features and visual design that may affect users' perceptions, therefore, we develop our own set of consistent, simple prototypes for exploring text summarization specifically. Each interactive prototype, implemented in  $\mathrm{Figma}^4$  or Google Docs, allowed participants to read an online news document and generate summaries with the support of a hypothetical AI model.
All prototypes were built based on the "Wizard-of-oz" prototyping concept (Kelley, 1983), commonly used in user studies on intelligent systems. This concept allows users to interact with intelligent systems that are not fully implemented; instead, "system outputs" are prepared manually by the research team. This method allows designers of intelligent systems to rapidly test design concepts, understand user experience, and iterate on their design. This method has been used to prototype NLP systems, for example, in the design of chatbots (Zhou et al., 2019; Avula, 2018).
In our study, participants interacted with "Wizard-of-oz" prototypes instead of implemented AI models, so that we could explore human perspectives and user needs for the different interaction types without being limited by model performance or other system characteristics, such as unpredictability and slow updates. Each of our five prototypes used the same original text (a news article from the set used in the warm-up activity, explained in §3.2) that needed to be summarized. The prototypes differed by the interaction interface they supported (from Study 1). Depending on the interaction type, the prototype presented users with "AI-generated" summaries, outputs or suggestions that were pre-defined and written by the research team. While the human-written outputs used in this study might not necessarily imitate the quality and style of AI-generated summaries, outputs, or suggestions, they were intended to provide concrete examples of the interaction types and
<sup>4</sup>https://www.figma.com/
461 4
elicit participant's needs and expectations. Future work should explore how more realistic model outputs affect human perceptions. The specific design of each prototype (and screenshots) are described in the Appendix A.3.
Participants. We recruited 16 participants (10 females, 6 males, all based in the U.S.) from Upwork with varied professional backgrounds and varied familiarity with the domains of Reddit posts, online news, and U.S. government bills. We intentionally recruited participants who have at least some level of familiarity with text editing or summarization from professional or educational settings, which ensures that participants could speak about for which summarization tasks they desire assistance and describe their needs for such interactions. See Appendix A.2 for demographic details. The study took 2.5 hours, and we paid each participant $60.
Procedure. Each participant first did a 60-minute offline warm-up activity less than 48 hours before the interview, where they summarized six articles (two Reddit posts on scams or finance, $^{7}$  two news articles from CNN/Daily Mail, $^{8}$  and two U.S. government bills $^{9}$ ). This activity aimed to expose participants to summarization with articles written in different styles and with varied domain context.
Then, during the 1-on-1 semi-structured recorded video interviews (90 minutes), participants first reflected on their experience in the warm-up summarization tasks and then interacted with all five prototypes, in random order, as users of AI-assisted summarization systems. Participants were shown a news article from the warm-up task (all prototypes used the same article) and also asked to imagine using the prototypes for the other documents from the warm-up. They interacted with the interfaces and received pre-determined outputs that mimicked AI assistance:
1. Guiding Model Output: participants could change the desired summary length and style (formal or informal) using sliders and highlight parts of the original text that should be
in the summary. We asked participants what additional guidance they wanted to offer to the model.
2. Selecting or Rating Model Output: participants chose from three AI-generated summaries.
3. Post-editing: participants saw an editable AI-generated summary (text box) and talked through how they would edit it.
4. Interactive Editing: given an AI-generated summary (text box), participants chose possible edits to the first sentence (dropdown menu) and then requested the model to update the summary based on those edits. We asked participants to imagine an alternate interface where they could edit anywhere in the summary.
5. Writing with Model Assistance: following a "wizard-of-oz" prototyping method (Kelley, 1983), a researcher acted as an AI bot in a Google Doc. As the participants typed their summaries, the "bot" provided suggested next sentences and added comments.
Participants were then prompted to talk through experience with each prototype, including what they liked or disliked regarding efficiency, control, and trust, and how they would improve them. See Appendix A.3 for screenshots of each prototype and A.5 for the interview questions and instructions used in the study. We collected and transcribed 22.6 hours of interview recordings, which were analyzed using thematic analysis (Guest et al., 2011). We performed two rounds of open coding and developed themes reported in the following sections. We refer to participants as P1-16 with gender non-specific pronouns (i.e., they, them). We present findings regarding efficiency, control, and trust—common themes in our interviews and key dimensions in the human-AI interaction literature (Amershi et al., 2019; Shneiderman, 2020). We include additional findings about user expectation and needs in writing summaries in Appendix A.4.
3.3 Findings
3.3.1 General Expectations & Needs
Expectation on AI to improve summarization. The warm-up summarization tasks were challenging and tedious. Summarizing the Government Bills was slow for many due to the unfamiliar domain, jargon, and "super dry, super repetitive" (P3) style. Summarizing informally-written and opin
<sup>5</sup>https://www.upwork.com/
6 Adequate payment in the United States.
7Extracted from r/scam and r/wallstreetbets
<sup>8</sup>https://paperswithcode.com/sota/xt-summarization-on-cnn-daily-mail-2
<https://www.tensorflow.org/datasets/talog/billsum>
ionated Reddit posts was also difficult as many were unsure about whether to keep the authors' perspective or summarize neutrally. Therefore, participants hoped that AI could speed up and ease this process. Many envisioned AI-generated summaries as a useful starting point: "determining where to get started can be a big roadblock for some writers...being able to have that auto-generated summary as your baseline to develop your ideas off I think would be really helpful" (P13) Further, participants hope to use AI suggestions to improve the content and structure of their writing: "it gives more of a third party look at things...just kind of compare and contrast it to what I've done, to make sure that I'm on the right track" (P15).
Different desire for control. Most agreed they at least wanted the ability to proofread the AI-generated summaries, or to “have the final say” on whether it was good as a final product (P3). Some said this responsibility was a habit of professionalism; others were cautious of the work done by AI and wanted to ensure quality: “it was drilled into my head that these devices are tools and they can fail...we’re always responsible for overseeing what the computer does” (P7). Beyond simple editing, participants had a varied desire for control. Some felt summarization was “not necessarily a creative enterprise” and, therefore, were willing to “relinquish a little bit of control to AI” for efficiency (P3), while others wanted to participate in the entire generation process. These participants preferred to compose their own summary using AI strictly as an aid, e.g., “it would just simply be used as a tool for me, not as something to replace my work” (P12). Many felt uncomfortable using AI-generated summaries directly or after only proofreading edits due to the sense of “plagiarism”, and as a result, wanted the ability to rewrite summaries into their own words. Desired control could also vary by situation. For example, P7 wanted more control when summarizing for the bills because that was a more “serious and important task,” while P8 would be more lenient when summarizing the Reddit posts: “even [the summary] doesn’t capture everything, it is good as long as the summary kind of outlines the the key points of the article.”
Need to understand AI to reduce over-reliance and boost trust. Participants were concerned that they might rely on AI too much and lacked confidence to correct it even when it was wrong.