SlowGuess/NAACL_2022_OCR huggingface dataset:
Linguistics, Vancouver, Canada, 43-48. https://aclanthology.org/P17-4008
Jesús González-Rubio, Daniel Ortiz-Martínez, José Miguel Benedí, and Francisco Casacuberta. 2013. Interactive machine translation using hierarchical translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 244-254.
Jesús González-Rubio, Daniel Ortiz-Martínez, Francisco Casacuberta, and José-Miguel Benedí. 2016. Beyond prefix-based interactive translation prediction. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. 198-207.
Spence Green, Jeffrey Heer, and Christopher D. Manning. 2013. The Efficacy of Human Post-Editing for Language Translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 439-448. https://doi.org/10.1145/2470654.2470718
Spence Green, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster, and Christopher D. Manning. 2014. Human Effort and Machine Learnability in Computer Aided Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1225-1236. https://doi.org/10.3115/v1/D14-1130
Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011. Applied thematic analysis. Sage Publications.
Junxian He, Wojciech Krysciński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020. Ctrlsum: Towards generic controllable text summarization. arXiv preprint arXiv:2012.04281 (2020).
Suzanne Hidi and Valerie Anderson. 1986. Producing Written Summaries: Task Demands, Cognitive Operations, and Implications for Instruction. Review of Educational Research 56, 4 (1986), 473-493. http://www.jstor.org/stable/1170342
Cheng-Zhi Anna Huang, Hendrik Vincent Koops, Ed Newton-Rex, Monica Dinculescu, and Carrie J. Cai. 2020. AI Song Contest: Human-AI Co-Creation in Songwriting. *ArXiv* abs/2010.05388 (2020).
Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021. The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey. arXiv preprint arXiv:2104.14839 (2021).
Daphne Ippolito, David Grangier, Chris Callison-Burch, and Douglas Eck. 2019. Unsupervised Hierarchical Story Infilling. In Proceedings of the First Workshop on Narrative Understanding. Association for Computational Linguistics, Minneapolis, Minnesota, 37-43. https://doi.org/10.18653/v1/w19-2405
John F Kelley. 1983. An empirical methodology for writing user-friendly natural language computer applications. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 193-196.
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg,
Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel S Weld. 2021. Genie: A leaderboard for human-in-the-loop evaluation of text generation. arXiv preprint arXiv:2101.06561 (2021).
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. 2018. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627 (2018).
Antonio L. Lagarda, Daniel Ortiz-Martínez, Vicent Alabau, and Francisco Casacuberta. 2015. Translating without in-domain corpus: Machine translation post-editing with online learning techniques. Computer Speech & Language 32, 1 (2015), 109-134. https://doi.org/10.1016/j.csl.2014.10.004 Hybrid Machine Translation: integration of linguistics and statistics.
Tsz Kin Lam, Julia Kreutzer, and Stefan Riezler. 2018. A reinforcement learning approach to interactive-predictive neural machine translation. arXiv preprint arXiv:1805.01553 (2018).
Tsz Kin Lam, Shigehiko Schamoni, and Stefan Riezler. 2019. Interactive-predictive neural machine translation through reinforcement and imitation. arXiv preprint arXiv:1907.02326 (2019).
Donghui Lin. 2011. Humans in the Loop of Localization Processes. In The Language Grid. Springer, 201-213.
Chien-Liang Liu, Chia-Hoang Lee, Ssu-Han Yu, and Chih-Wei Chen. 2011. Computer assisted writing system. Expert Systems with Applications 38, 1 (2011), 804-811.
Elena Lloret, Laura Plaza, and Ahmet Aker. 2013. Analyzing the capabilities of crowdsourcing services for text summarization. Language resources and evaluation 47, 2 (2013), 337-369.
Francesco Moramarco, Alex Papadopoulos Korfiatis, Aleksandar Savkov, and Ehud Reiter. 2021. A preliminary study on evaluating Consultation Notes with Post-Editing. arXiv preprint arXiv:2104.04402 (2021).
Khanh Nguyen, Hal Daume III, and Jordan Boyd-Graber. 2017. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv:1707.07402 (2017).
Hiroyuki Osone, Jun-Li Lu, and Yoichi Ochiai. 2021. *BunCho: AI Supported Story Co-Creation via Unsupervised Multitask Learning to Increase Writers' Creativity in Japanese*. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411763.3450391
Vishakh Padmakumar and He He. 2021. Machine-in-the-Loop Rewriting for Creative Image Captioning. arXiv preprint arXiv:2111.04193 (2021).
Tatiana Passali, Alexios Gidiotis, Efstathios Chatzikyriakidis, and Grigorios Tsoumakas. 2021. Towards Human-Centered Summarization: A Case Study on Financial News. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing. Association for Computational Linguistics, Online, 21-27. https://aclanthology.org/2021.hcinlp-1.4
Álvaro Peris and Francisco Casacuberta. 2018. Active Learning for Interactive Neural Machine Translation
of Data Streams. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 151-160. https://doi.org/10.18653/v1/K18-1015
Álvaro Peris and Francisco Casacuberta. 2019a. Interactive-predictive neural multimodal systems. In Iberian Conference on Pattern Recognition and Image Analysis. Springer, 16-28.
Alvaro Peris and Francisco Casacuberta. 2019b. Online learning for effort reduction in interactive neural machine translation. Computer Speech & Language 58 (2019), 98-126.
Bruno Pouliquen, Christophe Mazenc, and Aldo Iorio. 2011. Tapa: A user-driven translation system for patent documents based on domain-aware Statistical Machine Translation. In Proceedings of the 15th Annual conference of the European Association for Machine Translation. European Association for Machine Translation, Leuven, Belgium. https://aclanthology.org/2011.eamt-1.2
Rudolf Rosa, Ondrej Dusek, Tom Kocmi, David Marevcek, Tomavs Musil, Patricia Schmidtova, Dominik Jurko, Ondvrej Bojar, Daniel Hrbek, David Kovsvtak, Martina Kinsk'a, Josef Dolevzal, and Klara Vosecka. 2020. THEaiTRE: Artificial Intelligence to Write a Theatre Play. In AI4Narratives@IJCAI.
Sebastin Santy, Sandipan Dandapat, Monojit Choudhury, and Kalika Bali. 2019. INMT: Interactive neural machine translation prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations. 103-108.
Chunqi Shi, Donghui Lin, and Toru Ishida. 2013. Agent Metaphor for Machine Translation Mediated Communication. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (Santa Monica, California, USA) (IUI '13). Association for Computing Machinery, New York, NY, USA, 67-74. https://doi.org/10.1145/2449396.2449407
Ben Shneiderman. 2020. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. International Journal of Human-Computer Interaction 36, 6 (2020), 495-504. https://doi.org/10.1080/10447318.2020.1741118 arXiv:https://doi.org/10.1080/10447318.2020.1741118
Nisan Stiannon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325 (2020).
Hendrik Strobelt, Jambay Kinley, Robert Krueger, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2021. GenNI: Human-AI Collaboration for Data-Backed Text Generation. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 1106-1116.
Bruno Akio Sugiyama, Junia Coutinho Anacleteo, and Helena Medeiros Caseli. 2011. Assisting users in
a cross-cultural communication by providing culturally contextualized translations. In Proceedings of the 29th ACM international conference on Design of communication. 189-194.
Daniel Torregrosa, Mikel L. Forcada, and Juan Antonio Pérez-Ortiz. 2014. An open-source web-based tool for resource-agnostic interactive translation prediction. Prague Bull. Math. Linguistics 102 (2014), 69-80.
Daniel Torregrosa Rivero, Juan Antonio Pérez-Ortiz, Mikel L Forcada, et al. 2017. Comparative human and automatic evaluation of glass-box and black-box approaches to interactive translation prediction. (2017).
Marco Turchi, Matteo Negri, and Marcello Federico. 2014. Data-driven annotation of binary MT quality estimation corpora based on human post-editions. Machine translation 28, 3-4 (2014), 281-308.
Anne M Turner, Loma Desai, Kristin Dew, Nathalie Martin, and Katrin Kirchhoff. 2015. Machine Assisted Translation of Health Materials to Chinese: An Initial Evaluation. Studies in health technology and informatics 216 (2015), 979. http://europepmc.org/abstract/MED/26262281
Jiayi Wang, Ke Wang, Niyu Ge, Yangbing Shi, Yu Zhao, and Kai Fan. 2020. Computer assisted translation with neural quality estimation and automatic post-editing. arXiv preprint arXiv:2009.09126 (2020).
Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces. 402-412.
Rongxiang Weng, Hao Zhou, Shujian Huang, Lei Li, Yifan Xia, and Jiajun Chen. 2019. Correct-and-memorize: Learning to translate from interactive revisions. arXiv preprint arXiv:1907.03468 (2019).
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 (2021).
Takuro Yamaguchi, Reiko Hishiyama, Daisuke Kitagawa, Yuu Nakajima, Rieko Inaba, and Donghui Lin. 2013. Evaluation of rewriting service in language translation web services workflow. In 2013 International Conference on Culture and Computing. IEEE, 21-26.
Jia Yao, Muyun Yang, Junmao Meng, Dongqing Xiao, Tiejun Zhao, and Sheng Li. 2012. How Good is Web Based MT: A Case Study on Paper Title Translation. In 2012 International Conference on Asian Language Processing. IEEE, 213-216.
Marzieh Zarinbal, Azadeh Mohebi, Hesamoddin Mosalli, Razieh Haratinik, Zahra Jabalameli, and Farnoush Bayatmakou. 2019. A new social robot for interactive query-based summarization: Scientific document summarization. In International Conference on Interactive Collaborative Robotics. Springer, 330-340.
Yi Zhang, Dingding Wang, and Tao Li. 2011. iDVS: an interactive multi-document visual summarization
system. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 569-584.
Tianxiang Zhao, Lemao Liu, Guoping Huang, Huayang Li, Yingling Liu, Liu GuiQuan, and Shuming Shi. 2020. Balancing quality and human involvement: An effective approach to interactive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 9660-9667.
Michelle X Zhou, Carolyn Wang, Gloria Mark, Huahai Yang, and Kevin Xu. 2019. Building Real-World Chatbot Interviewers: Lessons from a Wizard-of-Oz Field Study.. In IUI Workshops.
A.1 Study 1 Method Details: Data Collection, Inclusion Criteria, and Manual Coding Procedures
We collected academic papers by searching three large online indices: Web of Science $^{10}$ , Engineering Village $^{11}$ , and ACM Digital Library $^{12}$ . Because this is a rapidly growing field and some relevant papers may not be officially published, we also searched arxiv.org $^{13}$  for e-print papers. In addition, we included relevant papers that we encountered in previous research activities and snowballed relevant references (for example, we included papers from the workshops of Hum-eval $^{14}$  and HCI-NLP $^{15}$ ).
The specific keywords we used to query the papers include: 1) A word or phrase about AI: AI OR "artificial intelligence" OR machine OR model; 2) A word or phrase related to human-AI collaboration: collaborate OR interact OR assist OR co-author OR co-write OR co-work OR in-the-loop OR co-create OR "human feedback" OR "human-centered"; and 3) A word or phrase about the task: "text summarization" OR "document summarization" OR "article summarization" OR "text generation" OR "document generation" OR "article generation" OR "caption generation" OR "machine translation" OR "style transfer". For each keyword, we searched for all of its forms and/or tenses.
The comprehensive search resulted in 692 papers. After a period of team discussion and iteration, we developed the criteria of the papers that
we would like to include in our analysis: First, the paper needed to directly contribute to the problem space of text-to-text generation tasks. Papers about image/video caption generation, speech recognition, and speech-based machine translation were excluded. Second, the paper had to contribute an interface, workflow, or user study that involve human-AI collaboration. For this reason, survey papers were excluded. In addition, because our goal was to identify types of human-AI collaboration that were not obvious previously, we intentionally excluded papers that in which the only human-AI collaboration is human evaluation for model generated final results or human generating training data.
The first author and two other authors then independently coded a random sample of 150 papers out of the 692 collected papers for inclusion or exclusion. Specifically, the first author coded all 150 papers, and the other two researchers coded 75 respectively. They reached a high inter-rater reliability by comparing the last two researchers' coding with the first authors' respectively (average Cohen's Kappa  $= 0.9$ ), validating the inclusion coding strategy. The first author then coded the remaining 542 papers for inclusion or exclusion. This led to 106 papers that met our inclusion criteria. Due to the rapid advancement in natural language generation in the recent ten years (Dong et al., 2021), we believe insights from recent papers would be most relevant to the design and development of future systems and research. Therefore, we excluded papers published earlier than 2011. All these procedures resulted in a final collection of 70 papers selected for analysis.
Finally, we analyzed the 70 papers following the procedures of thematic analysis (Guest et al., 2011), a manual coding method in qualitative research with which researchers iteratively develop themes from qualitative data. The first author conducted two rounds of open thematic coding on the papers, focusing on aspects such as human actions, collaboration goals, and problem domains. During this process, the team discussed, iterated and refined the themes.
A.2 Demographic Details of Study 2 Participants
The demographic details of Study 2 participants can be found in Table 2.
10https://clarivate.com/   
webofsciencegroup/solutions/   
web-of-science/   
11https://www.engineeringvillage.com/ 12https://dl.acm.org/ 13https://arxiv.org/ 14https://humeval.github.io/ 15https://aclanthology.org/2021.   
hcinlp-1.0/
ID	Gender	Age	Occupation	Education	Experience:summa-rization	Experience:editing	Familiarity:government bills	Familiarity:online news	Familiarity:Reddit posts
P1	M	50-59	Newspaper writer	Bachelor	6	6	6	6	5
P2	M	20-29	Student	Bachelor	6	6	6	6	6
P3	M	30-39	Student	Bachelor	5	7	5	7	7
P4	F	60-69	Freelance editor	Bachelor	5	7	5	7	5
P5	F	30-39	Freelance editor	Doctorate	7	7	5	7	5
P6	F	30-39	Project manager	Master	6	6	5	7	7
P7	M	30-39	Freelancer editor	Bachelor	5	4	3	6	5
P8	F	30-39	Freelance writer	Master	7	7	1	6	1
P9	F	30-39	Marketing consultant	Master	5	7	3	7	7
P10	F	20-29	Student	Bachelor	7	5	2	6	6
P11	F	50-59	Publicist	Bachelor	7	7	4	7	7
P12	M	60-69	Artist	Bachelor	6	6	1	6	1
P13	F	30-39	Freelance writer	Bachelor	6	7	2	7	5
P14	M	40-49	Engineer	Bachelor	5	7	1	7	7
P15	F	20-29	Student	High school	7	7	6	7	7
P16	F	30-39	Student	Bachelor	7	7	7	7	7
Table 2: Demographic information of Study 2 participants. All the information are self-reported by the participants. All the participants were based in the United States. Column "Experience: summarization" reports their answers to the question "rate the following statement: 'I am experienced in text summarization' on a scale of 1 to 7, with 1 being least experienced and 7 being most experienced." The other columns on experience or familiarity reports their answers to questions in the same format.
A.3 Prototype Interfaces Developed for Study 2 Interviews
Figure 3-7 show the screenshots of prototype interfaces explained in §3.2. All interfaces except Writing with Model Assistance contain the same original text (a news article from the articles used in the warm-up activity) that needs to be summarized on the left. The representations of different types of human-AI interactions are on the right side of the interface.
A.4 Study 2 Additional Findings: User Expectation and Needs in Writing Summaries
Summarizing with a specific audience in mind. Many participants had a personal template for writing summaries that they learned in school or work. For example, P6 always checked for "the who, the what, the where, and the why." P1 looked for "the who—Who is it about? What was it about? And then what was the outcome?" and stated that they used the same strategy in all of the summarization tasks in Part 1.
Participants expressed the need to know the target audience of the summary, so that they could determine what kind of information would be useful to them. As P5 said: "If I don't know who really is my audience in writing these summaries, I don't know what detail would matter to them." They would also like to customize the summary to suit the needs of different audiences, especially when the original article was less factual and had more room for interpretation, such as the Reddit posts: "your audiences is going to determine what's important to put in the summary. An attorney is going to perhaps want different information then your common Joe out on the streets. That affects of how something is summarized, you know, because there's always a choice." (P12)
Support on background knowledge. Participants reported that they faced many challenges in terms of comprehending the original article and writing the summaries. When reading the original articles, many were hindered by the lack of background knowledge. In P10's words, "it would
471 14
Figure 3: The interface for Guiding Model Output. Users can change the desired summary length and style (formal or informal) using sliders and highlight parts of the original text that they want to include in the summary. Users can press the "Generate" button to get the "AI-generated" summary based on their inputs.
SumAssist
Figure 4: The interface for Selecting or Rating Model Output. Users can chose the final product from three "AI-generated" candidate summaries.
have helped me to have someone explain what that situation was because I had a very hard time understanding the context behind that situation." Related to this was the difficulty in understanding jargon specific to a domain that the participants were unfamiliar with. For example, P13 is unfamiliar with the legislative jargon in the Government Bills but considered they "are really essential for
being able to understand kind of the larger overall picture of the text." P7 shared that the news article about health website contained technical jargon that "slows down the process and makes it harder." Therefore, participants hope to get support for the lack of background knowledge. P15 imagined "an in-text dictionary support for explaining jargon.
Figure 5: The interface for Post-editing. Users see an "AI-generated" summary in the text box that they can hypothetically edit.
Figure 6: The interface for Interactive Editing. Users see an "AI-generated" summary in the text box. They can use the drop-down menu to change certain words in the first sentence. They can then press "Predict" to request the model to update the rest of the summary based on those edits.
Summarizing formal and informal texts. Participants found summarizing different genres of original text challenging in different ways. Participants who thought comprehending the Government Bills challenging described them as "super dry, super repetitive," (P3) and "not designed as an article." (P2) It was difficult to identify the crucial information from a Bill since everything seems
important: "they want you to take all of it away. (P13) On the other hand, although easy to understand, Reddit posts posed a different challenge due to their informal, unfocused style. For example, in P1's words, the Reddit posts were "more of a narrative... there's no formula to (summarize) them." Furthermore, many found it challenging to compose the summaries in the same personalized style
Figure 7: The interface for Writing with Model Assistance. In a Google Doc, users can see the original article on the top and they can write their summary under the section "Write your summary here:". First, the user types a sentence for their summary, then a Bot (played by a researcher who log in with the "SumAssist Bot account") will insert the next sentence in gray fonts. The Bot will also insert comments on words in the user written sentence and suggest them to make changes.
as the original poster: "I'm not really familiar with summarizing something in somebody else's voice or in somebody else's tone." (P3) Contrast to the summarization for the Government Bills, described as having "only one outcome", and being "factual," (P7) and "more direct, more to the point," (P10) Reddit posts come with "an element of opinion to it," (P7) and therefore could be summarized through different angles. Some thought they should write in the same perspective as the original poster and preserve the opinions, as P9 said, "I would make sure I picked up what they were trying to, to lay out." others considered a summary as a neutral representation of information and thus they had to "pull the opinions out and give the general message." (P14)
A.5 Interview protocol and interaction instruction.
Please see the interview protocol and interaction instruction on the next page. Since we need to protect the privacy of our participants, we are not able to share the full interview transcripts.
474 17
A.5 Interview Protocol
(§3.3 contains interaction instruction and questions for each of the five prototypes)
First of all thank you for spending the time and talking with me today. We are researchers studying how we can help people summarize text. Your participation in this study could help us understand this topic, and contribute to scientific knowledge - ultimately we may publish a research paper about insights from your participation in our study.
2. Consent
This interview will last about 90 minutes. I would like to talk to you about your experience with text summarization. You should have already finished the warm-up for the study, and I will ask you questions about it. I will also have you try out some new tools that we developed to support text summarization and ask you questions along the way.
There are no right or wrong answers to my questions. I want to hear your personal experiences and opinions. You also absolutely can decline to answer any of the questions that I ask. At any point if you feel uncomfortable or need to pause or take a break, just let me know.
Everything that is said in this session will be completely confidential and used for research purposes only. Your responses may be published as quotes in a future academic research paper, but no identifiers will tie your username or any other identifiable factors to those responses.