SlowGuess/NAACL_2021_OCR huggingface dataset:
We developed a web application, enabling session collection with real users, that follows the INTSUMM schema described in §3: for an input document set, it first presents an initial summary, and then iteratively outputs a summary expansion response per given user request. Specifically, our application supports interactive requests in the form of textual queries from user free-text, summary span highlights and system suggested queries. A system response aims to simultaneously maximize relevance to the query and salience, while refraining from repeating previously presented contents.
An initial version of the application was assessed via a small-scale user study of 10 users, with an SUS questionnaire (Brooke, 1996) and the think-aloud protocol (Lewis, 1982) for feedback. Figure 1 displays the improved web application, on the topic "Native American Challenges". The left screenshot shows the initial summary with user rating [R.1] in [1], an example of a free-text query in the query box [2] and the list of suggested queries in [3]. The right screenshot shows the response to the query entered in the first screenshot. [4] reiterates the last submitted query, with the system response and user rating [R.2] in [5]. The last query can also be repeated via a button [6], to obtain additional information on that query. Users can highlight a span from the presented summary, to be automatically pasted to the query box. Initial summaries and expansions are extractive and in bullet-style.
In accordance to this interaction flow presented, we implemented two back-end algorithm schemes, denoted  $S_{1}$  and  $S_{2}$ , to demonstrate comparison of two INTSUMM systems via our evaluation framework. Each implementation consists of three components: (1) the initial summary generation, (2) the query-response generation and (3) extraction of suggested queries from the source documents. All system outputs must comply to required interaction latency standards (Anderson, 2020; Attig et al., 2017), e.g., a few seconds for the initial summary and a few hundred milliseconds for a query response. While we experimented with some more advanced techniques for MDS generation (e.g., Christensen et al., 2013; Yasunaga et al., 2017), sentence representation (Reimers and Gurevych, 2019) and sentence similarity (Zhang* et al., 2020), we found that these are not practical for incorpo
<sup>3</sup>https://www.mturk.com
ration within the interactive low-latency setting, or that they could not handle the relatively large document set inputs. Instead, we developed the two back-end schemas described next (with further details in Appendix A).
$S_{1}$  runs a sentence clustering initial summary algorithm. Query-responses are generated in MMR-style (Goldstein et al., 2000a) based on semantic similarity between query and sentences. Suggested queries are frequent bigrams and trigrams.  $S_{2}$  uses TextRank (Mihalcea and Tarau, 2004) for both the initial summary and suggested queries, and a query-response generation approach combining semantic and lexical similarity between query and sentences.
The two systems enable experimentation on our evaluation framework and, as we show, demonstrates its viability. Moreover, as apparent in our experimental results (§5.4 and user feedback in Appendix C), users attest to the real-world usefulness of these systems. Using our framework, including the baseline systems, future work can develop and examine more advanced methods for INTSUMM, accounting for the latency and input-size challenges.
5.2 Crowd Experimental Setup
Following our controlled session collection procedure from §4, we released the trap task in AMT and found 48 of 231 workers qualified for the second stage, out of which 25 accepted. 10 workers passed the training stage, from which we recruited 8 highly qualified ones. For the third stage, we collected sessions for 20 topics from DUC 2006, on  $S_{1}$  and  $S_{2}$ . Each worker could explore 10 different topics on each system, amounting to 160 possible sessions of which 153 were completed (with at least 3 sessions per combination of topic and system). Since  $S_{1}$  and  $S_{2}$  share a common frontend application, users were unaware of which system they are exploring on, and the order was randomized. A minimum exploration time constraint of 150 seconds was set. Initial summaries were ≥ 75 tokens (average of 85) and responses were two sentences long.
The full controlled crowdsourcing process took one author-work-week, and cost \(370. In comparison, "wild" crowdsourcing described in \(\S 4\) required a couple days' work and \(\$ 240\) (achieving, as discussed, inferior results), and running a noncrowdsourced user-study of the same magnitude would likely require more work time, and cost an estimated \(\$ 480\) (32 net hours of 16 workers at a
commonly acceptable $15 hourly wage). Furthermore, the results of a user study would not necessarily be of higher quality (Zuccon et al., 2013). To our judgement, the controlled crowdworkers are more suitable since they fathom the task before choosing to complete it. In a user study, workers are often unaware of the task before commencing, and may not be fully qualified for or desiring of it.
5.3 Simulated Bounds
In addition to real user experiments, we simulate each of our two systems on scripted query lists. Simulated sessions provide a means for quick development cycles and quality estimation.
The first of two query lists,  $L^{\mathrm{Sug}}$ , is constructed fully automatically: it consists of the top-10 ordered phrases in the system's suggested queries component per topic. This mimics a "lower bound" user who adopts the simplest strategy, namely, clicking the suggested queries in order without using judgment even to choose among these queries.
The second list,  $L^{\mathrm{Oracle}}$ , consists of 10 randomly chosen crowdsourced summary content units (SCUs) (Shapira et al., 2019) for each of the topics. Since the SCUs were extracted from the reference summaries of the corresponding topics, they mimic a user who searches for the exact information required to maximize similarity to the same reference summaries which we then evaluate against. While this is not necessarily the optimal query list due to the randomized sampling of SCUs for queries, we consider it our (non-strict) "upper bound" for the sake of experimentation.
The two "bounds" are relative to the system on which the simulations are carried on. Also, for fair comparison to real sessions, the simulation initial summary and response lengths are similarly set at  $\geq 75$  words and two sentences respectively.
We next present the results attained on the 153 sessions collected (§5.2), with the purpose of analyzing our full evaluation framework. We gain an understanding on the consistency between automatic and human measurement, and on the comprehensiveness of the full set of measures.
Figure 3 presents the average recall-curves and corresponding  $[P.1]$  averaged AUC scores of the  $S_{1}$  bounds (§5.3) and of the user sessions on  $S_{1}$  and  $S_{2}$ . AUC is computed between word-lengths 105 to 333 (the maximum intersection of all sessions). Table 2 shows  $[P.2]$  averaged ROUGE-1
Figure 3: The average recall-curves, along with corresponding AUC scores (unrelated to the x-axis) and their confidence intervals  $(\geq 95\%)$ , of the upper and lower bound sessions and of user sessions of the two systems.
Sessions	S@L 150	S@L 250	S@L 350
\( S_1 \) \( L^{Oracle} \)	.328 (±.012)	.400 (±.010)	.414 (±.012)
\( S_1 \) Real	.324 (±.010)	.382 (±.011)	.392 (±.012)
\( S_1 \) \( L^{Sug} \)	.319 (±.011)	.375 (±.012)	.382 (±.011)
\( S_2 \) \( L^{Oracle} \)	.333 (±.011)	.402 (±.013)	.412 (±.014)
\( S_2 \) Real	.321 (±.009)	.379 (±.013)	.388 (±.012)
\( S_2 \) \( L^{Sug} \)	.320 (±.011)	.374 (±.014)	.386 (±.013)
based Score@Length. Scores rank consistently on ROUGE-2, ROUGE-L and ROUGE-SU (see Appendix C).
It is evident from Figure 3 and Table 2 that the results on collected sessions indeed fall between the two bounds in all measures. This demonstrates the effectiveness of interactive summarization, even when using relatively simple algorithms: the algorithm enables fast information processing of input texts, and users effectively direct the algorithm to salient areas.
Additionally, the scores of  $S_{1}$  and  $S_{2}$  are close, providing no significant insights when comparing these two systems, which is surprising due to their distinct implementations. Manually reviewing the results, we were convinced that the systems indeed happen to perform at similar quality overall. However, when assessing the systems' separate components and inspecting user-provided ratings, we gain awareness of some interesting distinctions.
Table 3 shows a trend of consistency between ROUGE scores on each separate component and
Table 2: ROUGE-1  $F_{1}$ -based average Score@Length of simulated sessions vs. real user sessions. Scores at 350 words are approximate as few sessions were shorter. Scores rank consistently on ROUGE-2, -L and -SU. Intervals at  $\geq 95\%$  confidence.
Metric	S1	S2
Initial	Initial summary ROUGE-1 [R.1] Initial summary rating	0.232	0.225
3.89 (0.98)	3.71 (1.0)
Query	Que-Resp LOracle ROUGE-1 [R.2] Avg. response rating [R.3] Query responsiveness	0.156	0.161
3.17 (1.32)	3.35 (1.28)
3.61 (1.02)	3.83 (1.03)
Overall	[R.4a] System effectiveness [R.4a] System ease of use [R.4] System UMUX-Lite	3.81 (1.0)	4.05 (0.80)
4.51 (0.71)	4.63 (0.62)
74.2 (12.5)	77.1 (10.3)
Table 3: Average and (StD) scores of metrics comparing  $S_{1}$  and  $S_{2}$ . Users appear to be more satisfied with  $S_{2}$  overall, likely due to the query response component.
the ratings provided by the users. The initial summaries' ROUGE-1  $F_{1}$  scores are computed against the reference summaries, with a slight advantage for  $S_{1}$  over  $S_{2}$  - similar to the users' initial summary ratings. For the query-response component, we compute the average ROUGE  $F_{1}$  score of the independent responses to the queries in  $L^{\mathrm{Oracle}}$ , against the reference summaries. Again, user ratings reflect a similar trend that the query-response component of  $S_{2}$  slightly outscores that of  $S_{1}$ . Overall we see that  $S_{1}$  provides a better initial summary while  $S_{2}$  handles queries better. Also, users tend to be more satisfied by  $S_{2}$ , likely due to its ability to respond better to queries. This claim is evident from the positive correlation between [R.3] and [R.4a],  $r = 0.68, p < 0.001$  in  $S_{1}$  and  $r = 0.63, p < 0.001$  in  $S_{2}$ . In terms of absolute UMUX-Lite scores [R.4], 68 is considered average, and above 80 is considered excellent, meaning both  $S_{1}$  and  $S_{2}$  got high usability scores, with a preference for  $S_{2}$ .
An additional analysis finds a positive correlation between per-iteration response [R.2] scores and the relative per-iteration increase in ROUGE recall (e.g. for ROUGE-1  $r = 0.36$ ,  $p < 0.001$  in  $S_{1}$  and  $r = 0.33$ ,  $p < 0.001$  in  $S_{2}$ ), hinting at the credibility of correlation between human ratings and relative increase in ROUGE within sessions.
To conclude, our findings are favorable in terms of the framework's internal consistency of measures and soundness of the computed scores. For a more conclusive appraisal of the full evaluation framework, additional systems are to be run through the process, regardless of the accidental similarity between our two baselines.
We proposed a comprehensive evaluation framework for user-guided expansion-based interactive
summimization - a vital ingredient for the methodological advancement of interactive summarization research which was unaccounted for until now. Our controlled crowdsourcing procedure makes INTSUMM system session collection accessible, scalable and replicable. The evaluation measures in our framework provide a thorough assessment with absolute scores that enable comparison of INTSUMM systems. Our framework provides the means to advance INTSUMM research on system development and improved evaluation. All solutions, including our implemented baseline systems, are publicly available to enable comparison of new INTSUMM systems to ours on any MDS dataset.
In future work, it is worthwhile to separately assess the effectiveness of individual interaction modes, including ones incorporated in our implementation and others, e.g., full questions input by users. These would require further experimentation, additional evaluation metrics, and the possible use of datasets from tasks other than MDS. Within our expansion-based framework, we can consider additional measures of textual consistency, coherence, and relevance of responses to queries. We may also test additional approaches for summarization: e.g., abstractive summarization for flexible synthetic summary generation, requiring further evaluation of factuality and truthfulness. Beyond our framework, that targets objective quality, INTSUMM systems should also be evaluated according to their compatibility with personalized, subjective use.
User-study. Our system-testing user-study (mentioned in §5.1) was conducted on a university campus, and students within different age groups and from different backgrounds were recruited through a social media group for hiring for experiments and user studies. We required a high level of English for participation. People were accepted until the required amount of participants (10) was reached, without any targeted filtering. An individual study lasted around 30 minutes for a payment of around $10.
Crowdsourcing. There were several rounds of crowdsourcing, with varying tasks. Due to the need for fluent English speaking workers, a location filter was set on the AMT platform for English (as primary language) speaking countries. At least one of the authors tested each task before its release to estimate worst-case task completion duration. The
payment was then set according to  $9 per hour for the estimated required time. In practice, almost all tasks were completed in less than the time estimated, and payment was well above$ 9 per hour. Very few assignments were rejected in cases of clear insencereness (unreasonably fast submission or senseless behavior).
Dataset usage. As pointed out throughout the paper, the DUC 2006 dataset was utilized. It was obtained through the required means on the DUC website (duc.nist.gov). There was no possibility to reconstruct the dataset (document sets and reference summaries) within any of the conducted user study and crowdsourcing tasks.
Application. Our INTSUMM systems' outputs are extracts from the input document sets. As described in Appendix A, the algorithms for initial summary and query-response generation do not contain any intentional biasing.
The intended purpose of any INTSUMM system is to allow readers to make sense of large bodies of text through assisted exploration. Future work may open the door to more personalized algorithms and abstractive outputs. This would require extra care in making sure systems are ethically sound by adding targeted evaluation measures.
Compute time. As emphasized in the paper, INTSUMM systems require low latency and are hence relatively computationally cheap. During our research we ran some algorithms, to test for our systems, that required up to several hours of compute time per run, on a standard server.
We would like to thank Guiseppe Carenini for his helpful advice, and the anonymous reviewers for their constructive comments. This work was supported in part by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grants DA 1600/1-1 and GU 798/17-1); by the BIU Center for Research in Applied Cryptography and Cyber Security in conjunction with the Israel National Cyber Bureau in the Prime Minister's Office; by the Israel Science Foundation (grants 1157/16 and 1951/17); by a grant from the Israel Ministry of Science and Technology; by the NSF-CAREER Award #1846185; and by a Microsoft PhD Fellowship.
Shaun Anderson. 2020. How fast should a website load? https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds. Accessed: 2020-05-19.
Christiane Attig, Nadine Rauh, Thomas Franke, and Josef F Krems. 2017. System latency guidelines then and now-is zero latency really considered necessary? In International Conference on Engineering Psychology and Cognitive Ergonomics, pages 3-14. Springer.
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2014. Query-chain focused summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 913-922, Baltimore, Maryland. Association for Computational Linguistics.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704.
Pia Borlund. 2003. The iir evaluation model: a framework for evaluation of interactive information retrieval systems. Information research, 8(3):8-3.
John Brooke. 1996. Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4-7.
Ziqiang Cao, Wenjie Li, Sujian Li, Furu Wei, and Yanran Li. 2016. Attsum: Joint learning of focusing and summarization with neural attention. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 547-556.
Janara Christensen, Mausam, Stephen Soderland, and Oren Etzioni. 2013. Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1163-1173, Atlanta, Georgia. Association for Computational Linguistics.
Janara Christensen, Stephen Soderland, Gagan Bansal, and Mausam. 2014. Hierarchical summarization: Scaling up multi-document summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 902-912, Baltimore, Maryland. Association for Computational Linguistics.
Philipp Christmann, Rishiraj Saha Roy, Abdalghani Abujabal, Jyotsna Singh, and Gerhard Weikum. 2019. Look before you hop: Conversational question answering over knowledge graphs using judicious context expansion. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 729-738.
Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1-12.
Hoa Trang Dang. 2006. Overview of duc 2006. In Document Understanding Conference.
Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. In TAC.
Hal Daumé III and Daniel Marcu. 2006. Bayesian query-focused summarization. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 305-312. Association for Computational Linguistics.
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query-focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 961-964.
Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Jamie Callan. 2000a. Creating and evaluating multi-document sentence extract summaries. In Proceedings of the ninth international conference on Information and knowledge management, pages 165-172.
Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Mark Kantrowitz. 2000b. Multi-document summarization by sentence extraction. In Proceedings of the 2000 NAACL-ANLP Workshop on Automatic summarization, pages 40-48. Association for Computational Linguistics.
Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362-370.
Abram Handler and Brendan O'Connor. 2017. Rookie: A unique approach for exploring news archives. In Proceedings of Data Science + Journalism workshop at KDD, Halifax, Nova Scotia, Canada. Association for Computing Machinery.
Chathra Hendahewa and Chirag Shah. 2017. Evaluating user search trails in exploratory search tasks. Information Processing & Management, 53(4):905-922.
Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 712-721, Gothenburg, Sweden. Association for Computational Linguistics.
Peter Ingwersen. 1992. Information retrieval interaction, volume 246. Taylor Graham London.
Diane Kelly and Jimmy Lin. 2007. Overview of the trec 2006 ciqa task. In ACM SIGIR Forum, volume 41, pages 107-116. ACM New York, NY, USA.
Anton Leuski, Chin-Yew Lin, and Eduard Hovy. 2003. iNeATS: Interactive multi-document summarization. In The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pages 125-128, Sapporo, Japan. Association for Computational Linguistics.
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710.
Clayton Lewis. 1982. "Using the" thinking-aloud" method in cognitive interface design. IBM TJ Watson Research Center Yorktown Heights, NY.
James R Lewis, Brian S Utesch, and Deborah E Ma-her. 2013. Umux-lite: when there's no time for the sus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2099-2102.
Wenjie Li, Furu Wei, Qin Lu, and Yanxiang He. 2008. Pnr2: ranking sentences with positive and negative reinforcement for query-oriented update summarization. In Proceedings of the 22nd international conference on computational linguistics (Coling 2008), pages 489-496.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jimmy Lin. 2007. Is question answering better than information retrieval? towards a task-based evaluation framework for question series. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 212-219.
Jimmy Lin and Dina Demner-Fushman. 2006. Will pyramids built of nuggets topple over? In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 383-390. Association for Computational Linguistics.
Jimmy Lin, Nitin Madnani, and Bonnie Dorr. 2010. Putting the user in the loop: Interactive maximal marginal relevance for query-focused summarization. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 305-308, Los Angeles, California. Association for Computational Linguistics.