SlowGuess/NAACL_2021_OCR huggingface dataset:
James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281-297. Oakland, CA, USA.
Gary Marchionini. 2006. Exploratory search: from finding to understanding. Communications of the ACM, 49(4):41-46.
Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2014. Incremental update summarization: Adaptive sentence selection based on prevalence and novelty. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pages 301-310.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404-411.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
NIST. 2014. Document understanding conferences. https://duc.nist.gov/. Accessed: 2020-05-19.
Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, pages 1-9, Montreal, Canada. Association for Computational Linguistics.
Emilie Palagi, Fabien Gandon, Alain Giboin, and Raphaël Troncy. 2017. A survey of definitions and models of exploratory search. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics, pages 3-8.
Dragomir R Radev, Hongyan Jing, Małgorzata Stys, and Daniel Tam. 2004. Centroid-based summarization of multiple documents. Information Processing & Management, 40(6):919-938.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Controlled crowdsourcing for high-quality qa-srl annotation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. To Appear.
Gaetano Rossiello, Pierpaolo Basile, and Giovanni Semeraro. 2017. Centroid-based text summarization through compositionality of word embeddings. In Proceedings of the MultiLing 2017 Workshop on Summarization and Summary Evaluation Across Source Types and Genres, pages 12-21, Valencia, Spain. Association for Computational Linguistics.
Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 682-687, Minneapolis, Minnesota. Association for Computational Linguistics.
Ori Shapira, David Gabay, Hadar Ronen, Judit Bar-Ilan, Yael Amsterdamer, Ani Nenkova, and Ido Dagan. 2018. Evaluating multiple system summary lengths: A case study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 774-778, Brussels, Belgium. Association for Computational Linguistics.
Ori Shapira, Hadar Ronen, Meni Adler, Yael Amsterdamer, Judit Bar-Ilan, and Ido Dagan. 2017. Interactive abstractive summarization for event news tweets. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 109-114, Copenhagen, Denmark. Association for Computational Linguistics.
Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77-82.
Dingding Wang and Tao Li. 2010. Document update summarization using incremental hierarchical clustering. In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 279-288.
Patrick Wessa. 2020. Free statistics software, office for research development and education, version 1.2.1. https://www.wessa.net/. Accessed: 2020-05-19.
Ryen W White, Gary Marchionini, and Gheorghe Muresan. 2008. Evaluating exploratory search systems. Information Processing and Management, 44(2):433.
Ryen W White and Resa A Roth. 2009. Exploratory search: Beyond the query-response paradigm. 3. Morgan & Claypool Publishers.
Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52.
Rui Yan, Jian-Yun Nie, and Xiaoming Li. 2011. Summarize what you are interested in: An optimization framework for interactive personalized summarization. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1342-1351, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462, Vancouver, Canada. Association for Computational Linguistics.
Wenpeng Yin and Yulong Pei. 2015. Optimizing sentence modeling and selection for document summarization. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with bert.* In International Conference on Learning Representations.
Lin Zhao, Lide Wu, and Xuanjing Huang. 2009. Using query expansion in graph-based approach for query-focused multi-document summarization. Information processing & management, 45(1):35-41.
Markus Zopf. 2018. Estimating summary quality with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1687-1696, New Orleans, Louisiana. Association for Computational Linguistics.
Markus Zopf, Eneldo Loza Mencia, and Johannes Furnkranz. 2016. Sequential clustering and contextual importance measures for incremental update summarization. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1071-1082.
Guido Zuccon, Teerapong Leelanupab, Stewart Whiting, Emine Yilmaz, Joemon M Jose, and Leif Azzopardi. 2013. Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Information retrieval, 16(2):267-305.
A INTSUMM System Implementation Details
According to the INTSUMM schema described, we implemented two back-end baseline systems, sharing a front-end application. The user interactions supported are textual queries and the responses aim to maximize relevance to the query, similarly to query-focused static summarization, while refraining from repeating previously presented contents, somewhat similarly to update summarization. Specifically, the implementations support queries from free-text, highlights and system suggestions. Both systems output extractive bullet-style initial summaries and summary expansion responses.
A.1 Algorithm Variants
Each of our back-end INTSUMM implementations consists of three main components, as follows.
A.1.1 Initial Summary
We first consider the generation of the initial summary  $\sigma_0$ . Our experimentation with some classic and modern MDS implementations have indicated that most do not meet the interactivity response-time requirements, and hence we provide two implementations for this component based on standard extractive MDS methods.
The first algorithm denoted  $I^{\mathrm{CL}}$ , is clustering-based with ideas inspired by Rossiello et al. (2017) and Hong and Nenkova (2014). All sentences in the document set are separately assigned a representation by averaging the 300-dimensional word2vec (w2v) embeddings (Mikolov et al., 2013) within each sentence. For this we use the SpaCy library. Each vector's dimension is reduced to 20 with PCA (Wold et al., 1987), and then sentence vectors are clustered to 30 components with k-means (MacQueen et al., 1967). PCA and k-means are implemented with sklearn. We then order clusters by size, and select a representing sentence starting from the largest cluster, and continuing to following clusters, until the summary word-limit has reached. A cluster is skipped if its representing sentence is too similar (cosine similarity of 0.95) to previously selected sentences. The sentence selected to represent a cluster is the one whose words are on average most frequent within the full document set. On a standard server, the implementation
can generate a summary of 25 news articles from a common MDS dataset in a few seconds (2 to 10 seconds).
Hyper-parameters: For reduced sentence-representative vector dimension (20) we tested several values between 10 and 100. For number of sentence clusters (30) we tested values between 10 and 50, and for similarity threshold (0.95) we tested several options within the 0 to 1 range. The hyperparameters were lightly adjusted by computing ROUGE scores of some outputs against reference summaries, ensuring fast processing, and rational eyeballing.
We also experimented with Sentence-BERT (Reimers and Gurevych, 2019) representations, drawing on the 'roberta-base-nli-stsb-mean-tokens' model in place of the word2vec-based ones, however this did not improve output summaries, and, more importantly, had high latency (tens of seconds to a few minutes).
The second algorithm is TextRank (Mihalcea and Tarau, 2004), denoted  $I^{\mathrm{TR}}$ , coded with the syntaxtrank pipeline component in the SpaCy library. The implementation supports an argument with which we can limit the initial summary word-length. It is slightly slower than  $I^{\mathrm{CL}}$ , but runs within approximately the same time range.
A.1.2 Query Response Generation
This component computes similarity of a given query to all sentences not yet presented in previous iterations, and outputs a few of the best matches. Process-time is up to a few hundred milliseconds.
The first variant  $Q^{\mathrm{SEM}}$ , utilizes the semantic (w2v-based) sentence representations prepared in the initialization process (either averaged word2vec representation or Sentence-BERT). It computes the cosine similarity between the query's representation and all unused source sentences. In MMR-style (Goldstein et al., 2000a), the sentences most similar to the query which pass a dissimilarity threshold from already selected sentences are selected. An MMR dissimilarity of  $\leq 0.05$  gave the best results (on simulated sessions).
The second variant  $Q^{\mathrm{LEX}}$ , additionally considers lexical similarity by scoring the similarity of a query and a sentence as the product of the
4https://spacy.io/usage/vectors-similarity5https://scikit-learn.org
https://github.com/UKPLab/ sentence-transformers https://spacy.io/universe/project/ spacy-pytextrank
w2v similarity with three ROUGE-precision scores. I.e., for query  $q$  and sentence  $s$ ,  $\mathrm{sim}(q,s) = (\mathrm{cosine}(w2v(q),w2v(s)) + 1)*(R1_p(q,s) + 1)*(R2_p(q,s) + 1)*(RL_p(q,s) + 1)$ , where  $R1_{p}, R2_{p}$  and  $RL_{p}$  are ROUGE-1, ROUGE-2 and ROUGE-L precision respectively. The highest scoring unused sentences are output. Compared to the first variant, this method yields lexically-stricter search results.
Finally, we also experimented with BERTScore (Zhang* et al., 2020) as the similarity score between the query and each of the relevant sentences. Here too, the high latency makes this approach impractical for our purposes.
A.1.3 Suggested Queries
Our systems support an interaction mode enabling information expansion by clicking a system-suggested query from a list.
The first approach for preparing the list,  $\mathrm{Sug^{FREQ}}$ , is selecting the most frequent bigrams and trigrams within the source documents, disregarding stop words. A trigram is preferred when it contains a bigram with the same frequency. An n-gram with a Levenshtein distance (Levenshtein, 1966) of less than 2 from an already selected n-gram is skipped.
The second approach  $\mathrm{Sug}^{\mathrm{TR}}$ , utilizes the top-ranked phrases extracted from the TextRank algorithm (as part of the graph-based extractive summarization procedure).
A.1.4 Overall Systems
For the purpose of applying our evaluation framework, we picked two combinations of the above three components. (Assessing additional combinations is out of the scope of this paper.) The first combination, denoted System  $S_{1}$ , is comprised of  $I^{\mathrm{CL}}$ ,  $Q^{\mathrm{SEM}}$  and  $\mathrm{Sug}^{\mathrm{FREQ}}$ . The second combination, System  $S_{2}$ , consists of  $I^{\mathrm{TR}}$ ,  $Q^{\mathrm{LEX}}$  and  $\mathrm{Sug}^{\mathrm{TR}}$ .
A.2 Web Application
The front-end web application, which communicates with an INTSUMM system in the back-end (in our case  $S_{1}$  or  $S_{2}$ ), is seen in Figure 1 in the main paper. Some further details regarding the application:
A previous version of the web application included a button for additional general information, which was originally supported by System  $S_{1}$ . This sent an empty query to the system, for which the
sentence from the next unused cluster in the  $I^{\mathrm{CL}}$  algorithm was returned (rotating to the first cluster if all clusters have been used, and taking the next best unused representing sentence). In our preliminary user study and crowdsourcing experiments, we found that this feature was mainly a distraction and induced exploration laziness.
The Web application is implemented in HTML, CSS and Javascript, and the backend in Python. The app communicates with the backend over standard HTTP Post requests in JSON format.
A.3 Server Specifications
We ran experiments, and run our INTSUMM systems on an Intel Xeon CPU E5-2670 v3 @ 2.30GHz server with 50GB RAM running CentOS Linux 7. Run times are similar on an Intel Core i7-6600 CPU @ 2.60GHz laptop with 16GB RAM running Windows 10. We noticed some differences in the  $I^{\mathrm{CL}}$  algorithm's outputs when run on the different hardware (consistent within but diverging between), even though the software environments were identical. This is likely due to a different 'random' implementations in the two settings.
To check if the query-response component with BERTScore would be more practical on a GPU server, we tested it on an Nvidia Titan X GPU server. A single query response took 40-50 seconds to compute.
B Controlled Crowdsourcing Details
The controlled crowdsourcing protocol finds high quality users for the collection of system sessions. The use-case we enforced was producing an informative summary draft text which a journalist could use to best produce an overview of the given topic for the general public. This use case attempts to follow the informational goal of the reference summaries, which are practically generically written.
B.1 Trap Task
This task, consisting of three questions, aims to discover workers with an ability to apprehend salient information within text. It was implemented standardly within the Amazon Mechanical Turk<sup>8</sup> platform.
Question 1. This question tests apprehension of the notion of a general summary, by asking the user to choose a sentence that would be best to
include in an overview of some topic, given the topic name and four relevant sentences of varying informativeness.
For our journalistic scenario, we randomly selected 10 topics from the DUC<sup>9</sup> 2007 MDS dataset that include Pyramid SCUs (Nenkova and Passonneau, 2004), and for each topic manually chose one SCU with a weight of 4, and three SCUs with a weight of 1. The SCU with the higher weight would be expected to be the more appropriate choice for the journalistic overview, since all four reference summaries of the topic include it.
Question 2. This question simulates a scenario closer to interactive summarization. A three-sentence "initial summary" of the same topic (as Question 1) is presented, and the worker is asked to choose the best of three possible interaction possibilities that would provide more information on the topic from a theoretical search tool. We presented three pairs of queries of varying relevance and informativeness as interaction possibilities.
The "initial summary" is the lead-three sentences of one of the reference summaries of the topic. We prepared the three choices of two queries based on salient phrases manually found within the reference summaries. One pair of queries is worthy, one pair is somewhat worthy, and a third pair is unworthy.
Question 3. This question tests attentiveness to the task and creativity by asking users to suggest another interaction (query) to the theoretical search tool. Such an open-ended question requires more thought and hence filters careless or guessing workers.
Task preparation. We ran the 10 tasks internally with research colleagues to find vulnerabilities, and edited some questions accordingly. In addition we recorded the average work time to set a fair task payment on the crowdsourcing platform. In retrospect, the time it took crowd-workers was about two-thirds of the time it took internal workers. Moreover, the time was an additional indication of better workers – those completing the task correctly in shorter time were likely superior.
We paid  $0.50 for each trap task assignment, estimating about 3.5 minutes of work time. Good workers completed the task in 2.5 minutes on average, which should fairly pay only$ 0.30.
Task assessment. The first two questions are automatic filters for insincere workers. A meaningful answer to the third question, assessed manually, serves as a sanity check which we found useful for additional filtering.
The workers passing this phase were contacted via email. The message included an explanation and estimated payment of the subsequent tasks.
B.2 Practice Task
This task is an external question done within an IFrame on Mechanical Turk.
Two practice tasks were prepared from DUC 2006, separate from the 20 used for real session collection. Workers completing both tasks with predominantly relevant queries (checked manually) were asked to continue on to the final task.
We emphasized the use case of preparing a journalistic overview by instructing to "produce an informative summary draft text which a journalist could use to best produce an overview of the topic".
We paid $0.90 for each practice task, estimating 6-7 minutes of work per assignment. Our estimate was about right.
B.3 Evaluation Session Collection Task
As before, this external question task is done within an IFrame on Mechanical Turk.
For the session collection tasks we paid  $0.70 per topic, estimating 5 minutes of work. We promised to give a bonus for good work, to motivate completion of more assignments, and in higher standards. We awarded$ 0.15 to $0.30 bonus according to the quality (assessed manually), per assignment. All sessions were of very high quality, but some made an extra effort and provided comments and feedback.
B.4 Wording of Human Ratings
For our journalistic use-case, the ratings within a session are worded as follows:
- [R.1] "How useful is this for the journalist's generic overview of the topic?"
- [R.2] “How much useful info does this add to the journalist's overview (regardless of how well it matched your query)?”
- [R.3] “During the interactive stage, how well did the responses respond to your queries?”
- [R.4] "As a system for exploring information on a topic,
$9$ https://duc.nist.gov/
-  $[R.4a]$  "its capabilities meet the need to efficiently collect useful information for a journalistic overview."
-  $[R.4b]$  "it is easy to use."
B.5 Wild Crowdsourcing
For quality control, at the end of a session the user filled a questionnaire, in which they mark whether 10 statements are covered in their generated session. Of those statements, five were (separately) crowdsourced summary content units (SCUs) from the topic's reference summaries (Shapira et al., 2019), one of those SCUs was repeated to test for identical markings, two statements were SCUs from another topic, and two statements were the two shortest sentences output by the session. We thus know the answers to 4 statements and have a repeating statement test. Sessions with minimal mistakes could hypothetically be considered sincere.
Systems were evaluated using data from the DUC 2006 MDS dataset. 20 topics were used (all those with Pyramid (Nenkova and Passonneau, 2004) evaluations). These are: D0601, D0603, D0605, D0608, D0614, D0615, D0616, D0617, D0620, D0624, D0627, D0628, D0629, D0630, D0631, D0640, D0643, D0645, D0647, D0650. The practice tasks in the controlled crowdsourcing procedure used topics D0602, D0606. The 10 topics in the trap task are based on document sets with Pyramid evaluations from DUC 2007. These are: D0701A, D0703A, D0704A, D0705A, D0706B, D0707B, D0710C, D0711C, D0714D, D0716D.
Similar to the results on ROUGE-1 in the main paper, Tables 4, 5 and 6 are for metrics ROUGE-2, ROUGE-L and ROUGE-SU respectively.
Sessions	S@L 150	S@L 250	S@L 350
\( S_1 \) \( L^{Oracle} \)	.063 (±.011)	.085 (±.013)	.096 (±.013)
\( S_1 \) Real	.064 (±.010)	.077 (±.010)	.082 (±.010)
\( S_1 \) \( L^{Sug} \)	.065 (±.009)	.078 (±.010)	.082 (±.012)
\( S_2 \) \( L^{Oracle} \)	.067 (±.011)	.085 (±.013)	.094 (±.013)
\( S_2 \) Real	.058 (±.010)	.072 (±.011)	.077 (±.013)
\( S_2 \) \( L^{Sug} \)	.056 (±.010)	.068 (±.011)	.073 (±.011)
Table 4: ROUGE-2  $F_{1}$ -based average scores of simulated sessions vs. controlled crowdsourced sessions. Scores at 350 words are approximate as a few sessions were shorter. Intervals at  $\geq 95\%$  confidence.
Sessions	S@L 150	S@L 250	S@L 350
\( S_1 \) \( L^{Oracle} \)	.270 (±.013)	.328 (±.012)	.333 (±.014)
\( S_1 \) Real	.271 (±.010)	.314 (±.010)	.319 (±.010)
\( S_1 \) \( L^{Sug} \)	.258 (±.010)	.299 (±.011)	.302 (±.011)
\( S_2 \) \( L^{Oracle} \)	.275 (±.012)	.327 (±.015)	.332 (±.015)
\( S_2 \) Real	.270 (±.011)	.313 (±.014)	.315 (±.014)
\( S_2 \) \( L^{Sug} \)	.271 (±.011)	.311 (±.014)	.313 (±.013)
Table 5: ROUGE-L  $F_{1}$ -based average scores of simulated sessions vs. controlled crowdsourced sessions. Scores at 350 words are approximate as a few sessions were shorter. Intervals at  $\geq 95\%$  confidence.
Sessions	S@L 150	S@L 250	S@L 350
\( S_1 \) \( L^{Oracle} \)	.091 (±.008)	.145 (±.008)	.156 (±.011)
\( S_1 \) Real	.090 (±.007)	.137 (±.009)	.145 (±.009)
\( S_1 \) \( L^{Sug} \)	.089 (±.007)	.133 (±.009)	.139 (±.008)
\( S_2 \) \( L^{Oracle} \)	.093 (±.008)	.145 (±.010)	.156 (±.013)
\( S_2 \) Real	.090 (±.006)	.137 (±.010)	.141 (±.011)
\( S_2 \) \( L^{Sug} \)	.090 (±.007)	.133 (±.012)	.140 (±.011)
Table 6: ROUGE-SU  $F_{1}$ -based average scores of simulated sessions vs. controlled crowdsourced sessions. Scores at 350 words are approximate as a few sessions were shorter. Intervals at  $\geq 95\%$  confidence.
C.3 Length@Score Metric
Sessions	R1 .37	R2 .075	RL .31	RSU .14
S1 Oracle	193	191	199	232
S1 Real	218	233	233	266
S1 S\(ug\)	231	200	253	N/A
S2 Oracle	192	180	197	232
S2 Real	221	288	237	269
S2 S\(ug\)	236	N/A	245	310
Table 7: The Length@Score measurements for ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-SU  $F_{1}$  scores. This answers how many words on average are needed to reach the specified ROUGE  $F_{1}$  score. Values are calculated from the averaged overall session of a system, and not as a macro-average. When a value is 'N/A', the system did not reach the score.