SlowGuess/NAACL_2022_OCR huggingface dataset:
We now consider the second subtask of INTSUMM: generating lists of suggested queries. The list is regenerated after every interaction, to yield queries that focus on sub-topics that were not yet explored.
Reusing the notations of  $M_{Summ}$  in §3, we define a model,  $M_{Sugg}$ , for suggested queries list generation, that receives an input tuple  $(\mathcal{D},\pmb{E}^{\mathrm{in}},m)$  (notice that a query is not needed here). Here, the  $j$ -th phrase in  $\mathcal{D}$  is denoted  $\rho_j$ , when the documents in  $\mathcal{D}$  are concatenated, and accordingly, history  $\pmb{E}^{\mathrm{in}}$  is a list of phrases extracted from the session's current accumulated summary.  $m$  is the number of suggested queries to output. The model outputs phrase sequence  $\pmb{E}^{\mathrm{out}} = \{e_1^{\mathrm{out}},e_2^{\mathrm{out}},\dots,e_m^{\mathrm{out}}\}$  from  $\mathcal{D}$ , accounting for history  $\pmb{E}^{\mathrm{in}}$ . As in  $M_{Summ}$ 's setting,  $\mathcal{D}$  is paired with a set of generic reference summaries  $\mathcal{R}$ .
We adopt and adjust the architecture in §3.2 for this subtask. Similar to  $M_{Summ}$ ,  $M_{Sugg}$  selects input units one-by-one considering a history, with the main difference being the absence of query injection. Additionally, inputs and outputs are processed on the phrase- rather than the sentence level.
Phrase and state representation. For the given document set, all noun phrases are extracted using a standard part-of-speech regular expression method (Mihalcea and Tarau, 2004; Wan and Xiao, 2008).
We obtain document-level contextual phrase embeddings,  $c_{j}$  for phrase  $\rho_{j}$ , with the CNN and biLSTM networks, and softly attend the embeddings with a standard MMR score:
$$
\begin{array}{l} m _ {j} ^ {t} = \lambda \cdot \operatorname {S I M} (\rho_ {j}, \mathcal {D} ^ {\oplus}) \\ - (1 - \lambda) \cdot \max  _ {e \in \boldsymbol {E} _ {t}} \operatorname {S I M} \left(\rho_ {j}, e\right) \tag {12} \\ \end{array}
$$
The MMR-based phrase representations then pass through the glimpse attention procedure, which culminates in the phrase probability distribution for selecting the next output phrase.
Reinforcement learning. The policy in  $M_{Sugg}$  is trained with a single reward function that measures how prominent the selected phrase is within the reference summaries, and how different it is from previously seen phrases. Formally, at time step  $t$ , the reward  $r_t$  of selected phrase  $e_t^{\mathrm{out}}$  is:
$$
\begin{array}{l} r _ {t} = \operatorname {P F} \left(e _ {t} ^ {\text {o u t}}, \mathcal {R}\right) - \gamma_ {1} \cdot \operatorname {P F M A X} \left(e _ {t} ^ {\text {o u t}}, E ^ {\text {i n}}, \mathcal {R}\right) \\ - \gamma_ {2} \cdot \operatorname {P F M A X} \left(e _ {t} ^ {\text {o u t}}, \boldsymbol {E} _ {t} \backslash \boldsymbol {E} ^ {\text {i n}}, \mathcal {R}\right) \tag {13} \\ \end{array}
$$
$$
\operatorname {P F} \left(e _ {t} ^ {\text {o u t}}, \mathcal {R}\right) = \operatorname {a v g} _ {r \in \mathcal {R}} \left(\operatorname {a v g} _ {w \in e _ {t} ^ {\text {o u t}}} \operatorname {T F} (w, r)\right) \tag {14}
$$
$$
\operatorname {P F M A X} \left(e _ {t} ^ {\text {o u t}}, L, \mathcal {R}\right) = \max  _ {e \in L} \operatorname {P F} \left(e _ {t} ^ {\text {o u t}} \cap e, \mathcal {R}\right) \tag {15}
$$
where  $\mathrm{TF}(w, r)$  is the relative frequency of word  $w$  in reference summary  $r$ . Namely, PF computes the average term frequency of a phrase over its words and across the reference summaries, as an estimate of the phrase importance within the topic. PFMAX computes the highest PF against a list of phrases, which is used to lower the reward of a phrase that is redundant to phrases used earlier. Different weights are given to the PFMAX against the input history  $(\gamma_{1})$  and that of the phrases output so far  $(\gamma_{2})$ .
Similarly to  $M_{Summ}$ , we first pre-train the base model to get a warm start on embedding formation and salience detection. The reduced architecture of  $M_{Summ}$  and  $M_{Sugg}$  for pre-training are identical.
We use the same DUC 2007 training data, with document sets and reference summaries, and additionally prepare three "histories" per topic: one empty and two non-empty. An empty history mimics generating a session's initial list of suggested queries, while a non-empty history trains the model to consider previously known information. Training with two non-empty histories per topic prepares a model for varying informational states. These are curated from a generic summary (from a trained  $M_{Summ}$  model) that is truncated at two random sentence-lengths between 1 and 12. Overall, the model is trained on three versions of each topic, each time with a different history.
Similarly to  $M_{Summ}$ , validation is guided by the average normalized area under the recall curve. Here, the accumulating  $r_t$  scores from Equation 13 are used as the recall of the expanding suggested queries list. I.e., a higher reward means better suggested queries are output earlier. The AUC is normalized with the total token-length of all suggested queries to mitigate for lengthy phrase extractions.
We ran several experiments for the assessment of our  $M_{Summ}$  and  $M_{Sugg}$  models, applying the INTSUMM evaluation framework of Shapira et al. (2021b). The goals of the experiments are to compare varying configurations of our models and to evaluate against an INTSUMM baseline system. The experiments include both simulations and interactive sessions with human users.
5.1 Compared Algorithms
The  $M_{Summ}$  model architecture (§3.2) has several configurable components: encoding the query into sentences, considering the query in the MMR function (both at train and inference time), and the dual reward mechanism. We compared several variations of these using simulations, presented in §5.2.
In addition, we compare, both via simulations ( $\S 5.2$ ) and real sessions ( $\S 5.3$ ), against the (better-performing) baseline system in (Shapira et al., 2021b), named  $S_{2}$ .  $S_{2}$ 's initial summary algorithm is TextRank, and the query-response generator extracts sentences via lexical+semantic similarity to the query, somewhat resembling QSIM in Equation 10, fully neglecting the summary-so-far, in contrast to  $M_{Summ}$ .  $S_{2}$ 's suggested queries list contains TextRank's top salient topic phrases. Since these too do not account for the summary-so-far, they are computed at the session beginning and are not updated along the session, in contrast to  $M_{Sugg}$ .
5.2 Simulated Experiments
The INTSUMM task involves human users by definition. Nevertheless, running on simulated query lists and session histories is pertinent for efficient system evaluation and comparison of methods.
To simulate the query-assisted summarization algorithms, we utilize the real sessions recorded by Shapira et al. (2021b): 3-4 user sessions on 20 topics from DUC 2006 collected with  $S_{2}$ . In our simulation, each summary-so-far from a recorded session is fed as input to the system together with the following recorded user query. We then measure  $R_{1\Delta}^{\mathrm{recall}}$  (difference of ROUGE-1 recall incurred by the query response compared with the input summary-so-far). Additionally, we use  $R_{1}^{F_{1}}$  (ROUGE-1  $F_{1}$ ) for initial summary informativeness. Both are measured w.r.t. the reference summaries, normalized by the output length, and averaged per session recording, and then over all sessions and topics, to get an overall system infor
mativeness score. We also measure system query-responsiveness using the QSIM metric.
Table 1 presents a representative partial ablation of the  $M_{Summ}$  model. All variants were configured to output sentences of up to 30 tokens, initial summaries are 75 tokens, and query responses are 2 sentences. Configurations  $i - iv$  use the query in training, while  $v$  and  $vi$  do not. Each configuration is measured for informativeness (columns marked with  $\dagger$ ), and for query-responsiveness (QSIM column). Out of configurations  $i - iv$ , config.  $i$ , where we employ all mechanisms for query inclusion, yields the best overall scores in both informativeness and query-responsiveness, despite the inherent tradeoff between the two. In the second set of configurations  $(v - vi)$ , we observe that ignoring the query at train time substantially degrades query-responsiveness, and this is expectedly further exacerbated when also ignoring the query at inference time. However, disregarding the query gives more informative expansions with respect to reference summaries, since the model was trained only to optimize content informativeness, and is less likely to sidetrack to the query-related information.
Compared to  $S_{2}$  (last row), our model significantly improves informativeness. Query-responsiveness is better in the  $S_{2}$  baseline since its query-response generator simply invokes a function similar to QSIM, but for the price of lower informativeness. Still, this does not lead to inferior overall user experience, see § 5.3.
5.3 Real Session Collection and Evaluation
We collect real user sessions via controlled crowdsourcing (which provides high quality work, see Appendix D) with the use of an INTSUMM web application<sup>5</sup> running either our  $M_{Summ} + M_{Sugg}$  models or the  $S_{2}$  baseline algorithms, enabling a comparative assessment of the two systems. Notably, our algorithms have the low latency required for the interactive setting (Attig et al., 2017), i.e., responding almost immediately.<sup>6</sup>
Using the DUC 2006 INTSUMM test set, we prepared two complementing user sets of 20 topics, each with 10 of the topics to be run on our system and the other 10 on the baseline. We apply the evaluation metrics of Shapira et al. (2021b): (1) The
<sup>5</sup>Minimally modified from (Shapira et al., 2021b) to support updating the suggested queries list after each interaction.
$^6 M_{Summ}$  generates summaries in under a second and  $M_{Sugg}$  prepares the list of suggested queries in a few seconds. See Appendix E.2 for more details.
MSumm Model Configuration	Simulation Results († = informativeness metric, R1 = ROUGE-1)
#	Query in Encoding	Query in MMR	Query in Reward (Dual)	Query in MMR at Inference	† Initial Summ Norm R1F1(x10-3)	Initial Summ Token-Length	† Expansion Norm R1Reacell(x10-3)	Expansion Token-Len.	QSIM Query Responsiveness
i.	yes	yes	yes	yes	3.09 (±0.11)	86.7 (5.6)	0.913 (±0.055)	49.7 (2.7)	0.488 (±0.021)
ii.	yes	yes	no	yes	3.04 (±0.12)	87.4 (8.3)	0.897 (±0.054)	49.4 (2.7)	0.482 (±0.022)
iii.	yes	no	no	yes	3.00 (±0.14)	88.0 (8.7)	0.892 (±0.058)	50.1 (2.9)	0.479 (±0.020)
iv.	no	yes	yes	yes	2.98 (±0.17)	85.1 (6.5)	0.892 (±0.057)	51.3 (2.8)	0.462 (±0.025)
v.	no	no	no	yes	3.05 (±0.12)	85.4 (8.1)	0.955 (±0.046)	51.8 (2.9)	0.423 (±0.027)
vi.	no	no	no	no	3.05 (±0.12)	85.4 (8.1)	0.988 (±0.056)	52.8 (4.0)	0.311 (±0.023)
S2Baseline (Shapira et al., 2021b)	2.75 (±0.20)	85.1 (21.8)	0.799 (±0.040)	49.1 (2.8)	0.601 (±0.021)
Table 1: Simulation results on previously collected sessions, yielding a partial ablation of our  $M_{Summ}$  model, and the results on the baseline system which was originally used to collect those sessions. Intervals at 95% confidence.
Metric	Ours	S2 Baseline
R1AUC @ [106, 250]	43.42 (±1.54)	40.01 (±1.52)
R1F1@ initial	0.256 (±.011)	0.231 (±0.014)
R1F1@ 250	0.396 (±.015)	0.378 (±.015)
QSIM query-resp.	0.471 (±.028)	0.623 (±.023)
Manual query-resp.	3.96 (±0.19)	4.03 (±0.23)
Manual UMUX-Lite	78.9 (±2.5)	78.6 (±3.4)
Table 2: Average scores of our system (configuration  $\nu$  ) and baseline system  ${S}_{2}$  on actual user sessions. Our system exposes topical information better, while the user experience is very good despite the slight degradation in query-responsiveness. Intervals at 95% confidence.
area under the sessions' ROUGE recall curves, in a common word-length interval across all sessions and topics, which demonstrates how fast salient information is exposed in sessions. (2) ROUGE  $F_{1}$  at the initial summary and at 250 tokens, that indicate how effectively the interactive system can generate summaries at pre-specified, comparable lengths. (3) Manually assigned query-responsiveness score (1 to 5 scale), which expresses how well users think the system responded to their requests. And (4) manual UMUX-Lite (Lewis et al., 2013) score for system usability (effectiveness and ease of use), where 68 is considered "acceptable" and 80.3 is considered "excellent". We also measure automatic query-responsiveness with QSIM.7
We conducted two such comparative collection and assessment experiments, either employing  $M_{Summ}$  configuration  $\nu$  or  $i$ , namely the best of the two configuration sets. In both cases, the  $M_{Sugg}$  model used was set with  $\gamma_1 = 0.5$  and  $\gamma_2 = 0.9$  after some hyperparameter tuning (Appendix B.4). The first experiment (with configuration  $\nu$ ) is described here, and the other in Appendix E.1.
We hired 6 qualified workers using the controlled crowdsourcing procedure, and collected 2-3 sessions per topic per system (111 total sessions). In
the sessions, users explore their given topic by submitting queries with a common generic informational goal in mind (Appendix D).
Overall system assessment. Table 2, presenting average scores over the collected sessions, shows that our system is significantly more effective for exposing salient information, as depicted in the first three rows. Users indicate a slight degradation in query-responsiveness of our system, consistent with QSIM scores (row 4-5). Note that the observed difference in QSIM scores, between simulations and user sessions, partly stems from the fact that they were computed over different sets of queries. The varying queries issued by the users in user sessions form a less stable query responsiveness comparison than the one in Table 1, where QSIM scores are computed using consistent queries for all systems. Despite the gap in QSIM scores between our system and  $S_{2}$  in Table 2, the overall usability scores are slightly better (last row). This may suggest that users appreciate the informativeness of the produced summary even when they are aware that the summary is less biased on their queries; thus our system improves informativeness while still providing a favorable user experience.
Assessment of suggested queries functionality. We analyzed the types of queries users submitted throughout their sessions, to assess the utility of updating suggested queries, with  $M_{Sugg}$ , as opposed to a static list of suggestions, with  $S_2$ . To that end, we tallied suggested query clicks and query submissions via other modes, binning the tallies to three sequential temporal segments within their respective sessions (Appendix E.3). We found that, on average, the usage of suggested query clicks increased by  $\sim 13\%$  when nearing the end of a session with  $M_{Sugg}$ , and conversely decreased by  $\sim 24\%$  with  $S_2$ . While the decrease in use of the static list is expected, since appealing queries are likely exhausted earlier in a session, it is encouraging to
7While QSIM is a reasonable automatic measure for estimation of query-responsiveness, it is left for future work to assess its true reliability for such use.
witness the usefulness of updated queries as the session progresses. This behavior suggests that the updated list contains suggested queries that are indeed engaging for learning more about the topic.
Interactive summarization for information exploration is a task that requires compliance to user requests and session history, while comprehensively handling a large input document set. These requirements pose a challenge for advanced text processing methods due to the need for fast reaction time. We present novel deep reinforcement learning based algorithms that answer to the task requirements, improving salient information exposure while satisfying user queries and keeping user experience positive.
We note that while  $M_{Summ}$  is designed for the INTSUMM task, it may potentially be serviceable for standard MDS, QFS, update summarization and combinations thereof. This can be accommodated by a proper choice of input, e.g., QFS can be addressed by giving  $M_{Summ}$  as input a query, an empty history and target summary length. In future work, we may study the performance of our solutions for such tasks, as well as strive to further improve their performance on both ends of the INTSUMM task – selecting topically salient information and responding to user queries.
We thank the anonymous reviewers for their constructive comments and suggestions. This work was supported in part by Intel Labs; by the Israel Science Foundation (grants no. 2827/21 and 2015/21); by a grant from the Israel Ministry of Science and Technology; by the NSF-CAREER Award #1846185; and by a Microsoft PhD Fellowship.
Shaun Anderson. 2020. How Fast Should A Website Load? https://www.hobo-web.co.uk/your-website-design-should-load-in-4-seconds. Accessed: 2021-09-25.
Christiane Attig, Nadine Rauh, Thomas Franke, and Josef F. Krems. 2017. System Latency Guidelines Then and Now – Is Zero Latency Really Considered Necessary? In Engineering Psychology and Cognitive Ergonomics: Cognition and Design, pages 3–14, Cham. Springer International Publishing.
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2014. Query-Chain Focused Summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 913–922, Baltimore, Maryland. Association for Computational Linguistics.
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2016. Topic Concentration in Query Focused Summarization Datasets. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, page 2573-2579. AAAI Press.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models. arXiv preprint arXiv:1801.07704.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv preprint arXiv:2004.05150.
John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194):4-7.
Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, page 335-336, New York, NY, USA. Association for Computing Machinery.
Yen-Chun Chen and Mohit Bansal. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686, Melbourne, Australia. Association for Computational Linguistics.
Janara Christensen, Stephen Soderland, Gagan Bansal, and Mausam. 2014. Hierarchical Summarization: Scaling Up Multi-Document Summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 902–912, Baltimore, Maryland. Association for Computational Linguistics.
Günes Erkan and Dragomir R. Radev. 2004. LexRank: Graph-Based Lexical Centrality as Salience in Text Summarization. Journal of Artificial Intelligence Research, 22(1):457-479.
Corina Florescu and Cornelia Caragea. 2017. Position-Rank: An Unsupervised Approach to Keyphrase Extraction from Scholarly Documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105-1115, Vancouver, Canada. Association for Computational Linguistics.
Dan Gillick and Yang Liu. 2010. Non-Expert Evaluation of Summarization Systems is Risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 148–151, Los Angeles. Association for Computational Linguistics.
Aria Haghighi and Lucy Vanderwende. 2009. Exploring Content Models for Multi-Document Summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362-370, Boulder, Colorado. Association for Computational Linguistics.
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, page 1693-1701, Cambridge, MA, USA. MIT Press.
Eran Hirsch, Alon Eirew, Ori Shapira, Avi Caci-ularu, Arie Cattan, Ori Ernst, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, and Ido Dagan. 2021. iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 283–297, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2021. Linguistic Features - spaCy Usage Documentation. https://spacy.io/usage/linguistic-features#vectors-similarity. Accessed: 2021-11-01.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. CoRR, abs/1508.01991.
Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2021. CoMSum and SIBERT: A Dataset and Neural Model for Query-Based Multi-document Summarization. In Document Analysis and Recognition - ICDAR 2021, pages 84-98, Cham. Springer International Publishing.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2020. WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5647-