SlowGuess/NAACL_2022_OCR huggingface dataset:
5654, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141, Brussels, Belgium. Association for Computational Linguistics.
Anton Leuski, Chin-Yew Lin, and Eduard Hovy. 2003. iNeATS: Interactive Multi-Document Summarization. In The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics, pages 125–128, Sapporo, Japan. Association for Computational Linguistics.
James R. Lewis, Brian S. Utesch, and Deborah E. Maher. 2013. UMUX-LITE: When There's No Time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13, page 2099-2102, New York, NY, USA. Association for Computing Machinery.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pretraining for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jimmy Lin, Salman Mohammed, Royal Sequiera, Luchen Tan, Nimesh Ghelani, Mustafa Abualsaud, Richard McCreadie, Dmitrijs Milajevs, and Ellen Voorhees. 2017. Overview of the TREC 2017 Real-Time Summarization Track.
Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. Multi-document Summarization with Maximal Marginal Relevance-guided Reinforcement Learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1737-1751, Online. Association for Computational Linguistics.
Richard McCreadie, Craig Macdonald, and Iadh Ounis. 2014. Incremental Update Summarization: Adaptive Sentence Selection Based on Prevalence and Novelty. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, page 301-310, New York, NY, USA. Association for Computing Machinery.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing Order into Text. In Proceedings of the 2004
Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA. PMLR.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-Reward Reinforced Summarization with Saliency and Entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computational Linguistics.
Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley, Chenyan Xiong, Yizhe Zhang, Mohit Bansal, and Jianfeng Gao. 2021a. Data Augmentation for Abstractive Query-Focused Multi-Document Summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13666-13674.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021b. Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4768-4779, Online. Association for Computational Linguistics.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence Level Training with Recurrent Neural Networks. In ICLR.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Ori Shapira, Ramakanth Pasunuru, Ido Dagan, and Yael Amsterdamer. 2021a. Multi-Document Keyphrase Extraction: A Literature Review and the First Dataset. arXiv preprint arXiv:2110.01073.
Ori Shapira, Ramakanth Pasunuru, Hadar Ronen, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2021b. Extending Multi-Document Summarization Evaluation to the Interactive Setting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 657-677, Online. Association for Computational Linguistics.
Ori Shapira, Hadar Ronen, Meni Adler, Yael Amsterdamer, Judit Bar-Ilan, and Ido Dagan. 2017. Interactive Abstractive Summarization for Event News Tweets. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 109-114, Copenhagen, Denmark. Association for Computational Linguistics.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order Matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391.
Xiaojun Wan and Jianguo Xiao. 2008. Single Document Keyphrase Extraction Using Neighborhood Knowledge. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI'08, page 855-860. AAAI Press.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogeneous Graph Neural Networks for Extractive Document Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6209-6219, Online. Association for Computational Linguistics.
Ronald J Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 8(3):229-256.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021. PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization. arXiv preprint arXiv:2110.08499.
Yujia Xie, Tianyi Zhou, Yi Mao, and Weizhu Chen. 2020. Conditional Self-Attention for Query-based Summarization. arXiv preprint arXiv:2002.07338.
Datasets. The DUC 2006 and 2007 datasets were obtained according to the DUC website (duc.nist.gov) requirements. It was not possible for others to reconstruct the document sets and reference summaries of the dataset from the crowdsourcing tasks.
The datasets are composed of new articles mainly from the late 1990s from large news outlets, compiled by NIST. All data exposed by our systems are directly extracted from those articles. For extraction, we do not intentionally add in any rules for ignoring or boosting certain information due to an opinion.
Crowdsourcing. Due to the need for English speaking workers, a location filter was set on the Amazon Mechanical Turk (https://www.mturk.com) tasks for the US, UK and Australia. All tasks paid according to a $10 per hour wage, according to the estimated required time of each task. The payment was either paid per assignment, or as a combination with a bonus.
Compute resources. Our  $M_{Summ}$  and  $M_{Sugg}$  models required between 2 and 20 hours of training (usually around 4 hours), depending on the configuration. We trained on one NVIDIA GeForce GTX 1080 Ti GPU with 11GB memory. The pretrained base model was trained once and reused in all subsequent training. Outputting at inference time is computationally cheap:  $M_{Summ}$  runs up to about 1 second, but mostly in a few hundred milliseconds, and  $M_{Sugg}$  runs up to about 7 seconds, but mostly in under 4 seconds. Training with a batch size of 8 used about 3GB GPU memory for  $M_{Summ}$ , and about 9GB memory for  $M_{Sugg}$  (since there are many more input units per document set, i.e., all noun phrases versus sentences).
B.1 Pre-training Technicalities
To provide a warm start for training  $M_{Summ}$  and  $M_{Sugg}$ , a reduced version of the models, which is the same for both, is first pre-trained for generic extractive single-document summarization using the CNN/Daily Mail corpus (Hermann et al., 2015) with about 287k samples, as proposed by Chen and Bansal (2018). In this reduced model,  $\hat{c}_j^t$  is replaced by  $c_j$  in Equations 5, 7 and 8. Further-
more, there is a single reward function for learning the policy, computed per selected sentence  $e_t^{\mathrm{out}}$  as ROUGE- $L$ $F_1$  w.r.t. the (single) reference summary's sentence at index  $t$ . The reduced model pre-trains the full model for contextual sentence representation and for salient-sentence selection in the single-document generic setting. This allows training  $M_{Summ}$  and  $M_{Sugg}$  with a relatively small dataset for their final purposes.
B.2 Training Technicalities
Following (Mao et al., 2020), the pre-trained base model is the rnn-ext + RL model from Chen and Bansal (2018), and is trained like in Lebanoff et al. (2018). Both  $M_{Summ}$  and  $M_{Sugg}$  are further trained on our adjusted DUC 2007 data using an Adam optimizer with a learning rate of 5e-4 and no weight decay. A discount factor of 0.99 is used for the reinforcement learning rewards. The batch size was 8. Training was halted once 30 consecutive epochs did not improve the validation score.
The MMR function within our models uses TF-IDF vector cosine similarity for all SIM instances (in Equations 1 and 12). The TF-IDF vectorizer is initialized with the document set on which the MMR score is computed.
As is commonly practiced, selection of an output sentence/phrase  $e_t^{\mathrm{out}}$  is done by sampling probability distribution  $\pmb{p}^t$  (in Equation 8) at train time, and by extracting the maximum scoring sentence/phrase at inference time.
The MLP in Equation 3 transforms the MMR score with a feed-forward network with one-hidden layer of dimension 80 following (Mao et al., 2020).
B.3 Query-Assisted Summarization Model
Model configurations. The architecture of the  $M_{Summ}$  model and its training allowed for much creativity in the configuration process. Other than the combinations mentioned in the paper in Table 1, we also experimented with other components. We list here many of the experiments, without formal results. Anecdotes are taken by looking at validation scores and some eyeballing.
(1) The  $\beta$  value in the query-focused MMR function in Equation 2, that impacts the weight of the query on a sentence versus the document set on the sentence. We tried out a few  $\beta$  values and mainly noticed that a value of 0.5 kept validation results more stable across configurations, or kept training time shorter. In our experiments, to cancel out this component (both at training and inference
time), we simply set  $\beta = 1$  so that the query is not considered.
(2) Different summary reward functions. ROUGE $_{\Delta}$  recall (instead of  $F_{1}$ ) was also a good alternative, but gave somewhat less stable results across configurations. ROUGE (not as  $\Delta$ ) was also less stable with recall and  $F_{1}$ , and gave too short and irrelevant sentences with precision. We also tried sentence level ROUGE- $L$ , like in (Mao et al., 2020), eventually outputting sentences that were much less compliant to queries.
(3) Using only the query similarity reward instead of the dual reward mechanism worked surprisingly well. This may be due to the queries on which the model was trained on. These queries were very relevant to the gold reference summaries, hence possibly implicitly providing a strong signal to salient sentences within the document set. Still, this was less productive than our final choice of reward.
(4) Adding training data (additional DUC MDS datasets) did not impact the results. Importantly, since DUC 2007 is most similar to the test DUC 2006 set, it seems to be more beneficial to include DUC 2007 in the training set.
(5) We also tried representing the query in the input by concatenating it's raw text to each input sentence before get the sentence representations.
(6) To represent the sentences, we also tried using average w2v vectors (Honnibal and Montani, 2021) and Sentence-BERT (Reimers and Gurevych, 2019) instead of the CNN network. These did not show any apparent improvements, and were notably expensive in terms of execution time.
(7) For the sentence similarity in the query-MMR component, we tried w2v and SentenceBERT representations instead of TF-IDF vectors. Similarly to (6), they did not show improvements over using TF-IDF, and were very time-costly.
(8) Instead of the dual-reward mechanism that alternates between the two rewards from batch to batch, we also considered using a weighted average of the two rewards, consistently over all batches. Further experimentation is required on this technique for a more conclusive judgment.
Queries used for training. The queries used for training the  $M_{Summ}$  model can affect the way it learns to respond to a query. Seemingly, the most natural approach would be to train the model as close as possible to the model's use at inference time. This would mean training  $M_{Summ}$  with
queries from real sessions. However, a session's queries are dependent on outputs previously produced by the used system. It is therefore not certain that the sequence of queries from a different system's usage would necessarily benefit the training process when compared to a synthesized sequence of queries. I.e., it's not actually possible to train with "real sessions" in a conventional way.
Also, as stated in §3.3, the synthetic queries we eventually used direct the model to select salient sentences, which can support our dual-objectives: to get a sentence that is both globally salient to the topic, as well as responsive to the query. We tried training on other query types, synthesized with various keyphrase extraction techniques, and found that our final choice of queries more consistently gave good results overall.
Sentence length. We segmented the sentences in the document sets with the NLTK<sup>8</sup> sentence tokenizer, and removed sentences that contain quotes in them or do not end with a period.
During training we did not constrain the input sentences in any way. Some of the configuration experiments described above were done to check how the configuration might influence the length of the selected sentences. The best configurations, including the one we eventually used in our tests, tended to output somewhat longer sentences. Very long sentences are usually tedious for human readers, and we hence limited the sentences to 30 tokens at inference time. We found that this length constraint caused a slight degradation in simulation score results of our models, however still gave superior informativeness results compared to the baseline system.
Initial summary length. Sentences are accumulated until surpassing 75 tokens. Therefore summaries are not shorter than 75 tokens, but mostly not much longer than that.
B.4 Suggested Queries Extraction Model
Model configurations. We experimented with different configurations and hyper-parameter finetuning in the  $M_{Sugg}$  model as well. Tuning was performed in accordance to the validation scores and generic keyphrase extraction scores on the MK-DUC-01 multi-document keyphrase extraction dataset of Shapira et al. (2021a).
<sup>8</sup>https://www.nltk.org
(1) In the reward function in Equation 13, we set  $\gamma_{1} = 0.5$  and  $\gamma_{2} = 0.9$ , i.e., the preceding output phrases are more strongly accounted for than the phrases in the session history. We tested several values between 0 and 1 for both hyper-parameters.
(2) We implemented altered versions of the reward function in Equation 13. Instead of phrase unigram-level frequency, we tried computing the full phrase frequency and computing partial phrase frequency, i.e., a maximal phrase template match within a reference summary. All functions tested were adequate overall, though our final choice of reward function was closest to the keyphrase extraction task unigram overlap metric, and gave best results overall.
(3) We also attempted noun phrase extraction with the spaCy $^9$  noun chunker and named entity recognizer. This combined approach misses some noun phrases within the text, but mainly is also more computationally heavy than the simple POS regex search that we use.
Extracting phrases with regular-expression. We extracted all noun- phrases from the document set by first mapping all tokens to their part-of-speech tags, and then applying a regularexpression chunker with regex:  $\{\left(<  \mathrm{JJ} > \star \right.$  <NN.  $^ { \text{喜} } { > } +$  <IN>)?<JJ  $>\star$  NN.  $^ { \text{喜} } { > } + \}$
These steps were accomplished with NLTK.
Phrase length. There is no limit set on the phrase length. We tried training and inferring with a phrase length constraint of 4 words, but found that this gave worse results overall.
History sentences to phrases.  $M_{Sugg}$  works on the phrase level. Meanwhile, in our extractive interactive setting, the history is a set of sentences already presented to the reader. Therefore, when extracting phrases from  $\mathcal{D}$ , we also link each phrase to its source sentence, and obtain  $\pmb{E}^{\mathrm{in}}$  by compiling the phrases linked from the history sentences.
C Dataset Notes
While DUC 2006 (our test set) and 2007 (our train/validation set) were originally designed for the query-focused summarization task, they contain excessive topic concentration due to their long and descriptive topic queries (Baumel et al., 2016). Hence, their reference summaries can practically be considered generic.
D Session Collection
Controlled crowdsourcing protocol. We followed the controlled crowdsourcing protocol of Shapira et al. (2021b), which includes three steps: (1) a trap task for finding qualified workers; (2) practice tasks for explaining the interface and the purpose, as well as reiterating the generic information goal (see below) during exploration; (3) the session collection tasks. We used the Amazon Mechanical Turk HITs prepared by Shapira et al. (2021b).
Process cost. We paid  $0.40 for a trap task assignment, with 400 assignments released, and$ 0.90 for a practice task assignment, with 28 assignments completed. The session collection assignment paid $0.70, and a bonus mainly according to the length of interaction and additional comments provided. The bonus was between $0.15 and $0.35. A total of 111 sessions were recorded from 6 high quality workers. The full process cost about $385 in total (including the Mechanical Turk fees) for the experiment including configuration v in Table 1.
The second round of experiments done on another variant of our system (configuration  $i$ ) also included 28 practice tasks and compiled 10 final workers for a total of 180 collected sessions. Bonuses ranged from  $0.10 and$ 0.40 on the session collection task. The full process cost of the second experiment was about $475 in total (including the Mechanical Turk fees).
Session collection data preparation. We used the same 20 test topics as Shapira et al. (2021b), and created 2 batches of tasks. For the first batch, in alternating order of topics, 10 topics were paired with our system, and the other 10 were paired with the  $S_{2}$  baseline. The other batch consisted of the complementing topic-system pairings. The workers were assigned a batch to work on such that half of the workers would work on each batch.
User informational goal. Since all sessions on a topic are evaluated against the same reference summaries, it is important that users aim to explore similar information. Following Shapira et al. (2021b), during practice tasks all users received a common informational goal to follow, so that the sessions are comparable. The emphasized description was: "produce an informative summary draft text which a journalist could use to best produce an overview of the topic".
$^{9}$ https://spacy.io/
Sessions filtering. In the first experiment, we filtered out 7 sessions that accumulated less than 250 tokens (from 2 different workers).
In the second experiment, 9 of the 10 workers completed at least 19 of the 20 topics One worker completed only 3 tasks and we disregarded those sessions. We also threw away 9 sessions that accumulated less than 250 tokens.
INTSUMM user interface. We used the same user interface developed by Shapira et al. (2021b) with a small change to enable suggested query list updates after each interaction (the interface was designed for the baselines, where the suggested-query list is static). To refrain from any possible user experience bias, we made the UI change as least apparent as possible.
System response time.  $M_{Summ}$  is able to generate summaries mostly in under a second, and  $M_{Sugg}$  prepares the list in a few seconds. The summary expansion is hence presented to the user almost immediately after query submission, and the suggested queries list is shown shortly afterwards, before the user finishes reading the expansion. The small delay in suggested query updating is hence almost unnoticed. The baseline summarizer responds similarly fast to  $M_{Summ}$ , making response-time difference unperceivable between the systems.
User feedback. Many of the users provided feedback about the session collection tasks after finishing their assignment batch. The overall impression was that there was no strong preference for either system. For example, one user wrote: "I did not discern a consistent difference between the two systems that would result in having a clear preference." This kind of comment was repeated by several users. Generally, there were no explicit comments about the difference in quality of the summary outputs, and topics were mostly scored or commented on similarly between the two systems since the complexity of the topic influenced the ability of the systems to comply to the user.
A comment in favor of updating suggested queries during interaction said: "It was nice to have a new list as you progressed through the task, it helped me think of where to go next if I got stuck..." This specific comment was written by a user that explored topics quite deeply. On the other hand, a user that explored more shallow liked that used suggested queries in the static list were marked: "I did notice...the red font color on the used queries.
That was helpful." It therefore seems that updating suggested queries are more useful for lengthy exploration, but for quick navigation, the static list might naturally be enough.
E.1 Overall System Assessment
Metric	Ours	S2Baseline
R1AUC @ [106, 250]	42.52 (±1.65)	40.34 (±1.40)
R1F1@ initial	0.260 (±.011)	0.231 (±0.014)
R1F1@ 250	0.390 (±.015)	0.382 (±.014)
QSIM query-resp.	0.527 (±.016)	0.603 (±.022)
Manual query-resp.	3.66 (±0.29)	3.79 (±0.25)
Manual UMUX-Lite	73.8 (±3.6)	75.8 (±2.9)
Table 3: Average scores of our system and a baseline INTSUMM system on real user sessions, in an experiment using a different  $M_{Summ}$  configuration (configuration i) compared to the experiment of Table 2 (configuration v). Our system exposes topical information better, while the overall user experience is not significantly harmed. Intervals at  $95\%$  confidence level.
We conducted two comparative session collection and analysis experiments, one using  $M_{Summ}$  model configuration  $\nu$  (from Table 1), as presented in §5.3 and Table 2, and another with  $M_{Summ}$  model configuration  $i$ . As explained in §5.2, these two configurations performed best, on simulations, out of their respective configuration sets.