SlowGuess/NAACL_2022_OCR huggingface dataset:
We show here results of the second experiment, where we used  $M_{Summ}$  model configuration  $i$ , with the same  $M_{Sugg}$  model as in the first experiment. The  $S_2$  baseline was similarly used for comparison. We also kept the same AUC length limits (106 to 250 tokens) for easy comparability to Table 2. Table 3 shows the results. Here too, while less substantially, informativeness is improved with our system without significantly harming the user experience. Overall, it seems that users were somewhat more satisfied with the INTSUMM system that uses  $M_{Summ}$  configuration  $\nu$  than configuration  $i$ . Interestingly, it seems the users may have appreciated the slightly better informativeness of configuration  $\nu$  even if the query-responsiveness was not as good as in configuration  $i$ , as shown through the QSIM score. In addition, we see that absolute manual scores in Table 3 are lower than in Table 2, but trends are generally similar. It is common that scaling of manually supplied scores can fluctuate (e.g. Gillick and Liu, 2010).
Figures 3 and 4 show the averaged (per topic and then over all topics) recall curves of the collected
sessions in the experiment described in §5.3 and above, respectively. The x-axis is the accumulating token-length of the session, and the y-axis is the ROUGE-1 recall. The points on the curve are the average interpolated values from all the sessions. The vertical dashed lines are the intersecting bounds of the sessions, from 106 tokens to 250. The area under the curve (AUC) is computed for each of the curves, and reported in the first row of Tables 2 and 3. The higher AUC scores obtained from the recall curves of our models, compared to those of the  $S_{2}$  baseline, highlight the ability to expose more salient information earlier in the session.
E.2 Execution Time of Systems
Systems that are made for interacting with humans must respond quickly in order to keep the user's engagement. The exact amount of time does not affect the user experience as long as it does not surpass some limit, after which the user starts losing interest or feeling irritated (Attig et al., 2017; Anderson, 2020).
As mentioned in Appendix D,  $M_{Summ}$  generates summaries in under a second and  $M_{Sugg}$  prepares the list in a few seconds. The baseline summarizer also responds in under a second. The difference between the systems is virtually unperceivable during interaction. There were no comments from the users in our experiments that stated any issue with execution time.
E.3 Assessment of Suggested Queries Functionality
In this analysis, we assessed what modes of query submission users relied on over the course of a session. To that end, (1) we divided each session to three segments (first, second and third part of the session), and counted the types of queries. The types are "suggested query", "free-text", "highlight" (a span from the summary text) and "repeat" (repeating the last submitted query). (2) We then computed the percentage of each mode in each segment. (3) The percentages over all sessions and all topics were computed for each of the three segments.
This process was conducted only for sessions between 4 and 20 interactions, as the few long and short sessions often show different behavior. For the first experiment, this left 43 sessions with avg. 8.63 (std. 2.32) interactions for our system, and 50 sessions with 8.44 (2.48) interaction for  $S_{2}$ . For the second experiment, it left 72 sessions with 10.24 (4.82) interactions for our system, and 74 sessions with 9.59 (4.42) interactions for  $S_{2}$ .
We focus here on the use of suggested queries versus all other query types. In the first experiment we observe a change of  $+9\%$  from the first to the third segment in our system, and  $-20\%$  in  $S_{2}$ . In the second experiment we see  $+18\%$  and  $-28\%$  in  $S_{2}$ . As discussed in §5.3, this suggests the effectiveness of updated suggested queries, especially by the end of a session.
Figure 3: Averaged recall curves of our system and the  $S_{2}$  baseline system in the experiment described in  $\S 5.3$  and Table 2 (using  $M_{Summ}$  configuration  $\nu$  from Table 1). The intersecting range is bounded by dashed lines (between 106 and 250 tokens).
Figure 4: Averaged recall curves of our system and the  $S_{2}$  baseline system in the experiment described here in Appendix E.1 and Table 3 (using  $M_{Summ}$  configuration  $i$  from Table 1). The intersecting range is bounded by dashed lines (between 106 and 250 tokens).
F Further Explanations on Evaluation Metrics
The normalized AUC score for the validation metric (explained in §3.3) is computed over the recall curve produced from the accumulating summary expansions. Each point on the curve marks an accumulating token-length (x-axis) and an accumulating recall score (y-axis) of an interactive state, as depicted in Figures 3 and 4 (although these figures show the averaged session recall curves with bounds, whereas during validation the curve is for a single session and there are no bounds set). By computing the area under the full curve, and dividing by the full length, the normalized AUC score is obtained. The normalization gives an approximate absolute value that can be compared at different lengths (although at large length differences this is not comparable due to the decaying slope of the curve).
The manual query-responsiveness score, reported in Tables 2 and 3, is obtained by asking users, at the end of a session, "During the interactive stage, how well did the responses respond to your queries?", for which they rate on a 1-to-5 scale. The scores are averaged over the topic and then over all topics. This follows the evaluation defined in Shapira et al. (2021b).
The UMUX-Lite score (Lewis et al., 2013), reported in Tables 2 and 3, is obtained by asking users to rate (1-to-5) two statements at the end of a session: (1) "The system's capabilities meet the need to efficiently collect useful information for a journalistic overview" and (2) "The system is easy to use". The first question refers to the users' informational goal that they received, in order to follow a consistent objective goal during their exploration. The final score is a function of these two scores, and is used as a replacement for the popular SUS metric (Brooke, 1996) (with a much longer questionnaire), to which it shows very high correlation, thus offering a cheaper alternative. This also follows the evaluation defined in Shapira et al. (2021b).
All confidence intervals in Tables 1, 2 and 3 are computed as margins-of-error, on the topic-level, over the standard error of the mean with  $95\%$  confidence.[10]
The token-length values in Table 1 are averages with standard deviations.
G A2C Policy Learning
A policy gradient-based reinforcement learning approach (Williams, 1992) allows optimizing on nondifferentiable metrics, and eliminates the exposure bias that occurs with traditional training methods, like cross-entropy, on generation tasks (Ranzato et al., 2016).
Specifically, we use the Advantage Actor Critic (A2C) policy gradient training method. See technical explanations in the appendix of (Chen and Bansal, 2018). At a high level, an output reward (subtracted by a baseline reward – computed on a version of the model without MMR attention) is used to weight the output selection in the loss function. In so, outputs with higher rewards increase the likelihood of those outputs and lower rewards decrease the likelihood. Since the reward function is not differentiable, it is used as a weight on the probability of the selected output, which is then given to the loss function.
H INTSUMM Example
We show in Figure 5 an example of an INTSUMM system using the web application of Shapira et al. (2021b) and our our  $M_{Summ}$  (configuration  $i$  from Table 1) and  $M_{Sugg}$  models in the backend.
10E.g., see https://wwwcalculator.net/ standard-deviation-calculator.html
(c)
Figure 5: An INTSUMM system using the web application of Shapira et al. (2021b), with our  $M_{\text{summ}}$  and  $M_{\text{sugg}}$  models run in the backend, on one of the topics in DUC 2006 with 25 news documents about "Global Warming". Sub-figure (a) shows the initial summary and the initial list of suggested queries. Sub-figure (b) shows the result of clicking the "carbon dioxide gas" suggested query (with the query response and updated suggested queries list). Sub-figure (c) shows the result of subsequently submitting the query "water level". Query responses should be informative for the general topic, while also complying to the user queries. System summaries and expansions must be output fast in order to allow smooth interaction and human engagement.