SlowGuess/NAACL_2021_OCR huggingface dataset:
We also computed a "Length@Score" measurement assessing at what word length, on average, a given content score can be reached by the system. It might forecast how much interaction is required to reach a certain information coverage, indicating system effectiveness. This is computed on the overall averaged session, and not as a macro-average, since some topics do not reach the specified score. By looking at the averaged session, we have a better overlook at the system's capability. Table 7 presents these values. Here, the ROUGE scores for which to compute the resulting lengths are chosen based on numbers within a range of scores found in
MDS literature employing the DUC 2006 dataset on extractive summarization systems. We see an overall similar trend where the controlled user sessions fall between the upper and lower bounds, and that  $S_{1}$  scores slightly better than  $S_{2}$ . As an analysis, we can compare, e.g., the upper and lower bounds on ROUGE-1, and observe that the lower bound requires about 40 words more (20%) to contain a similar amount of salient content.
C.4 Standard Metric Implementations
Confidence intervals. All confidence intervals were calculated with a Python bootstrapping library, $^{10}$  and sometimes validated with an online tool (Wessa, 2020). The confidence level is  $\geq 95\%$  throughout the paper.
ROUGE. ROUGE scores were obtained either with the rouge Python package<sup>11</sup> for quick sentence comparisons, or the pyrouge Python package<sup>12</sup> for file comparison (long text to reference summaries).
UMUX-Lite When on a 5-point scale, the UMUX-Lite (Lewis et al., 2013) score is computed:
$$
0. 6 5 * \left(\left([ R. 4 a ] + [ R. 4 b ] - 2\right) * (1 0 0 / 8)\right) + 2 2. 9
$$
68 is considered average, and above 80 is excellent.
C.5 Real Session Examples
Tables 8 and 9 present real sessions on Systems  $S_{1}$  and  $S_{2}$  respectively, both on the topic "El Niño". The two users here have different querying strategies for acquiring information. Their strategies are not consistent across topics, as they depend on the complexity and breadth of the topics.
C.6 Simulation Samples
Table 10 shows the lists of queries in the simulations used for upper and lower bounds, for the topic "El Niño".
C.7 Feedback from Users
The session collection task also had a comment box to send any kind of feedback. Table 11 shows a few of these comments (some shortened for brevity). The comments strongly emphasize the users' sincerity in following the use-case, and note usefulness of the system as well as provide ideas for
improvements. Some comments attest to the need for prolonged exploration and human assistance for finding salient information.
10https://pypi.org/project/   
bootstrapped/   
11https://pypi.org/project/rouge/   
12https://pypi.org/project/pyrouge/
Table 8: A real session in System  $S_{1}$  on the topic "El Niño". [I] stands for 'initial summary', [H] stands for 'highlight' and [S] stands for 'suggested query'. The user here starts out with highlighted queries, and then uses suggestions.
LOracle
[I]
“El Nino can cause storms in California, tornadoes in Florida, a mild winter in the northern states.”
“Scientists in the United States, Australia, Israel, and Germany are using sophisticated computer simulations.”
“La Nina works in reverse of El Nino.”
“La Nina is the phenomenon of rapidly cooling equatorial waters in the central Pacific.”
“Computer modeling a simulation are used to study El Nino and El Nina patterns.”
“El Nino typically lasts a year.”
“Scientific technologies and techniques for studying these phenomena include computer modeling.”
“El Nino may lessen global warming by temporarily stemming the release of carbon dioxide from the Pacific.”
“Computer module studies and satellite systems allow for a better understanding of how El Nino and La Nina form.”
“The results of El Nino and El Nina can have severe economic impacts, disease and death.”
LFreqSug	LFreqTR
[I]
“el nino”
“la nina”
“pacific ocean”
“carbon dioxide”
“weather patterns”
“equatorial pacific”
“south america”
“global warming”
“weather phenomenon”
“surface temperatures”	[I]
“el nino years”
“el nino phenomenon”
“el nino events”
“el nino activity”
“next el nino”
“el nino behavior”
“el nino update”
“el ninos”
“global weather”
“normal weather patterns”
Table 10: The query lists for the topic "El Niño" used in the simulations for the upper ( $L^{\mathrm{Oracle}}$ ) and lower ( $L^{\mathrm{Sug}}$ ) bounds.  $\mathrm{Sug}^{\mathrm{FREQ}}$  is used in System  $S_{1}$  and  $\mathrm{Sug}^{\mathrm{TR}}$  is used in System  $S_{2}$ . [I] stands for "initial summary".
Topic	Comment
S1	School Safety	"A...mix of information...from...statistics...to facts about... specific incidents, probably because it's such a large topic."
S1	EgyptAir Crash	"Searching for "time" didn't give any kind of date or actual time of crash, however, searching for "date" tended to give actual date and time together..."
S1	EgyptAir Crash	"I noticed the search engine returned flexible dates this time (I searched 1991 and got 1996 results, for example) and I really appreciated that."
S1	Osteoarthritis	"There wasn't much...when trying to find specifics like symptoms... "Treatment" pulled up the closest and most relevant results, but the others went into the weeds or pulled up things that were tangential to the search terms."
S1	Evolution Teaching	"...This is the first of these where I thought the system really did not meet the need to efficiently collect useful info for a journalistic overview."
S1	Stephen Lawrence Killing	"I was very satisfied with the information provided...a topic with which I was totally unfamiliar. More generally...the system consistently did quite well...if I were a journalist writing overviews of these topics...I would be very pleased with...the information provided by the system [and] its ease of use!..."
S1	Quebec Separatist Movement	"I think anything that requires a higher level of background knowledge is a lot harder to research with this system, since you only get snippets."
S2	Wetlands	"I was able to find information on many different aspects of the topic."
S2	EgyptAir Crash	"...I [tried to] find out if there were other crashes...which did not turn up any info, but then later found that information when looking up a different search term."
S2	Concorde Aircraft	"The search results don't always seem to correspond to the terms keyed in..."
S2	Concorde Aircraft	"It seems like I'm always looking for more general info...Things I would want included in an overview or even in an article dealing with a specific incident such as in this case..."
S2	Elian Gonzales	"Most of the responses matched pretty well with the keyword search..."
S2	Elian Gonzales	"I find that I'm using the system the same way I use Google; whatever I'm wondering about I just ask in the form of a question."
S2	US Affordable Housing	"This set was very responsive and got results that I had not expected..."
S2	Kursk Submarine	"Outstanding! I feel like I could write an overview of this right now!"
S2	Jimmy Carter International	"In this one, the topic...was so general that it took me a bit to figure out exactly what I was supposed to be looking for. Once I got it, everything worked fine!"
S2	El Niño	"Great! Tons of useful information for a journalistic overview!"
Table 11: Some of many comments provided by the controlled crowdsourcing users for the two systems  $S_{1}$  and  $S_{2}$  on different topics (some shortened for brevity). The comments indicate that users follow the use-case. Notice that some comments show the need for prolonged exploration and human assistance for finding salient information.