SlowGuess/NAACL_2021_OCR huggingface dataset:
Ori Shapira<sup>1</sup>, Ramakanth Pasunuru<sup>2</sup>, Hadar Ronen<sup>3</sup>, Mohit Bansal<sup>2</sup>, Yael Amsterdamer<sup>1</sup>, and Ido Dagan<sup>1</sup>
$^{1}$ Bar-Ilan University  $^{2}$ UNC Chapel Hill  $^{3}$ Peres Academic Center
{obspp18, hadarg}@gmail.com {ram, mbansal}@cs.unc.edu amstery, dagan}@cs.biu.ac.il
Allowing users to interact with multi-document summarizers is a promising direction towards improving and customizing summary results. Different ideas for interactive summarization have been proposed in previous work but these solutions are highly divergent and incomparable. In this paper, we develop an end-to-end evaluation framework for interactive summarization, focusing on expansion-based interaction, which considers the accumulating information along a user session. Our framework includes a procedure of collecting real user sessions, as well as evaluation measures relying on summarization standards, but adapted to reflect interaction. All of our solutions and resources are available publicly as a benchmark, allowing comparison of future developments in interactive summarization, and spurring progress in its methodological evaluation. We demonstrate the use of our framework by evaluating and comparing baseline implementations that we developed for this purpose, which will serve as part of our benchmark. Our extensive experimentation and analysis motivate the proposed evaluation framework design and support its viability.
Large bodies of texts on a topic oftentimes contain extensive information that is challenging for a potential reader to handle. Traditionally, information seeking tasks, like search, question-answering (QA) and multi-document summarization (MDS), are single-round input-output processes that can serve the information seeker only to a limited extent. This calls for an interactive setting where a user can guide the information gathering process. For search and QA, this type of research has been gaining momentum recently in areas such as exploratory search (Marchionini, 2006) and conversational QA (Reddy et al., 2019).
For MDS, where interaction would allow a user to affect summary content, only sporadic works have been seen over the years (e.g., Leuski et al., 2003; Lin et al., 2010; Yan et al., 2011; Baumel et al., 2014; Christensen et al., 2014; Shapira et al., 2017; Handler and O'Connor, 2017). A key gap in the development and adoption of interactive summarization (denoted here INTSUMM) solutions is the lack of evaluation methodologies and benchmarks for meaningful comparison of systems, similarly to those for static (non-interactive) summarization (e.g., NIST, 2014). The previous works on interactive or customizable summarization of multi-document sets are distinct, with proprietary evaluations that do not admit comparison. Furthermore, the evaluation processes are often not scalable and replicable, or do not give a comprehensive enough assessment.
In this paper we develop an end-to-end evaluation framework for INTSUMM systems. The framework starts with real user session collection on a system, via a concrete process of controlled crowdsourcing that we designed for this task. The sessions are then measured to produce absolute scores for the system, allowing for robust system comparison. Our framework supports a general notion of expansion-based interactive summarization, where the textual summary gradually expands in response to user interaction. Figure 1 presents an INTSUMM system that we implemented to illustrate this notion (§5.1). To ensure our evaluation framework is sound, we developed the framework in multiple cycles accompanied by user studies and extensive crowdsourcing experimentation. Our main contributions are as follows.
(1) Evaluation measures. We propose a set of automatic and manual evaluation measures for INTSUMM systems, which build upon a combination of established notions in static summarization and interactive systems, and enable utilizing available multi-document summarization (MDS)
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 657-677 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Our INTSUMM web application, implemented for testing our evaluation framework. The left screenshot shows the automatically generated initial summary for 25 articles on "Native American Challenges" (from the DUC 2006 MDS dataset), and a follow-up expansion query that the user might enter. The right screenshot shows the response to the query. [1] Initial summary; [2] box for entering free-text queries (or highlighted spans from the summary pane); [3] list of clickable system suggested queries; [4] last user query; [5] system response to the last query; [6] button to expand further on the last query. Subsequent queries and responses are continuously appended at the bottom of the summary pane, allowing exploration of the documents' content.
datasets. Our measures are aggregated over multiple interactive sessions and document sets to obtain an overall system evaluation. In contrast to static summarization, our measures apply to the steps along the interaction to reflect the progress of information acquisition rather than just its final result. This is done by converting an interactive session to a sequence of incrementally growing static summaries, and measuring the accumulating information gain with a recall metric. See §3.
(2) Crowdsourced session collection process. Adequate INTSUMM system evaluation and comparison requires collecting realistic user sessions in a consistent manner, on which the measurements are conducted. Previous work mostly turn to in-house user-studies, which are less replicable, not scalable, and not always easily attainable. In contrast, standard crowdsourcing induces noise and overly tolerates subjective behavior, hindering replicability and comparability. We describe a controlled crowdsourcing procedure that overcomes the above obstacles, making the evaluation process reliable and much more accessible for researchers interested in pursuing INTSUMM research. See §4.
We demonstrate the use of our full evaluation framework on two INTSUMM systems that we implemented, which apply different algorithms but share a common user interface, with the DUC 2006 (Dang, 2006) MDS dataset. Analysis shows
favorable results in terms of internal consistency between sessions, users, and different evaluation measures, indicating that our solutions may serve as a promising benchmark for future INTSUMM research. See §5. The evaluation procedures and systems are available publicly. $^1$
Traditional MDS has been researched extensively (e.g. Goldstein et al., 2000b; Radev et al., 2004; Haghighi and Vanderwende, 2009; Yin and Pei, 2015). It encompasses variants of query-focused summarization (Dang, 2005), orienting the output summary around a given query (e.g. Daumé III and Marcu, 2006; Zhao et al., 2009; Cao et al., 2016; Feigenblat et al., 2017; Baumel et al., 2018), and incremental update summarization (Dang and Owczarzak, 2008), generating a summary of a document set with the assumption of prior knowledge on an earlier set (e.g. Li et al., 2008; Wang and Li, 2010; McCreadie et al., 2014; Zopf et al., 2016). Evaluation approaches predominantly include automatic ROUGE (Lin, 2004) measurement, i.e. word overlap against reference summaries, and manual responsiveness (Dang, 2006) scores or pairwise comparison (Zopf, 2018) between summaries.
'https://github.com/OriShapira/ InterExp
In the related QA task (Voorhees et al., 1999), a system extracts an answer for a targeted question. Similarly, in the interactive setting, a conversational QA (Reddy et al., 2019) system extracts answers to a series of interconnected questions with a clear informational goal. To check correctness in both cases, a system answer is simply compared to the true answer via text-comparison. On the contrary, in the exploratory style of INTSUMM, where the knowledge desired is less certain, evaluation must consider dynamically accumulating information.
Exploratory search (Marchionini, 2006; White and Roth, 2009) addresses the need for converting big data to knowledge via human-machine cooperation. For example, interactive information retrieval (Ingwersen, 1992) focuses on fine-tuning document retrieval interactively, and complex-interactive-QA (ciQA) (Kelly and Lin, 2007) involves interacting with a system to generate a passage that answers a complex question. Evaluation is a major challenge in dealing with these tasks (White et al., 2008; Palagi et al., 2017; Hendahewa and Shah, 2017). Firstly, real users must use the system being evaluated by completing a task-appropriate assignment, requiring large-scale user studies that highly increase the cost and complexity of evaluation. Furthermore, varying user behavior could mean distorted session comparison. Then, a system is measured on the basis of its final outputs, mostly disregarding the evolvement of the interactional session.
Among interactive summarization, in the query-chain focused summarization task (Baumel et al., 2014), a chain of queries yields a sequence of short summaries, each refraining from repeating content. The task's evaluation relies solely on pre-defined sequences of queries with a respective reference summary per iteration (laboriously prepared by experts) that disregards previous outputs by the system. Other interactive summarization systems, such as Christensen et al. (2014); Shapira et al. (2017), present a preassembled summary with several levels of detail, allowing a user to drill down to or expand on information of interest. These works do not evaluate in a manner that is comparable to others, and do not consider information variation due to interaction. They perform small-scale user-studies for preference between their system and static variants, or a single automatic assessment of the fully expanded final summary.
We address all of the above-mentioned evalua
tion issues, specifically targeting the INTSUM task, where the interaction-induced outputs are purely textual summary snippets of the input document set.
3 INTSUMM Evaluation
An INTSUMM system is evaluated by measuring its performance on multiple sessions produced as a result of human operation. The input of a session,  $\Sigma$ , is a set of documents,  $D$ , on which to explore. A session comprises an automatically generated initial summary,  $\sigma_0$ , and a sequence of user-posed requests,  $q_{i}$ , and corresponding output responses,  $r_i$ . The responses can be viewed as expansions of  $\sigma_0$ . Consequently, the overall interactive summary resulting from  $\Sigma$  defines a sequence of incrementally expanding snapshots  $[\sigma_0,\sigma_1,\dots ,\sigma_{|\Sigma |}]$  where  $\sigma_{i} = \sigma_{0}\cup \bigcup_{j = 1}^{i}r_{j}$  is the union of accumulative (summarized) information presented to the user after  $i$  interactions. Each snapshot may thus be regarded as a static summary, allowing static summarization measures to be applied on it.
For compared INTSUMM systems  $S_{1}, \ldots, S_{m}$ , we require at least  $u$  sessions of distinct users interacting with  $S_{i}$  on each test document set  $D \in \{D_{1}, \ldots, D_{n}\}$ . Assuming such sessions, we next define automatic and manual evaluation measures, and defer details on adequate session collection to §4. Importantly, all measures are based on established evaluation mechanisms used in static summarization and interactive systems, that we extend or adapt for the INTSUMM setting, and that are practically linear in time to the length of the session sequence. Together, the set of measures we define provide an encompassing assessment adequate for the evaluation of interactive summary systems.
3.1 Automatic Measures
Viewing a session as a sequence of incrementally expanding static summary snapshots, we would first like to obtain comparable scores for each static summary that will capture the information gained along the session up to the current interaction. Existing static MDS benchmarks provide reference summaries at a single length for the purpose of evaluating a summary at a similar length. This presumably means we would require a series of reference summaries that differ by small length gaps for the sequence of lengthening snapshots, which is difficult and costly to produce. To address this obstacle, we leverage a finding by Shapira et al. (2018) show
Figure 2: Example recall-curves of two sessions on an INTSUMM system. Points plotted per interaction snapshot within a session. Range of intersection between observed summary lengths is bounded by dashed lines.
ing that a reference summary of a single length can be used to relatively evaluate varying length summaries on a topic with a recall measure such as ROUGE. Thus, utilizing existing MDS datasets is indeed possible for measuring information gain throughout a session's snapshot sequence.
Based on this observation we now define three indicators for system performance, first over a single session and then aggregated over all sessions of a system.
Per-session indicators. (1) To illustrate the gradual information gain along a session we adopt a recall-by-length curve (Kelly and Lin, 2007; Lin, 2007), see for example Figure 2. The curve's x-axis is the snapshot word-length, chosen as the dominant factor affecting quality, as opposed to number of queries or interaction time, which are not necessarily comparable between sessions. The y-axis is a summary content recall score, such as ROUGE-recall against constant reference summaries. For session  $\Sigma$  with snapshots  $\sigma_0, \sigma_1, \dots, \sigma_{|\Sigma|}$ , each  $\sigma_i$  with word length  $x_i$  and content recall score  $y_i$  is plotted on the graph at  $(x_i, y_i)$ .
(2) We consider the area under the recall-curve (AUC). Intuitively, it is desirable for an INTSUMM system to generate more salient information earlier: assuming salient information is more relevant to users, this property means interaction is ceased sooner, as soon as the information needs are met. Accordingly, AUC is higher when content is more relevant and is retrieved earlier. AUC is defined between start and end x-values, fixed for comparable
measurement (see Figure 2), with y-value scores interpolated at these limits when a curve does not have a snapshot at the specific length(s).
(3) We consider the Score@Length metric that reports a score, such as standard ROUGE  $F_{1}$ , at pre-specified word-lengths, and demonstrates the informational effectiveness of a system at those lengths. This metric enables fair comparison to static summaries at the specified lengths. The inverse Length@Score measure is also examined, and detailed further in Appendix C.
Aggregated indicators. Our final system-level performance indicators are computed from the respective session indicators, as follows.
- The average recall curve, illustrating overall gradual information gain, is computed from individual session recall-at-length curves by interpolating y-values at constant x-value increments and averaging correspondingly. E.g., Figure 3.
- [P.1] is the average AUC computed from the individual session AUCs by first averaging per topic and then averaging the results over all topics, to give equal weight to each topic.
- [P.2] is the average Score@Length computed similarly to average AUC from individual session Score@Lengths.
3.2 Human Ratings
Automatic evaluation is convenient for fast assessment and consistent comparison, however manual appraisal more accurately forecasts the quality of a summarization system (Owczarzak et al., 2012). Thus, using manual metrics alongside automatic ones is important despite the higher cost it incurs.
Our evaluation framework allows doubly leveraging the involvement of human users by asking them to rate different system aspects during the session. We propose the following rating layout, with each measure being scored on a 1-to-5 scale.
- [R.1] After reading the initial summary, the user rates how informative it is for the given topic. This resembles the DUC manual summary content responsiveness rating (Dang, 2006).
- [R.2] To measure the information gain throughout the session, the user rates how much useful information each interaction's response adds. As this rating is scored per interaction, the session average measures overall ability to expose interesting information.
- [R.3] After the session, the user rates how generally well the system responded to the requests
2Any standard summary content recall measure can be used as long as it is consistent, including, e.g., manual mechanisms like Pyramid or nugget-style scoring (Nenkova and Passonneau, 2004; Lin and Demner-Fushman, 2006).
throughout the session.
- [R.4] As all human-involved systems should measure perceived usability, the user rates the two UMUX-Lite (Lewis et al., 2013) questionnaire statements:  $[R.4a]$  the system's capabilities meet the requirements and  $[R.4b]$  the system is easy to use. The UMUX-Lite score is a function of these two scores (although they are separately useful) and shows high correlation to the popular, and longer, SUS questionnaire (Brooke, 1996), thus offering a cheaper alternative.
Similarly to our automatic measures, these ratings are collected separately per session and then averaged, first per topic and then over all topics, to obtain comparable system scores.
The evident advantages of our proposed evaluation framework are: (1) scores are absolute and comparable from one session/system to another; (2) our framework fundamentally and conveniently extends upon prevailing static summarization evaluation practices and utilizes existing standard MDS dataset reference summaries.
4 Session Collection
The evaluation of interactive systems requires real user sessions, as explained in §3. Using a prototype INTSUMM system, described in §5.1, we conducted several cycles of session collection which uncovered multiple user-related challenges, in line with previous work on user task design (Christmann et al., 2019; Roit et al., 2020; Zuccon et al., 2013). In particular, recruited users may make undue interactions due to insincere or experimental behavior, yielding noisy sessions that do not reflect realistic system use. Additionally, without an objective informational goal, a user interacts with the system according to subjective interests, producing sessions that are objectively incomparable.
Controlled crowdsourcing method. Employing experts to use an interactive system in a user study is usually unnecessary and hinders scalability and accessibility for researchers, making crowdsourcing an appealing and less expensive alternative. While crowdsourcing is ordinarily used for annotation jobs, we show its suitability for system session collection. We designed a three-stage controlled crowdsourcing protocol that mitigates the aforementioned session collection challenges, while filtering out unsatisfactory workers (further details in Appendix B).
The first stage is a trap task whose aim is to
efficiently filter out insincere workers, and, conversely, discover workers with an ability to apprehend salient information within text. The second stage assigns practice tasks that familiarize the workers to the INTSUM system interface to prevent experimentation in the actual sessions to be evaluated. Here, the users are also presented with a grounding use-case, or 'cover-story' as termed by Borlund (2003). The use-case states an objective common goal to follow in interacting with the system, to minimize the effect of subjective preferences, and allow comparison against respective reference summaries with a similar objective goal. An example use-case to follow, applied in our experiments, is "produce an informative summary draft text which a journalist could use to best produce an overview of the topic". The use-case is strongly emphasized during practice sessions with integrated guidelines. Workers completing two practice assignments with predominantly relevant interactions are invited to continue on to the final stage.
The evaluation session collection stage involves interacting with the evaluated system, for a minimum amount of time per session (e.g., 150 seconds in our experiments), to produce a summary on a topic in light of the same assigned use-case as in the practice stage. Each worker may explore a topic once, and the overall goal is recording sufficiently many sessions per combination of system and topic. Generally in interactive tasks, systems are manually examined over a rather small number of instances (e.g. topics), with only a few users per instance, due to the high cost and complexity of collecting such sessions with experimenters. For example, Christensen et al. (2014) assessed their system on 10 topics, and the ciQA benchmark (Kelly and Lin, 2007) had 6 topics per tested subtask. Our session collection technique provides a more scalable approach, facilitating larger collection processes (e.g., in our experiments in §5 we used 20 topics and  $\geq 3$  sessions per system per topic).
We note that, in use cases or domains where experts are required, the proposed three-stage session collection protocol is still fully relevant. It is not limited to the crowdsourcing setting, and can be applied within controlled user-studies if needed.
Wild versus controlled crowdsourcing. We illustrate the benefit of the controlled crowdsourcing procedure described above by comparing its results with a "wild" crowdsourcing preliminary experiment. The latter experiment applied basic worker
Measure	Controlled	Wild
# interactions	12.3	7.0
Approx. explore time	250 sec.	170 sec.
% suggested query	36.2%	62.7%
% free-text query	25.3%	2.2%
% ΔAUC from lower bound	+1.8%	-1.4%
Table 1: Qualitative measures of improved session collection through controlled crowdsourcing against sincere wild crowdsourcing. Values were computed per session and averaged over all sessions on System  $S_{1}$  (see  $\S 5.1$ ).
filtering (99% approval rate and 1000 approved assignments on Amazon Mechanical Turk³ (AMT)) and did not apply the trap and practice tasks. For quality control, a post-session questionnaire was assigned in order to catch insincere workers.
Analysis of the collected sessions showed a substantial improvement in querying behavior in controlled crowdsourcing over sincere wild crowdsourcing (filtering out the insincere wild crowd-workers) – the former scored higher than the latter on every evaluation metric. Table 1 presents some qualitative indications of this improvement: controlled users were more engaged (more iterations and more time exploring) and put more thought into their queries (more free-text queries and less suggested queries). Notably, unlike uncontrolled crowd-workers, controlled workers were able to do better than a comparable fully automated baseline, evident from the last table row: the percent difference in ROUGE-1 AUC score from a “lower bound” simulated baseline (explained in §5.3), is positive (better) for controlled sessions and negative (worse) for “wild” ones. Finally, the queries of controlled users almost exclusively adhered to the use-case and the many helpful comments from the workers indicated their attentiveness to the task (see Appendix C).
We carried out experiments that assess our full evaluation framework and demonstrate its utility. As the few existing INTSUMM systems were not readily available or suitable for adaptation to our experimental setup, we developed an INTSUMM system of our own, shown in Figure 1, with two different algorithmic implementations for comparison. We gathered user sessions with our controlled crowdsourcing procedure and evaluated their quality with
our defined measures.
5.1 Test-System Implementations