SlowGuess/NAACL_2022_OCR huggingface dataset:
Ori Shapira $^{1,3*}$ , Ramakanth Pasunuru $^{2}$ , Mohit Bansal $^{2}$ , Ido Dagan $^{1}$ , and Yael Amsterdamer $^{1}$
$^{1}$ Bar-Ilan University  $^{2}$ UNC Chapel Hill  $^{3}$ Amazon
obspp18@gmail.com
{ram,mbansal}@cs.unc.edu
{dagan,amastery}@cs.biu.ac.il
Interactive summarization is a task that facilitates user-guided exploration of information within a document set. While one would like to employ state of the art neural models to improve the quality of interactive summarization, many such technologies cannot ingest the full document set or cannot operate at sufficient speed for interactivity. To that end, we propose two novel deep reinforcement learning models for the task that address, respectively, the subtask of summarizing salient information that adheres to user queries, and the subtask of listing suggested queries to assist users throughout their exploration. In particular, our models allow encoding the interactive session state and history to refrain from redundancy. Together, these models compose a state of the art solution that addresses all of the task requirements. We compare our solution to a recent interactive summarization system, and show through an experimental study involving real users that our models are able to improve informativeness while preserving positive user experience.
Integrating human interaction into NLP tasks has been gaining the interest of the NLP community. Human-machine cooperation can improve the general quality of results, as well as provide a higher sense of control for the targeted consumer. We focus on the task of interactive summarization (INTSUMM:Shapira et al.,2021b) which enables information exploration within a document set on a topic, by means of user-guided summarization. As illustrated in Figure 1, a user can incrementally expand on a summary by submitting requests to the system, in order to expose the information of interest within the topic. A proper exploration session demands access to all information within the document set, and fast reaction time for smooth human
Figure 1: An INTSUMM system, ingesting a large document set. A user interactively submits queries in order to expand on the information. The system is required to process the full document set for comprehensive exploration, respond quickly, and expose nonredundant salient information that also complies to the input queries. See real example in Figure 5.
engagement (Anderson, 2020; Attig et al., 2017). In addition, presented information must consider the session history to refrain from repetitiveness.
While it is worthwhile to apply recent NLP advances that excel at extracting salient and query-biased information, those advances usually come at a cost of rather small input size limits or heavy computation time. Indeed, all previous interactive summarization systems we know of either apply traditional methods or are inadequate for real-time processing due to high latency (§2). Our goal is to overcome these obstacles, and leverage advanced methods to improve information exposure while keeping latency acceptable for interaction.
As depicted in Figure 1, an INTSUMM system provides an initial generic summary as an overview of the topic, after which a user can iteratively issue queries to the system for summary expansions on subtopics of interest. To support querying, the system offers a list of suggested queries, hinting at information concealed within the document set.
We address the INTSUMM task components through two subtasks: (1) generating the initial summary and query responses, and (2) generating lists of suggested queries. For each of the subtasks we propose a deep reinforcement learning (RL) algorithm that addresses the respective sub-
* This work was conducted prior to joining Amazon.
$^{1}$ Code and trained models at: https://github.com/OriShapira/InterExp_DepRL
Human Language Technologies, pages 2551 - 2568
task requirements. To enable comprehensive topic exploration, our models speedily process the full document set, as inspired by Mao et al. (2020). Additionally, they are able to peek at session history to comply to the current state of the interaction. The model for the query-assisted summarization subtask,  $M_{Summ}$ , incorporates the query sequence by (1) encoding a query into the contextual sentence representations, (2) attending the representations using a new query-biased variant of the maximal marginal relevance (MMR: Carbonell and Goldstein, 1998) function, and (3) a dual reward mechanism for policy optimization (Pasunuru and Bansal, 2018) which we adapt to consider both reference summaries and the query ( $\S 3$ ). The model for the suggested queries list generation subtask,  $M_{Sugg}$ , works at the phrase level, as opposed to the sentence level, to enable extraction of important phrases that serve as suggested queries. Similarly to  $M_{Summ}$ , the model learns importance with consideration to session history, but without an input query – as its role is to suggest such a query ( $\S 4$ ).
The models are trained on the DUC $^2$  2007 multi-document summarization (MDS) news-domain dataset, with adoptions for our task setting. For testing, we follow the INTSUM evaluation framework of Shapira et al. (2021b) to run simulations, collect real user sessions, and assess the results, using DUC 2006. In principle, summary informativeness, i.e. general salience, could potentially come at the expense of query responsiveness, but importantly, our results show that our RL-based solution is able to significantly improve information exposure over the baseline of Shapira et al. (2021b), without compromising user experience ( $\S 5$ ).
Interactive summarization facilitates user-guided information navigation within document sets. The task suffered from a lack of a methodological evaluation, until Shapira et al. (2021b) formalized the INTSUMM task with a framework consisting of a benchmark, evaluation metrics, a session collection process and baseline systems. This framework, that we leverage, enables comparison and analysis of systems, allowing principled research on the task and accelerated development of algorithms.
To the best of our knowledge, all previous works on INTSUMM have either applied more traditional text-processing methods or require costly prepro
cessing of inputs to facilitate seamless interaction. Leuski et al. (2003) used surface-form features for processing content, and Baumel et al. (2014) adapted classic MDS algorithms like LexRank (Erkan and Radev, 2004) and KLSum (Haghighi and Vanderwende, 2009). Christensen et al. (2014) optimized discourse graphs and Shapira et al. (2017) relied on a knowledge representation, both expensively pre-generating hierarchical summaries that limit expansions to pre-prepared information selections. Hirsch et al. (2021) applied advanced coreference resolution algorithms that take several hours for preprocessing a document set.
The two INTSUMM baseline systems of Shapira et al. (2021b) use sentence clustering or TextRank (Mihalcea and Tarau, 2004) for summarization, sentence similarity heuristics for query-responses, and n-gram frequency or TextRank for suggested query extraction. Moreover, their query-response generators strictly consider a given query, ignoring history or global informativeness. Our proposed algorithms significantly improve information exposure over the latter baselines, using advanced deep RL methods, working in real time. We next review some recent techniques in MDS, query-focused keyphrase extraction, all of which relate to the INTSUMM task and our choice of algorithms.
The subtask of query-assisted summarization. Non-interactive MDS has been researched extensively, with few recent neural-based methods that can handle relatively large inputs. For example, Wang et al. (2020) use graph neural networks to globally score sentence salience, Xiao et al. (2021) summarize using Longformers (Beltagy et al., 2020), and Pasunuru et al. (2021b) combine a Longformer with BART (Lewis et al., 2020) and incorporate graphical representation of information. Mao et al. (2020) apply deep RL for autoregressive sentence selection, and, in contrast to most other neural methods, can ingest the full document set.
In the query-focused summarization (QFS) task summaries are biased on a query. To accommodate a query, Xie et al. (2020) use conditional self-attention to enforce dependency of the query on source words. Pasunuru et al. (2021a) and Kulkarni et al. (2021) hierarchically encode a query with the documents. These and other QFS methods require large training sets, and limit the allowed input size (Baumel et al., 2018; Laskar et al., 2020). Relatedly, incremental update summarization (Mc
$^{2}$ https://duc.nist.gov/
Creadie et al., 2014; Lin et al., 2017) marks query-relevant information as reported texts stream in, avoiding repeating information marked earlier. Interactivity is not a constraining factor here, yielding solutions with relatively high computation time.
With respect to the above related work, we develop a model inspired by Mao et al. (2020), which is closest to our requirements. To facilitate an interactive setting, our model (1) enables query+history injection, (2) supports full input processing, necessary for complete information availability during exploration, (3) has low latency at inference time, and (4) requires a relatively small training set.
The subtask of suggested-queries list generation. Extracting suggested queries on a document set most resembles the multi-document keyphrase extraction (MDKE) task since it aims to identify salient keyphrases (Shapira et al., 2021a). MDKE was mostly addressed using traditional heuristics or graph-centrality algorithms applied over the documents (e.g. Mihalcea and Tarau, 2004; Florescu and Caragea, 2017). In contrast to MDKE, the suggested queries extraction subtask is a new paradigm that updates "keyphrases" with respect to session history. While previous methods for keyphrase extraction could potentially be adapted for our dynamic setting, we choose to focus in this work on a deep RL architecture for suggested queries that resonates our model for query-assisted summarization and allows sharing insights between the models.
3 Query-Assisted Summarization Model
The subtask of query-assisted summarization covers two main components of the INTSUMM task: the generators of an initial summary and of query-responses. The initial summary concisely specifies some central issues from the input topic (not biased on a query) to initiate the user's understanding of the topic and to motivate further exploration. Then, for each user submitted query, the query-response generator non-redundantly expands on the previously presented information with topically salient responses that are also biased around the query. We next formally define the subtask and then describe our RL model for it.
3.1 Subtask Formulation
The input to the query-assisted summarization subtask is tuple  $(\mathcal{D},q,\pmb{E}^{\mathrm{in}},m)$ , such that:  $\mathcal{D}$  is a document set on a topic where the  $j$ -th sentence in the concatenation of  $\mathcal{D}$ 's documents is denoted  $s_j$ ;  $q$
is a query, and can be empty (denoted  $\_$ ) for an unbiased generic summary;  $E^{\mathrm{in}} = \{e_1^{\mathrm{in}},\dots,e_k^{\mathrm{in}}\}$  is a sequence of sentences from  $\mathcal{D}$  termed the history, containing texts previously output in the session; and  $m$  is the number of sentences to output. The output is sentence sequence  $E^{\mathrm{out}} = \{e_1^{\mathrm{out}},\dots,e_m^{\mathrm{out}}\}$  from  $\mathcal{D}$  (extractive summarization). When inputting  $(\mathcal{D},\_ ,\{\},m)$ , the output is a generic summary of  $m$  sentences, that can serve as the initial summary; and when  $q$  and  $E^{\mathrm{in}}$  are not empty, the output is an expansion on  $E^{\mathrm{in}}$  in response to  $q$ , containing new salient information biased on  $q$ .
$\mathcal{D}$  is paired with a set of generic reference summaries  $\mathcal{R}$ , which is used for training or as a part of the evaluation effort.
Our query-assisted summarization model,  $M_{Summ}$ , is autoregressive, outputting the requested number of summary sentences one-by-one. At time step  $t$ , a sentence  $e_t^{\mathrm{out}}$  is output according to the current query and an encoding of the summary-so-far  $E_t = \{e_1^{\mathrm{in}},\dots,e_k^{\mathrm{in}},e_1^{\mathrm{out}},\dots,e_{t - 1}^{\mathrm{out}}\}$  to prevent information repetition. At inference time,  $M_{Summ}$  outputs the summary sentences with the given query and history (possibly empty). At train time, we emulate a session by invoking  $M_{Summ}$  with a sequence of differing queries,  $\mathcal{Q} = \{q_1,q_2,\dots,q_m\}$ , for which to generate the corresponding sequence of output sentences. I.e., output sentence  $e_t^{\mathrm{out}}$  is biased on query  $q_t$  and the summary-so-far  $E_t$  at time step  $t$ . We next describe the architecture of  $M_{Summ}$ , also illustrated in Figure 2.
Sentence encoding. The first step of the model is hierarchically encoding the sentences of the document set  $\mathcal{D}$  to obtain contextualized representation  $c_{j}$  for sentence  $s_j\forall j$ . A CNN (Kim, 2014) encodes  $s_j$  on the sentence level and then a bi-LSTM (Huang et al., 2015) forms representation  $c_{j}$  on the document level, given the CNN encodings.
Query encoding. Additionally, at each time step  $t$  we prepare sentence+query representations  $c_{j}^{t} = c_{j} \oplus \mathrm{CNN}(q_{t})$ , i.e., obtained by concatenating a sentence representation and the CNN-encoding of the current query. This sentence+query representen
<sup>3</sup>In general, the implementation choices weighed in the speed at which the full input document set can be processed. In comparison to other techniques (some of which are more recent), these choices gave as good or better results at lower latency. Alternative architectural choices and their behavior are discussed in Appendix B.
Figure 2: The  $M_{Summ}$  query-assisted summarization model architecture. Contextual sentence embeddings are concatenated to the current query embedding. The sentence+query representation is softly attended with a transformed query-focused MMR score, and a sentence selection distribution is obtained with a two-hop attention mechanism, considering a summary-so-far representation. A dual-reward mechanism, using the reference summaries and query, optimizes a policy to train the model for summary content quality and sentence-to-query resemblance. At inference time, an initial summary is generated with empty  $E^{\mathrm{in}}$  and  $q_{t}$ -s, while for an expansion they are not empty.
tation influences the relevance of a sentence with respect to the current input query.
Query-MMR score weighting. MMR has been shown to be effective in MDS, where information repeats across documents. It aims to select a salient sentence for a summary, that is non-redundant to previous summary sentences. We extend standard MMR so that the importance of the sentence is in regards to both the document set and the query. Formally, the query-focused MMR function defines a score  $m_j^t$  for each  $s_j$  at time step  $t$  as follows:
$$
\begin{array}{l} m _ {j} ^ {t} = \lambda \cdot \operatorname {B I S I M} (s _ {j}, \mathcal {D}, q _ {t}) \\ - (1 - \lambda) \cdot \max  _ {e \in \boldsymbol {E} _ {t}} \operatorname {S I M} (s _ {j}, e) \quad (1) \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {B I S I M} \left(s _ {j}, \mathcal {D}, q _ {t}\right) = \beta \cdot \operatorname {S I M} \left(s _ {j}, \mathcal {D} ^ {\oplus}\right) \\ + (1 - \beta) \cdot \operatorname {S I M} \left(s _ {j}, q _ {t}\right) \tag {2} \\ \end{array}
$$
where  $\lambda \in [0,1]$  balances salience and redundancy and  $\beta \in [0,1]$  balances a sentence's salience within its document set and its resemblance to the current query.  $\mathrm{SIM}(x,y)$  measures the similarity of texts  $x$  and  $y$ , and  $\mathcal{D}^{\oplus}$  is a fully concatenated version of document set  $\mathcal{D}$ . Following findings of Mao et al. (2020), SIM computes cosine similarity between the two compared texts' TF-IDF vectors. Redundancy to previous sentences is computed as the highest similarity-score against any of the previous sentences. We set  $\lambda = 0.6$  (following Lebanoff et al., 2018) and  $\beta = 0.5$  (see Appendix B.3).
The query-focused MMR scores are incorporated into  $M_{Summ}$  by softly attending on the sentence representations with their respective translated query-focused MMR scores:
$$
\boldsymbol {\mu} ^ {t} = \operatorname {s o f t m a x} \left(\operatorname {M L P} \left(\boldsymbol {m} ^ {t}\right)\right) \tag {3}
$$
$$
\hat {\boldsymbol {c}} _ {j} ^ {t} = \mu_ {j} ^ {t} \boldsymbol {c} _ {j} ^ {t} \tag {4}
$$
State representation. At time  $t$ , a representation  $z_{t}$  of the summary-so-far is computed by applying an LSTM encoder on  $\{c_{\mathrm{idx}(e_1^{\mathrm{in}})}, \dots, c_{\mathrm{idx}(e_k^{\mathrm{in}})}, c_{\mathrm{idx}(e_1^{\mathrm{out}})}, \dots, c_{\mathrm{idx}(e_{t-1}^{\mathrm{out}})}\}$ , i.e., on the plain sentence representations of  $E_{t}$ , where  $\mathrm{idx}(e)$  is the index of sentence  $e$ . Then, a state representation  $g_{t}$  considers  $z_{t}$  and all sentence representations with the glimpse operation (Vinyals et al., 2016):
$$
a _ {j} ^ {t} = \boldsymbol {v} _ {1} \tanh  \left(\boldsymbol {W} _ {1} \hat {\boldsymbol {c}} _ {j} ^ {t} + \boldsymbol {W} _ {2} \boldsymbol {z} _ {t}\right) \tag {5}
$$
$$
\boldsymbol {\alpha} ^ {t} = \operatorname {s o f t m a x} \left(\boldsymbol {a} ^ {t}\right) \tag {6}
$$
$$
\boldsymbol {g} ^ {t} = \sum_ {j} \alpha_ {j} ^ {t} \boldsymbol {W} _ {1} \hat {\boldsymbol {c}} _ {j} ^ {t} \tag {7}
$$
where  $\pmb{v}_1, \pmb{W}_1$  and  $\pmb{W}_2$  are model parameters, and  $\pmb{a}^t$  represents the vector composed of  $a_j^t$ .
Finally, a sentence  $s_j$  at time  $t$  is assigned a selection probability  $\text{softmax}(p^t)_j$  such that:
$$
p _ {j} ^ {t} = \left\{ \begin{array}{l l} \boldsymbol {v} _ {2} \tanh  \left(\boldsymbol {W} _ {3} \hat {\boldsymbol {c}} _ {j} ^ {t} + \boldsymbol {W} _ {4} \boldsymbol {g} ^ {t}\right) & \text {i f} s _ {j} \notin \boldsymbol {E} _ {t} \\ - \infty & \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
where  $\pmb{v}_2$ ,  $\pmb{W}_3$  and  $\pmb{W}_4$  are model parameters.
Reinforcement learning. As  $M_{Summ}$ 's goal is to incrementally generate a query-assisted
summary, it should strive to optimize (1) nonredundant salient-sentence extraction and (2) query-to-sentence similarity, that can be appraised with ROUGE (Lin, 2004) and text-similarity metrics, respectively. A policy gradient-based RL approach (Williams, 1992) allows optimizing on such nondifferentiable metrics. Specifically, we adopt the Advantage Actor Critic method (Mnih et al., 2016) for policy learning, and a dual-reward procedure (Pasunuru and Bansal, 2018) to alternate between the summary and query-similarity rewards.
At time step  $t$ , for selected sentence  $e_t^{\mathrm{out}}$  (based on  $\mathrm{softmax}(p^t)$ ), reward  $r_t$  is computed and weighted into  $M_{Summ}$ 's loss function. The reward function alternates, from one train batch to the next, between ROUGE $_\Delta(e_t^{\mathrm{out}}, E_t, \mathcal{R})$  and QSIM $(e_t^{\mathrm{out}}, q_t)$ . The former computes the ROUGE difference before adding  $e_t$  to  $E_t$  and after:
$$
\begin{array}{l} \text {R O U G E} _ {\Delta} \left(e _ {t} ^ {\text {o u t}}, \boldsymbol {E} _ {t}, \mathcal {R}\right) = \\ \operatorname {R O U G E} \left(\left(\boldsymbol {E} _ {t} \cup e _ {t} ^ {\text {o u t}}\right) ^ {\oplus}, \mathcal {R}\right) - \operatorname {R O U G E} \left(\boldsymbol {E} _ {t} ^ {\oplus}, \mathcal {R}\right) \tag {9} \\ \end{array}
$$
A larger ROUGE $_{\Delta}$  value implies that  $e_t$  concisely adds more information onto  $E_t$ , with respect to topic reference summaries  $\mathcal{R}$ . We use ROUGE-1  $F_1$  as the ROUGE function here. The query-similarity reward function
$$
\begin{array}{l} \operatorname {Q S I M} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right) = \\ \operatorname {a v g} \left(\operatorname {S E M S I M} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right), \operatorname {L E X S I M} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right)\right) \tag {10} \\ \end{array}
$$
computes an average of semantic and lexical similarities between the selected sentence and corresponding query. SEMSIM computes the cosine similarity between the average of word embeddings (spaCy: Honnibal and Montani, 2021) of  $e_t^{\mathrm{out}}$  and that of  $q_t$ . For lexical similarity,
$$
\begin{array}{l} \operatorname {L E X S I M} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right) = \\ \operatorname {a v g} \left(R _ {1} ^ {p} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right), R _ {2} ^ {p} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right), R _ {L} ^ {p} \left(e _ {t} ^ {\text {o u t}}, q _ {t}\right)\right) \tag {11} \\ \end{array}
$$
is the average of ROUGE-1, 2 and L precision scores between sentence and query. By alternating between the two rewards, we train a sentence-selection policy in  $M_{Summ}$  to balance summary informativeness and adherence to queries.
Overall system. Our  $M_{Summ}$  model adopts its base architecture from Mao et al. (2020) (for generic MDS). Chiefly, we modify their model for handling an input query-sequence and a sentence
history, and employ a different summarization reward function. The query is incorporated in the sentence representation, in the new query-focused MMR function and in the dual-reward mechanism.
Pre-training. To provide a warm start for training  $M_{Summ}$ , a reduced version of  $M_{Summ}$  is first pre-trained for generic extractive single-document summarization using the large-scale CNN/Daily Mail corpus (Hermann et al., 2015), as proposed by Chen and Bansal (2018). The reduced model pre-trains the full model for contextual sentence representation and for salient-sentence selection in the single-document generic setting. See Appendix B.1 for precise technical details.
Training data. After pre-training the reduced version of  $M_{Summ}$ , we train the full model using the DUC 2007 MDS dataset, with modifications for our query-assisted MDS task. The dataset includes 45 topics (split into 35/10 train/val), each containing 25 documents and 4 reference summaries.
For each topic, we generate an "oracle" extractive summary by greedily aggregating 10 sentences from  $\mathcal{D}$ , that maximizes the ROUGE $_{\Delta}-1$  recall against  $\mathcal{R}$ . Then for each sentence, we extract a bi- or trigram that is most lexically-unique to the sentence, in comparison to all other sentences in  $\mathcal{D}$ . This yields a sequence of 10 "queries" that could easily render the corresponding oracle summary. The intuition for this approach is that it would teach  $M_{Summ}$  that it is worthwhile to consider a given query when selecting a sentence that is informative with respect to the reference summaries. This further assists in fulfilling the dual requirements of selecting a globally informative sentence that also adheres to the query. $^4$  Appendix B.3 discusses usage of different query types for training.
Validation metric. As the interactive session progresses, a recall curve emerges, that maps the ROUGE recall score (here ROUGE-1) versus the expanding summary token-length. Once the session halts, the area under the curve indicates the efficacy of the session for information exposure. A higher value implies faster unveiling of salient
4 Seemingly, the most natural approach would be to train the model with queries from real sessions (collected using a different system). However, a session's queries are dependent on outputs previously produced by the used system. Hence, these do not benefit the training process more than a synthesized sequence of queries.
information. Normalizing by the final summary length allows approximate comparability between different length sessions. We hence use the average (over topics) length-normalized area under the recall curve for validating the training progress.
4 Suggested Queries Extraction Model
4.1 Subtask Formulation