A Set Prediction Network For Extractive Summarization
Year: 2023

Abstract:
Extractive summarization focuses on extracting salient sentences from the source document and incorporating them in the summary without changing their wording or structure. The naive approach for extractive summarization is sentence classification, which makes independent binary decisions for each sentence, resulting in the model cannot detect the dependencies between sentences in the summary. Recent approaches introduce an autoregressive decoder to detect redundancy relationship between sentences by step-by-step sentence selection, but bring train-inference gap. To address these issues, we formulate extractive summarization as a salient sentence set recognition task. To solve the sentence set recognition task, we propose a set prediction network (SetSum), which sets up a fixed set of learnable queries to extract the entire sentence set of the summary, while capturing the dependencies between them. Different from previous methods with an auto-regressive decoder, we employ a non-autoregressive decoder to predict the sentences within the summary in parallel during both the training and inference process, which eliminates the traininference gap. Experimental results on both single-document and multi-document extracted summary datasets show that our approach outperforms previous state-of-the-art models.

heading: 1 Introduction
text: Extractive summarization is the process of extracting a brief set of sentences from the source document to cover the salient information of it. Compared with abstractive summarization (Liu and Liu, 2021; Wu et al., 2021), extractive summarization is less likely to deviate from the source document, as well as more efficient in execution. Due to these advantages, it has become widely utilized for automatic summarization tasks.
Recently, most of the approaches (Nallapati et al., 2017; Liu and Lapata, 2019; Xu et al., 2020)
† Corresponding author.
formulate extractive summarization as a sequence labeling task, following the encoder-decoder framework. Sentence classification (Cheng and Lapata, 2016) is a naive solution for the task, which makes independent binary decisions for each sentence in the source document, leading to high redundancy of the summary, as shown in Figure 1 (a). To tackle the redundancy problem, Zhou et al. (2018); Narayan et al. (2020) introduce an autoregressive decoder, which select sentences step by step to construct the summary, until the length limit of the summary is reached, as shown in Figure 1 (b). Trigram Blocking (Paulus et al., 2018), as a plugand-play method during inference, has the same motivation, which allows the model to consider the sentences already selected when making binary decisions for the current sentence. Besides, some methods (Narayan et al., 2018b; Bae et al., 2019; Gu et al., 2022) incorporate reinforcement learning to optimize the final evaluation metric with a
4766
step-by-step selection strategy. Zhong et al. (2020); Chen et al. (2021) propose a method to construct a summary with two steps, where step one is for constructing candidates summary, step two is for selecting a summary from the candidates. However, these methods not only decrease the efficiency of training and inference but also suffer from the training-inference gap.
To address the above issues, we formulate extractive summarization as a salient sentence set recognition task, which treats summary as the salient set of sentences, rather than a set of salient sentences. To solve the sentence set recognition task, we propose a set prediction network based on the encoderdecoder framework, which sets up a fixed set of learnable queries to extract the entire sentence set of the summary, while capturing the dependencies between them. As shown in Figure 1 (c), different from previous approaches, we employ a nonautoregressive decoder to extract the sentence set of the summary in parallel during both training and inference. The non-autoregressive decoder receives the sentence representation and a set of learnable vectors called sentence queries to decode the final sentence set in the summary. To measure the difference between predictions and gold labels in an end-to-end manner, we employ a loss function based on bipartite matching, which can produce an optimal matching between predictions and gold labels with minimal assignment cost. Compared with the autoregressive approach, our set prediction network is efficient and can eliminate the gap between training and inference. Furthermore, the decoder is able to capture the dependencies between sentences through a self-attention mechanism between sentence queries and then make joint decisions on the entire sentence set. Experimental results on four single-document and one multi-document extracted summary datasets show that our approach outperforms previous state-of-the-art models.
Our main contributions are as follows:
• We formulate extractive summarization as a salient sentence set recognition task, which treats summary as a salient set of sentences, rather than a set of salient sentences.
• To solve the sentence set recognition task, we propose a set prediction network, which not only enables capturing the dependencies between sentences in the summary but also eliminates the training-inference gap.
• Experiments show that our proposed model achieves state-of-the-art performance on several single-document and multi-document datasets.

heading: 2 Related Work
text: Traditional approaches (Nallapati et al., 2017; Liu and Lapata, 2019; Xu et al., 2020) formulate extractive summarization as a sequence labeling task following the encoder-decoder framework. To improve the performance of extractive summarization, some methods introduce autoregressive decoder (Liu and Lapata, 2019; Narayan et al., 2020) or reinforcement learning (Dong et al., 2018; Gu et al., 2022). These methods construct summary step-by-step until the length limit of the summary is reached. In addition to the above paradigm of sequence labeling, Zhong et al. (2020); An et al. (2022) formulate the extractive summarization task as a semantic text matching problem. Tang et al. (2022) formulates extractive summarization as an Optimal Transport (OT) problem from document to summary. Jia et al. (2021) selects sentences simultaneously from the source document when the predicted sentence probabilities exceed a threshold. Although the above methods have made different degrees of progress in extractive summarization, they also encounter various challenges, such as insufficient decoding efficiency, training, inference gap, unstable results, etc.
Recently, the query-based approach is employed in the summarization task. For example, (Xu and Lapata, 2022) unifies that all summaries are a response to a query. (Zhang et al., 2022) use learnable queries as a control signal to control summary generation. These query-based methods focus on generating a highly relevant summary for a given query, in which queries can be observable or latent. Besides, these methods restrict queries to a single sample and semantic space. In this paper, we propose a set prediction network for extractive summarization based on sample-independent queries, which uses a non-autoregressive decoder to improve the decoding efficiency and unify the training and inference processes.

heading: 3 Method
text: In this section, we first introduce the task formulation in §3.1 and then describe each component of our method in detail. As shown in Figure 2, Our method consists of three components, a doc-
ument encoder §3.2, a non-autoregressive set decoder §3.3, and a bipartite matching loss §3.4.

heading: 3.1 Task Formulation
text: Given a training sample (D,G), where D = (s1, s2, . . . , sn) denotes the original document with n sentences, G denotes reference summary. Our goal is to select S∗ = {s∗1, s∗2, . . . , s∗m} from D to cover the sailent information of it, where m ≤ n. A set of gold sentences Y = {< yli, yri , yti >}m−1i=0 is derived by a greedy selection strategy, where yli, y r i ∈ [0, n − 1], yti ∈ {0, 1} represents the left boundary, right boundary, and label of the i-th sentence, respectively.

heading: 3.2 Document Encoder
text: The encoder is designed to get a contextual representation of the sentence in the document. Following Liu and Lapata (2019), we first concatenate the sentences together while inserting a [CLS] and a [SEP] token at the start and the end of each sentence, respectively, and then input them to BERT (Devlin et al., 2019) to obtain a contextual representation of each token. After BERT encoding, we further encode the document with a bidirectional LSTM layer. Then, we take the representation of all the [CLS] tokens as the representation of the sentences h = {h1, h2, · · · , hn}.
In order to improve the model’s ability to capture the inter-sentence relationships, we use a 3-layer Transformer (Vaswani et al., 2017) to encode it, which can be formulated as:
hs = Transformer(h) (1)
Finally, we get contextual representation for sentences hs = {hs1, hs2, · · · , hsn} in the source document.

heading: 3.3 Set Decoder
text: The purpose of the set decoder is to generate the entire sentence set of the summary in parallel based on the output of the document encoder. To achieve this purpose, we use a non-autoregressive decoder as the backbone of the set decoder.
Input The input of the set decoder consists of M learnable randomly initialized vectors eq ∈ RM×d and the output hs ∈ Rn×d of the document encoder. Each query corresponds to a prediction, and for M queries the set decoder generates M predictions. In order to decode all sentences in the summary, we set M greater than the maximum number of sentences contained in the summary.
Non-Autoregressive Decoder The set decoder is based on a non-autoregressive decoder. We use a N -stacked identical layer to construct the nonautoregressive decoder. Each layer incorporates a multi-headed self-attention mechanism to represent the relationship between queries eq, and a multi-headed cross-attention mechanism to fuse information of the sentence hs in the source document, which can be formulated as follows:
Hq = Decoder(eq;hs) (2)
where eq, hs denote initialized query vectors and sentence representation in the source document, respectively.
Through the non-autoregressive decoder, M sentence queries are transformed into M query embeddings, which are denoted as Hq ∈ RM×d. In contrast to the autoregressive decoder that needs to adopt the mask mechanism to prevent information leakage, the non-autoregressive decoder has no need to adopt the mask strategy to prevent the earlier decoding steps from obtaining information from the subsequent steps. Therefore, we do not add any causal mask in the multi-head selfattention mechanism.
Set Prediction Each query embedding hqi in H q predicts one sentence from document total M in parallel. Set prediction is a joint decision of boundary and label.
To get the boundary for each query hqi , we first interact the query with each sentence of the document by two linear layers. The fusion representation of the i-th query and j-th sentence is computed as:
hf r/l i,j = Tanh(h q iwi + h s jwj) (3)
where wi, wj ∈ Rd×d are trainable projection parameters, r/l denotes left or right. Then we get fuse representation of the i-th query with all sentence hf r/l
i = [h fr/l i0 , h fr/l i1 , · · · , h fr/l
in ]. According to the fuse representation, we calcu-
late the distribution of the left or right boundary:
p r/l i = Softmax(h
fr/l i ) (4)
Furthermore, we can get the label probability of the query by the i-th query belonging to label c:
pci = exp(hqiw c i + b c i )∑
c′∈C exp(h q iw c′ i + b c′ i )
(5)
where wc ′ i and b c ′ i are learnable parameter.
Finally, the i-th query predicts result is (τ li , τ r i , τ c i ). τ l i = argmax(p l i) and τ r i = argmax(pri ) are the left and right boundary, τ c i = argmaxc(p c i ) is the sentence label. Note that a special predicate label ϕ is included to indicate no sentence.

heading: 3.4 Bipartite Matching Loss
text: The main challenge of training is measuring the difference between the M decoding results Ŷ and gold sentence set Y in an end-to-end manner. We introduce a bipartite matching loss to overcome this challenge. The calculation of the loss can be broken down into two stages: finding the optimal matching and then calculating the loss based on the optimal matching.
Finding the Optimal Matching. We find the optimal matching between gold set Y and the model output Ŷ by minimizing the cost between them. Notably, a query only can assign one instance in gold set, and vice versa. Since the model predicts results size M larger than the gold sentence set size, we first pad Y to the size M with ϕ. Then the cost of assigning Ŷi with Yj is defined as:
Cmatch(Ŷi, Yj) = −1{ci ̸=ϕ}[prj(ri) +plj(li) + p c j(ci)]
(6)
Finally, we get the optimal permutation element of o∗ with the lowest cost, which is defined as:
o∗ = argmin o∈OM
M∑
i
Cmatch(Ŷo(i), Y j) (7)
where OM is the space of all M -length permutations and OM increases as M increases, resulting in computational efficiency challenges. To obtain the optimal assignment o∗ efficiently, we use the Hungarian algorithm (Kuhn, 1955). With this algorithm, the optimal matching o∗ can be easily computed in polynomial time (O(M3)).
Calculating the Loss. After obtaining the optimal matching o∗, we then calculate the loss for all matched pairs in o∗. We define the loss as:
L(Ŷ , Y ) = M∑
i
{−logpco∗(i)(ci) + 1{ci ̸=ϕ}[
−logplo∗(i)(li)− logpro∗(i)(ri)]} (8)

heading: 4 Experiment

heading: 4.1 Datasets and Evaluation Metrics
text: To demonstrate the effectiveness of our model, we conduct experiments on five singledocument datasets and a multi-document dataset: CNN/DailyMail (Hermann et al., 2015) is a widely used single-document news summarization dataset
containing article-highlight pairs. XSum (Narayan et al., 2018a) concludes one-sentence summaries of online articles from BBC. Reddit (Kim et al., 2019) is collected from social media platforms with weak lead bias and strong abstractive features. WikiHow (Koupaee and Wang, 2018) is a dataset extracted and constructed from an online knowledge base covering a wide range of topics and with high diversity styles. PubMed (Cohan et al., 2018) is a long-form dataset of scientific papers, and we use the truncated version like (Gu et al., 2022). Multi-News (Fabbri et al., 2019) is a multi-document news summarization dataset. More statistical information about the datasets we used in the experiment is shown in Table 1.
We evaluate the quality of generated summaries using the popular automatic evaluation method ROUGE (Lin, 2004). In ROUGE, unigram and bigram overlap (ROUGE-1, 2) is used to measure informativeness and the longest common subsequence (ROUGE-L) is used to measure fluency. For simplicity, ROUGE-1, ROUGE-2, and ROUGE-L are abbreviated as R-1, R-2, and R-L, respectively. In addition, we also apply human evaluation, as a complement to the automatic evaluation.

heading: 4.2 Baselines
text: Basic Extractive Methods: LEAD selects the first several sentences as a summary from the source document. ORACLE extracts sentences as a summary from the source document according to the gold labels. BERTEXT (Liu and Lapata, 2019) utilizes pre-trained BERT (Devlin et al., 2019) to get the sentence representation and assign a label to a sentence to decide whether a sentence is in the summary.
Auoregressive selection Methods: BERTEXT + RL (Bae et al., 2019) directly maximizes
summary-level ROUGE scores through reinforcement learning based on BERTEXT. BERTEXT + Tri-Blocking (Liu and Lapata, 2019) introduces trigram blocking during inference based on BERTEXT. DiscoBERT (Xu et al., 2020) focuses on capturing a more semantically rich representation of sentences based on the RST graph to improve the quality of the summary. Stepwise ETCSum (Narayan et al., 2020) enable stepwise summarization by injecting the previously planned summary content into the structured transformer.
Parallel Prediction Methods: MatchSum (Zhong et al., 2020) is a summary-level approach, which selects the best one from the candidate summaries to form the final summary. OTExtSum (Tang et al., 2022) is a non-learning-based approach, which formulates text summarization as an Optimal Transport (OT) problem from document to summary. ThresSumlarge (Jia et al., 2021) picks up sentences simultaneously by a non-autoregressive decoder when predicted sentence probabilities exceed a threshold.

heading: 4.3 Implementation Details
text: We use pre-trained BERT (Devlin et al., 2019) in the encoder. In order to make a fair comparison with other methods, we use the BERT-base version on all datasets in the experiments. The number of stacked transformer blocks in the encoder is set to 2-4, and the batch size is set to 12. The number of stacked transformer blocks in the non-autoregressive decoder is set to 6. We initialize all queries using the normal distribution N (0.0, 0.02). We apply a linear warmup-decay learning rate scheduler. All experiments are conducted on an NVIDIA GeForce RTX 3090.

heading: 5 Results and Analysis

heading: 5.1 Overall Performances
text: Results on CNN/DailyMail Table 2 shows the evaluation results of proposed methods and baselines on the CNN/DailyMail dataset. Compared with the best auto-regressive methods ETCSum, our method achieves +0.82, +0.01, and +1.01 improvements on R-1, R-2, and R-L, respectively. Compared with the best summary-level method MatchSum, our method has +0.40, +0.19, and +0.38 improvements on R-1, R-2, and R-L, respectively.
Besides, we can see that the introduction of RLbased autoregressive decoding slightly improves the quality of the summary, but is less effective than a simple method like Trigram Blocking. Furthermore, the summary-level methods are still remarkably effective extractive summarization methods at present, which consider the entire summary and can be seen as a special non-autoregressive method.
Results on Datasets with Short Summaries We conduct experiments on XSum and Reddit to prove the effectiveness of our method on short datasets. As shown in Table 3, we can see there is a pretty apparent improvement achieved on the XSum and Reddit datasets than baseline methods. In particular, on the Reddit dataset, our method achieves +0.59, +0.48, and +0.3 improvements on R-1, R-2,
and R-L, respectively. We think this is due to the fact that the token length of the source documents in the two dataset are shorter than other datasets, which enable the model is easy to get contextual sentence representation.
Results on Datasets with Long Summaries Table 4 shows the results on WikiHow, Pubmed, and Multi-News, where Multi-News is a multidocument dataset. According to our intuition, the summary of these datasets has a larger set of sentences compared with other datasets, which requires more challenges to the model to find the correct set of sentences to construct the summary. From the results, we can see that our method has slight improvements on PubMed and Multi-News datasets, which is consistent with our intuition. For example, on the Multi-News dataset, compared with the summary-level method MatchSum our approach achieves +0.13, +0.29, and +0.11 on R1, R-2, and R-L, respectively. On the WikiHow dataset, we achieve comparable results to MatchSum. We think there are two reasons for the result. First, WikiHow has longer summaries compared to Reddit dataset, even though they are both datasets with abstract characteristics. Second, unlike CNN/DailyMail where the main information is more concentrated, that in WikiHow is more scattered.

heading: 5.2 Ablation Studies & Analysis
text: To demonstrate the effectiveness of the components in the model, we performed a series of ablation experiments on the Reddit dataset.
Effects of Model Architecture To verify the effectiveness of the model architecture of SetSum, we remove the parallel decoder and find a significant drop of model results. Specifically, the results shown in Table 5 dropped by 1.38, 0.64, and 1.01 at R-1, R-2, and R-L, respectively, which means that the encoder-based sentence level classifier is prone to select the incorrect sentences. After applying Triam Blocking on an encoder-based sentence-level classifier, the results have improved slightly, which indicates that there is redundant in the results. This occurs because when the parallel decoder is removed, the sentences selected by the method are independent without any dependency. This demonstrates that the parallel decoder plays an extremely important role in the SetSum model.
Effects of Bipartite Matching Loss In our experiments, we introduce bipartite matching loss to address the challenge of measuring the difference between the decoding results and the gold sentence set in an end-to-end manner. We investigate its effect in detail. Specifically, we compare bipartite matching loss with widely used cross-entropy loss. However, there is an issue that the number of decoder prediction M is larger than the gold label N , which makes it impossible to use the cross-entropy loss directly. To address the problem, we adopt two strategies to sort gold sentence labels: Fix Order and Random Order. Fix Order means we keep the original order of sentences in the source document and keep unchanged during training then pad to query size M . Random Order means we randomly sort gold sentence labels for each document before
training then pad to query size M . From the results in Table 5, we find: (1) Compared with the Fix Order strategy, simply shuffling (Random Order) will not improve the performance. (2) Compared with Fix Order and Random Order, introducing bipartite matching loss gains 1.44, 0.67, and 1.22 in R-1, R-2, and R-L, respectively, which verifies the effectiveness of bipartite matching loss.
Effects of Sentence Query The sentence query is the most important part of the SetSum, and we conduct a series of experiments to demonstrate its effectiveness. To explore the learning ability of entity queries, we freeze the parameter of sentence queries during training. The results are shown in Table 5 dropped by 1.31, 0.60, and 1.06 at R-1, R-2, and R-L, respectively, which indicates that the sentence queries do learn the patterns of summary. To verify the effect of query size M on the model, we conduct comparison experiments in Table 6. We can see that the effect of the model does not always increase as M is added, but first increases and then decreases. This occurs most likely because when M is small, the query cannot fully learn the pattern of summary, but as M becomes larger and larger, the query learns the noise instead. Eventually, the query number M in our experiments is set to 60.
Effect of Different Decoder Layers To investigate the importance of the decoder, we explore the effect of decoder layers on the results, as shown in Table 7. We can see that the model performs better as the number of decoder layers increases. In general, the model is more capable of learning as the number of decoder layers is stacked. Furthermore, the results with the sentence level interaction are always better than the interaction at the token level, regardless of the number of layers, and the effect is more obvious as the number of layers increases. For example, when we change the interaction from Q-S to Q-T at layer 6, R-1 drops by 0.75, but at layer 4, R-1 only drops by 0.22.

heading: 5.3 Human Evaluation
text: We also conduct human evaluation following (Chen et al., 2021) on CNN/DailyMail dataset. We invite 2 volunteers who are major in journalism to review the output summaries of several representative models independently. Specifically, we select 100 samples from the CNN/DailyMail dataset, volunteers are asked to rank summaries produced by BERTEXT (Liu and Lapata, 2019), MatchSum (Zhong et al., 2020) and our SetSum according to the following criteria: (1) Informativeness: The summary should preserve the main meaning of the
original document (2) Coherence: The sentences in the summary should be coherent with each other. All of the systems were ranked by 1, 2, and 3 with 3, 2, and 1 scores, respectively. Finally, we get a weighted average score for each system to measure the overall quality of the summary. Results are shown in Table 8. From the results, we can see that the summary obtained by our SetSum outperforms other methods in terms of informativeness and coherence. In addition, the 4.95% improvement in coherence is more obvious than the 4.39% improvement in informativeness, which indicates that our system can learn more dependencies between summary sentences in addition to improving the informativeness of summaries. The results of human evaluation further validate the effectiveness of our method.

heading: 6 Conclusion
text: In this paper, we propose a set prediction network for extractive summarization task. Compared with previous sequence labeling methods, our approach formulates extractive summarization as a sentence set prediction problem. In our approach, a set of sentence queries are fed into a non-autoregressive decoder, which then predicts all sentences within the summary in parallel. To measure the difference between the parallel prediction results and the gold labels, we apply a bipartite matching loss to train the model. To demonstrate the effectiveness of our approach, we conduct experiments on single-document and multi-document datasets. The experimental results demonstrate that our method outperforms the previous state-of-the-art models.
Limitations
We propose a set prediction network for the extractive summarization task, which has worked well on some datasets but still has some limitations. Firstly, due to the use of pre-train BERT in the document encoder, our method is inadequate for long text summarization tasks. In general, the text length of a long document is much longer, so the model needs to be more capable to capture the dependency. Next, we will extend the method to long document summarization tasks. Secondly, the queries in the decoder are initialized with a normal distribution. If we can initialize the queries with the prior knowledge, our method may be able to find the set of sentences of the summary more accurately, which is another direction we need to focus on in the future.

heading: Acknowledgements
text: This work is supported by the Key Research and Development Program of Zhejiang Province, China (No. 2023C01152), the Fundamental Research Funds for the Central Universities (No. 226-2022- 00143), and MOE Engineering Research Center of Digital Library.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Datasets and Evaluation Metrics

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Datasets and Evaluation Metrics
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Datasets and Evaluation Metrics
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 1
C 3 Did you run computational experiments? Implementation Details
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Our method is not a parametric sensitive method.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Implementation Details

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Results and Analysis
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? The package is open source
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Human Evaluation

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Human Evaluation

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Human Evaluation

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Human Evaluation
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: If we can initialize the queries with the prior knowledge, our method may be able to find the set of sentences of the summary more accurately, which is another direction we need to focus on in the future