SlowGuess/NAACL_2022_OCR huggingface dataset:
with maximal marginal relevance-guided reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1737-1751, Online. Association for Computational Linguistics.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
Mir Tafseer Nayeem, Tanvir Ahmed Fuad, and Yllias Chali. 2018. Abstractive unsupervised multi-document summarization using paraphrastic sentence fusion. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1191-1204, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021. Efficiently summarizing text and graph encodings of multi-document clusters. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4768-4779, Online. Association for Computational Linguistics.
Dragomir R. Radev, Hongyan Jing, Magorzata Sty, and Daniel Tam. 2004. Centroid-based summarization of multiple documents. Inf. Process. Manag., 40:919-938.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Ori Shapira, Hadar Ronen, Meni Adler, Yael Amsterdamer, Judit Bar-Ilan, and Ido Dagan. 2017. Interactive abstractive summarization for event news tweets. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 109–114, Copenhagen, Denmark. Association for Computational Linguistics.
Kaiqiang Song, Lin Zhao, and Fei Liu. 2018. Structureinfused copy mechanisms for abstractive summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1717-1729, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 885-895, New Orleans, Louisiana. Association for Computational Linguistics.
Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond subbasic: Task-focused summarization with sentence simplification and lexical expansion. Information Processing & Management, 43(6):1606-1618. Text Summarization.
Xiaojun Wan and Jianwu Yang. 2008. Multi-document summarization using cluster-based link analysis. In SIGIR '08.
Dingding Wang, Tao Li, Shenghuo Zhu, and C. Ding. 2008. Multi-document summarization via sentence-level semantic analysis and symmetric matrix factorization. In SIGIR '08.
Dingding Wang, Shenghuo Zhu, Tao Li, Yun Chi, and Yihong Gong. 2011. Integrating document clustering and multidocument summarization. ACM Trans. Knowl. Discov. Data, 5:14:1-14:26.
Joe H. Ward. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58:236-244.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Yang Zhang, Yunqing Xia, Yi Liu, and Wenmin Wang. 2015. Clustering sentences with density peaks for multi-document summarization. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1262-1267, Denver, Colorado. Association for Computational Linguistics.
A Data Statistics
Dataset	#Topics	#Sentsper doc	#Wordsper doc
DUC 2003	30	259	6831
DUC 2004	50	265	6987
TAC 2008-2010	138	237	5978
TAC 2011	44	205	5146
Table 8: Dataset statistics, including the number of document sets (i.e. topics) and the average number of sentences or words per document. Number of documents per topic is constant (10) for all datasets.
B.1 Proposition Salience Model
Datasets. For many previous summarization systems these benchmarks were insufficiently large enough for training their models. Consequently, they pretrained on a large scale summarization
dataset, such as CNN/DailyMail (Hermann et al., 2015), and then finetuned on DUC/TAC datasets (e.g., Lebanoff et al., 2018; Mao et al., 2020). In our case, we avoid external sources. However, as DUC training data is much smaller than TAC's (30 topics vs. 138), and it was apparently too small for the salience model training, we adopted the trained salience model for TAC benchmark (that was trained with TAC 2008-2010) as a pretrained model and then finetuned it with DUC 2003. Accordingly, validating the TAC benchmark using DUC 2004 during the salience model training causes data leakage since this model is later finetuned to test on the same DUC 2004. To avoid that, during the salience model training we used part of TAC 2010 that was omitted from training data, as a validation set (instead of DUC 2004).
Training Parameters. We trained the model for 10 epochs with learning rate of 1e-5 and batch size of 6 instances on 3 V100 GPUs (meaning effective batch size was 18).
Training. The CDLM model is fed with a proposition within its document and the other documents in the set. Specifically, since CDLM's input size is limited to 4,096 tokens, it is infeasible to feed the full document set as a long sequence. Therefore, following Lebanoff et al. (2019), only the first 20 sentences of each document are considered. Accordingly, a candidate proposition is input within its full document (up to 20 sentences), while other documents, ordered by their date, are truncated evenly and concatenated to fill the remaining space (9 sentences per document on average).
Each instance contains a proposition marked with start and end special tokens, within its multiple document context. A discontinuous proposition is marked with special tokens before and after each of its parts. In addition, sentence special token separators and document special token separators are used, as required for CDLM.
In order to reduce computation complexity, CDLM uses "local attention" (of 512 tokens) for all tokens, while specific tokens are attended to all 4096 tokens ("global attention"). In our case, we assigned global attention to the CLS token and to the candidate proposition tokens, including their special tokens.
For classification, we have added a binary classifier layer on top of our CDLM. The classification layer gets the CDLM's CLS output representation
Figure 3: An example of OpenIE spans extracted from a sentence. First, a sentence is divided to OpenIE tuples, including a predicate (verb) and its arguments. Then all predicates and their arguments are concatenated together to a full span. This illustration uses AllenNLP's Demo<sup>9</sup>.
concatenated to the sum of the CDLM output representations of the candidate proposition tokens:
$$
C L S \odot \sum_ {i \in P r o p} T _ {i} \tag {1}
$$
where  $T_{i}$  is the CDLM output representative of the  $i$ -th token, and  $Prop$  contains the token indices of the candidate proposition.
As our proposition salience training dataset contains only a few positive (i.e., salient) propositions with respect to all propositions, it creates an unbalanced dataset that may strongly bias the model to give a negative prediction. To cope with this, we randomly filter out  $60\%$  of the non-salient propositions, while over sampling salient propositions until the dataset becomes balanced.
B.2 SuperPAL Usage
In this work we used the SuperPAL model (Ernst et al., 2021) as the similarity metric between propositions for the clustering step ( $\S 3.3$ ), and to create training data for the fusion model ( $\S 3.5$ ). Originally, SuperPAL was tuned with a validation set that contains three topics from DUC 2004 (taken from the full validation set which also contains 7 additional topics, not from DUC 2004). In our setting, it may cause leakage since DUC 2004 is used as the test data. To avoid such leakage, we tuned SuperPAL again without using DUC 2004 topics at all (using the other 7 topics as a validation set).
B.3 Cluster Ranking
For computation time consideration, we set a maximum number of clusters to be selected for each topic. Since in most topics the 100-word limit is
exceeded after 8-10 propositional sentences, we set the maximum number of clusters to 10. Accordingly, the 10 (or fewer) highest ranked clusters are selected for the summary of each topic.
B.4 Fusion Model
Training Parameters. We trained the model for 3 epochs with learning rate of 3e-5 and batch size of 10 instances on 3 V100 GPUs (meaning effective batch size was 30).
We compare our method to several strong abstractive baselines: Opinosis (Ganesan et al., 2010) generates abstracts from salient paths in a word cooccurrence graph; Extract+Rewrite (Song et al., 2018) selects sentences using LexRank and generates for each sentence a title-like summary; PG (See et al., 2017) runs a Pointer-Generator model that includes a sequence-to-sequence network with a copy-mechanism; PG-MMR (Lebanoff et al., 2018) selects representative sentences with MMR (Carbonell and Goldstein, 1998) and fuses them with a PG-based model; Hi-MAP (Fabbri et al., 2019) is a hierarchical version of the PG model that allows calculating sentence-level MMR scores; MDS-Joint-SDS (Jin and Wan, 2020) is a hierarchical encoder-decoder architecture that is trained with SDS and MDS datasets while preserving document boundaries.
We additionally compare to several strong extractive baselines: SumBasic (Vanderwende et al., 2007) extracts phrases with words that appear frequently in the documents; KLSumm (Haghighi and Vanderwende, 2009) extracts sentences that opti
mize KL-divergence; LexRank (Erkan and Radev, 2004) is a graph-based approach where vertices represent sentences, the edges stand for word overlap between sentences, and sentence importance is computed by eigenvector centrality; DPP-CapsComb (Cho et al., 2019) balances between salient sentence extraction and redundancy avoidance by optimizing determinantal point processes (DPP); HL-XLNetSegs and HL-TreeSegs (Cho et al., 2020) are two versions of a DPP-based span highlighting approach that heuristically extracts candidate spans by their probability to begin and end with an EOS token; RL-MMR (Mao et al., 2020) adapts a neural reinforcement learning single document summarization (SDS) approach (Chen and Bansal, 2018) to the multi-document setup and integrates Maximal Margin Relevance (MMR) to avoid redundancy.
We used Amazon Mechanical Turk<sup>10</sup> for all three crowdsourced tasks with a list of 90 pre-selected workers from English speaking countries. These workers accomplished high quality work in other NLP-related tasks we have conducted in the past.
The crowdsourcing instructions of the tasks mentioned in  $\S 4.4$  & 5 are as follows:
D.1 General Summarization System Evaluation.
Read the following four texts (Text A, B, C, and D) and answer the following questions.
Text A:
<Reference summary>
Text B:
$<$  System summary 1>
Text C:
$<$  System summary 2>
Text D:
<System summary 3>
- Which of the texts B, C, or D has the highest content overlap with text A?
- Which of the texts B, C, or D has the 2nd highest content overlap with text A?
- Which of the texts B, C, or D is the most readable and well-understood?
- Which of the texts B, C, or D is the 2nd most readable and well-understood?
- Which of the texts B, C, or D avoids grammar mistakes the best?
- Which of the texts B, C, or D avoids grammar mistakes the 2nd best?
- Which of the texts B, C, or D avoids information repetition the best?
- Which of the texts B, C, or D avoids information repetition the 2nd best?
D.2 Supporting Cluster Evaluation.
Read the following two text spans, and answer the question below.
Span Text A:
<The generated sentence>
Span Text B:
<A proposition from the cluster>
Is the main fact of Span Text A mentioned in Span Text B? (ignoring additional details)
D.3 Faithfulness Evaluation.
Read the following group of text spans A and text span B, and answer the questions below. You can assume that all text spans in group A describe the same event, and therefore can be consolidated together to imply Text Span B.
1. Group of Text Spans A:
- They arrested John.
John was arrested.
Text Span B:
The FBI arrested John
Is the Group of Text Spans A implies the fact in Text Span B?
Text Span B add a detail that is not mentioned in A. Therefore the answer is No.
2. Group of Text Spans A:
- there were 10-12 girls and 15 boys in the schoolhouse
- there were boys and girls in the schoolhouse
there were 1012 girls and 15 boys in the schoolhouse
10https://www.mturk.com
Text Span B contradicts Group A (instead of 10-12 girls it says 1012 girls). Therefore the answer is No.