SlowGuess/NAACL_2022_OCR huggingface dataset:
As seen in Table 3, our abstractive model, denoted ProClusterabs for Propositional Clustering, surpasses all abstractive baselines by a large margin in all measures on both TAC 2011 and DUC 2004. Moreover, while the abstractive system scores were typically inferior to extractive system scores, ProClusterabs notably outperforms all extractive baselines in both benchmarks. Overall, our ProClusterabs provides the new abstractive MDS state-of-the-art score in this setting. In Figure 4 we present an example of a ProClusterabs system summary along with previous abstractive and ex
tractive state-of-the-art system summaries and the reference summary.
As said in §3.6, we also developed an extractive version, denoted ProCluster $_{\text{ext}}$ . As ProCluster $_{\text{ext}}$  selects document propositions that have the highest overlap with ProCluster $_{\text{abs}}$  sentences, ProCluster $_{\text{ext}}$  achieves similar scores to ProCluster $_{\text{abs}}$ , yielding the new extractive MDS state-of-the-art results.
For comparison we selected strong baselines, including previous state-of-the-art in this setup, in both the extractive and abstractive settings. See in Appendix §C for more concise details over each baseline. For reference, we also present a proposition-based extractive upperbound for each dataset (Oracleprop), where document propositions were selected greedily to maximize ROUGE-1F-1 + ROUGE-2F-1 with respect to the reference summaries.
To better apprehend the contribution of each of the steps in our pipeline, Table 5 presents results of the system when applying partial pipelines.
First,  $Salience_{prop}$  generates summaries simply consisting of the highest scoring document propositions, according to the CDLM-based salience model (§3.2). We also trained the salience model on the sentence- rather than the proposition-level, and similarly generated summaries of salient sentences, denoted  $Salience_{sent}$ . The notable improvement of  $Salience_{prop}$  over  $Salience_{sent}$  in both
$^{6}$ ROUGE parameters: -c 95 -24 -U -r 1000 -n 4 -w 1.2 -a -1 100 -m.
Note that methods evaluated with ROUGE recall (instead of F1) or limited to 665 bytes (instead of 100 tokens) are not directly comparable to our approach.
RL-MMR
- An unknown number of cats and dogs suffered kidney failure and about 10 died after eating the affected pet food, menu foods said in announcing the north american recall.
- Menu foods said Saturday it was recalling dog food sold under 46 brands and cat food sold under 37 brands and distributed throughout the united states, canada and mexico.
- Pet owners were worried that the pet food in their cupboards could be deadly after millions of containers of dog and cat food sold at major retailers across north america were recalled.
- The company announced the recall after receiving complaints that cats and dogs were suffering kidney failure.
- Menu Foods recalled dog food sold under 48 brands and cat food
- A major manufacturer of dog and cat food recalled 60 million containers of dog food.
- The products were made by Menu Foods. The company
- Cat food sold under 40 brands including Iams, Nutro and Eukanuba
- The company began using wheat gluten from a new supplier
- The 10 cats and dogs whose deaths have been linked to pet food
- The food was distributed throughout the United States, Canada and Mexico.
- Pet food sold under Wal-Mart, Safeway, Kroger and other store brands.
PG-MMR
- An unknown number of cats and dogs suffered kidney failure and about 10 died after eating the affected pet food, menu foods said in announcing the north american recall.
- Menu foods, the ontario-based company that produced the pet food, said Saturday it was recalling dog food sold under 40 brands including iams, nutro and eukanuba.
- Menu foods is recalling only certain gravity-style pet food in cans and pouches it made from dec. 3 to march 6.
- Pet owners were worried that the pet food in their cupboard may be deadly after millions of containers of dog and cat food sold.
- On Friday, March 16, 2007, Menu Foods of Streetsville, Ontario, began recalling 60 million containers of pet food after reports of ten animal deaths.
- Menu's dog foods are sold under 48 brands and cat foods under 40 brands.
- This company sells its products in the U.S., Canada and Mexico, and provides its products to 17 of the top 20 North American retailers.
- The foods may have become contaminated by wheat gluten purchased from a new supplier which caused kidney failure in the animals.
- The recalls bear code dates of 6339 through 7073.
- The company will compensate owners of deceased animals.
Table 4: The system summaries and reference summary of topic D1104 in TAC 2011.
datasets reveals the advantage of working at the proposition level for exposing salient information. This observation is also apparent when comparing the proposition-based oracle  $(Oracle_{prop})$  to the sentence-based oracle method  $(Oracle_{sent})$ . The results indicate that proposition-based systems have a higher ROUGE upperbound across the board, supporting its merit for use in summarization.
Next, we would like to assess the contribution of the clustering step. Therefore, we applied Salience<sub>prop</sub> followed by clustering and ranking of clusters (Sections 3.2, 3.3 and 3.4), while leaving the fusion step aside. From each cluster we then select the proposition with the highest salience score to be in the system summary. In both datasets, the clustering stage provides added improvement, suggesting its contribution to our pipeline.
To further demonstrate the potential of our approach, we also present two additional oracle scores for extractive upperbound analysis. First, we examine the potential of optimally selecting cluster representatives for the summary. We greedily select a single representative per cluster following the original cluster ranking (§3.4) that optimizes the overall ROUGE-1F-1 + ROUGE-2F-1 score of all selected representatives with respect to the reference summaries (Oraclecluster-rep). These results express the improvement comparing to our final model (ProClusterabs), that a better cluster repre
sentative choice could produce, i.e., up to  $\sim 2$  R-2 points in TAC 2011 and  $\sim 1$  point in DUC 2004.
Another aspect to examine is the potential of enhanced cluster ranking. To that end, we first selected the highest salience-scoring proposition as a representative from each cluster. Then, we greedily selected representatives, one at a time, that maximized the overall ROUGE-1 $_{\mathrm{F - 1}}$  + ROUGE-2 $_{\mathrm{F - 1}}$  against the reference summaries. Effectively, this points to a greedily optimized cluster choice (Oracle_ranking). The potential improvement of better cluster ranking compared to our final model (ProCluster\_abs) is hence up to  $\sim 5$  R-2 points in TAC 2011 and  $\sim 3$  points in DUC 2004. Indeed, our approach leaves cluster ranking improvement to future work.
Overall, we observe that all components of our multi-step approach are indeed effective for MDS, and that there is a great potential for further improvements within this architecture.
We further assessed our primary system, ProClusterabs, through manual comparison against PG-MMR and RL-MMR, which are state-of-the-art MDS systems in the abstractive and extractive settings (respectively). Crowdworkers on Amazon Mechanical Turk were shown the
<sup>8</sup>https://www.mturk.com
method	R1	R2	RSU4
TAC 2011	Oraclesent	47.53	19.83	22.10
Oracleprop	49.65	21.82	23.19
Oraclecluster-rep	43.40	14.61	17.46
Oracleranking	46.38	17.59	19.88
Saliencesent	37.32	9.59	13.40
Salienceprop	39.92	11.53	15.12
Salienceprop + Clustering	41.05	12.40	15.73
ProClusterabs	41.45	12.75	16.16
DUC 2004	Oraclesent	43.91	14.50	17.39
Oracleprop	46.49	16.16	18.76
Oraclecluster-rep	39.74	10.76	14.56
Oracleranking	43.70	12.92	16.43
Saliencesent	37.38	9.09	12.90
Salienceprop	37.73	8.97	13.18
Salienceprop + Clustering	38.41	9.09	13.56
ProClusterabs	38.71	9.62	14.07
summaries for a given topic from the three systems in arbitrary order, along with a corresponding reference summary. They were asked to rank the systems with respect to Content (content overlap with the reference), Readability (the degree to which a summary is readable and well-understood), Grammaticality (avoiding grammar errors), and Non-Redundancy (avoiding information repetition). Focusing on evaluating our system, we extract from this ranking a pairwise comparison between our ProCluster $_{\text{abs}}$  and each of the two baseline systems, separately. For each topic, this procedure was repeated for each of the four available reference summaries. Each such evaluation instance was judged independently by three workers, taking their majority vote for each pairwise comparison.
Table 6 presents the results of these pairwise comparisons, showing the percentage of cases in which our system was preferred over each one of the two baselines, under each of the four evaluation criteria. As can be seen, our system was favored in all cases, for both datasets. Furthermore, preference is almost always by a large margin, except for Non-Redundancy against RL-MMR, which avoids redundancy at a similar success level. Notably, as our clustering-based method is focused on improving content selection, the large gap in favor of ProClusterabs in the content criterion supports
Table 5: Ablation ROUGE F1 scores on TAC 2011 and DUC 2004. Each additional step in our multi-step method improves the output summaries. The Oracle results indicate the potential of our approach. Specifically, the benefit of summarizing on the proposition level is quite evident.
method	Content	Read.	Grammar	Non-Red.
TAC	PG-MMR	93%	84%	81%	72%
RL-MMR	82%	70%	74%	52%
DUC	PG-MMR	81%	83%	82%	76%
RL-MMR	70%	72%	71%	54%
Table 6: Human pairwise comparisons between ProCluster $_{\mathrm{abs}}$  and each of the two prior baseline systems, over the TAC 2011 and DUC 2004 datasets. The cells in a row show the percentage of cases in which our system was preferred over the corresponding baseline, under each of the four evaluation criteria: content, readability, grammaticality and non-redundancy.
System	unigram	bigram	trigram	sent.
TAC	PG-MMR	98.36	94.42	91.97	50.11
ProClusterabs	99.08	91.40	81.07	24.39
Ref. Summs.	90.27	53.17	29.66	1.48
DUC	PG-MMR	98.34	94.99	90.91	50.82
ProClusterabs	98.86	89.72	78.28	23.50
Ref. Summs.	88.41	44.27	18.65	0.13
Table 7: Percentage of n-gram/sentence overlap between summaries and source documents in TAC 2011 and DUC 2004. Compared to PG-MMR, our system has substantially less sequential overlap, indicating its increased abstractiveness. Reference summaries are naturally highly abstractive.
its advantage, consistently with our ROUGE-score advantage in the automatic evaluation (§4.2).
While our summaries are (somewhat non-conventionally) structured as bullet-style lists of propositions rather than a coherent paragraph, evaluators preferred our style of summarization in terms of readability. Moreover, as Table 7 points out, ProClusterabs appears to be more abstractive than PG-MMR, as suggested by the reduced ngram and sentence overlap with source documents. Specifically, about half of the system summary sentences of PG-MMR are fully copied, compared to about a quarter in our method. While the intensified abstractiveness of our summaries could have potentially hindered readability, our system was nevertheless preferred along this aspect as well.
Our approach leaves fertile ground for further improving readability by fusing several clusters together to generate sentences containing multiple propositions, and by developing sentence planning and ordering models. Compatible training datasets for these models can be derived out of the gold reference summaries, as was done in this work for the salience (§3.2) and fusion (§3.5) models.
5 Paraphrastic Clusters as Summary Evidence
A unique advantage of a cluster-based summary is that each summary sentence is linked explicitly to a group of propositions from which the sentence was generated, in so providing an "explanation", or support evidence, for the output. These cluster explanations can expand the reader's knowledge and provide complementary facts from the nearby source context regarding the information from the generated sentence. Such a feature may be incorporated in interactive summarization systems, as applied in (Shapira et al., 2017), where a user can choose to expand on the facts within a sentence of the presented summary.
To assess the reliability of such feature, we verified that clusters indeed "explain" their generated sentences. To that end, we conducted a crowdsourced annotation, where a worker marked whether a cluster proposition mentions the main idea of its corresponding generated sentence. Each pair was examined by three workers, with the majority vote used for the final decision. On a random selection of  $25\%$  of the clusters, we found that, on average,  $89\%$  and  $84\%$  of a cluster's propositions in DUC 2004 and TAC 2011 support their corresponding generated sentence, with an average cluster size of 3.4 and 4.8 propositions, respectively.
Furthermore, given this strong alignment of a cluster to its generated sentence, a cluster facilitates effective verification of faithfulness of its corresponding generated abstractive sentence. Since the output sentence is based solely on its cluster propositions, the sentence's correctness can be verified against the "explaining" cluster instead of against the full document set. An example of an unfaithful abstraction is marked in red in Table 1. To the best of our knowledge, this is the first attempt for efficient manual assessment of faithfulness in MDS. We conducted a respective evaluation process, through crowdsourcing, to assess the faithfulness of our system summaries. A worker saw a cluster and its generated sentence and marked whether the sentence was faithful to its origin cluster or not. Overall, this task cost a reasonable price of  $240\mathrm{~S~}$  for both the DUC 2004 and TAC 2011 datasets together. Over the full test sets, the annotations showed that  $80\%$  and  $90\%$  of the DUC 2004 and TAC 2011 summary sentences, respectively, were faithful to their corresponding clusters.
We advocate the potential of proposition-level units as a cleaner and more accurate unit for summarization. To that end, we present a new proposition-level pipeline for summarization that includes an accurate paraphrastic propositional clustering component followed by fusion of cluster propositions, to generate concise and coherent summary sentences. Our proposed method outperforms state-of-the-art baselines in both automatic and human evaluation on the DUC and TAC MDS benchmarks. We provide an ablation study that indicates the benefit of each of the pipeline steps, as well as the potential for future improvement. Moreover, we demonstrate the utility of the clustering-based approach for providing source documents explanations and for manually validating summary faithfulness.
The work described herein was supported in part by the PBC fellowship for outstanding PhD candidates in data science, Intel Labs, the Israel Science Foundation grant 2827/21, and by a grant from the Israel Ministry of Science and Technology.
Computation. We ran on 3 GPUs for 20 minutes to finteune each of the salience model and the fusion model.
The summarization model runs 10 minutes on 4 GPUs to generate a summary. Most of the time is spent on the clustering step, in which we calculate the SuperPAL similarity score between all salient proposition pairs.
Dataset. The DUC 2003 and 2004 and TAC 2008-2011 datasets were acquired according to the required NIST guidelines (duc.nist.gov).
Crowdsourcing. All human annotations and evaluations conducted with crowdsourcing were compensated as a  $12\mathrm{~s~}$  per hour wage. We estimated the task payment by completing sample assignments and obtaining the average assignment time.
Kristjan Arumae, Parminder Bhatia, and Fei Liu. 2019. Towards annotating and creating summary highlights at sub-sentence level. In Proceedings of the 2nd Workshop on New Frontiers in Summarization,
pages 64-69, Hong Kong, China. Association for Computational Linguistics.
Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passonneau. 2015. Abstractive multi-document summarization via phrase selection and merging. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1587-1597, Beijing, China. Association for Computational Linguistics.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2648-2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
X. Cai and Wenjie Li. 2013. Ranking through clustering: An integrated approach to multi-document summarization. IEEE Transactions on Audio, Speech, and Language Processing, 21:1424-1433.
Xiaoyan Cai, Wenjie Li, You Ouyang, and Hong Yan. 2010. Simultaneous ranking and clustering of sentences: A reinforcement approach to multi-document summarization. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 134-142, Beijing, China. Coling 2010 Organizing Committee.
Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, page 335-336, New York, NY, USA. Association for Computing Machinery.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, and Fei Liu. 2019. Improving the similarity measure of determinantal point processes for extractive multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1027-1038, Florence, Italy. Association for Computational Linguistics.
Sangwoo Cho, Kaiqiang Song, Chen Li, Dong Yu, Hassan Foroosh, and Fei Liu. 2020. Better highlighting: Creating sub-sentence summary highlights. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6282-6300, Online. Association for Computational Linguistics.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res., 22:457-479.
Ori Ernst, Ori Shapira, Ramakanth Pasunuru, Michael Lepioshkin, Jacob Goldberger, Mohit Bansal, and Ido Dagan. 2021. Summary-source proposition-level alignment: Task, datasets and supervised baseline. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 310-322, Online. Association for Computational Linguistics.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Tanvir Ahmed Fuad, Mir Tafseer Nayeem, Asif Mahmud, and Yllias Chali. 2019. Neural sentence fusion for diversity driven abstractive multi-document summarization. Comput. Speech Lang., 58:216-230.
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A graph based approach to abstractive summarization of highly redundant opinions. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 340-348, Beijing, China. Coling 2010 Organizing Committee.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362-370, Boulder, Colorado. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS.
Hanqi Jin and Xiaojun Wan. 2020. Abstractive multi-document summarization via joint learning with single-document summarization. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2545-2554, Online. Association for Computational Linguistics.
Logan Lebanoff, Franck Dernoncourt, Doo Soon Kim, Walter Chang, and Fei Liu. 2020. A cascade approach to neural abstractive summarization with content selection and fusion. In Proceedings of the 1st
Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 529-535, Suzhou, China. Association for Computational Linguistics.
Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Scoring sentence singletons and pairs for abstractive summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2175-2189, Florence, Italy. Association for Computational Linguistics.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131-4141, Brussels, Belgium. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 137-147, Los Angeles. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Zhengyuan Liu and Nancy Chen. 2019. Exploiting discourse-level segmentation for extractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 116-121, Hong Kong, China. Association for Computational Linguistics.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text & Talk, 8:243 - 281.
Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. Multi-document summarization