SlowGuess/NAACL_2022_OCR huggingface dataset:
Ori Ernst $^{1}$ , Avi Caciularu $^{1*}$ , Ori Shapira $^{1*}$ , Ramakanth Pasunuru $^{2}$ , Mohit Bansal $^{2}$ , Jacob Goldberger $^{1}$ , and Ido Dagan $^{1}$
$^{1}$ Bar-Ilan University  $^{2}$ UNC Chapel Hill {oriern,avi.c33, obspp18}@gmail.com {ram,mbansal}  $@$  cs.unc.edu {jacob.goldberger@, dagan@cs.}biu.ac.il
Text clustering methods were traditionally incorporated into multi-document summarization (MDS) as a means for coping with considerable information repetition. Particularly, clusters were leveraged to indicate information saliency as well as to avoid redundancy. Such prior methods focused on clustering sentences, even though closely related sentences usually contain also non-aligned parts. In this work, we revisit the clustering approach, grouping together sub-sentential propositions, aiming at more precise information alignment. Specifically, our method detects salient propositions, clusters them into paraphrastic clusters, and generates a representative sentence for each cluster via text fusion. Our summarization method improves over the previous state-of-the-art MDS method in the DUC 2004 and TAC 2011 datasets, both in automatic ROUGE scores and human preference.<sup>1</sup>
Common information needs are most often satisfied by multiple texts rather than by a single one. Accordingly, there is a rising interest in Multi-Document Summarization (MDS) — generating a summary for a set of topically-related documents. Inherently, MDS needs to address, either explicitly or implicitly, several subtasks embedded in this summarization setting. These include salience detection, redundancy removal, and text generation. While all these subtasks are embedded in Single-Document Summarization (SDS) as well, the challenges are much greater in the multi-document setting, where information is heterogeneous and dispersed, while exhibiting substantial redundancy across linguistically divergent utterances.
An appealing summarization approach that copes with these challenges, and is especially rele
- But the man, Rabei Osman Sayed Ahmed - expected to be the first of 29 defendants to take the stand when the bombing trial begins Thursday in Madrid - also said in the recordings that the attack was carried out according to his plan.
- The trial opened Thursday of 29 mostly Moroccan suspects charged with involvement in the 2004 Madrid train bomb attacks, which killed 191 people and injured 1,824 in the worst terror strike to hit Spain.
- Of the 29 people who go on trial Thursday for the March 2004 Madrid train bombings, seven face some 40,000 years in jail if found guilty.
Figure 1: An example of a cluster of propositions, shown within their source sentence context, from TAC 2011 (topic D1103). Clustering these as sentences would yield noisy unaligned information, however grouping together only the marked propositions keeps information alignment clean. The first sentence is illustratively divided into propositions, where only one of them is aligned to those in the other sentences.
vant for MDS, is clustering-based summarization. In such an approach, the goal is to cluster redundant paraphrastic pieces of information across the texts, which roughly convey the same meaning. Repetition of information across texts, as captured by paraphrastic clustering, typically indicates its importance, and can be leveraged for salience detection. Moreover, representing a paraphrastic cluster may facilitate generating a corresponding summary that eliminates repetitions while fusing together complementary details within the cluster.
Traditionally, clustering-based approaches were widely used for summarization, mostly in extractive and unsupervised settings (Radev et al., 2004; Zhang et al., 2015; Nayeem et al., 2018). Notably, most of these works generated sentence-based clusters, which tend to be noisy since a sentence typically consists of several units of information that only partially overlap with other cluster sentences. As a result, such clusters often capture topically related sentences rather than paraphrases. Figure 1 exemplifies such a noisy cluster, which does contain paraphrastic propositions (marked in blue) within their full sentences (marked in black). Another line of research in summarization coped with
$^{1}$ Our code and model are publicly available at https://github.com/oriern/ProCluster.
Human Language Technologies, pages 1765 - 1779
such noisy sentence-based setting, and looked into the use of sub-sentential units for summarization, e.g., Li et al. (2016) summarizes with Elementary Discourse Units (EDUs), while Ernst et al. (2021) endorse using Open Information Extraction (OpenIE) -based propositions (Stanovsky et al., 2018) for summarization.
In this paper, we revisit and combine the clustering-based approaches along with subsentential setting, two research lines that were explored only individually and rather scarcely in recent years. Specifically, we apply clustering-based summarization at the more fine-grained propositional level, which avoids grouping non-aligned texts, yielding accurate paraphrastic clusters. These clusters also provide better control over the generated summary sentences – as the generation component is only required to fuse similar propositions.
Our model (§3) leverages gold reference summaries to derive training datasets for several summarization sub-tasks. First, salient document propositions were extracted, to train a salience model, by greedily maximizing alignment with the reference summaries. Then, an available proposition similarity model, trained from summary-source alignments (Ernst et al., 2021), provides the basis for agglomerative clustering (Ward, 1963). Finally, we created training data for a BART-based model for sentence fusion (Lewis et al., 2020) by aligning reference summary propositions with source proposition clusters. Similar to many other works, we leave inter-sentence coherence and sentence planning and ordering outside the scope of the current paper. Accordingly, our process produces a bullet-style summary of individual concise and coherent sentences.
Overall, our experiments (§4) show that this multi-step model outperforms strong recent end-to-end solutions, which do not include explicit modeling of propositions and information redundancy. To the best of our knowledge, our approach achieves state-of-the-art results in our setting on the DUC 2004 and TAC 2011 datasets, with an improvement of more than 1.5 and 4 ROUGE-1 F1 points respectively, over the previous best approach. Finally, we also suggest (§5) that clustering-based methods provide "explanations", or supporting evidence, for each generated sentence, in the form of the source cluster propositions (see an example in Table 1).
Clustering-based summarization. Clustering-based summarization approaches typically involve salience detection while avoiding redundancy. One such approach clustered topically-related sentences, after which cluster properties were leveraged for rating sentence salience (Radev et al., 2004; Wang et al., 2008; Wan and Yang, 2008). Another approach rated sentence salience and clustered sentences simultaneously, iteratively improving the two objectives (Cai et al., 2010; Wang et al., 2011; Cai and Li, 2013; Zhang et al., 2015). Recently, however, clustering methods have been gradually marginalized out, being replaced by neural techniques. More recently though, some approaches (Nayeem et al., 2018; Fuad et al., 2019) presented abstractive clustering-based summarization, where topically-related sentences in each cluster are fused together to generate a summary sentence candidate. While most of previous clustering approaches operated at the noisy sentence level, in our work we present more accurate proposition-level clustering that eventually enhances summarization.
Sub-sentence units in summarization. While many summarization approaches extract full document sentences, either for extractive summarization or as an intermediate step for abstractive summarization, there are methods that operated the sub-sentential level. Li et al. (2016); Liu and Chen (2019); and Xu et al. (2020) produced extractive summaries consisting of Elementary Discourse Units (EDUs) – clauses comprising a discourse unit according to Rhetorical Structure Theory (RST) (Mann and Thompson, 1988). Such extractive approaches usually focus on content selection, possibly disregarding the inferior coherence arising from the concatenation of sub-sentence units. Accordingly, Arumae et al. (2019) established the highlighting task, where salient sub-sentence units are marked within their document to provide surrounding context. Recently, Cho et al. (2020) proposed identifying heuristically self-contained subsentence units for the highlighting task.
Abstractive approaches have been extracting subsentence units as a preliminary step for generation. Such units range from words (Lebanoff et al., 2020; Gehrmann et al., 2018), to noun or verb phrases (Bing et al., 2015), to OpenIE propositions (Pasunuru et al., 2021). In our work, we follow the same extract-then-generate pipeline, using Ope
Figure 2: Our multi-document summarization process. (a) All propositions are extracted (OpenIE; Stanovsky et al., 2018) from the documents. (b) Propositions are classified by a salience score (fine-tuned CDLM; Caciularu et al., 2021). (c) Salient propositions are clustered (fine-tuned SuperPAL; Ernst et al., 2021), forming groups of paraphrastic information units. (d) Clusters are ranked, as an indicator for information importance. (e) For each cluster, its propositions are fused (fine-tuned BART; Lewis et al., 2020) to generate a concise and coherent abstractive sentence. (f) The output summary is obtained as a bullet-style ranked list of the concise sentences.
nIEs (Stanovsky et al., 2018) as propositions. Since propositions are meant to contain single standalone facts consisting of a main predicate and its arguments, they are beneficial for grouping mostly overlapping paraphrases (unlike sentential paraphrases). In addition, propositions extracted with OpenIE can be noncontiguous, while alternative options, like EDUs, are limited to contiguous sequences.
This section first provides an overview of our method, followed by subsections describing its components. We follow previous clustering-based approaches, where text segments are first clustered into semantically similar groups, exploiting redundancy as a salience signal. Then, each group is fused to generate a merged sentence, while avoiding redundancy. As we operate at the proposition-level, we first extract all propositions from the input documents (§3.1). Then, to facilitate the clustering step, we filter out non-salient propositions using a salience model (§3.2). Next, salient propositions are clustered based on their semantic similarity (§3.3). The largest clusters, whose information was most repeated, are selected to be included in the summary (§3.4). Finally, each cluster is fused to form a sentence for a bullet-style abstractive summary (§3.5). In addition, we provide an extractive version where a representative (source) proposition
is selected from each cluster (3.6). Overall, clustering explicit propositions induces a multi-step process that requires dedicated training data for certain steps. To that end, we derive new training datasets for the salience detection and the fusion models from the original gold summaries. The full pipeline is illustrated in Figure 2, where additional implementation details are in  $\S B$  in the Appendix.
3.1 Proposition Extraction
Aiming to generate proposition-based summaries, we first extract all propositions from the source documents using Open Information Extraction (OpenIE) (Stanovsky et al., 2018) $^{2}$ , following Ernst et al. (2021). To convert an OpenIE tuple containing a predicate and its arguments into a proposition string, we simply concatenate them by their original order, as illustrated in Figure 3 in the Appendix.
3.2 Proposition Salience Model
To facilitate the clustering stage, we first aim to filter non-salient propositions by a supervised model. To that end, we derive gold labels for proposition salience from the existing reference summaries. Specifically, we select greedily propositions that maximize ROUGE-1 $_{\mathrm{F - 1}} +$  ROUGE-2 $_{\mathrm{F - 1}}$  against their reference summaries (Nallapati et al., 2017; Liu and Lapata, 2019) and marked them as salient.
$^{2}$ https://demo.allennlp.org/open-information-extraction
- The agreement will make Hun Sen prime minister and Ranariddh president of the National Assembly.
- ...to a coalition deal...will make Hun Sen sole prime minister and Ranariddh president of the National Assembly.
- The deal, which will make Hun Sen prime minister and Ranariddh president of the National Assembly...ended more than three months of political deadlock
- Last week...Hun Sen's Cambodian People's Party and Ranariddh's FUNCINPEC party agreed to form a coalition that would leave Hun Sen as sole prime minister and make the prince president of the National Assembly.
- In a long-elusive compromise...opposition leader Prince Norodom Ranariddh will become president of the National Assembly
... opposition party leaders Prince Norodom Ranariddh and Sam Rainsy are out of the country
- Sam Rainsy and his then-ally Prince Norodom Ranariddh led an exodus of opposition lawmakers out of Cambodia
- Opposition leaders Prince Norodom Ranariddh and Sam Rainsy...said they could not negotiate freely in Cambodia
- Opposition leaders Prince Norodom Ranariddh and Sam Rainsy...citing Hun Sen's threats
ProCluster summary
A. The deal will make Hun Sen prime minister and Ranariddh president of the National Assembly
B. The opposition party leaders Prince Norodom Ranariddh and Sam Rainsy are out of the country
C. Hun Sen's Cambodian People's Party narrowly won the election.
D. A series of negotiations to forge a new government failed.
E. The U.N. accused him of being behind a plot against his life.
F. Hun Sen ousted Ranariddh in a coup last year.
G. The opposition alleging widespread fraud and intimidation by the CPP
H. The parties have refused to enter into a coalition with Hun Sen until their allegations of election fraud have been thoroughly investigated.
- Hun Sen's Cambodian People's Party narrowly won the polls
- Hun Sen's ruling party narrowly won a majority in elections in July
- Hun Sen's Cambodian People's Party narrowly won the election.
- the ruling party narrowly won.
Cluster D
- A series of negotiations to forge a new government
...any...in deadlocked negotiations to form a government.
- A series of negotiations to forge a new government have failed.
Cluster E
- Hun Sen accused him of being behind a plot against his life.
- Sam Rainsy...to take refuge in a U.N. office in September to avoid arrest after Hun Sen accused him of
- Sam Rainsy...to avoid arrest after Hun Sen accused him of being behind a plot against his life.
Cluster F
- Hun Sen ousted Ranariddh in a coup.
- The men served as co-prime ministers until Hun Sen overthrew Ranariddh in a coup last year.
- Hun Sen overthrew Ranariddh in a coup last year.
Reference Summary
Cambodia King Norodom Sihanouk praised formation of a coalition of the Countries top two political parties, leaving strongman Hun Sen as Prime Minister and opposition leader Prince Norodom Ranariddh president of the National Assembly.
The announcement comes after months of bitter argument following the failure of any party to attain the required quota to form a government.
Opposition leader Sam Rainey was seeking assurances that he and his party members would not be arrested if they return to Cambodia.
Rainey had been accused by Hun Sen of being behind an assassination attempt against him during massive street demonstrations in September.
Table 1: The proposition clusters and system and reference summaries for DUC 2004, topic D30001. Each summary sentence (lower left box) was fused from its corresponding cluster (top boxes) that also provides supporting source evidence. An example of an unfaithful abstraction is marked in red.
Using this derived training data, we fine-tuned the Cross-Document Language Model (CDLM) (Caciularu et al., 2021) as a binary classifier for predicting whether a proposition is salient or not. Propositions with a salience score below a certain threshold were filtered out. The threshold was optimized with the full pipeline against the final ROUGE score on the validation set. All propositions contained in the clusters in Table 1 are examples of predicted salient propositions. We chose to use CDLM as it was pretrained with sets of related documents, and was hence shown to operate well over several downstream tasks in the multi-document setting (e.g., cross-document coreference resolution and multi-document classification).
Next, all salient propositions are clustered to semantically similar groups. Clusters of paraphrastic
propositions are advantageous for summarization as they can assist in avoiding redundant information in an output summary. Furthermore, paraphrastic clustering offers redundancy as an additional indicator for saliency, while the former salience model (§3.2) does not utilize repetitions explicitly. To cluster propositions we utilize SuperPAL (Ernst et al., 2021), a binary classifier that measures paraphrastic similarity between two propositions. All pairs of salient propositions are scored with SuperPAL, over which standard agglomerative clustering (Ward, 1963) is applied. Examples of generated clusters are presented in Table 1.
3.4 Cluster Ranking
The resulting proposition clusters are next ranked according to cluster-based properties. We examined various features, listed in Table 2, on our validation sets. The features examined include: aver
age of ROUGE scores between all propositions in a cluster ('Avg. ROUGE'), average of SuperPAL scores between all propositions in a cluster ('Avg. SuperPAL'), average of the salience model scores of cluster propositions ('Avg. salience'), minimal position (in a document) of cluster propositions ('Min. position'), and cluster size ('Cluster size').
For each feature, (1) clusters were ranked according to the feature, (2) the proposition with the highest salience model score ( $\S 3.2$ ) was selected from each cluster as a cluster representative, (3) the representatives from the highest ranked clusters were concatenated to obtain a system summary. We also measured combinations of two features ('Cluster size + Min. position' for example), where the first feature is used for primary ranking, and the second feature is used for secondary ranking in case of a tie. In all options, if a tie is still remained, further ranking between clusters is resolved according to the maximal proposition salience score of each cluster. The resulting ROUGE scores of these summaries on validation sets are presented in Table 2. $^{3}$  We found that 'Cluster size' yields the best ROUGE scores as a single feature, and 'Min. position' further improves results as a secondary tie breaking ranking feature. Intuitively, a large cluster represents redundancy of information across documents thus likely to indicate higher importance.
3.5 Cluster Fusion
Next, we would like to merge the paraphrastic propositions in each cluster, while consolidating complementary details, to generate a new coherent summary sentence. As mentioned, this approach helps avoiding redundancy, since redundant information is concentrated separately in each cluster.
To train a cluster fusion model, we derived training data automatically from the reference summaries, by leveraging the SuperPAL model (Ernst et al., 2021) (which was also employed in §3.3). This time, the model is used for measuring the similarity between each of the cluster propositions (that were extracted from the documents) and each of the propositions extracted from the reference summaries. The reference summary proposition with the highest average similarity score to all cluster propositions was selected as the aligned summary proposition of the cluster. This summary proposi
Cluster Feature	DUC 2004	TAC 2011
R1	R2	R1	R2
Avg. ROUGE	35.9	7.48	38.14	9.93
Avg. salience	35.5	7.98	41.18	12.55
Min. position	37.25	8.89	38.86	11.37
Avg. SuperPAL	37.41	8.90	41.22	12.59
Cluster size	37.58	9.01	41.35	12.49
Cluster size + Avg. SuperPAL	37.54	8.96	41.45	12.71
Cluster size + Avg. salience	37.77	9.09	41.44	12.62
Cluster size + Min. position	38.05	9.21	41.68	12.78
Table 2: ROUGE F1 results on validation sets when ranking clusters according to differing features (DUC 2004 is the validation set of TAC 2011 and vice versa). Two combined features means ranking on the first feature, and breaking ties with the second feature.
tion was used as the target output for training the generation model. Although these target OpenIE propositions may be ungrammatical or non-fluent, a human examination has shown that BART tends to produce full coherent sentences (mostly containing only a single proposition), even though it was finetuned over OpenIE extractions as target. Examples of coherent generated sentences can be seen in Table 1.
Accordingly, we fine-tuned a BART generation model (Lewis et al., 2020) with this dedicated training data. As input, the model receives cluster propositions, ordered by their predicted salience score ( $\S 3.2$ ) and separated with special tokens. The final bullet-style summary is produced by appending generated sentences from the ranked clusters until the desired word-limit is reached.
3.6 Extractive Summarization Version
To support extractive summarization settings, for example when hallucination is forbidden, we created a corresponding extractive version of our method. In this version, we extracted a representative proposition for each cluster, which was chosen according to the highest word overlap with the sentence that was fused from this cluster by our abstractive version.
Datasets. We train and test our summarizer with the challenging DUC and TAC MDS benchmarks.
<sup>3</sup>We also tried training a regression model on a mixture of features that should predict the ROUGE score of a proposition, but results were comparable. Bettering the ranking process is left for future work.
4For the Hi-MAP and MDS-Joint-SDS approaches we present only DUC 2004 scores since TAC 2011 scores are not available for them.
5The outputs of DPP-Caps (Cho et al., 2019), HL-XLNet and HL-Tree (Cho et al., 2020) were re-evaluated using author released output.
Method	TAC 2011	DUC 2004
R1	R2	RSU4	R1	R2	RSU4
absentive	Opinosis (Ganesan et al., 2010)	25.15	5.12	8.12	27.07	5.03	8.63
Extract+Rewrite (Song et al., 2018)	29.07	6.11	9.20	28.9	5.33	8.76
PG (See et al., 2017)	31.44	6.40	10.20	31.43	6.03	10.01
Hi-MAP4 (Fabbri et al., 2019)	-	-	-	35.78	8.90	11.43
PG-MMR (Lebanoff et al., 2018)	37.17	10.72	14.16	36.88	8.73	12.64
MDS-Joint-SDS4 (Jin and Wan, 2020)	-	-	-	37.24	8.60	12.67
ProClusterabs (Ours)	41.45	12.75	16.16	38.71	9.62	14.07
extractive	SumBasic (Vanderwende et al., 2007)	31.58	6.06	10.06	29.48	4.25	8.64
KLSumm (Haghighi and Vanderwende, 2009)	31.23	7.07	10.56	31.04	6.03	10.23
LexRank (Erkan and Radev, 2004)	33.10	7.50	11.13	34.44	7.11	11.19
HL-XLNetSegs5 (Cho et al., 2020)	37.32	10.24	13.54	36.73	9.10	12.63
HL-TreeSegs5 (Cho et al., 2020)	36.70	9.68	13.14	38.29	10.04	13.57
DPP-Caps-Comb5 (Cho et al., 2019)	38.14	11.18	14.41	38.26	9.76	13.64
RL-MMR (Mao et al., 2020)	39.65	11.44	15.02	38.56	10.02	13.80
ProClusterext (Ours)	40.98	12.40	15.77	38.73	9.64	13.89
Oracleprop	49.65	21.82	23.19	46.49	16.16	18.76
Table 3: Automatic ROUGE F1 evaluation scores on the TAC 2011 & DUC 2004 MDS test sets. Our solutions (ProCluster) improve over the previous state-of-the-art methods both in the abstractive and extractive settings. Notably, our abstractive approach also surpasses the best extractive ones.
Specifically, following standard convention (Mao et al., 2020; Cho et al., 2019), we test on DUC 2004 using DUC 2003 for training, and on TAC 2011 using TAC 2008/2009/2010 for training. These sets contain between 30 and 50 topics each. For validation sets, we used DUC 2004 for the TAC benchmark and TAC 2011 for the DUC benchmark.
Automatic evaluation metric. Following common practice, we evaluate and compare our summarization system with ROUGE-1/2/SU4 F1 measures (Lin, 2004). Stopwords are not removed, and the output summary is limited to 100 words. $^{6}$