Promoting Topic Coherence and Inter-Document Consorts in Multi-Document Summarization via Simplicial Complex and Sheaf Graph
Authors: Yash Kumar Atri
Arun Iyer
Tanmoy Chakraborty
Vikram Goyal
Venue: EMNLP 2023 Main (2023)
Keywords: multi document summarization
abstractive summarization

Abstract: Multi-document Summarization (MDS) characterizes compressing information from multiple source documents to its succinct summary. An ideal summary should encompass all topics and accurately model cross-document relations expounded upon in the source documents. However, existing systems either impose constraints on the length of tokens during the encoding or falter in capturing the intricate cross-document relationships. These limitations impel the systems to produce summaries that are non-factual and unfaithful, thereby imparting an unfair comprehension of the topic to the readers. To counter these limitations and promote the information equivalence between the source document and generated summary, we propose FIBER, a novel encoder-decoder model that uses pre-trained BART to comprehensively analyze linguistic nuances, simplicial complex layer to apprehend inherent properties that transcend pairwise associations and sheaf graph attention to effectively capture the heterophilic properties. We benchmark FIBER with eleven baselines over four widely-used MDS datasets -- Multinews, CQASumm, DUC and Opinosis, and show that FIBER achieves consistent performance improvement across all the evaluation metrics (syntactical, semantical and faithfulness). We corroborate these improvements further through qualitative human evaluation.

Promoting Topic Coherence and Inter-Document Consorts in Multi-Document Summarization via Simplicial Complex and Sheaf Graph
Yash Kumar Atri', Arun Iyer", Tanmoy Chakraborty', Vikram Goyal 'IIT-Delli, India; MMicrosoft Research, India; 'ITT Delhi, India vieraj,,vik ac in; ariy@micron wttemmicrooot..oom com; tanchak@i td. ac. in
Abstract
models, often resulting in the generation of sum- maries that are the results of hallucination and they often lack faithfulness (Maynez et al., 2020). The development of Large Language Models (LLMs) such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020) has sig- nificantly advanced the field of text summarization. However, generating factually accurate and faithful summaries remains a persistent challenge (OpenAI, 2023).
Multi-document Summarization (MDS) char- acterizes compressing information from multi- ple source documents to its succinct summary. An ideal summary should encompass all top- ics and accurately model cross-document re- lations expounded upon in the source docu- ments. However, existing systems either im- pose constraints on the length of tokens during the encoding or falter in capturing the intri- cate tross-document relationships. These limi- tations impel the systems produce summaries that non-factual and unfaithful, thereby im- parting an unfair comprehension of the topic to the readers. counter these limitations and promote the information equivalence between the source document and generated summary, we propose FABRIC, novel encoder-decoder model that uses pre-trained BART compre- hensively analyze linguistic nuances, simplicial complex layer to apprehend inherent properties that transcend pairwise associations and sheaf graph attention to effectively capture the het- erophilic properties. We benchmark FABRIC with eleven baselines over four widely-used MDS datasets Multinews, CQASumm, DUC and Opinosis, and show that FABRIC achieves consistent performance improvement across all the evaluation metrics (syntactical, semantical and faithfulness). We corroborate these im- provements further through qualitative human evaluation. The source code available at https://github cww/ccS2ccccc//c/ LOm/LCS2-IITTD/FABRI IIITD/FABIC
Recent studies have focused on enhancing the faithfulness and factuality summarization mod- els, which is broadly categorized into three types - -() post editing models for correcting generated summaries (Fabbri et al., 2022; Balachandran et al., 2022), (ii) using multi-task problems like question- answering (Durmus al., 2020: Deutsch al., 2020), entailment (Roit al., 2023; Pasunuru and Bansal, 2018) etc., and (iii) using external knowl- edge (Huang et al., 2020; Mao et al., 2022; Chen and Yang, 2021; Mao et al., 2022; Lyu et al., 2022) to support the model during summary generation. In contrast to these methods, we propose a novel approach that promotes topic coherence and inter- document connections, all without the need for additional parameters, post-editing, external knowl- edge, or auxiliary tasks.
Additionally, capturing semantic connections and multi-level representation can further aid the model in discerning the redundant and pivotal in- formation in multiple documents. Graph neural networks (Scarselli et al., 2009) are constrained in terms of subpar performance in heterophilic settings' and oversmoothing (Song al., 2023) when utilized with multi-layer neural networks. To mitigate these concerns, we propose FABRIC in which we introduce simplicial complex (Giusti
1 Introduction
Multi-document summarization (MDS) aims to for- mulate a summary that captures the essence and main points of multiple documents on specific topic. In contrast single document summariza- tion (SDS) that focuses on generating summaries from single source, MDS faces additional chal- lenges such as dealing with larger search space, redundant documents, and conflicting opinions. These challenges pose difficulties for deep learning
'In a heterophilic setting, diverse node edge types can complicate the messuge-passing process GNN, therefore affecting its performance.
2 "FABRIC: FAireess using BaRt, simplicial Complex and sheaf

---

al.. 2022) layer and sheaf graph (Hansen and Gebhart, 2020) attention for multi-document sum- marization. Simplicial complexes are employed to apprehend the interconnections among diverse elements of the text, encompassing words, phrases, or sentences. By representing these associations as simplices (geometric shapes formed by combining vertices, edges, and higher-dimensional counter- parts), simplicial complex furnishes structure to examine the connectivity and coherence within the text. On the other hand, sheaf graphs facilitate the assimilation of diverse node types and attributes within the graph structure. Each node can symbol- ize specific element, such as a word or phrase. and convey its own attributes or features. By con- sidering these heterogeneous characteristics, sheaf graphs can apprehend and model the relationships among distinct types of nodes. This empowers FABRIC to comprehend various elements and their relationships, enabling the generation of faithful and accurate summaries.
imput length.
2. We propose FABRIC, novel encoder-decoder model that uses pre-trained BART to compre- hensively analyze linguistic nuances, Simplicial Complex layer to apprehend inherent properties that transcend pairwise associations and sheaf graph attention to more effectively apprehend the heterophilic properties.
3. We evaluate FABRIC using four standard met- rics ROUGE (Lin, 2004), BARTScore (Yuan et al., 2021), FactCC (Kryscinski al., 2020), and SummaC (Laban et al., 2022), to assess the quantitative, and qualitative performances and faithfulness. We showcase that FABRIC fares well against eleven widely-popular ab- stractive baselines. FABRIC beats the best base- line, PRIMERA (Xiao et al., 2022) by +0.34 ROUGE-L on Multinews, +4.90 Rouge-L on CQASumm, +2.94 on DUC and +2.28 Rouge- on the Opinosis dataset. Our qualitative anal- yses on semantic similarity by BARTScore and faithfulness by FactCC and SummaC also show considerable gains compared to the baseline- generated summaries. We further perform an exhaustive human evaluations and comparison with ChatGPT to compare the quality of the system-generated summaries.
Moreover, when dealing with MDS, critical challenge that neural networks often encounter is processing large documents. Many recent studies have attempted to concatenate multiple documents into a flat sequence and train SDS models on them (Fabbri et al., 2019). However, this approach fails to consider the inter-document relationship and the presence redundant long input vectors. Even very large language models like Long-T5 (Guo et al., 2021) and BART-4096 (Lewis et al., 2020), which are capable of handling input vectors be- yond 2000 tokens, face practical limitations due the quadratic growth of input memory space. Furthermore, such extensive global attention may exhibit lower performance compared to alterna- tive methods. To address this issue, we propose a novel approach, called lopic-assisted document segmentation. It involves using segment of the document context vector, compressing it into oracle summary while covering major topics, and appending it to the next segment to generate the subsequent context vector. By employing this method, summarization models can learn the rep- resentation of the document from compressed form, overcoming the challenges associated with processing large documents.
Related Works
Abstractive summarization. The task of text sum- marization has seen remarkable improvement with the introduction of attention (Bahdanau et al., 2015) and transformer (Vaswani et al., 2017) based ap- proaches. Later, transformers coupled with content selection (Gehrmann et al., 2018; Atri et al., 2021), attention approximations (Zaheer et al., 2020), win- dowed and task orientedd attention (Beltagy et al., 2020) were used to improve the performance fur- ther. The introduction of LLMs has also shown in- credible performance by fine-tuning them over few epochs. LMs such as BERT (Devlin et al., 201 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and Pegasus (Zhang et al., 2020) are pre-trained over vast corpus and later finetuned over the summarization datasets to achieve state-of-the-art results. Reinforcement learning (RL) algorithms have also been leveraged further improve the performance of the models. abstractive summa- rization, Lee and Lee (2017); Pasunuru and Bansal (2018): 8); Böhm et al. (2019); Atri et al. (2023b) intro-
In short, contributions as follows.
We propose novel topic assisted document seg- mentation method, which allows any language model to generate contextual vectors for any
Phtpe://opemaa.ccommbbbaaaaaaa

---

duced the notion of how RL-based models can aid in text summarization. Various reward measures like Rouge-L (Paulus et al., 2017), human feedback (Böhm et al., 2019) and combination of rewards (Pasunuru and Bansal, 2018) 8) have been used.
Transformer-based models. LMs like Longformer (Beltagy al., 2020) are able to process docu- ments in a linear complexity. However, they fail to efficiently capture the context of the whole doc- ument. To address this limitation, we propose a new data modelling strategy, called Topic Assisted Document Segmentation with Oracle proxies. At first, the source document D is split into segments D1, D2, D3.... D, based on major topics identi- fied. We use BERTopic (Grootendorst, 2022) for topical identification. Next, we formulate oracle summarics for segment D1 using an Integer Linear Programming (ILP) solver (Gillick et al., 2008) and penalize it with BERTopic (Grootendorst, 2022) to aid the module in generating oracle summaries and to cover all major topics of the document. We ap- pend the oracle summary obtained O₂ to the next segment O1 + D1 and formulate the next oracle summary using only O₃ D1. Each generated segment limited to 768 tokens for BART to pro- cess the input vector effectively. Each segment, therefore, obtains an independent encoder repre- sentation of the document, which when fused with the other modules, is fed to the BART decoder to generate the final summary.
Faithfulness. Existing methods for improving faithfulness include proposing new fact-corrected datasets (Balachandran et al., 2022), ensembling language models (Guo et al., 2022), adversarial techniques (Wu al., 2022), target correction (Fab- bri et al., 2022; Adams et al., 2022; Lee et al., 2022). Natural Language Inference (NLI) mod- els (Laban et al., 2022), rejecting noisy tokens (Cao et al., 2022), controlling irrelevant sentences (Ghoshal et al., 2022), and contrasting candidate generation (Chen al., 2021). Others use auxiliary information from source documents like additional input (Dou et al., 2021), entity (Zhang et al., 2022), knowledge graphs (Lyu et al., 2022: Huang et al., 2020; Zhao et al., 2021) or RL-based approaches with rewards as entailment feedback (Roit et al., 2023), and structured fact extraction (Zhang et al., 2021). Nevertheless, these methodologies either raise question upon the veracity of the datasets, cull out discordant data points or employ extrinsic data sources to alleviate the factual incongruities. Formulating a new dataset entails substantial costs, and winnowing existing ones might diminish the dataset pool for training competitive model. In contrast, our proposed approach surmounts these constraints, capitalizing on the entirety of acces- sible data during training and leveraging the dy- namics of simplicial complexes and sheaf graphs apprehend the inter- and intra-document rela- tionships, thereby engendering exceedingly faithful and factual summaries.
3.2 Simplicial Complex Layer
Summarizing documents does not always entail in- formation in pairwise connections. Consequently, methods based on single-dimensional graphs fall short in capturing the comprehensive context and its intricate relationships. Conversely, the utiliza- tion of simplicial complex layer enables the in- corporation of higher-order graphs, where the in- clusion of simplices (vertices within single-order graph) empowers the modeling of information that extends beyond pairwise associations.
3 Proposed Methodology
A simplicial complex is a mathematical struc- ture that represents relationships between different levels of "simplices." A simplex is a generalization of a triangle (2-simplex) to higher dimensions. In the context of summarization, simplicial complexes used to capture relationships between multiple documents that might be directly connected, by considering their shared properties or interactions. This approach formulates an exceedingly diverse and extensive feature set, facilitating the model's 's ability to comprehend inter-document relations in more comprehensive and nuanced manner.
We present FABRIC, novel multi-encoder-devoode model for multi-document abstractive text summa- rization. Our approach harnesses the power of topic assisted document segments and leverages higher order topological spaces to adeptly extract essential information. Figure 1 shows schematic diagram of FABRIC. This section explains each component of FABRIC in detail.
3.1 Topic Assisted Document Segments
Transforming the entire document to contex- tual vector is very expensive pertaining to the quadratic growth of memory and computations in
We use the simplicial complex self-attention setting similar to (Giusti et al., 2022).
(SC) layer in the

---

Sheaf Attention
BART Encoder
F (v)
(e)
F (U)
X
Probs
F
F
Self Attention
Multi-Head
Add Norm
Forward
Forward
Feed
Feed
A-N and FF
T[X, X
f(Xy X
BART Decoder
Multi-Head
Self Attention
Ext 1
Ext 2
Add & Norm
Seg 1
Seg 2
Seg 3
ILP
Multi-Head
Self Attention
Simplicial Complex
X
Yn
Source Input
Outputs
Figure I: A schematic architecture of FABRIC. We adept the BART encoder and introduce simplicial complex layer and sheaf graph attention and fuse them with multi-head attention of BART. The fused representation is passed the BART decoder to generate candidate summaries.
Mathematically, we define the input SC layer as SC; and learnable weights as w... The transforma- tions are applied to the lower irrotational filter I/', and upper solenoidal filter H The transformations are represented by H [S RRi++,, and Ha [ E RF The representation H' and H are combined for each iteration, and self-attention is computed using the following equa- tions,
output signals, and the non- non-linearity ()(-) are hy- perparameters to be chosen at each layer. The final representation is passed through a feed-forward layer.
3.3 Sheaf Graph Attention
Similar to graph neural networks (GNNs) (Scarselli et al., 2009) and graph attention networks (GATs) (Veličković et al., 2018), the contextual represen- tation of simplicial complex (SC) layer encoun- ters limitations in performance, particularly in het- erophilic settings and over-smoothing. Moreover. relying single module to capture the inter- document relationship may introduce biases in rep- resenting factual knowledge. To overcome this limitation, we introduce a new module, called sheaf graph attention in the MDS setting, taking inspiration from the prior sheaf attention networks (Hansen and Gebhart, 2020).
i,a =0 h/pp (i.)))
(1) (1) (1()
for Ag(") (ENEN
(d)) (5)
h1ppp 1 ((2))
for c N(((
C ai
These representations are normalized using the softmax function (a) softmax,(c""). (w)), and (8) softmax,(e)) (e)).. The attention weights of aj
the SC layer are computed using,
(1) 2 ip Z+1 ai (L.
Σ (1"YZW!
A sheaf is a mathematical construct used to asso- ciate local data with certain topological spaces. a sheaf, data are organized in a way that respects the local structures of the underlying space. In the context of graphs, sheaf graphs extend the idea of traditional graph structures by incorporating not only node and edge connections but also the in- formation associated with these connections. practical terms, sheaf graphs can represent relation- ships between data points while also encoding ad- ditional context or attributes associated with those
where the coefficients of the upper and lower atten- tional Laplacians, [ and I, respectively
obtained The filter weights (wl)), I.p
(d) W (u) Jp"
(())) and the attention mechanism parameters aj
(d)
and a are learnable parameters, while the order j(()) and l(") the filters, the number F of

---

relationships. This makes them suitable for captur- ing complex interactions, semantic relationships. and context in various applications, including natu- ral language processing tasks like summarization. Mathematically, we define the input vector as X, and introduce the learnable weight matrices Wa and Wo The identity matrix is denoted by I. and the sheaf Laplacian is represented as vb.. We perform Kronecker product between I and Wa. The sheaf Laplacian & is defined as a vector in an undirected graph G., where vertices are defined by and edges by e. Mathematically, it can be expressed as follows:
utilizing the encoder representation and the point- ing mechanism. Much like pointer generator, the pointing mechanism enables the model to directly copy text from the source document.
4 Datasets
We benchmark our study on four widely popu- lar datasets. (i) Multinews (Fabbri et al., 2019)
comprises news articles as source documents and human- summaries as target summaries. (ii) CQASumm (Chowdhury and Chakraborty, 2019) dataset consists of question-answer pairs, where the accepted answer is used as the target summary and the remaining answers form the source document. (iii) DUC (DUC, 2002) includes news articles as source documents, with target summaries manu- ally annotated by editors. (iv) Opinosis (Ganesan et al., 2010) combines user reviews from various platforms, where individual opinions serve the source document, while the target summaries are -anotatdd..
Sheas(X) = relu((I- (T @ W.)XW, (1)
Here, X is the data transformation block, and @ is Kronecker product.
3.4 Encoder Setting
The first module introduces document segments via topic guidance, which enables FABRIC to gen- crate multiple target sequence vectors with seg- mented source documents aided with oracle prox- ies. The sequence vectors are learned using the BART encoder model. These feature vectors helps the model to understand the linguistic properties and retain multiple context vectors. The second module introduces the simplician complex layer, helping the model to understand the inter-document relations and aiding the weights of the vectors to promote the factualiy and faithfulness. As the fea- ture vector from the SC layer are not same as at- tention input, the output passed through feed- forward layer and fused with the multi-head atten- tion module of BART. Finally, the vectors from the sheaf graph attention model also undergo normal- ization via a feed-forward layer for feature vector normalization, followed by fusion with the heads of the multi-head attention in the BART encoder. The SC layer and sheaf graph contribute to the model's understanding of inter and intra-document relationships and the weighting of contexts, thereby promoting factual and faithful representations for the decoder generator model.
A Abstractive Baselines
(i) The Pointer Generator (PG) approach (See et al., 2017) combines attention, and pointing mech- anism to apprehend inter-document relationships. (ii) Himap (Fabbri et al., 2019) amalgamates MMR sentence weights in the PG network to emphasize pivotal information in the summary. (iii) Bottom- up Transformers (Gehrmann et al., 2018) employ the Transformer architecture (Vaswani et al., 2017) with one random attention head functioning as the copy pointer. (iv) BERT (Devlin et al., 2019) is an encoder-only language model trained using token masking techniques. (v) BART (Lewis et al., 2020) employs an encoder-decoder model trained on the text span masking technique. (vi) T5 (Raffel et al.. 2020), akin to BART, is mncoder-decoddr-baseee model that treats all downstream tasks as text-to- text problems during training. (vii) LongT5 (Guo et al.. 2021) scales the T5 architecture and utilizes sentence masking during pretraining to enrich lan- guage generation. (viii) Pegasus (Zhang et al., 2020) adopts an innovative sentence masking tech- nique in the language model to capture sentence- level representations. et al., 2020) incorporates: a Transformer with sparse attention handle elongated sequences.(x) BRIO (Liu al., 2022) employs a stochastic approach, rather than maximum likelihood, to train the net- (xik.(xi) PRIMERA (Xiao et al., 2022) extends
3.5 Decoder Setting
(ix) Longformer (Beltagy
The amalgamated representation derived from the encoder module of FABRIC, consisting of the BART encoder, SC layer, and sheaf graph attention, is for- warded to the BART decoder module. In this con- figuration, the BART decoder generates summaries

---

Multinews
CQASumm
DUC
Opinosis
System
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-
PG
41.85
12.91
20.79
31.09
5.52
21.85
31.43
6.03
23.08
19.65
1.29
20.08
HiMAP
44.17
16.05
24.38
27.13
4,48
19.87
31.44
6.11
23.17
18.02
1,46
19.84
Transfor.
44.32
15.11
28.07
27.52
4.53
21.0
32.14
6,24
23.34
20.46
1.41
21.35
BERT
44,27
16.23
31.41
27.32
4.35
21.14
34.64
6.39
24.18
20.41
4.62
21.84
BART
48.47
18.41
33,21
27.84
4.65
21.74
35.41
6.48
24.67
19.8
6.82
26.87
T5
44.08
16.39
37.68
28.11
3.74
21.45
34.13
6.32
24.13
27.41
6.97
27.41
LongT'
48.17
19.43
38.94
28.74
4.84
22.12
34.58
6.37
24.32
29.46
7.04
28.64
Pegasus
41.79
16.58
39.72
28.24
4.87
22.41
34.61
6.41
24.71
31.28
7.15
29.39
Longfor.
46.89
18.50
41.83
28.01
5.05
23.86
36.31
6.76
27.11
33.32
8.63
30.49
BRIO
47.24
19.34
44.57
29.13
5.12
24.17
37.18
7.02
29.37
31.15
8.52
29.64
PRIMERA
49.90
21.11
46.23
31.54
5.58
26.57
37.84
7.21
31.18
34.64
9.12
33.20
FABRIC
50.68
21.26
46.57
35,81
6.91
31.47
39.64
8.75
34.12
39.21
11.42
35.48
A bsin
0.78
1 0.11
1 0.34
4.27
1.33
4.90
1 1.80
1.54
1 2.94
4.57
2.30
t 2.28
48.24
20.54
44.87
33.24
29.14
36.21
8.29
32.71
38.54
10.84
34,29
45.58
19.12
42.35
31.85
27.65
35.78
8.11
37.68
10.51
32.18
42.18
18.74
41.42
29.18
26.19
33.19
8.02
30.12
36.84
10.18
31.73
Table I: Comparative analysis on four datasets Multinews, CQASumm, DUC and Opinosis. We report ROUGE-I (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) for eleven baselines. Our model FABRIC beats all the competing baselines. We also perform ablations over FABRIC. Addition of simplicial complex layer (SC) and Sheaf graph attention (SD) further improves the performance.
the Longformer architecture (Beltagy et al., 2020) by pretraining on the downstream task.
6.2 Human Evaluation Setup
We conducted a human evaluation to assess the qualitative aspects of the summaries generated by FABRIC. The evaluation focused on five parameters - -Ifformativeness (Inf), Relevance (Rel), Coherence (Coh), Fluency (Flu), and Topic coherence (Topic). To ensure representative evaluation, we randomly selected 50 test samples from the system-generann summaries and engaged 30 annotators' with exper- tise in the evaluation process. To minimize bias, each sample was annotated by at least two anno- tators, and annotations with divergence of more than two degrees were excluded to maintain con- sistency. The assigned scores by the annotators were then averaged for each sample. For additional details, please refer Appendix B.
6 Evaluation Setup
We benchmark FABRIC on the quantitative met- rics 1 (R1), Rouge-2 (R2) and Rouge-L. (RL) (Lin, 2004), to evaluate the lexical overlap, as well as on qualitative metrics BARTScore (Yuan et al., 2021) to compute the semantic overlap, and FactCC (Kryscinski et al., 2020), and SummaC (Laban et al., 2022) for faithfulness assessment.
6.1 Evaluation Metrics
To compare the quality of the summaries, em- ploy the following evaluation metries. (i) Rouge computes the lexical overlap between the target and the generated summary. (ii) BARTScore computes the semantic overlap between target and the gener- ated summary using the pre-trained BART (Lewis et al., 2020) LM. (iii) SummaC evaluates the con- sistency of summary sentences w.r.t the source ar- ticle, taking into account any inconsistencies that may occur throughout the source text. (iv) C leverages model trained assess the consistency between text/summary and its source article using synthetic data generated through various transfor- mations.
7 Experimental Results
We perform quantitative and qualitative evaluation the four datasets. We benchmark the datasets over eleven baselines and present quantitative and qualitative results.
7.1 Quantitative Evaluation
Table presents the lexical overlap between the various generated summaries and the reference summaries. Our results demonstrate that FABRIC
"The annotators were linguistics/sujeec experts and their ranged between 25-35.

---

Multinews
CQASumm
DUC
Opinosis
System
BS (1)
FC (t)
Sa (t)
BS (1)
FC (t)
Sa (t)
BS (1)
FC (1)
Sa (1)
BS (t)
FC (1)
Sa (1)
PG
-2.81
62.4
70.4
-3.21
48.3
47.6
-4.74
45.1
70.5
-4.37
55.4
56.1
HiMAP
-2733
65.8
71.4
-3.17
48.5
47.7
-4.88
45,2
71.4
-4.33
56.1
57.6
Transf.
-2,43
67.8
73.8
-3.11
49.8
48.1
.4.21
47.1
72.1
-4.02
58.4
59.6
BERT
-2.66
73.1
79.8
-3.13
51.7
49.7
-4.37
49.6
74,2
-4,28
60.1
62.8
BART
-2.51
76.8
81.2
-3,08
53,9
51.8
-4.34
51.8
75.6
-4,21
62.7
64.1
T5
-2.34
74.2
81.4
-3.02
51.8
50.1
-4.31
50.1
74.2
-4.18
61.2
62.8
LongT5
-1.92
75.4
83.4
-2.86
54.2
50.4
-3.87
50.8
73.5
-4.13
62.1
63.4
Pegasus
2.114
76.2
84.2
-2.82
55.1
51.1
-3.95
51.4
76,2
-4.25
62.4
64.7
Longf.
-1889
76.8
86.7
-2.73
56,2
52.9
-3.71
52,9
77.8
-4.02
62.7
65.8
BRIO
-1.85
75.4
85.2
-2.68
55.4
51.2
-3.60
52.5
76,4
-3.97
61.2
62.8
PRIMERA
-1.73
77.3
87.1
-2.62
57.1
53.8
-3.58
53.6
78.7
-3.86
63.4
66.5
FABRIC
-1.71
78.1
88.2
22.60
57.6
54.2
-3.57
54,9
79.1
-3.77
64.8
68.3
Table 2: Performance of the competing method in terms of BARTScore (BS), FactCC (FC), and SummaC (Sa).
Lower BS indicates better performance.
Multinews
CQASumm
System
Inf
Coh
Rel
Flu
Topic
BERTopic
Inf
Coh
Rel
Flu
Topic
BERTopic
BART
2.31
2.28
1.92
2.35
2.14
2.52
1.47
Longformer
2.92
2.43
2.34
2.19
1.93
2.37
2.15
2.51
1.49
PRIMERA
3.27
3.12
2.47
2.37
2.24
2.33
1.97
2.17
2.53
1.50
FABRIC
3.25
3.16
2.94
2.49
2.49
2.39
2.32
2.35
2.02
2.17
2.59
1.53
Table 3: Scores five human evaluation metrics Informativeness (Inf), Relevance (Rel), Coherence (Coh), Fluency (Flu) and Topic Coherence (Topic) and one automatic metric BERTopic over three baselines and our proposed model, FABRIC.
attains 50.68 R1 and 46.57 RL, beating the best baseline (PRIMERA) by +0.78 R1 and +0.34 RL
7.2 Qualitative Evaluation
We also conducted a human evaluation to assess the performance of FABRIC. Table 3 presents the results, demonstrating that our model generates high-quality summaries that are redundant, faith- ful, and easily comprehensible humans. In ad- dition the human evaluation, we employed the BARTScore (Yuan et al., 2021), FactCC (Kryscin- ski et al., 2020), and SummaC (Laban et al., 2022) metrics to evaluate the semantic overlap and faith- fulness between the wystem-generann summaries and the reference summaries.
points. For CQASumm, FABRIC attains 35.81 R1 and 31.47 RL beating PRIMERA by +4.27 and +4.90 RL points. Similarly, for DUC and Opinosis, FABRIC attains 39.64 R1 and 34.12 RL, respectively and 34.12 RI and 35.48 RL respec- tively, beating PRIMERA by +1.80 R1, +4.57 RI, +2.94 RL, and +2.28 RL respectively.
To demonstrate the effectiveness of each module in FABRIC, we conduct ablation study. The re- sults of each ablation presented in Table 1. Our base model, augmented with the simplicial com- plex (SC) layer, enhances the RI and RL scores compared the multinews baseline by +2.89 and +-9.14 4 RL points, respectively. Furthermore, the integration of sheaf graph attention contributes to additional improvements, boosting the perfor- mance by 2.66 RI points and 2.52 RL points. These findings corroborate our hypothesis that lan- guage models alone insufficient capture inter- document relationships. However, when combined with context captured on higher dimension, they significantly enhance the model's ability to achieve high performance.
Table 2 provides an assessment of the semantic overlap, faithfulness, and factuality of the gener- ated summaries. The results demonstrate a signif- icant improvement in all three qualitative metrics. In terms of semantic overlap, FABRIC achieves a score of 11.71 compared to multinews, exhibiting an improvement of + 0.02 over the best baseline, PRIMERA. Similarly, for the FactCC and SummaC metrics, FABRIC shows improvements of +0.8 and +1.1, respectively. For the CQASumm dataset, FABRIC achieves a semantic overlap of -2.60, a FactCC score of 57.6, and a SummaC score 54.2, surpassing the best baseline, PRIMERA, by

---

Model
Ouiput
Target-1
teenage would-be bank robber in Detroit found himself imprisoned and presumably regretting two life choices- within seconds of being handed money at bank on Monday, police say. A police spokesman tells the Detroit Free Press that the 15-year-old gave teller note claiming he had bomb and demanding money. [...]
FABRIC
A bank teller in Detroit managed to stop a would-be bank robber before he could make it to the sidewalk. Police suy 16-year-old boy walked into a Chase Bank the citys east side Monday afternoon and told the teller he was strapped with homb and to give him all the money, reports the Detroit Free Press. The teller did, and the teen set off for the doorway, "likely exhilarated by the scent of escuping with his new wealth," " reports.[...]
ChatGPT
Kelly Preston, in an interview with Natalic Morales on 'Today,' discusses the "silent birth" technique commonly practiced by Scientologists, This technique involves refraining from spoken words during childbirth to create peaceful environment, Preston credits this technique for the calm and peaceful demeanor of her children. [...]
Target-2
preservationist in New York City thinks hes found the site a 1750s lavern [...] Adam Woodward says found what he believes to be hand hewn and hand-plann joists and foundation walls in the basement.
FABRIC
So he searched the basement. and "found myself in what I am pretty certain is the 1750s historic tavern," " he says. Specifically, found what he thinks hand-hewn and hand-planed joists and foundation walls from Colonial.
ChatGPT
Adam Woodward believes he has found evidence that a building in Manhattan could be the former site of the Bull's Head Tavem, famous 18th-century tavemn where George Washington is believed to have visited during the American Revolution.
Table 4:
Comparison of target summary with the summary generated by FABRIC and ChatGPT.
+0.02, +0.6, and +0.4 points, respectively. Table 2 summarizes the improvements across all base- lines on four datasets. These findings confirm our hypothesis that language models alone are insuf- ficient to comprehend the relationships between multiple entities, necessitating the incorporation of additional modules to ensure the accuracy and coherence of the generated summaries.
comprehensively conveys the main incident but also provides additional relevant information. This additional information is captured by the simplicial complex layer and the sheaf graph attention, which pass it on to the decoder for inclusion in the final summary. However, in the case of sample #2, the source document presents the information about the year 1750 as hypothesis, while another docu- ment presents it as quotation. This discrepancy leads FABRIC to treat this information as defini- tive and present it as a quoted fact in the summary. This highlights the challenge of properly handling such cases and emphasizes the notion that target summaries and quantitative metrics alone may not suffice as the true performance measure (Li et al., 2021) for the task of abstractive summarization.
The quantitative improvements achieved by our model are further supported by human assessments. Table 3 presents the results of these assessments, re- vealing F FABRIC's 's consistent performance across all datasets. With the exception of informativeness on the multinews dataset compared to the PRIMERA baseline, FABRIC achieves the highest scores in all metries on multinews and CQAsumm. This indi- cates that the generated summaries are highly faith- ful, relevant, and coherent when compared to the other baselines. Although FABRIC exhibits some shortcomings in terms of informativeness accord- ing to the human evaluations, it still outperforms other baselines by significant margin. A detailed examination of the generated summaries and an analysis of the findings can be found in Section 8.
FABRIC ChatGPT
We conducted comparison between the sum- maries generated by FABRIC and CharGPT as shown in Table We randomly generated 50 sum- maries from the multinews test set for this analy- sis. When comparing the two, we found that the summaries generated by ChatGPT are often overly general, lacking relevance and informativeness in relation to the source documents. For instance, in sample #1, ChatGPT discusses general topic that deviates from the main focus by expanding on its own knowledge graph. Although it showcases lin- guistic capabilities, it fails to align the generated summary with the specific factual information from the source document. Similarly, in sample #2, Chat-
8 Error Analysis
As indicated in Table 3. significant improvements are observed across all four metrics of human eval- uation. The generated summaries successfully cap- ture the essence of the source documents and cover the major topics discussed. For instance, in sam- ple #1 in Table 4, the generated summary not only

---

GPT provides a summary that captures the main idea of the source but neglects mention important factual details such as the year or any correlations with specific locations. This comparison highlights that general purpose LLMs like ChatGPT have a tendency to focus on linguistic aspects but strug- gle to ensure fidelity to the factual information and alignment with the original source. As result, they can be considered unfaithful they deviate from the source and expand the generated output based on their own knowledge.
they may limit its practical applicability in certain real-time or resource-constramm settings. Over- all, while FABRIC shows promising advancements in multi-document abstractive summarization, fur- ther research is needed to address these limitations and extend its applicability to broader range of scenarios.
13 Ethics Statement
The benchmarked datasets Multinews, CQA- Summ, DUC and Opinosis are available under open-source licenses. The baselines were also cov- ered by the open-source licenses. The human eval- uation for the quality assessment of summaries were done by students from academic institute. All experiments were performed over Nvidia A100 (80GB) GPU.
10 Conclusion
In this study, we presented FABRIC, a encoder- decoder model designed to enhance topic co- herence and inter-document relations for multi- document abstractive summarization. The key com- ponents of FABRIC include BART for capturing lin- guistic aspects and implicial complex Layer and sheaf graph attention for capturing inter-document relationships. We compared FABRIC with eleven baselines on four widely popular MDS datasets. The results consistently demonstrated that FABRIC surpasses existing systems and achieves significant improvements across both quantitative and qualita- tive measures. These findings were further backed by human evaluation, validating the effectiveness FABRIC in generating more accurate and faithful summaries that better represent the content of the source document.
References
Griffin Adams, Han-Chin Shing, Qing Sun, Christo- pher Winestoek, Kathleen McKeown, and Noémie Elhadad. 2022. Learning revise references for faithful summarization. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pages 4009-4027, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yash Kumar Atri, Vikram Goyal, and Tanmoy Chakraborty. 2023a. Fusing multimodal signals on hyper-complex space for extreme abstractive text summarization (ul:dr) scientific contents. Pro- ceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, page 3724-3736, New York, NY, USA. Association for Computing Machinery.
11 Acknowledgement
The authors acknowledge the support of ihub- Anubhuti-iitd Foundation at IIT-Delhi.
Yash Kumar Atri, Vikram Goyal, and Tanmoy Chakraborty. 2023b. Multi-document summariza- tion using selective attention span and reinforcement learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 1-10.
12 Limitations
While our study presents promising results for FABRIC in the context of multi-document abstrac- tive summarization, there are certain limitations that should be acknowledged. Firstly, FABRIC demonstrated improvements in both quantitative and qualitative evaluation metrics, the subjective nature of summarization quality makes it challeng- ing to establish a universally agreed-upon stan- dard for evaluation. Secondly, the performance of FABRIC could be influenced by factors such as the size and quality of the training data, hyperpa- rameter tuning, and architectural choices, which should be further investigated. Lastly, the com- putational complexity and resource requirements of FABRIC should be taken into consideration, as
Yash Kumar Atri, Shraman Pramanick, Vikram Goyal, and Tanmoy Chakraborty. 2021. See, hear, read: Leveraging multimodality with guided attention for abstractive text summarization. Knowledge -Based Systems, 227:107152.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. 2022. Correcting diverse factual errors in abstractive summarization via post- editing and language model infilling. In Proceedings

---

of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 9818-9831 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faith- fulness assessment in abstractive summarization. In ACL, pages 5055-5070, Online.
Iz. Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150
Alex Fabbri, Prafulla Kumar Choubey, Jesse Vig, Chien- Sheng Wu, and Caiming Xiong. 2022. Improving fac- tual consistency in summarization with compression- based post- post-editing. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 9149-9156, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Tryna Gurevych. 2019. Bet- ter rewards yield better summaries: Learning to summarise without references. In EMNLP-IJCNLP, pages 31 10-3112. ACL.
Meng Cao. Yue Dong, Jingyi He, and Jackie Chi Kit Cheung. 2022. Learning with rejection for abstrac- tive text summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 9768-9780, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019. Multi-news: a large-scale multi nulti-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv: /906.0176999
Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: graph-based approach to abstrac- tive summarization of highly redundant opinions. Proceedings of the 23rd International Conference on Computational Linguisties, pages tion for Computational Linguistics.
Jiaao Chen and Diyi Yang. 2021. Structure-aware ab- stractive conversation summarization via discourse and action graphs. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 1380-1391 1391. Online. As- sociation for Computational Linguistics.
Associa-
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Com- putational Linguistics.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021. Improving faithfulness in abstractive sum- marization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 5935-5941, Online. Association for Computational Linguistics.
Asish Ghoshal, Arash Einolghozati, Ankit Arun, Haoran Li, Lili Yu, Yashar Mehdad, Scott Wen tau Yih, and Asli Celikyilmaz. 2022. Improving faithfulness of ab- stractive summarization by controlling confounding effect of irrelevant sentences.
Tanya Chowdhury and Tanmoy Chakraborty, 2019. Cqa- summ: Building references for community question answering summarization corpora. Association for Computing Machinery.
Daniel Gillick, Benoit Favre, and Dilek Z. Hakkani-Tür. 2008. The iesi summarization system at tac 2008. Theory and Applications of Categories.
Daniel Deutsch. Tania Bedrax- Weiss, and Dan Roth. 2020. Towards Question-Answering an Automatic Metric for Evaluating the Content Quality of a Sum- mary.
L. Giusti, C. Battiloro, P. Di Lorenzo, S. Sardellitti, and S. Barbarossa. 2022. Simplicial attention neural networks.
Maarten Grootendorst. 2022. Bertopie: Neural topic modeling with a class- tf-idf procedure, arXiv preprint arXiv:2203.05794,
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL, pages 4171-4186.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text transformer for long sequences.
Zi-Yi Dou, Pengfei Liu. Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4830-4842, Online. Association for Computational Linguistics.
Yanzhu Guo, Chloé Clavel. Moussa Kamal Eddine, and Michalis Vazirgiannis. 2022. Questioning the valid- ity of summarization datasets and improving their factual consistency. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 5716-5727, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
DUC. 2002. Document Understanding Conferences. [online] Available at: https://duc.nist.gov. Accessed I Mar. 2020.

---

Jakob Hansen and Thomas Gebhart. 2020. Sheaf neural networks.
Dublin, Ireland. Association for Computational Lin- guistics.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computationa Linguistics, pages 5094-5101 On- line. Association for Computational Linguistics.
Yuanjie Lyu, Chen Zhu, Tong Xu, Zikai Yin, and En- hong Chen, 2022. Faithful abstractive summarization transformer.
via fact-aware maasiistency-costraiie In Proceedings of the 31st ACM International Con- ference on Information & Knowledge Management, CIKM '22, page 1410-1419, New York, NY, USA. Association for Computing Machinery.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computa- tional Linguistics.
Qianren Mao, Jianxin Li, Hao Peng, Shizhu He, Lihong Wang, Philip S. Yu, and Zheng Wang. 2022. Fact- driven abstractive summarization by utilizing multi- granular multi-relational knowledge. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 30: 166-1678.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan Thomas Medonald. 2020. On faithfulness and factuality in abstractive summarization. In ACL, pages 1906-1919, Online.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI-
based models for inconsistency detection in summa- rization. Transactions of the Association for Compu-
tational Linguistics. 10:163-177.
OpenAL. 2023. Gpt-4 technical report.
Gyoung Ho Lee and Kong Joo Lee. 2017. Automatic text summarization using reinforcement learning with embedding features. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 193- 197, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. In NAACL, pages 646-653, New Orleans, Louisiana.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. CoRR, abs/1 700.04304
Hwanhee Lec. Cheoneum Park, Scunghyun Yoon, Trung Bui, Franck Dernoncourt, Juae Kim, and Ky- omin Jung. 2022. Factual error correction for abstrac- tive summaries using entity retrieval. In Proceedings of the 2nd Workshop on Natural Language Genera- tion, Evaluation, and Metrics (GEM), pages 439-444, 444, Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. JMIRR, 21 21(140))1-67.
Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Ge- offrey Cideron, Robert Dadashi, Matthicu Geist, Ser- tan Girgin, Léonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieil- lard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin, and Idan Szpektor. 2023. Factually consistent summarization via reinforcement learning with textual entailment feedback.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zetlemmoer. 2020. BART: Denoising sequence-to-sen to-seuuecce pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880, Online. Association for Computa- tional Linguistics.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. JEEE Transactions on Neural Networks, 20(1): 1006088000
Lei Li, Wei Liu, Marina Litvak, Natalia Vanetik, Ji- acheng Poi, Yinan Liu, and Siya Qi. 2021. Subjective bias in abstractive summarization.
A Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver. Canada. Association for Computa- tional Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81. Barcelona, Spain. Association for Computational Linguistics.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig, 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Pupers), pages 2890-2903,
Yunchong Song, Chenghu Zhou, Xinbing Wang, and Zhouhan Lin. 2023. Ordered GNN: Ordering mes- sage passing to deal with heterophily and over- smoothing. In The Eleventh International Confer- ence on Learning Representations.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
sentence summarization. Information Processing Management, 58(3)1002478.
Lulu Zhao, Zeyuan Yang, Weiran Xu, Sheng Gao, and Jun Guo. 2021. Improving abstractive dialogue sum- marization with conversational structure and factual knowledge.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
A
Appendix
A.1
System Setup/Implementation Details
For Multinews and CQASumm, we used the origi- nal cleaned corpus and segmented the source douc- ments as per the top-5 topics coverage. We trained the BERTopic over the source corpus and gener- ated topics later. On average, we segmented the source document into three documents as per the upper limit of BART encoder and number of topics found. For DUC and Opinosis, we used the model trained on the multinews dataset for inference. For modeling, we used PyTorch 1.14 for prototyping. We initially set the learning rate 0.002, and later warm-up is applied for the first 10000 steps. The model is trained for 8 epochs for multinews and CQASumm.
Wenhao Wu, Wei Li, Jiachen Liu, Xinyan Xiao, Ziqiang Cao. Sujian Li. and Hua Wu. 2022. FRSUM: To- wards faithful abstractive summarization via enhanc- ing factual robustness. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages (6400-3544) Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid- based masked sentence pre-training for multi-document summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages -245- Dublin, Ireland. Association for Computational Linguisties.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph- based neural multi-docummnt summa- rization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462, Vancouver, Canada. Associa- tion for Computational Linguistics.
B Human Evaluation
We evaluate the qualitative aspects the generated summaries on 5 metrics - Informativeness, Rele- vance, Coherence, Fluency, and Topical modeling. We define these metrics as follows:
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems, volume 34, pages 27263-27277. Curran As- sociates, Inc.
(1) Informativeness: The degree to which sum- mary provides accurate and comprehensive infor- mation about the source text, conveying the main points effectively.
(2) Relevance: The extent to which a summary is directly related and applicable to the topic and content of the source text, cap- turing the key aspects and avoiding irrelevant de- tails. (3) Coherence: The coherence of summary refers to its logical flow and smooth organization of ideas, ensuring that the sentences and paragraphs are well-connected and cohesive. (4) Fluency: Flu- ency measures the readability and naturalness of summary's ' language, including grammar, syntax, and smoothness of expression, allowing for easy comprehension and readability. (5) Topical mod- eling: The overarching subject or theme that the summary focuses on, capturing the main idea and central topic of the source text, while maintaining coherence and relevance in conveying the essential information.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed, 2020. Big bird: Trans- formers for longer sequences. In NearlPS, pages 17297-
Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski, Kazuma Hashimoto, and Yingho Zhou. 2022. Im- proving the faithfulness abstractive summarization via entity coverage control. In Findings of the Asxo- ciation for Computational Linguistics: NAACI. 2022, pages 528-535. Seattle, United States. Association for Computational Linguistics.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. PEGASUS: Pre-training with ex- tracted gap-sentences for abstractive summarization. In ICML, pages 11339.
Mengli Zhang, Gang Zhou, Wanting Yu, and Wenfen Liu. 2021. Far-ass: Fact-aware reinforced abstractive
We also show few more examples of our pro- posed FABRIC in Table 5.

---

C Discussions
Model
Output
Target
The unemployment ticked down from 5.9% to 5.8% last month, its lowest level since July 2008, the AP reports. And 214,000 jobs were added; 233,000 had been expected, per the Wall Street Journals forecust, with the unemployment rate expected to stick at 5.9%. August and September numbers were also revised upward:
Why not use traditional graphs: To benchmark with the traditional graph-based methods, we uti- lized (Yasunaga et al.. 2017) to model the intercon- nections between the documents. However, when tested, the performance improvement was only seen for the DUC dataset. For Multinews, CQA- Summ and Opinosis, the performance was inconsis- tent over the syntactical and semantic metrics. We infer that traditional graph-basee approaches strug- gle to capture nuanced dependencies and higher- order connections between the documents. In con- trast, Simplicial Complex and Sheaf Graph capture the underlying semantics and meaning of the docu- ments, not just their surface-level links. It provides a way to incorporate semantic context and rele- vance into the model, facilitating a more holistic understanding of how information is distributed across the documents.
FABRIC
The economy added 214,000 jobs in October, below the 233,000 economists expected, but the unemployment rate dropped 5,8% from 5.9%, reports the Wall Street Journal. That's the lowest rate since July 2008, notes the AP. The Labor Department also added 31,000 more jobs in Au- gust and Septemher than previously estimated. Employers have now added at least 200,000 Jobs for nine straight months, the longest such stretch since 1995, [...]
Target
The US won't he competing in men's soccer at the Olympics, thanks to heartbreaking goal that came literally the last minute. El Sal- vador battled the US to 3-3 draw last night, the AP reports; the US, which had been heavily favored win its group in CONCACAF quali- fying, needed win after being upset by Canada 2-0. It was back-and-forth affair, with the US scoring the first minute, then falling behind, and finally coming back lead 3-2.
How do Sheaf Graphs improve performance in heterophilic settings and counter over- smoothing: In heterophilic setting, where nodes have varying degrees of relevance and con- nections across different subgroups or categories, traditional graph attention mechanisms might strug- gle to capture diverse relationships effectively. Sheaf graph attention excels in this scenario by allowing for adaptive information flow based on the local structures. Unlike standard graph atten- tion, which computes weights based on node sim- ilarities, sheaf graph attention can tailor its atten- tion weights for each context, considering both local and global patterns. This ability to differ- entiate and adapt to heterophilic relationships en- ables better representation of information spread across different subgroups, leading to improved performance. Over-smoothing is common issue in graph neural networks (GNNs), where nodes' representations become overly similar after multi- ple graph convolutions, causing loss of discrimina- tive information. Sheaf graph attention mitigates over-smoothing by incorporating higher-order rela- tionships and context-specine attention. Instead of relying solely on aggregating neighbouring nodes' features, sheaf graph attention considers simplicial structures and focuses on more complex interac- tions, preventing overemphasis on similar nodes. This allows nodes to maintain their distinct charac- teristies even after multiple iterations of informa- tion propagation.
FABRIC
The US mens soccer team is of Olympic qualifying for the 2012 London Games after 3- 3 tie with El Salvador last night. The Americans led 3-2 in the final minute of stoppage time, but substitute keeper Scan Johnson couldní handle long shot from El Salvadors Jaime Alas, and the ball bounced off his hands, up over him, and into the net, reports AP. The tie means the US will miss the Olympics for the second time in three games. I...]
Target
If this story didní already cause you develop mild case of gephyrophobia. then this one may do the trick: Some 11% of Americaá bridges structurally deficient and need of repair, according a new report from Transportation for America. Its you consider this next one: 260 million trips are made across the 66,405 problematic bridges each day.
made more serious when
FABRIC
More than in nine bridges in the US is struc- turally deficient, meaning its in danger of col- lapse, according
new report from Transporta- tion for America, and thats problem thats only going get worse the nations most heavily traveled bridges near the end of their expected life spans, USA Today reports. "Youfe sceing the aging the system," says co-author of the report. "It really does parallel the (aging of) the Baby Boomers in a startling way." The report found that least 66,405 bridges 11% of the total are at least 65 years old on average, [...]
Table 5: Comparison of target summary with our FABRIC model generated summaries.


Reviews:
[{"review_id": "oz18RSQRXA", "reviewer": "Reviewer_PARH", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluating topic segmentation:** Thank you for the suggestion. We employ topic segmentation to divide the document into multiple segments during preprocessing. In relation to this module, we do not evaluate this module due to the absence of gold labels. Instead, we assess the performance of the topic segmentation within the context of summarization, utilizing an automated metric (BERTopic), which is detailed in Table 3.\n\n**Is EMNLP the right venue?** The task of summarization closely aligns with the area of EMNLP, and we believe that EMNLP is the right platform for presenting this research. We will simplify the methodology section and add further background and motivation in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is generally well structured, motivated\n- Good experimental setup and ample evaluation \n- Improvement over state of the art.", "reasons_to_reject": "- It would have been good to evaluate the topic segmentation on its own.\n- The proposed method is very difficult to understand for intended audience for this work (EMNLP attendees). One could imagine many NLP practitioners to be confused about the proposed method. I myself am unable to properly critique the work. Given the nature of the presented method, perhaps a venue such as NeurIPS would be more appropriate", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work introduces FABRIC, a multi-document summarizer.  FABRIC consist of a BART encoder decoder model, augmented with a simplicial complex and topic segmentation components. The proposed method is comprehensive and well evaluated. Authors report both quantitative and qualitative evaluations.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EjGjdCm8uj", "reviewer": "Reviewer_W8oR", "summary": "", "questions": "1. Why do you use simplicial complex and sheaf graph to model the innerconnections of the given documents? Why not use traditional graphs or hypergraphs? \n2. Why does the proposed sheaf graph attention improve the performance in heterophilic setting and also alleviate the over-smoothing problem?\n3. In these MDS dataset used by this paper, the average number of input documents per sample is low. Authors may consider using datasets [1,2] with a higher average number of documents per sample to verify the effectiveness (Topic Coherence and Inter-Document Consorts) of the proposed method.\n\n[1] Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles\n[2] Generating a Structured Summary of Numerous Academic Papers: Dataset and Method", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Missing notations in equations:** Thank you for your suggestion. In line 274, the symbol \"F\" corresponds to a bank of Simplicial Filters. Furthermore, we have validated and updated all the notations used in the algorithm. We will update these in the updated version of the paper.\n\n**Graph construction method is unclear:** We do not explicitly construct a graph; instead, we pass the token embeddings ‘Xs' into a Sheaf Graph function to formulate a graph representation of the features, which are learned by the module during the training phase. This is similar to graph formulation and learning in the GCN and GAT approaches.\n\n**Ablations of proposed modules:** We performed ablation analyses for every module, as shown in Table 1 (main paper). The proposed Sheaf Graph (SD) module achieves a Rouge-1 score of 29.18 on CQASumm, 33.19 on DUC, and 36.84 on Opinosis. Further, the introduction of Simplicial Complex enhances the performance of Rouge-1 to 31.85 on CQASumm, 35.78 on DUC, and 37.68 on Opinosis.\n\n**Motivation and rationale behind using Simplicial Complex and Sheaf Graphs:** Thank you for this suggestion. We will include the following  motivation and background on Simplicial Complex and Sheaf Graphs in the main paper and append supplementary sections with the required mathematical foundation needed.\n\n`Simplicial Complex: A simplicial complex is a mathematical structure that represents relationships between different levels of \"simplices.\" A simplex is a generalization of a triangle (2-simplex) to higher dimensions. In the context of summarization, simplicial complexes are used to capture relationships between multiple documents that might not be directly connected, by considering their shared properties or interactions.`\n\n`Sheaf graphs: A sheaf is a mathematical construct used to associate local data with certain topological spaces. In a sheaf, data are organized in a way that respects the local structures of the underlying space. In the context of graphs, sheaf graphs extend the idea of traditional graph structures by incorporating not only node and edge connections but also the information associated with these connections. In practical terms, sheaf graphs can represent relationships between data points while also encoding additional context or attributes associated with those relationships. This makes them suitable for capturing complex interactions, semantic relationships, and context in various applications, including natural language processing tasks like summarization.`\n\n\n**Why not use traditional graphs and hypergraphs:** To benchmark with the traditional graph-based methods, we utilized [1] to model the interconnections between the documents. However, when tested, the performance improvement was only seen for the DUC dataset. For Multinews, CQASumm and Opinosis, the performance was inconsistent over the syntactical and semantic metrics. We infer that traditional graph-based approaches struggle to capture nuanced dependencies and higher-order connections between the documents. In contrast, Simplicial Complex and Sheaf Graph capture the underlying semantics and meaning of the documents, not just their surface-level links. It provides a way to incorporate semantic context and relevance into the model, facilitating a more holistic understanding of how information is distributed across the documents. As for the hypergraphs, we did not run any ablations but theorized that hyperedges, when connected to multiple nodes, can lead to difficulties in interpreting the semantic relationships they represent. In abstractive summarization, understanding the precise meaning and context of these relationships is crucial. Secondly, Hypergraphs can become increasingly complex as the number of nodes and hyperedges grows.\n\n**How do Sheaf Graphs improve performance in heterophilic settings and counter over-smoothing:** In a heterophilic setting, where nodes have varying degrees of relevance and connections across different subgroups or categories, traditional graph attention mechanisms might struggle to capture diverse relationships effectively. Sheaf graph attention excels in this scenario by allowing for adaptive information flow based on the local structures. Unlike standard graph attention, which computes weights based on node similarities, sheaf graph attention can tailor its attention weights for each context, considering both local and global patterns. This ability to differentiate and adapt to heterophilic relationships enables better representation of information spread across different subgroups, leading to improved performance. Over-smoothing is a common issue in graph neural networks (GNNs), where nodes' representations become overly similar after multiple graph convolutions, causing loss of discriminative information. Sheaf graph attention mitigates over-smoothing by incorporating higher-order relationships and context-specific attention. Instead of relying solely on aggregating neighbouring nodes' features, sheaf graph attention considers simplicial structures and focuses on more complex interactions, preventing overemphasis on similar nodes. This allows nodes to maintain their distinct characteristics even after multiple iterations of information propagation.\n\n**Benchmarking with additional datasets:** Thank you for your suggestion. We conducted benchmarking on these datasets using our proposed model, and the results are summarized in the table below. Unfortunately, due to time constraints, we were unable to run the baselines. However, we will include the baseline results for these datasets in the revised paper.\n\n| Dataset        | Rouge-1    | Rouge-2    | Rouge-L    |\n|----------------|-------|-------|-------|\n| Multi-XScience | 35.82 | 7.81  | 31.97 |\n| BigSurvey-MDS  | 45.13 | 13.62 | 18.39 |\n\n[1] Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based Neural Multi-Document Summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 452–462, Vancouver, Canada. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper adopts the simplical complexes structure to model the interconnections of multiple documents and also the sheaf graphs also facilitate the heterogeneous setting.\n2. The experiment shows superiority in multiple document summarization task compared with other 11 baselines.\n3. The paper is well-organized.", "reasons_to_reject": "1. The equations are not well explained. For instance, in line 274, it did not explain the meaning of F. Please check the rest.\n2. The graph construction method is not clear.\n3. Ablation study should be conducted to reveal the contribution of each module.\n4. The writing needs improvements. More method and rationale details are suggested to be given.", "questions_to_authors": "1. Why do you use simplicial complex and sheaf graph to model the innerconnections of the given documents? Why not use traditional graphs or hypergraphs? \n2. Why does the proposed sheaf graph attention improve the performance in heterophilic setting and also alleviate the over-smoothing problem?\n3. In these MDS dataset used by this paper, the average number of input documents per sample is low. Authors may consider using datasets [1,2] with a higher average number of documents per sample to verify the effectiveness (Topic Coherence and Inter-Document Consorts) of the proposed method.\n\n[1] Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles\n[2] Generating a Structured Summary of Numerous Academic Papers: Dataset and Method", "paper_topic_and_main_contributions": "This paper proposes FABRIC, a new model for multi-document summarization. It utilizes a simplicial complex layer to understand complex relationships, and sheaf graph attention to capture diverse properties. FABRIC outperforms other baselines.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dhnbh4rrjx", "reviewer": "Reviewer_EADd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Human evaluation IAA and divergence assessment:** We computed Cohen's Kappa score to determine the Inter-Annotator Agreement (IAA), resulting in an aggregated Kappa score of 0.8731. Regarding divergence, 7.82% of annotators exhibited discrepancies. To address this, we reassigned these summary samples to new human evaluators, considering agreements exceeding 75%. At the final evaluation, 3.49% of samples were excluded to ensure consistency in human evaluation.\n\n**Definition of Informativeness, Relevance, and Topic Modelling:** You are correct. The source documents and the generated summaries were provided for the human evaluations. We will mention this explicitly in the revised paper.\n\n**Improvement in human evaluations across datasets:** In terms of automatic evaluations, when compared to PRIMERA, Multinews demonstrates an improvement of +0.78 R1 and +0.34 RL. CQASumm shows gains of +4.27 R1 and +4.90 RL. Similarly, for DUC and Opinosis, improvements of +1.80 R1, +2.94 RL and +4.57 R1, +2.28 RL are achieved, respectively. Regarding human evaluations, the benchmarks were conducted on random samples, revealing consistent improvements across metrics and datasets. This suggests that the proposed model exhibits strong generalization and consistent performance across diverse datasets.\n\n**ChatGPT vs. proposed model evaluations:** Due to space constraints, we presented limited examples; however, we plan to provide all samples upon the final paper's release. Regarding the comparison, human evaluators were provided with the source document, ChatGPT-generated summary, and proposed model-generated summary. The evaluation yielded an aggregated kappa score of 0.8382. The evaluation criteria were akin to those outlined in section 6.2.\n\n**Typos and Grammars:** We apologise for the typos and grammatical mistakes. We have gone through the paper and have revised it. We have also corrected the baseline name — PRIMERA.\n\n**Background about Simplicial Complex and Sheaf Graphs:** Thank you for the suggestion. We will include the motivation and background behind using Simplicial Complex and Sheaf Graphs in the main text. We will also append the supplementary with the required mathematical background.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new model for multi-document summarization that consistently outperforms state-of-the-art models on 4 benchmarks according to automatic metrics.", "reasons_to_reject": "The paper claims that FABRIC improves faithfulness and factuality. However, this claim is not assessed in the results because the automatic faithfulness metrics (i.e., Summac and FactCC) were used between the generated summary and the reference, and the human evaluation also doesn’t assess faithfulness.\n\nRegarding the human evaluation:\n\n\n- What was the IAA for the human evaluation task? Also, the authors mentioned in lines 439-441 that “annotations with a divergence of more than two degrees were excluded to maintain consistency”. A large divergence between annotators might also reflect that the instructions are subjective or not sufficiently clear or that the annotators didn’t understand the task. How many cases were excluded because of a large divergence? \n\n\n- From the definition of informativeness: “The degree to which a summary provides accurate and comprehensive information about the source text”, relevance: “The extent to which a summary is directly related and applicable to the topic and content of the source text” and topic modeling: “The overarching subject or theme that the summary focuses on, capturing the main idea and central topic of the source text”, one can understand that annotators were given all the source documents and asses the generated summary to the entire set of input documents. Is this the case or the generated summary was only compared to the reference? If so, the definitions should be updated accordingly. \n\n\n- Overall, the improvement is rather minor compared to PRIMERA (0.02 on average for all evaluation aspects). \n\n\nThe comparison to ChatGPT in Section 9 is a bit misleading, showing some cherry picking examples in Table 4. Who conducted the comparison? Did the annotators compare the generated summary to the source articles or to the reference? What was the IAA between the annotators? What are the criteria of evaluation? Are they the same as the human evaluation in Section 6.2?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new model for multi-document summarization using (1) simplicial complex layers and (2) sheaf graph attention. Along with this model, the paper suggests obtaining contextual vectors for any input length. \n\nThe authors show that the proposed model consistently outperforms 11 models on 4 popular MDS datasets according to automatic metrics (lexical and model-based metrics). In addition, the authors conduct a human evaluation, showing similar trends.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 130: input length input → input length     \nLine 146: primera  \nLine 246: segment generated —> generated segment  \nLine 248: “LM to process” ?\nLine 359: Datasets    \nLine 364: Missing references to CQASumm     \nLine 480: do you mean table 3?   \nLine 940: , instead of . \n \t\t\t\t\t\n\nSometimes, the authors write primer, sometimes primera, the actual name is primera. \n\nSome BG about simplicial and sheaf graphs is needed  \n\nThe third contribution is more a requirement than a contribution. For me, the paper proposes two contributions, (1) a method for processing multiple documents by compressing documents and (2) a new encoder-decoder model for MDS based on simplicial complex and sheaf graphs."}}]

Author rebuttal:

Meta-review: The paper tackles multi-document summarisation, by introducing a novel idea that uses simplicial complex and sheaf graph attention to capture complex and diverse cross-document relationships. Reviewers all thought the paper is generally well-structured, the experiments are extensive, and the results are impressive. Some concerns are raised by the reviewers, and the two main ones that I can see are: (1) certain parts surrounding the methodology are difficult to follow, and they require more description (e.g. explaining the idea of simplicial complex) and motivation (e.g. why do we need simplicial complex?); and (2) the human evaluation results require some clarification - in particular its inter-annotator agreement and the conclusion drawn from the ChatGPT results. That said, I believe the contribution of the paper is significant, and that these concerns are presentational issues that could be fixed in the revision.
Decision: Accept-Main