Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders
Authors: Qianren Mao
Shaobo Zhao
Jiarui Li
Xiaolei Gu
Shizhu He
Bo Li
Jianxin Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Bipartite Graph
Graph Pre-training
Unsupervised Extractive Summarization
Graph Convolutional Auto-Encoders
TL;DR: Our proposed graph pre-training auto-encoder models intra-sentential features and inter-sentential features through sentence-word bipartite graphs, enabling extract significant sentences for unsupervised  summarization.

Abstract: Pre-trained sentence representations are crucial for identifying significant sentences in unsupervised document extractive summarization.
However, the traditional two-step paradigm of pre-training and sentence-ranking, creates a gap due to differing optimization objectives. 
To address this issue, we argue that utilizing pre-trained embeddings derived from a process specifically designed to optimize informative and distinctive sentence representations helps rank significant sentences.  
To do so, we propose a novel graph pre-training auto-encoder to obtain sentence embeddings by explicitly modelling intra-sentential distinctive features and inter-sentential cohesive features through sentence-word bipartite graphs. 
These fine-tuned sentence embeddings are then utilized in a graph-based ranking algorithm for unsupervised summarization.
Our method is a plug-and-play pre-trained model that produces predominant performance for unsupervised summarization frameworks by providing summary-worthy sentence representations. 
It surpasses heavy BERT- or RoBERTa-based sentence representations in downstream tasks.

Bipartite Graph Pre-training for Unsupervised Extractive Summarization
with Graph Convolutional Auto-Encoders
Qianren Mao', Shaobo Zhao", Jiarui Li', Xiaolei Gu², Shizhu He', Bo Li Jianxin Li
Zhongguancun Laboratory, Beijing, P.R.China.
2 School of Software, Beihang University, Beijing, P.R.China.
3 Institute of Automation, Chinese Academy of Sciences, Beijing, P.R.China.
School of Computer Science and Engineering, Beihang University, Beijing, P.R.Chima
{maoqr. 1ibo, www.lxigcxxicc.bbb..oooo edu. cn, zhu. he@nlpr ia. an
4204 11t.::1001.000222241111111111111 142.195777....111111224411111111 edu. cn.
Abstract
during friendly match between Argentin Worlze's s Stadium Beijing.
the
Pre-trained sentence representations are crucial for identifying significant sentences unsu-
Chinese
hug soccer legend Linnel Mess
pervised document extractive summarization. However, the traditional two-step paradigm of pre-training and sentence-ranking, creates gap due to differing optimization objectives. To address this issue, we argue that utilizing pre-trained embeddings derived from pro- cess specifically designed to optimize cohen- sive distinctive sentence representations helps rank significant sentences. To do so, we propose novel graph pre-training auto- encoder to obtain sentence embeddings by ex- plicitly modelling intra-sentential distinctive features and inter-sentential cohesive features through sentence-word bipartite graphs. These pre-trained sentence representations are then utilized in a graph- based ranking algorithm for unsupervised summarization. Our method pro- duces predominant performance for unsuper- vised summarization frameworks by providing summary- worthy sentence representations. It surpasses heavy BERT- or RoBERTa-haas based sen- tence representations in downstream tasks.
S
W2
appиers initially shocked. stretchiing his arms and hugging the fan back
clips show
W3
Sc
W3
Beilng
The graph structure of bipartite sentence-word graph. The sentences connect with unique nodes (mo- nopolized by a single sentence node), and common nodes (shared multiple sentence nodes).
Figure 1:
et al., 2020), modelling graph-based ranking meth- ods through global view of the document.
For unsupervised document summarization, learning semantic sentence embeddings is crucial, alongside the sentence ranking paradigm. Textual pre-training models like skip-thought model (Kiros et al., 2015), TF-IDF, and BERT (Devlin et al., 2019) generate sentential embeddings, enabling extractive systems to produce summaries that cap- ture the document's central meaning (Yasunaga et al., 2017; Xu et al., 2019; Jia et al.. 2020; Wang et al., 2020). By combining sentence representa- tions generated from pre-trained language models, prominent performances have been achieved with graph-based based sentence ranking methods (Zheng and Lapata, 2019; Liang et al., 2021; Liu et al., 2021).
1 Introduction
Unsupervised document summarization involves generating a shorter version of a document while preserving its essential content (Nenkova and McK- cown, 201 1). It typically involves two steps: pre- training learn sentence representations and sen- tence ranking using sentence embeddings to select the most relevant sentences within a document.
Despite the effectiveness of graph- based rank- ing methods that incorporate pre-trained sentential embeddings, there are some underexplored issues. Firstly, a significant gap exists between the two- step paradigm of textual pre-training and sentence graph-ranking,
Most research focuses on graph-baser based sen- tence ranking methods, such as TextRank (Mihal- cea and Tarau, 2004) and LexRank (Erkan and Radev, 2004), to identify the significant sentence by utilizing topological relations. Continual im- provements have been demonstrated by several at- tempts (Narayan et al., 2018; Zhou et al., 2018; Wang al., 2019; Xiao and Carenini, 2019; Wang
the optimization objectives di- verge in these two steps. The pre-trained frame- work is primarily designed represent sentences with universal embeddings rather than summary- worthy features. By relying solely on the universal embeddings, the nuanced contextual information the document may be overlooked, resulting in sub-
Jianxin Li is the corresponding author.

---

optimal summaries. Secondly, the existing graph formulation (c.g., GCNs (Bruna et al., 2014)) only encodes distinctive sentences but not necessarily cohensive ones, which may limit the extraction of summary-worthy sentences.
tence node representations are utilized for down- stream extractive summarization.
Our pre-trained sentence representations obtain superior performance in both single document sum- marization on the CNN/DailyMail dataset (Her- mann et al., 2015) and multiple document sum- marization on the Multi-News dataset (Sandhaus, 2008) within salient extractive summarization frameworks. i) To our knowledge, we are the first to introduce the bipartite word-sentence graph pre- training method and pioneer bipartite graph pre- trained sentence representations in unsupervised extractive summarization. ii) Our pre- pre-trained sen- tence representation excels in downstream tasks us- ing the same summarization backbones, surpassing heavy BERT- or RoBERTa-based representations and highlighting its superior performance.
In summarization, cohensive sentences reveal how much the summary represents document, and distinctive sentences involve how much com- plementary information should be included in summary. To exemplify how these sentence fea- tures come from words, we analyze a sentence- word bipartite graph as depicted in Figure 1.
The connections -, w2, S. - w4 capture detra-sentential information, where the unique word nodes W1 = Bejing, w2 =fan, wa = shocked contribute distinctive features to their respective sentence nodes Sa. Sb, Se.
Background & Related Work
The connections S. ww. Sb-wo, Sp-WSSSW-W W3, Se-vi W3 capture inter-sentential information, where the shared word nodes wo = Argentina, W3 = Messi contains cohensive features for their connected sentence nodes Sa. Sb. Sc-
2.1 Sentence Ranking Summarization
Traditional extractive summarization methods are mostly unsupervised (Yin and Pei, 2015; Nallap- ati et al., 2017; Zheng and Lapata, 2019; Zhong et al., 2019: Mao al., 2022). Among them, graph- based sentential ranking methods are widely used. Two popular algorithms for single-document sum- marization are unsupervised LexRank (Erkan and Radev, 2004) and TextRank (Mihalcea and Tarau, 2004), estimating the centrality score of each sen- tence node among the textual context nodes. In contrast to LexRank and TextRank construct- ing an undirected sentence graph, the model of PACSUM (Zheng and Lapata, 2019) builds a directed graph. Its sentence centrality computed by ag- gregating its incoming and outgoing edge weights:
Clearly, sentence's unique features come from its individual word nodes, while its cohensive fea- tures come from shared word nodes with other sen- tences. Based on this observation, we argue that optimizing cohensive and distinctive sentence rep- resentations during pre-training is ultimately ben- eficial for ranking significant sentences in down- stream extractive summarization. To achieve this, we propose novel graph pre-training paradigm using a sentence-word bipartite graph with graph convolutional auto-encoder (termed as Bi-GAE!) to learn sentential representations.
Centrality(s) = 21 Σ jzi
ea + A2 Σ joi
In detail, we pre-train the bipartite graph by pre- dicting the word-sentence edge centrality score in self-supervision. Intuitively, more unique nodes imply smaller edge weights, as they are not shared with other nodes. Conversely, when there are more shared nodes, their edge weights tend to be greater. We present a novel method for bipartite graph encoding, involving the concatenation of an inter-sentential GCNinter and an intra-sentential GCNimr These two GCNs allocate two encoding channels for aggregating inter-sentential cohesive features and intra-sentential distinctive features dur- ing pre-training. Ultimately, the pre-trained sen-
euj
(1)
where hyper-parameters 4, d2 are different weights for forwardand backward-looking directed edges and di + d2 = 1. ei, is the weights the edges ei, E E and computed using word co-occurrence statistics, such as the similarity score. Building upon the achievements of PAC- Sum(Zheng and Lapata, 2019), recent models such as FAR (Liang et al., 2021) and DASG (Liu et al., 2021) have aimed improve extractive summa- rization by integrating centrality algorithms. These models primarily focus on seeking central sen- tences based on semantic facets (Liang et al., 2021) or sentence positions (Liu et al., 2021).
'Code and data available at: https://g/t thub. com/ OpenSUM/BiGar

---

2.2
Sentential Pre-training
3.2 Bipartite Graph Pre-training
PLM's pre-training, such as BERT and GPT, is cru- cial for identifying meaningful sentences in down- stream summarization tasks. The previously men- tioned graph-based summarization methods, such PacSum(Zheng and Lapata, 2019), FAR(Liang al., 2021), and DASG (Liu et al., 2021) uti- lize pre-trained BERT representations for sentence ranking. STAS (Xu et al., 2020) takes a different approach by pre-training a Transformer-based LM to estimate sentence importance. However, STAS is not plug-and-play and requires separate pre- training model for each downstream task.
We reform the original VGAE (Kipf and Welling. 2016) pre-training framework by optimizing edge weight prediction in bipartite graphs. The pre- training optimizer learns to fit the matrices between the input weighted adjacency matrix and the constructed adjacency matrix in typical way of self-supervised learning. By integrating intra- sentential GCNim and an inter-sentential GCNiner in the VGAE (Kipf and Welling, 2016) self- supervised framework, our pre- pre-training method en- ables effective aggregation of intra-sentential and inter-sentential information, allowing for the rep- resentation of high-level summary-worthy features in the bipartite graph pre-training. Bipartite Graph Initializers. Let X,, &R( and X, R(mid represent the input feature ma- trix of the word and sentence nodes respectively, where dy and d, are the dimension of word em- bedding vector and sentence representation vector respectively. We first use convolutional neural net- works (CNN) (LeCun et al., 1998) with different kernel sizes to capture the local n-gram feature for each sentence SĆ and then use the bidirectional long short-term memory (BiLSTM) (Hochreiter and Schmidhuber, 1997) layer to get the sentence- level feature S/. The concatenation of the CNN local feature and the BiLSTM global feature is used as the sentence node initialized feature Xs, = :5; The initialized representations are used as inputs to the graph autoencoder module. Bipartite Graph Encoder. To model summary- worthy representations, we encode the bipartite graph by concatenation of an intra-sentential GCNintaa and an inter-sentential GCNinter in which two GCNs assign two encoding channels for ag- gregating intra-sentential distinctive features and inter-sentential cohesive features. The GCNintra
Despite the success of the aforementioned unsu- pervised extractive summarization methods, it still maintains a gap between the PLMs' pre-training and the downstream sentence ranking methods. Ad- ditionally, low-quality representations can result in incomplete or less informative summaries, nega- tively affecting their quality. Pre-training models typically produce generic semantic representations instead of generating summary summary-worthy worthy representa- tions, which can result in suboptimal performance in unsupervised summarization tasks.
3 Methodology
In what follows, we describe our pre-training model Bi-GAE (as shorthand for Bipartite Graph Pre-training with Graph Convolutional Auto- Encoders ) used for unsupervised extractive sum- marization. We will introduce bipartite graph en- coding and the pre-training procedure using our BI-GAE. Ultimately, we will utilize the pre-trained sentence representations for the downstream unsu- pervised summarization.
3.1 Document as Bipartite Graph
Formally, We denote the constructed bipartite word- sentence graph & (V.AA, &, X). where V Vn U V,, Here, V. denotes [V unique words of the document and V, corresponds to sentences in the document. A (e11...., Ci.j..., Cnm) defines the adjacency relation- ships among nodes, and ei, E (0, 11"00m indicates the edge weight from source node i to target node matrix containing
(H" X, A Aweight O), can be seen as form of message passing to aggregate intra-sentential dis- tinctive features. The first GCNium layer generates lower-dimensional feature matrix. Its node-wise formulation is given by:
the |V3l-
intra
intra
(2)
v vādd
j. X E [n+mm)xd is termed the representation of all nodes. The node repre- sentations will be iteratively updated by aggregat- ing summary-worthy features (intra-sentential and inter-sentential messages) between word and sen- tence nodes via the bipartite graph autoencoder.
where et Aweight denotes the edge weight from source node to target node j. Here we use the betweenness centrality 2 as the edge
html

---

Encoding
Sampling
Decoding
Constructed Bipartite Graph
GCN
Reconstructed Bipartite Graph
gintre
z N (H,,
word
word
sentence
sentence
z (ppttri]zzlune
Aweight aa (2
AB A9
- (HH,.
(E, (E,Aweight X)
(E, (E, 2)
GCNiater
2 onter
Reconstruction Loss (AEwiiggghe Aweight)
Figure 2: Overall architecture our pre-training model B--GAF. We construct sentence-wond bipartite graph to optimize both distinctive intra-sentential and cohensive inter-sentential nodes, by predicting the word-sentence edge centrality scores using self-supervised graph autoencoder.
weights. The betweenness centrality of edge is the sum of fractions of the shortest paths passing through it. The first GCNiua layer makes features neighbour nodes with fewer as- sociation relationships aggregated and enlarged and outputs lower-dimensional feature matrix H. Then, the second GCN layer generates nntra GCN Hintra Aweight) and log(orintre))
OZ (Z)mtra) - (z. ( .
(6)
Generative Decoder. Our generative decoder is given by inner product between latent variables Z. The output of our decoder is reconstructed adjacency matrix Å, which is defined as follows:
N
N
p(Az
p(A.
(7)
GCNo (Hintra Weeight
i=1
The GCNiner (Ho X. Aweight. b) can be seen form of message passing to aggregate inter- sentential cohensive features:
where p(A,,lz,z,) = olz, f 2,), and Arj are the ele- ments of o(-) is the logistic sigmoid function. Edge Weights Prediction as the Pre-training Ob- jective. We use edge weight reconstruction as the training objective to optimize our pre-trained Bi- GAE. Specifically, the pre-training optimizer learns to fit the matrices between the input weighted ad- jacency matrix weight and the reconstructed adja- cency matrix Aweight
inter
Σ
e.
inter
JENCWUUA
dj
The graph convolution operator GCNinter will ag- gregate neighbour node features with more associa- tion relationships aggregated and enlarged. Analo- gously, we can obtain intee and log( which are parameterized by the two-layer GCNiteer
L MSE(p((Aeee) (AAeiii Aweight) (L(g
generate the latent variable Z as output of bipartite graph encoder by sampling from GCNister and GCNimia and then concatenating sam- pled two latent variables inter and Z inira.
Then
The loss function of the bipartite graph pre-training has two parts. The first part is MSE loss which mea- sures how well the pre-training model reconstructs the structure of the bipartite graph. KL works as a regularizer in original VGAE, and p(Z) = N(O, 1) is Gaussian prior.
N
g inter inter intra
(attn [ 1=1
inter
| [=]
q(z (itta
(4)
4 EXPERIMENTS
where q(zlhtter and qiz(ntra) from GCNs, isfying independent distribution conditions. Here,
During the graph pre-training, the Bi-GAE is opti- mized by the prediction of edge weights in a self- supervised manner. Subsequently, we utilize the
gizenteer = N p.tter diag((oo/(ter>))) (5)

---

pre-trained sentence representations of BI-GAE to replace those used in state-of-the-art unsupervised summarization backbones. This allows us to as- sess the effectiveness of the pre-trained sentence representation in downstream tasks.
centrality ranking for sentence selection. We now introduce extractive summarization backbones.
TextRank and LexRank utilize PageRank cal- culate node centrality based on a Markov chain model recursively.
4.1 Downstream Tasks and Datasets
PACSUM (Zheng and Lapata, 2019) constructs graphs with directed edges. The rationale be- hind this approach is that the centrality of two nodes is influenced by their relative position in the document, as illustrated by Equation 15.
We evaluate our approach on two summarization datasets: the CNN/DailyMail (Hermann et al., 2015) dataset and the Multi-news (Fabbri et al., 2019) dataset. The CNN/DailyMail comprises ar- tieles from CNN and Daily Mail news websites, summarized by their associated highlights. We follow the standard splits and preprocessing steps used in baselines (See et al., 2017; Liu and La- pata, 2019; Zheng and Lapata, 2019; Xu et al., 2020; Liang et al., 2021), and the resulting dataset contains 287,226 articles for training, 13,368 for validation, and 11,490 for the test. The Multi- news is large-scale multi-document summariza- tion (MDS) dataset and comes from diverse set of news sources. contains 44,972 articles for training, 5,622 for validation, and 5,622 for testing. Referring to prior works (Fabbri et al., 2019; Liu et al., 2021), we create sentence discourse graphs for each document and cluster them, with each cluster yielding summary sentence.
DASG (Liu et al., 2021) selects sentences for summarization based on the similarities and rela- tive distances among neighbouring sentences. It incorporates a graph edge weighting scheme Equation 15, using a coefficient that maps a pair of sentence indices to a value calculated by their relative distance.
FAR (Liang et al., 2021) modifies Equation 15 by applying a facet-aware centrality- ranking model to filter out insignificant sentences. FAR also incorporates similarity constraint between candidate summary representation and document representation to ensure the selected sentences are semantically related the entire text, thereby facilitating summarization.
4.2 Pre-training Datasets
The main distinction among the extractive frame- works mentioned above lies in their centrality al- gorithms. A comprehensive comparison of these algorithms can be found in Appendix 8
We construct a bipartite graph with word and sen- tence nodes, determining edge weights through graph centrality. The centrality-based weights de- noted as weight serve as inputs for the BI-GAE model. During pre-training, we use MSE loss to measure the average squared difference between the predicted edge values Awsighr and the true val- ues input Aweihht as it indicates more minor errors between the predicted and true values. We conve- niently utilize training datasets without their sum- marization labels as the corpus to pre-train sentence representations by our BI-GAE.
4.4 Compared Sentence Embeddings
We evaluate three sentence representations for com- puting sentence centrality. The first compared sen- tence embedding employs TF-IDF- ap- proach, where each vector dimension calcu- lated based on the term frequency (TF) of the word in the sentence and the inverse document frequency (IDF) of the word across the entire cor- pus of documents. The second representation is based on the Skip-thought model (Kiros et al., 2015), an ancoder-decoded model trained rounding sentences using sentence-level distribu- tional hypothesis (Kiros et al., 2015). We utilize the publicly available skip-thought model to obtain sentence representations. The third approach relies on BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) to generate sentence embeddings. httpp: / http//gg.tttt.tt////ee wtw.:ith.....aaiiiieeeeeeeoo com./ryankirrosskiippttttta thoughts
4.3 Backbones of Summarization Approaches There are several simple unsupervised sum- marization extraction frameworks, including TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004), as well as more robust graph-based ranking methods such as PAC- SUM (Zheng and Lapata, 2019), FAR (Liang et al., 2021), DASG (Liu al., 2021). Graph-based rank- ing methods take sentence representations as in- put, using the algorithm of graph- sentence

---

Table I: ROUGE F1 performance of the single docu- ment extractive summarization the CNN//ailyMail. is reported in Xu et al. (2020), is reported in Zheng and Lapata (2019) and $ reported in Liang (2021). means our careful re-implementation due to the absence of publicly accessible source code for these methods the experiment was missing from the pub- lished paper. The best results are in-bold.
Table 2: ROUGE performance of the multi-document extractive summarization the Multi-News. is ported in Fabbri et al. (2019), is reported in Li et al. (2020) and is reported in Wang et (2020). means careful implementation due to the absence of pub- liely accessible source code for these methods or the experiment was missing from the published paper. The best results are in-bold.
Method
LM
ROUGE-1
ROUGE-2
ROUGE-L
Method
LM
ROUGE-1
ROUGE-2
ROUGE-L
ORACLE
54.70
30.40
50.80
ORACLE
$2.32
22.32
47.93
LEAD-3
40.49
17.66
36.75
FIRST-3
40.21
12.13
37.13
ST
31.40
10.20
28.60
TF-IDF
38.44
13.10
13.50
TF-IDF
33.20
11.80
29.60
BERT*
41.95
13.86
38.07
TextRank
TextRank
BERT"
30.80
9.60
27.40
BERT
42.56
13.69
38.47
B-GAF
36.60
14.58
32.91
Bi-GAR
43.20
14.76
38.95
STVee
31.91
10.33
28.36
TF-IDF
38.27
12.70
13.20
TF-IDF
34.68
12.82
31.12
TF-IDF
41.01
12.69
18.00
LexRank
LexRank
BERT
27.50
7.38
24.63
BERT
41.77
13.81
37.87
Bi-GAE
39,73
16.81
36.02
BERT
40.97
12.93
37.21
Seevec
38.60
16.10
35.30
Br-GAE
42.91
14.28
38.83
TF-IDF
39,20
16.30
35,30
BERT
43.27
14.16
38.25
PACSUM
BERT
40.70
17.80
36.90
PACSUM
RoBERTa
41.33
13.33
37.59
BERT*
40.69
17.82
36,91
B--GAR
43,53
14.42
39.26
RoBERTa
40.74
17.82
36.96
BERT
42.60
13.22
16.15
DASG
Bi-GAE
41.29
18.22
37.49
RoBERTa
41.73
13.33
37.59
BERT
40.83
17.85
36.91
B--GAE
43,39
14.27
39.22
RoBERTa
40.87
17.42
36.31
BERT
43.40
14.35
36.26
FAR
Bi- Bi-A
41.26
18.
37.40
RoBERTa
43.08
14.07
39.00
BERT"
40.89
17.68
37.10
Bi-GAE
43.58
14.58
39.30
DASG
RoBERTaa
40.90
17.76
37.12
BI-GAE
41.37
18.25
37.56
tences that optimize the mean of ROUGE-1 and ROUGE-2 scores. The results indicate the fol- lowing: (i) Our pre-trained sentence representa- tion, which incorporates TextRank and LexRank extractive frameworks, yields prominent improve- ments in ROUGE-1/2/L performances. (ii) Our model acquires intra-sentential distinctive and inter- sentential cohensive features of sentences via pre- training sentence-word bipartite graphs, aiding graph- ranking for unsupervised summariza- tion. (iii) Our pre-trained sentence representation outperforms all other robust sentence representa- tion methods across all summarization frameworks.
4.5
Implementation Details and Metrics
pre-training the Bi-GAE, we choose the best model and hyper-parameters based on their perfor- mance on the validation set. Appendix 8.2 provides detailed information on the hyper-parameters used during the Bı-GAE pre-training procedure. For fine- tuning the unsupervised extractive summarization frameworks, there are few hyper- parameters to be tuned for computing centrality scores. The main syper-parammttes for the extractive summarization frameworks are listed in Appendix 8.3. We have kept the remaining hyper-parameters in the back- bones of summarization frameworks unchanged.
In contrast, sentence representations initialized with BERT or RoBERTa perform poorly in Tex- tRank and LexRank frameworks. This could be attributed to the collapse of BERT-derived sentence representations, resulting in high similarity scores for all sentences and thus failing to leverage the po- tential centrality in TextRank and LexRank. How- ever, our methods surpass BERT and RoBERTa
5 Results and Analysis
5.1 Single-Document Experiments
Our results on the CNN/Daily Mail are summa- rized in Table The Oracle upper bound extracts gold standard summaries by greedily selecting sen-

---

Table 3: ROUGE Fl performance of the extractive summarization. Pre-trained encoder in BI-GAE is equipped with one kind of GCNs (GCNizer or GCN FAR and DASG are two extractive frameworks, respec- tively, and are tested in the CNN/DailyMail dataset. The pre-training corpora used also is the downstream CNN/DailyMail dataset without summarization labels.
Table 4: ROUGE FI performance of Bi-GAE on the downstream CNNi summarization and Bi- GAE is pre-trained on the Multi-news dataset.
Method
ROUGE-1
ROUGE-2
ROUGE-L
TextRank
36.69 0.09
14.91 0.33
33.19 0.28
LexRank
40.13 1 0.40
17.16 0.35
36.41 T 0.39
Method
LM
ROUGE-1
ROUGE-2
ROUGE-L
PACSUM
41.12 1 0.17
18.09 0.13
37.34 0.15
B--GAE
GCN
41.18
18.18
37.37
FAR
41.17 0.03
18.18 0.01
37.41 0.02
W. GCN jastr
41.20
18.19
37.40
DASG
41.27 L 0.10
18.16 1 0.07
37.50 & 0.06
W. GoNe
41.27
18.15
37.46
Br-GAE
GCNista
41.26
18.13
37.44
Table 5: ROUGE performance on the downstream
Multi-news extractive summarization, in which the model is pre-trained on the CNN/DailyMail dataset.
the FAR and DASG summarization frameworks, showcasing the effectiveness of sentence represen- tations pre-trained by graph auto-encoders.
Method
ROUGE-1
ROUGE-2
ROUGE-I.
TextRank
43.27 0.07
14.76 0.00
39.02 0.07
5.2 Multi-Document Experiments
LexRank
42.87 0.04
14.28 0.00
38.81 0.02
Table 2 shows the comparison of Multi-news marization. Given that all frameworks employ- ing our pre-trained representations outperform the FIRST-3 baseline, our approach effectively mitigates position bias (Dong et al., 2021). This bias often re- sults in incomplete summaries that neglect essential information located in the middle of the document. The results demonstrate two key findings: (i) Our method adeptly captures essential summary-worthy sentences, thereby consolidating the process of sen- tence clustering and, turn, improving extractive accuracy. (ii) The embedded, intra-sentential dis- tinctive features and inter- sentential cohensive fea- tures are crucial in ranking significant sentences across multiple documents.
PACSUM
43.46 0.07
14.52 0.10
39.26 0.00
DASG
43.27 1 0.12
14.43 0.116
0.16
39.15 1 0.07
FAR
43.54 1 0.04
14.61 f 0.03
39.30 0.00
content of the document.
5.4 Effects of Pre-training Datasets
To evaluate the impact of different pre-training datasets, we test summarization frameworks using two types of representations pre-trained on distinct corpora. In Table 4 and Table 5, we can observe pre-training on the Multi-news dataset showed min- imal performance degradation or limited changes CNN/DailyMail summarization, and vice versa the similarity between the two news corpora leads to consistent results in downstream tasks.
5.3 Component-wise Analysis
To comprehend how modelling intra-sentential fea- tures and inter-sentential features contribute to sentence-wee word bipartite graphs, conducted an ablation study on the CNN/DailyMMil dataset. As shown in Table 3, we can observe that the Bi-GAE model equipped solely with GCNinter or solely with GCNintee performs well. When combined with both, Br-GAE yields the best results across all met- rics. This highlights the importance of incorpo- rating intra-sententia and inter-sentential features for effective summarization. Combining the two GCNs leads complementary effects, enhancing the model's overall performance. On the contrary, using only GCNinter or GCNimtre individually re- sults in poor performance, as it fails to capture either the semantically cohensive or the distinctive
5.5 Density Estimation of Summarization
There three measures density, coverage, and compression introduced by Grusky et al. (2018) and Fabbri al. (2019) to assess the extractive nature of an extractive summarization dataset. this paper, we adopt these measures to evaluate the quality of extracted summaries, as illustrated in Figure 3. The coverage (x-axis) measure assesses the degree to which summary is derived from the original text. The density (y-axis) measures the extent to which a summary can be described as a series of extractions. Compression c, on the other hand, refers to the word ratio between two texts Text A and Text B. Higher compression pose a challenge as it necessitates capturing the essential

---

figgnnest Jeaako
50
(809)
frggment, 144
1989
20003
fnggure
52
0.5
0.5
covertgge
(a) comp (O', M*)
(b) comp (B", M*)
(c) comp (B', M*)
(d) comp (B', O3)
Extrastve: Gagpent Stastty
Aaas: 9523
Exteetive Bagnen daealty
Extrstive Gauncat disstt
50
9027
9633
10
25.2
Extrrvtto frggnen
C-
0.5 fragmett coverge
Eagment coverag
fragmeet coverage
Bragoent coverage
(e) comp (BS.M MM)
(1) comp (B*, .M))
(g) comp (B.M.)
(h) comp (BI,0)
on CNN/Daily (subfigures (a), (b), (c), (d)) and Multi-News (subfigures (e). (f), (g), (h)) datasets. Each box represents a normalized bivariate density plot, showing extractive fragment coverage on the x-axis density on the y-axis. The top corner of each plot shows the number of text and the median compression ratio between text A and text B. The comp(A, B) denotes the comparison elements are the text used for comparing, and the text B used as the reference. comp(O', ) the Oracle and the manual summary, comp(B M"):: the extracted summary of BERT-based DASG and the manual summary. comp(B', M*): the extracted summary of our Bi-GAE based DASG and the manual summary, comp(B', O'): the extracted summary of our Bı-GA GAE based DASG and the Oracle.
Figure 3: Density and coverage distributions of extractive compression
aspects of the reference text with precision. For de- tailed mathematical definitions of these evaluation measures, please refer Appendix 8.6.
suggests greater inclination of the model to ex- tract fragments (novel words) that differ from stan- dard sentences. Our model also outperforms the BERT-based DASG in compression score (0.6522) to compare subfigures (b) and (c). Regarding the y- axis (fragment density) in subfigure (d), our model shows variability in the average length of copied sequences to the Oracle summary, suggesting vary- ing styles of word sequence arrangement. These advantages persist in the multi- dataset.
We utilize three measures that quantify the level of text overlap between the Oracle summary and the manual summary (subfigures (a) and (e)). (ii)
the summary extracted by the BERT-based DASG and the manual summary (subfigure (b) and (f)), (iii) the summary extracted by our Bi-GAE based DASG and the manual summary (subfigure (c) and (g)), and (iv) the summary extracted by our Bi-GAE based DASG and the Oracle (subfigure (d) and (h)). These measures are plotted using kernel density estimation Figure Among them, subfigure (a) displays the comparison between the Oracle summary compared to the manual summary, which serves as the upper bound for the density and cov- erage distributions of extractive compression score in extractive summarization. Subfigure (e) shows this score in the multi-news dataset.
Conclusion
In this paper, we introduce pre-training process that optimizes summary-worthy representations for extractive summarization. Our approach em- ploys graph pre- preitraining autoencoders learn intra- sentential and inter-sentential features on sentence- word bipartite graphs, resulting in pre-trained em- beddings useful for extractive summarization. Our model is easily incorporated into existing unsu- pervised summarization models and outperforms salient BERT-based and RoBERTa-based summa- rization methods with predominant ROUGE-1/2/L. score gains. Future work involves exploring the po- tential of our pre-trained sentential representations for other unsupervised extractive summarization tasks and text-mining applications.
Comparing the extractive summary of our Bi- GAE based DASG (DASG integrated by the sen- tence representation of our Bi-GAE) and the tractive Oracle summary in subfigures (a), (b), and (c), we have observed variability in copied word percentages for diverse sentence extraction in CNilDailyMail. lower score on the x-axis

---

7 Limitations
Conference of the Association for Computational Lin- guisties, ACL, Volume 1: Long Papers, pages 1074- 1084, Association for Computational Linguistics.
We emphasize the importance of pre-trained sen- tence representations in learning meaningful rep- resentations for summarization. In our approach, we pre-train the sentence-wond bipartite graph by predicting the edge betweenness score in a self- supervised manner. Exploring alternative centrality scores (such as TF-IDF score or current-flow be- tweenness for edges) as optimization objectives for MSE loss would be a viable option.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL-HLT, Volume (Long Papers), pages 708-719. Association for Com- putational Linguistics.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems, NewrlPS, pages 1693 1701.
Additionally, we seek to validate the effective- ness of the sentence representations learned from BI-GAE in other unsupervised summarization back- bones and tasks.
Acknowledgements
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8): 1780.
This work is supported by the National Natural Science Foundation of China (No.U20B2053 and No.62376270).
Ruipeng Jia, Yanan Cao, Hengzhu Tang, Fang Fang, Cong Cao, and Shi Wang. 2020. Neural extractive summarization with hierarchical attentive heteroge- neous graph network. In EMNLP, pages 3622-3631. Association for Computational Linguisties.
References
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. 2014. Spectral networks and locally connected networks on graphs. In 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2016. Variational graph encoeen. CORR, abs/1611.07308.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Pro- cessing Systems, NeuralPS, pages 3294-3302.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. 2020. Simple and deep graph con- volutional networks. In Proceedings of the 37th In- termational Conference Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pages 1725-1732 PMLR.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278- 2324.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL-HILT, pages 4171-4186, Associ- ation for Computational I. Linguistics.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph improve abstractive molti-document summarization. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL, pages 6232-6243, Association for Computational Linguis- tics.
Yue Dong. Andrei Romascanu, and Jackie Chi Kit Che- ung. 2021. Discourse-awaan aware unsupervised summa- rization for long scientific documents. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, EACL. pages 1002. Association for Computational I Linguistics.
Xinnian Liang, Shuangzhi Wu, Mu Li, and Zhoujun Li. 2021. Improving unsupervised extractive sum- marization with facet-aware modeling. In Findings of the Association for Computational Linguistics: ACL/IJCNLP, volume ACL/UJCNPP 2021 of Find- ings of ACL, pages 1685-1697. Association for Com- putational Linguistics.
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based based lexical centrality as salience in text sum- marization. J. Artif. Intell. Res., 22:457-479.
Alexander R. Fabbri, Irene I., Tianwei She, Suyi Li, and Dragomir& R. Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th
Jingzhou Liu, Dominic J. D. Hughes, and Yiming Yang. 2021. Unsupervised extractive text summarization with augmented sentence graphs. In SIGIR '11: The 44th International ACM SIGIR Conference

---

on Research and Development Information Re- trieval. pages 2313-233 2317. ACM.
extractive summarization. In ACL, Volume Long Papers. pages 2221 2227. Association for Computa- tional I Linguisties.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing. EMNLP-IJCNIP, pages 3728-3738. Association for Computational Linguistics.
Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In EMNLP-IJCNLP, pages 3009-3019. Association for Computational Linguis- tics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Discourse-aware neural extractive model for text summarization. CoRR, abs/1910.14142.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1162..
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomo- hiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80 of Proceedings of Ma- chine Learning Research, pages
Qianren Mao, Jianxin Li, JiaZheng Wang, Xi Li, Peng Hao, Lihong Wang. and Zheng Wang. 2022. Explic- itly modeling importance and coherence for timeline summarization. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP. pages 8062-80660 IEEE.
PMLR.
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei, and Ming 7hou, 2020. Unsupervised extractive summa- rization by pre-training hierarchical transformers. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL, pages 1784 1795. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, EMNLP, pages 404 411. Association for Computational Linguistics.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: recurrent neural network based sequence model for extractive summarization of doc- uments. In AAAI, pages 3075-3081. AAAT Press.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareck, Krishnan Srinivasan, and Dragomir R. Radev. 2017. Graph-based neural multi-document summa- rization. In CoNI., pages 452-462, Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summariza- tion with reinforcement learning. In NAACL-HLT, Volume I (Long Papers), pages 1747-1759. Associa- tion for Computational Linguistics.
Wenpeng Yin and Yulong Pei. 2015, Optimizing sen- tence modeling and selection for document summa- rization. In Proceedings of the Twenty-Fourt Inter- national Joint Conference on Artificial Intelligence, IJCAI, pages 1383-1389. AAAI Press.
Ani Nenkova and Kathleen McKeown. 2011. Auto- matic summarization. Found. Trends Inf. Retr. 5(2- 3): 103-333..
Hao Zheng and Mirella Lapata. 2019. Sentence cen- trality revisited for unsupervised summarization. Proceedings of the 57th Conference of the Associa- tion for Computational Linguistics, ACI. pages 6236- 6247. ACL.
Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia. 6( 6(12):22752.
Abigail See, Peter J. Liu, and Christopher D. Man- ning. 2017. Get to the point: Summarization with gointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics, ACL, Volume 1: Long Papers, pages 1073-1083. Association for Computational Linguistics.
Ming Zhong, Pengfei Liu, Danqing Wang. Xipeng Qiu, and Xuanjing Huang. 2019. Searching for effective neural extractive summarization: What works and what's 's next. In ACL, Volume 1: Long Papers, pages 1049-1058. Association for Computational Linguis- tics.
Qingyu Zhou, Nan Yang. Furu Wei, Shaohan Huang. Ming Zhou, and Tiejun Zhao. 2018 Neural docu- ment summarization by jointly learning score and select sentences. In ACL, Volume Long Papers, pages 654-663. Association for Computational Lin- guistics.
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, and Xuanjing Huang. 2020. Heterogencous graph neural networks for extractive document sum- marization. In ACI. pages 6209-6219. Association for Computational Linguistics.
Hong Wang, Xin Wang, Wenhan Xiong, Mo Yu, Xi- aoxiao Guo. Shiyu Chang, and William Yang Wang. 2019. Self-supervised learning for contextualized

---

8 Appendix
vocabulary 50,000 and initialize tokens with 300- dimensional GloVe 840B embeddings'. We filter stop words and punctuations when creating word nodes and truncate the input document to a maxi- mum length of 50 sentences. To eliminate the noisy common words, we remove 10% of the vocabulary with low TF-IDF values over the whole dataset. We initialize sentence nodes with d, 150. We use batch size of during pre-training and apply the Adam optimizer with a learning rate 5e-5 for CNNl and 2e-5 for Multi-News. The dropout is 0.1. The pre-training model trained for 210,000 steps, and the warm-up step set to 8000. Attempts made to invoke certain model inter- faces in PYG have revealed that using JKNET (Xu et al., 2018) and GCNII (Chen et al., 2020) the encoder backbone in the pre-training process sults in performance for downstream tasks that are essentially indistinguishable from those of GCN.
8.1 Details about Centrality Algorithms
The key idea of graph-based ranking to calculate the centrality score of each sentence (or vertex) described in section 2.1. In this section, we give the differences in centrality algorithms among several salient summarization backhone models.
The PACSUM (Zheng and Lapata, 2019) method enhances the centrality of two nodes in sentence graphs, considering how their relative positions in a document influence their importance.
Centrality(s)) (s)= d1 Σ eij d2
Σ pii
eih
jRi
where hyper-parameters d1, d2 are different weights for forward and backward-looking directed edges and d1 + d2 = 1. Ci.j is the normalized simi- larity score.
The FAR (Liang et al., 2021) approach enhances the centrality of two nodes with distance constraints in sentence graphs by considering how their relative positions in document influence their importance.
8.3 Hyper-parameters in Summarization
We begin by using Stanford NLP to split tences and preprocess the dataset. The source text has maximum sentence length of 512, while the summary is limited to a maximum sentence length of 140. During the tuning process for extractive summarization, we fine-tune the parameters related to the centrality algorithm within a narrow range of [-1.0, 2.0]. Table 6 presents the optimal hyper- parameters for each extractive summarization back- bones, utilizing B1-GAE pre-trained sentence representations. For the CNN/DailyMann dataset, we select the top-3 sentences for the summarization based on the average length of the Oracle human- written summaries, whereas, for Multi-New, we choose the top-9 sentences.
Centrality(s:) (::ralltt(ss)=A))) =1 Σ
Max((ei)-e)).)
jKi
(10)
+ A2
Max((eij - O),
where E = B. (max(er,) - min(ee,))) For $1, the threshold e can be seen as a diameter, $1 is the centre. B is hyper-parameter to control the scale of diameter.
The DASG (Liu al., 2021) method enhances the centrality of two nodes sentence graphs by taking into account their relative position and se- mantic facets within document.
8.4 Sentence Similarity Computation
Centrality(s) =
+11
eij
The crucial aspect of the unsupervised graph rank method downstream tasks lies the calculation of similarity between two sentences. In this regard, we examine two methods for calculating similarity, both of which draw inspiration from the similarity calculation approach utilized in PAcSuM(Zheng and Lapata, 2019). The first one can employ pair- wise dot product to compute an unnormalized similarity matrix Ejj = v vj. and the second one is cosine similarity E, = cos(v, v,). The final normalized
(11)
Σ pz
Cijj
where 1f..... dt and 2T..... are fixed hyper- parameters and set to be 3 empirically.
8.2 Hyper-parameters in Bi-GAE pre-training We mainly use PyTorch Geometric, PYG 4 to im- plement BI-GAE More specifically, we limit the
htttps://githbbe thub. www.hlu...cooppooaaee/a teagg-tttem/ptorrct geometric
hhttps://nlp.ste http://llpp.ttafor.... wdu/projects//looee http://///saaaaacccoo//ssttttttp /Corell

---

Datasets
Methods
Hyper-parameters
PacSum
Ai -1.0,
Maail
FAR
A = -0.5. = 0.9
DASG
B= 0=0.05 =
PacSum
=
Multi-News
FAR
Ai -0.5,
DASG
1.-11 45 2.0
B = - -5
Table 6: Main hyper-paramettee parameters of centrality algorithms for tuning extractive summarization with our Bi-GAE pre-trained sentence representations.
Table 7: ROUGE Fl performance of the extractive sum- marization. The pre-trained encoder our BI-GAE equipped with extractive frameworks DASG FAR, respectively, and is tested in CNN/DailyMai dataset. The pre-training corpora used also MNii dataset without summarization labels.
Table 8: ROUGE Fl performance of the extractive sum- marization. The pre-trained encoder in our B1-GAE is equipped with extractive frameworks DASG FAR, respectively, and is tested in the Multi-news dataset. The pre-training corpora used is the Multi-news dataset without summarization labels.
Method
Sim
ROUGE-I
ROUGE-2
ROUGE-L
Method
Sim
ROUGE-1
ROUGE-2
ROUGE-L
Bi-GAE DASG
41.13
17.97
37.34
B-GAE DASG
cos
43.39
14.27
39.22
dot
41.37
18.25
37.56
dot
43.12
14.16
38.99
41.20
18.19
37.40
cos
42.97
14.34
38.87
Bi-GAE FAR
B+GAE FAR
dot 41.26
18.24
37.45
dot 43.58
14.58
39.30
similarity matrix E is defined as:
the verification set. As shown in Figure 4, our find- ings indicate that the optimal prediction accuracy for both corpora typically ranges between 0.60 and 0.65. Based on these observations, we formulated the following hypothesis: when there are more unique nodes, their edge weights should be smaller since they are not shared by other nodes. Con- versely, when there are more shared nodes, their edge weights should be greater. The improvement in performance on downstream tasks validates the soundness of our hypothesis.
Eij- F.j- [minE + BlmaxE minE)].
(12)
where E,y is designed mitigate the influence of absolute values and instead emphasize the relative contributions of different similarity scores. The hyper-pann hyper-parameter B E [O, 1] controls the threshold below which the similarity score of Eij is set to O. Figure and Figure 8 illustrate the testing results of models using two similarities. Through empiri- cal analysis, we have discovered that the paie-wise dot product yields better performance in most cases on CNN//uillmmil summarization and Multi-news summarization. This finding aligns with the results reported in PAcSuM(Zheng and Lapata, 2019).
8.6 Characterizing Summarization Strategies
As shown in Figure 3, each box is normalized bi- variate density plot of extractive fragment coverage (x-axis) and density (y-axis), and the top left corner of each plot shows the median compression ratio c between text A and text B.
8.5 B--GAE Pre-training Validation
We meticulously fine-tune a multitude of param- eters in our process. For the pre-training of the CNN/Daily Mail corpus, we find that the optimal learning rate for model is 5e-5, with batch size of Similarly, in the pre-training of the Multi- news corpus, we find that the optimal learning rate is 2e-5 while maintaining batch size of 8.
Fragment Coverage Extractive fragment coverage is the percentage of words in the summary that are from the source article, measuring the extent which summary is derivative of a text:
Σ In. FEFLA,B)
COVERAGEIA. B)
(13)
To assess the pre-traine pre-training performances, we con- duct accuracy tests of the edge weight prediction on
where F (A, B) is the set of shared sequences

---

Acceney Valittionn
text and article text include: the comparison between extracted summary and manual summary. the comparison between the extractive Oracle and the manual summary. or the comparison between extracted summary and Oracle summary.
0.64
0.60
0.58
0.56
20
40
60
80
100
lieration Stops (10?)
(a) Pre-training on CNN/Daily Mail
Agcanec Validation
0.60
0.59
0.58
0.57
0.56
0.55
0.54
60
100
Iterationn Stops (10")
(b) Pre-training on Multi-news
Figure 4: Verification results of edge prediction accu- racy during Bi-GAE pre-training on CNN/Daily Mail and Multinews corpora.
tokens in A and B and is identified as extractive in greedy manner. For example, a summary (text B) with 10 words that 7 words are the same as its article (text A) and include new words will have COVERAGE(A, B) =0.7.
Fragment Density The density measure quantifies the average length of the extractive fragment to which each word in the text belongs.
1
Σ LAP
BENSITY(A,B)
(14)
B
JEF(A.B)
For instance, a summary (text B) might contain many individual words from the article (text A) and therefore have high coverage. For instance, mary might contain many individual words from the article and therefore have high coverage. For an article (text A) with a 10-word summary (text B) made of two extractive fragments of lengths and 4 would have COVERAGE(A, S) = 0.7 and DENSITY(A B) =2.5.
Compression Ratio The compression ratio is defined as the word ratio between the article and summary:
IA|
COMPRESSIONAL B)
(15)
[
Summarizing with higher compression is challeng- ing requires capturing more precisely the criti- cal aspects of the article text.
Among our settings about the above metrics, we have expanded the comparison between summary


Reviews:
[{"review_id": "UvQMLAVfsc", "reviewer": "Reviewer_psgw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer psgw:\n\nMany thanks for your positive feedback! We are pleased you found our paper: a clear and concise description, an interesting approach, and a detailed explanation. Here, we will carefully address your concerns and questions in detail:\n\n*1. Concern about 'our paper lacks novelty and the topic of extractive summarization using graph ranking is not new'.*\n\n- To the best of our knowledge, **our approach stands as the first effort in introducing the bipartite word-sentence graph pre-training method. Furthermore, we are also the first to employ bipartite graph pre-trained sentence representations in an unsupervised extractive summarization**. \n\n- In fact, **the other two reviewers confirm the novelty of our paper**: \"A new self-supervised learning method for extractive text summarization.\" [Reviewer VXBy] and an \"In-Depth Exploration of Graph Pre-training Auto-Encoder\" [Reviewer vmKw]. [Reviewer vmKw] also aptly affirmed, \"Our paper introduces a pioneering approach to pre-train sentence embeddings using a graph-based auto-encoder.\"\n\n- Additionally, we would like to reaffirm our initial motivation to clear up any possible confusion. Our intention is not to propose a new unsupervised summarization ranking method but rather to introduce a sentence representation approach suitable for unsupervised summarization, stemming from an upstream pre-training task. In other words, **our improvement is focused on the upstream of unsupervised summarization tasks**.\n\n*2. Concern about 'the improvement is not convincingly demonstrated.'*\n\n- We clarify that Sec 5, Tables 1 and 2 display the superior performance of our approach. Such improvement excludes the influence of irrelevant factors like data fluctuation and hypermeters since the unsupervised paradigm of downstream tasks. We will also release the source code. \n\n- In this context, we have summarized the performance of our model surpassing RoBERTa-based sentence representations in downstream tasks, as shown in the table. Notably, our model demonstrates significant enhancement, with an **average increase of 0.57 and 1.17 points in ROUGE scores across the two datasets**.\n\nCNN/Daily | ROUGE-1 | ROUGE-2 | ROUGE-L\n---|---|---|---\nROBERTa (+PacSum) | 40.74 | 17.82 | 36.96 |\nOurs (+PacSum) | 41.29($\\uparrow$0.55) | 18.22($\\uparrow$0.40) | 37.49($\\uparrow$0.53) |\nROBERTa (+FAR) | 40.87 | 17.42 | 36.31 |\nOurs (+FAR)| 41.26($\\uparrow$0.39) | 18.14($\\uparrow$0.72) | 37.40($\\uparrow$1.09) |\nROBERTa (+DASG) | 40.90 | 17.76 | 37.12 |\nOurs (+DASG)| 41.37($\\uparrow$0.47) | 18.25($\\uparrow$0.49) | 37.56($\\uparrow$0.44) |\naverage gains | **0.47** | **0.54** | **0.69** |\n\n\n\nMultiNews | ROUGE-1 | ROUGE-2 | ROUGE-L\n---|---|---|---\nROBERTa (+PacSum) | 41.33 | 13.33 | 37.59 |\nOurs (+PacSum) | 43.53($\\uparrow$2.20) | 14.42($\\uparrow$1.09)| 39.26($\\uparrow$1.67) |\nROBERTa (+FAR) | 43.08 | 14.07 | 39.00 |\nOurs (+FAR)| 43.58($\\uparrow$0.50) | 14.58($\\uparrow$0.51) | 39.30($\\uparrow$0.30) |\nROBERTa (+DASG) | 41.73 | 13.33 | 37.59 |\nOurs (+DASG)| 43.39($\\uparrow$1.66) | 14.27($\\uparrow$0.94) | 39.22($\\uparrow$1.63) |\naverage gains | **1.45** | **0.85** | **1.20** |\n\n\n- Our pre-training is tailor-made for the summarization task to alleviate the \"gap\" between upstream pre-training and downstream sentence ranking tasks. However, this gap persists in baseline pre-training methods, mostly unaddressed for unsupervised tasks due to infeasible fine-tuning in unsupervised scenarios. BERT's sentence bias leads to the collapse problem, confirmed by [Yan et al., ACL2020]. \n\n- Reviewer 3 has also affirmed our convincing contributions, such as: 'The experimental results prove the embeddings' effectiveness in two extractive summarization datasets, further strengthening its credibility. '\n\n- While our method has achieved exceptional performance, the article does not explicitly claim statistical significance for the improvements. We appreciate your insightful suggestions and plan to conduct rigorous significance testing to explore whether our results truly represent a significant enhancement in the final version of the paper.\n\n---\nAgain, we sincerely appreciate your review and feedback. We hope our responses have provided you with a clearer picture of our work and have addressed your concerns. Besides, we consider the comment 'improvement is not convincingly demonstrated' to be quite broad; we cannot provide any explanations without explicit mentions. We hope to have more explicit opinions and suggestions. \n\nSincerely & Best wishes\n\n---\n**References:**\n[1] Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, Weiran Xu: ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. ACL/IJCNLP (1) 2021: 5065-5075", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "One strength of the paper is the clear and concise description of the proposed method. The use of graph pre-training auto-encoders to model intra-sentential and inter-sentential features through sentence-word bipartite graphs is an interesting approach. The paper also provides a detailed explanation of how pre-trained sentence embeddings are used in the graph-based ranking algorithm, making it easier to understand and replicate the experiments.", "reasons_to_reject": "The main weakness of the paper lies in its limited novelty. The topic of extractive summarization using graph ranking is not new, and the proposed method does not demonstrate significant improvements over previous techniques. While the paper claims to outperform BERT- and RoBERTa-based methods, the extent of this improvement is not convincingly demonstrated through rigorous experiments and comprehensive comparisons with state-of-the-art approaches.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the task of unsupervised extractive summarization using graph-based ranking with pre-trained embeddings. However, the paper lacks novelty as the topic of extractive summarization with graph ranking has been explored extensively in the past. The proposed method does not demonstrate significant advancements over previous approaches, which weakens its contribution to the field.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MMrlzPgJst", "reviewer": "Reviewer_VXBy", "summary": "", "questions": "Question A: What is the computation cost for building the word sentence graph?  what could be the maximum input length when using your method?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VXBy:\n\nWe thank you for all the constructive feedback and positive comments. In the following, we address each concern raised by the reviewer individually.\n\n*1. Concern about other 'pre-trained methods'.*\n\n- Thank you for your suggestion！The two models suggested models, BART (Lewis et al., ACL2020) and PEGASUS (Zhang et al., ICML2020), are pre-training methods based on sequence-to-sequence architectures, primarily suitable for supervised abstractive summarization tasks. Whether they can be directly adapted for unsupervised extractive summarization tasks requires further discussion and dedicated design. \n\n*2. Concern about 'more evaluation metrics'.*\n\n- Thanks for your constructive comments. Except for the ROUGE, we also introduce three other metrics: Fragment Coverage, Fragment Density, and Compression Ratio. These three types of metrics have been widely employed as primary indicators to reflect the nature of a summary (by Fabbri et al., ACL2019, Sharma et al., ACL2019, Huang et al., NAACL 2021, Salemi et al., EMNLP 2021). Given this, we have opted for widely acknowledged metrics as the methods for assessing the characteristics of summaries, apart from ROUGE. \n\n- We created a pre-training method to capture informative aspects of sentence representations. As the described motivation of our paper [in Lines 070-074], the existing graph approach (like GCNs by Bruna et al., 2014) emphasizes encoding distinct sentences rather than always informative ones, potentially limiting the extraction of crucial sentences. For a deeper check on whether our model captures informative content, we prefer using Fragment Coverage, Fragment Density, and Compression Ratio to validate summary qualities thoroughly.\n\n- (can be discussed) We concur with the reviewers that coherence is a practical evaluative approach for further analyzing summary characteristics. However, the above three metrics we employ are more aligned with the original intention of our paper's focus on informative summaries than coherence. Moreover, achieving an objective and broadly accepted assessment might prove challenging due to the potential bias and subjectivity of human evaluations. We expect our approach to be validated using human validation and coherence metrics in the future. \n\n*3. Concern about 'not sure whether it works for clustering methods'.*\n\n-   Thank you for your careful review. All summarization backbones introduced in Chapter 9.1 are all sentence-ranking methods. In order to achieve fair comparison and maintain consistency, we have not used graph clustering modules for backbones. \n\n- Our pre-training is customized for summarization to address the \"gap\" between upstream pre-training and sentence ranking in summarization. In upstream pre-training, we explicitly capture intra-sentential distinct and inter-sentential informative features via sentence-word bipartite graphs. These embeddings aid in identifying distinct and informative sentences during ranking sentences in downstream summarization [our motivation is described in Lines 100-113]. Hence, we emphasize ranking-based models based on our graph pre-training approach. In future work, we can talk about more modules (like graph clustering), but not the focus of our work now. \n\n- Missing references will be cited in the final paper. Thanks again for your appreciation and constructive comments on our work.\n\n*Q\\&A.*\n\nThank you for your question. The computation cost of building the sentence-word bipartite graph is negligible (O(n)). Moreover, its advantages are evident since constructing a sentence-word bipartite graph does not require reliance on external tools or consideration of the error propagation issue.\n\n---\n\nThanks again for your appreciation and constructive comments on our work!\n\nBest regards\n\n---\n**References:**\n\n[1] Mike Lewis et al., BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. ACL.2020\n\n[2] Jingqing Zhang et al., PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. ICML.2020\n\n[3] Alexander R. Fabbri et al., Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. ACL.2019\n\n[4] Eva Sharma et al., BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization. ACL.2019\n\n[5] Luyang Huang et al., Efficient Attentions for Long Document Summarization. NAACL-HLT.2021\n\n[6] Alireza Salemi et al., ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization. EMNLP.2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Well written paper, easy to follow.\n* Experiment setup and comparison is clear.\n* A new self-supervised learning method for extractive text summarization. ", "reasons_to_reject": "* Since BERT and Roberta is pre-trained with local context, it's better to compare with other \"summarization-focused\" pre-trained method, for example, take the encoder part of BART or PEGASUS. \n* Apart from ROUGE, more evaluation metrics for summary redundancy/coherence is needed. Also consider some human evaluation. \n* The method is only evaluated with ranking based models, not sure whether it works for clustering based approach such as [1].\n\n[1] Zhao, Jinming, et al. \"Summpip: Unsupervised multi-document summarization with sentence graph compression.\" In Proceedings of the 43rd international acm sigir conference on research and development in information retrieval, pp. 1949-1952. 2020. \n", "questions_to_authors": "Question A: What is the computation cost for building the word sentence graph?  what could be the maximum input length when using your method?", "paper_topic_and_main_contributions": "This paper proposed a graph auto-encoder based pre-training method to obtain sentence representation, the graph is constructed by considering both intra and inter sentence word connection. Compared with other representation methods such as TFIDF and BERT, the proposed method works better when fed with graph ranking models for extractive text summarisation. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Zhao, Jinming, et al.  \"Summpip: Unsupervised multi-document summarization with sentence graph compression.\" In Proceedings of the 43rd international acm sigir conference on research and development in information retrieval, pp. 1949-1952. 2020."}}, {"review_id": "66r4w0yi4k", "reviewer": "Reviewer_vmKw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer vmKw:\n\nWe would like to thank you for strongly supporting our paper! [Reviewer psgw, VXBy] have also affirmed our contributions, such as:\n1. An interesting approach [Reviewer psgw]\n2. Provides a detailed explanation [Reviewer psgw]\n3. Making it easier to understand and replicate the experiments [Reviewer psgw]\n4. Clear and concise description [Reviewer psgw] Well-written paper, easy to follow [Reviewer VXBy]\n5. Experiment setup and comparison is clear [Reviewer VXBy]\n6. A new self-supervised learning method for extractive text summarization [Reviewer VXBy]\n \nHere we will carefully address your concerns in detail.\n\n- The dataset we chose is an intersection of the datasets used in several baselines in the original papers. These baselines are evaluated on both single-document summarization and multi-document summarization tasks. \n\n- As far as I know, dialogue summarization datasets (ICSI or AMI dataset) are aligned with abstractive summarization, making them more suitable for validating abstractive summarization methods. Furthermore, our baselines have yet to be experimented with or have their performance results disclosed on these two datasets. Reproducing the results of baselines on a new dataset would entail costs and meticulous replication.\n\n- Besides, the ARXIV dataset has domain disparities compared to news corpora. Whether the pretraining on news-based corpora is also warranted in a new domain becomes a topic worthy of exploration. For experiments on new-domain datasets, the initial step involves constructing a bipartite graph based on the source documents, followed by bipartite graph pretraining and downstream summarizaztion. Considering the invisible workload, more complementary experiments（like other extractive summarization datasets）can be explored in future endeavours. \n\n- Our pretraining approach was conducted on news-related datasets (the downstream news summarization dataset we employed). The BERT and RoBERTa models we are comparing were also primarily pre-trained on extensive news corpora. This ensures a relatively fair domain consistency for our pretraining corpus. \n\n- Missing References will be cited in our final manuscript. \n\n---\n\nThank you for all the constructive feedback and positive comments. \n\nSincerely & Best wishes", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **In-Depth Exploration of Graph Pre-training Auto-Encoder:** The paper presents a thorough investigation into the graph-based pre-training auto-encoder, enabling the learning of both intra-sentential and inter-sentential features. This unique approach contributes to the development of highly valuable pre-trained embeddings specifically tailored for extractive summarization.\n2. **Versatility and Effectiveness in Different Summarization Methods:** By showcasing the adaptability of the proposed embeddings in diverse unsupervised summarization techniques, the paper substantiates the practicality of its method. The experimental results provide strong evidence of the embeddings' effectiveness in two extractive summarization datasets, further strengthening its credibility.", "reasons_to_reject": "1. **Confusing Textual Description:** Some portions of the paper lack clarity and context. For instance, the text \"These fine-tuned sentence embeddings\" in line 17 is ambiguous and does not provide sufficient explanation regarding the fine-tuning process for sentence embeddings. Enhancing the clarity of such descriptions will ensure a better understanding of the methodology.\n2. **Limited Experimentation Domains:** While the paper exhibits promising results on the news domain dataset, it would be beneficial to broaden the scope of experimentation to include various domains, such as scientific articles (arxiv summarization dataset) and dialogue summarization (ICSI dataset or AMI dataset). Conducting experiments in diverse domains will enhance the generalizability and robustness of the proposed method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces a pioneering approach to pre-train sentence embeddings using a graph-based auto-encoder. The proposed method effectively bridges the gap between pre-training and sentence-ranking tasks in unsupervised summarization. By leveraging intra-sentential and inter-sentential features from a sentence-word bipartite graph, the pre-trained embeddings exhibit promising utility in extractive summarization. Notably, the experimental results demonstrate that these embeddings can be seamlessly integrated into various sentence-ranking methods, such as TextRank, PacSum, and FAR, yielding superior performance compared to traditional embeddings like BERT and RoBERTa.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Sun, Shichao, et al. \"Improving Sentence Similarity Estimation for Unsupervised Extractive Summarization.\" *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2023.\n\n@inproceedings{sun2023improving,\n  title={Improving Sentence Similarity Estimation for Unsupervised Extractive Summarization},\n  author={Sun, Shichao and Yuan, Ruifeng and Li, Wenjie and Li, Sujian},\n  booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},\n  pages={1--5},\n  year={2023},\n  organization={IEEE}\n}"}}]

Author rebuttal:

Meta-review: The paper is the clear and concise description of the proposed method, However, there are some insufficient works, such as evaluation metrics for summary redundancy/coherence,  human evaluation, etc. 
Moreover, more advanced datasets and flexible tasks are needed to contribute the development of extractive summarization, other than CNN/DM or MDS. 
Therefore, I would suggest the paper accepted as Findings in the conference.
Decision: Accept-Findings