Venue: EMNLP 2020

# A Spectral Method for Unsupervised Multi-Document Summarization

Kexiang Wang\({}^{1}\), Baobao Chang\({}^{1,2}\) and Zhifang Sui\({}^{1,2}\)

\({}^{1}\)Key Laboratory of Computational Linguistics, Ministry of Education,

School of Electronics Engineering and Computer Science, Peking University, Beijing, China

\({}^{2}\)Peng Cheng Laboratory, Guangdong, China

{wkx,chbb,szf}@pku.edu.cn

###### Abstract

Multi-document summarization (MDS) aims at producing a good-quality summary for several related documents. In this paper, we propose a spectral-based hypothesis, which states that the goodness of summary candidate is closely linked to its so-called spectral impact. Here spectral impact considers the perturbation to the dominant eigenvalue of affinity matrix when dropping the summary candidate from the document cluster. The hypothesis is validated by three theoretical perspectives: semantic scaling, propagation dynamics and matrix perturbation. According to the hypothesis, we formulate the MDS task as the combinatorial optimization of spectral impact and propose an accelerated greedy solution based on a surrogate of spectral impact. The evaluation results on various datasets demonstrate: (1) The performance of the summary candidate is positively correlated with its spectral impact, which accords with our hypothesis; (2) Our spectral-based method has a competitive result as compared to state-of-the-art MDS systems.

+
Footnote †: November 4, 2020. ©2020 Association for Computational Linguistics

## 1 Introduction

Given a cluster of documents related to the same topic or event, the task of multi-document summarization (MDS) centers on a brief summary of the cluster. As emphasized by Lebanoff et al. (2018), for this task, the labeled training data (i.e. cluster-summary pairs) are scarce. Hence dealing with it in an unsupervised paradigm becomes a reasonable choice. For the unsupervised MDS task, the automatic summarizer is required to discover the main content of the document cluster without the guidance of golden summaries. To preserve the fluency and grammaticality of summary, we mainly focus on the extractive method in which summary sentences are extracted from the original document cluster.

In this paper, we propose a novel spectral-based hypothesis for the unsupervised MDS task. The hypothesis states that the goodness (or effectiveness) of any summary candidate is closely linked with its spectral impact on the document cluster. The spectral impact of a summary candidate quantifies the perturbation to the dominant eigenvalue (in modulus) of affinity matrix when dropping the candidate from the document cluster. In other words, the hypothesis points out the spectral impact as an indicator of the MDS task, which is the first attempt to characterize MDS from a spectral viewpoint explicitly. As a representation of the document cluster, the affinity matrix supports the definition of spectral impact. Adjusting the building of the affinity matrix can bring out the best in the hypothesis. To validate the proposed hypothesis, we provide both theoretical explanations and empirical evidence. _Theoretically_, the spectral impact caused by dropping a summary from the cluster can be characterized from three different perspectives (see SS2.4). _Empirically_, for any summary candidate, the real dataset witnesses a positive correlation between its performance and computed spectral impact. For a particular MDS task, applying the hypothesis leads to a constrained optimization problem where the objective function is spectral impact. Our summarize utilizes an accelerated greedy algorithm based on a surrogate of spectral impact. The competitive results of our summarizer have been obtained on various datasets.

The differences between prior works and our method are clarified for unsupervised MDS:

(1) _Underlying hypothesis_. The hypothesis indicates the mechanism for the summarization. For instance, manifold-ranking-based methods share the hypothesis that a good summary sentence has a high ranking on the low-dimensional manifold that documents reside in Wan et al. (2007); Cheng et al. (2011); Li et al. (2011). However, the reasonableness

of this manifold hypothesis has not been directly evaluated. Another hypothesis in the sparse-coding-based methods (Li et al., 2015; Liu et al., 2015; Yao et al., 2015) regards the original sentences as a linear combination of summary sentences. This leads to an intuitive reconstruction, whereas linear combination is more a simplification than a necessity. Our proposed hypothesis offers a spectral viewpoint and will be explicitly validated on the real dataset.

(2) _Optimization objective_. Multi-criteria optimization is suitable for MDS as various criteria (goals) exist in the task, such as relevancy criterion and non-redundancy criterion. For instance, Lin and Bilmes (2011) is a bi-criteria case that imposes the submodularity constraint on each criterion. Multi-criteria loss functions in neural-network-based methods (Ma et al., 2016; Chu and Liu, 2019; Zheng et al., 2019) include the reconstruction errors from different spaces. In the above cases, the overall objective functions used include some hyperparameters for gluing singletons. Comparatively, our proposed objective (spectral impact) has a compact form. It avoids the hyperparameter setting and simulates the non-separable processing of multiple MDS criteria by human beings.

(3) _Model complexity_. There is a trade-off between model complexity and model interpretability. For instance, the reported performance of the aforementioned deep-neural-based models is elusive, and there exists no general principle to further improve them. Our summarizer realizes the interpretable behavior based on verified hypothesis while preserving enough model complexity by the flexible affinity matrix (as a plug-in).

Our main contributions are twofold: (1) A novel spectral-based hypothesis for unsupervised MDS, which gains support from both theoretical and empirical sides; (2) An accelerated greedy algorithm for solving the hypothesis-driven optimization problem.

The rest of the paper is organized as follows. SS2 gives the details of our method, including the spectral-based MDS hypothesis and the greedy algorithm to solve the spectral optimization problem. Evaluation results, related work and conclusions are covered in SS3, SS4 and SS5 respectively.

## 2 Spectral-based MDS

What role does the summary play in the process of MDS? Our proposed hypothesis offers a spectral insight and brings out a workable formulation of MDS.

### Notations

We use calligraphic fonts for sets, capital bold letters for matrices and lower-case bold letters for vectors. The universal set \({\cal C}\) is formed by splitting and gathering the sentences from document cluster, i.e. \({\cal C}\!=\!\left\{s_{1},s_{2},...,s_{n}\right\}\) (\(s_{i}\) represents the \(i\)-th sentence and \(n\) is the total number of sentences). Each sentence has its ordinal number, e.g. \(o_{i}\) of sentence \(s_{i}\) indicates it is the (\(o_{i}\))-th sentence in the document that \(s_{i}\) belongs to. The summary candidate (subset of \({\cal C}\)) is denoted as \({\cal S}\). We represent the affinity matrix of document cluster as: \({\bf A}\!=\!\left\{a_{ij}\right\}_{n\times n}\). In addition, the dominant eigenvalue (in modulus) and the corresponding eigenvector of \({\bf A}\) are denoted as \(\lambda({\bf A})\) and \({\bf v}\), respectively.

**Dropping a set from a matrix**: emptying all the rows and columns whose indexes occur in the set. Consider the operation of dropping \({\cal S}\) from \({\bf A}\). If we denote the operation itself and the resultant matrix as \({\bf A}\backslash{\cal S}\), then

\[{\bf A}\backslash{\cal S}=\begin{cases}0,&s_{i}\in{\cal S}~{}\text{or}~{}s_{j }\in{\cal S},\\ a_{ij},&\text{otherwise}.\end{cases}\]

### Spectral Hypothesis

When representing the document cluster as a matrix, the matrix spectrum (i.e. a collection of eigenvalues) can uncover its different facets. Note that the dominant eigenvalue especially corresponds to the key facet, which gives a clue as to the main content that the summarizer needs to discover. For the extractive MDS, we propose the spectral-based hypothesis:

Given: Affinity matrix \({\bf A}\), the matrix representation of document cluster; set \({\cal S}\), any summary candidate including some original sentences.

Definition: Spectral impact of \({\cal S}\) is the perturbation to dominant eigenvalue of \({\bf A}\) when dropping \({\cal S}\) from \({\bf A}\), i.e. \(\Delta\lambda({\cal S})\!\triangleq\!\lambda({\bf A})\!-\!\lambda({\bf A} \backslash{\cal S})\).

Hypothesis: Goodness of \({\cal S}\) as a summary has a close link with its spectral impact \(\Delta\lambda({\cal S})\).

The above hypothesis tells us that the goodness (effectiveness) of any summary candidate can be determined by the proposed spectral impact, which reflects the change of dominant eigenvalue when the summary candidate \({\cal S}\) is left out. Affinity matrix \({\bf A}\) supports the definition of spectral impact, which 

stores the pairwise affinity of sentences as the name suggests.

For a specific MDS task, applying the hypothesis leads to an optimization formulation as follows:

\[{\cal S}^{*}=\operatorname*{arg\,max}_{{\cal S}\subseteq{\cal C}} \Delta\lambda({\cal S}),\quad\text{s.t.}~{}~{}|{\cal S}|\leqslant k. \tag{1}\]

The set \({\cal C}\) denotes the universal set and the number \(k\) specifies the maximum capacity of candidate \({\cal S}\).

The above formulation sets the spectral impact to be the objective function. The inherent rationality can be verified partially by these properties:

(a) _monotonicity_: \(\Delta\lambda({\cal S}_{1})\leqslant\Delta\lambda({\cal S}_{2})\) for any \({\cal S}_{1}\!\subseteq\!{\cal S}_{2}\) (see Li et al., 2012, Theorem 1);

(b) _normalization_: \(\Delta\lambda(\Phi)=0\), \(\Delta\lambda({\cal C})=\lambda({\bf A})\) (\(\Phi\) denotes empty set).

In the context of MDS, property (a) points out that a whole summary has more goodness than its components and property (b) regulates a reasonable range of the goodness of any summary candidate.

An overview of our hypothesis can be found in Figure 2.1. The document cluster with its matrix representation **A** is depicted in the left part, while the right part gives the dropping operation and the computation of spectral impact for two summary candidates \({\cal S}_{p}\) and \({\cal S}_{q}\). The goodness of each candidate is judged by their spectral impacts, and the winner \({\cal S}^{*}\) stands out with the largest spectral impact.

Notice that cardinality constraint is adopted in Problem (1) to specify the length limit of summary. Other reasonable constraints are also available, such as the knapsack constraint and the non-uniform matroid constraint (Welsh, 1976). The relevant conclusions and solutions discussed in the following sections continue to be applicable for those constraints.

### Affinity Matrix

Many prior works have adopted **A** for the MDS task, such as Yang et al. (2018) and Yang et al. (2019). Each element in the affinity matrix **A** is a pairwise affinity of two different sentences. Since our hypothesis depends on **A**, a better MDS performance can be expected by adjusting the building of **A**. Sentence embeddings play a vital role in the process of building **A**, since affinity \(a_{ij}\) can be set to be the cosine similarity of the embeddings of sentences \(s_{i}\) and \(s_{j}\) (i.e. \(a_{ij}\!=\!a_{ji}\) and \(a_{ii}\!=\!0\)). For comparison purposes, we consider the following three strategies of building sentence embeddings.

**Tf-isf**: the simple tf-idf model with a finer granularity. More details can be found in Wan et al. (2007) and Wang et al. (2017).

**ESE**: the enhanced feature embedding model (Yang et al., 2019). The embedding of each sentence is the concatenation of all components: paragraph vector, positional embedding and three feature embeddings (namely word-part-of-speech, bigram and trigram).

**BERT**: the sentence encoder that learns vector representations by pre-training a deep bi-directional Transformer network (Devlin et al., 2019). The advantage is that BERT is context-sensitive when considering the word embedding.

Notice that the leading sentences in each document should have priority in the summary extraction. For injecting this knowledge, \(a_{ij}\) is multiplied

Figure 2.1: An example depicting the spectral-based hypothesis for the task of unsupervised MDS (\(n\!=\!5\), \(k\!=\!2\)). The hypothesis suggests using spectral impact to judge whether a summary candidate is good or not.

 

by the average positional weight \(1/(o_{i}+o_{j})\). This can differentiate the sentences across documents and preserve the symmetry of **A**.

### Justifications of Hypothesis

We validate our spectral-based hypothesis by the following three complementary perspectives:

**Semantic scaling**: _dominant eigenvalue of affinity matrix determines the vector scaling in semantic space_. The \(n\)-dimensional semantic space is constructed as follows: Each sentence in the document cluster represents a different dimension and the \(i\)-th axis of the space corresponds to the \(i\)-th sentence. Then the affinity matrix \(\textbf{A}_{n\times n}\) can be seen as a linear operator on this semantic space and the pairwise affinity \(a_{ij}\) regulates the transformation between the \(i\)-th axis and \(j\)-th axis. Given an arbitrary nonzero vector **x** in the space, the transformed vector is **Ax**. Then the property holds:

\[\|\textbf{Ax}\|\leqslant\lambda(\textbf{A})\|\textbf{x}\|. \tag{2}\]

Notation \(\|\cdot\|\) denotes the Euclidean norm of vector. This is a sharp bound as equality holds only if **x** is the dominant eigenvector of operator **A**.

The property shows that the scaling up of any vector (namely \(\big{\lvert}\textbf{Ax}\big{\rvert}/\|\textbf{x}\|\)) is not larger than \(\lambda(\textbf{A})\). Hence, the dominant eigenvalue \(\lambda(\textbf{A})\) characterizes the ability of operator **A** to scale up any vector in the semantic space that document cluster resides in. When dropping the summary candidate \(\mathcal{S}\), the transformations to and from all axes covered by \(\mathcal{S}\) will no longer exist for operator **A**. In other words, there is no contribution for scaling up vectors from the \(i\)-th axis for any sentence \(s_{i}\in\mathcal{S}\). When the best-quality summary is dropped, the main components of operator **A** are emptied, which causes the largest reduction of its ability to scale up vectors. Therefore, the dominant eigenvalue, indicator of this ability, can be used to locate the multi-document summary, as proposed in our hypothesis.

**Propagation dynamics**: _isolating the summary blocks the information dynamics_. In this perspective, there is a spread of information over the document cluster according to the underlying network specified by matrix **A**. The pairwise affinity \(a_{ij}\) indicates the propagation rate between sentences \(s_{i}\) and \(s_{j}\) (more similar they are, more rapid the propagation occurs). The question that arises here is whether the information propagated from a few seed sentences will form a pandemic or become extinct in the long term. In epidemiology, the virus (information) will form a pandemic only if the basic reproduction number \(R_{0}\) of this virus is larger than \(1\)(Jones, 2007). For instance, \(R_{0}\) of COVID-19 is about \(3.28~{}(>1)\)(Liu et al., 2020), which uncovers the inevitable propagation of this virus.

Many works (Wang et al., 2003; Prakash et al., 2012; Chen, 2018) have found out that \(R_{0}\) is proportional to the dominant eigenvalue of the underlying information network. Thus a small dominant eigenvalue corresponds to a small value of \(R_{0}\), which hinders the information propagation. For the MDS task, when isolating the best-quality summary (i.e. \(\textbf{A}\backslash\mathcal{S}\)), the remainder of the document cluster will become the hardest for information propagation. Our hypothesis is consistent with this finding as the summary \(\mathcal{S}\) found by solving Problem (1) is able to reduce \(\lambda(\textbf{A}\backslash\mathcal{S})\) the most.

**Matrix perturbation**: _spectral impact considers both the relevancy and non-redundancy goal of MDS_. For analyzing the behavior of spectral impact, we expand it using first-order matrix perturbation theory (Stewart, 1990) as follows:

\[\begin{split}\Delta\lambda(\mathcal{S})&=\textbf{u }^{\prime}\,\textbf{Eu}+\mathcal{O}(\left\lVert\textbf{E}\right\rVert^{2})\\ &=2{\sum_{s_{i}\in\mathcal{S}}{u_{i}}^{2}\lambda(\textbf{A})}{ \sum_{s_{i},s_{j}\in\mathcal{S}}{u_{i}}{a_{ij}}u_{j}}+\mathcal{O}(\left\lvert \textbf{E}\right\rVert^{2}),\\ \text{where}&\textbf{E}=\textbf{A}-\textbf{A} \backslash\mathcal{S},\textbf{Au}=\lambda(\textbf{A})\textbf{u},\left\lvert \textbf{u}\right\rvert=1.\end{split} \tag{3}\]

Let us analyze each term of the expansion shown in Eq. (3). The first sums up the score of \(2{u_{i}}^{2}\lambda(\textbf{A})\) for any sentence \(s_{i}\in\mathcal{S}\). The value of \(u_{i}\) is a measure for the relevancy of sentence \(s_{i}\), since eigenvector centrality has been typically used for ranking sentences (Erkan and Radev, 2004; Bellaachia and Al-Dhelean, 2014; Al-Dhelean, 2015). Hence, the first term is an indicator for the **relevancy** of summary \(\mathcal{S}\). The second term is a penalty that considers every pair of summary sentences. Specifically, the penalty is \(u_{i}a_{ij}u_{j}\) for sentences \(s_{i}\) and \(s_{j}\). When \(a_{ij}\) is large (sentences are redundant), the penalty becomes prominent. Thus the second term measures the **non-redundancy** of summary \(\mathcal{S}\). The third term \(\mathcal{O}(\left\lVert\textbf{E}\right\rVert^{2})\) is relatively small compared to the preceding two because matrix **E** is nearly dominated by zeros (\(\left\lvert\mathcal{S}\right\rvert<<n\)). Hence the third term will not change the main behavior of \(\Delta\lambda(\mathcal{S})\).

### Algorithm

The naive idea for solving Problem (1) is to enumerate all possible combinations and find the best summary. The time complexity is \(\binom{n}{k}n^{2}\) if it takes \(\mathcal{O}(n^{2})\) time to compute the dominant eigenvalue 

of matrix (say using the method proposed by Lanczos (1950)). This exact enumeration algorithm is _infeasible_ even when \(n\) is 500 and \(k\) is 5.

Theoretically, Problem (1) falls into spectral optimization that has been proved to be NP-hard in many cases Van Mieghem et al. (2011). To avoid the time-consuming eigen-decomposition, some works resort to the QR decomposition of matrix Li et al. (2015a; Chen et al. (2018)). However, to actually compute QR decomposition, they depend on the Gram-Schmidt process that is inherently numerically unstable, which impedes the optimization process. In this paper, we bypass all these matrix decomposition and propose a straightforward surrogate for spectral impact, which is both effective and efficient. Based on a bound for dominant eigenvalue of \({\bf A}\backslash{\cal S}\) (Theorem 2.14 in Stevanovic (2014)), the surrogate is proposed as follows:

**Surrogate of spectral impact:**

\[\Delta\lambda({\cal S})\approx\frac{\sum_{s_{i}\in{\cal S}}v^{2}_{ i}\lambda-\sum_{s_{i},s_{j}\in{\cal S}}v_{i}a_{ij}v_{j}}{\sum_{s_{i}\in{\cal C }}v^{2}_{i}-\sum_{s_{i}\in{\cal S}}v^{2}_{i}}\]

where \(\lambda\) is the dominant eigenvalue \(\lambda({\bf A})\) and \({\bf v}^{1}\) is its corresponding eigenvector of matrix \({\bf A}\).

By using the surrogate for acceleration, we consider a greedy strategy to iteratively select \({\cal S}\), which is listed in Alg. 2.1. First, we compute the dominant eigenvalue \(\lambda\) and eigenvector \({\bf v}\) of \({\bf A}\) (line 1). At each iteration (lines 3 to 7), the sentence \(s_{\tau}\) maximizing the marginal gain of \(\Delta\lambda({\cal S})\) is extracted based on the previously selected set \({\cal S}\) (i.e. maximizing \(\Delta\lambda({\cal S}\cup\{s_{j}\})-\Delta\lambda({\cal S})\), line 4) and added to \({\cal S}\) (line 5). Also, the auxiliary vector \({\bf w}\) and scalar \(x\) should be updated according to the numerator and denominator of the surrogate (lines 6, 7). The operator '\(\odot\)' and '\(\cdot\)', for any two vectors, are their Hadamard product and inner product, respectively.

```
Input: the affinity matrix \({\bf A}\) and the budget \(k\) Output: the summary \({\cal S}\)
1 Compute the dominant eigen-pair \((\lambda,{\bf v})\) of \({\bf A}\); Initialize: \({\cal S}\leftarrow\Phi\), \({\bf w}\!\shortrightarrow\!\lambda{\bf v}\odot{\bf v}\), \(x\!\shortrightarrow\!{\bf v}\cdot{\bf v}\);for\(i\gets 1:k\)do
2 Let \(\tau\!\shortleftarrow\!\arg\max_{j}\{\frac{w_{j}}{x-v^{2}_{j}}|j\!\in\![1,n]; s_{j}\!\notin\!{\cal S}\}\); Add \(s_{\tau}\) to \({\cal S}\); Update: \(w_{j}\!\shortleftarrow\!w_{j}-2v_{j}a_{j\tau}v_{\tau}\) for all \(s_{j}\!\notin\!{\cal S}\); Update: \(x\!\shortrightarrow\!x-v^{2}_{\tau}\); return\({\cal S}\)
```

**Algorithm 2.1**Accelerated Spectral MDS

The lemma below demonstrates that Alg. 2.1 has a quadratic time complexity, which is evidently better than the exponential one of naive enumeration.

**Lemma 2.1**.: _The time complexity of Alg. 2.1 is \({\cal O}(n^{2}+kn)\)._

Proof.: Computing the dominant eigen-pair of matrix \({\bf A}\) takes \({\cal O}(n^{2})\) time. The initializations of the vector \({\bf w}\) and scalar \(x\) are both linear time operations, i.e. \({\cal O}(n)\). At each iteration, all \(n\) sentences need to be traversed for extracting \(s_{\tau}\) and updating \({\bf w}\). Thus the total complexity is \({\cal O}(n^{2}+kn)\). 

To get the final summary, we reorder the summary sentences returned by Alg. 2.1 according to their positions in the corresponding document.

## 3 Experiments

### Datasets

Three datasets are selected in the following experiments to provide a complete evaluation of our method. Two domains have been taken into account: news (DUC and Multi-News) and business reviews (Yelp). Table 3.1 lists some key characteristics of these datasets.

**DUC 20042** (task 2): the DUC task that contains a benchmark dataset. There are 50 document clusters, each of which includes 10 documents about the same news event. In addition, four human-written summaries are offered for each cluster to be the reference (golden) summary.

Footnote 2: [https://duc.nist.gov/duc2004/tasks.html](https://duc.nist.gov/duc2004/tasks.html)

**Yelp3**: an all-purpose dataset that can be utilized for MDS. We only use the subset that has the reference summary (the test split offered by Chu and Liu (2019)): 100 businesses (document clusters), each of which includes 8 reviews (documents). One reference summary was collected for each cluster using crowdsourcing. More details of building the dataset can be found in Chu and Liu (2019).

Footnote 3: [https://www.yelp.com/dataset](https://www.yelp.com/dataset)

**Multi-News4**: a large-scale dataset collected from news aggregator Fabbri et al. (2019). It has 5622 document clusters (in the test split offered by the original paper), and multiple documents are present 

in each cluster. Furthermore, each cluster is attached with one human-written reference summary.

### Experimental Details

For the extractive MDS method (including ours), the pre-processing includes paragraph splitting, sentence splitting and word tokenization. In our method, all the splitted sentences are gathered in set \({\cal C}\). The input of Alg. 2.1 includes the affinity matrix **A** which is built according to the strategies stated in SS2.3. Specifically, the strategy utilizing tf-isf vectors has a word bag that contains all the stemmed words found in the dataset (word stemming using Porter's stemmer5). For the strategy ESE, we pre-trained all different sentence embeddings on Daily Mail dataset Hermann et al. (2015) by following the guideline of Yang et al. (2019) (the dimension of concatenated embedding is 800). For the strategy BERT, we used the uncased BERTBase model6 pre-trained on Wikipedia, through bert-as-service7 to obtain the sentence embedding of 768 dimensions. All the experiments are performed on a machine with two CPUs (3.5GHz) and one GPU (16G memory).

Footnote 5: [https://tartarus.org/martin/PorterStemmer/](https://tartarus.org/martin/PorterStemmer/)

Footnote 6: [https://github.com/google-research/bert](https://github.com/google-research/bert)

Footnote 7: [https://github.com/hanxiao/bert-as-service](https://github.com/hanxiao/bert-as-service)

The extractive MDS methods need a length limit of summary to terminate the extraction of summary sentences. We adopt 100 words as the length limit in the DUC dataset, instead of 665 bytes specified by the official task. The change has also been made to provide the same setting for evaluating various methods in Hong et al. (2014) and Zheng et al. (2019). For the Yelp dataset, we set the limit to be the 99.5th percentile less than the maximum length of any document; for Multi-News, the limit is set as 300 words. The same settings have been adopted in Chu and Liu (2019) and Fabbri et al. (2019), respectively.

### Evaluation Metrics

We adopt ROUGE Lin (2004) as the automatic metric, which has been observed in a good agreement with human judgment Owczarzak et al. (2012). It measures the overlap of \(N\)-grams (R-N) and skip-bigrams with a maximum distance of four words (R-SU4). Also, it can be computed based on the longest common subsequence (R-L). Each version of ROUGE has their scores oriented to recall, precision and F1.

In the experiments, we report the different combinations of ROUGE scores for each dataset, which have been recommended and adopted by previous works. Specifically, the recall scores of R-1,2,4 will be reported for the DUC 2004 dataset according to Hong et al. (2014), Wang et al. (2017) and Zheng et al. (2019); the F1 scores of R-1,2,L will be reported for Yelp as in Chu and Liu (2019); the F1 scores of R-1,2,SU4 will be reported for Multi-News as in Fabbri et al. (2019). The toolkit for computing ROUGE metrics is ROUGE-1.5.8 and its option is set to be '-m -c 95 -r 1000 -f A -p 0.5 -t 0'.

Footnote 8: [https://github.com/andersjo/pyrouge/tree/master/tools/ROUGE-1.5.5](https://github.com/andersjo/pyrouge/tree/master/tools/ROUGE-1.5.5)

### Comparing Methods

We compare our method with both traditional and state-of-the-art MDS methods.

**Lead**: The documents in a cluster are randomly shuffled, and the first sentence of the document is added to the summary until the length limit is reached.

**LexRank Erkan and Radev (2004)**: It performs the sentence relevancy estimation by the random walk process on the sentence graph.

**CLASSY04 Conroy et al. (2004)**: It ranked first in the official evaluation of DUC 2004. As a supervised method, it uses a Hidden Markov Model to rank sentences and a QR decomposition to produce the summary.

**C-Attention Li et al. (2017)**: The cascaded attention based auto-encoder is proposed for estimating the relevancy of words and sentences.

**GRU-GCN Yasunaga et al. (2017)**: It is a supervised method that employs a Graph Convolutional Network on sentence graph. The sentence embedding obtained from a Recurrent Neural Network serves as the input node feature.

**ParaFuse**Nayeem et al. (2018)**: MDS is formulated as multi-sentence compression. As the state 

of-the-art on DUC 2004, however, it needs some extra resource and toolkit, such as paraphrase bank and keyword extractor.

**Best Review**Chu and Liu (2019): A simple baseline selecting the best document to be summary based on word overlap.

**Centroid**Rossiello et al. (2017): Word embeddings are exploited to boost the performance of centroid-based methods.

**MeanSum**Chu and Liu (2019): An end-to-end neural model is put forward to implement the abstractive summarization of business review documents. The summary is decoded from the mean of the representations of input reviews.

**PG**See et al. (2017): It introduces a pointer-generator (PG) network that motivates the summarizer to copy original words from input via pointing, while preserving the ability to generate new words.

**Hi-MAP**Fabbri et al. (2019): It proposes the integration of sentence-level MMR scores into the PG network in order to adapt the attention weights on a word-level. The MMR score is computed by the Maximal Marginal Relevance algorithm Carbonell and Goldstein (1998), which gives the goodness of the available sentence given already selected ones.

**Our method Spectral**: This is our spectral-based method specified in Alg. 2.1. According to different building strategies in SS2.3, it has three versions: Spectral-tfisf, Spectral-ESE and Spectral-BERT.

Notice that on DUC 2004, the supervised methods are trained on datasets of earlier DUC evaluations or CNN and Daily Mail datasets Hermann et al. (2015) according to their original papers. On Multi-News, they are both trained and tested on the dataset itself.

### Main Results

Table 3.2 demonstrates the ROUGE results of various methods on the DUC dataset. The method Parause is previously state-of-the-art on this dataset. From the table, our method Spectral-BERT outperforms ParaFuse by 1.2 percent in R-1 score and has a slightly lower R-2 and R-4 score (still ranking second). Notice that ParaFuse, as mentioned in SS3.4, is not exactly a self-contained system. Compared with CLASSY04 (winner of the official evaluation), Spectral-BERT has an enormous advantage (say 3.7% and 2.7% higher R-1 and R-2 score, respectively). Notice that all supervised methods have a relatively low performance, since they are trained on datasets different from DUC 2004. This observation is consistent with the results reported in Fabbri et al. (2019).

Table 3.3 and Table 3.4 show ROUGE results on Yelp and Multi-News, respectively. Our method Spectral-BERT has also beaten other unsupervised methods by a wide margin (say 1.3% higher R-L score than MeanSum and 2.1% higher R-SU4 score than C-Attention). Compared with the state-of-the-art supervised system on Multi-News (namely Hi-MAP), Spectral-BERT cannot rival its performance. However, Spectral-BERT has beaten the other supervised system (i.e. PG) according to R-2 and R-SU4 score.

We observe that a better matrix building strategy (stated in SS2.3) has led to a considerable improvement of our method on all three datasets. Specifically, the BERT encoder brings about one percent improvement in R-2 score as compared with the tf-if model. It proves that our method is flexible and can benefit from recent off-the-shelf pre-training techniques Devlin et al. (2019); Clark et al. (2019).

\begin{table}
\begin{tabular}{l l|c c} \hline \hline Method & R-1 & R-2 & R-L \\ \hline Lead & 0.268 & 0.038 & 0.144 \\ Centroid & 0.246 & 0.029 & 0.138 \\ Best review & 0.280 & 0.035 & 0.153 \\ MeanSum & 0.289 & 0.037 & 0.159 \\ \hline Spectral-tfisf & 0.283 & 0.036 & 0.147 \\ Spectral-ESE & 0.291 & 0.037 & 0.165 \\ Spectral-BERT & **0.302** & **0.045** & **0.172** \\ \hline \hline \end{tabular}
\end{table}
Table 3.3: ROUGE results of unsupervised methods on the Yelp dataset (our model is Spectral). Best scores are in bold, and second best are underlined.

 



### Linguistic Quality

To further assess the linguistic quality of different summaries, we employ Amazon Mechanical Turk9 workers to judge the performance of three summarizers on a random sample of Multi-News (200 document clusters). A worker is asked to rate each summary on a scale of 1 (poor) to 5 (excellent) according to three criteria: **relevancy** (does the summary cover all the key information of document cluster? ) and two criteria adopted by DUC 2005 evaluation (**non-redundancy** and **grammaticality**) [4]. Table 3.5 shows the results. We observe that our method Spectral has the highest relevancy and non-redundancy score. Abstractive method C-Attention has a relatively low score of grammaticality. Notice that the non-redundancy scores of all summarizers are generally low, which shows that humans are more sensitive to the redundancy existing in the summary.

Footnote 8: [https://www.mturk.com/](https://www.mturk.com/)

### Hypothesis Validation

We provide the empirical evidence of our proposed spectral-based hypothesis. For each cluster of documents on DUC 2004, we construct a sample set of 500 summary candidates \({\cal S}\), each of which contains 3 original sentences selected randomly from the documents. The Pearson correlation coefficient (denoted by \(r\)) of spectral impact and the candidate goodness as a summary, when applied to the sample set, is computed and the derived histogram is shown in Figure 3.1. Each correlation coefficient falls into their corresponding bins. In the figure, our method Spectral-BERT and the ROUGE metrics are utilized because: (1) Spectral-BERT has been reported with a better empirical performance in SS3.5; (2) The goodness of summary candidate in this scenario can be measured by the precision-oriented ROUGE scores, esp. R-1 and R-2, in that the word count of candidate \({\cal S}\) is varied in the sample. We note that there are no bins corresponding to negative correlation coefficients (\(r\) ranges from -1 to 1), and quite a few \(r\)'s have a large score beyond \(0.5\) (the widely accepted threshold of a large \(r\) recommended by Cohen (2013)). This demonstrates that the two variables have a positive linear correlation, which supports our hypothesis. Similar results can be obtained when \({\cal S}\) contains a different number of sentences.

## 4 Related Work

**Unsupervised MDS.** There are a bunch of works working on different hypotheses and models in this field. PageRank alike algorithms [6, 7, 12] utilize random walks with some redundancy avoiding measures. Regarding the document cluster as a manifold structure, [13, 4] use the manifold ranking process on data. There are also quite a few neural architecture based models for a hidden semantic representation of sentences, documents or subtopics, such as [11, 12, 13, 4].

**Spectral optimization.** Optimizing eigen-related

Figure 3.1: Pearson correlation coefficients \(r\) of spectral impact (SI) and ROUGE (R-1 or R-2) on DUC 2004. The sample size is the number of document clusters, i.e. 50. Best viewed in colors.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & Method & R-1 & R-2 & R-SU4 \\ \hline Supervised & PG & 0.419 & 0.129 & 0.165 \\ method & Hi-MAP & **0.435** & **0.149** & **0.174** \\ \hline \multirow{6}{*}{Unsupervised} & Lead & 0.394 & 0.118 & 0.145 \\  & LexRank & 0.383 & 0.127 & 0.132 \\ \cline{1-1}  & C-Attention & 0.386 & 0.125 & 0.146 \\ \cline{1-1} \cline{2-5} method & Spectral-tfisf & 0.397 & 0.121 & 0.144 \\ \cline{1-1}  & Spectral-ESE & 0.396 & 0.130 & 0.159 \\ \cline{1-1}  & Spectral-BERT & _0.409_ & _0.136_ & _0.167_ \\ \hline \hline \end{tabular}
\end{table}
Table 3.4: ROUGE results on the Multi-News dataset (our model is Spectral). The best scores are in bold, the second best are underlined, the best among unsupervised methods are in _italics_.



metrics often leads to a specific collective optimization problem, which is believed to be hard in nature unless \({\rm P}={\rm NP}\)Cook (1971). Some particular examples Van Mieghem et al. (2011); Chen et al. (2016) have been proved NP-hard. The typical solvers adopt the heuristics based on either perturbation theory Chen et al. (2016) or QR decomposition Li et al. (2015a); Chen et al. (2018).

## 5 Conclusion

We propose a novel hypothesis-driven method for unsupervised MDS, where the goodness of any summary candidate can be determined from a spectral perspective when dropping it from the document cluster. Various MDS tasks of different sizes and domains show a promising result of our method. Extending our method to an abstractive setting is meaningful future work.

## Acknowledgments

We would like to thank all the reviewers for their helpful advice on various aspects of this work. This work is supported by National Natural Science Foundation of China (No. 61936012 and No. U19A2065) and Beijing Academy of Artificial Intelligence (BAAI).

## References

* Al-Dhelaan (2015) Mohammed Al-Dhelaan. 2015. Starsum: A simple star graph for multi-document summarization. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 715-718.
* Bellaachia and Al-Dhelaan (2014) Abdelghani Bellaachia and Mohammed Al-Dhelaan. 2014. Multi-document hyperedge-based ranking for text summarization. In _Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management_, pages 1919-1922.
* Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_, pages 335-336.
* Chen et al. (2018) Chen Chen, Ruiyue Peng, Lei Ying, and Hanghang Tong. 2018. Network connectivity optimization: Fundamental limits and effective algorithms. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1167-1176.
* Chen et al. (2016) Chen Chen, Hanghang Tong, B Aditya Prakash, Tina Eliassi-Rad, Michalis Faloutsos, and Christos Faloutsos. 2016. Eigen-optimization on large graphs by edge manipulation. _ACM Transactions on Knowledge Discovery from Data (TKDD)_, 10(4):49.
* Chen (2018) Zesheng Chen. 2018. Epidemic thresholds in networks: Impact of heterogeneous infection rates and recovery rates. In _2018 IEEE International Conference on Communications (ICC)_, pages 1-6. IEEE.
* Cheng et al. (2011) Xue-Qi Cheng, Pan Du, Jiafeng Guo, Xiaofei Zhu, and Yixin Chen. 2011. Ranking on data manifold with sink points. _IEEE Transactions on Knowledge and Data Engineering_, 25(1):177-191.
* Chu and Liu (2019) Eric Chu and Peter Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. In _International Conference on Machine Learning_, pages 1223-1232.
* Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In _International Conference on Learning Representations_.
* Cohen (2013) Jacob Cohen. 2013. _Statistical power analysis for the behavioral sciences_. Academic press.
* Conroy et al. (2004) John M Conroy, Judith D Schlesinger, Jade Goldstein, and Dianne P O'leary. 2004. Left-brain/right-brain multi-document summarization. In _Proceedings of the Document Understanding Conference (DUC 2004)_.
* Cook (1971) Stephen A Cook. 1971. The complexity of theorem-proving procedures. In _Proceedings of the third annual ACM symposium on Theory of computing_, pages 151-158. ACM.
* Dang (2005) Hoa Trang Dang. 2005. Overview of duc 2005. In _Proceedings of the document understanding conference_, volume 2005, pages 1-12.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Erkan and Radev (2004) Gunes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. _Journal of artificial intelligence research_, 22:457-479.
* Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1074-1084, Florence, Italy. Association for Computational Linguistics 

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in neural information processing systems_, pages 1693-1701.
* Hong et al. (2014) Kai Hong, John Conroy, Benoit Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_, pages 1608-1616, Reykjavik, Iceland. European Language Resources Association (ELRA).
* Jones (2007) James Holland Jones. 2007. Notes on r0. _Stanford University_.
* Lanczos (1950) Cornelius Lanczos. 1950. _An iteration method for the solution of the eigenvalue problem of linear differential and integral operators_. United States Governm. Press Office Los Angeles, CA.
* Lebanoff et al. (2018) Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4131-4141, Brussels, Belgium. Association for Computational Linguistics.
* Li et al. (2012) Cong Li, Huijuan Wang, and Piet Van Mieghem. 2012. Bounds for the spectral radius of a graph when nodes are removed. _Linear Algebra and its Applications_, 437(1):319-323.
* Li et al. (2011) Huiying Li, Yue Hu, Zeyuan Li, Xiaojun Wan, and Jianguo Xiao. 2011. Pkutm participation in tac2011. _Proceeding RTE_, 7.
* Li et al. (2015a) Liangyue Li, Hanghang Tong, Yanghua Xiao, and Wei Fan. 2015a. Cheetah: fast graph kernel tracking on dynamic graphs. In _Proceedings of the 2015 SIAM International Conference on Data Mining_, pages 280-288. SIAM.
* Li et al. (2015b) Piji Li, Lidong Bing, Wai Lam, Hang Li, and Yi Liao. 2015b. Reader-aware multi-document summarization via sparse coding. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
* Li et al. (2017a) Piji Li, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. 2017a. Cascaded attention based unsupervised information distillation for compressive summarization. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2081-2090, Copenhagen, Denmark. Association for Computational Linguistics.
* Li et al. (2017b) Piji Li, Zihao Wang, Wai Lam, Zhaochun Ren, and Lidong Bing. 2017b. Salience estimation via variational auto-encoders for multi-document summarization. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lin and Bilmes (2011) Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 510-520, Portland, Oregon, USA. Association for Computational Linguistics.
* Liu et al. (2015) He Liu, Hongliang Yu, and Zhi-Hong Deng. 2015. Multi-document summarization based on two-level sparse representation model. In _Twenty-ninth AAAI conference on artificial intelligence_.
* Liu et al. (2020) Ying Liu, Albert A Gayle, Annelies Wilder-Smith, and Joacim Rocklov. 2020. The reproductive number of covid-19 is higher compared to sars coronavirus. _Journal of travel medicine_.
* Ma et al. (2016) Shulei Ma, Zhi-Hong Deng, and Yunlu Yang. 2016. An unsupervised multi-document summarization framework based on neural document model. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 1514-1523, Osaka, Japan. The COLING 2016 Organizing Committee.
* Mei et al. (2010) Qiaozhu Mei, Jian Guo, and Dragomir Radev. 2010. Divrank: the interplay of prestige and diversity in information networks. In _Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1009-1018. Acm.
* Najceum et al. (2018) Mir Tafsser Najceum, Tanvir Ahmed Fuad, and Yllias Chali. 2018. Abstractive unsupervised multi-document summarization using paraphrastic sentence fusion. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1191-1204, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Owczarzak et al. (2012)Karolina Owczarzak, John M. Conroy, Hoa Tang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In _Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization_, pages 1-9, Montreal, Canada. Association for Computational Linguistics.
* Prakash et al. (2012) B Aditya Prakash, Deepayan Chakrabarti, Nicholas C Valler, Michalis Faloutsos, and Christos Faloutsos. 2012. Threshold conditions for arbitrary cascade models on arbitrary networks. _Knowledge and information systems_, 33(3):549-575.
* Rossiello et al. (2017) Gaetano Rossiello, Pierpaolo Basile, and Giovanni Semeraro. 2017. Centroid-based text summarization through compositionality of word embeddings. In _Proceedings of the MultiLine 2017 Workshop on Summarization and Summary Evaluation Across 

Source Types and Genres_, pages 12-21, Valencia, Spain. Association for Computational Linguistics.
* See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Stevanovic (2014) Dragan Stevanovic. 2014. _Spectral radius of graphs_. Academic Press.
* Stewart (1990) G. W. Stewart. 1990. Matrix perturbation theory.
* Van Mieghem et al. (2011) Piet Van Mieghem, Dragan Stevanovic, Fernando Kuipers, Cong Li, Ruud Van De Bovenkamp, Daijie Liu, and Huijuan Wang. 2011. Decreasing the spectral radius of a graph by link removals. _Physical Review E_, 84(1):016101.
* Wan et al. (2007) Xiaojun Wan, Jianwu Yang, and Jianguo Xiao. 2007. Manifold-ranking based topic-focused multidocument summarization. In _IJCAI_, volume 7, pages 2903-2908.
* Wang et al. (2017) Kexiang Wang, Tianyu Liu, Zhifang Sui, and Baobao Chang. 2017. Affinity-preserving random walk for multi-document summarization. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 210-220, Copenhagen, Denmark. Association for Computational Linguistics.
* Wang et al. (2003) Yang Wang, Deepayan Chakrabarti, Chenxi Wang, and Christos Faloutsos. 2003. Epidemic spreading in real networks: An eigenvalue viewpoint. In _22nd International Symposium on Reliable Distributed Systems, 2003. Proceedings._, pages 25-34. IEEE.
* Welsh (1976) DJA Welsh. 1976. Matroid theory. 1976.
* Yang et al. (2018) Kang Yang, Kamal Al-Sabahi, Yanmin Xiang, and Zuping Zhang. 2018. An integrated graph model for document summarization. _Information_, 9(9):232.
* Yang et al. (2019) Kang Yang, Hongye He, Kamal Al-Sabahi, and Zipping Zhang. 2019. Ecforest: Extractive document summarization through enhanced sentence embedding and cascade forest. _Concurrency and Computation: Practice and Experience_, 31(17):e5206.
* Yao et al. (2015) Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2015. Compressive document summarization via sparse optimization. In _Twenty-Fourth International Joint Conference on Artificial Intelligence_.
* Yasunaga et al. (2017) Michihiro Yasunaga, Rui Zhang, Khalitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 452-462, Vancouver, Canada. Association for Computational Linguistics.
* Zheng et al. (2019) Xin Zheng, Aixin Sun, Jing Li, and Karthik Muthuswamy. 2019. Subtopic-driven multi-document summarization. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3153-3162, Hong Kong, China. Association for Computational Linguistics.