Introduction
Today's increasing flood of information on the web creates a need for automated multi-document summarization systems that produce high quality summaries. However, producing summaries in a multi-document setting is difficult, as the language used to display the same information in a sentence can vary significantly, making it difficult for summarization models to capture. Given the complexity of the task and the lack of datasets, most researchers use extractive summarization, where the final summary is composed of existing sentences in the input documents. More specifically, extractive summarization systems output summaries in two steps : via sentence ranking, where an importance score is assigned to each sentence, and via the subsequent sentence selection, where the most appropriate sentence is chosen, by considering 1) their importance and 2) their frequency among all documents. Due to data sparcity, models heavily rely on well-designed features at the word level BIBREF0, BIBREF1, BIBREF2, BIBREF3 or take advantage of other large, manually annotated datasets and then apply transfer learning BIBREF4. Additionally, most of the time, all sentences in the same collection of documents are processed independently and therefore, their relationships are lost.
In realistic scenarios, features are hard to craft, gathering additional annotated data is costly, and the large variety in expressing the same fact cannot be handled by the use of word-based features only, as is often the case. In this paper, we address these obstacles by proposing to simultaneously leverage two types of sentence embeddings, namely embeddings pre-trained on a large corpus that capture a variety of meanings and domain-specific embeddings learned during training. The former is typically trained on an unrelated corpus composed of high quality texts, allowing to cover additional contexts for each encountered word and sentence. Hereby, we build on the assumption that sentence embeddings capture both the syntactic and semantic content of sentences. We hypothesize that using two types of sentence embeddings, general and domain-specific, is beneficial for the task of multi-document summarization, as the former captures the most common semantic structures from a large, general corpus, while the latter captures the aspects related to the domain.
We present SemSentSum (Figure FIGREF3), a fully data-driven summarization system, which does not depend on hand-crafted features, nor additional data, and is thus domain-independent. It first makes use of general sentence embedding knowledge to build a sentenc semantic relation graph that captures sentence similarities (Section SECREF4). In a second step, it trains genre-specific sentence embeddings related to the domains of the collection of documents, by utilizing a sentence encoder (Section SECREF5). Both representations are afterwards merged, by using a graph convolutional network BIBREF5 (Section SECREF6). Then, it employs a linear layer to project high-level hidden features for individual sentences to salience scores (Section SECREF8). Finally, it greedily produces relevant and non-redundant summaries by using sentence embeddings to detect similarities between candidate sentences and the current summary (Section SECREF11).
The main contributions of this work are as follows :
We aggregate two types of sentences embeddings using a graph representation. They share different properties and are consequently complementary. The first one is trained on a large unrelated corpus to model general semantics among sentences, whereas the second is domain-specific to the dataset and learned during training. Together, they enable a model to be domain-independent as it can be applied easily on other domains. Moreover, it could be used for other tasks including detecting information cascades, query-focused summarization, keyphrase extraction and information retrieval.
We devise a competitive multi-document summarization system, which does not need hand-crafted features nor additional annotated data. Moreover, the results are competitive for 665-byte and 100-word summaries. Usually, models are compared in one of the two settings but not both and thus lack comparability.
Method
Let $C$ denote a collection of related documents composed of a set of documents $\lbrace D_i|i \in [1,N]\rbrace $ where $N$ is the number of documents. Moreover, each document $D_i$ consists of a set of sentences $\lbrace S_{i,j}|j \in [1,M]\rbrace $, $M$ being the number of sentences in $D_i$. Given a collection of related documents $C$, our goal is to produce a summary $Sum$ using a subset of these in the input documents ordered in some way, such that $Sum = (S_{i_1,j_1},S_{i_2,j_2},...,S_{i_n,j_m})$.
In this section, we describe how SemSentSum estimates the salience score of each sentence and how it selects a subset of these to create the final summary. The architecture of SemSentSum is depicted in Figure FIGREF3.
In order to perform sentence selection, we first build our sentence semantic relation graph, where each vertex is a sentence and edges capture the semantic similarity among them. At the same time, each sentence is fed into a recurrent neural network, as a sentence encoder, to generate sentence embeddings using the last hidden states. A single-layer graph convolutional neural network is then applied on top, where the sentence semantic relation graph is the adjacency matrix and the sentence embeddings are the node features. Afterward, a linear layer is used to project high-level hidden features for individual sentences to salience scores, representing how salient a sentence is with respect to the final summary. Finally, based on this, we devise an innovative greedy method that leverages sentence embeddings to detect redundant sentences and select sentences until reaching the summary length limit.
Method ::: Sentence Semantic Relation Graph
We model the semantic relationship among sentences using a graph representation. In this graph, each vertex is a sentence $S_{i,j}$ ($j$'th sentence of document $D_i$) from the collection documents $C$ and an undirected edge between $S_{i_u,j_u}$ and $S_{i_v,j_v}$ indicates their degree of similarity. In order to compute the semantic similarity, we use the model of BIBREF6 trained on the English Wikipedia corpus. In this manner, we incorporate general knowledge (i.e. not domain-specific) that will complete the specialized sentence embeddings obtained during training (see Section SECREF5). We process sentences by their model and compute the cosine similarity between every sentence pair, resulting in a complete graph. However, having a complete graph alone does not allow the model to leverage the semantic structure across sentences significantly, as every sentence pair is connected, and likewise, a sparse graph does not contain enough information to exploit semantic similarities. Furthermore, all edges have a weight above zero, since it is very unlikely that two sentence embeddings are completely orthogonal. To overcome this problem, we introduce an edge-removal-method, where every edge below a certain threshold $t_{sim}^g$ is removed in order to emphasize high sentence similarity. Nonetheless, $t_{sim}^g$ should not be too large, as we otherwise found the model to be prone to overfitting. After removing edges below $t_{sim}^g$, our sentence semantic relation graph is used as the adjacency matrix $A$. The impact of $t_{sim}^g$ with different values is shown in Section SECREF26.
Based on our aforementioned hypothesis that a combination of general and genre-specific sentence embeddings is beneficial for the task of multi-document summarization, we further incorporate general sentence embeddings, pre-trained on Wikipedia entries, into edges between sentences. Additionally, we compute specialised sentence embeddings, which are related to the domains of the documents (see Section SECREF35).
Note that 1) the pre-trained sentence embeddings are only used to compute the weights of the edges and are not used by the summarization model (as others are produced by the sentence encoder) and 2) the edge weights are static and do not change during training.
Method ::: Sentence Encoder
Given a list of documents $C$, we encode each document's sentence $S_{i,j}$, where each has at most $L$ words $(w_{i,j,1}, w_{i,j,2}, ..., w_{i,j,L})$. In our experiments, all words are kept and converted into word embeddings, which are then fed to the sentence encoder in order to compute specialized sentence embeddings $S^{\prime }_{i,j}$. We employ a single-layer forward recurrent neural network, using Long Short-Term Memory (LSTM) of BIBREF7 as sentence encoder, where the sentence embeddings are extracted from the last hidden states. We then concatenate all sentence embeddings into a matrix $X$ which constitutes the input node features that will be used by the graph convolutional network.
Method ::: Graph Convolutional Network
After having computed all sentence embeddings and the sentence semantic relation graph, we apply a single-layer Graph Convolutional Network (GCN) from BIBREF5, in order to capture high-level hidden features for each sentence, encapsulating sentence information as well as the graph structure.
We believe that our sentence semantic relation graph contains information not present in the data (via universal embeddings) and thus, we leverage this information by running a graph convolution on the first order neighborhood.
The GCN model takes as input the node features matrix $X$ and a squared adjacency matrix $A$. The former contains all sentence embeddings of the collection of documents, while the latter is our underlying sentence semantic relation graph. It outputs hidden representations for each node that encode both local graph structure and nodes's features. In order to take into account the sentences themselves during the information propagation, we add self-connections (i.e. the identity matrix) to $A$ such that $\tilde{A} = A + I$.
Subsequently, we obtain our sentence hidden features by using Equation DISPLAY_FORM7.
where $W_i$ is the weight matrix of the $i$'th graph convolution layer and $b_i$ the bias vector. We choose the Exponential Linear Unit (ELU) activation function from BIBREF8 due to its ability to handle the vanishing gradient problem, by pushing the mean unit activations close to zero and consequently facilitating the backpropagation. By using only one hidden layer, as we only have one input-to-hidden layer and one hidden-to-output layer, we limit the information propagation to the first order neighborhood.
Method ::: Saliency Estimation
We use a simple linear layer to estimate a salience score for each sentence and then normalize the scores via softmax and obtain our estimated salience score $S^s_{i,j}$.
Method ::: Training
Our model SemSentSum is trained in an end-to-end manner and minimizes the cross-entropy loss of Equation DISPLAY_FORM10 between the salience score prediction and the ROUGE-1 $F_1$ score for each sentence.
$F_1(S)$ is computed as the ROUGE-1 $F_1$ score, unlike the common practice in the area of single and multi-document summarization as recall favors longer sentences whereas $F_1$ prevents this tendency. The scores are normalized via softmax.
Method ::: Summary Generation Process
While our model SemSentSum provides estimated saliency scores, we use a greedy strategy to construct an informative and non-redundant summary $Sum$. We first discard sentences having less than 9 words, as in BIBREF9, and then sort them in descending order of their estimated salience scores. We iteratively dequeue the sentence having the highest score and append it to the current summary $Sum$ if it is non-redundant with respect to the current content of $Sum$. We iterate until reaching the summary length limit.
To determine the similarity of a candidate sentence with the current summary, a sentence is considered as dissimilar if and only if the cosine similarity between its sentence embeddings and the embeddings of the current summary is below a certain threshold $t_{sim}^s$. We use the pre-trained model of BIBREF6 to compute sentence as well as summary embeddings, similarly to the sentence semantic relation graph construction. Our approach is novel, since it focuses on the semantic sentence structures and captures similarity between sentence meanings, instead of focusing on word similarities only, like previous TF-IDF approaches ( BIBREF0, BIBREF1, BIBREF3, BIBREF4).
Experiments ::: Datasets
We conduct experiments on the most commonly used datasets for multi-document summarization from the Document Understanding Conferences (DUC). We use DUC 2001, 2002, 2003 and 2004 as the tasks of generic multi-document summarization, because they have been carried out during these years. We use DUC 2001, 2002, 2003 and 2004 for generic multi-document summarization, where DUC 2001/2002 are used for training, DUC 2003 for validation and finally, DUC 2004 for testing, following the common practice.
Experiments ::: Evaluation Metric
For the evaluation, we use ROUGE BIBREF10 with the official parameters of the DUC tasks and also truncate the summaries to 100 words for DUC 2001/2002/2003 and to 665 bytes for DUC 2004. Notably, we take ROUGE-1 and ROUGE-2 recall scores as the main metrics for comparison between produced summaries and golden ones as proposed by BIBREF11. The goal of the ROUGE-N metric is to compute the ratio of the number of N-grams from the generated summary matching these of the human reference summaries.
Experiments ::: Model Settings
To define the edge weights of our sentence semantic relation graph, we employ the 600-dimensional pre-trained unigram model of BIBREF6, using English Wikipedia as source corpus. We keep only edges having a weight larger than $t_{sim}^g = 0.5$ (tuned on the validation set). For word embeddings, the 300-dimensional pre-trained GloVe embeddings BIBREF12 are used and fixed during training. The output dimension of the sentence embeddings produced by the sentence encoder is the same as that of the word embeddings, i.e. 300. For the graph convolutional network, the number of hidden units is 128 and the size of the generated hidden feature vectors is also 300. We use a batch size of 1, a learning rate of $0.0075$ using Adam optimizer BIBREF13 with $\beta _1=0.9, \beta _2=0.999$ and $\epsilon =10^{-8}$. In order to make SemSentSum generalize better, we use dropout BIBREF14 of $0.2$, batch normalization BIBREF15, clip the gradient norm at $1.0$ if higher, add L2-norm regularizer with a regularization factor of $10^{-12}$ and train using early stopping with a patience of 10 iterations. Finally, the similarity threshold $t_{sim}^s$ in the summary generation process is $0.8$ (tuned on the validation set).
Experiments ::: Summarization Performance
We train our model SemSentSum on DUC 2001/2002, tune it on DUC 2003 and assess the performance on DUC 2004. In order to fairly compare SemSentSum with other models available in the literature, experiments are conducted with summaries truncated to 665 bytes (official summary length in the DUC competition), but also with summaries with a length constraint of 100 words. To the best of our knowledge, we are the first to conduct experiments on both summary lengths and compare our model with other systems producing either 100 words or 665 bytes summaries.
Experiments ::: Sentence Semantic Relation Graph Construction
We investigate different methods to build our sentence semantic relation graph and vary the value of $t_{sim}^g$ from $0.0$ to $0.75$ to study the impact of the threshold cut-off. Among these are :
Cosine : Using cosine similarity ;
Tf-idf : Considering a node as the query and another as document. The weight corresponds to the cosine similarity between the query and the document ;
TextRank BIBREF16 : A weighted graph is created where nodes are sentences and edges defined by a similarity measure based on word overlap. Afterward, an algorithm similar to PageRank BIBREF17 is used to compute sentence importance and refined edge weights ;
LexRank BIBREF9 : An unsupervised multi-document summarizer based on the concept of eigenvector centrality in a graph of sentences to set up the edge weights ;
Approximate Discourse Graph (ADG) BIBREF2 : Approximation of a discourse graph where nodes are sentences and edges $(S_u,S_v)$ indicates sentence $S_v$ can be placed after $S_u$ in a coherent summary ;
Personalized ADG (PADG) BIBREF3 : Normalized version of ADG where sentence nodes are normalized over all edges.
Experiments ::: Ablation Study
In order to quantify the contribution of the different components of SemSentSum, we try variations of our model by removing different modules one at a time. Our two main elements are the sentence encoder (Sent) and the graph convolutional neural network (GCN). When we omit Sent, we substitute it with the pre-trained sentence embeddings used to build our sentence semantic relation graph.
Experiments ::: Results and Discussion
Three dimensions are used to evaluate our model SemSentSum : 1) the summarization performance, to assess its capability 2) the impact of the sentence semantic relation graph generation using various methods and different thresholds $t_{sim}^g$ 3) an ablation study to analyze the importance of each component of SemSentSum.
Experiments ::: Results and Discussion ::: Summarization Performance
We compare the results of SemSentSum for both settings : 665 bytes and 100 words summaries. We only include models using the same parameters to compute the ROUGE-1/ROUGE-2 score and recall as metrics.
The results for 665 bytes summaries are reported in Table TABREF28. We compare SemSentSum with three types of model relying on either 1) sentence or document embeddings 2) various hand-crafted features or 3) additional data.
For the first category, we significantly outperform MMR BIBREF18, PV-DBOW+BS BIBREF19 and PG-MMR BIBREF20. Although their methods are based on embeddings to represent the meaning, it shows that using only various distance metrics or encoder-decoder architecture on these is not efficient for the task of multi-document summarization (as also shown in the Ablation Study). We hypothesize that SemSentSum performs better by leveraging pre-trained sentence embeddings and hence lowering the effects of data scarcity.
Systems based on hand-crafted features include a widely-used learning-based summarization method, built on support vector regression SVR BIBREF21 ; a graph-based method based on approximating discourse graph G-Flow BIBREF2 ; Peer 65 which is the best peer systems participating in DUC evaluations ; and the recursive neural network R2N2 of BIBREF1 that learns automatically combinations of hand-crafted features. As can be seen, among these models completely dependent on hand-crafted features, SemSentSum achieves highest performance on both ROUGE scores. This denotes that using different linguistic and word-based features might not be enough to capture the semantic structures, in addition to being cumbersome to craft.
The last type of model is shown in TCSum BIBREF4 and uses transfer learning from a text classifier model, based on a domain-related dataset of $30\,000$ documents from New York Times (sharing the same topics of the DUC datasets). In terms of ROUGE-1, SemSentSum significantly outperforms TCSum and performs similarly on ROUGE-2 score. This demonstrates that collecting more manually annotated data and training two models is unnecessary, in addition to being difficult to use in other domains, whereas SemSentSum is fully data driven, domain-independent and usable in realistic scenarios.
Table TABREF32 depicts models producing 100 words summaries, all depending on hand-crafted features. We use as baselines FreqSum BIBREF22 ; TsSum BIBREF23 ; traditional graph-based approaches such as Cont. LexRank BIBREF9 ; Centroid BIBREF24 ; CLASSY04 BIBREF25 ; its improved version CLASSY11 BIBREF26 and the greedy model GreedyKL BIBREF27. All of these models are significantly underperforming compared to SemSentSum. In addition, we include state-of-the-art models : RegSum BIBREF0 and GCN+PADG BIBREF3. We outperform both in terms of ROUGE-1. For ROUGE-2 scores we achieve better results than GCN+PADG but without any use of domain-specific hand-crafted features and a much smaller and simpler model. Finally, RegSum achieves a similar ROUGE-2 score but computes sentence saliences based on word scores, incorporating a rich set of word-level and domain-specific features. Nonetheless, our model is competitive and does not depend on hand-crafted features due to its full data-driven nature and thus, it is not limited to a single domain.
Consequently, the experiments show that achieving good performance for multi-document summarization without hand-crafted features or additional data is clearly feasible and SemSentSum produces competitive results without depending on these, is domain independent, fast to train and thus usable in real scenarios.
Experiments ::: Results and Discussion ::: Sentence Semantic Relation Graph
Table TABREF34 shows the results of different methods to create the sentence semantic relation graph with various thresholds $t_{sim}^g$ for 665 bytes summaries (we obtain similar results for 100 words). A first observation is that using cosine similarity with sentence embeddings significantly outperforms all other methods for ROUGE-1 and ROUGE-2 scores, mainly because it relies on the semantic of sentences instead of their individual words. A second is that different methods evolve similarly : PADG, Textrank, Tf-idf behave similarly to an U-shaped curve for both ROUGE scores while Cosine is the only one having an inverted U-shaped curve. The reason for this behavior is a consequence of its distribution being similar to a normal distribution because it relies on the semantic instead of words, while the others are more skewed towards zero. This confirms our hypothesis that 1) having a complete graph does not allow the model to leverage much the semantic 2) a sparse graph might not contain enough information to exploit similarities. Finally, Lexrank and ADG have different trends between both ROUGE scores.
Experiments ::: Results and Discussion ::: Ablation Study
We quantify the contribution of each module of SemSentSum in Table TABREF36 for 665 bytes summaries (we obtain similar results for 100 words). Removing the sentence encoder produces slightly lower results. This shows that the sentence semantic relation graph captures semantic attributes well, while the fine-tuned sentence embeddings obtained via the encoder help boost the performance, making these methods complementary. By disabling only the graph convolutional layer, a drastic drop in terms of performance is observed, which emphasizes that the relationship among sentences is indeed important and not present in the data itself. Therefore, our sentence semantic relation graph is able to capture sentence similarities by analyzing the semantic structures. Interestingly, if we remove the sentence encoder in addition to the graph convolutional layer, similar results are achieved. This confirms that alone, the sentence encoder is not able to compute an efficient representation of sentences for the task of multi-document summarization, probably due to the poor size of the DUC datasets. Finally, we can observe that the use of sentence embeddings only results in similar performance to the baselines, which rely on sentence or document embeddings BIBREF18, BIBREF19.
Related Work
The idea of using multiple embeddings has been employed at the word level. BIBREF28 use an attention mechanism to combine the embeddings for each word for the task of natural language inference. BIBREF29, BIBREF30 concatenate the embeddings of each word into a vector before feeding a neural network for the tasks of aspect extraction and sentiment analysis. To our knowledge, we are the first to combine multiple types of sentence embeddings.
Extractive multi-document summarization has been addressed by a large range of approaches. Several of them employ graph-based methods. BIBREF31 introduced a cross-document structure theory, as a basis for multi-document summarization. BIBREF9 proposed LexRank, an unsupervised multi-document summarizer based on the concept of eigenvector centrality in a graph of sentences. Other works exploit shallow or deep features from the graph's topology BIBREF32, BIBREF33. BIBREF34 pairs graph-based methods (e.g. random walk) with clustering. BIBREF35 improved results by using a reinforced random walk model to rank sentences and keep non-redundant ones. The system by BIBREF2 does sentence selection, while balancing coherence and salience and by building a graph that approximates discourse relations across sentences BIBREF36.
Besides graph-based methods, other viable approaches include Maximum Marginal Relevance BIBREF37, which uses a greedy approach to select sentences and considers the tradeoff between relevance and redundancy ; support vector regression BIBREF21 ; conditional random field BIBREF38 ; or hidden markov model BIBREF25. Yet other approaches rely on n-grams regression as in BIBREF39. More recently, BIBREF1 built a recursive neural network, which tries to automatically detect combination of hand-crafted features. BIBREF4 employ a neural model for text classification on a large manually annotated dataset and apply transfer learning for multi-document summarization afterward.
The work most closely related to ours is BIBREF3. They create a normalized version of the approximate discourse graph BIBREF2, based on hand-crafted features, where sentence nodes are normalized over all the incoming edges. They then employ a deep neural network, composed of a sentence encoder, three graph convolutional layers, one document encoder and an attention mechanism. Afterward, they greedily select sentences using TF-IDF similarity to detect redundant sentences. Our model differs in four ways : 1) we build our sentence semantic relation graph by using pre-trained sentence embeddings with cosine similarity, where neither heavy preprocessing, nor hand-crafted features are necessary. Thus, our model is fully data-driven and domain-independent unlike other systems. In addition, the sentence semantic relation graph could be used for other tasks than multi-document summarization, such as detecting information cascades, query-focused summarization, keyphrase extraction or information retrieval, as it is not composed of hand-crafted features. 2) SemSentSum is much smaller and consequently has fewer parameters as it only uses a sentence encoder and a single convolutional layer. 3) The loss function is based on ROUGE-1 $F_1$ score instead of recall to prevent the tendency of choosing longer sentences. 4) Our method for summary generation is also different and novel as we leverage sentence embeddings to compute the similarity between a candidate sentence and the current summary instead of TF-IDF based approaches.
Conclusion
In this work, we propose a method to combine two types of sentence embeddings : 1) universal embeddings, pre-trained on a large corpus such as Wikipedia and incorporating general semantic structures across sentences and 2) domain-specific embeddings, learned during training. We merge them together by using a graph convolutional network that eliminates the need of hand-crafted features or additional annotated data.
We introduce a fully data-driven model SemSentSum that achieves competitive results for multi-document summarization on both kind of summary length (665 bytes and 100 words summaries), without requiring hand-crafted features or additional annotated data.
As SemSentSum is domain-independent, we believe that our sentence semantic relation graph and model can be used for other tasks including detecting information cascades, query-focused summarization, keyphrase extraction and information retrieval. In addition, we plan to leave the weights of the sentence semantic relation graph dynamic during training, and to integrate an attention mechanism directly into the graph.
Acknowledgments
We thank Michaela Benk for proofreading and helpful advice.

Question: How big is dataset domain-specific embedding are trained on?
Answer: Unanswerable

Question: How big is unrelated corpus universal embedding is traned on?
Answer: Unanswerable

Question: How better are state-of-the-art results than this model? 
Answer: we achieve better results than GCN+PADG but without any use of domain-specific hand-crafted features,  RegSum achieves a similar ROUGE-2 score