Question: How many summarization approaches did they appliy to the TAC 2014 scientific summarization dataset to evaluate the effectiveness of ROUGE.?
Evidence: We study the effectiveness of Rouge and our proposed method (Sera) by analyzing the correlations with semi-manual human judgments. Very few teams participated in TAC 2014 summarization track and the official results and the review paper of TAC 2014 systems were never published. Therefore, to evaluate the effectiveness of Rouge, we applied 9 well-known summarization approaches on the TAC 2014 scientific summarization dataset. Obtained Rouge and Sera results of each of these approaches are then correlated with semi-manual human judgments. In the following, we briefly describe each of these summarization approaches.
LexRank BIBREF9 : LexRank finds the most important (central) sentences in a document by using random walks in a graph constructed from the document sentences. In this graph, the sentences are nodes and the similarity between the sentences determines the edges. Sentences are ranked according to their importance. Importance is measured in terms of centrality of the sentence — the total number of edges incident on the node (sentence) in the graph. The intuition behind LexRank is that a document can be summarized using the most central sentences in the document that capture its main aspects.
Latent Semantic Analysis (LSA) based summarization BIBREF10 : In this summarization method, Singular Value Decomposition (SVD) BIBREF11 is used for deriving latent semantic structure of the document. The document is divided into sentences and a term-sentence matrix INLINEFORM0 is constructed. The matrix INLINEFORM1 is then decomposed into a number of linearly-independent singular vectors which represent the latent concepts in the document. This method, intuitively, decomposes the document into several latent topics and then selects the most representative sentences for each of these topics as the summary of the document.
Maximal Marginal Relevance (MMR) BIBREF12 : Maximal Marginal Relevance (MMR) is a greedy strategy for selecting sentences for the summary. Sentences are added iteratively to the summary based on their relatedness to the document as well as their novelty with respect to the current summary.
Citation based summarization BIBREF13 : In this method, citations are used for summarizing an article. Using the LexRank algorithm on the citation network of the article, top sentences are selected for the final summary.
Using frequency of the words BIBREF14 : In this method, which is one the earliest works in text summarization, raw word frequencies are used to estimate the saliency of sentences in the document. The most salient sentences are chosen for the final summary.
SumBasic BIBREF15 : SumBasic is an approach that weights sentences based on the distribution of words that is derived from the document. Sentence selection is applied iteratively by selecting words with highest probability and then finding the highest scoring sentence that contains that word. The word weights are updated after each iteration to prevent selection of similar sentences.
Summarization using citation-context and discourse structure BIBREF16 : In this method, the set of citations to the article are used to find the article sentences that directly reflect those citations (citation-contexts). In addition, the scientific discourse of the article is utilized to capture different aspects of the article. The scientific discourse usually follows a structure in which the authors first describe their hypothesis, then the methods, experiment, results and implications. Sentence selection is based on finding the most important sentences in each of the discourse facets of the document using the MMR heuristic.
KL Divergence BIBREF17 In this method, the document unigram distribution INLINEFORM0 and the summary unigram distributation INLINEFORM1 are considered; the goal is to find a summary whose distribution is very close to the document distribution. The difference of the distributions is captured by the Kullback-Lieber (KL) divergence, denoted by INLINEFORM2 .
Summarization based on Topic Models BIBREF17 : Instead of using unigram distributions for modeling the content distribution of the document and the summary, this method models the document content using an LDA based topic model BIBREF18 . It then uses the KL divergence between the document and the summary content models for selecting sentences for the summary.
We study the effectiveness of ROUGE and our proposed method (SERA) by analyzing the correlations with semi-manual human judgments. Very few teams participated in TAC 2014 summarization track and the official results and the review paper of TAC 2014 systems were never published. Therefore, to evaluate the effectiveness of ROUGE, we applied 9 well-known summarization approaches on the TAC 2014 scientific summarization dataset.
Answer: 9 well-known summarization approaches.