Question: Are there any variants of ROUGE?
Evidence: Rouge has been the most widely used family of metrics in summarization evaluation. In the following, we briefly describe the different variants of Rouge:
Rouge-L, Rouge-W, Rouge-S and Rouge-SU were later extended to consider both the recall and precision. In calculating Rouge, stopword removal or stemming can also be considered, resulting in more variants.
In the summarization literature, despite the large number of variants of Rouge, only one or very few of these variants are often chosen (arbitrarily) for evaluation of the quality of the summarization approaches. When Rouge was proposed, the original variants were only recall-oriented and hence the reported correlation results BIBREF1 . The later extension of Rouge family by precision were only reflected in the later versions of the Rouge toolkit and additional evaluation of its effectiveness was not reported. Nevertheless, later published work in summarization adopted this toolkit for its ready implementation and relatively efficient performance.
The original Rouge metrics show high correlations with human judgments of the quality of summaries on the DUC 2001-2003 benchmarks. However, these benchmarks consist of newswire data and are intrinsically very different than other summarization tasks such as summarization of scientific papers. We argue that Rouge is not the best metric for all summarization tasks and we propose an alternative metric for evaluation of scientific summarization. The proposed alternative metric shows much higher and more consistent correlations with manual judgments in comparison with the well-established Rouge.
ROUGE has been the most widely used family of metrics in summarization evaluation. In the following, we briefly describe the different variants of ROUGE: – ROUGE-N: ROUGE-N was originally a recall oriented metric that considered N-gram recall between a system generated summary and the corresponding gold human summaries. In later versions, in addition to the recall, precision was also considered in ROUGE-N, which is the precision of N-grams in the system generated summary with respect to the gold human summary. To combine both precision and recall, F1 scores are often reported. Common values of N range from 1 to 4. – ROUGE-L: This variant of ROUGE compares the system generated summary and the human generated summary based on the Longest Common Subsequences (LCS) between them. The premise is that, longer LCS between the system and human summaries shows more similarity and therefore higher quality of the system summary – ROUGE-W: One problem with ROUGE-L is that all LCS with same lengths are rewarded equally. The LCS can be either related to a consecutive set of words or a long sequence with many gaps. While ROUGE-L treats all sequence matches equally, it makes sense that sequences with many gaps receive lower scores in comparison with consecutive matches. ROUGE-W considers an additional weighting function that awards consecutive matches more than non-consecutive ones. – ROUGE-S: ROUGE-S computes the skip-bigram co-occurrence statistics between the two summaries. It is similar to ROUGE-2 except that it allows gaps between the bigrams by skipping middle tokens. – ROUGE-SU: ROUGE-S does not give any credit to a system generated sentence if the sentence does not have any word pair co-occurring in the reference sentence. To solve this potential problem, ROUGE-SU was proposed which is an extension of ROUGE-S that also considers unigram matches between the two summaries. ROUGE-L, ROUGE-W, ROUGE-S and ROUGE-SU were later extended to consider both the recall and precision. In calculating ROUGE, stopword removal or stemming can also be considered, resulting in more variants. In the summarization literature, despite the large number of variants of ROUGE, only one or very few of these variants are often chosen (arbitrarily) for evaluation of the quality of the summarization approaches. When ROUGE was proposed, the original variants were only recall-oriented and hence the reported correlation results (Lin, 2004).
Answer: Yes. Such as ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, ROUGE-2, ROUGE-SU,.