## 2 Related Work  
Previous work has looked at extensions to (word) n-gram approaches for judging the quality of a summary relative to one or more human-generated summaries. [PERSON] and [PERSON] [5] analyzed summarization evaluation data from the Document Understanding Conferences (DUC [6], the TAC predecessor) for the years 2005-2007. They proposed using robust regression to find canonical correlation coefficients to improve the correlation between the automatically-generated metric and human-generated metrics. Their approach used several variants of the ROUGE (word) n-gram scoring to produce a new metric, called ROUGE Optimal Summarization Evaluation (ROSE). ROSE was successful in improving the correlation within a given year, which would be useful for researchers producing new summarization methods to build an interpolation model to compare their new method with methods that were evaluated by humans. However, the method was not successful in improving the correlation when a model was built using one year's data, say 2005, and then applying it to another year's data, say 2006. The authors did demonstrate that if the actual linguistic quality scores were known, then such a cross-year improvement could be attained. For \"update summaries\", the base summary (or summaries), which encapsulates the readers' knowledge of the topic so far, must be taken into account. An update summary is a text summary that is generated on an \"evolving topic\" and summaries are generated for new documents with a focus on what is novel in the new set of documents. Evaluation of such summaries can pose a challenge. An approach that uses the ROUGE content metrics comparing the information in the update summary to that of the previously generated summaries on the topic was proposed by [PERSON], [PERSON] and [PERSON] [6]. Nouveau-ROUGE computes a linear combination of the similarity of the summary with not only the current human-generated summaries on the topic but those corresponding to the previously known information. Such an approach was shown to improve correlation with both pyramid and overall responsiveness metrics, two important human-generated scores given to summaries. [PERSON] _et al_. [7] proposed a system called Valuation using Enhanced Rationale Technique (VERT-F). VERT-F compares the (word) bigrams of summaries to be evaluated with human-generated summaries by computing a statistic which is a combination of 4 others: the \(\chi^{2}\) statistic, and the three metrics from information retrieval--precision, recall, and the geometric mean of these (\(F-\)measure). [PERSON] _et al_. [8] proposed a novel (character) n-gram approach, AutoSummENG (AUTOmatic SUMMARY Evaluation based on N-gram Graphs), which was designed to be language independent. The approach builds graphs for a given summary based on n-gram counts. Graph similarity metrics are then used to compare summaries. The resulting metrics were shown to be competitive with ROUGE scoring. ([PERSON] _et al_. [9] turned the evaluation around and used it as a method to generate summaries.) The Merged Model Graph method (MeMoG) [10] is a variation on AutoSummENG where instead of comparing summaries to summaries, a summary's graph is compared to a merged graph of 3 human-generated summaries. We will compare the performance of our approach to AutoSummENG, MeMoG, and VERT-F, as well as several variations of ROUGE in the Results section. [PERSON], [PERSON], and [PERSON] [11] went in a different direction and sought to automatically evaluate the linguistic quality of a summary. They tested numerous high-level and low-level features such as cosine similarity between sentences, co-reference information, and number of syllables in a sentence's words in an attempt to separately predict several aspects of linguistic quality, including grammaticality, non-redundancy, referential clarity, focus, and structure and coherence. Their best results were 90% accuracy for pairwise comparisons of competing systems and 70% accuracy for ranking summaries.