## 6 Results  
To understand the results from the Text Analysis Conference, one more task subdivision needs to be mentioned. For both the update summaries and the initial summaries, participants submit up to four sets of score predictions (for a total of eight). However, the number is actually sixteen, since participants train each of the eight models twice. The first iteration of each is when participants are trying to predict only the scores of machine-generated summaries (called the \"no models\" subtask). In the second, participants train one model to simultaneously predict the scores of both human summaries and machine summaries (called the \"all peers\" subtask). Figure 1, presented in Section 1, demonstrates why this second iteration exists. When predicting only the machine summaries, the best-fit line for ROUGE-2 and ROUGE-SU4 goes through most of the data, but does not correctly fit the human summaries. This situation improves significantly for several of our systems and is shown in Figure 6 with System 25 (our canonical correlation model aimed at predicting responsiveness). System 25 shrinks the gap for initial summaries and almost eliminates it for update summaries. Figures 2, 3 and 7 give the Pearson correlation against overall responsiveness, pyramid scoring, and readability for the 2011 update summary tasks for the top 14 performing AESOP submissions (out of 25 total evaluation metrics). Our metrics are usually stronger for the update tasks, but in general, all of our submissions were among the top performers. We note that as indicated by the error bars, many of the best performing systems, including a number of our metrics, significantly outperformed the baselines of the three ROUGE methods in the \"all peers\" task. On the other hand, while ROUGE is not the best for the no models data, it is always within the 95% confidence interval of the best metric. The two left-hand plots are ROUGE-2 and ROUGE-SU4, two of the standard baseline evaluation systems. The right-hand plots show our System 25, a canonical correlation model aimed at predicting overall responsiveness. The regression line in each plot only goes as far as the lowest-scoring human summary. System 25 closes the gap quite well for update summaries and is also a significant improvement for initial summaries. Figure 6: The performance of our System 25 at TAC 2011. Systems 1-3 are the baseline evaluation systems (see Results section). Our systems are numbers 6, 8, 23 and 25. Systems 12 (AutoSummENG) and 18 (MeMoG) are due to [PERSON] _et al_. [8]. System 4 is VERT-F, due to [PERSON] _et al_. The vertical bar shown for each system is a 95% confidence interval for its Pearson correlation with the human metric listed at the top of the figure. Figure 7: Pearson correlation with responsiveness top 14 AESOP metrics all peers and no models (2011). For the update summaries (Figures 7, 2 and 3), there was no clear winner among our systems. Each of our four submissions was at least once our best system. System 6 (non-negative least squares model aimed at predicting responsiveness) seems to be our best for predicting the quality of machine summaries alone, and was the overall best in predicting responsiveness and pyramid for that subset. However, system 6 was twice our worst for evaluating the humans and machines together. [PERSON]'s system 4 (VERT-F) was the best at predicting readability in the tasks with and without human summaries, and [PERSON] system 18 did the best at predicting responsiveness when the humans were included. Perhaps most impressively, all four of our systems (6, 8, 23 and 25) were among the top six performers (out of 25) for predicting any of the three metrics with human summaries included (_i.e._, the \"all peers\" task). In order to further validate our results, we applied our sixteen tuned models to the TAC 2008 summarization data. Of our TAC 2008-2011 data, 2008 is the only other year (besides 2011) that our models did not use for training.