An Automatic Method for Summary Evaluation
Using Multiple Evaluation Results by a Manual Method
Hidetsugu Nanba
Faculty of Information Sciences,
Hiroshima City University
3-4-1 Ozuka, Hiroshima, 731-3194 Japannanba@its.hiroshima-cu.ac.jp

Abstract
To solve a problem of how to evaluate
computer-produced summaries, a numberof automatic and manual methods havebeen proposed. Manual methods evaluatesummaries correctly, because humans
evaluate them, but are costly. On theother hand, automatic methods, which
use evaluation tools or programs, are low
cost, although these methods cannot
evaluate summaries as accurately as
manual methods. In this paper, weinvestigate an automatic evaluation
method that can reduce the errors of
traditional automatic methods by using
several evaluation results obtained
manually.
We
conducted
someexperiments using the data of the Text
Summarization Challenge 2 (TSC-2). A
comparison with conventional automatic
methods shows that our method
outperforms other methods usually used.

1

Introduction

Recently, the evaluation of computer-produced
summaries has become recognized as one of the
problem areas that must be addressed in the fieldof automatic summarization. To solve thisproblem, a number of automatic (Donaway et al.,2000, Hirao et al., 2005, Lin et al., 2003, Lin,
2004, Hori et al., 2003) and manual methods(Nenkova et al., 2004, Teufel et al., 2004) havebeen proposed. Manual methods evaluatesummaries correctly, because humans evaluatethem, but are costly. On the other hand,automatic methods, which use evaluation tools or
programs, are low cost, although these methods
cannot evaluate summaries as accurately as
manual methods. In this paper, we investigate an

Manabu Okumura
Precision and Intelligence Laboratory,
Tokyo Institute of Technology
4259 Nagatsuta, Yokohama, 226-8503 Japanoku@pi.titech.ac.jp

automatic method that can reduce the errors of
traditional automatic methods by using several
evaluation results obtained manually. Unlike
other automatic methods, our method estimatesmanual evaluation scores. Therefore, our method
makes it possible to compare a new system withother systems that have been evaluated manually.
There are two research studies related to ourwork (Kazawa et al., 2003, Yasuda et al., 2003).
Kazawa et al. (2003) proposed an automatic
evaluation method using multiple evaluation
results from a manual method. In the field ofmachine translation, Yasuda et al. (2003)
proposed an automatic method that gives anevaluation result of a translation system as ascore for the Test of English for International
Communication(TOEIC).
Althoughtheeffectiveness of both methods was confirmed
experimentally, further discussion of four points,
which we describe in Section 3, is necessary fora more accurate summary evaluation. In thispaper, we address three of these points based on
Kazawa’s and Yasuda’s methods. We also
investigate whether these methods canoutperform other automatic methods.
The remainder of this paper is organized asfollows. Section 2 describes related work.
Section 3 describes our method. To investigate
the effectiveness of our method, we conducted
some examinations and Section 4 reports onthese. We present some conclusions in Section 5.

2

Related Work

Generally, similar summaries are considered toobtain similar evaluation results. If there is a setof summaries (pooled summaries) produced froma document (or multiple documents) and if theseare evaluated manually, then we can estimate amanual evaluation score for any summary to beevaluated with the evaluation results for thosepooled summaries. Based on this idea, Kazawa et(Step 1) Prepare summaries and theirevaluation results by a manual methodal. (2003) proposed an automatic method using
multiple evaluation results from a manual
method. First, n summaries for each document, m,were prepared. A summarization systemgenerated summaries from m documents. Here,
we represent the ith summary for the jth document
and its evaluation score as xij and yij, respectively.
The system was evaluated using Equation 1.
m(Step 2) Calculate the similarities between asummary to be evaluated and the pooledsummariesnscr ( x ) = ∑ ∑ w j yij Sim ( x, xij ) + b (1)(Step 3) Combine manual scores of pooledsummaries in proportion to their similarities
to the summary to be evaluatedi =1 j =1

The evaluation score of summary x was
obtained by summing parameter b for all thesubscores calculated for each pooled summary,xij. A subscore was obtained by multiplying aparameter wj, by the evaluation score yij, and thesimilarity between x and xij.
In the field of machine translation, there isanother related study. Yasuda et al. (2003)
proposed an automatic method that gives anevaluation result of a translation system as ascore for TOEIC. They prepared 29 humansubjects, whose TOEIC scores were from 300s to
800s, and asked them to translate 23 Japaneseconversations into English. They also generated
translations using a system for each conversation.
Then, they evaluated both translations using an
automatic method, and obtained WH, whichindicated the ratio of system translations thatwere superior to human translations. Yasuda et al.
calculated WH for each subject and plotted thevalues along with their corresponding TOEIC
scores to produce a regression line. Finally, they
defined a point where the regression line crossed
WH = 0.5 to provide the TOEIC score for thesystem.
Though, the effectiveness of Kazawa’s andYasuda’s
methods
were
confirmed
experimentally, further discussions of four points,
which we describe in the next section, arenecessary for a more accurate summary
evaluation.

3

Investigation of an Automatic Method
using Multiple Manual Evaluation
Results

3.1

Overview of Our Evaluation Method
and Essential Points to be Discussed

We investigate an automatic method using
multiple evaluation results by a manual method
based on Kazawa’s and Yasuda’s method. The
procedure of our evaluation method is shown as
follows;

For each step, we need to discuss the following
points.
(Step 1)
1. How many summaries, and what type(variety) of summaries should be prepared?
Kazawa et al. prepared 6 summaries foreach document, and Yasuda et al. prepared
29 translations for each conversation.
However, they did not examine about the
number and the type of pooled summaries
required to the evaluation.
(Step 2)
2. Which measure is better for calculating thesimilarities between a summary to beevaluated and the pooled summaries?
Kazawa et al. used Equation 2 to calculate
similarities.

Sim ( x, xij ) =| xij ∩ x |min(| xij |, | x |)(2)

where xij ∩ x indicates the number of
discourse units1 that appear in both xij and x,
and | x | represents the number of words in x.
However, there are many other measuresthat can be used to calculate the topicalsimilarities between two documents (orpassages).
As well as Yasuda’s method does, using
WH is another way to calculate similarities
between a summary to be evaluated andpooled summaries indirectly. Yasuda et al.
(2003) tested DP matching (Su et al., 1992),
BLEU (Papineni et al., 2002), and NIST2 ,for the calculation of WH. However there aremany other measures for summary
evaluation.
1

Rhetorical Structure Theory Discourse Treebank.
www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalog
Id=LDC2002T07 Linguistic Data Consortium.
2

http://www.nist.gov/speech/tests/mt/mt2001/resource/How many summaries should be used to
calculate the score of a summary to beevaluated? Kazawa et al. used all the pooledsummaries but this does not ensure the best
performance of their evaluation method.
(Step 3)
4. How to combine the manual scores of thepooled summaries? Kazawa et al. calculated
the score of a summary as a weighted linear
sum of the manual scores. Applying
regression analysis (Yasuda et al., 2003) isanother method of combining several
manual scores.

used the similarity measure, which obtain
the best performance in Exp-1.
Exp-3: We also examined Point 2 based on
Yasuda’s method. As a similarity measure,
we tested cosine distance and 11 kinds of
ROUGE. Then, we examined Point 4 bycomparing the result of Yasuda’s methodwith that of Kazawa’s.
Exp-4: In the same way as Exp-2, wecompared the evaluation with otherautomatic methods, which we describe in
the next section, to investigate whether theevaluation based on Yasuda’s method canoutperform other automatic methods.

3.

3.2

z

z

Three Points Addressed in Our Study

We address the second, third and fourth points in
Section 3.1.
(Point 2) A measure for calculating
similarities between a summary to beevaluated and pooled summaries:
There are many measures that can calculate thetopical similarities between two documents (orpassages). We tested several measures, such as
ROUGE (Lin, 2004) and the cosine distance. We
describe these measures in detail in Section 4.2.
(Point 3) The number of summaries used to
calculate the score of a summary to beevaluated:
We use summaries whose similarities to asummary to be evaluated are higher than athreshold value.
(Point 4) Combination of manual scores:
We used both Kazawa’s and Yasuda’s methods.

4

Experiments

4.1

Experimental Methods

4.2

Automatic Evaluation Methods Used in
the Experiments

In the following, we show the automatic
evaluation methods used in our experiments.
Content-based evaluation (Donaway et al.,2000)
This measure evaluates summaries by comparing
their content words with those of the humanproduced extracts. The score of the contentbased measure is obtained by computing thesimilarity between the term vector using tf*idf
weighting of a computer-produced summary andthe term vector of a human-produced summary
by cosine distance.
ROUGE-N (Lin, 2004)
This measure compares n-grams of twosummaries, and counts the number of matches.
The measure is defined by Equation 3.
ROUGE − N =∑ ∑ Count

S ∈R gram N ∈Szz

Exp-1: We examined Points 2 and 3 basedon Kazawa’s method. We tested thresholdvalues from 0 to 1 at 0.005 intervals. We
also tested several similarity measures, suchas cosine distance and 11 kinds of ROUGE.
Exp-2: In order to investigate whether theevaluation based on Kazawa’s method canoutperform other automatic methods, we
compared the evaluation with otherautomatic methods. In this experiment, we( gram N )∑ ∑ Count ( gram )S ∈R gram N ∈S

To investigate the three points described in
Section 3.2, we conducted the following four
experiments.

match

(3)

N

where Count(gramN) is the number of an N-gram
and Countmatch(gramN) denotes the number of ngram co-occurrences in two summaries.
ROUGE-L (Lin, 2004)
This measure evaluates summaries by longest
common subsequence (LCS) defined by
Equation 4.
u∑ LCS (r , C )ROUGE − L = i =i

∪

i

(4)

m

where LCSU(ri,C) is the LCS score of the union’s
longest common subsequence between reference
sentences ri and the summary to be evaluated,and m is the number of words contained in areference summary.

ROUGE-S (Lin, 2004)
Skip-bigram is any pair of words in theirsentence order, allowing for arbitrary gaps.
ROUGE-S measures the overlap of skip-bigramsin a candidate summary and a reference
summary. Several variations of ROUGE-S are
possible by limiting the maximum skip distance
between the two in-order words that are allowed
to form a skip-bigram. In the following,
ROUGE-SN denotes ROUGE-S with maximum
skip distance N.
ROUGE-SU (Lin, 2004)
This measure is an extension of ROUGE-S; it
adds a unigram as a counting unit. In thefollowing, ROUGE-SUN denotes ROUGE-SU
with maximum skip distance N.
4.3

Evaluation Methods

In the following, we elaborate on the evaluation
methods for each experiment.
Exp-1: An experiment for Points 2 and 3
based on Kazawa’s method
We evaluated Kazawa’s method from theviewpoint of “Gap”. Differing from otherautomatic methods, the method uses multiple
manual evaluation results and estimates the
manual scores of the summaries to be evaluatedor the summarization systems. We therefore
evaluated the automatic methods using Gap,which manually indicates the difference between
the scores from a manual method and eachautomatic method that estimates the scores. First,
an arbitrary summary is selected from the 10
summaries in a dataset, which we describe in
Section 4.4, and an evaluation score is calculated
by Kazawa’s method using the other ninesummaries. The score is compared with a manualscore of the summary by Gap, which is definedby Equation 5.
m

Gap =

threshold value. Therefore, we checked thecoverage of the method, which is defined by
Equation 6.
The number of summaries
evaluated by the method
Coverage =The number of given summaries

Exp-2: Comparison of Kazawa’s method withother automatic methods
Traditionally, automatic methods have been
evaluated by “Ranking”. This means thatsummarization systems are ranked based on the
results of the automatic and manual methods.
Then, the effectiveness of the automatic method
is evaluated by the number of matches between
both rankings using Spearman’s rank correlation
coefficient and Pearson’s rank correlation
coefficient (Lin et al., 2003, Lin, 2004, Hirao etal., 2005). However, we did not use bothcorrelation coefficients, because evaluation
scores are not always calculated by a Kazawabased method, which we described in Exp-1.
Therefore, we ranked the summaries instead of
the summarization systems. Two arbitrarysummaries from the 10 summaries in a dataset
were selected and ranked by Kazawa’s method.
Then, Kazawa’s method was evaluated using“Precision,” which calculates the percentage ofcases where the order of the manual method of
the two summaries matches the order of theirranks calculated by Kazawa’s method. The twosummaries were also ranked by ROUGE and bycosine distance, and both Precision values were
calculated. Finally, the Precision value of
Kazawa’s method was compared with those of
ROUGE and cosine distance.
Exp-3: An experiment for Point 2 based on
Yasuda’s method
An arbitrary system was selected from the 10
systems, and Yasuda’s method estimated its
manual score from the other nine systems.
Yasuda’s method was evaluated by Gap, which
is defined by Equation 7.
mn∑∑ | scr ' ( xkl ) − y kl |k =1 l =1

m×n

(6)

Gap =

(5)

where xkl is the kth system’s lth summary, and ykl
is the score from a manual evaluation method forthe kth system’s lth summary. To distinguish ourevaluation function from Kazawa’s, we denote it
as scr’(x). As a similarity measure in scr’(x), we
tested ROUGE and the cosine distance.
We also tested the coverage of the automatic
method. The method cannot calculate scores if
there are no similar summaries above a given∑ | s( x ) − y |
k =1

k

m

k

(7)

where xk is the kth system, s(xk) is a score of xk by
Yasuda’s method, and yk is the manual score forthe kth system. Yasuda et al. (2003) tested DP
matching (Su et al., 1992), BLEU (Papineni et al.,2002), and NIST3, as automatic methods used intheir evaluation. Instead of those methods, we

3

http://www.nist.gov/speech/tests/mt/mt2001/resource/tested ROUGE and cosine distance, both ofwhich have been used for summary evaluation.
If a score by Yasuda’s method exceeds therange of the manual score, the score is modified
to be within the range. In our experiments, weused evaluation by revision (Fukushima et al.,2002) as the manual evaluation method. The
range of the score of this method is between zeroand 0.5. If the score is less than zero, it ischanged to zero and if greater than 0.5 it ischanged to 0.5.
Exp-4: Comparison of Yasuda’s method andother automatic methods
In the same way as for the evaluation of
Kazawa’s method in Exp-2, we evaluatedYasuda’s method by Precision. Two arbitrarysummaries from the 10 summaries in a dataset
were selected, and ranked by Yasuda’s method.
Then, Yasuda’s method was evaluated using
Precision. Two summaries were also ranked by
ROUGE and by cosine distance and both
Precision values were calculated. Finally, thePrecision value of Yasuda’s method wascompared with those of ROUGE and cosine
distance.
4.4

The Data Used in Our Experiments

We used the TSC-2 data (Fukushima et al.,2002) in our examinations. The data consisted ofhuman-produced extracts (denoted as “PART”),
human-produced abstracts (denoted as “FREE”),
computer-produced summaries (eight systems
and a baseline system using the lead method(denoted as “LEAD”)) 4 , and their evaluation
results by two manual methods. All thesummaries were derived from 30 newspaper
articles, written in Japanese, and were extracted
from the Mainichi newspaper database for theyears 1998 and 1999. Two tasks were conductedin TSC-2, and we used the data from a single
document summarization task. In this task,
participants were asked to produce summaries inplain text in the ratios of 20% and 40%.
Summaries were evaluated using a ranking
evaluation method and the revision method
evaluation. In our experiments, we used theresults of evaluation from the revision method.
This method evaluates summaries by measuring
the degree to which computer-produced
summaries are revised. The judges read the4

In Exp-2 and 4, we evaluated “PART”, “LEAD”,
and eight systems (candidate summaries) byautomatic methods using “FREE” as the reference
summaries.

original texts and revised the computer-produced
summaries in terms of their content andreadability. The human revisions were made with
only three editing operations (insertion, deletion,
replacement). The degree of the human revision,
called the “edit distance,” is computed from thenumber of revised characters divided by the
number of characters in the original summary. If
the summary’s quality was so low that a revisionof more than half of the original summary was
required, the judges stopped the revision and ascore of 0.5 was given.
The effectiveness of evaluation by the revision
method was confirmed in our previous work
(Nanba et al., 2004). We compared evaluation byrevision with ranking evaluation. We also tested
otherautomatic
methods:
content-based
evaluation, BLEU (Papineni et al., 2001) and
ROUGE-1 (Lin, 2004), and compared theirresults with that of evaluation by revision as
reference. As a result, we found that evaluation
by revision is effective for recognizing slightdifferences
betweencomputer-produced
summaries.
4.5

Experimental Results and Discussion

Exp-1: An experiment for Points 2 and 3
based on Kazawa’s method
To address Points 2 and 3, we evaluatedsummaries by the method based on Kazawa’s
method using 12 measures, described in Section
4.4, as measures to calculate topical similarities
between summaries, and compared thesemeasures by Gap. The experimental results forsummarization ratios of 40% and 20% areshown in Tables 1 and 2, respectively. Tables
show the Gap values of 12 measures for each
Coverage value from 0.2 to 1.0 at 0.1 intervals.
Average values of Gap for each measure are also
shown in these tables. As can be seen from
Tables 1 and 2, the larger the threshold value,
the smaller the value of Gap. From the result, we
can conclude for Point 3 that more accurate
evaluation is possible when we use similarpooled summaries (Point 2). However, thenumber of summaries that can be evaluated bythis method was limited when the thresholdvalue was large.
Of the 12 measures, unigram-based methods,
such as cosine distance and ROUGE-1, produced
good results. However, there were no significant
differences between measures except for when
ROUGE-L was used.

Table 1 Comparison of Gap values for several measures(ratio: 40%)
Coverage
Measure
R-1
R-2
R-3
R-4
R-L
R-S
R-S4
R-S9
R-SU
R-SU4
R-SU9
CosineThreshold

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.080 0.070
0.082 0.074
0.083 0.074
0.085 0.078
0.102 0.100
0.083 0.077
0.083 0.072
0.083 0.075
0.083 0.077
0.082 0.073
0.083 0.074
0.081 0.074
Small

0.067
0.070
0.075
0.076
0.097
0.073
0.071
0.069
0.070
0.069
0.070
0.065

0.057
0.070
0.071
0.073
0.094
0.073
0.069
0.070
0.071
0.069
0.068
0.062

0.064
0.069
0.069
0.069
0.091
0.069
0.066
0.067
0.069
0.065
0.066
0.059

0.062
0.063
0.063
0.064
0.090
0.067
0.066
0.066
0.068
0.068
0.067
0.056

0.058
0.059
0.059
0.060
0.089
0.064
0.060
0.066
0.064
0.063
0.066
0.057

0.3

0.2

Average0.045 0.041
0.051 0.042
0.051 0.045
0.051 0.043
0.082 0.078
0.060 0.045
0.054 0.044
0.057 0.046
0.057 0.043
0.051 0.043
0.054 0.046
0.039 0.043
Large

0.062
0.065
0.066
0.067
0.091
0.068
0.065
0.067
0.067
0.065
0.066
0.059

Table 2 Comparison of Gap values for several measures(ratio: 20%)
Coverage
Measure
R-1
R-2
R-3
R-4
R-L
R-S
R-S4
R-S9
R-SU
R-SU4
R-SU9
CosineThreshold

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.129 0.104
0.132 0.115
0.132 0.115
0.134 0.121
0.140 0.135
0.130 0.119
0.130 0.114
0.130 0.119
0.130 0.118
0.130 0.111
0.130 0.116
0.128 0.106
Small

0.102
0.107
0.116
0.121
0.134
0.113
0.109
0.113
0.109
0.107
0.108
0.102

0.976
0.109
0.111
0.112
0.125
0.106
0.105
0.105
0.109
0.106
0.105
0.094

0.090
0.096
0.102
0.103
0.117
0.098
0.102
0.095
0.097
0.100
0.096
0.091

0.089
0.093
0.092
0.090
0.110
0.099
0.092
0.097
0.098
0.090
0.090
0.090

0.089
0.079
0.080
0.080
0.105
0.089
0.085
0.095
0.088
0.086
0.085
0.079

Exp-2: Comparison of Kazawa’s method withother automatic methods (Point 2)
In Exp-1, cosine distance outperformed the other11 measures. We therefore used cosine distancein Kazawa’s method in Exp-2. We ranked
summaries by Kazawa’s method, ROUGE andcosine distance, calculated using Precision.
The results of the evaluation by Precision forsummarization ratios of 40% and 20% are shownin Figures 1 and 2, respectively. We plotted thePrecision value of Kazawa’s method by changing
the threshold value from 0 to 1 at 0.05 intervals.
We also plotted the Precision values of ROUGE2 as dotted lines. ROUGE-2 was superior to theother 11 measures in terms of Ranking. The X
and Y axes in Figures 1 and 2 show the thresholdvalue of Kazawa’s method and the Precision
values, respectively. From the result shown in
Figure 1, we found that Kazawa’s method

0.3

0.2

Average

0.083 0.082
0.081 0.082
0.078 0.079
0.080 0.078
0.769 0.060
0.089 0.087
0.088 0.085
0.085 0.084
0.089 0.079
0.084 0.087
0.085 0.082
0.080 0.057
Large

0.096
0.099
0.101
0.102
0.111
0.103
0.101
0.103
0.102
0.100
0.099
0.092

outperformed ROUGE-2, when the thresholdvalue was greater than 0.968. The Coveragevalue of this point was 0.203. In Figure 2, thePrecision curve of Kazawa’s method crossed thedotted line at a threshold value of 0.890. The
Coverage value of this point was 0.405.
To improve these Coverage values, we need toprepare more summaries and their manual
evaluation results, because the Coverage iscritically dependent on the number and variety ofpooled summaries. This is exactly the first point
in Section 3.1, which we do not address in thispaper. We will investigate this point as the next
step in our future work.

Table 3 Gap between the manual method andYasuda’s method

1

precision

0.8
0.6
0.4
0.2
0
0

0.2

0.4

0.6

0.8

1

threshold value
Kazawa's method

R-2

Figure 1 Comparison of Kazawa’s method and
ROUGE-2 (ratio: 40%)
1

precision

0.8
0.6
0.4
0.2
0
0

0.2

0.4

0.6

0.8

1

threshold value
Kazawa's method

R-2

Figure 2 Comparison of Kazawa’s method and
ROUGE-2 (ratio: 20%)
Exp-3: An experiment for Point 3 based on
Yasuda’s method
For Point 2 in Section 3.2, we also examined
Yasuda’s method. The experimental result by
Gap is shown in Table 3. When the ratio is 20%,
ROUGE-SU4 is the best. The N-gram and theskip-bigram are both useful when thesummarization ratio is low.
For Point 4, we compared the result by
Yasuda’s method (Table 3) with that of
Kazawa’s method (in Tables 1 and 2). Yasuda’s
method could accurately estimate manual scores.
In particular, the Gap values of 0.023 by
ROUGE-2 and by ROUGE-3 are smaller thanthose produced by Kazawa’s method with athreshold value of 0.9 (Tables 1 and 2). This
indicates that regression analysis used in
Yasuda’s method is superior to that used in
Kazawa’s method.

Cosine
R-1
R-2
R-3
R-4
R-L
R-S(∞)
R-S4
R-S9
R-SU(∞)
R-SU4
R-SU9

Ratio
20%
40%
0.037
0.031
0.033
0.022
0.028
0.023
0.028
0.023
0.036
0.024
0.040
0.038
0.051
0.060
0.025
0.040
0.042
0.052
0.027
0.055
0.037
0.022
0.023
0.048

Average
0.035
0.028
0.025
0.025
0.030
0.039
0.055
0.033
0.047
0.041
0.029
0.036

Exp-4: Comparison of Yasuda’s method withother automatic methods
We also evaluated Yasuda’s method bycomparison with other automatic methods interms of Ranking. We evaluated 10 systems by
Yasuda’s method with ROUGE-3, which
produced the best results in Exp-3. We also
evaluated the systems by ROUGE and cosine
distance, and compared the results. The results
are shown in Table 4.
Table 4 Comparison between Yasuda’s method andautomatic methods

Ratio
YasudaCosine
R-1
R-2
R-3
R-4
R-L
R-S(∞)
R-S4
R-S9
R-SU(∞)
R-SU4
R-SU9

20%
0.867
0.844
0.822
0.844
0.822
0.822
0.822
0.667
0.800
0.733
0.711
0.800
0.756

Average
40%
0.844
0.800
0.778
0.800
0.800
0.844
0.800
0.689
0.756
0.689
0.711
0.822
0.711

0.856
0.822
0.800
0.822
0.811
0.833
0.811
0.678
0.778
0.711
0.711
0.811
0.733

As can be seen from Table 4, Yasuda’s method
produced the best results for the ratios of 20%
and 40%. Of the automatic methods compared,
ROUGE-4 was the best.

As evaluation scores by Yasuda’s method
were calculated based on ROUGE-3, there wereno striking differences between Yasuda’s method
and the others except for the integration processof evaluation scores for each summary. Yasuda’s
method uses a regression analysis, whereas theother methods average the scores for eachsummary. Yasuda’s method using ROUGE-3
outperformed the original ROUGE-3 for bothratios, 20% and 40%.

5Conclusions

We have investigated an automatic method thatuses several evaluation results from a manual
method based on Kazawa’s and Yasuda’s
methods. From the experimental results based on
Kazawa’s method, we found that limiting thenumber of pooled summaries could produce
better results than using all the pooled summaries.
However, the number of summaries that can beevaluated by this method was limited. To
improve the Coverage of Kazawa’s method,more summaries and their evaluation results are
required, because the Coverage is criticallydependent on the number and variety of pooledsummaries.
We also investigated an automatic method
based on Yasuda’s method and found that themethod using ROUGE-2 and -3 could accurately
estimate manual scores, and could outperform
Kazawa’s method and the other automatic
methods tested. From these results, we canconclude that the automatic method performed
the best when ROUGE-2 or 3 is used as asimilarity measure, and a regression analysis isused for combining manual method.

References
Robert L. Donaway, Kevin W. Drummey and Laura
A. Mather. 2000. A Comparison of Rankings
Produced by Summarization Evaluation Measures.
Proceedings of the ANLP/NAACL 2000
Workshop on Automatic Summarization: 69–78.
Takahiro Fukushima and Manabu Okumura. 2001.
Text
Summarization
Challenge/Text
Summarization Evaluation at NTCIR Workshop2.
Proceedings of the Second NTCIR Workshop on
Research in Chinese and Japanese Text Retrieval
and Text Summarization: 45–51.
Takahiro Fukushima, Manabu Okumura and
Hidetsugu Nanba. 2002. Text Summarization
Challenge 2/Text Summarization Evaluation at
NTCIR Workshop3. Working Notes of the 3rd
NTCIR Workshop Meeting, PART V: 1–7.

Tsutomu Hirao, Manabu Okumura, and Hideki
Isozaki. 2005. Kernel-based Approach forAutomatic Evaluation of Natural Language
Generation
Technologies:
Application
toAutomatic Summarization. Proceedings of HLTEMNLP 2005: 145–152.
Chiori Hori, Takaaki Hori, and Sadaoki Furui. 2003.
Evaluation Methods for Automatic Speech
Summarization. Proceedings of Eurospeech 2003:
2825–2828.
Hideto Kazawa, Thomas Arrigan, Tsutomu Hirao andEisaku Maeda. 2003. An Automatic Evaluation
Method of Machine-Generated Extracts. IPSJ SIG
Technical Reports, 2003-NL-158: 25–30. (inJapanese).
Chin-Yew Lin and Eduard Hovy. 2003. Automatic
Evaluation of Summaries Using N-gram CoOccurrence Statistics. Proceedings of the Human
Language Technology Conference 2003: 71–78.
Chin-Yew Lin. 2004. ROUGE: A Package for
Automatic Evaluation of Summaries. Proceedingsof the ACL-04 Workshop “Text Summarization
Branches Out”: 74–81.
Hidetsugu Nanba and Manabu Okumura. 2004.
Comparison of Some Automatic and Manual
Methods for Summary Evaluation Based on the
Text Summarization Challenge 2. Proceedings ofthe Fourth International Conference on Language
Resources and Evaluation: 1029–1032.
Ani Nenkova and Rebecca Passonneau, 2004.
Evaluating Content Selection in Summarization:
The Pyramid Method. Proceedings of HLT-NAACL
2004: 145–152.
Kishore Papineni, Salim Roukos, Todd Ward andWei-Jing Zhu. 2001. BLEU: A Method forAutomatic Evaluation of Machine Translation.
IBM Research Report, RC22176 (W0109-022).
Keh-Yih Su, Ming-Wen Wu, and Jing-Shin Chang.
1992. A New Quantitative Quality Measure for
Machine Translation Systems. Proceedings of the
14th International Conference on Computational
Linguistics: 433–439.
Simone Teufel and Hans van Halteren. 2004.
Evaluating Information Content by Factoid
Analysis: Human Annotation and Stability.
Proceedings of EMNLP 2004: 419–426.
Kenji Yasuda, Fumiaki Sugaya, Toshiyuki Takezawa,
Seiichi Yamamoto and Masuzo Yanagida. 2003.
Applications of Automatic Evaluation Methods to
Measuring a Capability of Speech Translation
System. Proceedings of the Tenth Conference of
the European Chapter of the Association for
Computational Linguistics: 371–378.