Some Examinations of Intrinsic Methods for Summary Evaluation
Based on the Text Summarization Challenge (TSC)
Hidetsugu Nanba and Manabu Okumura
Precision and Intelligence Lab.,Tokyo Institute of Technology,
4259 Nagatsuta
 Midori-ku Yokohama, JAPAN, 226-8503
nanba, oku  @pi.titech.ac.jp
Abstract
Computer-produced summaries have traditionally been evaluated by comparing them with human-produced summaries using the Fmeasure. However, the F-measure is not appropriate when alternative sentences are possible in a human-produced extract. In thispaper, we examine some evaluation methods devised to overcome the problem, including utility-based evaluation. By giving scores formoderately important sentences that does not appear in the human-produced extract, utility-based evaluation can resolve the problem.
However, the method requires much effort from humans to provide data for evaluation. In this paper, we first propose a pseudo-utilitybased evaluation that uses human-produced extracts at different compression ratios. To evaluate the effectiveness of pseudo-utility-based
evaluation, we compare our method and the F-measure using the data of the Text Summarization Challenge (TSC), and show that pseudoutility-based evaluation can resolve this problem. Next, we focus on content-based evaluation. Instead of measuring the ratio of sentencesthat match exactly in the extract, the method evaluates extracts by comparing their content words to those of human-produced extracts.
Although the method has been reported to be effective in resolving the problem, it has not been examined in the context of comparing
two extracts produced from different systems. We evaluated computer-produced summaries by content-based evaluation, and compared
the results with a subjective evaluation. We found that the evaluation by content-based measure matched those by subjective evaluationin 93% of the cases, if the gap in content-based scores between two summaries is more than 0.2.

1. Introduction
Recently, the issue of how to evaluate computerproduced summaries has been recognized as one of the
problems that must be addressed in the field of automatic
summarization. Evaluation methods for text summarization can be divided into two categories: intrinsic andextrinsic(Sparck-Jones and Galliers, 1996). The methods
that evaluate system outputs (summaries) themselves areoften called intrinsic methods. On the other hand, extrinsic
methods evaluate the performance of a summarization system in a given task, such as information retrieval and text
categorization. In this paper, we focus on intrinsic methods.
Computer-produced summaries have traditionally been
evaluated by comparing them with human-produced summaries using the F-measure. Jing et al.(Jing, Barzilay,
McKeown, and Elhadad, 1998), however, pointed out that
the F-measure has a problem: it is not appropriate whenalternative sentences are possible in a human-produced extract. For example, if a human subject extracts sentence 1
and a system extracts sentence 2, the system obtains a lower
score, even when sentences 1 and 2 are interchangeable. In
this paper, we examine some of the evaluation methods devised to overcome this problem.
Several such methods have been proposed, including the utility-based evaluation proposed by Radev etal.(Radev, Jing, and Budzikowska, 2000). Utility is the importance of each sentence, as scored manually on a one-toten scale. Utility-based evaluation measures the coverageof utility scores of the human-produced extract. By giving scores for moderately important sentences that does notappear in the human-produced extract, utility-based evaluation can resolve the problem. However, the method requires
a great deal of effort for humans to assign scores manually.
Content-based evaluation(Donaway, Drummey, andMather, 2000) is another method. Instead of measuring
the ratio of sentences that match exactly in the extract, themethod evaluates computer-produced extracts by comparing their content words to those of human-produced extracts. The score for content-based evaluation is obtainedby computing the similarity between the term frequency(tf) vector of a computer-produced extract and the tf vector of a human-produced extract, using the cosine distance.
Content-based evaluation does not require much effort forhumans to make a data set for evaluation. Although theauthors reported that content-based evaluation was effective for resolving the problem, the method has not been examined in the context of comparing two extracts produced
from different systems.
In this paper, we first propose a pseudo-utility-based
evaluation that overcomes the problems of utility-based
evaluation. We can generally assume that sentences in an
extract at high compression ratios are more important thanthose at low compression ratios. Based on this assumption, we can assign an importance to each sentence in atext, when there are human-produced extracts at different
compression ratios. We can then use them for utility-based
evaluation. We think that pseudo-utility-based evaluation
is more practical than utility-based evaluation for making adata set for evaluation, because a number of data sets withextracts at different compression ratios have been made(e.g., (Jing, Barzilay, McKeown, and Elhadad, 1998)).
To evaluate the effectiveness of pseudo-utility-based
evaluation, we compare it with the F-measure using the dataof the Text Summarization Challenge (TSC)(Fukushima
and Okumura, 2001:a; Fukushima and Okumura, 2001:b),a subtask of NTCIR workshop 2, and show that pseudoutility-based evaluation can improve the F-measure.
Next, we focus on content-based evaluation. We com-pare content-based evaluation with subjective evaluation,and investigate the effectiveness of content-based evaluation for comparison of two computer-produced summaries.
In the following sections, we first briefly review some
intrinsic methods that overcome some problems of the Fmeasure. In Section 3, we propose pseudo-utility-based
evaluation. To reveal the effectiveness of pseudo-utilitybased evaluation, we evaluated computer-produced extracts
by our method and the F-measure. We report the results in
Section 4. We also report an examination of content-based
evaluation.

2. Related Work
Jing et al.(Jing, Barzilay, McKeown, and Elhadad,1998) conducted some examinations on intrinsic and extrinsic methods to investigate the factors affecting evaluation results. From their results, they found that the Fmeasure has at least the following two problems.
 Problem 1:
the F-measure is very sensitive to the compression ratio, i.e., the scores differ greatly according to the compression ratio.
 Problem 2:
the F-measure is not appropriate when alternative sentences are possible in a human-produced extract; forexample, if a human extracts sentence 1 and a system
extracts sentence 2, the system obtains a lower score,even when sentences 1 and 2 are interchangeable.
Several methods to reduce the effect of problem 1 havebeen proposed. Mittal et al.(Mittal, Kantrowitz, Goldstein,
and Carbonell, 1999) proposed that systems should be evaluated at a variety of compression ratios, and the results
should be reported in a manner similar to the 11-point precision score that is used in information retrieval.
They also pointed out that differences in inherent properties of corpora affect the results. To be able to compare
the performance of systems on different corpora, they suggested that a score of system performance should be normalized by a baseline score, which was defined to be theaverage performance of all possible extracts (randomly extracted sentences). Given a baseline score b and a score ofsystem performance p, the adjusted score is calculated by
the following equation.

 





Here, the baseline score of the F-measure is equivalent tothe compression ratio, and the F-score generally increases
when the baseline score (i.e., compression ratio) increases(see Tables 3 and 4). Mittal’s method, which adjusts thescore of system performance by the baseline score, reduces
the effect of problem 1.
Radev et al. improved Mittal’s measure (Radev, Jing,
and Budzikowska, 2000). In addition to Mittal’s proposal,
Radev et al. took account of inter-judge agreement J. Whenseveral human subjects are asked to make extracts from atext, the inter-judge agreement measures to what extent thesentences each judge extracts agree with each other. J isconsidered as an upper bound on the performance of a system. Given a baseline score R and a score of system performance S, a modified system performance S’ is calculatedusing the following equation.

  

 
 

Several methods for reducing the effect of problem 2have also been proposed. Jing et al. (Jing, Barzilay, McKeown, and Elhadad, 1998) proposed an evaluation methodthat took account of moderately important sentences that do
not appear in the human-produced extract. In this method,
the agreement between a sentence in a human-produced extract and a sentence in a computer-produced extract is represented as the degree of the human subjects’ agreement.
For example, if three of five human subjects extract sentence 1, and two subjects extract sentence 2, a system thatextracts sentence 1 will receive a score of 3/5, and a systemthat extracts sentence 2 will receive a score of 2/5, insteadof one or zero, respectively1 .
Radev et al.(Radev, Jing, and Budzikowska, 2000) proposed a utility-based evaluation. Utility is the importanceof each sentence according to a score assigned manually on
a one-to-ten scale. A utility-based score is calculated by dividing the sum of utilities of the computer-produced extractby that of the human-produced extract. By giving scores formoderately important sentences from the human-produced
extract, utility-based evaluation can resolve problem 2.
Donaway et al.(Donaway, Drummey, and Mather, 2000)
proposed two evaluation methods. In one, both human subjects and a system are asked to rank the sentences of a text
in order of their importance, and the computer-produced
extract is then evaluated by comparing the ranks of bothcomputer-produced and human-produced extracts. Thismethod classifies all sentences in the original text in termsof their importance, instead of classifying them into two
categories (important or unimportant).
Another method is content-based evaluation. Instead of
measuring the ratio of sentences that match exactly in theextracts, the method evaluates extracts by comparing theircontent words with those of human-produced extracts. The
score for content-based evaluation is obtained by computing the similarity between the term frequency (tf) vector of
a computer-produced extract and the tf vector of a humanproduced extract, using the cosine distance.
Donaway et al. conducted an examination for comparison of these two methods together with recall. They
prepared several pairs of human-produced extracts, whose
contents were highly similar, and use them for the evaluation of computer-produced extracts. The hypothesis of their
examination is good evaluation method should yield similar scores by comparing a computer-produced extract witha pair of human-produced extracts, if they are highly similar. They calculated correlation coefficients of both scoresfor each evaluation method. As a result, they concluded
that content-based evaluation was the best way to resolve
problem 2.
In Document Understanding Conference 2001,
computer-produced summaries were evaluated by comparing with human-produced summaries using the notionof model units (MUs) and peer units (PUs)(McKeown et1
If we consider sentences that more than half of the humansubjects extract as correct, sentence 1 is correct, and 2 is incorrect.

al., 2001). First, the human-produced summaries weremanually segmented into MUs, which are informational
units that should express a self-contained fact in the ideal
case. Second, computer-produced summaries were automatically segmented into PUs, which are always sentences.
Third, the assessor located the PU(s) that covered thecontent of each MU, if any. Finally, the scores of precisionwere calculated for each computer-produced summary as
the number of PUs matching some MU divided by the
number of PUs in the peer summary. The third step in thisprocedure can resolve problem 2, because a PU is locatedby an assessor, if only the PU covers the content of the
MU, though they were extracted from different parts in a
text.

Next, we compare two computer-produced extracts at
50% ratio. The F-scores of both systems are 0.6, because
three sentences of five in both computer-produced extracts
are correct. Of the three correct sentences of both systems,
S4 and S10 are common and the remaining ones are different. For their third sentences, system 1 extracts S7, andsystem 2 extracts S1. S1 is considered more important thanS7, because S1 is contained in the human-produced extract
at 10% ratio. As a result, the pseudo-utility scores of systems 1 and 2 are 0.419 and 0.806, respectively. This meansthat pseudo-utility-based evaluation can identify the difference between two computer-produced extracts.

3. Pseudo-utility-based Evaluation

To evaluate the effectiveness of pseudo-utility-based
evaluation, we conducted some tests using the data of theTSC. We also discuss content-based evaluation, which wasused as one of evaluation methods in the TSC.
In Section 4.1, we first explain the tasks and evaluationin the TSC. In Section 4.2, we report the analysis of bothmeasures based on the data of the TSC.

In this section, we propose a pseudo-utility-based evaluation that uses human-produced extracts at different compression ratios. When there are human-produced extracts
for a text at ratios of r1%, r2%, and r3% (r1  r2  r3),
we can classify each sentence in the text into the following
four categories: (a) sentences contained in the r1% extract,(b) sentences that are not contained in the r1% extract but
are contained in the r2% extract, (c) sentences that are notcontained in the r2% extract but are contained in the r3%
extract, and (d) other sentences2 . If we regard these categories as a one-to-four scale, we can use them as utilities
for pseudo-utility-based evaluation.
Now, we explain pseudo-utility-based evaluation using
an example shown in Table 1. Table 1 shows a humanproduced extract and two computer-produced extracts, allat 10%, 30%, and 50% ratios. Of the ten sentences (S1–
S10) in the original text, the extracted sentences are marked
as ‘+’ in the table. Now, we define the weight of each sentence w as 1/(compression ratio).
Of the five sentences (S3, S4, S7, S9, and S10) extracted
by system 1 at 50% ratio, three (S4, S7 and S10) are contained in the human-produced extracts. The F-score of system 1 at 50% ratio is 0.6 (3/5). Here, as the weights ofthe five sentences (S3, S4, S7, S9, S10) are 0, 1/30, 1/50,
0, 1/30, respectively, the total of the weights is 13/150 (0+ 1/30 + 1/50 + 0 + 1/30), while the total weight of thehuman-produced extract is 31/150 (1/10 +1/30 + 1/50 +1/50 + 1/30). The pseudo-utility score is calculated by
the total weight of a computer-produced extract divided by
the total weight of a human-produced extract. The pseudoutility score in this case is 0.419 (  ).
 of systems 1 and
The F-scores and pseudo-utilityscores
2 are shown in Table 2. In the table, as both systems extract S4 instead of S1, which was extracted by the humansubject, the F-score is zero. Here, as S4 is contained in ahuman-produced extract at 30% ratio, this sentence is considered as a moderately important sentence. In this example, the only possible score of the F-measure at 10% is zeroor one, while pseudo-utility-based evaluation makes it possible to evaluate extracts appropriately by taking account ofmoderately important sentences such as S4.
Here, an extract at  % must be contained in an extract at
! %, ( #" ! ), and an extract at ! % must be contained in an
extract at %$ % (! " %$ ).
2

4. Analysis of Evaluation Methods

4.1. Evaluation in the TSC
The Text Summarization Challenge (TSC) is a subtaskof NTCIR Workshop 2, which was held so that researchers
in the field could collect and share text data for summarization, and to make clear the issues of evaluation measuresand methods for summarization of Japanese texts. Threetasks were conducted in the TSC, and we describe two ofthem, task A-1 and A-2, as their evaluation uses an intrinsic
method (For further detail, please refer to (Fukushima andOkumura, 2001:a; Fukushima and Okumura, 2001:b)).

& Task A-1 (Extraction of important sentences):to extract important sentences at 10%, 30%, and 50%
ratios. Extracts were evaluated by F-measure.

& Task A-2 (Summaries to be compared with humanproduced summaries):to produce summaries in plain text at the ratios of 20%
and 40%. Summaries were evaluated in two ways:
content-based evaluation and subjective evaluation. In
subjective evaluation, human judges were asked toevaluate and rank the computer-produced summaries
in terms of coverage of important contents and readability. Judges were given four types of summaries:
two human-produced summaries, a system result, andan extract produced by a lead-based method.
4.1.1. Texts
Thirty newspaper articles were extracted from theMainichi newspaper database for 1994 and 1998. In termsof genre, editorials and articles on social issues were used.
The editorials were grouped into two sets of about 1200
and 2400 characters in length, while the social issue articles were grouped into three sets with lengths about 600,
900 and 1200 or more characters.
4.1.2. Evaluation methods for each task
Task A-1
For task A-1, recall, precision, and F-measures wereused, where

S1
S2
S3
S4
S5
S6
S7
S8
S9
S10

Table 1: An example of pseudo-utility-based evaluation
Human subject
Importance
System 1
System 2
10% 30% 50%
w
10% 30% 50% 10% 30% 50%
+
+
+
1/10
+
+
0
0
+
+
+
1/30
+
+
+
+
+
+
0
0
+
+
+
1/50
+
+
1/50
0
+
+
+
+
+
1/30
+
+
+

Table 2: An example of evaluation by F-measure and pseudo-utility-based measure
F-measure
10%

0.000

')(*,+

30%

0.667'0/1+
/5

50%

0.600

' +

System 1
pseudo-utility evaluation*.-/
0.333 ' *.-* ( +( -/0.400 ' *.-* 0( -/ + 5
(3-24/ 0 ( *- 5(3-2 / ( - +0.419 ' *.-* 0(3240 (6240 (8 Recall = the number of correct sentences marked by
the system / the total number of correct sentences
marked by the human subject

8 Precision = the number of correct sentences marked
by the system / the total number of sentences marked
by the system

8 F-measure = 2 9 Recall 9 Precision / (Recall + Precision)
After calculating these scores for each article, we computed the average of them. We also evaluated the results of
two baseline systems. One was based on the lead method(Lead), and the other was based on term frequency (tf).
Task A-2(i) Subjective evaluation
The following four kinds of summaries as well as theoriginal texts were prepared.

8 Summaries produced by extracting important parts ofthe sentences in the text (PART)

8 Freely summarized texts (FREE)

8 Summaries produced by a system (SYS)

8 Summaries produced by using the lead method(BASE)First, the evaluator (one person) read the original text
and its summaries (4 kinds). Then, the person evaluatedand scored the summaries in terms of how readable theywere, and how well the content of the text was described.
The scores were 1, 2, 3, or 4, where 1 is the best and 4 is
the worst, i.e., a lower score indicates a better evaluation.
(ii) Content-based evaluation
Like Donaway et al.(Donaway, Drummey, and Mather,
2000), we tried to find out how close the two summaries
were by examining the content words. Morphological analysis was applied to the system and human summaries, andF-measure0.000'(*,+

0.667'.0/1+
/5

0.600

' +

System 2
pseudo-utility evaluation*.-/' *.-* ( +*.-* ( *.-/
0.800 ' *.-* (32-/ ( +
(3*-24* 0 ( -/ 5(6-27/ 0 ( - +0.806 ' *.-*(3240 (6240 (0.333

only content words (keitaiso) were selected for both ofthem. The distances between the word-frequency vector ofsystem- and human-produced summaries were then computed. We used both PART and FREE as keys.
4.2.

Analysis of Evaluation Methods

To show the effectiveness of pseudo-utility-based evaluation, two examinations are necessary.
One is acomparison between pseudo-utility-based evaluation andthe F-measure, and the other is a comparison between
utility-based evaluation and pseudo-utility-based evaluation. However, as it requires a great deal of effort for humans to make a data set for utility-based evaluation, we did
not conduct the latter examination.
In this section, we first compare pseudo-utility-based
evaluation with the F-measure, and the results are reported
in Section 4.2.1. We also conducted an examination oncontent-based evaluation, and the results are reported in
Section 4.2.2.
4.2.1. Comparison of pseudo-utility-based evaluation
and F-measure (task A-1)
First, we investigated the effectiveness of pseudoutility-based evaluation by comparing human-produced extracts with computer-produced extracts. Figure 1 shows atypical example in which pseudo-utility was applied effectively. The theme of the original newspaper article is ‘theinfection of AIDS in Asia’. Both a human-produced extract and a computer-produced extract at 10% ratio (the task
was to extract only one sentence from article 940702171)
are shown in the figure. As can be seen from the figure,the extracts (sentences) do not match each other, so the Fscore is zero. Both sentences, however, describe almost thesame topic, ‘the number of patients has been increasing recently in Asia’. Now, as the computer-produced extract is
contained in the human-produced*.- / extracts at 30% ratio, thepseudo-utility score is 0.333 ( *.- (;: * ), which seems more ap-(;:

Figure 1: An example showing how pseudo-utility evaluation was applied effectively (1)propriate than the F-score of zero.
Newspaper articles in the TSC generally consist of 10
to 20 sentences. The human-produced extract at 10% ratio therefore consists of only one or two sentences. In
this case, even though a system can extract a moderately
important sentence, it will not be reflected in the F-score.
Pseudo-utility-based evaluation, however, takes account ofsuch sentences, so more appropriate evaluation is possible
in cases such as the example shown in Figure 1.
Another example is shown in Figure 2. The numberof sentences extracted at 10% ratio from article 940715208
was three. In this example, the F-score is 0.333, because thefirst sentence of the three in the computer-produced extract
is contained in the human-produced extract. Here, one ofthe other two sentences in the computer-produced extract is
contained in the human-produced extracts at 30% ratio, andanother one is in 50%. As a result, the pseudo-utility scoreis 0.511 ( <.=>;? <@A<=.>;? B.@C<=.>;? D ). Comparing the computer>;? < the human-produced extract, senproduced extractB=.withtences (2) and (3) in the computer-produced extract are examples of the description ‘universities, together with othereducational organizations, are attempting to devise a countermeasure’ in sentence (2) in the human-produced extract.
From these results, pseudo-utility-based evaluation is effective in assigning scores when the system extracts moderately important sentences.
From the result of this analysis, we can conclude thatpseudo-utility-based evaluation can reduce the effect ofproblem 2, as pointed out by Jing et al..
We then calculated the average scores for pseudoutility-based evaluation and F-measure for each system.
The results are shown in Table 3 for F-measure and 4 for

Figure 2: An example showing how pseudo-utility evaluation was applied effectively (2)pseudo-utility-based evaluation3 . Ten systems from sevengroups attempted task A-1. The system IDs are shown fromI to IX in the tables. Different systems from the same groupare shown by dash (VII’ and IX’).
Systems I and II are ranked first and second in Table 3,
but second and first in Table 4. The ranks of many othersystems also changed; in particular the rank of system Vchanged from 9 by F-measure to 5 by pseudo-utility-based
evaluation. To investigate the reliability of these ranks bypseudo-utility-based evaluation, we focused on systems I
and II, and compared the extracts of both systems.
Among 90 pairs (30 texts E three ratios(10%, 30%,50%)) of computer-produced extracts, we chose 16 pairs
with the same F-scores and different pseudo-utility scores.
Among the 16 pairs, the pseudo-utility scores of system
I are larger than those of system II in 10 cases. A typical example of these cases is shown in Table 5. We show
several sentences extracted by systems I and II at the 10%
ratio from article 980500136 and the weights of each sen3

We eliminated four articles (940701189, 940702187,
940716331, 980203053) from this examination, as they did notmeet the condition in footnote 2.

Table 3: F-score of each system
SYSTEM
I
II
V
VI
VI’
VII
VII’
VIII
IX
IX’
TF
Lead
Ave F

10%
0.363
0.337
0.251
0.305
0.282
0.305
0.241
0.199
0.358
0.268
0.284
0.276
0.289

30%
0.435
0.452
0.447
0.431
0.435
0.474
0.497
0.399
0.420
0.409
0.433
0.367
0.433

50%
0.589
0.612
0.574
0.568
0.572
0.586
0.578
0.590
0.571
0.570
0.586
0.530
0.577

total (rank)
0.463 (2)0.467 (1)0.424 (9)0.435 (6)0.429 (8)0.455 (3)0.439 (5)0.396 (11)0.450 (4)0.416 (10)0.434 (7)0.391 (12)0.433Table 4: Pseudo-utility score of each system
SYSTEM
I
II
V
VI
VI’
VII
VII’
VIII
IX
IX’
TF
Lead
Ave F

10%
0.518
0.450
0.410
0.444
0.420
0.433
0.401
0.330
0.463
0.388
0.406
0.401
0.422

30%
0.559
0.603
0.546
0.537
0.516
0.560
0.556
0.515
0.544
0.509
0.526
0.481
0.537

50%
0.664
0.673
0.641
0.608
0.607
0.651
0.636
0.654
0.616
0.612
0.657
0.549
0.630

total (rank)
0.581 (1)0.569 (2)0.527 (5)0.521 (8)0.504 (9)0.541 (3)0.525 (6)0.495 (11)0.535 (4)0.498 (10)0.525 (6)0.468 (12)0.530

tence used for pseudo-utility-based evaluation. Here, all
the sentences, which have weights of 1/10, show a humanproduced extract at 10% ratio. Among the five sentences
extracted by system I, two sentences (S44 and S54) were
contained in the human-produced extract, so the F-score is0.4. System I also extracted S30, which occurs in humanproduced extracts at 30%, and both S3 and S4, which occur
in human-produced extracts at 50%, so the pseudo-utility
score is 0.547.
System II extracted S26 and S43, so the F-score is 0.4(the score is same as system I). Among the other three sentences extracted by system II, S3 and S4 are common tosystem I and the weight of the other sentence (S31) is zero.
Consequently, the pseudo-utility score of system II is 0.480,
which is lower than that of system I (Table 6).
S22 brings up an important question that is a main
theme of this article. S22 is contained in a human-produced
extract at 10%, though neither system extracted it. However, system I extracted S50, which is a solution to the question posed in S22. This sentence is actually contained in ahuman-produced extract at 30%. It is therefore appropriatethat pseudo-utility-based evaluation differentiates systems
I and II, because of the extraction of S50 by system I.
4.2.2. Comparison of content-based evaluation andsubjective evaluation (task A-2)
First, we describe the results of subjective evaluationand content-based evaluation in the TSC. Second, we compare them and discuss the effectiveness of content-based

Table 8: Percentage of cases where the order of contentbased scores of two summaries matched the order of theirranks by subjective evaluation (whole data)

20% summary
40% summary

FREE-based
91.4% (1371/1500)
89.3% (1339/1500)
Average: 89.8%

PART-based
88.6% (1329/1500)
90.0% (1350/1500)

Table 9: Percentage of cases where the order of contentbased scores of two summaries matched the order of theirranks by subjective evaluation (SYS vs BASE)

20% summary
40% summary

FREE-based
PART-based
64.3% (193/300) 58.0% (174/300)
58.7% (176/300) 63.7% (191/300)
Average: 61.2%

evaluation.
Results of subjective evaluation
Table 7 shows the ratios for four types of summaries(FREE, PART, SYS, BASE) ranked first, second, third andfourth, in terms of coverage of important contents (CONT)
and readability (READ)4 . As can be seen from the table,
FREE is higher than the others (73.5%) in the number ofcases that it ranked first, with PART, SYS, and BASE following. However, the difference between SYS and BASE
is very small compared with that between FREE and PART.
The quality of the four types of summaries can be ordered
as follows:(1)FREE G (2)PART G (3)SYS and BASE
Comparison of subjective evaluation and content-based
evaluation
We compared the results of subjective evaluation andcontent-based evaluation. We calculated the percentage ofcases where the order of content-based scores of two summaries matched the order of their ranks by subjective evaluation. The possible combinations of the four types of summaries are ‘FREE-PART’, ‘FREE-SYS’, ‘FREE-BASE’,‘PART-SYS’, ‘PART-BASE’, and ‘SYS-BASE’. Amongthem, we eliminate ‘FREE-PART’ from our examination,
because both FREE and PART are used as keys for evaluation in the TSC.
The results are shown in Table 8. As can be seen fromthe table, the evaluation by content-based measure matched
the subjective evaluation in 90% of cases at compression
ratios of both 20% and 40%.
As described above, the difference between SYS and
BASE is very small. We therefore focused on them and calculated the percentage of cases where the order of contentbased scores of the two summaries matched the order of
their ranks by subjective evaluation. The results are shownin Table 9. We can conclude that the content-based measurecannot detect the slight difference of quality between twosummaries, such as SYS and BASE. However, it is reliable
at detecting larger differences between two summaries.
To investigate the reliability of content-based evaluation, we calculated the percentage of cases where the order4

two ratios (20% and 40%) H 30 texts H 10 systems = 600.

Table 5: Extracts produced by computer systems I and II (10% ratio)

Table 6: F-scores and pseudo-utility scores of systems I and II (article 980511036 at 10% ratio)I
II
F-measure0.400 0.400
pseudo-utility 0.547 0.480
of content-based scores of two summaries matched the order of their ranks by subjective evaluation, using gaps in thecontent-based score from zero to one at 0.1 intervals. The
results are shown in Table 10. We found that the evaluationby content-based measure matched the subjective evaluation in 93% of cases, if the gap in the content-based scores
between two summaries was more than 0.2.
We examined the cases when the gap in the contentbased scores between SYS and BASE was more than 0.2
in Table 9. The results are shown in Table 11. The caseswith gaps larger than 0.2 make up only 14.5% between SYS
and BASE, while 59.5% in Table 105 . From these results,
we reconfirm that the difference between SYS and BASE issmall.
5

The number of cases with gaps smaller than 0.2 is2430(1242(0.0–0.1)+1188(0.1–0.2)) among 6000 cases. So, theratio that the gaps is larger than 0.2 is 0.595 ( IKJMLONQP)VSN WSR6WSLSW LUTST ).

Next, we calculated the percentage of cases where theorder of content-based scores of two summaries matchedthe order of their ranks by subjective evaluation, when the
gap was more than 0.2. The result is shown in Table 12.
As can be seen from the table, the reliability of contentbased evaluation increases by more than 10%. The ratio of
71.3%, however, is still lower than the 92.8% in Table 10.

5. Conclusion and Future Research
In this paper, we first proposed pseudo-utility-based
evaluation, and conducted an examination to investigate
the effectiveness of our method. We evaluated computerproduced extracts by the F-measure and by pseudo-utilitybased evaluation, and found that pseudo-utility-based evaluation could diminish the problems of the F-measure.
We also focused on content-based evaluation. We
conducted tests in the context of comparing two summaries produced from different systems. We evaluatedTable 7: The ratios of four types of summaries ranked to first, second, third and fourth

FREE

CONT
READ
TOTAL
CONT
READ
TOTAL
CONT
READ
TOTAL
CONT
READ
TOTAL

PART

SYS

BASE

1st
69.8% (419/600)
77.7% (466/600)
73.5% (885/1200)
49.0% (294/600)
40.6% (244/600)
44.8% (538/1200)
2.3% (14/600)
11.2% (67/600)
6.6% (79/1200)
0.0% (0/600)
6.5% (39/600)
3.2% (39/1200)

2nd28.7% (172/600)
19.0% (114/600)
23.8% (286/1200)
49.0% (294/600)
47.5% (285/600)
48.3% (579/1200)
3.3% (20/600)
10.3% (62/600)
6.8% (82/1200)
0.8% (5/600)
8.0% (48/600)
4.4% (53/1200)

Table 10: Effectiveness of content-based measure
Gap between
content-based scores
0.0 - 0.1
0.1 - 0.2
0.2 - 0.3
0.3 - 0.4
0.4 - 0.5
0.5 - 0.6
0.6 - 0.7
0.7 - 0.8
0.8 - 0.9
0.9 - 1.0

Percentage of cases in which
content-based evaluation matchedwith subjective evaluation (%)
57.8 (718/1242)
77.1 (916/1188)
92.8 (966/1041)
95.9 (805/839)
96.4 (588/610)
98.8 (589/596)
99.4 (336/338)
99.0 (103/104)
100.0 (26/26)
100.0 (16/16)

Table 11: Percentage of cases when the gap between twosummaries is more than 0.2 (SYS vs BASE)
FREE-based
PART-based
20% summary 17.0% (51/300) 23.0% (69/300)
40% summary 10.0% (30/300) 8.0% (24/300)
Average: 14.5%
Table 12: Percentage of cases where the order of contentbased scores of two summaries matched the order of theirranks by subjective evaluation and the gap between twosummaries is more than 0.2 (SYS vs BASE)

20% summary
40% summary

FREE-based
PART-based
74.5% (38/51) 73.9% (51/69)
60.0% (18/30) 70.8% (17/24)
Average: 71.3%

computer-produced summaries by content-based evaluation, and compared the results with a subjective evaluation. We found that the evaluation by content-based measure matched the subjective evaluation in 93% of cases, if
the gap in the content-based scores between two summaries
was more than 0.2.
We showed that human-produced extracts at different
compression ratios could be used for utility-based evaluation. Although we used human-produced extracts at 10%,30%, and 50% ratios for evaluation, we should look for optimal combinations of compression ratios.

3rd
1.5% (9/600)
3.2% (19/600)
2.3% (28/1200)
1.8% (11/600)
8.5% (51/600)
5.3% (64/1200)
68.0% (408/600)
43.3% (260/600)
55.7% (668/1200)
38.2% (229/600)
52.7% (316/600)
45.4% (545/1200)

4th
0.0% (0/600)
0.2% (1/600)
0.1% (1/1200)
0.2% (1/600)
3.0% (18/600)
1.6% (19/1200)
26.3% (158/600)
38.8% (233/600)
32.6% (391/1200)
61.0% (366/600)
32.8% (197/600)
46.9% (563/1200)

In this paper, we used ‘1/compression ratio’ as theweights for sentences for pseudo-utility-based evaluation.
In future, we should also study the optimal weights of eachsentence used for pseudo-utility-based evaluation.

6.

References

Donaway, R.L., Drummey, K.W., and Mather, L.A. 2000.
A Comparison of Rankings Produced by Summarization
Evaluation Measures. Proceedings of the ANLP/NAACL
2000 Workshop on Automatic Summarization, pages 69–
78.
Fukushima, T. and Okumura, M. 2001. Text Summarization Challenge Text Summarization Evaluation at NTCIR Workshop2. Proceedings of the Second NTCIR
Workshop Meeting, pages 45–51.
Fukushima, T. and Okumura, M. 2001. Text Summarization Challenge Text Summarization Evaluation in Japan.
Proceedings of NAACL 2001 Workshop Automatic Summarization, pages 51–59.
Jing, H., Barzilay, R., McKeown, K., and Elhadad, M.
1998. Summarization Evaluation Methods: Experiments
and Analysis. Technical Report SS-98-06, Intelligent
Text Summarization, AAAI Press, pages 51–59.
Mani, I. 2001. Automatic Summarization. John Benjamins
Pub Co.
McKeown, K.R., Barzilay, R., Evans, D., Hatzivassiloglou,
V., Kan, M. Y., Schiffman, B. and Teufel, S. 2001.
Columbia Multi-Document Summarization: Approachand Evaluation. Proceedings of Document Understanding Conference 2001, pages 43–63.
Mittal, V., Kantrowitz, M., Goldstein, J., and Carbonell, J.
1999. Selecting Text Spans for Document Summaries:
Heuristics and Metrics. Proceedings of the 16th National
Conference on Artificial Intelligence, pages 467–473.
Radev, D.R., Jing, H., and Budzikowska, M. 2000.
Centroid-based Summarization of Multiple Documents:
Sentence Extraction, Utility-based Evaluation, and User
Studies. Proceedings of the ANLP/NAACL2000 Workshop on Automatic Summarization, pages 21–29.
Sparck-Jones, K. and Galliers, J. 1996. Evaluating Natural Language Processing Systems: An Analysis andReview. Lecture Notes in Artificial Intelligence 1083,
Berlin: Springer.