Text Summarization Challenge
Text summarization evaluation at NTCIR Workshop2

Takahiro Fukusima
Otemon Gakuin University
fukusima@res.otemon.ac.jp

Abstract
We describe the outline of Text Summarization
Challenge (TSC hereafter), a text summarization
evaluation conducted as one of the tasks at theNTCIR Workshop2.
First, we introduce TSC
explaining its background and purpose. Then we
describe briefly types of summarization andsummarization evaluation methods in general. Next,
we focus on TSC, including the participants, the threetasks in TSC, data used, evaluation methods for eachtask, brief report on the results as well as the featuresof the Challenge. The future directions for TSC

are mentioned at the end as conclusion.
Keywords:evaluation

1text

summarization,summarization

Introduction

As research on automatic text summarization isbecoming a hot topic in NLP, we also see the needs todiscuss and clarify the issues on how to evaluate text
summarization systems. SUMMAC in May 1998 as apart of TIPSTER (Phase III) project ([1], [2]) andon-going TIDES program shows the need andimportance of the evaluation for text summarization.
In Japan, there has been a lot of research on
automatic text summarization. However, since theevaluation of such systems was done individuallywith their own evaluation measures at universities
and industrial research organizations, and there havebeen few discussions about evaluation measures,
which makes it difficult to compare text
summarization systems. Moreover, we did not haveenough language resources such as human-preparedsummaries.
Thus, we have chosen automatic text
summarization as an NTCIR-2 task in order for theManabu Okumura
Tokyo Institute of Technology
oku@pi.titech.ac.jp

researchers in the field to collect and share text data
for summarization, and to make clear the issues ofevaluation measures for summarization of Japanese
texts. Before we started TSC formally, we hadchosen 33 organizing committee members and hold
two preliminary meetings in 1999.
We alsoinitiated a mailing list specifically for discussing
issues related to TSC. At the same time, we have
set up a website for TSC written in Japanese andEnglish at http://galaga.jaist.ac.jp:8000/tsc/. What
we discussed at the two meetings and in the mailinglist has been archived and can be seen in the website.

2Types of summarization

We would like to introduce briefly types of text
summarization, and summarization evaluation
methods in this and the following sections.
Text summarization is a task of producing a shorter
text from the source, while keeping the information
content in the source, and summaries are the resultsof such task.
There are several ways to classify summaries. The
following threes factors are considered to beimportant for text summarization research ([3]).
Input factors: text length, genre, single vs.
multiple documents
Purpose factors: who is the user, the purpose ofsummarization
Output factors: running text or headed text etc.
Summaries can be classified with respect to thenumber of the source text (single text vs. multiple
texts summarization), and as to whether they are
tailored to particular users. If they are targeted forspecific users, they may be called “user-focused”,
and if they are intended to users in general, suchsummaries are “generic”.
In terms of summarization purposes, summaries
can be “indicative” or “informative”. Users canmake use of indicative summaries before referring to
the source, e.g. to judge relevance of the source text.
On the other hand, users may use summaries in placeof the source text (informative summaries).
Summaries can also be classified into extracts andabstracts in terms of how they are composed.
Conventional text summarization systems produce
summaries by using sentences or paragraphs as basic
unit, giving them degree of importance, sorting thembased on the importance, and gathering the importantsentences.
In short, summaries are a set ofimportant sentences extracted from the source text
and called “extracts”.
In contrast, summaries may contain newly
produced text, and they are called “abstracts” ([4]).

3

Evaluation methods

It is said that evaluation methods for text
summarization can be largely divided into two
categories: intrinsic and extrinsic.
The quality of summaries is judged directly withsome norms, typically “ideal” summaries producedby hand, or important sentences selected by hand
(intrinsic evaluation) ([5], [6]).
The quality of summaries can also be judged bymeasuring how it influences the achievement of someother task (extrinsic evaluation). Such tasks can bequestion-answering, comprehension task, as well as
relevance judgement of a document to a topic ([2]).
In TSC, we have conducted evaluations for text
summarization of varied lengths, varied genres, andsingle-document using both intrinsic and extrinsicmethods.

4Participants

We had 9 participating systems for Task A-1 andA-2, and 7 systems for Task B at the Dryrun. We
have 10 participating systems for Task A-1, 9systems for Task A-2, and 9 systems for Task B at theFormal run. As group, we had 9 participatinggroups that are all Japanese, from universities,
governmental research institute or companies inJapan. Table 1 shows the breakdown of the groups.
University
Governmental
research instituteCompany

315

Table 1 Breakdown of Participants(Please note that one group consists of a company
and a university.)

5Three Tasks
ScheduleinTSC

andits

TSC has three tasks. First two tasks are forintrinsic evaluation (called Task A) and they aresummarization tasks where participants are given
texts to be summarized automatically andsummarization rates (or lengths of summary). They
should submit the results of their summarization
system according to the summarization rates.
There are two types of summarizing tasks in Task
A.
Task A-1 is to extract important sentences.
Summarization rate is given as a ratio between thenumber of chosen sentences and the total number ofthe sentences in the article. The rates are 10%, 30%,and 50%.
Task A-2 is to produce summaries in plain text tobe compared with human-prepared summaries.
Summarization rate is a rate between the number ofcharacters in the summary and the total number ofcharacters in the original article. The rates are about
20% and 40%.
The third and final task is for extrinsic evaluation,
called here Task B where summaries are evaluated byconducting IR task. Given queries and retrieved
documents based on the queries, participants submit
summaries. The length of the summaries is not
limited, however, the summaries should be in plain
text. They should make one summary for eachdocument (not a summary from multiple documents).
The retrieved documents may include irrelevant
documents to the queries.
The schedule of evaluations at TSC is as follows:
the Dryrun was conducted in September 2000 and theFormal run was in December 2000. The final
evaluation results have been reported to theparticipants by the end of December 2000.

6Data Used for TSC

We use newspaper articles from the Mainichi
newspaper database of 1994, 1995, 1998. As key data(human prepared summaries), we prepare thefollowing types of summaries.
Extract-type summaries:
We asked captioners who are well experienced insummarization to select important sentences fromeach article. The summarization rates are 10%,30%, and 50%. These summaries are used for
Task A-1.
Abstract-type summaries:
We ask the captioners to summarize the originalarticles in two ways. The first is to choose
important parts of the sentences recognized
important in extract-type summaries (abstract-typetype1). The second is to summarize the originalarticles “freely” without worrying about sentenceboundaries, trying to obtain the main ideas of thearticles (abstract-type type2). Both types ofabstract-type summaries are used for Task A-2.
In terms of genre, we used editorials and articles
on social issues for the formal run evaluation. The
lengths are grouped into two kinds for the editorials:
about 1200 and 2400 characters, for social issue
articles, the lengths are three kinds: about 600, 900,
and 1200 or more characters.

7

Evaluation Methods for each task

We use summaries prepared by human as key
data for evaluation. We would like to describe
evaluation methods for each task below.

7.1

Evaluation methods for Task A

For Task A-1, we use recall, precision, and F
measures where:
Recall = the number of correct sentences marked by
the system / the total number of correctsentences marked by humanPrecision = the number of correct sentences marked
by the system / the total number of sentences
marked by the system
F-measures = 2 x Recall x Precision /(Recall + Precision)
After calculating these scores for each article, we
compute the average of them and make it its finalscore. We also evaluate the results of two baseline
systems. One is based on lead method (lead), andthe other based on term frequency (tf).
For Task A-2, the evaluation is not as simple as
A-1 above. The system results are compared withhuman-prepared summaries in two ways (A-2-1 andA-2-2) as explained below.
A-2-1 (content-based evaluation)

We try to find out how close the two summaries areby examining the content words ([7]). Morphologicalanalysis is done to the system results and humansummaries, and only content words (keitaiso) areselected. Then, the distance between theword-frequency vector of human summary andsystem result is computed.
We use both abstract-type summaries (abstract-type
type 1 and typ e 2) as key.
A-2-2 (subjective evaluation)
We ask human judges, who are experienced in
producing summaries, to evaluate and rank thesystem summaries in terms of two points of views.
- How much the system summary covers theimportant content of the original article.
- How readable the system summary is.
The judges are given 4 types of summaries to beevaluated and ranked in 1 to 4 scale (1 is the best, 2for the second, 3 for the third best, and 4 for theworst). The first two types are human-produced
abstract-type type1 and type2 summaries. The thirdis system result, and the fourth is summaries
produced by lead-based method.
7.2

Evaluation method for Task B

For Task B, the evaluation is an extrinsic one
using information retrieval task. Human subjects aregiven queries and summaries of the retrieveddocuments. They read the summaries and judge howrelevant the documents are. The evaluation method isbasically the same as SUMMAC ([2], [8]). The
measures for evaluation are recall, precision andF-measures as well as the time to indicate how long it
takes to carry out the task.
Recall = the number of documents judged relevantcorrectly by human subjects /the total number of relevant documents
Precision = the number of documents judged relevantcorrectly by human subjects / the total number ofdocuments judged relevant by subjectsF-measures = 2 x Recall x Precision /(Recall + Precision)

8Results

Here are the results for Task A-1 for selecting
important sentences, and summarization rates are10%, 30%, and 50%. Since recall and precision

scores turned out to be the same for the all systems,
F-measures scores are used in Figure 1.
0.7

For Task A-2 with subjective evaluation, we
had the following results. In Figure4, digits standfor summarization rates (20%, 40%), and R for
readability, and C for content information.

0.5
4.00

0.4
0.3
10%
30%
50%

0.20.10TF

Lead

10

98

76543

2

1System

3.50
Ranking score

F-measures

0.6

3.00
2.50

20R
20C
40R
40C

2.00
1.50
1.00

Figure 1 Evaluation Results for Task A-1(F-measures for 10, 30, 50%)
The next is the results of Task A-2. Next, here
are results for Task A-2 with content-basedevaluation.
First, we show the results forimportant-part summaries (abstract type 1), and forfree summaries (abstract type 2)

123

45 6System

78

9TF

Figure 4 Evaluation Results for Task A-2-2
For Task B, we have the following results.
First, the result for level-A and B documents
10.7
0.6
0.5
0.4
0.3
0.2
0.1
00.8
0.6
20%
40%

0.4

RecallPrecision

0.2
00.5
0.4
0.3
0.2

20%

0.1

40%

0TF

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

RecallPrecision
Lead

TF

Full text

9876

54

3

System2

Figure 3 Evaluation Result for Task A-2-1(abstract-type 2)

Figure 5 Task B results (level A documents)

1

Lead

987654321System

Lead

0.6

TFFigure 2 Evaluation Results for Task A-2-1(abstract-type 1)

Full text

98

7

65

4

3

21

TF

Lead

98

7

6543

2

1

SystemSystem

Figure 6 Task B results (level B documents)

And here is the result of time to finish the task.
The scores are average time for conducting the IRtask for 50 texts (one query) in minutes.
System1
2
3
4
5
6
7
8
9
Full text
TF
Leadminutes
9:41
12:48
6:25
6:44
8:33
9:01
10:16
9:16
9:31
13:46
8:44
7:32

IVV

VIVIITable 2 Time needed for Task B
Though the results are computed as scores, we do
not consider they are definite. Described below as
one of the features of TSC, we would like to focus onround-table evaluation at the workshop, which weplan to include in the final paper of the workshop.

9VIII

Uses similarity information among
original documents by hierarchicalclustering and information gain ratio forTask B.
The system uses a hybrid method ofterm-frequency based method and leadsentence extraction.
The approach is based on passage
importance with Hanning window(density of words) for Task A-1 andTask B. For Task A-2, the system
makes use of dependency structure
analysis, TF-IDF and similarity to leadsentences.
The system is a combination of theoriginal system and lead-based method.
The original system computes scoresfor ranking sentences and paragraphs
based on relevance degrees between
two sentences or paragraphs.
The approach utilizes keyword-based
sentence extraction algorithm andthematic hierarchy based sentence
extraction algorithm.
Table 3 System Technologies

Participant system technologies

The following table shows brief descriptions of theparticipating systems. It is based on the eight
papers submitted to the workshop by eachparticipating group. Please note that system numberin the next table does not correspond to those in theother tables and figures in the paper or in theevaluation result section.

SystemI

II

III

Reported methods and features
The system uses a scoring function
integrating sentence location, sentence
length, TF-IDF, similarity to the title.
The system uses word frequency,
location, and content of the sentence(opinion or not) for extract typesummaries. The system for abstract
type summaries deletes multiple
modifiers
andillustration
withparaphrasing technique.
Key concept is phrase-representedsummarization.
It selects core
relations among words, and constructs
phrases as summary

10 Features of TSC evaluation
For Task A, we use several summarization rates([9]), and prepared various lengths and genres of thetexts and used them for evaluations. The lengths
vary from 600, 900, 1200 and 2400 characters, andthe genres include business news, social issues aswell as editorials.
As for Task A-2, since it is difficult to haveintrinsic evaluation for informative summaries, wewill hold “round-table evaluation” where we justpresent the results of content-based and subjective
evaluations, and offer all the participants theopportunity to discuss issues, including how toevaluate summaries, at NTCIR workshop 2.

11 Conclusions
We have described the outline of the Text
Summarization Challenge. We are planning tocontinue our efforts for text summarization
evaluation as TSC2. Though the full details are notyet decided, we would like to continue to produce
human summaries of the same kind with variedlengths and genres of newspaper articles, and we liketo conduct multiple-document evaluation in thesecond Challenge.

Acknowledgements
We would like to thank all the members of theorganizing committee, and to give special thanks to
all the people who helped us with the evaluations,and Dr. Kiyoaki Shirai at Tokyo Institute ofTechnology who is not an organizing member,
however, helped us much.

References[1] Proceedings of The Tipster Text Program Phase
III, Morgan Kaufmann, 1999.
[2] Mani, I., et al. The TIPSTER SUMMAC Text
Summarization Evaluation, Technical Report, MTR
98W0000138 The MITRE Corp. 1998.
[3] Sparck-Jones, K. “Automatic Summarizing:Factors and Directions” in Mani, I. And Maybury, M.,
editors, Advances in Automatic Text Summarization,
pp.1-12. MIT Press, 1999.
[4] Mani, I. And Maybury, M., editors, Advances inAutomatic Text Summarization, MIT Press, 1999.
[5] Paice, C. Constructing literature abstracts bycomputer: Techniques and prospects. Information
Processing Management, 26(1) 1990.
[6] Edmundson, H.P. New methods in automaticabstracting. The Association for ComputationalMachinery, 16(2) 1969.
[7] Donaway, R.L., Drummey, K.W. and Mather,L.A, "A Comparison of Rankings Produced bySummarization Evaluation Measures", Proc. of theANLP/NAACL2000 Workshop on Automatic
Summarization. Pp. 69-78.
[8] http://www.itl.nist.gov/div894/894.02/
related_projects/tipster_summac/index.html[9] Jing, H. and Barzilay, R. and McKeown, K. andElhadad, M, "Summarization Evaluation Methods:Experiments and Analysis", pp. 51-59 in Intelligent
Text Summarization Technical Report SS-98-06,
AAAI Press 1998.