Corpus and Evaluation Measures for Multiple Document Summarization
with Multiple Sources
Tsutomu HIRAONTT Communication Science Laboratories
hirao@cslab.kecl.ntt.co.jp

Takahiro FUKUSIMA
Otemon Gakuin University
fukusima@res.otemon.ac.jp

Manabu OKUMURA
Tokyo Institute of Technology
oku@pi.titech.ac.jp

Chikashi NOBATA
Communication Research Laboratories
nova@crl.go.jp

Hidetsugu NANBA
Hiroshima City University
nanba@its.hiroshima-cu.ac.jp
Abstract
In this paper, we introduce a large-scale test collection for multiple document summarization, the Text
Summarization Challenge 3 (TSC3) corpus. We
detail the corpus construction and evaluation measures. The significant feature of the corpus is that it
annotates not only the important sentences in a document set, but also those among them that have thesame content. Moreover, we define new evaluation
metrics taking redundancy into account and discuss
the effectiveness of redundancy minimization.

1

Introduction

It has been said that we have too much information on our hands, forcing us to read through a great
number of documents and extract relevant information from them. With a view to coping with this situation, research on automatic text summarization has
attracted a lot of attention recently and there havebeen many studies in this field. There is a particularneed to establish methods for the automatic summarization of multiple documents rather than single
documents.
There have been several evaluation workshops
on text summarization. In 1998, TIPSTER SUMMAC (Mani et al., 2002) took place and the Document Understanding Conference (DUC)1 has been
held annually since 2001. DUC has included multiple document summarization among its tasks since
the first conference. The Text Summarization Challenge (TSC)2 has been held once in one and a halfyears as part of the NTCIR (NII-NACSIS Test Collection for IR Systems) project since 2001. Multiple
document summarization was included for the firsttime as one of the tasks at TSC2 (in 2002) (Okumura
et al., 2003). Multiple document summarization is
now a central issue for text summarization research.
12

http://duc.nist.gov
http://www.lr.pi.titech.ac.jp/tsc

In this paper, we detail the corpus constructionand evaluation measures used at the Text Summarization Challenge 3 (TSC3 hereafter), where multiple document summarization is the main issue. We
also report the results of a preliminary experimenton simple multiple document summarization systems.

2 TSC3 Corpus
2.1 Guidelines for Corpus Construction
Multiple document summarization from multiple
sources, i.e., several newspapers concerned with thesame topic but with different publishers, is more difficult than single document summarization since it
must deal with more text (in terms of numbers ofcharacters and sentences). Moreover, it is peculiar
to multiple document summarization that the summarization system must decide how much redundant information should be deleted3 .
In a single document, there will be few sentenceswith the same content. In contrast, in multiple documents with multiple sources, there will be many
sentences that convey the same content with different words and phrases, or even identical sentences.
Thus, a text summarization system needs to recognize such redundant sentences and reduce the redundancy in the output summary.
However, we have no way of measuring the effectiveness of such redundancy in the corpora for
DUC and TSC2. Key data in TSC2 was given as
abstracts (free summaries) whose number of characters was less than a fixed number and, thus, it
is difficult to use for repeated or automatic evaluation, and for the extraction of important sentences.
Moreover, in DUC, where most of the key data wereabstracts whose number of words was less than a3
It is true that we need other important techniques such as
those for maintaining the consistency of words and phrases thatrefer to the same object, and for making the results more readable; however, they are not included here.

fixed number, the situation was the same as TSC2.
At DUC 2002, extracts (important sentences) wereused, and this allowed us to evaluate sentence extraction. However, it is not possible to measure theeffectiveness of redundant sentences reduction since
the corpus was not annotated to show sentence withsame content. In addition, this is the same even if
we use the SummBank corpus (Radev et al., 2003).
In any case, because many of the current summarization systems for multiple documents are basedon sentence extraction, we believe these corpora to
be unsuitable as sets of documents for evaluation.
On this basis, in TSC3, we assumed that the process of multiple document summarization consistsof the following three steps, and we produce a corpus for the evaluation of the system at each of thethree steps4 .
Step 1 Extract important sentences from a given setof documents
Step 2 Minimize redundant sentences from the result of Step 1
Step 3 Rewrite the result of Step 2 to reduce thesize of the summary to the specified number ofcharacters or less.
We have annotated not only the important sentences in the document set, but also those among
them that have the same content. These are the corpora for steps 1 and 2. We have prepared humanproduced free summaries (abstracts) for step 3.
In TSC3, since we have key data (a set of correct important sentences) for steps 1 and 2, we conducted automatic evaluation using a scoring program. We adopted an intrinsic evaluation by humanjudges for step 3, which is currently under evaluation. We provide details of the extracts preparedfor steps 1 and 2 and their evaluation measures in
the following sections. We do not report the overall
evaluation results for TSC3.
2.2 Data Preparation for Sentence Extraction
We begin with guidelines for annotating importantsentences (extracts). We think that there are twokinds of extract.
1. A set of sentences that human annotators
judge as being important in a document set(Fukusima and Okumura, 2001; Zechner,
1996; Paice, 1990).
4This is based on general ideas of a summarization system
and is not intended to impose any conditions on a summarization system.

(a)
(b)
(c)
(d)

Doc. x

Mainichi articles

abstract

Doc. yYomiuri articles

Figure 1: An example of an abstract and its sources.
2. A set of sentences that are suitable as a sourcefor producing an abstract, i.e., a set of sentences in the original documents that correspond to the sentences in the abstracts(Kupiec
et al., 1995; Teufel and Moens, 1997; Marcu,
1999; Jing and McKeown, 1999).
When we consider how summaries are produced,
it seems more natural to identify important segments in the document set and then produce summaries by combining and rephrasing such information than to select important sentences and revise
them as summaries. Therefore, we believe that second type of extract is superior and thus we prepared
the extracts in that way.
However, as stated in the previous section, withmultiple document summarization, there may be
more than one sentence with the same content, andthus we may have more than one set of sentencesin the original document that corresponds to a given
sentence in the abstract; that is to say, there may be
more than one key datum for a given sentence in the
abstract5 .
we have two sets of sentences that correspond tosentence in the abstract.
(1)  of document  , or(2) a combination of  and  of document
This means that  alone is able to produce , andcan also be produced by combining   and   (Figure 1).
We marked all the sentences in the original documents that were suitable sources for producing thesentences of the abstract, and this made it possible
for us to determine whether or not a summarization system deleted redundant sentences correctlyat Step 2. If the system outputs the sentences in
the original documents that are annotated as corresponding to the same sentence in the abstract, it
5

We use ‘set of sentences’ since we often find that morethan one sentence corresponds to a sentence in the abstract.

Table 1: Important Sentence Data.
Sentence ID of Abstract1
2
3

Set of Corresponding Sentences  
        

has redundancy. If not, it has no redundancy. Returning to the above example, if the system outputs
  ,  ,and  , they all correspond to sentence in the
abstract, and thus it is redundant.

3

Evaluation Metrics

We use both intrinsic and extrinsic evaluation. The
intrinsic metrics are “Precision”, “Coverage” and“Weighted Coverage.” The extrinsic metric is“Pseudo Question-Answering.”
3.1 Intrinsic Metrics
3.1.1 Number of Sentences System Should
Extract
Precision and Recall are generally used as evaluation matrices for sentence extraction, and we used
the PR Breaking Point (Precision = Recall) for theevaluation of extracts in TSC1 (Fukusima and Okumura, 2001). This means that we evaluate systems
when the number of sentences in the correct extract is given. Moreover, in TSC3 we assume thatthe number of sentences to be extracted is known
and we evaluate the system output that has the same
number of sentences.
However, it is not as easy to decide the number ofsentences to be extracted in TSC3 as in TSC1. We
assume that there are correspondences between sentences in original documents and their abstract as in
Table 1. An ASCII space, ” ”, is the delimiter forthe sets of corresponding sentences in the table. As
shown in the table, we often see several sets of sentences that correspond to a sentence in the abstract
in multiple document summarization.
An ‘extract’ here is a set of sentences needed
to produce the abstract. For instance, we can obtain ‘extracts’ such as “   ,  , ! , " , # , "# ”, and“  $# , % ,  , ! , " , # ,   ,  ” from Table 1 6 . Often
there are several ‘extracts’ and we must determine
which of these is the best. In such cases, we definethe ‘correct extract’ as the set with the least numberof sentences needed to produce the abstract because
it is desirable to convey the maximum amount of
information with the least number of sentences.
Finding the minimum set of sentences to produce
the abstract amounts to solving the constraint sat6

In fact, it is possible to produce the abstract with other sentence combinations.

isfaction problem. In the example in Table 1, we
obtain the following constraints from each sentencein the abstract:&(' * )+-,/.#*0/ 1 ,&(' 2)+30/!304" ,&(' 2)+.#504%604718,4.-04#504"#71

With these conditions, we now find the minimum
set that makes all the conjunctions true. We needto find the minimum set that makes ' 90 ' /0' :)<;= >-? @ In this case, the minimum cover isAB CB%!7B%"CB%#DB%"#CE , and so the system should
extract six sentences.
In TSC3, we computed the number of sentencesthat the system should extract and then evaluated thesystem outputs, which must have the same numberof sentences, with the following precision and coverage.
3.1.2 Precision
Precision is the ratio of how many sentences in thesystem output are included in the set of the corresponding sentences. It is defined by the following
equation.
Precision FHG IKJ(1)

where L is the least number of sentences needed
to produce the abstract by solving the constraint
satisfaction problem and M is the number of ‘correct’ sentences in the system output, i.e., the sentences that are included in the set of corresponding sentences. For example, the sentences listed
in Table 1 are ‘correct.’ If the system output is“  #CB  B%!B  NB%"#DB%"  ”, then the Precision is asfollows:
Precision FPOQRFRSUT

Q%QWV
T

(2)

for “  XB  #DB%B CB%!B "# ”, the Precision is as follows:
Q
Precision F Q FZY[T(3)
3.1.3 Coverage
Coverage is an evaluation metric for measuring how
close the system output is to the abstract taking into
account the redundancy found in the set of sentencesin the output.
The set of sentences in the original documentsthat corresponds correctly to the \ -th sentence oftheabstract
is denoted here as]_^a` human-produced]b^a`]_^a` d]_^a` eB
CB%c%c cB
B%c%c cB
. In this case, we have]_^a` dsets of corresponding sentences. Here,
indicates a set of elements each of which corresponds tothe sentencedeA7g ^ing ^a` doriginalg documents,]_^a` d number` d` the`^` d` h)
B ] ` 7B%c c%cB g ` ` B c%g c ` c E ` . For
noted asinstance,from1,  i)  XB    andg ` `g ` Table`  3)+ $#CB   j)+  .
Then, we define the evaluation score ? .$\1 for the\ -th sentence in the abstract as equation (1).
z {}|~ z u laa w   ok[lnmo F:prqts  w J(4)uvxwvUy
f

W.C. F

where .1 is defined by the following equation.
 lao F
S

if the system outputs otherwiseY

If the system extracts “  $# ,   , ! , %N , "# , "  ”,
? .\ 1 is computed as follows:
max l SUJ[Y o F
max l SUT t o F
max l SUJSUT % o FYS T %
U
SUT %

and its Coverage is 0.553. If the system extracts“   , %$# , % ,  , ! , "# ”, then the Coverage is 0.780.
ktl Y o F
ktlxo F
ktl  o F

max l Y[JXY o FZY
QWV
QWVmax l SUT o FSUT
QWV
QWV
max l SUJSUT o FRSUT

3.1.4 Weighted Coverage
Now we define ‘Weighted Coverage’ since eachsentence in TSC3 is ranked A, B or C, where “A” isthe best. This is similar to “Relative Utility” (Radev
et al., 2003). We only use three ranks in order tolimit the ranking cost. The definition is obtained bymodifying equation (6).
W.C. F u lnxlnmook[lnmol onl  o l onl o  la¡5onla¡5o J

Y  Y  S7T °  SUT %  U
S T  U
S T %
Q%±%²
F/SUT
T





Y
Y
SUT °
Y
S T
U
Y

(8)

(5)

]b^a` dis outFunction ? returns 1 (one) when any
puted completely. Otherwise it returns a ]_partial^nd.
score according to the number of sentences Given function ? and the number of sentences in
the abstract  , Coverage is defined as follows: k[lnmo uTCoverage F(6)k[l Y o Fk[lWo F
k[l  o F

where =.\ 1 denotes the ranking of the \ -th sentenceof the abstract and ¢£.$=.$\11 is its weight. ¤.$= ¤¥1 is
the number of sentences whose ranking is = ¦¥ inthe abstract. Suppose the first sentence is ranked A,
the second B, and the third] C in Table 1, and theirweights are given as ¢9. 1)¨§ ,¢£.©ª1«)¨¬7@®­ and¢9. ' 16)+¬7@®¯ 7 .
Asbefore,ifthesystemextracts
then the Weighted“  # ,   ,  ! ,  N ,  "# ,  "  ”,
Coverage is computed as follows:(7)

3.2 Extrinsic Metrics
3.2.1 Pseudo Question-Answering
Sometimes question-answering (QA) by humansubjects is used for evaluation (Morris et al., 1992;
Hirao et al., 2001). That is, human subjects judge
whether predefined questions can be answered byreading only a machine generated summary. However, the cost of this evaluation is huge. Therefore,
we employ a pseudo question-answering evaluation,i.e., whether a summary has an ‘answer’ to the question or not. The background to this evaluation is inspired by TIPSTER SUMMAC’s QA track (Mani etal., 2002).
For each document set, there are about five questions for a short summary and about ten questions
for long summary. Note that the questions for the
short summary are included in the questions for thelong summary. Examples of questions for the topic“Release of SONY’s AIBO” are as follows: “Howmuch is AIBO?”, “When was AIBO sold?”, and“How many AIBO are sold?”.
Now, we evaluate the summary from the ‘exact
match’ and ‘edit distance’ for each question. ‘Exact match’ is a scoring function that returns one
when the summary includes the answer to the question. ‘Edit distance’ measures whether the system’ssummary has strings that are similar to the answer
strings. The score ³ e´ based on the edit distance isnormalized with the length of the sentence and theanswer string so that the range of the score is [0,1]:
Sed F

length of the sentence µ edit distanceTlength of the answer strings(9)

The score for a summary is the maximum valueof the scores for sentences in the summary. The
7 ¶¸·º¹ ·n»¼¼
may be computed differently. It is 1/rank (onedivided by rank) here.

Table 2: Description of TSC3 Corpus.
# of doc. sets
# of articles (The Mainichi)# of articles (The Yomiuri)
Total# of Sentences

30
175
177
352
3587

score is 1 if the summary has a sentence that includes the whole answer string.
It should be noted that the presence of answer
strings in the summary does not mean that a humansubject can necessarily answer the question.

4

Preliminary Experiment

In order to examine whether our corpus is suitable
for summarization evaluation, our evaluation measures significant information and redundancies inthe system summaries.
Below we provide the details of the corpus, evaluation results and effectiveness of the minimizationof redundant sentences.
4.1 Description of Corpus
According to the guidelines described in section
two, we constructed extracts and abstracts of thirty
sets of documents drawn from the Mainichi andYomiuri newspapers published between 1998 to1999, each of which was related to a certain topic.
First, we prepared abstracts (their sizes were 5%
and 10% of the total number of the characters in
the document set), then produced extracts using theabstracts. Table 2 shows the statistics.
One document set consists of about 10 articles
on average, and the almost same number of articles
were taken from the Mainichi newspaper and the
Yomiuri newspaper. Most of the topics are classified
into a single-event according to McKeown (2001).
The following list contains all the topics.
0310 Two-and-half-million-year old new hominid species
found in Ethiopia.
0320 Acquisition of IDC by NTT (and C&W).
0340 Remarketing of game software judged legal by Tokyo
District Court.
0350 Night landing practice of carrier-based aircrafts of the
Independence.
0360 Simultaneous bombing of the US Embassies in Tanzania
and Kenya.
0370 Resignation of President Suharto.
0380 Nomination of Mr. Putin as Russian prime minister.
0400 Osama bin Laden provided shelter by Taliban regime in
Afghanistan.
0410 Transfer of Nakata to A.C. Perugia.
0420 Release of Dreamcast.
0440 Existence of Japanese otter confirmed.
0450 Kyocera Corporation makes Mita Co. Ltd. its subsidiary.
0460 Five-story pagoda at Muroji Temple damaged by typhoon.
0470 Retirement of aircraft YS-11.

0480 Test observation of astronomical telescope ‘Subaru’started.
0500 Dolly the cloned sheep.
0510 Mass of neutrinos.
0520 Human Genome Project finishes decoding of the 22nd
chromosome.
0530 Peace talks in Northern Ireland at the end of 1999.
0540 Debut of new model of bullet train (700 family).
0550 Mr. Yukio Aoshima decides not to run for gubernatorial
election.
0560 Mistakes in entrance examination of Kansai University.
0570 Space shuttle Endeavour, from its launch to return.
0580 40 million-year-old fossil of new monkey species foundby research group at Kyoto University.
0590 Dead body of George Mallory found on Mt. Everest.
0600 Release of SONY’s AIBO.
0610 e-one, look-alike of iMac.
0630 Research on Kitora tomb resumes.
0640 Tidal wave damage generated by earthquake in Papua
New Guinea.
0650 Mistaken bombing of the Chinese embassy by NATO.

4.2 Compared Extraction Methods
We used the lead-based method, the TF c IDF-based
method (Zechner, 1996) and the sequential patternbased method (Hirao et al., 2003), and compared
performance of these summarization methods on
the TSC3 corpus.
Lead-based Method
The documents in a test set were sorted in chronological and ascending order. Then, we extracted asentence at a time from the beginning of each document and collected them to form a summary.
TF c IDF-based Method
The score of a sentence is the sum of the significant
scores of each content word in the sentence. We
therefore extracted sentences in descending order^ ofimportance score. The sentence score Stfidf . 1 isdefined by the following.
Stfidf la½  o F ¾¿[À  ºl Á J ÂjÃ o J|

where ¢9.;XB Ä³_1 is defined as follows:  Â lnÁ JÂÅÃ o F ÁÆÇlnÁ JÂÅÃ o}ÈÉºÊ%Ë Ì ÆÍlºÁ o T

(10)

(11)

;[ÎÏ.$;XB%Ä«³b1 is the frequency of word ; in the document set, ÐÎ.$;X1 is the document frequency of ; , and Ä©4 is the total number of documents in the set. In
fact, we computed these using all the articles published in the Mainichi and Yomiuri newspapers forthe years 1998 and 1999.

Sequential Pattern-based Method
The score of a sentence is the sum of the significant scores of each sequential pattern in the sentence. The patterns used for scoring were decided

Table 3: Evaluation results for “Precision”, “Coverage” and “Weighted Coverage.”
MethodLead
TF Ñ IDF
Pattern

Length
Short
Long
Short
Long
Short
Long

Prec.
.426
.539
.497
.604
.613
.665

Cov.
.212
.259
.292
.325
.305
.298

W.C.
.326
.369
.397
.434
.403
.418

MethodTF Ñ IDF
Pattern

Table 4: Evaluation results for “Pseudo QuestionAnswering.”
MethodLead
TF Ñ IDF
Pattern

Length
Short
Long
Short
Long
Short
Long

Exact
.300
.275
.375
.393
.390
.370

Edit
.589
.602
.643
.659
.644
.640

where ¢£.$ÕÏ1 is defined as follows: lºÔÍo F(13)

Î.ÕÏBÄ³_1 is the sentence frequencyof pattern Õ in]the document set and Î.ÕÏB ³b1 is] the sentence frequency of pattern Õ in all topics.  ³r is the numberof sentences in all topics and Û$?¦.ÕÏ1 is the patternlength.

4.3 Evaluation Result
Table 3 shows the intrinsic evaluation result. All
methods have lower Coverage and Weighted Coverage scores than Precision scores. This means thatthe extracted sentences include redundant ones. In
particular, the difference between “Precision” and“Coverage” is large in “Pattern.”
Although both “Pattern” and “TF c IDF” outperform “Lead,” the difference between them is small.
In addition, we know that “Lead” is a good extraction method for newspaper articles; however, this is
not true for the TSC3 corpus.
Table 4 shows the extrinsic evaluation results.
Again, both “Pattern” and “TF c IDF” outperform“Lead”, but the difference between them is small.
We found a correlation between the intrinsic and extrinsic measures.

Length
Short
Long
Short
Long

Prec.
.430
.533
.531
.620

Cov.
.297
.345
.289
.338

W.C.
.377
.455
.390
.456

Table 6: Effects of clustering (Pseudo QuestionAnswering).
MethodTF Ñ IDF
Pattern
by using a statistical significance test such as the Ò
metric test and using 1,000 patterns. This is an extension of Lin’s method^ (Lin and Hovy, 2000). The
sentence score Spat . 1 is defined by the following.
(12)
Spat la½  o F ÓX¿[À  lnÔÍo J|

z {ÍÖÇz
ÉÊ%ËWlaÆlnÔ J ÂÅÃ o} Y oÇÈÉÊ%ËWl × Ø Ó  {ÍÖ Ù oÚ klºÔÍo
T

Table 5: Effects of clustering (“Precision”, “Coverage”, “Weighted Coverage”).

Length
Short
Long
Short
Long

Exact
.401
.377
.392
.380

Edit
.650
.648
.650
.655

4.4 Effect of Redundant Sentence
Minimization
The experiment described in the previous section
shows that a group of sentences extracted in a simple way includes many redundant sentences. To
examine the effectiveness of minimizing redundantsentences, we compare the Maximal Marginal Relevance (MMR) based approach (Carbonell and Goldstein, 1998) with the clustering approach (Nomoto
and Matsumoto, 2001). We use ‘cosine similarity’with a bag-of-words representation for the similarity measure between sentences.
Clustering-based Approach
After computing importance scores using equations(10) and (12), we conducted hierarchical clustering
using Ward’s method until we reached L (see Section 3.1.1) clusters for the first ¯7L sentences. Then,we extracted the sentence with the highest scorefrom each cluster.
Table 5 shows the results of the intrinsic evaluation and Table 6 shows the results of the extrinsic evaluation. By comparison with Table 3, theclustering-based approach resulted in TF c IDF andPattern scoring low in Precision, but high in Coverage. When comparing Table 4 with Table 6, thescore is improved in most cases. These results imply that redundancy minimization is effective for
improving the quality of summaries.
MMR-based Approach
After computing importance scores using equations(10) and (12), we re-ranked the first ¯DL sentences by
MMR and extracted the first L sentences.
Table 7 and 8 show the intrinsic and extrinsic
evaluation results, respectively. We can see the effectiveness of redundancy minimization by MMR.
Notably, in most cases, there is a large improvement
in both the intrinsic and extrinsic evaluation results
as compared with clustering.

Table 7: Effects of MMR (“Precision”, “Coverage”,“Weighted Coverage”).
MethodTF Ñ IDF
Pattern

Length
Short
Long
Short
Long

Prec.
.469
.565
.469
.577

Cov.
.306
.376
.332
.377

W.C.
.403
.475
.429
.500

Table 8: Effects of MMR (Pseudo QuestionAnswering).
MethodTF Ñ IDF
Pattern

Length
Short
Long
Short
Long

Exact
.386
.405
.417
.390

Edit
.647
.667
.663
.656

These results show that redundancy minimization
has a significant effect on multiple document summarization.

5

Conclusion

We described the details of a corpus constructed for
TSC3 and measures for its evaluation, focusing on
sentence extraction. We think that a corpus in which
important sentences and those with the same content
are annotated for multiple documents is a new andsignificant feature for summarization corpora.
It is planned to make the TSC3 corpus available(even if the recipient is not a TSC3 participant) byexchanging memoranda with the National Instituteof Informatics in Japan. We sincerely hope that this
corpus will be useful to researchers who are interested in text summarization and serve to facilitate
further progress in this field.

References
J. Carbonell and J. Goldstein. 1998. The Use of
MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In
Proc. of the 21th ACM-SIGIR, pages 335–336.
T. Fukusima and M. Okumura. 2001. Text Summarization Challenge: Text Summarization Evaluation in Japan. In Proc. of the NAACL 2001 Workshop on Automatic summarization, pages 51–59.
T. Hirao, Y. Sasaki, and H. Isozaki. 2001. An
Extrinsic Evaluation for Question-Biased Text
Summarization on QA tasks. In Proc. of the
NAACL 2001 Workshop on Automatic Summarization, pages 61–68.
T. Hirao, J. Suzuki, H. Isozaki, and E. Maeda.
2003. Multiple Document Summarization using
Sequential Pattern Mining (in Japanese). In The
Special Interest Group Notes of IPSJ (NL-158-6),
pages 31–38.

H. Jing and K. McKeown. 1999. The Decomposition of Human-Written Summary Sentences.
Proc. of the 22nd ACM-SIGIR, pages 129–136.
J. Kupiec, J Petersen, and F. Chen. 1995. A Trainable Document Summarizer. In Proc. of the 18th
SIGIR, pages 68–73.
C-Y. Lin and E. H. Hovy. 2000. The AutomatedAcquisition of Topic Signatures for Text Summarization. In Proc. of the 16th COLING, pages
495–501.
I. Mani, G. Klein, D. House, L. Hirschman, T. Firman, and B. Sundheim. 2002. SUMMAC: a text
summarization evaluation. Natural Language
Engineering, 8(1):43–68.
D. Marcu. 1999. The Automatic Constructionof Large-scale Corpora for Summarization Research. Proc. of the 22nd ACM-SIGIR, pages
137–144.
K. McKeown, R. Barzilay, D. Evans, V. Hatzivassilogou, M. Y. Kan, B. Schiffman, and S. Teufel.
2001. Columbia Multi-Document Summarization: Approach and Evaluation. In Proc. of theDocument Understanding Conference 2001.
A. H. Morris, G. M. Kasper, and D.A. Adams.
1992. The Effects and Limitations of Automatic
Text Condensing on Reading Comprehension.
Information System Research, 3(1):17–35.
T. Nomoto and M. Matsumoto. 2001. A New Approach to Unsupervised Text Summarization. In
Proc. of the 24th ACM-SIGIR, pages 26–34.
M. Okumura, T. Fukusima, and H. Nanba. 2003.
Text Summarization Challenge 2, Text Summarization Evaluation at NTCIR Workshop 3. In
Proc. of the HLT/NAACL 2003 Text Summarization Workshop, pages 49–56.
C. Paice. 1990. Constructing Literature Abstracts
by Computer: Techniques and Prospects. Information Processing and Management, 26(1):171–
186.
D. R. Radev, S. Teufel, H. Saggion, W. Lam,
J. Blitzer, H. Qi, A. Celebi, D. Liu, and
E. Drabek. 2003. Evaluation challenges in largescale document summarization. In Proc. of the
41st ACL, pages 375–382.
S. Teufel and M. Moens. 1997. Sentence Extraction as a Classification Task. In Proc. of the ACL
Workshop on Intelligent Scalable Text Summarization, pages 58–65.
K. Zechner. 1996. Fast Generation of Abstracts
from General Domain Text Corpora by Extracting Relevant Sentences. In Proc. of the 16th
COLING, pages 986–989.