Phrase-representation Summarization Method
and Its Evaluation
OKA Mamiko and UEDA Yoshihiro
Fuji Xerox Co., Ltd.
430 Sakai, Nakai-machi, Ashigarakami-gun, Kanagawa 259-0157, Japan{oka.mamiko, Ueda.Yoshihiro}@fujixerox.co.jp

Abstract
We have developed a summarization method thatcreates a summary suitable for the process of sifting
information retrieval results. Unlike conventional
methods that extract important sentences, this method
constructs short phrases to reduce the burden ofreading long sentences. We developed an improved
task-based evaluation method and applied to provethe effectiveness of phrase-represented summary. In
the task-based evaluation in TSC, phrase-represented
summary provided the fastest judgement amongsystems that achieved almost the same accuracy.
However, the experiment method has some problems,
and we must try to design tasks closer to the realworld IR in the future TSC.
Keywords: Phrase-representation summarization,
Phrase-represented summary, At-a-glance, Indicative,
Task-based evaluation

1

Introduction

Summaries are used to select relevant documentfrom information retrieval results. The goal ofsummarization for such “indicative” use is to provide
fast and accurate judgement.
Most automatic summarization systems adopt the“sentence extraction” method. It gives a score to
every sentence on the basis of its characteristics, suchas word frequency, the position in which it appears,
etc. and selects sentences with high scores.
The sentences collected in such a way tend to be
so long and complex that the reader must reconstruct
the structure while reading them. Reading suchsentences involves some annoyance.
Our aim is to reduce this burden by providing an“at-a-glance” summary.
Phrase-representation
summarization is a method to create the “at-a-glance”
summary for the Japanese language.
In this paper, we present the concept, the algorithm,
and evaluation of the efficacy of the summary
produced by a prototype based on this method. The

results and the problems of NTCIR Text
Summarization Workshop (TSC) are also mentioned.

2

The concept

Examples of an “at-a-glance” summary are headlines of news articles.
The headline provides
information for judging whether a reader should readthe article or not and, in this sense, it is really“indicative.” The characteristics are brevity (short inlength) and simplicity (less embedded sentences).
We use “phrases” to represent the simplicity1 andset our goal to create phrase-represented summaries.
They provide a reader with an outline of the
document, avoiding reading stress by enumerating
short phrases containing the important words andconcepts composed from these words.
The method we adopted to achieve this goal is toconstruct such phrases from the relations between
words.
The phrase-represented summary has thefollowing characteristics.
(1) At-a-glance comprehension
Because each unit is short and simple, a user isable to grasp the meaning at a glance.
(2) Adequate informativeness
Unlike extracted sentences, phrases created bythis method are not accompanied by information
unnecessary for relevance judgement.
(3) Wide coverage of topics
Units composing a summary are relatively short,and point various positions of the original text.
Therefore, even a generic summary includes
various topics written in a document.

1

The word “phrase” used here is not of the linguistic sense but anexpression for “short” and “simple.” In Japanese, there is no
rigid distinction between “phrase” and “clause.”Original
表 4.2 に示すように、「句」による要約は、手がかりのカバー率と適切な情報量の両方を満たしており、……
(As shown in Figure 4.2, the summaries composed by phrase satisfy
both the coverage of the clues and adequate quantity of information…)(1) selection ofcore relation
句(phrase)Analysis graph手がかり(clues)の(OF)

要約(summaries)
による(ELEMENT)(2) Addition of relations

カバー率(coverage)
と
(AND)

情報量
(quantity of
information)

適切な(adequate)

の
(OF)

両方
(both)

は(THEME)満たす(satisfy)

を(OBJ)(3) generation
Obtained phrase
句による要約はカバー率と情報量の両方を満たす(The summaries composed by phrase satisfy both the coverage and quantity of information.)
Figure 1: Outline of phrase construnction.

3

Summarization method

3.1

Outline of phrase construction

Here we give the outline of the algorithm toconstruct a phrase using the example shown in Figure
12. The method consists of the following three steps:(1) Selecting a core relation from a text analyzed
the relationships between words.
(2) Adding relations necessary for the unity of thephrase’s meaning.
(3) Generating a surface phrase from the selected
relations.
On the first step, a core relation of a phrase is
selected from the given text. The sentences in the
text are analyzed to produce directed acyclic graphs(DAGs) constructed from relation units. Each unit
consists of two nodes (words) and an arc (relation
between the words). Each node is not only a single
word but also can be a word sequence (noun group).

2

In this paper, the examples are represented in Japanese, because
TSC evaluates Japanese text summarization.
However,translated words or particle functions in English are attached asmuch as possible. Applicability of the phrase-representation
summarization to other languages is discussed in [1].

In Figure 1, the arc connecting the two shaded nodes
represents the core relation.
The core relation alone carries insufficient
information to convey the content of the original
document. On the second step, additional relations
are attached to specify the information the phrase
supplies into further detail. In Figure 1, the nodes
and arcs with double line are attached.
On the third step, a short phrase can be generated
from the selected nodes and arcs in the graph.
Phrase-represented summary enumerates such
short phrases to give the readers enough information
to grasp the outline of a document. This algorithm is
explained in the next section.

3.2

Outline of summary generation

A phrase-represented summary consists of several
phrases, and there are two algorithms to generate asummary. One is based on important relations,another is based on important phrases.
Using the relation-based algorithm, an important
relation is selected first and it is used as the core of aphrase. The required number of core relations are
first selected. Using the phrase-based algorithm, allpossible phrases are created first, and importantphrases are selected from them. Here, we introduce
the flow of the phrase-based algorithm, because thesummaries submitted to TSC are generated by thisalgorithm. The flow is shown in Figure 2.
The other method, the relation-based algorithm isintroduced in [1].

Input
Relation analysis
Selection of a relation

Further description of each step

Here, we describe each step in Figure 2.

Additional relation attachment

Relation Analysis
Syntactic analysis is applied to each sentence inthe document to produce a DAG of the relations ofwords. We use a simple parser based on patternmatching [2], one of whose rules always judges eachcase dependent on its nearest verb. Some of themisanalysis will be hidden by “ambiguity packing” in
the “additional relation attachment” step.

Yes

A Relation remains?

Construction of a phrase

3.3

Document

No
Relation scoring
Phrase scoring

Core relation selection
Selection of an important phrase

A relation unit (two nodes and an arc connectingthem) is selected as the core of a phrase.
Because different core relations can produce thesame phrase by attaching additional relations (in thenext step), it is not necessary to select every relation
as a “core”.
In the current implementation, most relations are
selected as the core and omitting multiple phrases in
the phrase selection step, because we are in theprocess of considering what is the good phrase bycomparing the phrases generated by the system.

No

Yes
Generation of surface phrases
OutputSummary
Figure2. Flow of thephrase-based algorithm.

Additional relation attachment
The information that the core relation carries isusually insufficient. Additional relations are attachedto make the information the phrase supplies more
specific and to give the reader sufficient information
to infer the content of the original document. The
following relations are a part of the relations to be
attached.
(1) Mandatory cases
Relations that correspond to mandatory cases areattached to verbs. Mandatory case is defined for eachverb except for those that share the common
mandatory case list, which includes “は” (THEME),“も” (ALSO) and null-marker.
Ex.) パ ン ダ (panda) が (AGENT) 上 野 動 物 園(Ueno Zoo) に (DATIVE) 贈 ら れ た (waspresented)
In the examples in this section, underlined words
consist of the core relation, and double underline
represents the attached words.
(2) Noun modified by a verb
In Japanese, the “verb – noun” structure represents
an embedded sentence, and the noun usually fillssome gap in the embedded sentence. If the verb in
the core relation (noun – verb) consists of such a verb

Terminate?

– noun relation, the modified noun is also assumed tocarry important information, even if it does not fill
the mandatory case (though the case is not analyzed
in the current algorithm). Thus the verb – noun
relation is attached to the core.
Ex.) 日本中(all over Japan)を(OBJ)席巻する(dominating)パンダブーム(panda boom)(3) Ambiguity packing
The analysis graphs often contain errors because
the pattern-based parser does not resolve ambiguities.
For example, the structure
V 0-THAT3 N1 “の”(OF) N2 (Ving N1’s N2)
is ambiguous in Japanese (V can modify either N1 or
N2 but the parser always analyzes N1 as modified).
If the V – N1 relation is selected as the core, the N1 –N2 relation is always attached to the core to include
the possible V – N2 relation.

3"0" shows that there are no particles or any other words
connecting two words. Japanese does not require anything likerelative pronouns.

Ex.) 椅子(chair)に(DATIVE)座った(sat)魯迅(Lu Xun)の(OF)像(statue)(4) Modifiers of generic nouns
The concepts brought by generic nouns such as“もの” (thing), “こと” (“that” of that-clause), “場合”(case), “ 時 代 ” (era) are not so specific that theyusually accompany modifiers to be informative. Heresuch modifiers are attached to make them informative.
Ex.) 句(phrase)は(THEME)要約(summary)に(DATIVE)適した(suitable)単位(unit)である(is)words used in the last phrase to avoid frequent use of
the same words.
Score reduction is achieved by multiplying thepredefined cut-down ratio R (0 < R < 1) by the scoresof the words used.

Generation of surface phrases
The surface phrases are produced from selected
relations to connect the surface strings of the nodes
and their belonging words in the original order. In
this step, the parts that are not selected in a summaryphrase are replaced by “ … ”. An example is asfollows.
Ex.) …「句」による要約は、…カバー率と…情報量の両方を満たしており、…

Relation scoring
An importance score is provided for each relation
unit.
First, every word is scored by its importance. Thisscore is calculated based on the tf*IDF value4 [3]. In
the summaries submitted in TSC, the score for theword in the headlines (Task A) or the queries (Task
B) is weighted more than its tf*IDF value.
Then, the relation score is calculated as follows:
Score = Srel * (S1 + S2)
Here, S1 and S2 are the scores of the two words
connected by relations. The score of a word sequenceis calculated by decreasing the sum of the scores of
its constituent words according to the length of theword sequence.
Srel is the importance factor of the relation. The
relations that play central roles in the meaning, suchas verb cases, are given high scores, and thesurrounding relations, such as “AND” relations, arescored low.

4

Implementation

We developed a prototype of the summarization
system based on this algorithm. The system isdeveloped in Java and C++, and working on
Windows 95/98/NT and Solaris 2.6.
The time consumed by summarization process is in
proportion to the text length and it takes about 800
msec to generate a summary for an A4 sized
document (2000 Japanese characters) using a Sun
Enterprise 4505.

5

Evaluation

5.1

Evaluation method for the phraserepresented summary

Phrase selection

The
aim
of
the
phrase-representation
summarization is to give fast and accurate judgement
in selecting relevant documents from IR results.
Thus, task-based evaluation on information retrieval[4] is adequate to evaluate the effectiveness of thesummarization method.
We have conducted an evaluation experiment in1999 [5], and participated in task-based evaluation(Task B) in TSC.

The phrase with the highest score among allphrases is selected to compose a summary.

5.2

Phrase scoring
An importance score is provided for each phrase.
The basic algorithm is to sum up the scores of
constituent relations in a phrase. To remove theinfluence of the phrase length, the total score isrelaxed according to the number of constituent
relations.

Terminative condition
Whether the summaries created so far are
sufficient is judged. Currently, either the number ofphrases or characters defines the condition.

Re-scoring of relations
If the condition is not satisfied, another phrase is
selected. Before them, relation scores and phrasescores are re-calculated reducing the scores of theTask-based evaluation has recently drawn theattention in the summarization field, and someexperiments on information retrieval were reported[6][7]. However, there is no standard evaluation
method, and we consider that there are someshortcomings in the existing methods. Thus, wedeveloped an improved evaluation method andcarried out a relatively large-scale experiment.

5
4

IDF is calculated from Mainichi Newspapers in 1995 (CD-ROM).

Our task-based evaluation in 1999

Java , Solaris and Sun are the trademarks of Sun Microsystems.
Windows is the trademarks of Microsoft and Intel, respectively.

briefly

introduceourtask-based

Evaluation method
We compared four types of summary: (a) leading
fixed-length characters, (b) tf*IDF-based sentence
extraction summaries [8], (c) phrase-represented
summaries and (d) tf*IDF-based keywords. The
evaluation criteria were the accuracy of subjects’
relevance assessment and the time to assess.
The characteristics of our evaluation method are as
follows:(1) The summary length is regulated to sixty to eighty
characters.
(2) To reduce the diversity of the assessment, wemade the assumed situation realistic and specify it
into details including the purpose of the search.
(3) We assigned ten subjects per summary sample toreduce the influence of each person's assessment.
(4) For subjects to assess the relevance, we
introduced four relevance levels (from higher tolower: L3, L2, L1 and L0, which is judged to beirrelevant).

subject of a document or not is independent of
whether a user needs the document or not.
We can not simply compare all summaries in
Figure 4 because of the diversity of the summary
length.
0.85
0.8

0.85
0.8

Phrases

0.7

Sentences

0.65
0.6

S9 S1
S5
S6
TF S8 S7

S2
Full

Lead

0.65
0.6
0.55
0.5
0.45
0.4
300

400

500

600

700

800

Figure 4. The relationshipbetween f-measures and time.
The summary length is not regulated in TSC-task
B. We submitted two different lengths of phraserepresented summaries: within 100 characters(System 3) and 150 characters (System 4) to fit it to
the purpose of the task (IR sifting).
However, other participants submitted much
longer summaries. The summary length of eachsystem is compared in Figure 5. The broken line
represents the average character numbers of allsummaries except full text. Figure 5 shows thatsummaries by System 3 and System 4 are much
shorter than the average length.

Keywords
800

0.55

700

0.5

Lead

0.45
0.4
220

240

260

280

300

320

Time [sec]Figure 3. The relationshipbetween f-measures and time.

600
500
400
300
200
100

The relationship between f-measures and time in
TSC-task B is shown in Figure 4. We use the resultof “answer level B”, because whether a topic is the

S8
S9
Fu
ll
TF
Le
ad

S7

S5
S6

S3
S4

S2

Task-based evaluation in TSC

S1

0

5.3

900

Time [sec]Number of characters

F-measures

0.75

S4

0.7

Experiment results
The relationship of the accuracy and the time isshown in Figure 3. To represent the accuracy, fmeasures are used in Figure 3.
The result proves that the phrase-represented
summaries serve accurate judgement in a relatively
short time.

S3

0.75

F-measures

Here weevaluation.

Figure 5. Summary length.

Generally speaking, longer text has larger
information, and requires a longer time to read. The
relationship between the summary length and the fmeasures is shown in Figure 6, and the relationshipbetween the summary length and time is shown in
Figure 7.
These figures support the above
assumption.
0.84

S2

0.82

S9

F-measures

0.8

S1

S5

S3

0.78
0.76

S6
TF S7
S4
S8

Full

Lead

0.7
0

100 200 300 400 500 600 700 800 900

Number of characters

Figure 6. The relationship between
summary length and f-measures.

900

Time [sec]

800

Full
S2

700
600
500
400

S3

S7
S9 S1
S8
S6
S5 TF
Lead
S4

300
0

5.4

Toward future task-based evaluation

The comparison between our experiment and TSCtask B is shown in Table 1. From the experiences ofdesigning an experiment method and participating in
TSC-task B, we would like to make some suggestions
for the future TSC.
If the task-based evaluation on information
retrieval will be carried out also in the future, wemust try to design tasks closer to the real situation.
Here we discuss them from three aspects: summary,task and document.

Summaries to be readIn the standard WWW search engines, about 100
characters are used as the summary and the lengthmust be adequate as a replacement of the document
itself. Not only regulate the length, but also selecting
an appropriate length must be needed.

0.74
0.72

(2) The phrase-represented summaries are about
1/8 length of full text and get almost the samef-measures.

Task
In the assumed IR on TSC, each topic is given as asimple concept (e.g. “performances at Kabuki-za”)
and thus the subjects must judge the relevancewhether the concept is included in the document or
not. Such IR is largely different from the real world
IR. We usually have some purpose and retrieval isjust a mean to achieve the goal, and judge whether
the document itself contains information to fulfill the
purpose by reading the summary. The criteria
depends on the situations, for example, “the searcher
must make an survey of reviews of the performancesat Kabuki-za,” or “the searcher wants to reserve
tickets of coming performance at Kabuki-za”.
The f-measures of full texts support our view. In
the task-based evaluation, it is assumed that subjects
can judge relevance correctly by full texts, and thesystem can be evaluated by how good they canachieve the task using summaries instead of full texts.
In the experiment result shown in Figure 4, the fmeasures of the full texts are relatively low. It
represents that many subjects cannot achieve theirtask even if they are given the full information.

100 200 300 400 500 600 700 800 900

Number of characters
Figure 7. The relationship between
summary length and time.

However, at least we can get the following
information from these results.
(1) The phrase-represented summaries are much
faster than the systems that get almost thesame f-measures.

Documents to be retrieved
The document source in TSC is newspaper articles,
but usually we search various types of documents at
once. In the WWW for example, there are various
styles or forms of documents, such as essays, articles,
papers, advertisements, mail archives or diaries.
Though there are many problems (e.g. copyright) toobtain and use various types of document, we should
try to make a test collection of inconsistent
documents and use them in evaluations like TSC.

Table 1. Comparison of the experiment method.
Document source
IR question
Number of questions
Number of documents perquestion
Summary length
Subjects

Our experiment in 1999
WWW
Newly created with detailed
situations
3
10

12
50

60-80 characters
Not regurated
40 persons who usually use WWW 36 students
search103Number of subjects per
summary sample
Relevance levels in subject's 4 levels
assessment(L3: the answer to the question isfound in a summary.
L2: a clue to the answer is found ina summary.
L1: clues are not found in asummary, but the document may be
relevant.
L0: irrelevant.)
Relevancelevelsof 2 levels
document(relevant or irrelevant)

5.5

TSC-task B
Newspaper
Selected topics from IREXThe evaluation of the phrase style

Though TSC-task A is for informative summaries,
we have participated also in task A-2. In task A-2,
the system results are compared with human-preparedsummaries by two ways: one is content-based
evaluation (task A-2-1) and another is subjective
evaluation (task A-2-2).
The result of task A-2-2 shows that theacceptability of the phrase-represented summaries is
not so highly evaluated by human. We suppose there
are two reasons. One is that the form of phraserepresented summary extremely differs from that ofwhich general human reminds as “summary”. The
other
is
that
phrase-represented
summaryoccasionally contains a phrase that is grammaticallyincorrect, whereas sentence extraction does notcontains such errors, because each constituent is theoriginal sentence itself.
One of the purposes that we have participated intask A-2 is to evaluate the phrase style. As describedin Section 3.3, the parts not selected in a summary are
replaced by “…”. This style has both advantages andshortcomings. One of the advantages of using “…” isto allow reader to imagine the meaning even if thephrase is not perfect. One of the shortcomings is that“…” disturbs smooth reading and understanding.
We submitted two types of summary: (1) All parts
not selected in a phrase are replaced by “…” (System

2 levels(relevant or irrelevant)3 levels(Level A: given topic is subject ofan article.
Level B: given topic is not subject
but written in an article.
Level C: irrelevant.)

3). (2) Only the end of a phrase is replaced by “…”(System 4).
The result is that the (2) type summaries were
better in both readability and similarity to the humansummaries. We would like to take this into
consideration in designing future systems.

5.6

The content-based evaluation

The result of task A-2-1, the similarity between thesystem results and the human-prepared summaries isshown in Figure 8. System 3 is omitted here, because
System 3 and System 4 have almost the samecontents.
In Figure 8, the phrase-represented summaries
occupy relatively high positions among all systems.
One of the factors that raise the similarity comesfrom the summary creating method. The phraserepresented summaries are constructed so that they
include the important concepts of the document, andthis goal resembles that of the human free summaries.
And each phrase is a part of the original sentence as
the sentences of the important-part summaries. On
the contrary, thesentences which mostsummarization systems output necessarily include notso important parts of the sentences to reduce thesimilarity score.
If we tune up our system for newspaper summaries
or human-prepared summaries used in TSC dryrun,

the similarity score might be much higher. One
reason why we avoided tune-up is that our target is
not limited to one specific document type and tuning
toward it is inappropriate. Another reason is that we
consider there is no single “correct” summary and thehuman-prepared summary provided for TSC is just
one sample and inappropriate as a target of tune-up.

0.7

Important-part (40%)

0.65

Similarity

0.6

Free (40%)

0.55

Important-part (20%)

0.5
0.45

Free (20%)

0.4
0.35

TF
Le
ad

S9

S8

S7

S6

S5

S4

S2

S1

0.3

Figure 8. The Similarity between
system results and human summaries.

6

ConclusionWe
introduced
thephrase-representation
summarization method. The task-based evaluation in
TSC shows that the summaries are effective for fastsifting, however there seems to be some problems in
the experimental method. Toward the future TSC, weshould try to design tasks closer to the real world IR.

References[1] Ueda, Y., Oka, M., Koyama, T. and Miyauchi, T.
Toward the "At-a-glance" Summary: Phraserepresentation Summarization Method. Proceedings of
COLING-2000: 878-884, 2000.
[2] Miyauchi, T., Oka, M. and Ueda, Y. Key-relation
technology for text retrieval. Proceedings of
SDAIR’95: 469-483, 1995.
[3] Salton, G.
Automatic Text Processing: The
Transformation, Analysis, and Retrieval of
Information by Computer. Addison-Wesley, 1989.
[4] Hand, T. F. A Proposal for Task-based Evaluation of
Text Summarization Systems. Proceedings of
ACL/EACL'97 Workshop on Intelligent Scalable Text
Summarization: 31-38, 1997.
[5] Oka, M. and Ueda, Y.
Evaluation of Phraserepresentation Summarization based on Information
Retrieval Task. Proceedings of ANLP/NAACL 2000
Workshop on Automatic Summarization: 59 - 68, 2000.
[6] Jing, H., Barzilay, R., McKeown, K. and Elhadad, M.
Summarization Evaluation Methods: Experiments andAnalysis. Intelligent Text Summarization: 51-59,
AAAI Press, 1998.
[7] Mani, I., House, D., Klein, G., Hirschman, L., Obrst,
L., Firmin, T., Chizanowski, M., and Sundheim, B.
The TIPSTER SUMMAC Text Summarization
Evaluation. Technical Report MTR 98W0000138,
MITRE Technical Report, 1998.
[8] Zechner, K. Fast Generation of Abstracts from
General Domain Text Corpora by Extracting Relevant
Sentences. Proceedings of. COLING-96: 986-989,
1996.