Automatic Evaluation of Texts by Using Paraphrases
Kazuho Hirahara, Hidetsugu Nanba, Toshiyuki Takezawa

Manabu Okumura

Hiroshima City University{hirahara, nanba, takezawa}@ls.info.hiroshima-cu.ac.jp

Tokyo Institute of Technology
oku@pi.titech.ac.jp

Abstract
The evaluation of computer-produced texts has been recognized as an important research problem for automatic text summarization and machine translation. Traditionally, computer-produced texts were evaluated automatically by n-gram overlap withhuman-produced texts. However, these methods cannot evaluate texts correctly, if the n-grams do not overlap between computer-produced and human-produced texts, even though the two texts convey the same meaning. We explored the use of paraphrases for the refinement of traditional automatic methods for text evaluation. To confirm the effectiveness of our method, we
conducted some experiments using the data from the Text Summarization Challenge 2. We found that the use of paraphrases
created using a statistical machine translation technique could improve the traditional evaluation method.

Keywords: text summarization, machine translation, text evaluation, synonyms
grams of the reference translation, and counts the numberof matches. These matches are position independent. The
1. Introduction
quality of the candidate translation depends on the numThe evaluation of computer-produced texts has been recber of matches.
ognized as an important research problem for text sumROUGE-N (Lin and Hovy, 2003; Lin, 2004) is a stanmarization and machine translation. Traditionally, comdard
evaluation measure in automatic text summarization.
puter-produced texts were evaluated by n-gram overlap
Themeasurecompares the n-grams of the two summaries,
with human-produced texts (Papineni, 2002; Lin andand counts the number of matches. The measure is deHovy, 2003; Lin, 2004). However, these methods cannot
fined by the following equation.
evaluate texts correctly, if the n-grams do not overlap
between the computer-produced and human-produced
texts, even though the two texts convey the same meanCountmatch ( gramN )ing. Therefore, we explore the use of paraphrases for the
S R gramN S
refinement of traditional automatic methods for text evalROUGE  N uation.
Count ( gramN )Several evaluation methods using paraphrases were proS R gramN S
posed in text summarization (Zhon et al., 2006) and machine translation (Kauchak and Barzilay, 2006; Kanayawhere N is the length of the n-gram, gramN, and Countma, 2003; Yves and Etienne, 2005), and their effectiven-grams comatch(gramN) is the maximum number ofness was confirmed. However, these studies did not disoccurring in a candidate summary and a set of reference
cuss what paraphrases techniques gave more accurate text
summaries. Lin examined ROUGE-N with values of N
evaluation. We analyzed 318 paraphrases in texts to befrom one to four, and reported that ROUGE-N had a highevaluated, and classified them into five categories. Thencorrelation with manual evaluation when N was one or
we examined several paraphrase methods that covered
two. In our work, we focus on evaluation of computerfour of those categories (about 70% of the 318 paraphrasproduced summaries, and use ROUGE-N as a baselinees). We evaluated texts using the data of the Text Summethod for text evaluation.
marization Challenge 2. We found that our method could
improve a traditional evaluation method.
2.2. Text Evaluation Using Paraphrases
The remainder of this paper is organized as follows.
Section 2 describes related work. Section 3 describes the
Several evaluation methods using paraphrases were probenefits of paraphrases in text evaluation. Section 4 exposed in text summarization (Zhon et al., 2006) and maplains our evaluation method using paraphrases. To inchine translation (Kauchak and Barzilay, 2006; Kanayavestigate the effectiveness of our method, we conducted
ma, 2003; Yves and Etienne, 2005). Zhou et al. (2006)
some experiments, and Section 5 reports on these. We
proposed a method "ParaEval" to obtain paraphrases aupresent some conclusions in Section 6.
tomatically using a statistical machine translation (SMT)
technique. If translations of two terms X and Y are the2. Related Worksame term, then the terms X and Y are considered to beparaphrases. Based on this idea, they automatically obWe describe the related studies of "automatic evaluation
tained paraphrases from a translation model, which wasof texts" and "text evaluation using paraphrases" in Seccreated from pairs of English and Chinese sentences ustions 2.1. and 2.2., respectively.
ing the SMT technique. They then used these paraphrases
for the improvement of ROUGE-N. In our work, we also
2.1. Automatic Evaluation of Texts
use paraphrases acquired by the SMT technique as paSeveral measures for evaluating computer-produced texts
raphrase method.
have been proposed (Papineni, 2001; Lin and Hovy,
In addition to the SMT-based paraphrases, we examined
2003; Lin, 2004). BLEU (Papineni, 2001) was developed
another method for automatic acquisition of paraphrases.
as a measure of automatic evaluation for machine translaLin (1998) and Lee (1999) proposed a method for calcution. It compares the n-grams of the candidate with the n-

 
 

lating the similarity between terms, called "distributional
similarity". The underlying assumption of their approachis that semantically similar words are used in similar contexts. Therefore, they define the similarity between two
terms as the amount of information contained in thecommonality between the terms, divided by the amountof information in the contexts of the terms. In our work,
we use "distributional similarity" as a method for acquiring paraphrases.

3. The Benefits of Paraphrases in Text Evaluation
To investigate the benefits of paraphrases in text evaluation, we compared multiple summaries created from thesame text.

3.1. Data
In this investigation, we used 30 Japanese editorials1 from
the Mainichi newspaper databases of 1998 and 1999. For
each editorial, we asked 10 human subjects to create abstract-type summaries for a summarization ratio of 20%,
which were produced to try to obtain the main idea of theeditorial without worrying about sentence boundaries. We
compared these summaries and obtained 318 paraphrases.

3.2. Paraphrases in Text Evaluation
We classified the 318 paraphrases into the following fivecategories.
A) Synonymous expressions with different Japanesecharacters
The senses of two expressions are the same, but they
were expressed with different Japanese characters. There
were 64 (20.1%) paraphrases in this category.
B) Word-level synonymous expressions
Two words have the same sense, but are different, such as"時間" (time) and "時" (moment). There were 78 (24.5%)
paraphrases in this category. Synonym dictionaries are
required for paraphrases in this category.
C) Phrase-level synonymous expressions
Two phrases have the same sense, but are expressed differently, such as "切り離せない" (cannot be divided)and "繋がっている" (linked to each other). There were38 (11.9%) paraphrases in this category. Phrase-level
synonym dictionaries are required for paraphrases in thiscategory.
D) Clause-level synonymous expressions
Two clauses have the same sense, but are expressed differently, such as "X がなければ Y はできなかった" (If it
were not for X, Y would not succeed.) and "X により Yができた" (Y succeeded because of X). Changing voices
and transitive/intransitive alternations are also classified
in this category. There were 39 (12.3%) paraphrases inthis category. Clause-level synonym dictionaries or moresophisticated paraphrasing techniques are required forparaphrases in this category.
E) Other paraphrases
We can recognize that two expressions are the same
meaning by guessing from their contexts. There were 99(31.2%) paraphrases in this category. Taking account of1

These editorials were used in Text Summarization Challenge (Fukushima et al., 2002), which is an evaluation
workshop of text summarization, conducted in the
NTCIR workshop.

paraphrases in this category is very difficult using current
natural language processing technologies.
Among these paraphrases, we focused on categories A,
B, C, and D, and examined several paraphrase resources
for these four categories.

4. An Automatic Method of Text Evaluation using Paraphrases
In this section, we describe our text evaluation method
using paraphrases. In Section 4.1., we describe the procedure for our method. In Section 4.2., we explain several
paraphrase methods for categories A, B, C, and D.

4.1. Procedure for Text Evaluation
We evaluated texts using the following procedure, which
resembles Zhou's ParaEval (Zhou et al., 2006).
Step 1: Search using a greedy algorithm to find (C)phrase-level or (D) clause-level paraphrases matches.
Step 2: The non-matching fragments from Step 1 are thensearched using a greedy algorithm to find (A) paraphrases
using different Japanese characters or (B) word-level
paraphrases or synonym matches.
Step 3: Search by literal lexical unigram matching on theremaining text.
Step 4: Count the agreed words in a reference summaryfrom Steps 1, 2, and 3, and output the Recall value for thereference summary as an evaluation score.

4.2. Paraphrase Methods
We used the following four paraphrase methods forsummary evaluation.
SMT (automatic): Paraphrases using the statisticalmachine translation (SMT) technique.
DS (automatic): Paraphrases using the distributional
similarity method.
Word (manual): WordNet dictionary.
NTT (manual): NTT Goi-Taikei dictionary.

In the following, we explain the details of each paraphrase method.
Paraphrases using the statistical machine translation
technique (SMT)
If translations of two expressions X and Y are the same
expression, then the expressions X and Y are considered
to be paraphrases. Therefore, we constructed a translation
model from 150,000 pairs of English-Japanese sentences
automatically extracted (Utiyama and Isahara, 2003) fromthe Yomiuri newspaper database and Daily Yomiuri using a translation tool Giza++2. In this translation model,
we deleted English-Japanese expression pairs, in which
the number of words and parts of speech of each word
were different. For example, we don’t consider a nounphrase and a verb phrase to be a paraphrase. From theremainder of the English-Japanese expression pairs, we
obtained 85,858 pairs of paraphrases.

2

http://www.fjoch.com/GIZA++.html

CategoryA

Paraphrase level
character

Number of
cases

SMT

DS

WordNetNTT

Required techniques

△

△

△

◎

character-level paraphrases◎

20.1% (64/318)

B

word

24.5% (78/318)

○

◎

C

phrase11.9% (38/318)

○

△

D

clause12.3% (39/318)△Eothersword-level paraphrases△

phrase-level paraphrases
clause-level paraphrases
paraphrases based on context analysis

31.2% (99/318)

Table 2: The classification of the paraphrases and necessary correspondence
Paraphrase
methodTarget POS

Automatic/
ManualWe automatically collected paraphrases using distributional similarity in the following procedure.

SMT

All

automatic

1.

DS

Noun, NounPhrase, Verb

automatic

Word

Noun, Verb

manualNTT

Noun, Verb, Adjective

manualParaphrases using distributional similarity

2.
3.
4.
5.
6.

Analyze the dependency structures of all sentences
in a total of 56 years of Japanese newspapers from
the Mainichi, Yomiuri, and Nikkei newspaper databases using the Japanese parser CaboCha3.
Extract noun-verb pairs that have dependency relations from the dependency trees obtained in Step 1.
Count the frequencies of each noun-verb pair.
Collect verbs and their frequencies for each noun,
creating indices for each noun.
Calculate the similarities between two indices ofnouns using the SMART similarity measure (Salton,
1971).
Obtain a list of synonymous nouns4.

In Step 2, we also extracted noun-phrase-verb pairs, instead of noun-verb pairs, and obtained a list of synonymous noun phrases using the same Steps 3 to 6.
As well as collecting verbs for each noun in Step 4, we
similarly collected nouns for each verb, and obtained alist of synonymous verbs.
WordNet dictionary
WordNet (Bond et al., 2009) is a most widely used lexical resource in natural language processing. This databaselinks nouns, verbs, adjectives, and adverbs to sets of synonyms (synsets) that are in turn linked through semantic
relations that determine word definitions. We considereda set of words linked in the same synset as paraphrases
and used them for text evaluation.
NTT Goi-Taikei dictionary
NTT Goi-Taikei is a Japanese thesaurus produced by
NTT Communication Science Laboratories. In this dictionary, a list of synonymous expressions of nouns, adjectives, and verbs with different Japanese characters is included.
The four paraphrase methods are summarized in Table 1.
Table 2 shows the relations between the four paraphrase
methods and four categories of paraphrases in Section 3.2.

Table 1: Paraphrases for text evaluation

5. Experiments
To investigate the effectiveness of our method, we
conducted several experiments.
5.1. Experimental Settings
Correct data sets
In addition to the 300 abstract-type summaries createdfrom 30 editorials by 10 human subjects, we preparedanother 300 extract-type summaries, which were produced by extracting important parts of the original texts.
Three human subjects assigned evaluation scores manually on a one-to-four scale to each of these 600 summaries.
Alternatives
We conducted examinations using 16 proposed methods
and a baseline method ROUGE-1, shown in Table 3. 5
The proposed methods used different combinations of thefour kinds of paraphrases: SMT, DS, NTT, and Word.
Experimental method
We used the top extract-type summary and the top abstract-type summary in each topic as reference summaries.
Then we conducted the following experiments for each
topic.

53

http://chasen.org/~taku/software/cabocha/
4
For each noun, we extracted the top 20 similar nouns,
and used them for text evaluation.

We employed ROUGE-1 as a baseline method, because
ROGUE-1 obtained the best performance among a seriesof ROUGE family in this dataset (Nanba and Okumura,2006).

Combination of
Paraphrases
OurS
method
D
W
N
SD
SW
SN
DW
DN
WN
SDW
SDN
SWN
DWN
SDWN
Baseline ROUGE-1
method

SMT
(S)
○

DS
(D)

Word
(W)

NTT
(N)

○
○
○
○
○
○

○
○
○
○
○

○
○
○
○

○
○
○
○

○
○
○
○
○
○

○
○
○
○
○
○

Table 3: List of 16 proposed methods and a baseline
method.


EX-1: Evaluate nine extract-type summaries using
the top extract-type summary as reference summaries.
EX-2: Evaluate nine extract-type summaries using
the top abstract-type summary as reference summaries.
EX-3: Evaluate nine abstract-type summaries using
the top extract-type summary as reference summaries.
EX-4: Evaluate nine abstract-type summaries using
the top abstract-type summary as reference summaries.

In each experiment, evaluation scores were calculated bytaking the reference summary. We then ranked summaries by our methods and ROUGE-1, and compared themwith a manual ranking by Spearman rank-order correlation coefficients.

Combination of Extract
Abstract
Paraphrases(EX-1)(EX-2)Our
S (SMT)
0.296
0.332
methodD (DS)
0.368
0.328
W (WordNet)
0.357
0.344
N (NTT)
0.358
0.329
SD
0.319
0.355
SW
0.331
0.353
SN
0.316
0.339
DW
0.365
0.346
DN
0.331
0.370
WN
0.358
0.348
SDW
0.333
0.357
SDN
0.332
0.337
SWN
0.321
0.342
DWN
0.368
0.348
SDWN
0.314
0.332
Baseline
ROUGE-1
0.358
0.310
Table 4: Evaluation results using an extract-type reference summary
Combination of Extract
Abstract
Paraphrases(EX-3)(EX-4)Our
S (SMT)
0.346
0.386
methodD (DS)
0.322
0.367
W (WordNet)
0.327
0.397
N (NTT)
0.323
0.400
SD
0.377
0.401
SW
0.330
0.427
SN
0.394
0.426
DW
0.302
0.364
DN
0.317
0.368
WN
0.321
0.402
SDW
0.329
0.436
SDN
0.394
0.419
SWN
0.346
0.432
DWN
0.303
0.371
SDWN
0.358
0.426
Baseline
ROUGE-1
0.313
0.389
Table 5: Evaluation results using an abstract-type reference summary

5.2. Experimental Results

5.3. Discussion

We show the experimental results in Tables 4 and 5,
which show the Spearman rank-order correlation coefficients for the 17 methods using two extract-type reference summaries and two abstract-type reference summaries, respectively.
As can be seen from Table 4, all of our methods could
evaluate abstract-type summaries more accurately than
ROUGE-1. Of our 16 methods, the combination of"SDW" gave the best performance in abstract-type summaries evaluations. This combination could improve
ROUGE-1 by 0.047 (15%).
In Table 5, the combination of SN was better than
ROUGE-1 by 0.113 (36%) when evaluating extract-typesummaries, while the combination of "SDW" was betterby 0.047 (12%) when evaluating abstract-type summaries.

Effectiveness of paraphrases in text evaluation
More than half of our methods performed worse than
ROUGE-1 in the experiment Ex-1, which indicates thatparaphrases were not effective for the evaluation of extract-type summaries using extract-type reference summaries. On the other hand, most of our methods improved
ROUGE-1 in the experiments Ex-2, 3, and 4. We considered that they were valid experimental results, because
paraphrases are generally used in abstract-type summaries.
Effectiveness of SMT
In the experiment Ex-4, the combinations of "DW" and"SDW" obtained 0.364 and 0.436 of Spearman rank ordercorrelation coefficients, respectively. This indicates that
the SMT-based paraphrases contributed to improve ascore from 0.364 to 0.436. We can also confirm the effec-

tiveness of SMT-based paraphrases from "W" and "SW".
To confirm the effectiveness of the SMT-based paraphrases more precisely, we calculated Spearman rank-ordercorrelation coefficients for each topic and counted thenumber of topics that "SDW" and "SW" were superior to"DW" and "W", respectively. The result is shown in Table 6. As can be seen from Table 6, the SMT-based paraphrases are useful to improve the combination of "DW",
because the number of topics that the combination of"SDW" improved "DW" was much larger than the opposite cases. On the other hand, the combination of "SW"impaired "W" in 13 topics. In the SMT-based paraphrases,
there were cases that a term X can be paraphrased into Y,
but Y cannot be paraphrased into X. A pair of "判決"
(adjudication) and "敗訴" (unsuccessful litigation) is oneof such paraphrases. Recently, detecting such paraphrases
has studied in the field of textual entailment recognition.
In future, the SMT-based paraphrases may be improved
using techniques in the field.
Improve
Same
Impair"SDW" vs.
17 (0.56 )4 (0.13)9 (0.30)
"DW"
"W"
vs.
11 (0.37)
6 (0.20)
13 (0.43)"SW"
Table 6: The Number of Topics that the combinations of
SDW and SW Improved DW and W
Effectiveness of distributional similarity
Distributional similarity did not contribute to improve
ROUGE-1. For example, the combinations of "SW" and"SDW" in Ex-4 obtained 0.427 and 0.436 of Spearman
rank order correlation coefficients, respectively. In thiscase, the distributional-similarity-based paraphrases contributed to improve ROUGE-1 by only 0.009. In anothercase, "SDWN" impaired "SWN" from 0.432 to 0.426.
The distributional similarity collected more related terms
rather than synonyms. A pair of "イギリス" (England)and "フランス" (France) is one of such paraphrases. As
the method expresses the senses of each noun or nounphrase with a set of verbs having dependency relations intexts, it tends to collect terms that have the same properties.

6. Conclusions
We explored the use of paraphrases for the refinement oftraditional automatic methods for text evaluation. We
analyzed 318 paraphrases in texts to be evaluated, andclassified them into five categories. Then we examinedseveral paraphrase methods that covered four of thosecategories (about 70% of the 318 paraphrases). To confirm the effectiveness of our method, we conducted someexperiments using the data from the Text Summarization
Challenge 2. We found that the use of the combination of
three kinds of paraphrases (SMT, distributional similarity,and WordNet) improved the traditional evaluation method ROUGE-1 from 0364 to 0.436.

References
Bond, F., Isahara, H., Uchimoto, K., Kuribayashi, T. andKanzaki, K. (2009). Extending the Japanese WordNet. Proc. 15th Annual Meeting of the Association for
Natural Language Processing, pp. 80-83.

Fukushima, T., Okumura, M. and Nanba, H. (2002). Text
Summarization Challenge 2 / Text Summarization
Evaluation at NTCIR Workshop 3. Working Notes of
the 3rd NTCIR Workshop Meeting, PART V, pp. 1-7.
Nanba, H. and Okumura, M. (2006). An Automatic
Method for Summary Evaluation Using Multiple
Evaluation Results by a Manual Method. Proc. COLING/ACL 2006 Main Conference Poster Sessions, pp.
603-610.
Kanayama, H. (2003). Paraphrasing Rules for Automatic
Evaluation of Translation into Japanese. Proc. FirstInternational Workshop on Paraphrasing, pp. 88-93.
Kauchak, D. and Barzilay, R. (2006). Paraphrasing for
Automatic Evaluation. Proc. 2006 Human Language
Technology Conference of the North American
Chapter of the Association for Computational Linguistics, pp. 455-462.
Lee, L. (1999). Measures of Distributional Similarity.
Proc. 37th Annual Meeting of the Association for
Computational Linguistics, pp. 25-32.
Lin, C. Y. and Hovy, E. (2003). Automatic Evaluation ofSummaries Using N-gram Co-occurrence Statistics.
Proc. 4th Meeting of the North American Chapter ofthe Association for Computational Linguistics andHuman Language Technology, pp. 150-157.
Lin, C. Y. (2004). ROUGE A Package for Automatic
Evaluation of Summaries. Proc. ACL-04 Workshop“Text Summarization Branches Out”, pp. 74-81.
Lin, D. (1998). Automatic Retrieval and Clustering of
Similar Words. Proc. 36th Annual Meeting of the Association for Computational Linguistics and the 17th
International Conference on Computational Linguistics, pp. 768-774.
Papineni, K., Roukos, S., Ward, T. and Zhu, W. J. (2002)
BLEU: a Method for Automatic Evaluation of MachineTranslation. Proc40 thAnnual Meeting ofthe Association for Computational Linguistics, pp. 311-318.
Salton, G. (1971). The SMART Retrieval System - Experiments in Automatic Document Processing. Prentice-Hall, Inc., Upper Saddle River, NJ.
Utiyama, M. and Isahara, H. (2003). Reliable Measures
for Aligning Japanese-English News Articles andSentences. Proc. 41st Annual Meeting of the Association for Computational Linguistics, pp. 72-79.
Yves, L. and Etienne, D. (2005). Automatic Generationof Paraphrases to be used as Translation Referencesin Objective Evaluation Measures of Machine Translation. Proc. Third International Workshop on Paraphrasing.
Zhou, L., Lin, C. Y., Munteanu, D. S. and Hovy, E.
(2006). ParaEval Using Paraphrases to Evaluate
Summaries Automatically. Proc. 2006 Human Language Technology Conference of the North American
Chapter of the Association for Computational Linguistics, pp. 447-454.