Hiroshima City University at Evaluation Subtask
in the NTCIR-8 Patent Translation Task
Hidetsugu Nanba

Kazuho Hirahara

Toshiyuki Takezawa

Graduate School of Information
Sciences, Hiroshima City University
3-4-1 Ozukahigashi, Hiroshima
731-3194, Japan

Graduate School of Information
Sciences, Hiroshima City University
3-4-1 Ozukahigashi, Hiroshima
731-3194, Japan

Graduate School of Information
Sciences, Hiroshima City University
3-4-1 Ozukahigashi, Hiroshima
731-3194, Japan

ABSTRACTThe evaluation of computer-produced texts is an importantresearch problem for automatic text summarization and machinetranslation. Traditionally, computer-produced texts were
evaluated automatically by n-gram overlap with human-produced
texts. However, these methods cannot evaluate texts correctly, if
the n-grams do not overlap between computer-produced andhuman-produced texts, even though the two texts convey the same
meaning. We explore the use of paraphrases for the refinement oftraditional automatic methods for text evaluation. In our previous
work, we devised an evaluation method for text summarization
using multiple paraphrase methods. Our goal in NTCIR-8 is toconfirm the effectiveness of our method for machine translation.
We evaluated 1200 computer-produced translations by six
proposed methods and two baseline methods, and confirmed theeffectiveness of our methods.

Categories and Subject Descriptors
H.3.3 [Information Search and Retrieval]: Search process
H.3.4 [Systems and Software]: Performance evaluation
H.3.5 [Online Information Services]: Data sharing

General Terms
Measurement, Performance, Experimentation

Keywords
text evaluation, paraphrase, machine translation

1. INTRODUCTION
The evaluation of computer-produced texts is an importantresearch problem for text summarization and machine translation.
Traditionally, computer-produced texts were evaluated by n-gram
overlap with human-produced texts (Papineni, 2002; Lin and
Hovy, 2003; Lin, 2004). However, these methods cannot evaluate
texts correctly, if the n-grams do not overlap between thecomputer-produced and human-produced texts, even though thetwo texts convey the same meaning. Therefore, we explore the useof paraphrases for the refinement of traditional automatic methods
for text evaluation.
Several evaluation methods using paraphrases have been proposedin text summarization (Zhou et al., 2006) and machine translation(Kauchak and Barzilay, 2006; Kanayama, 2003; Yves and Etienne,2005), and their effectiveness has been confirmed. In our previous
work, we also proposed an evaluation method for text

summarization using multiple paraphrase methods (Hirahara et al.,2009). Our goal in NTCIR-8 (Fujii et al., 2010) is to confirm theeffectiveness of the method in machine translation.
The remainder of this paper is organized as follows. Section 2
describes related work. Section 3 explains our evaluation method
using paraphrases. To investigate the effectiveness of our method,
we conducted some experiments, and we report on these in
Section 4. We present some conclusions in Section 5.

2. RELATED WORKWe describe the related studies of "automatic evaluation of texts"
and "text evaluation using para-phrases" in Sections 2.1 and 2.2,
respectively.

2.1 Automatic Evaluation of Texts
Several measures for evaluating computer-produced texts havebeen proposed (Papineni, 2001; Lin and Hovy, 2003; Lin, 2004).
BLEU (Papineni, 2001) was developed as a measure of automatic
evaluation for machine translation. It compares the n-grams of thecandidate with the n-grams of the reference translation, andcounts the number of matches. These matches are position
independent. The quality of the candidate translation depends on
the number of matches.
ROUGE-N (Lin and Hovy, 2003; Lin, 2004) is a standard
evaluation measure in automatic text summarization. The measurecompares the n-grams of the two summaries, and counts thenumber of matches. The measure is defined by the following
equation:  Count ( gram )ROUGE  N   Count ( gram )S R gramN S

S R gramN S

match

N

N

where N is the length of the n-gram, gramN, andCountmatch(gramN) is the maximum number of n-grams cooccurring in a candidate summary and a set of reference
summaries. Lin examined ROUGE-N with values of N from one
to four, and reported that ROUGE-N had a high correlation withmanual evaluation when N was one or two. In our work, we focus
on evaluation of computer-produced translations, and use
ROUGE-1 as a baseline method.

2.2 Text Evaluation Using Paraphrases
Several evaluation methods using paraphrases have been proposedin text summarization (Zhou et al., 2006, Hirahara et al., 2009)and machine translation (Kauchak and Barzilay, 2006; Kanayama,
2003; Yves and Etienne, 2005). Zhou et al. (2006) proposed amethod "ParaEval" to obtain paraphrases automatically using astatistical machine translation (SMT) technique. If translations of
two terms X and Y are the same term, then the terms X and Y areconsidered to be paraphrases. Based on this idea, the researchers
automatically obtained paraphrases from a translation model, theparaphrases were created from pairs of English and Chinese
sentences using the SMT technique. They then used theseparaphrases for the improvement of ROUGE-N. In our work, we
also use paraphrases acquired by the SMT technique as aparaphrase method.
In addition to the SMT-based paraphrases, Hirahara (2009)
examined other three paraphrase methods: distributional similarity(Lin, 1998; Lee, 1999), WordNet dictionary, and NTT GoiTaikei
dictionary (Hirahara et al., 2009), and experimentally confirmed
the effectiveness of their method for evaluating sum-maries
written in Japanese. In our work, we applied Hirahara's method tothe evaluation of computer-produced translations written in
English.

3. AN AUTOMATIC EVALUATION OFTEXTS USING PARAPHRASESIn this section, we describe our text evaluation method using
paraphrases based on Hirahara's method (Hirahara et al. 2009). In
Section 3.1, we de-scribe the procedure for our method. In
Section 3.2, we explain two paraphrase methods.

(Fujii et al., 2008) using the translation tool Giza++ 1 . In thistranslation model, we deleted English-Japanese expression pairs,
in which the number of words and parts of speech of each word
were different. For example, we do not consider a noun phrase to
be a paraphrase of a verb phrase.

3.2.2 WordNet dictionary (WN)
WordNet 2 is a very widely used lexical resource in natural
language processing. This database links nouns, verbs, adjectives,and adverbs to sets of synonyms (synsets) that are linked in turnthrough semantic relations that determine word definitions. We
considered a set of words linked in the same synset as paraphrases
and used them for evaluation.

4. EVALUATION
4.1 Experimental Method
4.1.1 Data
We used 1200 English sentences, which were translated from 100
Japanese sentences by 12 machine translation systems (Fujii et al.,2008).

4.1.2 Alternatives
We examined the following six proposed methods and twobaseline methods. Here, "Tagger" indicates that all words in eachtranslation were lemmatized by the part-of-speech tagging tool
TreeTagger3.

3.1 Procedure for Text Evaluation

Our methods

We evaluated texts using the following procedure, which
resembles Zhou's ParaEval (Zhou et al., 2006).



HCU-3 (S+T): ROUGE+SMT+Tagger



HCU-4 (S): ROUGE+SMT



HCU-5 (W+T): ROUGE+WN+Tagger

Step 2: The non-matching fragments from Step 1 are thensearched using a greedy algorithm to find word-level paraphrases
or synonym matches.

HCU-6 (W): ROUGE+WN



HCU-7 (SW+T): ROUGE+SMT+WN+Tagger

Step 3: Search by literal lexical unigram matching on theremaining text.

HCU-8 (SW): ROUGE+SMT+WN

Step 4: Count the agreed words in a reference translation from
Steps 1, 2, and 3, and output the Recall value for the referencetranslation as an evaluation score.



HCU-1 (base+T): ROUGE+Tagger



HCU-2 (base): ROUGE

3.2 Paraphrase Methods

4.1.3 Evaluation

We used the following two paraphrase methods for evaluation ofcomputer-produced translations.

In the following, we explain the details of each paraphrase method.

In each experiment, evaluation scores were calculated by taking
the reference translation. We then ranked the 12 computerproduced translations by our methods and baseline methods, andcompared them with manual ranking4 using Spearman rank-ordercorrelation coefficients and Pearson's correlation coefficient. The
details of the data and the evaluation procedure were described in
the overview paper (Fujii et al., 2010).

3.2.1 Paraphrases using the SMT technique1

http://www.fjoch.com/GIZA++.html

2

http://wordnet.princeton.edu/

3

http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/

4

Computer-produced sentences were ranked in terms of adequacy
and fluency.

Step 1: Search using a greedy algorithm to find phrase-level orclause-level paraphrase matches.



SMT (automatic): Paraphrases using the SMT technique.



WN (manual): WordNet dictionary.

If translations of two expressions X and Y are the same
expression, then the expressions X and Y are considered to beparaphrases. Therefore, we constructed a translation model from1,800,000 pairs of English and Japanese sentences automaticallyextracted from patent documents published during 1993-2000

Baseline methods

4.2 Results and Discussion
The experimental results are shown in Tables 1-4. In thefollowing, we discuss these results.

Effect of lemmatization
As can be seen from Tables 1 and 3, the values of HCU-1, 3, 5,
and 7 in the row of "ALL" are higher than those of HCU-2, 4, 6,
and 8, respectively. The former four methods used lemmatization,and this indicates that lemmatization is effective in our evaluation.

Effect of paraphrasing
The WordNet dictionary is considered to be a useful paraphrase
method, because HCU-5 (W+T) is the only method that
performed better than two baseline methods in the evaluation ofadequacy (the row of "ALL" in Tables 1 and 3). Although, SMTbased paraphrases could also improve baseline methods in theevaluations of several systems (e.g., HCU-5 for system 2 in Table1), the overall performances of our methods using the SMT-based
paraphrases (HCU3, 4, 7, and 8) was worse than that of the twobaseline methods.

Effect of our methods on fluency
Our methods performed worse than the baseline methods in theevaluation of fluency, because our methods were originallydeveloped for the evaluation of text summarization. Traditionally,
the creation of extract-type summaries has been considered an
important research problem in text summarization, andresearchers in this field have focused on the evaluation ofsummaries in terms of adequacy using word-level matches. In ourevaluation procedure, we also employed word-level matches in
Steps 2 and 3, which we described in Section 3.1. If we employ
combinations of n-gram matches, such as BLEU (Papineni, 2002)
instead of word-level matches, our methods might be improved in
the evaluation of fluency.

5. CONCLUSIONS
We participated in the evaluation subtask in the NTCIR-8 Patent
Translation Task. We constructed six proposed methods using
paraphrase methods and compared them with two baseline
methods. From the experimental results, we confirmed that one ofour methods HCU-5, which used the WordNet dictionary as aparaphrase method, was an improvement over the baseline
methods.

6. REFERENCES[1] Fujii, A., Utiyama, M., Yamamoto, M., Utsuro, T., Ehara, T.,
Echizen-ya, H., and Shimohata, S. 2010. Overview of the
Patent Translation Task at the NTCIR-8 Workshop. In Proc.
8th NTCIR Workshop Meeting on Evaluation of Information

Access Technologies: Information Retrieval, Question
Answering and Cross-lingual Information Access.
[2] Fujii, A., Utiyama, M., Yamamoto, M., and Utsuro, T. 2008.
Overview of the Patent Translation Task at the NTCIR-7
Workshop. In Proc. 7th NTCIR Workshop Meeting on
Evaluation of Information Access Technologies: Information
Retrieval, Question Answering and Cross-lingual
Information Access, 389-400.
[3] Hirahara, K., Nanba, H., Takezawa, T., and Okumura, M.
2009. Automatic Evaluation of Texts by Using Paraphrase.
In Proc. 4th Language & Technology Conference. LTC'09.
370-374.
[4] Kanayama, H. 2003. Paraphrasing Rules for Automatic
Evaluation of Translation into Japanese. In Proc. FirstInternational Workshop on Paraphrasing, 88-93.
[5] Kauchak, D. and Barzilay, R. 2006. Paraphrasing for
Automatic Evaluation. In Proc. 2006 Human Language
Technology Conference of the North American Chapter ofthe Association for Computational Linguistics, 455-462.
[6] Lee, L. 1999. Measures of Distributional Similarity. In Proc.
37th Annual Meeting of the Association for Computational
Linguistics, 25-32.
[7] Lin, C. Y. and Hovy, E. 2003. Automatic Evaluation ofSummaries Using N-gram Co-occurrence Statistics. In Proc.
4th Meeting of the North American Chapter of the
Association for Computational Linguistics and Human
Language Technology, 150-157.
[8] Lin, C. Y. 2004. ROUGE A Package for Automatic
Evaluation of Summaries. In Proc. ACL-04 Work-shop “TextSummarization Branches Out”, 74-81.
[9] Lin, D. 1998. Automatic Retrieval and Clustering of Similar
Words. In Proc. 36th Annual Meeting of the Association for
Computational Linguistics and the 17th International
Conference on Computational Linguistics, 768-774.
[10] Papineni, K., Roukos, S., Ward, T., and Zhu, W. J. 2002.
BLEU: a Method for Automatic Evaluation of MachineTranslation. In Proc. 40th Annual Meeting of the Association
for Computational Linguistics, 311-318.
[11] Yves, L. and Etienne, D. 2005. Automatic Generation of
Paraphrases to be used as Translation References in
Objective Evaluation Measures of Machine Translation. In
Proc. Third International Workshop on Paraphrasing.
[12] Zhou, L., Lin, C. Y., Munteanu, D. S. and Hovy, E. 2006.
ParaEval Using Paraphrases to Evaluate Summaries
Automatically. In Proc. 2006 Human Language Technology
Conference of the North American Chapter of the
Association for Computational Linguistics, 447-454.

Table 1. Pearson's correlation coefficient in adequacysystem 1

system 2

system 3

system 4

system 5

system 6

system 7

HCU-1 (base+T)

0.36220.1091

0.32240.25920.17180.2752

0.4695

HCU-2 (base)0.29930.07880.25510.24610.18540.2322

0.3462

HCU-3 (S+T)

0.32030.1116

0.38970.16690.13120.19930.4424

HCU-4 (S)0.27180.04970.25560.17420.14050.19340.3431

HCU-5 (W+T)

0.3586

0.11700.29930.26370.19330.27600.4181

HCU-6 (W)

0.28800.07660.21790.2556

0.19490.23650.3112

HCU-7 (SW+T)

0.30430.12440.36000.16350.17950.19230.3948

HCU-8 (SW)

0.25380.04580.20570.17180.16510.1786

0.3185system 10

system 11

system 12

Avg.

All

system 8

system 9

HCU-1 (base+T)

0.24000.35640.29050.37590.35830.29920.2463

HCU-2 (base)0.25460.44850.19790.24570.37100.26340.1977

HCU-3 (S+T)

0.18700.36340.21430.28580.33150.26190.2211

HCU-4 (S)0.16660.47710.15390.1781

0.34840.22940.1802

HCU-5 (W+T)

0.22280.38280.29330.3436

0.39350.29680.2507

HCU-6 (W)

0.23640.46450.19330.21870.39020.25700.1990HCU-7 (SW+T)

0.16070.36690.22400.26280.37700.25920.2217

HCU-8 (SW)

0.1502

0.46830.15430.16080.38950.22190.1772

Table 2. Pearson's correlation coefficient in fluency
system 1

system 2

system 3

system 4

system 5

system 6

system 7

HCU-1 (base+T)

0.34510.16820.37510.14340.27110.16420.4429HCU-2 (base)0.30740.03460.32080.22950.31080.15580.3947

HCU-3 (S+T)

0.28020.16350.3811

0.04120.2541

0.09530.3998

HCU-4 (S)0.2621

0.06440.25720.13950.28650.12810.3784

HCU-5 (W+T)

0.31990.14720.35590.12250.27240.15780.3931

HCU-6 (W)

0.28340.0546

0.28820.20830.31000.15720.3591

HCU-7 (SW+T)

0.25380.16400.34190.03840.2807

0.09450.3558

HCU-8 (SW)0.24000.06020.21310.13020.29330.11650.3474system 10

system 11

system 12

Avg.

All

system 8

system 9

HCU-1 (base+T)

0.11530.27350.15940.29450.37750.26080.2285

HCU-2 (base)0.16850.28610.13570.23110.2941

0.23910.1976

HCU-3 (S+T)

0.06370.26040.06640.20180.38030.2156

0.1949HCU-4 (S)0.10950.30150.05440.15850.28900.20240.1711

HCU-5 (W+T)

0.11490.32080.16260.25150.39360.25100.2243

HCU-6 (W)

0.15410.31110.12680.19370.29560.22850.1898

HCU-7 (SW+T)

0.06960.29080.08130.15770.39720.21050.1923

HCU-8 (SW)

0.10680.30130.07180.11950.30280.19190.1639

Table 3. Spearman's rank correlation coefficient in adequacysystem 1

system 2

system 3

system 4

system 5

system 6

system 7

HCU-1 (base+T)

0.31540.11280.33800.22310.14910.2296

0.4561

HCU-2 (base)0.24040.1276

0.23670.20400.13750.1701

0.3282

HCU-3 (S+T)

0.28110.11120.40120.16320.10630.16520.4338

HCU-4 (S)0.21500.05930.23470.13730.06920.13150.3155

HCU-5 (W+T)

0.32810.11210.31210.23140.15770.2342

0.3946

HCU-6 (W)0.23340.11550.19640.23710.13280.1696

0.2977

HCU-7 (SW+T)

0.27830.1101

0.36180.14560.14480.13640.3738

HCU-8 (SW)0.19230.04100.18180.13570.08380.10860.2934system 10

system 11

system 12

Avg.

All

HCU-1 (base+T)

system 80.2324system 90.24390.24270.35350.35780.27120.2234

HCU-2 (base)0.28780.37450.16650.22670.35380.23780.1654

HCU-3 (S+T)

0.16310.28650.21150.2456

0.3616

0.24420.2065

HCU-4 (S)0.15810.40120.11990.15750.34750.19550.1411

HCU-5 (W+T)

0.2097

0.29630.26320.29890.40720.27050.2274

HCU-6 (W)

0.2786

0.40500.17270.20130.37190.23430.1673

HCU-7 (SW+T)

0.15440.30460.22380.22710.41820.23990.2042

HCU-8 (SW)

0.14380.40270.11830.14490.37770.18530.1363

Table 4. Spearman's rank correlation coefficient in fluency
system 1

system 2

system 3

system 4

system 5

system 6

system 7

HCU-1 (base+T)

0.30710.17500.37830.09040.25160.13700.4445

HCU-2 (base)0.24420.00780.25810.21670.24180.11920.3655

HCU-3 (S+T)

0.23280.18750.3846

0.03430.22690.06820.4009HCU-4 (S)0.21320.02960.18610.11620.21320.0844

0.3356

HCU-5 (W+T)

0.29310.14810.3676

0.09440.23170.12690.3925

HCU-6 (W)

0.21290.02820.23050.20980.2361

0.0931

0.3287

HCU-7 (SW+T)

0.20610.18530.34060.02940.2356

0.05250.3448

HCU-8 (SW)0.19250.03100.13710.12050.20240.0507

0.2969system 10

system 11

system 12

Avg.

All

system 8

system 9

HCU-1 (base+T)

0.13280.23640.11010.30390.41550.24860.2126

HCU-2 (base)0.23830.24320.12870.2206

0.31340.21650.1705

HCU-3 (S+T)

0.07760.23250.05710.19480.42350.21010.1868

HCU-4 (S)0.14650.24600.04170.14850.31670.17310.1400

HCU-5 (W+T)

0.12990.29680.13440.24940.41420.23990.2064

HCU-6 (W)

0.21620.27820.12610.19480.28530.20330.1596

HCU-7 (SW+T)

0.08470.25730.08360.14430.4222

0.19890.1786

HCU-8 (SW)

0.12570.2516

0.06890.11290.29600.15720.1265