言語処理学会 第20回年次大会 発表論文集 (2014年3月)

Analyzing the Inﬂuence of Parsing Errors
on Pre-reordering Performance for SMT
Dan Han1,2

Pascual Martínez-Gómez2,3
Yusuke Miyao1,2
4
Katsuhito Sudoh
Masaaki Nagata4
1
The Graduate University For Advanced Studies
2
National Institute of Informatics, 3 The University of Tokyo
4
NTT Communication Science Laboratories, NTT Corporation{handan,pascual,yusuke}@nii.ac.jp{sudoh.katsuhito,nagata.masaaki}@lab.ntt.co.jp

1

Introduction

Word alignment for long distance language
pairs is problematic in state-of-the-art phrasebased statistical machine translation. Linguistically motivated reordering models have been
widely studied to conquer this challenge. One of
the most popular and eﬀective methods is called
pre-reordering, where words in sentences fromthe source language are re-arranged with the objective to resemble the word order of the target
language. There are mainly two ways to formulate re-arranging rules. One is to learn automatically from the data (Xia and McCord, 2004;Genzel, 2010); while another one is to handcraft reordering rules based on linguistic studies (Isozaki et al., 2010; Han et al., 2012; Hanet al., 2013). In both methods, syntactic information are obtained by using automatic parsers.
However, although these parsers produce parsing errors, current reordering methods do notinclude any parsing error identiﬁcation or correction mechanism. In order to improve the robustness of pre-reordering method, it is useful
to explore the relationship between parsing andreordering.
In this work, we use both empirical and descriptive approaches to analyze the eﬀects ofparsing errors on pre-reordering performance for
Chinese-to-Japanese statistical machine translation. We examine the impact of parsing errorsthat are produced by a dependency parser called
Corbit1 (Hatori et al., 2011), on a pre-reordering
framework called unlabeled dependency parsing1

http://triplet.cc/software/corbit

based pre-reordering for Chinese (DPC) (Han etal., 2013). We not only quantify the distributionof general parsing errors along with reordering
quality, but also examine the inﬂuence of concrete parsing errors on reordering.

2

Gold Data

In order to reveal how and which parsing errors inﬂuence reordering, we contrast the reordering based on error-free parse trees, which
are considered as Gold-Trees, with the reordering based on parse trees that are generated byparser, which are referred as Auto-Trees. We
select sentences from Chinese Penn Treebank
ver.7.0 (CTB-7) to build up gold data. CTB7 is a human annotated treebank that comprises sentences from ﬁve genres: broadcast
news (BN), broadcast conversations (BC), news
magazine (NM), newswire (NS), and newsgroupweblogs (NW). Since our parser was trained on
this corpus, in order to maximize the analysis
accuracy, we obtain sentences from the development set (Wang et al., 2011).
We ﬁrst randomly sampled 517 unique sentences (set-1) from all ﬁve genres. But sentencesin BC and NW tend to include faults like repetitions, incomplete sentences, corrections, andso on. We thus randomly selected another 2, 126
unique sentences (set-2) from other three genres:
NS, NM, and BN. Table 1 lists the statistics ofthe selected sentences in ﬁve genres.
To obtain Japanese references, professional
human translators translated all the sentences
in set-1 and set-2. Based on the Japanese references, a bilingual speaker reordered Chinese― 824 ―

Copyright(C) 2014 The Association for Natural Language Processing.
All Rights Reserved.

set-1
set-2
TotalAL.
Voc.

BN
100
797
897
29.8
5.5K

BC100
100
20.0
690

NM
100
578
678
33.5
5K

NS
117
751
868
28.4
5.1K

NW
100
100
25.9
972

Total
517
2, 1262, 643
29.8
9.5K

Table 1: Statistics of Selected Sentences in Five
Genres of CTB-7. AL. stands for the average
length of sentences, while Voc. for vocabulary.

sentences in set-1 to follow the word order of
their Japanese counterparts. After constructing
these two data sets, we use Penn2Malt2 (Nivre,2006) to convert CTB-7 sentences to dependency Gold-Trees. We added three new headrules for Penn2Malt converting as follows: FLR(Fillers) and DFL (Disﬂuency) head on righthand branch; INC (Incomplete sentences) follows the same head rule as FRAG (Fragment).

3Analysis Method

We ﬁrst use an empirical approach to provide
a general idea of the sensitiveness of DPC prereordering method on parsing errors, and also
test an upper bound of the reordering performance. Then, we dig more on the eﬀects ofconcrete parsing errors from the aspect of Partof-Speech tag (POS tag) to reveal what typesof parsing errors inﬂuence the most. In bothstages, we calculate Kendall’s tau (τ ) rank correlation coeﬃcient (Kendall, 1938) to evaluate
the reordering quality (Isozaki et al., 2010).
3.1

General analysis

In the ﬁrst stage, we set up two analysis scenarios since we have one gold reordered Chinese
data set and one Japanese references set, which
can be used as two benchmarks. In scenario 1,
manually reordered Chinese sentences are usedas the benchmark. We measure the word ordersimilarities which reveal the reordering quality
in two sentence pairs. One pair is composedof the benchmark sentences and the Gold-Treebased reordered sentences, while another pair iscomposed of the benchmark sentences and theAuto-Tree based reordered sentences. By comparing the measurement results, we quantify the2

http://stp.lingﬁl.uu.se/ nivre/research/Penn2Malt.html

extent of parsing errors that inﬂuence reordering. Meantime, the former measurement gives
a general idea of the upper bound of the reordering method. In scenario 2, instead of using
handcrafted reordered Chinese, Japanese references are used as the benchmark and we calculate the word order similarities in the same way
as in scenario 1.
In both scenarios, we carry out reordering
method DPC. Accordingly, there are two automatically reordered data sets that are producedby two reordering systems: Gold-Tree based
reordering system (Gold-DPC) and Auto-Treebased reordering system (Auto-DPC). AutoTrees are automatically generated by Corbit.
The baseline system uses unreordered Chinesesentences. Due to the fact that the reordering
method is identical but the Auto-Trees may contain errors, we will be able to observe reordering
diﬀerences directly caused by parsing errors.
Scenario 1
Although there are totally
517 sentences in set-1, 26 sentences were failed
during the converting from CTB-7 parsed text
to parse trees. For comparison, 491 available(Gold- and Auto-) dependency trees are used
to reorder sentences. Our ﬁrst observation on
the eﬀects of parsing errors to reordering performance is to examine word order similarities
between manually reordered Chinese sentences
and automatically reordered Chinese sentences.
Figure 1 shows the distribution of τ values of
the 491 sentences. Comparing to baseline, bothGold-DPC and Auto-DPC show higher average τ values which imply that DPC have positively reordered the Chinese sentences and improved the word alignment between Chinese andJapanese. However, comparing with Gold-DPC,
Auto-DPC has reordered more sentences withlower τ value and less sentences with high τ ,which shows that DPC is sensitive to parsingerrors. Since the sentence number of set-1 islimited, in order to enhance the conclusions, weincreased the test data by adding set-2 for further experiment in scenario 2.
Scenario 2
We merge set-1 and set2, then use Japanese references as benchmark
in this scenario. In order to calculate thevalue of Kendall’s tau between Chinese sentences and their Japanese counterparts, we use― 825 ―

Copyright(C) 2014 The Association for Natural Language Processing.
All Rights Reserved.

60%
50%

Auto-DPC

40%

Gold-DPC

30%
20%10%

oo.

0%

0.5

0.6

0.70.8Kendall's tau (τ)

0.9

1.0

Figure 1: Distribution of τ for 491 sentences.

25%

Gold-DPC

20%
15%
10%
5%

PinYin:.
Chinese:.
English:.
POS. tag:

ta1.
他.
He.
PN.

R-Chinese:
English:Japanese:

彼 (は)

0.6

0.7
0.8
Kendall's tau (τ)

0.9.

o.

.

mai3.
买.
buy.
VV.

他

书店
bookstore

去went(to)书hebook

行って本 (を)買った

本屋 (に)

le0.
了.
-ed.
AS.

o

shu1dian4.
书店...
bookstore.
NN.

shu1.
书.
book.
NN.

.
。
.
.
PU.

买了。

buy-ed。

o.

ROOT.

o.

Dependency errors by POS tag

In this stage, we deﬁne two types of dependency parsing errors given by POS tags,
namely dependent-error and head-error. As
a dependent-error, a token points to a wrong
head, whereas a head-error means that a token is wrongly recognized as a head. We collect all the POS tags of the tokens which are in-oqu4.
去.
went. (to)
VV.

ta1.
他.
He.
PN.

R-Chinese:English:他书店书

hebookstore

bookJapanese:

彼 (は)行って本 (を)

本屋 (に)

shu1dian4.
书店...
bookstore.
NN.

.

PinYin:.
Chinese:.
English:.
POS. tag:

1

MGIZA++ to obtain the alignment ﬁle, chja.A3.ﬁnal. Therefore, the comparison implies
how monotonically the Chinese sentences havebeen reordered to align with their Japanese references. There are totally 2, 164 available (Goldand Auto-) trees. Figure 2 shows the distribution of τ values and baseline system was using
unreordered Chinese sentences.
In Figure 2, disregarding the alignment accuracy of MGIZA++, both Gold-DPC and AutoDPC produced more sentences with higher τvalues against the baseline system, which indicate the same conclusion that DPC improved
word alignment as shown in scenario 1. However, comparing with Gold-DPC, Auto-DPC has
about 2% drop in 0.9 < τ <= 1, which showsthat reordering method DPC is sensitive to parsing errors. Furthermore, the performance of reordering system Gold-DPC sketches the ﬁgureof the upper bound of DPC.

o.

.

Figure 2: Distribution of τ for 2, 164 sentences.

3.2

oqu4.
去.
went. (to)VV.

o0%
0.5.

(a) Gold dependency tree
Percent of Sentences30%

Auto-DPC.

oEnglish Translation: He went to bookstore and bought a book.

35%
Baseline

ROOT.

Percent of Sentences

Baseline

mai3.
买.
buy.
VV.

o.

le0.
了.
-ed.
AS.

shu1.
书.
book.
NN.

.
。
.
.
PU.

买了-ed

去went(to)

。

buy

買った。

(b) Possible wrong dependency parse tree

Figure 3: Example for calculating parsing errors
in terms of POS tag.

volved in either parsing error in sentences, andthen calculate the proportion of diﬀerent POS
tags in terms of the parsing error types. Figure 3 shows an example of a gold parse tree
(Fig. 3a) with an error-containing tree (Fig. 3b).
By comparing these two parse trees, tokens of“他 (he, PN)”, “去 (went, VV)”, “书店 (bookstore, NN)”, “买 (buy, VV)”, and “。(., PU)”are dependent-errors, since they point to wrong
heads. “去 (went, VV)” and “买 (buy, VV)” arehead-errors since they are wrongly recognized asheads. Therefore, in this sentence, dependenterror includes twice of VV and once of PN, NN,
PU; head-error includes twice of VV.
With the objective of observing the eﬀects ofparsing errors on reordering, we group sentences
based on their τ values according to Figure 2,and plot the distributions of POS tags that areinvolved in parsing errors for dependent-error
and head-error separately. Figure 4a gives thedistribution of three most frequent POS tags
that point to wrong heads, and Figure 4b shows― 826 ―

Copyright(C) 2014 The Association for Natural Language Processing.
All Rights Reserved.

80%

VV
PU
NN
2 per. Mov. Avg. (VV)2 per. Mov. Avg. (PU)
2 per. Mov. Avg. (NN)

Percent of Errors

70%

60%
50%
40%

30%20%
10%0%0.1

0.2

0.3

0.4
0.5
0.6
Kendall's tau (τ)

0.7

0.8

0.9

1

Percent of Errors(a) Distribution of top three dependent-error POS tags
and their trend lines.
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%0%

VV
NN
2 per. Mov. Avg. (VV)2 per. Mov. Avg. (NN)

References
0.1

0.20.30.4
0.5
0.6
Kendall's tau (τ)

0.7

0.8

0.9

1(b) Distribution of top two head-error POS tags andtheir trend lines.

Figure 4: Distribution of top inﬂuential error
POS tags for two types of errors.

the distribution of two most frequent POS tags
that are wrongly recognized as heads. VV represents most of verbs except a few exceptions,such as predicative adjective and copula. NN
represents most of nouns except proper noun or
temporal noun, etc., and PU is punctuation.
Both ﬁgures exhibit that verbal tokens takethe largest proportion in both dependent-error
and head-error than other types of tokens in
low reordering performance sentences. Therefore, the parsing errors on verbal tokens inﬂuence more on the reordering quality. On theother hand, parsing errors on nouns do not inﬂuence severely on reordering. Moreover, since theproportion of PU involved in dependent-error isconsistent, we can not say whether parsing errors on punctuations are inﬂuential or not.

4

tive approaches to examine the eﬀects of different parsing errors on pre-reordering performance. We achieved three objectives that wequantiﬁed eﬀects of parsing errors on reordering, estimated upper bounds in performance ofthe pre-reordering method, and examined theeﬀects of speciﬁc parsing errors on reorderingfrom the aspect of POS tags. The current study
on exploring inﬂuential parsing errors is not exhaustive. There are more analysis possibilities,
such as parsing errors that are produced by different parsers and their eﬀects on diﬀerent prereordering models. We believe that such relationship analysis between two types of techniques can beneﬁt the improvement of both.

Conclusion

We carried out linguistically motivated analysis methods by using empirical and descrip-Dmitriy Genzel. 2010. Automatically learningsource-side reordering rules for large scale machinetranslation. In Proc. of COLING, pages 376–384.
Dan Han, Katsuhito Sudoh, Xianchao Wu, Kevin
Duh, Hajime Tsukada, and Masaaki Nagata.
2012. Head ﬁnalization reordering for Chinese-toJapanese machine translation. In Proc. of SSST-6,pages 57–66.
Dan Han, Pascual Martínez-Gómez, Yusuke Miyao,
Katsuhito Sudoh, and Masaaki Nagata. 2013.
Using unlabeled dependency parsing for prereordering for Chinese-to-Japanese statistical machine translation. In Proc. of HyTra, pages 25–33.
Jun Hatori, Takuya Matsuzaki, Yusuke Miyao, andJun’ ichi Tsujii. 2011. Incremental joint POS
tagging and dependency parsing in Chinese. InProc. of IJCNLP, pages 1216–1224.
Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada,and Kevin Duh. 2010. Head ﬁnalization: A simple
reordering rule for SOV languages. In Proc. ofWMT and Metrics MATR, pages 244–251.
Maurice G Kendall. 1938. A new measure of rankcorrelation. Biometrika, 30(1/2):81–93.
Joakim Nivre. 2006. Inductive dependency parsing.
Springer.
Yiou Wang, Jun’ichi Kazama, Yoshimasa Tsuruoka,
Wenliang Chen, Yujie Zhang, and Kentaro Torisawa. 2011. Improving Chinese word segmentation and POS tagging with semi-supervised methods using large auto-analyzed data. In Proc. ofIJCNLP-5, pages 309–317.
Fei Xia and Michael McCord. 2004. Improving astatistical MT system with automatically learned
rewrite patterns. In Proc. of COLING, pages 508–514.

― 827 ―

Copyright(C) 2014 The Association for Natural Language Processing.
All Rights Reserved.