A machine translation system combining rule-based machinetranslation and statistical post-editing

Terumasa EHARA
Yamanashi Eiwa College
999, Yokone, Kofu, Yamanashi, JAPANeharate ＠ yamanashi-eiwa.ac.jp

The distortion limit in the tuning process and decoding process for the SPE part is set to 1, because
both the source and target languages on the SPEpart are the same.

Abstract
System architecture, experimental settings and evaluation results of the EIWA
in the WAT2014 Japanese to English (jaen) and Chinese to Japanese (zh-ja) tasks
are described. Our system is combining
rule-based machine translation (RBMT)and statistical post-editing (SPE). Evaluation results for ja-en task show 19.86
BLEU score, 0.7067 RIBES score, and22.50 human evaluation score. Evaluation
results for zh-ja task show 33.57 BLEU
score, 0.8114 RIBES score, and 15.00 human evaluation score.

1

Evaluation andtranslation selection

Post edited
target sentence

Statistical post editing(SPE)

Rule-based MT from S to T(RBMT)translation phasetraining phase

Grammar
Dictionaries

Introduction

One of the architectures of combining rule-based
technique and statistical technique in the machinetranslation field is combining a rule-based machine translation (RBMT) and a statistical postediting (SPE) (Dugast et al., 2007; Simard et al.,2007; Ehara, 2007).
The RBMT part translates source documents totarget documents using rule-based machine translation. The SPE part automatically post-edits theoutput of the RBMT part to be more accurate target documents.

2Machine translated
target sentence

Source sentence

Outputtarget sentence

Translation model(TM)Language model(LM)TM training partLM training part

Data selection

Rule-based MT from S to T(RBMT)

Sentence aligned
Source part of
the training data

Target part of
the training data

Figure 1: Basic system architecture for theja-en task
2.1

System architecture and experimental
setting for the ja-en task

Translation model adaptation

Translation model (TM) training for SPE partdoes not use whole training data for ja-en task
(3,008,500 sentence pairs) but selected data
adapted to the development data or the test data.
For the SPE for the development data, 134,634
sentence pairs are used and for the SPE for the testdata, 127,925 sentence pairs are used. The methodof this filtering differ from the previous method(Ehara, 2010). The new method is as follows: At

Our basic system architecture for ja-en task isshown in Figure 1 that is the same in the previousworks (Ehara, 2007; Ehara, 2010; Ehara, 2011;
Ehara, 2013). We use commercial based translation software for the RBMT part and the phrase
based Moses (Koehn et al., 2003) for the SPE part.
50

Proceedings of the 1st Workshop on Asian Translation (WAT2014), pages 50‒54,Tokyo, Japan, 4th October 2014.

a first step, we select training sentences which include a low frequency word in the development
sentences (or the test sentences). The low frequency word means the count of such word in thetraining sentences are less than a threshold (we set300). The word “ＳＡＲＳ”, which occurs 13
times in the development set, occurs 189 times inthe training set. So, it is a low frequency word in
the development data. On the other hand, the word“化学”, which occurs 29 times in the development
set, occurs 40,964 times in the training set. So, itis not a low frequency word in the development
data. At a second step, we add training sentences
which include a word in the development sentences that is not in the set made by the first step.
However, we do not add any sentences by the second step. For example, the word “標題” occurs 19
times in the development set and is not includedin the set made by the first step but it occurs notimes in the whole training set. We make a TMfrom this adapted training data. We refer to thisTM as TM1. The training sentence pairs for theTM1 for the test data are selected by the similar
method.
We make another TM. Adding to the training
sentences of TM1, we add additional training sentences from the RBMT outputs of the development data (or the test data). This means SPE partmakes easier not to rewrite RBMT outputs. We
refer to this TM as TM2. The training data for theTM2 of the development set includes 134,634
sentence pairs and the training data for the TM2of the test set includes 129,737 sentence pairs.
2.2

3

For the zh-ja task, we use the same base system ofja-en task. But, we do not use TM adaptation by
the data selection part in the Figure 1 and translation selection by the translation evaluation and selection part in the Figure 1, as the ja-en task. We,however, add some new things to our base system:
・To use a user dictionary in the RBMT part.
・To adapt a language model.
・To make additional sentence segmentationby the full-width space position.
3.1

User dictionary

A user dictionary for the RBMT part is made fromthe zh-ja training data. We make the phrase tablefrom the training data using phrase-based Moses
and extract high scored phrase pairs. We filter
these selected phrase pairs using part of speech.
As a result, we get the user dictionary having
1,246,274 entries all of which are nouns.
3.2

Translation model

We use whole 672,315 zh-ja training sentencepairs to make the TM for the SPE part. After additional sentence segmentation described the section 3.4, we get 679,292 training segment pairs.
We do not use training data filtering like in the jaen task.
3.3

Language model adaptation

For LM training, in addition to zh-ja training data,
we add more sentences from the Japanese side of
the ja-en training data. The method for adding islike TM adaptation for the ja-en task. For example,LM for the development set is adapted as follows.
In the first step, we select training sentences of jaen task (only Japanese side) which include a low
frequency word in the RBMT output of the Chinese side of the development data. The low frequency word means the count of such word in thetraining sentences are less than a threshold (we set300). In a second step, we add training sentencesof ja-en task (only Japanese side) which include aword in the RBMT outputs of the Chinese side ofthe development data and that is not in the set
made by the first step. LM adaptation for the test
set is similar.
As the result, we get 1,109,647 Japanese sentences to train LM for the development set and1,092,850 Japanese sentences to train LM for theLanguage model

For language model (LM) training for SPE partuses 2,000,000 sentences from train-1.txt andtrain-2.txt. We use SRILM (Stolcke, 2002) with
5-grams order and modified Kneser-Ney discounting.
2.3

System architecture and experimental
setting for the zh-ja task

Translation selection

We use a translation selection method with aquality estimation method described in Ehara(2013). Translation candidates are from threetranslations: RBMT output and two SPE outputs.
One SPE method uses TM1 and the other SPE
method uses TM2, which are described in 2.1. The
bonus score (Ehara, 2013) for two SPE outputs areset to 0.1. As the result, 126, 850 and 836 outputs
are selected from the outputs of RBMT, SPE with
TM1 and SPE with TM2, respectively.

51

test set. By this adaptation, mean value of the perplexity for the Japanese side of the development
set drops from 66.72 to 59.69.
3.4

zh-ja task are 197, 66 and 137, respectively. An
example of EIWA’s win case is shown in Table21.
The EIWA’s result is very similar to the reference.

Additional sentence segmentation

この発見は角野の推測に新たな根拠を与えていると考えられる。
この発見により与えられると考えられるbase line
対角野の推測のために新しい。
このことは角野の推測に新たな根拠をreference与えるものと考えられる。
可以认为这个发现对角野的推测给予「北方的水生食物连锁污染数据库 调查工具」
source新的根据。
「北方の水食物連鎖汚染データベース 調査道具」
EIWA

Several sentences in the zh-ja task include a fullwidth space character (" ") which is used to separate segments. For example, the following sentences in the development set include such fullwidth space.

Table 2: Example of EIWA’s win case

So, we make an additional segmentation to thetraining, development and test sets. When thenumber of the full-width spaces between Japaneseside and Chinese side in training or development
sentence pair is the same, the sentences are segmented at the full-width space position. For thetest set, we make this segmentation at all fullwidth space position in Chinese sentences.

4Some examples of EIWA’s loss cases areshown in Table 3.
灯油漏洩を防止処置を扱った研究は非常に多く，化学処理方法の検討中の段階であった。
石油流出の防止処理対策についてのbase line 研究は非常に多く，化学の防犯処理方法は研究段階にある。
流出油の防除手法についてはかなり研
reference 究が進んでいるが，ケミカルの防除手法はまだ調査研究の途中段階である。
关于石油泄漏的防范处理措施的研究
source
非常多，通过化学的防范处理方法正处在研究阶段。
EIWA

Issues for context-aware machinetranslation

TM adaptation described in 2.1 and LM adaptation described in 3.3 can be considered contextaware machine translation. Our method can be extended to document level adaptation. However,
we do not make experiments that use non adapted
models or adapted models not with the test setlevel but the test document level. One shortcoming of our adaptation method is that it needs retraining of TM and/or LM adapted to the input
document, which is time consuming.

5(a) Lexical mistranslation
着目し，地盤材料として利用する人工石
炭灰粒子として，すべての粒子の特性EIWAに基づいて，粒子の形状，物理的特性と単粒子の粉砕特性について検討を行った。
個々の粒子の特性に基づいて，地盤材料としての利用の人工石炭灰粒子に着base line 目し，粒子の形状，物理特性および単個の粒子の破砕特性について検討した。
造粒石炭灰を地盤材料として利用することに着目し，粒子個々の特性に基づいreferenceて，粒子の形状や物理的特性および単粒子破砕特性について検討した。
着眼于作为地基材料予以利用的人造煤灰颗粒，基于每个粒子的特性，对粒
source子的形状、物理特性以及单颗粒子的破碎特性进行了讨论。

Evaluation results

Automatic evaluation results for the whole test
sentences (zh-ja: 2107, ja-en: 1812) and humanevaluation results for the selected test sentences(400) by the organizer are shown in Table 1 with
our system rank and the number of all evaluatedsystems up to September 14th.
Task
ja-en
zh-ja

BLEU
19.86 (9/27)
33.53 (12/19)

RIBES
0.7067 (5/27)
0.8114 (6/19)

HUMAN
22.50 (7/16)
15.00 (5/11)

Table 1: Evaluation results
5.1Human evaluation results(b) Syntactic mistranslation

Comparing with the baseline system, the numberof wins, ties and losses in human evaluation in theTable 3: Examples of EIWA’s loss cases

1

All sample sentences (source and reference sentences) in this document are provided by the AsianScientific Paper Excerpt Corpus.
52

In Table 3 (a), EIWA’s translation of the Chinese expression “石油泄漏” is “灯油漏洩” compared with the baseline translation “石油流出”.
This case has lexical mistranslation in EIWA.
In Table 3 (b), Chinese subordinate clause including “着眼” is incorrectly parsed by EIWA’s
RBMT part. This case has syntactic mistranslation
in EIWA.

0.6

0.4

0.2

DIFF_IMPACT

5.2

tive because sentence structures of EIWA and reference is more similar than the sentence structuresof baseline and reference.

Correlation between automatic and human evaluation results

For human evaluated 400 sentences of zh-ja task,
we conduct automatic evaluations with RIBES(Isozaki et al., 2010) and IMPACT (Echizen-yaand Araki, 2007).
Here we use “human average score (HUM)”
which means the average value of scores of thethree human annotators. DIFF_RIBES is the difference of RIBES scores of EIWA’s output andbaseline output. DIFF_IMPACT, also, means thedifference of IMPACT scores of EIWA’s output
and baseline output. All these scores are in theinterval [-1,1]. The scattering graph between
HUM and DIFF_RIBES is shown in the Figure 2.
The scattering graph between HUM and
DIFF_IMPACT is shown in the Figure 3.
Pearson’s correlation coefficient between
HUM and DIFF_RIBES is 0.3618 and Pearson’scorrelation coefficient between HUM andDIFF_IMPACT is 0.3686. They are weak correlations.

-0.2

-0.4

-0.6
-1.5

-1-0.5

00.5

11.5

HUM

Figure 3: Correlation between HUM andDIFF_IMPACTAt Table 4 (b), the main part of the sentence,“Ｅｍｍａ／ＱｏＳを提案した” in the reference, is correctly translated in EIWA. So, HUM
may be 1. It is not clear that DIFF_RIBES is negative.
At Table 4 (c), baseline translation is perfectly
equal to the reference translation. So, RIBES
score of baseline is 1 then DIFF_RIBES is negative. However, two annotators evaluated EIWA’swin and one annotator evaluated baseline’s win.
The reason may be that EIWA’s translation ismore literal than baseline translation. Another reason may be that the references were not shown toannotators when they made evaluation. Multiplereferences by additional translation or scramblingof reference (Isozaki et al., 2014) may make
DIFF_RIBES be smaller.

0.60.40.2

DIFF_RIBES

006-0.2

System architecture, experimental setting andevaluation results of EIWA are described. We arein the middle position in human and automatic
evaluation. One of the future issues is to improveparsing accuracy in the RBMT part. Syntacticallycollapsed RBMT outputs cannot be recovered bythe SPE part. Other future issue is to combine
rule-based technique and statistical techniquemore tightly beyond RBMT plus SPE method.

-0.4

-0.6
-1.5

-1

-0.5

00.5

1Conclusion

1.5

HUM

Figure 2: Correlation between HUM andDIFF_RIBES
Table 4 shows examples having big difference
between HUM and DIFF_RIBES. At Table 4 (a),lexical error “暗号陽２３号” may affect HUM
score to be negative, while DIFF_RIBES is posi-53水稲のカドミウムの高吸収特性を示す品種「暗号陽２３号」を用いた。
カドミウム高吸収特性の品種「密阳２３base line号」を示すイネを採用した．
イネはカドミウム高吸収特性を示す品種
reference「密陽２３号」を用いた。
稻子采用了显示镉高吸收特性的品种
source“密阳２３号”。

Terumasa Ehara. 2007. Rule Based Machine Translation Combined with Statistical Post Editor for Japanese to English Patent Translation. Proceedings ofMachine Translation Summit XI, Workshop on Patent Translation, pages13-18.

EIWA

Terumasa Ehara. 2010. Machine translation for patentdocuments combining rule-based translation andstatistical post-editing. Proceedings of NTCIR-8Workshop Meeting, pages 384-386.

(a) HUM= -1, DIFF_RIBES=0.222

Terumasa Ehara. 2011. Machine translation system forpatent documents combining rule-based translationand statistical post-editing applied to the PatentMT
Task. Proceedings of NTCIR-9 Workshop Meeting,pages 623-628.

本稿では，ＡＬＭプランＥｍｍａ／ＱｏＳを提案し，オーバレイネットワークでは，同EIWA時に複数のビデオを伝送を実現する場合，分散制御のＱｏＳである．
Ｅｍｍａ／ＱｏＳはオーバレイネットワー
クでは，同時に複数のビデオを伝送をbase line実現する方案を提案した分散制御のＱ
ｏＳをＡＬＭ
本論文では，オーバレイネットワークにおける複数ビデオの同時配信時にＱｏＳreferenceを分散制御で実現するＡＬＭプロトコル
Ｅｍｍａ／ＱｏＳを提案した．
本文提出了ＡＬＭ方案Ｅｍｍａ／ＱｏＳ，source
在Ｏｖｅｒｌａｙ网络中，实现同时传输复数视频时分散控制ＱｏＳ。

Terumasa Ehara. 2013. Machine translation system forpatent documents combining rule-based translationand statistical post-editing applied to the NTCIR-10
PatentMT Task. Proceedings of the 10th NTCIRConference, pages 335-338.
Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada andKevin Duh. 2010. Head finalization: A simple reordering rule for SOV languages. Proceedings of theJoint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 244-251.

(b) HUM=1, DIFF_RIBES= -0.1678

Hideki Isozaki, Natsume Kouchi and Tsutomu Hirao.
2014. Dependency-based Automatic Enumerationof Semantically Equivalent Word Orders for Evaluating Japanese Translations. Proceedings of theNinth Workshop on Statistical Machine Translation,
pages 287–292.

表３に人工株式市場の各エージェントの予測の木で用いる記号を構成することを示した。
人工株式市場を構成する各エージェント
base line の予測の木に用いる記号を表３に示す．
人工株式市場を構成する各エージェント
reference の予測の木に用いる記号を表３に示す．
表３显示构成人工股份市场的各代理
source
商的预测树中使用的记号。
EIWA

Philipp Koehn, Franz J. Och and Daniel Marcu. 2003.
Statistical Phrase-Based Translation, Proceedingsof HLT-NAACL 2003, pages 48-54.
Michel Simard, Nicola Ueffing, Pierre Isabelle andRoland Kuhn. 2007. Rule-based Translation With
Statistical Phrase-based Post-editing. Proceedingsof the Second Workshop on Statistical MachineTranslation, pages 203-206.

(c) HUM=0.3333, DIFF_RIBES=-0.4873

Andreas Stolcke. 2002. SRILM - An Extensible Language Modeling Toolkit. Proceedings of the International Conference on Spoken Language Processing.

Table 4: Examples having big difference between HUM and DIFF_RIBES

ReferenceLoïc Dugast, Jean Senellart and Philipp Koehn. 2007.
Statistical Post-Editing on SYSTRAN’s Rule-BasedTranslation System. Proceedings of the Second
Workshop on Statistical Machine Translation, pages
220-223.
Hiroshi Echizen-ya and Kenji Araki. 2007. Automatic
Evaluation of Machine Translation based on Recursive Acquisition of an Intuitive Common Parts Continuum, Proceedings of the Eleventh MachineTranslation Summit (MT SUMMIT XI), pages 151158.

54