System Combination of RBMT plus SPE and Preordering plus SMT

Terumasa EHARA
Ehara NLP Research Laboratory
Seijo, Setagaya, Tokyo, JAPANeharategmail.com

RBMT and SMT conduct lexical transfer andstructural transfer simultaneously. On the other
hand, monotone SPE and monotone SMT, which
are technically the same process, conduct lexicaltransfer only. Preordering conducts structuraltransfer only.
We have made researches combining a rule
based method and a statistical method that is
RBMT plus SPE (Ehara, 2014). This year, we add
preordering plus SMT part to our system for
WAT2015. This new system is also the combination of rule based method (RBMT and preordering) and statistical method (SPE and SMT).

Abstract
System architecture, experimental settings and evaluation results of EHR groupin the en-ja, zh-ja, JPCzh-ja and JPCko-ja
tasks are described. Our system concept iscombination of a rule based method and astatistical method. System combination ofrule-based machine translation (RBMT),
RBMT plus statistical post-editing (SPE)and preordering plus statistical machinetranslation (SMT) is conducted. From themultiple outputs of three systems, candidate selection part selects the best output
by language model score. For JPCzh-ja
task devtest data translation, SPE improves BLEU score by 17.81, preordering
improves BLEU score by 1.89 and system
combination improves BLEU score by
0.26.

1

2

Our basic system architecture is shown in
Figure 1.

Input
RBMT

Introduction

Lex. Trans.
✔
✔
✔

Preordering

Candidateselection

Struct. Trans.
✔
✔

OutputFigure 1: Basic system architecture.

✔

Input sentence is fed to RBMT system, RBMT
plus SPE system and Preordering plus SMT system1. From outputs of three systems, candidate selection part selects best output as the system output. Here, our SPE and SMT are semi-monotone,✔

Table 1: Classification of MT andrelated techniques.

1

RBMT

Semi-monotone Semi-monotone
SPE
SMT

Two main processes of machine translation are
lexical transfer and structural transfer. Machinetranslation techniques and related techniques areclassified in terms of these two processes asshown in Table 1.
Technique
RBMT
SMT
Monotone SPE
Preordering
Monotone SMT

System architecture

For JPCko-ja task, we don’t use preordering part.
29
Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 29‒34,
Kyoto, Japan, 16th October 2015.
2015 Copyright is held by the author(s).

because distortion limit of decoder is set to 6 instead of 0. We will explain details of the each partof the system in the following subsections.
2.1

ja task and also Japanese side of NTCIR-10’s
training corpora of PatentMT task EJ subtask(Goto et al. 2013)3. Total number of sentences forthis LM training is 5M. This LM is also built by
lmplz with order 6.
Distortion limit (DL) of the decoder is set to 6.
Usual setting of DL between Japanese and English or between Japanese and Chinese is 10 or over.
So we call our SPE part semi-monotone SPE.

RBMT part

We use a commercial based RBMT system for the
RBMT part. We, also, use user terminology dictionaries for zh-ja, JPCzh-ja and JPCko-ja tasks.
For zh-ja and JPCzh-ja tasks, we use a part of Chinese to Japanese technical term dictionary of JPO(Japan Patent Office) (Japio, 2015) 2 . Original
JPO’s dictionary includes 2,210,294 words(nouns and verbs). We filter out all verbs and thenouns which have identical Japanese translations
with the commercial based RBMT outputs. As theresult, we select 1,463,265 terms for the user dictionary for JPCzh-ja and zh-ja tasks. For JPCkoja task, we make a user dictionary from the training corpus of the task. We get 434,334 terms forthe user dictionary for the JPCko-ja task. For enja task, we don’t use any user dictionary.
We also use sentence pattern dictionary for
JPCzh-ja task. We use only 13 sentence patterns
for the task.
2.2

2.3

Preordering part

Preordering is one of the effective technique toimprove structural transfer accuracy (Isozaki,2010). Our preordering method uses context freeparsing rules with reordering rules. Figure 2
shows examples of parsing rules with reorderingrules and example of parsed phrases4. First example is English grammar rule with reordering rule.
The right hand side of the parsing rule “ADVPVBN PP” is reordered to “PP ADVP VBN” by thereordering rule “2 0 15.” Second example is Chinese grammar rule with reordering rule. Reordering rules are built by a heuristic algorithm.
VP ⇒ ADVP VBN PP 2 0 1 (widely utilized in many fields)

SPE part

VP ⇒ VV NP IP 1 2 0 (使 各个 电动机 13 旋转 驱动)

SPE part intends to improve the translation qualityof the output of the RBMT part. All target languages of the tasks are Japanese. So SPE parttranslates Japanese to Japanese. We use phrase
based Moses (Koehn et al., 2003) with default options as the SPE engine. Word segmenter for Japanese is Juman ver.7.01 (Kurohashi et al., 1994).
Translation models (TM) of each task is built
from RBMT output and reference translation of
the training corpus of each task. Training corpus
size (number of sentence pairs) will be listed inTable 3.
Language model (LM) is built from the following monolingual corpora. LM for en-ja task andzh-ja task is built from target side of the training
corpora both of the en-ja task and zh-ja task (3.6M
sentences). LM is built by lmplz tool in Moses(KenLM) with order 6. LM for JPCko-ja task and
JPCzh-ja task is built from target side of the training corpora both of the JPCko-ja task and JPCzh-

Parsing accuracy directly affects preordering
accuracy. We use Stanford Chinese word segmenter (Tseng et al., 2005; Chang et al., 2008) andBerkeley parser (Petrov et al., 2006) as the parsing
engine of our preordering part. Two improvements for the parsing are conducted. First one isgrammar improvement for Chinese grammar. For
en-ja task, we use the original English grammar of
the Berkeley parser, eng_sm6.gr. For JPCko-ja
task, we don’t conduct preordering because of thesimilarity of word order of Korean and Japanese.
For zh-ja task and JPCzh-ja task, we improve theoriginal Chinese grammar, chn_sm5.gr. It will be
explained in section 2.3.1. Second improvementof parsing is reranking of k-best parse trees thatwill be explained in section 2.3.2.

2

4

Figure 2: Example of parsing rules andreordering rules with examples

https://alaginrc.nict.go.jp/resources/jpo-info/jpolist.html
3Dev, devtest and test data of JPCko-ja task and
JPCzh-ja task are extracted from Japanese patent documents published in 2013. On the other hand,
NTCIR-10’s training corpora is extracted from Japanese patent documents published in 1990 to 2005.
They are not overlapped.

All sample sentences and phrases in this paper are
from ASPEC Corpora or JPO Patent Corpora provided by the workshop organizer.
5
Reordering rule “2 0 1” means that position of righthand side tags permutates from “0 1 2” to “2 0 1”.
Then, “ADVP VBN PP” is reordered to “PP ADVP
VBN”.
30

2.3.1

put the second ranked tree to our tree bank in thiscase.
From 1M JPCzh-ja training corpus, the number of
second or lower ranked tree is selected is about
151K. Re-training Berkeley Chinese grammar using this 151K tree bank, we get new grammar
named
chn_jpo.gr.
Comparingoriginal
chn_sm5.gr and chn_jpo.gr, the agreement rate ofsentence head word of top ranked parse tree andgold standard for devtest data is shown in Table
26.
Agreement rate rises about 13%.

Grammar improvement

The idea for grammar improvement is to use wordalignment of JPCzh-ja bilingual training corpus.
Firstly, word alignment is conducted from JPCzhja training corpus (1M sentence pairs) by GIZA++(Och and Ney, 2003). For each sentence pairs, we
decide sentence head word both for Japanese andChinese using word alignment. Since Japanese isa typical head final language, head word of Japanese sentence is positioned at the end of the sentence. So it is easy to find the sentence head word
for Japanese sentences. We find Chinese sentencehead word as the aligned word to the Japanese
sentence head word. For example, in the ja-zh sentence pair shown in Figure 3, Japanese sentencehead word “発生する” is aligned to Chinese word
“产生”. So “产生” is decided as the head word ofthis Chinese sentence. We consider it as the gold
standard head word.

GrammarAgreement rate (%)
chn_sm5.gr
50.5
chn_jpo.gr
63.0

Table 2: Agreement rate of sentence head word
for the devtest data
2.3.2

シンチレータ ５４ は 、 Ｘ線 を 受けて 光 を 発生 する 。

Another improvement of preordering part is reranking of k-best parse trees. For the training corpus, reordered source sentence is compared togold standard reordered source sentence. Here,gold standard reordered source sentence is determined using alignment to a target sentence. For
example, Chinese sentence shown in Figure 3 “闪烁器 54 接收 X 射线 产生 光 。”is gold standard reordered to “闪烁器 54 X
射线 接收 光 产生 。” using alignment
to the target sentence. This comparison is measured by word error rate and select the parse tree
which has the minimum word error rate in the kbest parse trees as the best parse tree. The parsetree shown in Figure 4 (a) is reordered to “闪烁器54 X 射线 光 产生 接收 。” and theparse tree shown in Figure 4 (b) is reordered to“ 闪 烁 器 54
X 射线 接收 光 产
生 。”. Then, Figure 4 (b) is selected as the bestparse tree in this case.
For the dev, devtest and test sets, we use LM
based reranking to select the best parse tree.
Firstly, we make reordered source sentence corpus from the training corpus by the above method
and build LM using this corpus. Next, we selectthe best parse tree which has the maximum LM
score in the k-best reordered sentences in dev,
devtest and test sets. Here, LM score of a sentence
is a score calculated by the tool “query” in Moses
divided by the number of words in the sentence.

闪烁器 54 接收 X 射线 产生 光 。

Figure 3: Example of zh-ja word alignment.
┗IP┣NP┃┣NR━闪烁器
┃┗NN━54┣VP┃┣VV━接收┃┣NP┃┃┣NN━X┃┃┗NN━射线┃┗IP┃ ┗VP┃ ┣VV━产生┃ ┗NP┃ ┗NN━光┗PU━。

Reranking of k-best parse trees┗IP┣NP┃┣NR━闪烁器
┃┗NN━54┣VP┃┣VP┃┃┣VV━接收┃┃┗NP┃┃ ┣NN━X┃┃ ┗NN━射线┃┗VP┃ ┣VV━产生┃ ┗NP┃ ┗NN━光┗PU━。

(a) top ranked tree(b) second ranked tree
Figure 4: Parsed trees.
Next step is to make a tree bank. Chinese sentences of training corpus are parsed by the original
grammar i.e. chn_sm5.gr, and we get k-best parsetrees for each sentence (k is set to 100). Then we
select the best parse, in which the sentence head
word is same as the gold standard head word. For
example, the Chinese sentence in Figure 3 is kbest parsed shown in Figure 4. The top ranked tree
has “接收” as the sentence head word and the second ranked tree has “产生” as the sentence headword, which is same as the gold standard. So we
6

Devtest data is not included in the training data forthe grammar improvement. Gold standard of Chinese

sentence head for devtest data is determined by themethod described above.
31

2.4

SMT part
Another ad-hoc processing is to convert all half
width characters in RBMT and SMT outputs tofull width characters, because Japanese document
tend to use full width characters.

SMT part uses phrase based Moses same as SPE
part. For JPCko-ja task, SMT part translates
source sentences to target sentences as the usualphrase based SMT. As the segmenter for Korean,
we use MeCab-ko7. For JPCzh-ja task, zh-ja task
and en-ja task, SMT part translates reordered
source sentences to the target. Reordering is madeby the method described in 2.3. Distortion limit is
set to 6 both JPCko-ja task and other tasks. So, wecall our SMT semi-monotone SMT. LM for SMT
is same as LM for SPE. TM is trained from thetraining corpus provided by the workshop organizer. Training corpus sizes (number of sentencepairs) are listed in Table 3.
Task
JPCko-ja
JPCzh-ja
zh-ja
en-ja

2.7

We have no consideration for context-awarenessin our system.

Corpus size
994,998
995,385
668,468
2,351,575

3.1

Translation results

System
BLEU
RIBES
HUMAN
RBMT+SPE70.13 0.9419
6.500
JPCko-ja SMT
70.81 0.9430-Combination
70.67 0.9430 10.250
RBMT+SPE40.35 0.8195
8.250
JPCzh-ja Preordering+SMT 40.70 0.8243-Combination
41.06 0.8270 22.000
RBMT+SPE35.59 0.8158-zh-jaPreordering+SMT 39.43 0.8377-Combination
37.90 0.8260 25.750
RBMT+SPE30.46 0.7685-en-jaPreordering+SMT 29.78 0.7536 32.500
Combination
30.88 0.7657--

Candidate selection part

Table 4: Evaluation results of the translation

Other ad-hoc processing

In JPCko-ja task and JPCzh-ja task, system
combination using candidate selection by LM
score is more accurate than RBMT+SPE systemboth in automatic and human evaluation. In zh-ja
task, Preordering+SMT system has higher BLEU
and RIBES than system combination. However,
we don’t have human score for preordering+SMT
system for the zh-ja task.

For JPCko-ja task, we conduct an ad-hoc preprocessing for Korean source sentences of the train,dev, devtest and test corpora and their RBMT outputs. It is deletion of brackets surrounding thenumber, because the use of brackets between Korean and Japanese is different shown in Figure 5.
In Korean sentence, number “2” is surrounded by
the brackets. However, in Japanese sentence,
number “２” is not surrounded by the brackets. So
we delete brackets surrounding the number in Korean side to improve alignment accuracy of brackets.

3.2

Candidate selection results

Table 5 shows the candidate selection results.
Most of outputs of RBMT part are not selected.
Outputs of RBMT+SPE part and outputs of preordering+SMT part are selected about half and half.

또한, 산화피막(2)이 존재하는 경우에는, ・・・
また、酸化皮膜２が存在する場合には、・・・

Figure 5: Different bracket usage in Korean
and Japanese.
7

Experimental results

Task

The last part of our translation system is a candidate selection part. This part select the candidate
which has the maximum LM score from the outputs of RBMT part, RBMT+SPE part and Preordering+SMT part. Here, LM score is calculated
from the LM for SMT part by the method described in section 2.3.2.
2.6

3

Table 4 shows the official evaluation results of thetranslation of the test data (Nakazawa et al., 2015).
In all cases, BLEU and RIBES are calculated using Japanese segmenter Juman.

Table 3: Training corpus size
2.5

Issues for context-aware machine translation

https://bitbucket.org/eunjeon/mecab-ko/
32

JPCko-ja

25

Preordering
Total
+SMT
1177
798
2000

JPCzh-ja
zh-ja

2
9

875
1270

1123
828

2000
2107

en-ja

5

658

1149

1812

Task

RBMT

RBMT+SPE

Table 5: The number of each system outputs
selected by the candidate selection part.

１５” is -71.1855. These two examples indicate
the limitation of LM score based candidate selection method.
SMT
SPE
BASE

To confirm effectiveness of candidate selection
process, we compare LM scores and human evaluation scores for JPCzh-ja task. Table 6 shows human evaluation score of SPE8 outputs and SMT
outputs when the case of LM score for SMT output exceeds LM score for SPE output.

REF
SRC

SMT

SMT＼SPE
-1
0
1

-1
6
25
8

0
4
101
32

1
2
26
18

SPE

Table 6: The number of human evaluation scoresfor SPE outputs and SMT outputs when the caseof LM score of SMT output exceeds LM score of
SPE output.

BASE

REF

From the Table 6, we can see this candidate selection process makes human score better in 65
cases (SMT > SPE) and worse in 32 cases (SPE >SMT). The number of tie cases is 125.
To investigate worsen cases, we show severaltranslation examples. Table 7 shows SMT output
and SPE output, baseline (BASE) output, reference (REF) and source (SRC) of two translation
examples. In these cases, LM score of SMT output
is greater than LM score of SPE output. But human score of SMT output (-1) is less than humanscore of SPE output (1).
In the first example, the word “オートコリメ
ータ” is less general than the word “コリメータ”.
Actually, LM score of the former word is-5.61676 and LM score of the latter word is-4.12944. Then LM score of the former sentenceis less than LM score of the latter sentence.
In the second example, “アノード４１０と陰極４１５” is less general than “アノード４１５とカソード４１０” in our LM. Actually, LMscore of SMT output sentence is -66.1355 and theLM score of the sentence which is converted theterm “アノード４１５とカソード４１０” of
SMT output to “アノード４１０とカソード４

SRC

図５は、コリメータ装置による調整動作を説明する
ための概略図である。
図５は、オートコリ メータ装置による調整動作を説明するための概略図である。
図５は、自動コリメータ装置による調整動作を説明するための概略図である。
図５は、オートコリ メータ装置による調整動作を説明するための概略図である。
图5是用于说明由自 动准直 仪装置 进行的调整操
作的概略图。
図４は、名称からア ノード４１５とカソード４１０との間に挟まれた発光層４０５のＯＬＥＤ４００とを含む第１の例示的な実施の形態について説明する。
図４は、概念上からア ノード４１０と陰極４１５との間の発光層４０５のＯＬＥＤ４００の第１の例示的実
施形態を含むと説明する。
図４は、陽極４１０とカソード４１５との間の発光層４０５のＯＬＥＤ４００の第１で概念的に説明は、例示
的な実施形態が挟まれている。
図４は、ア ノード４１０とカソード４１５とに挟まれた発光層４０５を含むＯＬＥＤ４００の第１実施例の概念図である。
图4从概念上说明包括夹在阳极4 1 0 与阴极4 1 5之间的发射层405的OLED 400的第一示例性实施方案。

Table 7: Example of candidate selection partmaking worse output.
3.3

Other results

Table 8 shows other evaluation results for JPCzhja task on devtest data translation.
System
RBMT
RBMT
RBMT+SPE
SMT
Preordering+SMT
Preordering+SMT
Preordering+SMT
System combination

Additional feature
User dictionary
User dictionary(DL=10)original grammar
improved grammar
improved grammar +reranking of parse
trees

BLEU
16.55
23.54
41.35
40.86
40.15
41.84

RIBES
0.71920.75100.82030.80710.81640.821842.75

0.823743.01

0.8265

Table 8: Evaluation results for JPCzh-ja devtest
data translation
User dictionary, SPE and preordering greatly
improve RIBES score. Improving of grammar, reranking of parse trees and system combinationslightly improve RIBES score. For BLEU score,8

In this section, the term “SPE” is used for
RBMT+SPE and the term “SMT” is used for preordering+SMT for the simplicity.
33

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham
Neubig, Sadao Kurohashi and Eiichiro Sumita. 2015.
Overview of the 2nd Workshop on Asian Translation, Proceedings of the 2nd Workshop on AsianTranslation (WAT2015).

results are almost similar as RIBES case. However, preordering with original grammar makes
BLEU score worse compared with simple SMT.
RBMT+SPE with user dictionary improves
BLEU score by 17.81 compared with simple
RBMT. Preordering+SMT with the improved
grammar and reranking of parse trees improves
BLEU score by 1.89 compared with simple SMTwith DL 10. System combination improves BLEU
score by 0.26 compared with preordering+SMT.

4

Franz Josef Och, Hermann Ney. 2003. A SystematicComparison of Various Statistical Alignment Models, Computational Linguistics, volume 29, number
1, pages 19-51.
Slav Petrov, Leon Barrett, Romain Thibaux and Dan
Klein. 2006. Learning Accurate, Compact, and Interpretable Tree Annotation, Proceedings of COLING-ACL 2006, pages 433-440.

Conclusion

System architecture, experimental settings andevaluation results of the EHR group in the
WAT2015 tasks are described. Our system designconcept is combining of rule-based method andstatistical method and it gives the good effect tothe translation accuracy. One of the future issues
is to improve parsing accuracy both in RBMT partand preordering part.

Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel
Jurafsky and Christopher Manning. 2005. A Conditional Random Field Word Segmenter. In Fourth
SIGHAN Workshop on Chinese Language Processing.

Reference
Pi-Chuan Chang, Michel Galley and Chris Manning.
2008. Optimizing Chinese Word Segmentation for
Machine Translation Performance. Proceedings of
the Third Workshop on Statistical Machine Translation, pages 224-232.
Terumasa Ehara. 2014. A machine translation system
combining rule-based machine translation and statistical post-editing, Proceedings of the 1st Workshop on Asian Translation (WAT2014), pages 50-54.
Isao Goto, Ka Po Chow, Bin Lu, Eiichiro Sumita andBenjamin K. Tsou. 2013. Overview of the Patent
Machine Translation Task at the NTCIR-10 Workshop, Proceedings of the 10th NTCIR Conference,
pages 260-286.
Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada and
Kevin Duh. 2010. Head finalization: A simple reordering rule for SOV languages. Proceedings of the
Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 244-251.
Japio (Japan Patent Information Organization). 2015.
Investigation report of dictionary improvement andquality evaluation of machine translation for Chinese patent documents (in Japanese).
Philipp Koehn, Franz J. Och and Daniel Marcu. 2003.
Statistical Phrase-Based Translation, Proceedingsof HLT-NAACL 2003, pages 48-54.
Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto and Makoto Nagao. 1994. Improvements ofJapanese morphological analyzer JUMAN. Proceedings of The International Workshop on Sharable Natural Language Resources, pages 22-28.
34