Toshiba MT System Description for the WAT2015 Workshop
Satoshi Sonoh and Satoshi Kinoshita
Knowledge Media Laboratory, Corporate Research & Development Center,Toshiba Corporation.
1, Komukai Toshiba-cho, Saiwai-ku, Kawasaki 212-8582, JAPAN{satoshi.sonoo,satoshi.kinoshita}@toshiba.co.jp
Asian parallel corpora including ASPEC 1 ,NTCIR PatentMT2 and JPO Patent Corpus3 areavailable for the research of machine translation
systems. By using the parallel corpora, we haveconfirmed advantages which apply statisticalpost editing (SPE) to RBMT in domain adaptation (Suzuki, 2011).
In the last workshop (Nakazawa et al., 2014),
we participated in Japanese-English and Japanese-Chinese tasks with SPE approach and obtained higher evaluation results than RBMT.
Meanwhile, RBMT showed better performance
than SPE in the direct and relative comparison(Sonoh et al., 2014). In this workshop(WAT2015), we participated in all tasks including Japanese-English (ja-en), English-Japanese(en-ja), Japanese-Chinese (ja-zh) and ChineseJapanese (zh-ja) for “scientific paper subtask”,and Chinese-Japanese (JPCzh-ja) and KoreanJapanese (JPCko-ja) for “patents subtask”. Patents subtask is newly added, and its parallel
corpus has 4 sections (Chemistry, Electricity,
Mechanical Engineering and Physics).
In all the tasks, we submitted SPE translation
results based on our RBMT and SMT. In addition, we submitted system combination results
between SPE and SMT with recurrent neurallanguage model (RNNLM; Mikolov el al., 2010).
Section 2 and 3 describe the overview of oursystems and some pre/post processing. The experimental results and official results are shownin Section 4 and 5. The analysis for the official
results is discussed in Section 6 and finally, Section 7 concludes this paper. As for a contextaware translation, the description was omitted
because our baseline system is the same as thelast workshop (see Sonoh et al., 2014).

Abstract
This paper provides the system description ofToshiba Machine Translation System for the 2ndWorkshop on Asian Translation (WAT2015).
We participated in all tasks that consist of “scientific papers subtask” and “patents subtask”.
We submitted statistically post edited translation (SPE) results based on our rule based translation system and SMT for each language pair.
In addition, we submitted system combination
results between SPE and SMT with a recurrent
neural language model (RNNLM). In experimental results, the system combination achieved
higher BLEU scores than single system withreranking. We also obtained improvements in
Chinese translation in crowdsourcing evaluations.

1

Introduction

Recently, statistical machine translation (SMT)
has been broadly developed and successfully
used in the portion of practicable systems.
However, it is costly to make a large volume ofparallel corpora in a wide range of domains forcommercial use. For this reason, we have developed rule based machine translation (RBMT)
system using a monolingual corpus in the target
language. For example, target word selection ispossible based on co-occurrence relationshipextracted from a monolingual corpus (Suzuki etal., 2005). Furthermore, we have developed aword sense disambiguation based on a monolingual corpus in the target domain, and it has been
applied to Japanese-Korean and KoreanJapanese translation systems (Kumano 2013,
Tanaka et al., 2014). On the other hand, open1

http://lotus.kuee.kyoto-u.ac.jp/ASPEC/http://research.nii.ac.jp/ntcir/permission/ntcir10/perm-en-PatentMT.html
3
http://lotus.kuee.kyoto-u.ac.jp/WAT/patent/
248
Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 48‒53,
Kyoto, Japan, 16th October 2015.
2015 Copyright is held by the author(s).

2Overview of Toshiba System

2.1

RBMT Systemusing both basic features and RNNLM score. In
tuning, we combined 100-best candidates ofboth SPE and SMT for dev-set, and ran MERT
tuning by adding the RNNLM score to the basicfeatures. In decoding, we re-ranked combined
candidates by product-sum of the features including RNNLM score and tuned weights.
For ja-en, en-ja, ja-zh and zh-ja, we used
RNNLMs trained by the first 500k sentences inthe training data of ASPEC. For JPOzh-ja and
JPOko-ja, we used 500k sentences which wereevenly extracted from 4 sections in JPO PatentCorpus. All RNNLMs were trained with 500
hidden layers and 50 classes by RNNLM toolkit5.

Our RBMT system is basically a transfer-based
machine translation (Izuha et al., 2008). Thecore framework consists of morphological analysis, syntactic/semantic analysis, target word
selection, structural transfer, syntactic generation and morphological generation. Furthermore,huge amount of rules as translation knowledge
including word dictionaries can realize both hightranslation performance and flexibility of customization. As for Japanese-Korean translation,
syntactic analysis and transfer are omitted because the languages are grammatically similar.
2.2

Statistical Post Editing

Tuning
RBMTprocessing

3.1

Technical Term Dictionaries

3.2

pre/post-English Word Correction

To improve translation of sentences including
misspelled words in English, we applied correction processing based on an edited distance. We
replaced the word considered as misspellingwith a word which had the smallest edited distance in the training data. However, because
SMT and SPE basically have robustness to themisspelling, we confined words to be replaced towords which remain as unknown words in SMT
and SPE results.

System Combination using RNNLM

Although both SPE and SMT are based on a statistical model from the given corpora, they generate different translation candidates because
SPE has some features from RBMT. If a bettersystem can be selected from the candidates ineach translation, we can get a better translation
result.
Thus, we realized a system combination between SPE and SMT as n-best reranking using aRNNLM. The n-best reranking can be achieved

3.3Japanese KATAKANA Normalization

In the case where a target language is Japanese;
we applied normalization of KATAKANA notation. In advance of translation, we counted thefrequency of KATAKANA notation, which has
fluctuations of prolonged sound mark, in the5

http://www.fit.vutbr.cz/~imikolov/rnnlm/
https://alaginrc.nict.go.jp/resources/jpo-info/jpooutline.html#jpo-dic-zh
64andAs the preparation for each task, we selected
technical term dictionaries by the same principle
in the last workshop (Sonoh el al., 2014). For
JPOzh-ja, we used an additional patent dictionary, which is extracted from JPO ChineseJapanese dictionary6. Furthermore, for JPOko-ja,
we used n-gram probability dictionary, which
was made from monolingual patent resources, inorder to resolve word sense disambiguation(Tanaka et al., 2014).

SPE using phrase-based SMT has been proposed
and it is an efficient framework which is able toadapt translation output to target domains(Michel et al., 2007).
We first translated source sentences of training data in ASPEC and JPO Patent Corpus byRBMT. Then we trained phrase-based model
between translated sentences and reference sentences using Moses toolkit (Kohen et al., 2007).
In the training, we used 1M sentences for ja-en,en-ja, JPCzh-ja and JPCko-ja, 0.67M for ja-zh
and zh-ja in the training data. Japanese sentenceswere tokenized by JUMAN 4 , and Moses tokenizer for English, and Kytea (Neubig et. al,2011) for Chinese. We also trained 5-gram language models using KenLM (Heafield et al.,2013). In tuning and decoding, we set distortion
limit to 0 for JPOko-ja in consideration ofgrammatical similarity and 6 for other language
pairs.
2.3

3

http://nlp.ist.i.kyoto-u.ac.jp/index.php?JUMAN
49

Table 1: Overall BLEU and RIBES scores for “scientific papers subtask”.
ja-en
en-ja
ja-zhzh-ja
System Rerank
BLEU RIBES BLEU RIBES BLEU RIBES BLEU RIBESRBMT No
15.31 0.677 14.78 0.685 19.51 0.767 15.39 0.723
No
17.41 0.620 25.17 0.642 28.20 0.810 36.34 0.815
SMTYes
17.85 0.619 25.37 0.643 28.46 0.809 36.69 0.815
No
22.65 0.717 31.10 0.767 29.48 0.809 35.76 0.822
SPEYes
22.92 0.718 31.73 0.770 29.49 0.809 36.06 0.823
COMB Yes
23.00 0.716 31.82 0.770 29.60 0.810 37.47 0.827
Table 2: Overall BLEU and RIBES scores for “patents subtask”.
JPOzh-ja
JPOko-ja
System Rerank
BLEU RIBES BLEU RIBESRBMT No25.81 0.764 51.28 0.902
SMTNo38.77 0.802 70.17 0.943
Yes
39.18 0.805 70.89 0.944
SPENo
39.01 0.813 68.47 0.940
Yes
39.30 0.811 68.76 0.940
COMB Yes
40.23 0.813 70.40 0.942
target sentences of the training data. In the translation results, KATAKANA fluctuations werereplaced with those of highly-frequent notations,such as “from スクリュ to スクリュー” and“from サーバー to サーバ”. By applying normalization, we got improvements of about 0.5
BLEU in RBMT.
Furthermore, we replaced the ideographic
comma “、” in number expression with a normal comma “,” for translation results in Japanese.
3.4

means results of reranking using RNNLM (100best for SMT and SPE, 200-best for COMB). In
all tasks, SPE improves translation results ofRBMT on the BLEU and RIBES. In tasks except
JPOko-ja, SPE achieves performance equal to or
better than phrase-based SMT.
Moreover, in most tasks, Rerank improves
about 0.3-0.5 BLEU score, and COMB shows
better performance than other systems. In
JPOko-ja, SMT, SPE and COMB show very
high performances which are close to 70 BLEU,and SMT with reranking achieves the highest
BLEU and RIBES scores. In ja-en, en-ja, ja-zhand zh-ja, more than half of translations selectedfrom SPE and the others selected from SMT. In
particular SPE accounted for about 80% translations in ja-en, en-ja and zh-ja. On the other hand,more than half of translations selected from
SMT in JPOzh-ja and JPOko-ja. Table 3 shows
the translation examples that COMB achieves
better results than SPE with reranking in sentence-level BLEU.
Finally, we compared between phrase-basedmodel and hierarchical phrase-based model. Table 4 shows comparison in ja-zh task. In all systems including SPE, hierarchical phrase-basedmodel improves about 0.4 BLEU. We appliedhierarchical phrase-based model to ja-zh only,
because significant improvements were not confirmed in other language pairs.

Other Post Processing

In order to reduce unknown words in SMT, weapplied RBMT to SMT results. For example, inja-zh, we translated KATAKANA words, which
remain in SMT results, into Chinese or English
words, if the words were found in RBMT dictionaries. Also, Hangul words in SMT results ofJPOko-ja were translated into Japanese words.

4Experimental Results

This section shows experimental results of ourtranslation systems.
Table 1 and 2 show the overall BLEU andRIBES scores for “scientific papers subtask” and“patents subtask”, respectively. COMB means
results of the system combination and Rerank50

Table 3: Translation examples indicating that COMB achieves better results than SPE in sentece-level
BLEU.
SRC
揺動時に比べて，発電量は４０倍である。
REFIn comparison with the fluctuation, the electric power generation is the 40 twice.
ja-enSPE
Compared with the time of rocking, production of electricity is 40 times.
COMB Compared with the time of fluctuations, the electric power generation is 40 times.
SiO2 films showed excellent performance even at 430℃ or less, and the memory
SRC
effect of Si dot MOS capacitor was confirmed.
ＳｉＯ２膜は，４３０℃以下でも優れた性能を示し，ＳｉドットＭＯＳコンデンサのメモREFリ効果を確認した。
en-ja
ＳｉＯ２膜は４３０℃でも優れた性能を示す以下であり，ＳｉドットＭＯＳキャパシタのSPE記憶効果が確認された。
ＳｉＯ２膜は４３０℃でも優れた性能を示し，以下，ＳｉドットＭＯＳキャパシタの記憶COMB効果を確認した。
SRC
この擾乱からの回復についても考察した。
对这种干扰的恢复也进行了考察。
REFja-zhSPE考察了从该干扰恢复。
COMB 在这种干扰的恢复也进行了考察。
还对在完成来所登记之前的各个环节进行了介绍。
SRCREF
来所登録が完了するまでの流れ等も紹介した。
zh-jaSPE登録の完了までの各段階について紹介した。
COMB 登録が完了するまでの各段階について紹介した。
SRC
在固体中加入 BHT，混合物在丙酮中溶解。
ＢＨＴを固体に加え、混合物をアセトンに溶解する。
JPO REFzh-ja SPE固体ＢＨＴを加え、この混合物は、アセトンに溶解した。
COMB 固体ＢＨＴを加え、混合物をアセトンに溶解した。
SRC원재료 필름(1)에서의 비접합부의 적어도 일부를 덮도록 배치되어도 좋다.
原反フィルム１における非接合部の少なくとも一部を覆うように配置されてもよ
REF
い。
JPO
ko-ja SPE
原反フィルム１での非接合部の少なくとも一部を覆うように配置されてもよい。
原反フィルム１における非接合部の少なくとも一部を覆うように配置されてもよCOMB
い。
Table 4: A Comparison of Phrase-based Model.
ja-zh
System hierarchical
BLEURIBES

SMTSPECOMBNo
Yes
No
Yes
No
Yes

28.46
29.82
29.49
29.89
29.60
30.07

5

Official Results

This section shows official results of our translation systems.
We basically submitted two results, one isSPE 7 and the other is the system combination
between SPE and SMT. Furthermore, top twosystems on the BLEU scores were evaluated bythe crowdsourcing. In the crowdsourcing evaluation, pair-wise evaluation against the baseline
system (phrase-based SMT) was performed by 5evaluators, and HUMAN score was calculated

0.809
0.810
0.809
0.809
0.810
0.817

7In JPOko-ja, because SMT showed higher BLEU
score than SPE, we submitted SMT result.
51

Table 5: Overall official results for “scientific papers subtask”. B, R and H mean BLEU, RIBES,HUMAN, respectively. HUMAN was evaluated by 5 evaluators using crowdsorcing.
ja-en
en-ja
ja-zhzh-ja
System
B
R
H
B
RH
B
RH
B
R
H

SPE
22.89 0.719 25.00 32.06 0.771 40.25 30.17 0.813 2.50 35.85 0.825 -1.00
COMB 23.00 0.716 21.25 31.82 0.770
30.07 0.817 17.00 37.47 0.827 18.00
Table 6: Overall official results for “patents subtask”.
JPOzh-ja
JPOko-ja
SystemBRHBRH

SMT
71.01 0.944 4.50
SPE
41.12 0.822 24.25
COMB 41.82 0.821 14.50 70.51 0.942 3.00
based on majority voting (Nakazawa et al.,2014).
In WAT2015 results (Nakazawa et al., 2015),
we note that Toshiba systems were ranked as
one of the top three systems in human evaluationin ja-en, ja-zh and JPOzh-ja. Especially, ja-zh
achieved the highest score although the BLEU
score is lower than other systems. On the otherhand, as for JPOko-ja, we got a comparatively
high BLEU score, but were disappointed by its
low HUMAN score.
Table 5 and 6 are the overall official resultsfor each task, respectively. In ja-zh and zh-ja,
COMB shows higher HUMAN score than SPE.
On the other hand, SPE or SMT is higher than
COMB in ja-en, JPOzh-ja and JPOko-ja. Theseresults indicate that the system combination improves human evaluation of Chinese translationin the scientific documents, at least. We guessthat the system combination between equivalentsystems achieves complementary translation toimprove human evaluations. For example,BLEU scores of SPE and SMT are nearly equalin ja-zh and zh-ja (shown in Table 1).

HUMAN score to the baseline system, superior
results of COMB over Online A are shown in
the pair-wise evaluation.
We hypothesize that the significant difference between the crowdsourcing and the inner
evaluators occurs from the evaluation of thenumber expressions, such as “システム(100)”and “システム 100”. In the training data of
JPOko-ja, a lot of brackets of numbers in thesource sentences disappear in the target sentences. Thus, brackets are dropped in SPE and SMT.
As for well-translated target sentences such as
JPOko-ja, it is possible that evaluators in thecrowdsourcing judged faithful translation as better by focusing on existence of brackets.

67

Table 7: The relationship between automaticevaluations and human evaluations.
HUMAN
BLEU RIBES
COMB
Online A

Discussion

On receiving the crowdsourcing results, we analyzed differences between our system andOnline A, which obtained the highest HUMAN
score in JPOko-ja. Table 7 shows the comparison between our system (COMB) and Online A.
Here, ‘Baseline’ column is the HUMAN score in
the result of crowdsourcing (official results) andthe other was evaluated by inner evaluators. The
inner evaluation was conducted excluding expressional differences as described in detail below. Although Online A achieves a very high

70.51
55.05

0.94
0.91

Baseline COMB Online A

3.00
38.75 -10.75

10.75-Conclusion

The overview of Toshiba machine translation
systems, which applied the statistical post editing and the system combination with RNNLM,
is described in this paper.
SPE and reranking with RNNLM achieved
higher BLEU than phrase-based SMT in mostlanguage pairs. Furthermore, the system combination between SPE and SMT improved BLEU
score in Japanese-English pair and JapaneseChinese pair. In the other hand, a straightforward correlation between automatic evaluation52

and human evaluation is not confirmed in oursystem. We need to establish the combination ofmulti-systems for practical use purpose, taking
advantage of their characteristics and qualities.

Machine Translation. In Proc. of the ACL,
pages 177-180.
Graham Neubig, Yosuke Nakata and Shinsuke
Mori. 2011. Pointwise Prediction for Robust,
Adaptable Japanese Morphological Analysis.
In Proc. of ACL-HLT.

Reference

Kenneth Heafield, Ivan Pouzyrevsky, Jonathan
H. Clark, and Philipp Koehn. 2013. Scalable
modified Kneser-Ney language model estimation. In Proc. of the ACL, pages 690–696.

Hirokazu Suzuki and Akira Kumano. 2005.
Learning Translations from Monolingual
Corpora. In Proc. of MT Summit X.Akira Kumano. 2013. Korean Translation System forPatent Documents. In Japio YEAR BOOK
2013, pages 298-301. (In Japanese)

Toshiaki Nakazawa, Hideya Mino, Isao Goto,Graham Neubig, Sadao Kurohashi and Eiichiro Sumita. 2015. Overview of the 2nd Workshop on Asian Translation. In Proc. of the2nd Workshop on Asian Translation(WAT2015).

Hiroyuki Tanaka, Satoshi Sonoh, Satoshi Kinoshita and Satoshi Kamatani. 2014. Improvement of Korean machine translation using statistical word selection. In Proc. of the3rd Symposium on Patent Information Processing, pages 97-100. (In Japanese)
Hirokazu Suzuki. 2011. Automatic Post-Editing
based on SMT and its selective application bySentence-Level Automatic Quality Evaluation.
In Proc. of MT Summit XIII.
Toshiaki Nakazawa, Hideya Mino, Isao Goto,Sadao Kurohashi and Eiichiro Sumita. 2014.
Overview of the 1st Workshop on AsianTranslation. In Proc. of the 1st Workshop on
Asian Translation (WAT2014).
Satoshi Sonoh, Satoshi Kinoshita, Hiroyuki
Tanaka and Satoshi Kamatani. 2014. Toshiba
MT System Description for the WAT2014
Workshop. In Proc. of the 1st Workshop on
Asian Translation (WAT2014).
Tomas Mikolov, Martin Karafiat, Lukas Burget,
Jan Cernocky, and Sanjeev Khudanpur. 2010.
Recurrent neural network based language
model. In Proc. of INTERSPEECH 2010 pages 1045–1048.
Tatsuya Izuha, Akira Kumano and Yuka Kuroda.
2008. Toshiba Rule-Based Translation System at NTCIR-7 PAT MT. In Proc. ofNTCIR-7 Workshop Meeting, pages 430-434.
Michel Simard, Cyril Goutte and Pierre Isabell.
2007. Statistical Phrase-based Post-editing. In
Proc. of NAACL HLT 2007, ACL, pages 508515.
Philipp Kohen, Marcell Federuci, Brooke Cowan, Richard Zens, Chris Dyer, Ondrej Bojar
Alexandra Constantin and Evan Herbst. 2007.
Moses: Open Source Toolkit for Statistical53