Overview of the 4th Workshop on Asian Translation
Toshiaki Nakazawa
Shohei Higashiyama and Chenchen Ding
Japan Science andNational Institute ofTechnology Agency
Information andnakazawa@nlp.ist.i.kyoto-u.ac.jp
Communications Technology{shohei.higashiyama, chenchen.ding}@nict.go.jp
Hideya Mino and Isao GotoNHK
{mino.h-gq, goto.i-es}@nhk.or.jp

Hideto Kazawa
Googlekazawa@google.com

Yusuke Oda
Graham Neubig
Nara Institute ofCarnegie Mellon University
Science and Technology
gneubig@cs.cmu.edu
oda.yusuke.on9@is.naist.jp
Abstract

thermore, we invited research papers on topics related to machine translation, especiallyfor Asian languages. The submitted research
papers were peer reviewed by three programcommittee members and the committee accepted 4 papers, which focus on on neural machine translation, and construction and evaluation of language resources. We also launched
the small NMT task, which aims to build asmall NMT system that keeps a reasonabletranslation quality. There are, however, no
submissions to the task this year.
WAT is the uniq workshop on Asian language transration with the following characteristics:

This paper presents the results ofthe shared tasks from the 4th workshop on Asian translation (WAT2017)
including J↔E, J↔C scientific paper translation subtasks, C↔J, K↔J,
E↔J patent translation subtasks,
H↔E mixed domain subtasks, J↔E
newswire subtasks and J↔E recipe
subtasks. For the WAT2017, 12 institutions participated in the sharedtasks. About 300 translation results
have been submitted to the automatic
evaluation server, and selected submissions were manually evaluated.

1

Sadao Kurohashi
Kyoto University
kuro@i.kyoto-u.ac.jp• Open innovation platform
Due to the fixed and open test data, we
can repeatedly evaluate translation systems on the same dataset over years.
There is no deadline of translation result submission with respect to automatic evaluation of translation quality
and WAT receives submissions at any
time.

Introduction

The Workshop on Asian Translation (WAT)
is a new open evaluation campaign focusingon Asian languages. Following the success ofthe previous workshops WAT2014 (Nakazawa
et al., 2014), WAT2015 (Nakazawa et al.,2015) and WAT2016 (Nakazawa et al., 2016),
WAT2017 brings together machine translation
researchers and users to try, evaluate, shareand discuss brand-new ideas of machine translation. We have been working toward practical use of machine translation among all Asian
countries.
For the 4th WAT, we adopted new translation subtasks with English-Japanese news corpus and English-Japanese recipe corpus in addition to the subtasks at WAT2016 1 . Fur-

• Domain and language pairs
WAT is the world’s first workshopthat targets scientific paper domain,
and Chinese↔Japanese andKorean↔Japanese language pairs. In the
future, we will add more Asian languages
such as Vietnamese, Thai, Burmese andso on.

1

This year we did not conduct Indonesian-English
newswire subtask, which is conducted in WAT2016,
due to corpus license reasons.

• Evaluation method
Evaluation is done both automatically
1

Proceedings of the 4th Workshop on Asian Translation, pages 1–54,
Taipei, Taiwan, November 27, 2017. c 2017 AFNLP

Lang
JE
JC

Train
3,008,500
672,315

Dev
1,790
2,090

DevTest
1,784
2,148

pair is accompanied by a similarity score thatare calculated by the method and a field ID
that indicates a scientific field. The correspondence between field IDs and field names,
along with the frequency and occurrence ratios for the training data, are descripted in the
README file of ASPEC-JE.
The development, development-test andtest data were extracted from parallel sentences from the Japanese-English paper abstracts that exclude the sentences in the training data. Each dataset consists of 400 documents and contains sentences in each field at
the same rate. The document alignment was
conducted automatically and only documentswith a 1-to-1 alignment are included. It istherefore possible to restore the original documents. The format is the same as the training
data except that there is no similarity score.

Test
1,812
2,107

Table 1: Statistics for ASPEC.
and manually. For automatic evaluation, we use three metrics: BLEU, RIBES
and AMFM. As human evaluation, we
evaluate translation results by pairwise
evaluation and JPO adequacy evaluation.
JPO adequacy evaluation is conducted forthe selected submissions according to thepairwise evaluation results.

2

Dataset

WAT2017 uses the Asian Scientific Paper Excerpt Corpus (ASPEC) 2 , JPO Patent Corpus(JPC) 3 , JIJI Corpus 4 , IIT Bombay EnglishHindi Corpus (IITB Corpus) 5 and Recipe
Corpus 6 as the dataset.
2.1

2.1.2 ASPEC-JCASPEC-JC is a parallel corpus consisting ofJapanese scientific papers, which come fromthe literature database and electronic journalsite J-STAGE by JST, and their translation to
Chinese with permission from the necessary
academic associations. Abstracts and paragraph units are selected from the body text
so as to contain the highest overall vocabulary
coverage.
The development, development-test andtest data are extracted at random from documents containing single paragraphs across theentire corpus. Each set contains 400 paragraphs (documents). There are no documents
sharing the same data across the training, development, development-test and test sets.

ASPEC

ASPEC was constructed by the Japan Scienceand Technology Agency (JST) in collaboration
with the National Institute of Information andCommunications Technology (NICT). The
corpus consists of a Japanese-English scientific paper abstract corpus (ASPEC-JE),
which is used for J↔E subtasks, and aJapanese-Chinese scientific paper excerpt corpus (ASPEC-JC), which is used for J↔C subtasks. The statistics for each corpus are shownin Table 1.
2.1.1 ASPEC-JE
The training data for ASPEC-JE was constructed by NICT from approximately two
million Japanese-English scientific paper abstracts owned by JST. The data is a comparable corpus and sentence correspondences arefound automatically using the method from
(Utiyama and Isahara, 2007). Each sentence

2.2 JPC
JPC was constructed by the Japan Patent Office (JPO). The corpus consists of ChineseJapanese patent description corpus (JPC-CJ),
Korean-Japanese patent description corpus(JPC-KJ) and English-Japanese patent description corpus (JPC-EJ) with the sectionsof Chemistry, Electricity, Mechanical engineering, and Physics on the basis of International Patent Classification (IPC). Each corpus is partitioned into training, development,
development-test and test data. This corpus
is used for patent subtasks C↔J, K↔J andE↔J. The statistics for each corpus are shown2
http://lotus.kuee.kyotou.ac.jp/ASPEC/index.html
3
http://lotus.kuee.kyotou.ac.jp/WAT/patent/index.html
4
http://lotus.kuee.kyoto-u.ac.jp/WAT/jijicorpus/index.html
5
http://www.cfilt.iitb.ac.in/iitb_parallel/index.html
6
http://lotus.kuee.kyoto-u.ac.jp/WAT/recipecorpus/index.html

2

Lang
CJ
KJ
EJ

Train
1,000,000
1,000,000
1,000,000

Dev
2,000
2,000
2,000

DevTest
2,000
2,000
2,000

Lang
EJ

Test
2,000
2,000
2,000

Dev
2,000

DevTest
2,000

Test
2,000

Table 3: Statistics for JIJI Corpus.
Lang
H
EH
JH

Table 2: Statistics for JPC.
in Table 2.
The Sentence pairs in each data were randomly extracted from a description part ofcomparable patent documents under the condition that a similarity score between two sentences is greater than or equal to the thresholdvalue 0.05. The similarity score was calculatedby the method from (Utiyama and Isahara,2007) as with ASPEC. Document pairs whichwere used to extract sentence pairs for eachdata were not used for the other data. Furthermore, the sentence pairs were extracted
so as to be the same number among the four
sections. The maximize number of sentencepairs which are extracted from one document
pair was limited to 60 for training data and20 for the development, development-test andtest data.
The training data for JPC-CJ was madewith sentence pairs of Chinese-Japanese
patent documents published in 2012. For
JPC-KJ and JPC-EJ, the training data was
extracted from sentence pairs of KoreanJapanese and English-Japanese patent documents published in 2011 and 2012. The development, development-test and test data for
JPC-CJ, JPC-KJ and JPC-EJ were respectively made with 100 patent documents published in 2013.
2.3

Train
200,000

Train–
1,492,827
152,692

Dev
–
520
1,566

Test
–
2,507
2,000

Mono
45,075,279––

Table 4: Statistics for IITB Corpus. “Mono”
indicates monolingual Hindi corpus.

using the method from (Utiyama and Isahara,2007).
2.4 IITB Corpus
IIT Bombay English-Hindi corpus contains
English-Hindi parallel corpus (IITB-EH) as
well as monolingual Hindi corpus collectedfrom a variety of sources and corpora developed at the Center for Indian Language Technology, IIT Bombay over the years. This corpus is used for mixed domain subtasks H↔E.
Furthermore, mixed domain subtasks H↔J
were added as a pivot language task with aparallel corpus created using publicly available
corpora (IITB-JH) 7 . Most sentence pairs in
IITB-JH come from the Bible corpus. The
statistics for each corpus are shown in Table
4.
2.5 Recipe Corpus
Recipe Corpus was constructed by Cookpad
Inc. Each recipe consists of a title, ingredients, steps, a description and a history. Every
text in titles, ingredients and steps consists ofa parallel sentence while one in descriptions
and histories is not always a parallel sentence.
Although all of the texts in the training set canbe used for training, only titles, ingredientsand steps in the test set is used for evaluation.
The statistics for each corpus are described inTable 5.

JIJI Corpus

JIJI Corpus was constructed by Jiji Press, Ltd.
in collaboration with NICT. The corpus consists of news text that comes from Jiji Press
news of various categories including politics,
economy, nation, business, markets, sports
and so on. The corpus is partitioned intotraining, development, development-test andtest data, which consists of Japanese-English
sentence pairs. The statistics for each corpus
are shown in Table 3.
The sentence pairs in each data are identified in the same manner as that for ASPEC

3 Baseline Systems
Human evaluations were conducted as pairwise comparisons between the translation results for a specific baseline system and translation results for each participant’s system.
7
http://lotus.kuee.kyoto-u.ac.jp/WAT/Hindicorpus/WAT2017-Ja-Hi.zip

3

Lang
EJ

TextType
Title
Ingredient
Step

Train
14,779127,244108,993

Dev
500
4,274
3,303

DevTest
500
4,188
3,086

Test
500
3,935
2,804

Table 5: Statistics for Recipe Corpus.
That is, the specific baseline system was thestandard for human evaluation. A phrasebased statistical machine translation (SMT)system was adopted as the specific baseline
system at WAT 2017, which is the same system as that at WAT 2014 to WAT 2016.
In addition to the results for the baseline
phrase-based SMT system, we produced results for the baseline systems that consistedof a hierarchical phrase-based SMT system,a string-to-tree syntax-based SMT system,a tree-to-string syntax-based SMT system,
seven commercial rule-based machine translation (RBMT) systems, and two online translation systems. We also experimentally produced results for the baseline systems thatconsisted of an neural machine translation system using the implementation of (Vaswani
et al., 2017). The SMT baseline systems consisted of publicly available software, and theprocedures for building the systems and fortranslating using the systems were publishedon the WAT web page8 . We used Moses(Koehn et al., 2007; Hoang et al., 2009) as theimplementation of the baseline SMT systems.
The Berkeley parser (Petrov et al., 2006) wasused to obtain syntactic annotations. The
baseline systems are shown in Table 6.
The commercial RBMT systems and the online translation systems were operated by theorganizers. We note that these RBMT companies and online translation companies did
not submit themselves. Because our objective
is not to compare commercial RBMT systems
or online translation systems from companies
that did not themselves participate, the system IDs of these systems are anonymous inthis paper.

8

http://lotus.kuee.kyoto-u.ac.jp/WAT/

4

5
Table 6: Baseline Systems

ASPEC
JPC
IITB JIJI RECIPE
System ID System
Type JE EJ JC CJJE EJ JC CJ JK KJHE EH JE EJ JE
EJ
SMT PhraseMoses’ Phrase-based SMT
SMT ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓SMT Hiero Moses’ Hierarchical Phrase-based SMT
SMT ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓SMT S2T Moses’ String-to-Tree Syntax-based SMT and Berkeley parser SMT ✓
✓
✓
✓
✓
✓
SMT T2S Moses’ Tree-to-String Syntax-based SMT and Berkeley parser SMT
✓
✓
✓
✓
✓
✓
RBMT X The Honyaku V15 (Commercial system)
RBMT ✓ ✓
✓ ✓
✓ ✓ ✓
✓
RBMT X ATLAS V14 (Commercial system)
RBMT ✓ ✓✓ ✓RBMT X PAT-Transer 2009 (Commercial system)
RBMT ✓ ✓✓ ✓RBMT X PC-Transer V13 (Commercial system)
RBMT
✓ ✓ ✓
✓
RBMT X J-Beijing 7 (Commercial system)
RBMT
✓ ✓
✓ ✓
RBMT X Hohrai 2011 (Commercial system)
RBMT
✓ ✓
✓
RBMT X J Soul 9 (Commercial system)
RBMT✓ ✓RBMT X Korai 2011 (Commercial system)
RBMT✓ ✓Online X
Google translate
Other ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓Online X
Bing translator
Other ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓AIAYN
Google’s implementation of “Attention Is All You Need”
NMT ✓ ✓

3.1

Training Data

• Phrase score option: GoodTuring

We used the following data for training the
SMT baseline systems.

The default values were used for the other system parameters.

• Training data for the language model: Allof the target language sentences in theparallel corpus.
• Training data for the translation model:
Sentences that were 40 words or less inlength. (For ASPEC Japanese–English
training data, we only used train-1.txt,
which consists of one million parallel sentence pairs with high similarity scores.)• Development data for tuning: All of thedevelopment data.
3.2

3.4 Hierarchical Phrase-based SMT
We used the following Moses configuration forthe hierarchical phrase-based SMT system.
• max-chart-span = 1000
• Phrase score option: GoodTuring
The default values were used for the other system parameters.
3.5 String-to-Tree Syntax-based SMT
We used the Berkeley parser to obtain target language syntax. We used the following Moses configuration for the string-to-tree
syntax-based SMT system.

Common Settings for Baseline
SMT

We used the following tools for tokenization.

• max-chart-span = 1000
• Phrase score option: GoodTuring• Phrase extraction options: MaxSpan =1000, MinHoleSource = 1, and NonTermConsecSource.

• Juman version 7.09 for Japanese segmentation.
• Stanford Word Segmenter version 201401-0410 (Chinese Penn Treebank (CTB)
model) for Chinese segmentation.
• The Moses toolkit for English and Indonesian tokenization.
• Mecab-ko11 for Korean segmentation.
• Indic NLP Library12 for Hindi segmentation.

The default values were used for the other system parameters.
3.6 Tree-to-String Syntax-based SMT
We used the Berkeley parser to obtain source
language syntax. We used the following Moses
configuration for the baseline tree-to-string
syntax-based SMT system.

To obtain word alignments, GIZA++ andgrow-diag-final-and heuristics were used. We
used 5-gram language models with modified
Kneser-Ney smoothing, which were built using a tool in the Moses toolkit (Heafield et al.,2013).
3.3• max-chart-span = 1000
• Phrase score option: GoodTuring• Phrase extraction options: MaxSpan =1000, MinHoleSource = 1, MinWords =0, NonTermConsecSource, and AllowOnlyUnalignedWords.

Phrase-based SMT

We used the following Moses configuration forthe phrase-based SMT system.

The default values were used for the other system parameters.

• distortion-limit– 20 for JE, EJ, JC, and CJ– 0 for JK, KJ, HE, and EH– 6 for IE and EI• msd-bidirectional-fe lexicalized reordering

4 Automatic Evaluation
4.1 Procedure for Calculating
Automatic Evaluation Score
We evaluated translation results by three metrics: BLEU (Papineni et al., 2002), RIBES(Isozaki et al., 2010) and AMFM (Banchs
et al., 2015). BLEU scores were calculated using multi-bleu.perl which was distributed

9

http://nlp.ist.i.kyotou.ac.jp/EN/index.php?JUMAN
10
http://nlp.stanford.edu/software/segmenter.shtml
11
https://bitbucket.org/eunjeon/mecab-ko/
12https://bitbucket.org/anoopk/indic_nlp_library

6with the Moses toolkit (Koehn et al., 2007).
RIBES scores were calculated using RIBES.py
version 1.02.4 13 . AMFM scores were calculated using scripts created by the technical collaborators of WAT2017. All scores for eachtask were calculated using the correspondingreference.
Before the calculation of the automatic evaluation scores, the translation results were tokenized with word segmentation tools for eachlanguage. For Japanese segmentation, we used
three different tools: Juman version 7.0 (Kurohashi et al., 1994), KyTea 0.4.6 (Neubig et al.,2011) with Full SVM model 14 and MeCab
0.996 (Kudo, 2005) with IPA dictionary 2.7.0
15 . For Chinese segmentation, we used two
different tools: KyTea 0.4.6 with Full SVM
Model in MSR model and Stanford Word Segmenter (Tseng, 2005) version 2014-06-16 with
Chinese Penn Treebank (CTB) and Peking
University (PKU) model 16 . For Korean segmentation we used mecab-ko 17 . For English
segmentation, we used tokenizer.perl 18 in
the Moses toolkit. For Hindi segmentation,
we used Indic NLP Library 19 . The detailed
procedures for the automatic evaluation are
shown on the WAT2017 evaluation web page
20 .
4.2

Newswire subtask (J↔E),
Mixed domain subtask (H↔E, H↔J) or
Recipe subtask (J↔E);• Method:
SMT, RBMT, SMT and RBMT, EBMT,NMT or Other;• Use of other resources in addition to theprovided data ASPEC / JPC / IITB Corpus / JIJI Corpus / Recipe Corpus;• Permission to publish automatic evaluation scores on the WAT2017 web page.
Although participants can confirm only the
information that they filled or uploaded, theserver for the system stores all submitted information including translation results andscores. Information about translation resultsthat participants permit to be published is disclosed via the WAT2017 evaluation web page.
Participants can also submit the results for human evaluation using the same web interface.
This automatic evaluation system will remain
available even after WAT2017. Anybody canregister an account for the system by following
the procesures in the registration web page 21 .

Automatic Evaluation System

The participants submit translation results via
an automatic evaluation system deployed on
the WAT2017 web page, which automaticallygives evaluation scores for the uploaded results. Figure 1 shows the submission interface for participants. The system requires participants to provide the following information
when they upload translation results:• Subtask:
Scientific papers subtask (J↔E, J↔C),
Patents subtask (C↔J, K↔J, E↔J),
13

http://www.kecl.ntt.co.jp/icl/lirg/ribes/index.html
http://www.phontron.com/kytea/model.html
15
http://code.google.com/p/mecab/downloads/detail?
name=mecab-ipadic-2.7.0-20070801.tar.gz
16
http://nlp.stanford.edu/software/segmenter.shtml
17https://bitbucket.org/eunjeon/mecab-ko/
18
https://github.com/mosessmt/mosesdecoder/tree/
RELEASE-2.1.1/scripts/tokenizer/tokenizer.perl
19
https://bitbucket.org/anoopk/indic_nlp_library20
21
http://lotus.kuee.kyotohttp://lotus.kuee.kyotou.ac.jp/WAT/evaluation/index.html
u.ac.jp/WAT/WAT2017/registration/index.html
14

7

8
Figure 1: The submission web page for participants

5

Human Evaluation

score can be calculated by the following formula:

In WAT2017, we conducted 2 kinds of humanevaluations: pairwise evaluation and JPO adequacy evaluation.
5.1

P airwise = 100 ×Pairwise Evaluation

From the definition, the Pairwise score ranges
between -100 and 100.

The pairwise evaluation is the same as thelast year, but not using the crowdsourcing thisyear. We asked professional translation company to do pairwise evaluation. The cost of
pairwise evaluation per sentence is almost thesame to that of last year.
We randomly chose 400 sentences from the
Test set for the pairwise evaluation. We used
the same sentences as the last year for thecontinuous subtasks. Each submission is compared with the baseline translation (Phrasebased SMT, described in Section 3) and givena Pairwise score.
5.1.1

5.1.4 Confidence Interval Estimation
There are several ways to estimate a confidence interval. We chose to use bootstrap resampling (Koehn, 2004) to estimate the 95%
confidence interval. The procedure is as follows:
1. randomly select 300 sentences from the
400 human evaluation sentences, and calculate the Pairwise score of the selected
sentences

Pairwise Evaluation of Sentences
2. iterate the previous step 1000 times andget 1000 Pairwise scores

We conducted pairwise evaluation of each of
the 400 test sentences. The input sentence
and two translations (the baseline and a submission) are shown to the annotators, and theannotators are asked to judge which of thetranslation is better, or if they are of the same
quality. The order of the two translations areat random.
5.1.2

3. sort the 1000 scores and estimate the 95%
confidence interval by discarding the top
25 scores and the bottom 25 scores
5.2 JPO Adequacy Evaluation

Voting

The participants’ systems, which achieved the
top 3 highest scores among the pairwise evaluation results of each subtask22 , were also
evaluated with the JPO adequacy evaluation.
The JPO adequacy evaluation was carried out
by translation experts with a quality evaluation criterion for translated patent documents
which the Japanese Patent Office (JPO) decided. For each system, two annotators evaluate the test sentences to guarantee the quality.

To guarantee the quality of the evaluations,each sentence is evaluated by 5 different annotators and the final decision is made dependingon the 5 judgements. We define each judgement ji (i = 1, · · · , 5) as:
if better than the baseline 1−1 if worse than the baselineji =
0if the quality is the same
The final∑ decision D is defined as follows using
S=
ji :

 win (S ≥ 2)
loss (S ≤ −2)D=
tie (otherwise)
5.1.3

W −L
W +L+T

5.2.1 Evaluation of Sentences
The number of test sentences for the JPO adequacy evaluation is 200. The 200 test sentences were randomly selected from the 400
test sentences of the pairwise evaluation. The
test sentence include the input sentence, thesubmitted system’s translation and the reference translation.

Pairwise Score Calculation

Suppose that W is the number of wins compared to the baseline, L is the number of losses
and T is the number of ties. The Pairwise

22
The number of systems varies depending on thesubtasks.

9

5
4
3
2
1

All important information is transmitted correctly. (100%)
Almost all important information is transmitted correctly. (80%–)
More than half of important information istransmitted correctly. (50%–)
Some of important information is transmitted
correctly. (20%–)
Almost all important information is NOTtransmitted correctly. (–20%)

Table 7: The JPO adequacy criterion
5.2.2 Evaluation Criterion
Table 7 shows the JPO adequacy criterion
from 5 to 1. The evaluation is performed
subjectively. “Important information” represents the technical factors and their relationships. The degree of importance of each element is also considered to evaluate. The percentages in each grade are rough indications
for the transmission degree of the source sentence meanings. The detailed criterion can befound on the JPO document (in Japanese) 23 .

6

Participants List

Table 8 shows the list of participants for
WAT2017. This includes not only Japanese organizations, but also some organizations fromoutside Japan. 12 teams submitted one ormore translation results to the automatic evaluation server or human evaluation.

23

http://www.jpo.go.jp/shiryou/toushin/chousa/tokkyohonyaku_hyouka.htm

10

11

Table 8: List of participants who submitted translation results to WAT2017 and their participation in each subtasks.

RECIPEASPEC
JPC
IITBC JIJI TTL ING STETeam ID
OrganizationJEEJJCCJJEEJJCCJKJHEEHJEEJJEEJJEEJJEEJ
Kyoto-U (Cromieres et al., 2017)
Kyoto University✓ ✓ ✓ ✓TMU (Matsumura and Komachi, 2017)Tokyo Metropolitan University✓ ✓ ✓EHR (Ehara, 2017)
Ehara NLP Research Laboratory✓✓ ✓NTT (Morishita et al., 2017)
NTT Communication Science Laboratories✓ ✓✓ ✓JAPIO (Kinoshita et al., 2017)
Japan Patent Information Organization✓ ✓✓ ✓NICT-2 (Imamura and Sumita, 2017) National Institute of Information and Communications Technology ✓ ✓ ✓ ✓✓XMUNLP (Wang et al., 2017)
Xiamen University✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓The University of Tokyo✓
UT-IIS (Neishi et al., 2017)
CUNI (Kocmi et al., 2017)Charles University, Institute of Formal and Applied Linguistics✓✓✓
IITB-MTG (Singh et al., 2017)
Indian Institute of Technology Bombay✓ ✓u-tkb (Long et al., 2017)
University of Tsukuba✓ ✓ ✓ ✓NAIST-NICT (Oda et al., 2017)
NAIST/NICT✓

7

Evaluation Results

the row is better than the system in the column at a significance level of p < 0.01, 0.05
and 0.1 respectively. Testing is also done by
the bootstrap resampling as follows:

In this section, the evaluation results for
WAT2017 are reported from several perspectives. Some of the results for both automaticand human evaluations are also accessible atthe WAT2017 website24 .
7.1

1. randomly select 300 sentences from the
400 pairwise evaluation sentences, andcalculate the Pairwise scores on the selected sentences for both systems

Official Evaluation Results

Figures 2, 3, 4 and 5 show the official evaluation results of ASPEC subtasks, Figures 6,
7, 8, 9 and 10 show those of JPC subtasks,
Figures 11 and 12 show those of IITBC subtasks, Figures 13 and 14 show those of JIJI
subtasks and Figures 15, 16, 17, 18, 19 and 20
show those of RECIPE subtasks. Each figurecontains automatic evaluation results (BLEU,
RIBES, AM-FM), the pairwise evaluation results with confidence intervals, correlation between automatic evaluations and the pairwise
evaluation, the JPO adequacy evaluation result and evaluation summary of top systems.
The detailed automatic evaluation results
for all the submissions are shown in Appendix
A. The detailed JPO adequacy evaluation results for the selected submissions are shownin Table 9. The weights for the weighted κ(Cohen, 1968) is defined as |Evaluation1 −Evaluation2|/4.
From the evaluation results, the following
can be observed:

2. iterate the previous step 1000 times andcount the number of wins (W ), losses (L)and ties (T )3. calculate p = WL+L
Inter-annotator Agreement
To assess the reliability of agreement between
the workers, we calculated the Fleiss’ κ (Fleiss
et al., 1971) values. The results are shown in
Table 21. We can see that the κ values arelarger for X → J translations than for J → X
translations. This may be because the majority of the workers are Japanese, and the evaluation of one’s mother tongue is much easierthan for other languages in general.

8 Submitted Data
The number of published automatic evaluation results for the 14 teams exceeded 300 before the start of WAT2017, and 67 translation
results for pairwise evaluation were submitted
by 12 teams. Furthermore, we selected severaltranslation results from each subtask according to the pairwise evaluation scores and evaluated them for JPO adequacy evaluation. We
will organize the all of the submitted data forhuman evaluation and make this public.

• The translation quality of this year is better than that of last year for all the subtasks.
• There is no big difference between theneural network based translation models
according to the JPO adequacy evaluation results for ASPEC subtasks.
7.2

9 Conclusion and Future
Perspective

Statistical Significance Testing of
Pairwise Evaluation between
Submissions

This paper summarizes the shared tasks of
WAT2017. We had 12 participants worldwide,and collected a large number of useful submissions for improving the current machine translation systems by analyzing the submissions
and identifying the issues.
For the next WAT workshop, we plan tochange the baseline system from the PBSMT
to NMT because the pairwise scores are saturated for some of the subtasks. Also, weTables 10, 11 and 12 show the results of statistical significance testing of ASPEC subtasks,
Tables 13, 14 and 15 show those of JPC subtasks, Table 16 shows those of IITBC subtasks,
Table 17 shows those of JIJI subtasks and Tables 18, 19 and 20 show those of RECIPE subtasks. ≫, ≫ and > mean that the system in24

http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/12

are planning to do extrinsic evaluation of thetranslations.
Unfortunately, there was no participants forthe small NMT task this year. We will brushup the task definition and invite participants
for the next WAT.

Appendix A

Submissions

Tables 22 to 41 summarize all the submissions
listed in the automatic evaluation server at thetime of the WAT2017 workshop (27th, November, 2017). The OTHER column shows the useof resources such as parallel corpora, monolingual corpora and parallel dictionaries in addition to ASPEC, JPC, IITB Corpus, JIJI Corpus, RECIPE Corpus.

13Figure 2: Official evaluation results of ASPEC-JE.

14

Figure 3: Official evaluation results of ASPEC-EJ.

15

Figure 4: Official evaluation results of ASPEC-JC.

16

Figure 5: Official evaluation results of ASPEC-CJ.

17

Figure 6: Official evaluation results of JPC-JE.

18

Figure 7: Official evaluation results of JPC-EJ.

19

Figure 8: Official evaluation results of JPC-JC.

20

Figure 9: Official evaluation results of JPC-CJ.

21

Figure 10: Official evaluation results of JPC-KJ.

22

Figure 11: Official evaluation results of IITBC-HE.

23

Figure 12: Official evaluation results of IITBC-EH.

24

Figure 13: Official evaluation results of JIJI-JE.

25

Figure 14: Official evaluation results of JIJI-EJ.

26

Figure 15: Official evaluation results of RECIPE-TTL-JE.

27

Figure 16: Official evaluation results of RECIPE-TTL-EJ.

28

Figure 17: Official evaluation results of RECIPE-ING-JE.

29

Figure 18: Official evaluation results of RECIPE-ING-EJ.

30

Figure 19: Official evaluation results of RECIPE-STE-JE.

31

Figure 20: Official evaluation results of RECIPE-STE-EJ.

32

SYSTEM DATA Annotator A
Annotator B
allweightedID
ID average varianceaverage varianceaverage κκNTT1681 4.15
0.58
4.13
0.52
4.14 0.29 0.41
AIAYN
1736 4.16
0.67
4.05
0.75
4.10 0.26 0.42
ASPEC-JE
Kyoto-U
1717 4.11
0.69
4.09
0.54
4.10 0.26 0.40
0.68
4.01
0.67
3.89 0.21 0.31
2016 best
1246 3.76
0.56
4.28
0.49
4.41 0.33 0.43
NTT1729 4.54
AIAYN
1737 4.38
0.83
4.21
0.76
4.30 0.36 0.52
0.73
4.16
0.69
4.29 0.35 0.48
NICT-21479 4.43
ASPEC-EJ
Kyoto-U1731 4.37
0.84
4.15
0.74
4.26 0.39 0.54
NAIST-NICT 1507 4.36
0.69
4.06
0.57
4.21 0.26 0.36
2016 best
1172 3.97
0.76
4.07
0.85
4.02 0.35 0.49
0.73
3.71
0.98
3.98 0.10 0.18
NICT-21483 4.25
Kyoto-U1722 4.25
0.79
3.64
1.07
3.95 0.12 0.23
ASPEC-JC
AIAYN
1738 4.26
0.69
3.54
1.03
3.90 0.17 0.27
2016 best
1071 4.00
1.09
3.76
1.14
3.88 0.20 0.36
NICT-21481 4.63
0.47
3.99
0.98
4.31 0.17 0.23
Kyoto-U
1720 4.62
0.56
3.97
0.94
4.30 0.16 0.22
ASPEC-CJ
AIAYN
1740 4.59
0.61
3.96
1.04
4.27 0.14 0.23
2016 best
1256 4.25
1.04
3.64
1.23
3.94 0.23 0.34
0.26
4.78
0.51
4.79 0.34 0.42
JAPIO
1574 4.80
u-tkb
1472 4.24
1.26
4.08
2.27
4.16 0.43 0.64
JPC-JE
CUNI
1666 4.12
1.49
3.99
2.35
4.05 0.40 0.63
2016 best
1149 4.09
0.80
4.51
0.58
4.30 0.25 0.39
JAPIO1454 4.74
0.45
4.76
0.38
4.75 0.32 0.48
EHR1407 4.64
0.61
4.61
0.65
4.63 0.42 0.60
JPC-EJ
u-tkb
1470 4.39
1.07
4.42
0.99
4.40 0.43 0.61
2016 best
1098 4.03
0.91
4.51
0.57
4.27 0.23 0.41
u-tkb
1465 3.99
1.12
4.19
0.94
4.09 0.22 0.32
JPC-JC
1.72
3.02
1.75
3.25 0.27 0.51
2016 best
1150 3.49
JAPIO
1484 4.41
0.68
4.51
0.64
4.46 0.26 0.34
0.92
4.35
1.03
4.31 0.33 0.48
EHR1414 4.27
JPC-CJu-tkb
1468 3.84
1.16
4.04
1.36
3.94 0.23 0.43
1.89
3.27
1.76
3.44 0.26 0.52
2016 best
1200 3.61
JAPIO
1448 4.82
0.24
4.87
0.11
4.84 0.55 0.55
JPC-KJ
EHR
1417 4.76
0.30
4.86
0.23
4.81 0.35 0.47
2016 best
1209 4.58
0.32
4.66
0.30
4.62 0.33 0.36
XMUNLP
1511 3.43
1.64
3.60
1.74
3.51 0.22 0.45
IITBC-HE
IITB-MTG 1726 2.14
1.45
2.45
1.87
2.29 0.30 0.51
XMUNLP
1576 3.95
1.18
3.76
1.85
3.86 0.17 0.36
IITBC-EH
IITB-MTG 1725 2.78
1.74
2.58
1.87
2.68 0.15 0.38
2016 best
1032 3.20
1.33
3.53
1.19
3.36 0.10 0.16
Online A
1523 3.03
1.60
3.28
2.24
3.15 0.15 0.37
JIJI-JE
NTT
1599 1.87
1.25
2.23
1.69
2.05 0.26 0.46
XMUNLP
1442 1.91
1.26
2.19
1.56
2.05 0.24 0.44
Online A
1518 3.31
1.92
3.78
2.06
3.54 0.23 0.50
JIJI-EJ
NTT
1679 1.78
1.18
2.28
1.97
2.03 0.29 0.52
XMUNLP
1443 1.72
1.02
2.20
1.70
1.96 0.33 0.51
XMUNLP
1637 3.90
1.98
3.62
1.57
3.76 0.30 0.56
RECIPE-TTL-JE
Online A
1534 3.52
2.04
3.16
2.07
3.34 0.36 0.60
Online B
1533 4.56
0.55
3.84
2.02
4.20 0.25 0.35
RECIPE-TTL-EJ
XMUNLP
1636 4.54
0.62
3.62
2.40
4.08 0.26 0.34
XMULNP
1635 4.68
0.65
4.58
0.85
4.63 0.47 0.67
RECIPE-ING-JE
Online A
1544 4.29
1.44
4.23
1.57
4.26 0.55 0.76
XMUNLP
1634 4.71
0.43
4.43
1.03
4.57 0.40 0.53
RECIPE-ING-EJ
Online A
1542 4.50
0.95
4.54
0.91
4.52 0.50 0.65
XMUNLP
1632 4.61
0.76
3.98
0.96
4.29 0.13 0.28
RECIPE-STE-JE
Online A
1551 3.34
1.54
2.69
1.21
3.01 0.14 0.36
0.36
4.04
1.33
4.39 0.12 0.21
XMUNLP
1633 4.75
RECIPE-STE-EJ
Online A
1549 4.18
0.42
3.16
1.52
3.67 0.11 0.17
Subtask

Table 9: JPO adequacy evaluation results in detail.

33

TMU (1695)

ORGANIZER (1333)

TMU (1703)

CUNI (1665)

NICT-2 (1476)

Kyoto-U (1733)

NICT-2 (1480)

NTT (1681)
ORGANIZER (1736)
NTT (1616)- ≫≫≫≫≫≫≫≫≫> ≫ ≫ ≫≫≫≫≫≫- - ≫≫≫≫≫≫- ≫≫≫≫≫≫≫≫≫≫≫≫- ≫≫≫≫
≫ ≫≫≫
> ≫≫
- ≫
≫

Kyoto-U (1717)
NTT (1681)
ORGANIZER (1736)
NTT (1616)
Kyoto-U (1733)
NICT-2 (1480)
NICT-2 (1476)
CUNI (1665)
ORGANIZER (1333)
TMU (1703)

NTT (1729)
NICT-2 (1479)
ORGANIZER (1334)
NTT (1684)
NAIST-NICT (1507)
ORGANIZER (1737)
Kyoto-U (1731)
UT-IIS (1710)
NAIST-NICT (1506)
NICT-2 (1475)
TMU (1709)

TMU (1704)

TMU (1709)

NICT-2 (1475)

NAIST-NICT (1506)

UT-IIS (1710)

Kyoto-U (1731)

ORGANIZER (1737)

NAIST-NICT (1507)

NTT (1684)

NICT-2 (1479)
ORGANIZER (1334)

Table 10: Statistical significance testing of the ASPEC-JE Pairwise scores.

- - ≫≫≫≫≫≫≫≫≫- ≫ ≫≫≫≫≫≫≫≫- ≫≫≫≫≫≫≫≫≫ ≫ ≫ ≫≫≫≫≫- - > ≫≫≫≫
- > ≫≫≫≫> ≫≫≫≫
≫≫≫≫
- ≫≫
≫≫
≫

ORGANIZER (1342)

NICT-2 (1477)

ORGANIZER (1740)- > ≫≫≫≫- > ≫≫≫- ≫ ≫≫> ≫≫≫≫≫NICT-2 (1481)

TMU (1743)

NICT-2 (1478)

Kyoto-U (1577)

Kyoto-U (1722)
Kyoto-U (1642)
ORGANIZER (1738)
NICT-2 (1483)
NICT-2 (1478)
ORGANIZER (1336)

ORGANIZER (1336)

Kyoto-U (1642)
ORGANIZER (1738)
NICT-2 (1483)

Table 11: Statistical significance testing of the ASPEC-EJ Pairwise scores.

Kyoto-U (1720)≫≫≫≫≫
Kyoto-U (1577)- - - ≫
NICT-2 (1481)- - ≫ORGANIZER (1740)- ≫NICT-2 (1477)≫

Table 12: Statistical significance testing of the ASPEC-JC (left) and ASPEC-CJ (right) Pairwisescores.

34

EHR (1407)
EHR (1406)
JAPIO (1454)
u-tkb (1470)
ORGANIZER (1339)

JAPIO (1462)

ORGANIZER (1339)

u-tkb (1470)

JAPIO (1454)

EHR (1406)

u-tkb (1472)

JAPIO (1578)

CUNI (1666)

JAPIO (1574)

ORGANIZER (1338) ≫ ≫ ≫ ≫JAPIO (1574)- ≫≫JAPIO (1578)≫≫
CUNI (1666)≫- ≫≫≫≫
- ≫≫≫
≫≫≫
- ≫
≫

ORGANIZER (1341)

u-tkb (1468)

JAPIO (1447)

EHR (1408)

u-tkb (1465)

EHR (1414)

Table 13: Statistical significance testing of the JPC-JE (left) and JPC-EJ (right) Pairwise scores.

JAPIO (1484) ≫ ≫ ≫ ≫ ≫EHR (1414)
- ≫≫≫
≫≫≫
EHR (1408)
JAPIO (1447)≫ ≫u-tkb (1468)

ORGANIZER (1340) ≫JAPIO (1448)
JAPIO (1450)
EHR (1417)
EHR (1416)

ORGANIZER (1344)

EHR (1416)

EHR (1417)

JAPIO (1450)

Table 14: Statistical significance testing of the JPC-JC (left) and JPC-CJ (right) Pairwise scores.

- ≫≫≫
≫≫≫
≫≫
≫

IITB-MTG (1726)

IITB-MTG (1725)

Table 15: Statistical significance testing of the JPC-KJ Pairwise scores.

XMUNLP (1511) ≫XMUNLP (1576) ≫Table 16: Statistical significance testing of the IITBC-HE (left) and IITBC-EH (right) Pairwisescores.

35

ORGANIZER (1395)

XMUNLP (1443)

NTT (1603)

ORGANIZER (1514)

NTT (1679)

CUNI (1668)

NICT-2 (1473)

NICT-2 (1474)

ORGANIZER (1396)

XMUNLP (1442)

NTT (1677)

NTT (1599)

ORGANIZER (1526)

ORGANIZER (1523) ≫ ≫ ≫ ≫ ≫ ≫ ≫ ≫ORGANIZER (1526)≫≫≫≫≫≫≫NTT (1599)≫≫≫≫≫≫NTT (1677)≫≫≫≫≫
XMUNLP (1442)
≫≫≫≫
> ≫≫
ORGANIZER (1396)
NICT-2 (1474)≫≫
NICT-2 (1473)≫

ORGANIZER (1518) ≫ ≫ ≫ ≫ ≫ORGANIZER (1514)
≫≫≫≫
≫ ≫≫
NTT (1679)
NTT (1603)- ≫XMUNLP (1443)-

ORGANIZER (1534)
XMUNLP (1637)

ORGANIZER (1533)- ≫≫ORGANIZER (1528)

ORGANIZER (1531)

XMUNLP (1637)

Table 17: Statistical significance testing of the JIJI-JE (left) and JIJI-EJ (right) Pairwise scores.

XMUNLP (1636)≫≫≫
ORGANIZER (1533)

XMUNLP (1635)≫≫
ORGANIZER (1544)≫

ORGANIZER (1537)

ORGANIZER (1542)

ORGANIZER (1539)

ORGANIZER (1544)

Table 18: Statistical significance testing of the RECIPE-TTL-JE (left) and RECIPE-TTL-EJ(right) Pairwise scores.

XMUNLP (1634)≫≫
ORGANIZER (1542)≫

XMUNLP (1632)≫≫
ORGANIZER (1551)≫

ORGANIZER (1546)

ORGANIZER (1549)

ORGANIZER (1548)

ORGANIZER (1551)

Table 19: Statistical significance testing of the RECIPE-ING-JE (left) and RECIPE-ING-EJ(right) Pairwise scores.

XMUNLP (1633)≫≫
ORGANIZER (1549)≫

Table 20: Statistical significance testing of the RECIPE-STE-JE (left) and RECIPE-STE-EJ(right) Pairwise scores.

36

ASPEC-JE
SYSTEMDATA κ
Online D 1333 0.230
AIAYN 1736 0.204
Kyoto-U 1717 0.217
Kyoto-U 1733 0.204
TMU
1695 0.188
TMU
1703 0.191
1616 0.201
NTTNTT
1681 0.173
NICT-2 1476 0.274
NICT-2 1480 0.257
CUNI
1665 0.241
ave.
0.216

ASPEC-EJSYSTEMDATA κ
Online A
1334 0.290
AIAYN
1737 0.338
Kyoto-U1731 0.321
TMU
1704 0.269
TMU
1709 0.260
NTT1684 0.353
1729 0.341
NTT
NICT-21475 0.315
NICT-21479 0.395
UT-IIS
1710 0.305
NAIST-NICT 1506 0.301
NAIST-NICT 1507 0.339
ave.
0.319

ASPEC-JCSYSTEMDATA κ
Online D 1336 0.189
AIAYN 1738 0.183
Kyoto-U 1642 0.159
Kyoto-U 1722 0.128
TMU
1743 0.171
NICT-2 1478 0.222
NICT-2 1483 0.194
ave.
0.178

ASPEC-CJSYSTEMDATA κ
Online A 1342 0.215
AIAYN 1740 0.310
Kyoto-U 1577 0.284
Kyoto-U 1720 0.254
NICT-2 1477 0.191
NICT-2 1481 0.279
ave.
0.255

JPC-JE
JPC-EJ
JPC-JC
JPC-CJ
JPC-KJ
SYSTEMDATA κ SYSTEMDATA κ SYSTEMDATA κ SYSTEMDATA κ SYSTEMDATA κ
Online A 1338 0.424 Online A 1339 0.410 Online A 1340 0.185 Online A 1341 0.194 Online A 1344 0.257
JAPIO
1574 0.280 EHR1406 0.364 u-tkb
1465 0.176 EHR
1408 0.201 EHR
1416 0.413
1578 0.296 EHR1407 0.385 ave.
1414 0.170 EHR1417 0.459
JAPIO
0.180 EHR
CUNI
JAPIO
1666 0.249 JAPIO
1454 0.409
1447 0.257 JAPIO
1448 0.224
1472 0.380 JAPIO
1462 0.280
1484 0.247 JAPIO
1450 0.235
u-tkb
JAPIO
1470 0.349
u-tkb
1468 0.172 ave.
ave.
0.326 u-tkb
0.317
ave.
0.366
ave.
0.207
IITBC-HE
IITBC-EH
JIJI-JESYSTEM DATA κ SYSTEM DATA κ SYSTEM DATA κXMUNLP 1511 0.376 XMUNLP 1576 0.269 Hiero
1396 0.117
IITB-MTG 1726 0.626 IITB-MTG 1725 0.371 Online A 1523 0.035
ave.
0.501 ave.
0.320 RBMT B 1526 0.004
1599 0.095
NTTNTT
1677 0.077
NICT-21473 0.078
NICT-21474 0.064
XMUNLP 1442 0.070
CUNI
1668 0.060
ave.
0.067

JIJI-EJSYSTEM DATA κ
Hiero1395 0.104
RBMT A 1514 0.167
Online A 1518 0.179
1603 0.189
NTTNTT
1679 0.155
XMUNLP 1443 0.151
ave.
0.157

RECIPE-TTL-JE
SYSTEM DATA κ
RBMT B 1531 0.305
Online A 1534 0.333
XMUNLP 1637 0.366
ave.
0.334

RECIPE-TTL-EJ
SYSTEM DATA κ
RBMT B 1528 0.340
Online B 1533 0.356
XMUNLP 1636 0.341
ave.
0.345

RECIPE-STE-JE
SYSTEM DATA κ
RBMT B 1548 0.290
Online A 1551 0.289
XMUNLP 1632 0.261
ave.
0.280

RECIPE-STE-EJSYSTEM DATA κ
RBMT B 1546 0.108
Online A 1549 0.138
XMUNLP 1633 0.162
ave.
0.136

RECIPE-ING-JE
SYSTEM DATA κ
RBMT B 1539 0.537
Online B 1544 0.551
XMUNLP 1635 0.614
ave.
0.567

RECIPE-ING-EJ
SYSTEM DATA κ
RBMT B 1537 0.665
Online B 1542 0.515
XMUNLP 1634 0.618
ave.
0.599

Table 21: The Fleiss’ kappa values for the pairwise evaluation results.

37

38

SYSTEM ID
SMT Hiero
SMT Phrase
SMT S2T
Online D (2014)
RBMT ERBMT F
Online C (2014)
RBMT D (2014)
Online D (2015)
SMT S2T
RBMT D (2015)
Online C (2015)
Online D (2016)
Online D (2016/11)
AIAYN
Kyoto-U 1
Kyoto-U 2
TMU 1
TMU 2
NTT 1
NTT 2
NICT-2 1
NICT-2 2
CUNI 1RIBES
0.651066
0.645137
0.678253
0.643588
0.663851
0.661387
0.624827
0.683378
0.676609
0.678253
0.683378
0.622564
0.677412
0.733483
0.767577
0.761403
0.765464
0.725284
0.741175
0.764831
0.768880
0.747335
0.741329
0.741699

Table 22: ASPEC-JE submissionsID
METHOD OTHER BLEU
2
SMT
NO
18.72
6
SMT
NO
18.45
9
SMT
NO
20.36
35
Other
YES
15.08
76
Other
YES
14.82
79
Other
YES
13.86
87
Other
YES
10.64
96
Other
YES
15.29
775
Other
YES
16.85
877
SMT
NO
20.36
887
Other
YES
15.29
892
Other
YES
10.29
1042
Other
YES
16.91
1333
NMT
YES
22.04
1736
NMT
NO
28.06
1717
NMT
NO
27.53
1733
NMT
NO
27.66
1695
NMT
NO
21.00
1703
NMT
NO
23.03
1616
NMT
NO
27.43
1681
NMT
NO
28.36
1476
NMT
NO
24.79
1480
NMT
NO
26.76
1665
NMT
NO
23.43

AMFM
0.588880
0.590950
0.593410
0.564170
0.561620
0.556840
0.466480
0.551690
0.562270
0.593410
0.551690
0.453370
0.564270
0.584390
0.595580
0.585540
0.591160
0.585710
0.595260
0.597620
0.597860
0.574810
0.578150
0.583780

Pair
+7.75
—–
+25.50+13.75—–—–—–+23.00+0.25+7.00+16.75—–+28.00+63.00+75.25+77.75+74.50+56.75+61.00+75.00+77.25+68.75+69.75+66.00

39

ID

5
12
34
66
68
91
95
367
774
875
883
889
1041
1334
1737

SYSTEM ID

SMT Phrase
SMT T2S
Online A (2014)
RBMT B (2014)
RBMT A
Online B (2014)
RBMT CSMT Hiero
Online A (2015)
SMT T2S
RBMT B (2015)
Online B (2015)
Online A (2016)
Online A (2016/11)
AIAYN

SMT
SMT
Other
Other
Other
Other
Other
SMT
Other
SMT
Other
Other
Other
NMT
NMT

NO
NO
YES
YES
YES
YES
YES
NO
YES
NO
YES
YES
YES
YES
NO

METHOD OTHER

RIBES
kytea
0.691926
0.758031
0.723486
0.680748
0.676464
0.693390
0.672645
0.7469780.713960
0.758031
0.6807480.701966
0.715222
0.780217
0.847559
mecab
0.695390
0.760516
0.725848
0.682683
0.678934
0.698126
0.676018
0.747722
0.718150
0.760516
0.682683
0.703859
0.718559
0.782674
0.851471

Table 23: ASPEC-EJ submissions (Organizer)

BLEU
juman kytea mecab juman
27.48 29.80 28.27 0.683735
31.05 33.44 32.10 0.748883
19.66 21.63 20.17 0.718019
13.18 14.85 13.48 0.671958
12.86 14.43 13.16 0.670167
17.04 18.67 17.36 0.687797
12.19 13.32 12.14 0.668372
30.19 32.56 30.94 0.734705
18.22 19.77 18.46 0.705882
31.05 33.44 32.10 0.748883
13.18 14.85 13.48 0.671958
17.80 19.52 18.11 0.693359
18.28 19.81 18.51 0.706639
26.19 28.22 26.68 0.776787
40.79 42.55 41.50 0.844896
juman
0.736380
0.744370
0.695420
0.622930
0.626940
0.643070
0.594380
0.743900
0.6772000.744370
0.622930
0.646160
0.677020
0.727040
0.768630

AMFM
kytea
mecab
0.736380 0.736380
0.744370 0.744370
0.695420 0.695420
0.622930 0.622930
0.626940 0.626940
0.643070 0.643070
0.594380 0.594380
0.743900 0.743900
0.677200 0.677200
0.744370 0.744370
0.622930 0.622930
0.646160 0.646160
0.677020 0.677020
0.727040 0.727040
0.768630 0.768630—–+34.25+42.50
+0.75
—–
—–
—–
+31.50+34.25+30.00+9.75—–+49.75+74.00+69.75

Pair

40

ID

1731
1704
1709
1684
1729
1475
1479
1710
1506
1507

SYSTEM ID

Kyoto-U 1
TMU 1
TMU 2
NTT 1
NTT 2
NICT-2 1
NICT-2 2
UT-IIS 1
NAIST-NICT 1
NAIST-NICT 2

NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

METHOD OTHER

RIBES
kytea
0.835870
0.809649
0.818443
0.839981
0.841769
0.834448
0.848170
0.832054
0.827225
0.839321
mecab
0.839646
0.811057
0.821563
0.844326
0.846486
0.835255
0.849929
0.836394
0.830116
0.842337

Table 24: ASPEC-EJ submissions (Participants)

BLEU
juman kytea mecab juman
38.72 40.65 39.37 0.832472
32.65 35.05 33.72 0.802262
34.05 36.69 35.32 0.812926
39.80 42.27 40.47 0.835806
40.32 42.80 40.95 0.838594
36.85 38.94 37.87 0.826791
40.17 42.25 41.17 0.842206
36.26 38.93 37.06 0.827891
36.47 38.54 37.30 0.821989
38.25 40.29 39.05 0.834492
juman
0.7542200.740620
0.744890
0.757740
0.762170
0.759570
0.765580
0.746910
0.7633100.770480

AMFM
kytea
mecab
0.754220 0.754220
0.740620 0.740620
0.744890 0.744890
0.757740 0.757740
0.762170 0.762170
0.759570 0.759570
0.765580 0.765580
0.746910 0.746910
0.763310 0.763310
0.770480 0.770480+69.75+50.75+56.50+72.25+75.75+62.00+74.75+68.00+63.50+70.00

Pair

41

ID

3
7
10
37
216
243
244
777
881
886
891
1045
1336
1738
1642
1722
1743
1478
1483

SYSTEM ID

SMT Hiero
SMT Phrase
SMT S2T
Online D (2014)
Online C (2014)
RBMT B (2014)
RBMT C
Online D (2015)
SMT S2T
RBMT B (2015)
Online C (2015)
Online D (2016)
Online D (2016/11)
AIAYN
Kyoto-U 1
Kyoto-U 2
TMU 1
NICT-2 1
NICT-2 2

SMT
SMT
SMT
Other
Other
RBMT
RBMT
Other
SMT
Other
Other
Other
NMT
NMT
NMT
NMT
NMT
NMT
NMT

METHOD
NO
NO
NO
YES
YES
NO
NO
YES
NO
YES
YES
YES
YES
NO
NO
NO
NO
NO
NO

OTHER

RIBES
stanford(ctb)
0.809561
0.790263
0.809457
0.606328
0.613075
0.745885
0.648758
0.660847
0.809457
0.745885
0.615048
0.667382
0.7282700.850052
0.848107
0.849168
0.798736
0.846578
0.851893

Table 25: ASPEC-JC submissions

27.71
27.96
28.65
9.37
7.26
17.86
9.62
10.73
28.65
17.86
7.44
11.16
15.94
34.97
35.67
35.31
22.92
33.72
35.23

kytea

BLEU
stanfordstanfordkytea(ctb) (pku)27.70 27.35 0.809128
28.01 27.68 0.788961
28.65 28.35 0.807606
8.93
8.84
0.606905
7.01
6.72
0.612808
17.75 17.49 0.744818
9.96
9.59
0.642278
10.33 10.08 0.660484
28.65 28.35 0.807606
17.75 17.49 0.744818
7.05
6.75
0.611964
10.72 10.54 0.665185
15.68 15.38 0.728453
34.96 34.72 0.850199
35.30 35.40 0.849464
35.37 35.06 0.850103
22.86 22.74 0.798681
33.64 33.60 0.847223
35.23 35.14 0.852084
stanford(pku)
0.811394
0.790937
0.808417
0.604149
0.611563
0.743794
0.645385
0.660482
0.808417
0.743794
0.612158
0.666953
0.728284
0.848394
0.848318
0.847879
0.797969
0.846158
0.851548
0.7451000.749450
0.755230
0.625430
0.587820
0.667960
0.594900
0.634090
0.755230
0.667960
0.566060
0.639440
0.673730
0.787250
0.779400
0.7854200.700030
0.779870
0.785820

kytea

AMFM
stanford stanford
(ctb)
(pku)
0.745100 0.745100
0.749450 0.749450
0.755230 0.755230
0.625430 0.625430
0.587820 0.587820
0.667960 0.667960
0.594900 0.594900
0.634090 0.634090
0.755230 0.755230
0.667960 0.667960
0.566060 0.566060
0.639440 0.639440
0.673730 0.673730
0.787250 0.787250
0.779400 0.779400
0.785420 0.785420
0.700030 0.700030
0.779870 0.779870
0.785820 0.785820

+3.75
—–
+14.00-14.50—–-20.00—–-14.75+7.75-11.00—–-26.00+17.75+70.50+71.50+72.50+4.25+67.25+69.50

Pair

42

ID

4
8
13
36
215
239
242
776
879
885
890
1043
1342
1740
1577
1720
1477
1481

SYSTEM ID

SMT Hiero
SMT Phrase
SMT T2S
Online A (2014)
Online B (2014)
RBMT A (2014)
RBMT D
Online A (2015)
SMT T2S
RBMT A (2015)
Online B (2015)
Online A (2016)
Online A (2016/11)
AIAYN
Kyoto-U 1
Kyoto-U 2
NICT-2 1
NICT-2 2

SMT
SMT
SMT
Other
Other
RBMT
RBMT
Other
SMT
Other
Other
Other
NMT
NMT
NMT
NMT
NMT
NMT

NO
NO
NO
YES
YES
NO
NO
YES
NO
YES
YES
YES
YES
NO
NO
NO
NO
NO

METHOD OTHER

RIBES
kytea
0.798726
0.766384
0.820490
0.598172
0.596006
0.652402
0.626400
0.590393
0.820490
0.652402
0.592841
0.589397
0.717173
0.875511
0.878964
0.880069
0.868359
0.878580

Table 26: ASPEC-CJ submissions

BLEU
juman kytea mecab juman
35.43 35.91 35.64 0.810406
34.65 35.16 34.77 0.772498
36.52 37.07 36.64 0.825292
11.63 13.21 11.87 0.595925
10.48 11.26 10.47 0.600733
9.37
9.87
9.35
0.666277
8.39
8.70
8.30
0.641189
11.53 12.82 11.68 0.588285
36.52 37.07 36.64 0.825292
9.37
9.87
9.35
0.666277
10.41 11.03 10.36 0.597355
11.56 12.87 11.69 0.589802
18.75 20.64 19.04 0.719022
46.87 47.30 47.00 0.880815
48.43 48.84 48.51 0.883457
48.34 48.76 48.40 0.884210
44.26 44.90 44.50 0.871438
46.84 47.51 47.27 0.882356
mecab
0.807665
0.771005
0.825025
0.598573
0.600706
0.661730
0.633319
0.592887
0.825025
0.661730
0.597298
0.593361
0.720095
0.880368
0.884137
0.884745
0.871736
0.882195juman
0.750950
0.753010
0.754870
0.658060
0.636930
0.626070
0.586790
0.649860
0.754870
0.626070
0.628290
0.659540
0.692820
0.798110
0.799520
0.799840
0.788940
0.799680

AMFM
kytea
mecab
0.750950 0.750950
0.753010 0.753010
0.754870 0.754870
0.658060 0.658060
0.636930 0.636930
0.626070 0.626070
0.586790 0.586790
0.649860 0.649860
0.754870 0.754870
0.626070 0.626070
0.628290 0.628290
0.659540 0.659540
0.692820 0.692820
0.798110 0.798110
0.799520 0.799520
0.799840 0.799840
0.788940 0.788940
0.799680 0.799680

+4.75
—–
+16.00-21.75—–-37.75—–-19.00+17.25-28.00—–-51.25+22.50+78.50+79.50+82.75+78.00+79.00Pair

43

SYSTEM ID
SMT Phrase
SMT Hiero
SMT S2T
Online A (2016)
Online B (2016)
RBMT C (2016)
RBMT A (2016)
RBMT B (2016)
Online A (2016/11)
JAPIO 1JAPIO 2CUNI 1
u-tkb 1RIBES
0.730056
0.763030
0.793483
0.803661
0.688004
0.755017
0.750381
0.710992
0.878342
0.878298
0.873093
0.837425
0.841136

Table 27: JPC-JE submissionsID
METHOD OTHER BLEU
977
SMT
NO
30.80
979
SMT
NO
32.23
980
SMT
NO
34.40
1035
Other
YES
35.77
1051
Other
YES
16.00
1088
Other
YES
21.00
1090
Other
YES
21.57
1095
Other
YES
18.38
1338
NMT
YES
49.35
1574
NMT
YES
49.00
1578
NMT
YES
48.08
1666
SMT
NO
38.29
1472
NMT
NO
37.31

AMFM
0.664830
0.672500
0.672760
0.673950
0.486450
0.519210
0.521230
0.518110
0.722590
0.724710
0.715560
0.681520
0.697290

Pair
—–
+8.75
+23.00+32.25—–—–+23.75—–+71.50+68.50+67.00+58.00+51.50

44

ID

973
974
975
1036
1073
1085
1086
1087
1339
1406
1407
1454
1462
1470

SYSTEM ID

SMT Phrase
SMT Hiero
SMT T2S
Online A (2016)
Online B (2016)
RBMT D (2016)
RBMT F (2016)
RBMT E (2016)
Online A (2016/11)
EHR 1
EHR 2
JAPIO 1JAPIO 2
u-tkb 1

SMT
SMT
SMT
Other
Other
Other
Other
Other
NMT
NMT
NMT
NMT
SMT
NMT

NO
NO
NO
YES
YES
YES
YES
YES
YES
NO
NO
YES
YES
NO

METHOD OTHER

RIBES
kytea
0.728281
0.778657
0.796783
0.792471
0.735203
0.757341
0.769244
0.741985
0.877336
0.858466
0.864256
0.883481
0.861596
0.846888

Table 28: JPC-EJ submissions

BLEU
juman kytea mecab juman
32.36 34.26 32.52 0.728539
34.57 36.61 34.79 0.777759
35.60 37.65 35.82 0.797353
36.88 37.89 36.83 0.798168
21.57 22.62 21.65 0.743083
23.02 24.90 23.45 0.761224
26.64 28.48 26.84 0.773673
21.35 23.17 21.53 0.743484
50.60 51.65 50.83 0.879382
44.44 45.59 44.15 0.860998
44.63 45.94 44.53 0.866722
50.27 51.23 50.17 0.886403
51.79 52.23 51.75 0.864038
38.91 41.12 39.11 0.845815
mecab
0.729077
0.779049
0.798025
0.796308
0.740962
0.760325
0.773344
0.742300
0.878316
0.860659
0.866205
0.885747
0.8622000.845551

juman
0.711900
0.715300
0.717030
0.7191100.659950
0.647730
0.675470
0.646930
0.770480
0.747050
0.747770
0.776790
0.781150
0.734010

AMFM
kytea
mecab
0.711900 0.711900
0.715300 0.715300
0.717030 0.717030
0.719110 0.719110
0.659950 0.659950
0.647730 0.647730
0.675470 0.675470
0.646930 0.646930
0.770480 0.770480
0.747050 0.747050
0.747770 0.747770
0.776790 0.776790
0.781150 0.781150
0.734010 0.734010—–+21.00+30.75+20.00—–—–+12.75—–+48.50+58.25+60.00+56.25+41.00+49.50

Pair

45

ID

966
967
968
1038
1069
1118
1340
1465

SYSTEM ID

SMT Phrase
SMT Hiero
SMT S2T
Online A (2016)
Online B (2016)
RBMT C (2016)
Online A (2016/11)
u-tkb 1

SMT
SMT
SMT
Other
Other
Other
NMT
NMT

METHOD
NO
NO
NO
YES
YES
YES
YES
NO

OTHER

RIBES
stanford(ctb)
0.797888
0.799118
0.802805
0.760672
0.651070
0.708681
0.829122
0.826055

Table 29: JPC-JC submissions

BLEU
kytea stanfordstanfordkytea(ctb) (pku)30.60 32.03 31.25 0.787321
30.26 31.57 30.91 0.788415
31.05 32.35 31.70 0.793846
23.02 23.57 23.29 0.754241
9.42
9.59
8.79
0.642026
12.35 13.72 13.17 0.688240
33.04 33.92 33.34 0.824829
31.80 33.19 32.83 0.819791
stanford(pku)
0.794388
0.796685
0.800848
0.760148
0.643520
0.700210
0.829067
0.825025

0.710940
0.718360
0.720030
0.702350
0.527180
0.475430
0.735470
0.706720

kytea

AMFM
Pair
stanford stanford
(ctb)
(pku)
0.710940 0.710940—–
0.718360 0.718360 +4.75
0.720030 0.720030 +4.25
0.702350 0.702350 -23.00
0.527180 0.527180—–
0.475430 0.475430 -41.25
0.735470 0.735470 +32.50
0.706720 0.706720 +21.75

46

ID

430
431
432
647
648
759
760
1040
1341
1408
1414
1447
1484
1468

SYSTEM ID

SMT Hiero
SMT Phrase
SMT T2S
Online A (2015)
Online B (2015)
RBMT A (2015)
RBMT B
Online A (2016)
Online A (2016/11)
EHR 1
EHR 2
JAPIO 1JAPIO 2
u-tkb 1

SMT
SMT
SMT
Other
Other
RBMT
RBMT
Other
NMT
NMT
NMT
SMT
NMT
NMT

NO
NO
NO
YES
YES
NO
NO
YES
YES
NO
NO
YES
YES
NO

METHOD OTHER

RIBES
kytea
0.802059
0.778921
0.811350
0.707264
0.641255
0.664098
0.581837
0.702718
0.844918
0.856376
0.856784
0.843774
0.873390
0.833610

Table 30: JPC-CJ submissions

BLEU
juman kytea mecab juman
39.22 39.52 39.14 0.806058
38.34 38.51 38.22 0.782019
39.39 39.90 39.39 0.814919
26.80 27.81 26.89 0.712242
12.33 12.72 12.44 0.648996
10.49 10.72 10.35 0.674060
7.94
8.07
7.73
0.596200
26.99 27.91 27.02 0.707739
42.66 43.76 42.95 0.845858
47.08 47.44 46.83 0.859070
46.52 47.17 46.35 0.859619
50.52 51.25 50.57 0.847793
50.06 50.51 50.00 0.875398
38.79 40.47 38.99 0.832144
mecab
0.804523
0.781456
0.813595
0.711273
0.648742
0.667349
0.586941
0.706707
0.845794
0.858888
0.858353
0.846081
0.874822
0.831209

juman
0.729370
0.7231100.725920
0.693840
0.588380
0.5571300.502100
0.693720
0.747240
0.756350
0.761370
0.774660
0.779420
0.729580

AMFM
kytea
mecab
0.729370 0.729370
0.723110 0.723110
0.725920 0.725920
0.693840 0.693840
0.588380 0.588380
0.557130 0.557130
0.502100 0.502100
0.693720 0.693720
0.747240 0.747240
0.756350 0.756350
0.761370 0.761370
0.774660 0.774660
0.779420 0.779420
0.729580 0.729580—–—–+20.75-7.00—–-39.25—–-19.75+54.25+68.25+69.75+60.50+80.25+55.50

Pair

47

ID

438
439
651
652
653
654
963
964
1039
1344
1416
1417
1448
1450

SYSTEM ID

SMT Phrase
SMT Hiero
Online B (2015)
Online A (2015)
RBMT A (2015)
RBMT B
Online A (2015)
RBMT A (2015)
Online A (2016)
Online A (2016/11)
EHR 1
EHR 2
JAPIO 1JAPIO 2

SMT
SMT
Other
Other
Other
Other
Other
Other
Other
NMT
NMT
NMT
SMT
SMT

NO
NO
YES
YES
YES
YES
YES
YES
YES
NO
NO
NO
YES
YES

METHOD OTHER

RIBES
kytea
0.939729
0.935903
0.852263
0.909385
0.873734
0.849014
0.909385
0.873734
0.907652
0.854158
0.942940
0.944812
0.945754
0.945841

Table 31: JPC-KJ submissions

BLEU
juman kytea mecab juman
69.22 70.36 69.73 0.941302
67.41 68.65 68.00 0.937162
36.41 38.72 37.01 0.851745
55.05 56.84 55.46 0.909152
42.00 43.97 42.45 0.876396
34.74 37.51 35.54 0.845712
55.05 56.84 55.46 0.909152
42.00 43.97 42.45 0.876396
54.78 56.68 55.14 0.907320
44.42 45.14 44.72 0.857642
71.52 72.34 71.82 0.944516
71.36 72.26 71.65 0.946126
73.00 73.71 73.23 0.946880
73.00 73.73 73.25 0.946985
mecab
0.940756
0.936570
0.851945
0.908838
0.875146
0.846228
0.908838
0.875146
0.906743
0.857083
0.944219
0.945888
0.946645
0.946745

juman
0.8562200.850560
0.728750
0.800460
0.712020
0.643150
0.800610
0.712700
0.798750
0.783850
0.866060
0.8711100.872510
0.873200

AMFM
kytea
mecab
0.856220 0.856220
0.850560 0.850560
0.728750 0.728750
0.800460 0.800460
0.712020 0.712020
0.643150 0.643150
0.800610 0.800610
0.712700 0.712700
0.798750 0.798750
0.783850 0.783850
0.866060 0.866060
0.871110 0.871110
0.872510 0.872510
0.873200 0.873200

—–
+2.75
—–
+38.75
-7.25
—–
—–
—–
+8.00
-55.75+6.25+11.25+48.75+48.50Pair

48

SYSTEM ID
Online A (2016)
Online B (2016)
SMT Phrase
XMUNLP 1
IITB-MTG 1

RIBES
0.714537
0.683214
0.638090
0.750921
0.682902

Table 33: IITB-EH submissions

Pair+44.75+14.000.00+68.25+21.00AMFM
0.670660
0.668450
0.660860
0.688770
0.624780

AMFM
0.6211000.590520
0.574850
0.629530
0.557040

BLEU
RIBES
18.720000 0.716788
16.970000 0.691298
10.790000 0.651166
21.390000 0.749660
12.230000 0.688606

Table 32: IITB-HE submissionsID
METHOD OTHER BLEU
1031
Other
YES
21.37
1048
Other
YES
15.58
1054
SMT
NO
10.32
1511
NMT
NO
22.44
1726
NMT
NO
11.55

ID
METHOD OTHER RESOURCES
1032
Other
YES
1047
Other
YES
1252
SMT
NO
1576
NMT
NO
1725
NMT
NO

SYSTEM ID
Online A (2016)
Online B (2016)
SMT Phrase
XMUNLP 1
IITB-MTG 1

Pair+57.25+42.50—–+64.50+28.75

49

SMT Phrase
SMT Hiero
SMT T2S
RBMT-A
RBMT-BONLINE-A 1
NTT 1
NTT 2
XMUNLP 1

1393
1395
1397
1514
1515
1518
1603
1679
1443

SYSTEM ID ID

SMT
SMT
SMT
RBMT
RBMT
NMT
NMT
NMT
NMT

NO
NO
NO
NO
NO
NO
NO
NO
NO

METHOD OTHER

SYSTEM ID
SMT Phrase
SMT Hiero
SMT S2TONLINE-A 1
RBMT-A
RBMT-B
NTT 1
NTT 2
NICT-2 1
NICT-2 2
XMUNLP 1
CUNI 1RIBES
kytea
0.584892
0.601505
0.597791
0.515050
0.531603
0.605532
0.670920
0.684048
0.688497

Table 35: JIJI-EJ submissions

BLEU
juman kytea mecab juman
15.77 16.65 15.76 0.580284
16.22 16.95 16.22 0.594923
14.95 15.38 14.79 0.594072
5.31
6.68
5.69
0.505227
4.72
5.98
4.97
0.518416
11.29 13.12 11.84 0.597473
19.13 20.47 19.41 0.668517
20.37 21.82 20.68 0.680598
19.61 20.72 20.14 0.684120

AMFM
kytea
mecab
0.545240 0.545240
0.550260 0.550260
0.530370 0.530370
0.473940 0.473940
0.487320 0.487320
0.533120 0.533120
0.536970 0.536970
0.537800 0.537800
0.546360 0.546360

Pair—–+10.25—–+70.00—–+51.75+32.00+26.75+0.25+7.25+20.75-24.00juman
0.5452400.5502600.530370
0.473940
0.487320
0.533120
0.536970
0.537800
0.546360

AMFM
0.475740
0.470610
0.4771700.450850
0.391050
0.385600
0.476200
0.474360
0.459000
0.453420
0.465710
0.432700

mecab
0.585437
0.602516
0.599530
0.513580
0.532079
0.603374
0.676594
0.688863
0.691056

RIBES
0.554550
0.558225
0.556728
0.529844
0.472312
0.475760
0.638841
0.648931
0.642379
0.632638
0.637059
0.564797

Table 34: JIJI-JE submissionsID
METHOD OTHER BLEU
1394
SMT
NO
15.11
1396
SMT
NO
15.67
1398
SMT
NO
14.54
1523
NMT
NO
8.19
1525 RBMT
NO
4.36
1526 RBMT
NO
4.67
1599
NMT
NO
19.44
1677
NMT
NO
20.90
1473
NMT
NO
16.52
1474
NMT
NO
18.19
1442
NMT
NO
17.95
1668
SMT
NO
10.67

—–
+10.25—–+31.25—–+69.75+14.50+17.75+11.75Pair50

RBMT-A
RBMT-BONLINE-B 1
SMT Phrase
XMUNLP 1

1536
1537
1542
1570
1634

SYSTEM ID ID

RBMT
RBMT
NMT
SMT
NMT

NO
NO
NO
NO
NO

METHOD OTHER

SYSTEM ID
RBMT-A
RBMT-BONLINE-B 1
SMT Phrase
XMUNLP 1

RIBES
0.484800
0.520800
0.563419
0.830105
0.831261

RIBES
kytea
0.371584
0.363344
0.544581
0.740283
0.742770

juman
0.426550
0.445370
0.618010
0.775770
0.778530

AMFM
Pair
0.613950—–
0.607190 -24.75
0.656630 -15.50
0.859040—–
0.854970 +3.50

mecab
0.355798
0.372793
0.537147
0.741249
0.739909

Table 37: RECIPEING-EJ submissions

BLEU
juman kytea mecab juman
4.52
4.74
4.29
0.365913
5.44
4.95
5.07
0.385269
15.57 14.89 14.85 0.548026
31.39 30.61 29.60 0.749305
34.88 34.26 33.19 0.747521

Table 36: RECIPEING-JE submissionsID
METHOD OTHER BLEU
1538 RBMT
NO
11.14
1539 RBMT
NO
12.52
1544
NMT
NO
20.33
1571
SMT
NO
44.42
1635
NMT
NO
46.98

AMFM
Pair
kytea
mecab
0.426550 0.426550—–
0.445370 0.445370 -48.50
0.618010 0.618010 -24.25
0.775770 0.775770—–
0.778530 0.778530 -3.75

51

RBMT-A
RBMT-BONLINE-A 1
SMT Phrase
XMUNLP 1

1545
1546
1549
1568
1633

SYSTEM ID ID

RBMT
RBMT
NMT
SMT
NMT

NO
NO
NO
NO
NO

METHOD OTHER

SYSTEM ID
RBMT-A
RBMT-BONLINE-A 1
SMT Phrase
XMUNLP 1

RIBES
0.546642
0.565086
0.670484
0.705506
0.784235

RIBES
kytea
0.533326
0.532919
0.635888
0.698499
0.776469

juman
0.368710
0.441180
0.526680
0.626610
0.645050

AMFM
Pair
kytea
mecab
0.368710 0.368710—–
0.441180 0.441180 -65.50
0.526680 0.526680 -29.25
0.626610 0.626610—–
0.645050 0.645050 +45.50

AMFM
Pair
0.315930—–
0.268580 -60.25
0.415880 +2.75
0.595290—–
0.598050 +40.50

mecab
0.521713
0.529113
0.629012
0.695400
0.775689

Table 39: RECIPESTE-EJ submissions

BLEU
juman kytea mecab juman
3.06
4.55
3.43
0.518467
3.30
5.19
4.00
0.525116
8.14
11.23 8.97
0.623865
17.60 21.43 18.53 0.694179
22.55 26.87 24.00 0.776539

Table 38: RECIPESTE-JE submissionsID
METHOD OTHER BLEU
1547 RBMT
NO
5.37
1548 RBMT
NO
5.82
1551
NMT
NO
11.04
1569
SMT
NO
22.84
1632
NMT
NO
28.03

52

RBMT-A
RBMT-BONLINE-B 1
SMT Phrase
XMUNLP 1

1527
1528
1533
1566
1636

SYSTEM ID ID

RBMT
RBMT
NMT
SMT
NMT

NO
NO
NO
NO
NO

METHOD OTHER

SYSTEM ID
RBMT-A
RBMT-BONLINE-A 1
SMT Phrase
XMUNLP 1

RIBES
0.086378
0.100899
0.199338
0.451707
0.526993

RIBES
kytea
0.137884
0.300578
0.559142
0.576617
0.573466

juman
0.187540
0.402830
0.590440
0.571650
0.584980

AMFM
Pair
kytea
mecab
0.187540 0.187540—–
0.402830 0.402830 -50.25
0.590440 0.590440 +3.75
0.571650 0.571650—–
0.584980 0.584980 +23.75

AMFM
Pair
0.433380—–
0.445400 -25.75
0.509470 +10.25
0.571230—–
0.542690 +10.25

mecab
0.122371
0.270482
0.5321300.548811
0.558997

Table 41: RECIPETTL-EJ submissions

BLEU
juman kytea mecab juman
0.00
0.00
0.00
0.128134
2.57
2.30
2.08
0.299133
16.16 15.85 15.40 0.573771
17.16 16.23 16.57 0.600503
19.41 18.87 18.78 0.592087

Table 40: RECIPETTL-JE submissionsID
METHOD OTHER BLEU
1530 RBMT
NO
0.53
1531 RBMT
NO
0.56
1534
NMT
NO
2.19
1567
SMT
NO
9.72
1637
NMT
NO
15.57

References

wat2017. In Proceedings of the 4th Workshop
on Asian Translation (WAT2017), pages 140–
145, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Rafael E. Banchs, Luis F. D’Haro, and Haizhou
Li. 2015. Adequacy-fluency metrics: Evaluating mt in the continuous space model framework. IEEE/ACM Trans. Audio, Speech andLang. Proc., 23(3):472–482.

Tom Kocmi, Dušan Variš, and Ondřej Bojar. 2017.
Cuni nmt system for wat 2017 translation tasks.
In Proceedings of the 4th Workshop on AsianTranslation (WAT2017), pages 154–159, Taipei,
Taiwan. Asian Federation of Natural Language
Processing.

Jacob Cohen. 1968. Weighted kappa: Nominal
scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213 – 220.

Philipp Koehn. 2004. Statistical significance tests
for machine translation evaluation. In Proceedings of EMNLP 2004, pages 388–395, Barcelona,
Spain. Association for Computational Linguistics.

Fabien Cromieres, Raj Dabre, Toshiaki Nakazawa,and Sadao Kurohashi. 2017. Kyoto university
participation to wat 2017. In Proceedings of the
4th Workshop on Asian Translation (WAT2017),
pages 146–153, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Philipp Koehn, Hieu Hoang, Alexandra Birch,
Chris Callison-Burch, Marcello Federico, Nicola
Bertoldi, Brooke Cowan, Wade Shen, Christine
Moran, Richard Zens, Chris Dyer, Ondrej Bojar,
Alexandra Constantin, and Evan Herbst. 2007.
Moses: Open source toolkit for statistical machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL),
demonstration session.

Terumasa Ehara. 2017. Smt reranked nmt. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages 119–126, Taipei, Taiwan. Asian Federation of Natural Language Processing.
J.L. Fleiss et al. 1971. Measuring nominal scaleagreement among many raters. Psychological
Bulletin, 76(5):378–382.

T. Kudo. 2005.
Mecab :
Yet anotherpart-of-speech and morphological analyzer.
http://mecab.sourceforge.net/.

Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H.
Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In
Proceedings of the 51st Annual Meeting of the
Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696, Sofia,
Bulgaria. Association for Computational Linguistics.

Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements of Japanese morphological analyzer JUMAN. In Proceedings of The International
Workshop on Sharable Natural Language, pages
22–28.

Hieu Hoang, Philipp Koehn, and Adam Lopez.
2009. A unified framework for phrase-based,hierarchical, and syntax-based statistical machine translation. In Proceedings of the International Workshop on Spoken Language Translation, pages 152–159.

Zi Long, Ryuichiro Kimura, Takehito Utsuro,
Tomoharu Mitsuhashi, and Mikio Yamamoto.
2017. Patent nmt integrated with large vocabulary phrase translation by smt at wat 2017.
In Proceedings of the 4th Workshop on AsianTranslation (WAT2017), pages 110–118, Taipei,
Taiwan. Asian Federation of Natural Language
Processing.

Kenji Imamura and Eiichiro Sumita. 2017. Ensemble and reranking: Using multiple models in
the nict-2 neural machine translation system at
wat2017. In Proceedings of the 4th Workshop
on Asian Translation (WAT2017), pages 127–
134, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Yukio Matsumura and Mamoru Komachi. 2017.
Tokyo metropolitan university neural machinetranslation system for wat 2017. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages 160–166, Taipei, Taiwan. Asian Federation of Natural Language Processing.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010
Conference on Empirical Methods in Natural
Language Processing, EMNLP ’10, pages 944–
952, Stroudsburg, PA, USA. Association for
Computational Linguistics.

Makoto Morishita, Jun Suzuki, and Masaaki Nagata. 2017. Ntt neural machine translation systems at wat 2017. In Proceedings of the 4thWorkshop on Asian Translation (WAT2017),
pages 89–94, Taipei, Taiwan. Asian Federationof Natural Language Processing.

Satoshi Kinoshita, Tadaaki Oshio, and Tomoharu
Mitsuhashi. 2017. Comparison of smt and nmt
trained with large patent corpora: Japio at

Toshiaki Nakazawa, Chenchen Ding, Hideya
MINO, Isao Goto, Graham Neubig, and Sadao

53

Kurohashi. 2016. Overview of the 3rd workshop on asian translation. In Proceedings of the
3rd Workshop on Asian Translation (WAT2016),
pages 1–46, Osaka, Japan. The COLING 2016
Organizing Committee.

2017. Comparing recurrent and convolutionalarchitectures for english-hindi neural machinetranslation. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages
167–170, Taipei, Taiwan. Asian Federation of
Natural Language Processing.

Toshiaki Nakazawa, Hideya Mino, Isao Goto,
Sadao Kurohashi, and Eiichiro Sumita. 2014.
Overview of the 1st Workshop on Asian Translation. In Proceedings of the 1st Workshop
on Asian Translation (WAT2014), pages 1–19,
Tokyo, Japan.

Huihsin Tseng. 2005. A conditional random field
word segmenter. In In Fourth SIGHAN Workshop on Chinese Language Processing.
Masao Utiyama and Hitoshi Isahara. 2007. Ajapanese-english patent parallel corpus. In MT
summit XI, pages 475–482.

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro
Sumita. 2015. Overview of the 2nd Workshop
on Asian Translation. In Proceedings of the 2nd
Workshop on Asian Translation (WAT2015),
pages 1–28, Kyoto, Japan.

Ashish Vaswani, Noam Shazeer, Niki Parmar,
Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Boli Wang, Zhixing Tan, Jinming Hu, Yidong
Chen, and xiaodong shi. 2017. Xmu neural machine translation systems for wat 2017. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages 95–98, Taipei, Taiwan.
Asian Federation of Natural Language Processing.

Masato Neishi, Jin Sakuma, Satoshi Tohda, Shonosuke Ishiwatari, Naoki Yoshinaga, and Masashi
Toyoda. 2017. A bag of useful tricks for practical neural machine translation: Embedding
layer initialization and large batch size. In Proceedings of the 4th Workshop on Asian Translation (WAT2017), pages 99–109, Taipei, Taiwan.
Asian Federation of Natural Language Processing.
Graham Neubig, Yosuke Nakata, and Shinsuke
Mori. 2011. Pointwise prediction for robust,
adaptable japanese morphological analysis. In
Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 529–533, Stroudsburg,
PA, USA. Association for Computational Linguistics.
Yusuke Oda, Katsuhito Sudoh, Satoshi Nakamura,
Masao Utiyama, and Eiichiro Sumita. 2017. A
simple and strong baseline: Naist-nict neuralmachine translation system for wat2017 englishjapanese translation task. In Proceedings of the
4th Workshop on Asian Translation (WAT2017),
pages 135–139, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In
ACL, pages 311–318.
Slav Petrov, Leon Barrett, Romain Thibaux, and
Dan Klein. 2006. Learning accurate, compact,
and interpretable tree annotation. In Proceedings of the 21st International Conference
on Computational Linguistics and 44th Annual
Meeting of the Association for Computational
Linguistics, pages 433–440, Sydney, Australia.
Association for Computational Linguistics.
Sandhya Singh, Ritesh Panjwani, Anoop
Kunchukuttan, and Pushpak Bhattacharyya.

54