Overview of the 1st Workshop on Asian Translation
Toshiaki Nakazawa
Japan Science andTechnology Agency

Isao Goto
Hideya Mino
National Institute ofNHK
Information andCommunications Technology
nakazawa@pa.jst.jp hideya.mino@nict.go.jp goto.i-es@nhk.or.jp
Sadao Kurohashi
Kyoto University

Eiichiro Sumita
National Institute ofInformation andCommunications Technology
eiichiro.sumita@nict.go.jp

kuro@i.kyoto-u.ac.jp

LangPair
ASPEC-JE
ASPEC-JC

Abstract
This paper presents the results of the
1st workshop on Asian translation(WMT2014) shared tasks, which includedJ↔E translation subtasks and J↔C translation subtasks. As the first year of WAT,
12 institutions participated to the sharedtasks. More than 300 translation results
have been submitted to the automatic evaluation server, and selected submissions
were manually evaluated.

1

Train
3,008,500
672,315

Dev
1,790
2,090

DevTest
1,784
2,148

Test
1,812
2,107

Table 1: Statistics of ASPEC.

scientific papers as a domain and JapaneseChinese as a language pair. In the future, wewill add more Asian languages, such as Korean, Vietnamese, Indonesian, Thai, Myanmar and so on.
• Evaluation method
Evaluation will be done by both automaticand human evaluation. For human evaluation, WAT will use crowdsourcing, which is
low cost and allows multiple evaluations.

Introduction

The Workshop on Asian Translation (WAT) is a
new open evaluation campaign focusing on Asian
languages. We would like to invite a broad rangeof participants and conduct various forms of machine translation experiments and evaluation. Collecting and sharing our knowledge will allow us tounderstand the essence of machine translation andthe problems to be solved. We are working toward
the practical use of machine translation among all
Asian countries.
For the 1st WAT, we chose scientific papers as
the targeted domain, and selected the languages
Japanese, Chinese and English.
What makes WAT unique:

2 Dataset
WAT uses Asian Scientific Paper Excerpt Corpus(ASPEC)1 as the dataset. ASPEC is constructedby the Japan Science and Technology Agency(JST) in collaboration with the National Instituteof Information and Communications Technology(NICT). It consists of a Japanese-English scientificpaper abstract corpus (ASPEC-JE), which is used
for J↔E subtasks, and a Japanese-Chinese scientific paper excerpt corpus (ASPEC-JC), which isused for J↔C subtasks. The statistics of each corpus are described in Table1.

• Open innovation platform
The test data is fixed and open, so you can repeat evaluations on the same data and confirm
changes in translation accuracy over time.
WAT has no deadline for the automatic translation quality evaluation (continuous evaluation), so you can submit translation results at
any time.

2.1 ASPEC-JE
The training data of ASPEC-JE was constructedby the NICT from approximately 2 million
Japanese-English scientific paper abstracts ownedby the JST. Because the paper abstracts are kind

• Domain and language pairs
WAT is the world’s first workshop that uses1

http://lotus.kuee.kyoto-u.ac.jp/ASPEC/1
Proceedings of the 1st Workshop on Asian Translation (WAT2014), pages 1‒19,
Tokyo, Japan, 4th October 2014.

of comparable corpora, the sentence correspondences are automatically found using the methodof (Utiyama and Isahara, 2007). Each sentencepair is accompanied with the similarity score andthe field symbol. The similarity scores are calculated by the method of (Utiyama and Isahara,2007). The field symbols are single letters AZ and show the scientific field of each document2 . The correspondance between the symbols
and field names, along with the frequency and occurance ratios for the training data, are given in the
README of ASPEC-JE.
The development, development-test and test
data were extracted from parallel sentences fromJapanese-English paper abstracts owned by JSTthat are not contained in the training data. Eachdata set contains 400 documents. Furthermore, thedata has been selected to contain the same relative
field coverage across each data set. The documentalignment was conducted automatically and only
documents with a 1-to-1 alignment are included.
It is therefore possible to restore the original documents. The format is the same as for the training
data except that there is no similarity score.

baseline system at WAT 2014.
In addition to the results for the baseline phrasebased SMT system, we produced results for thebaseline systems that consisted of a hierarchicalphrase-based SMT system, a string-to-tree syntaxbased SMT system, a tree-to-string syntax-based
SMT system, five commercial rule-based machinetranslation (RBMT) systems, and two online translation systems. The SMT baseline systems consisted of publicly available software, and the procedures for building the systems and translating
using the systems were published on the WAT
2014 web page3 . We used Moses (Koehn et al.,2007; Hoang et al., 2009) as the implementationof the baseline SMT systems. The Berkeley parser(Petrov et al., 2006) was used to obtain syntactic
annotations. The baseline systems are shown in
Table 2.
The commercial RBMT systems and the onlinetranslation systems were operated by the organizers. We note that these RBMT companies and online translation companies did not submit themselves. Since our objective is not to compare commercial RBMT systems or online translation systems from companies that did not themselves participate, the system description of these systems
are anonymized in this paper.
We describe the detail of the baseline SMT systems.

2.2 ASPEC-JCASPEC-JC is a parallel corpus consisting ofJapanese scientific papers from the literature
database and electronic journal site J-STAGE of
JST that have been translated to Chinese after receiving permission from the necessary academic
associations. The parts selected were abstracts andparagraph units from the body text, as these contain the highest overall vocabulary coverage.
The development, development-test and test
data are extracted at random from documents containing single paragraphs across the entire corpus. Each set contains 400 paragraphs (documents). Therefore there are no documents sharing
the same data across the training, development,
development-test and test sets.

3

3.1 Data for Training
We used the following data for the training of the
SMT baseline systems.
• Training data for the language model: All of
the target language sentences in the parallel
corpus.
• Training data for the translation model: Sentences that were 40 words or less in length.
(For Japanese–English training data, we onlyused train-1.txt, which consisted of 1 million
parallel sentence pairs with high similarity
scores.)• Development data for tuning: All of the development data.

Baseline Systems

Human evaluations were conducted as pairwise
comparisons between the translation results for a
specific baseline system and translation results foreach participant’s system. That is, the specificbaseline system was the standard of human evaluation. A phrase-based statistical machine translation (SMT) system was adopted as the specific23.2 Common Settings for Baseline SMT
We used the following tools for tokenization.
• Juman version 7.04 for Japanese segmentation.
34

http://opac.jst.go.jp/bunrui/index.html
2

http://lotus.kuee.kyoto-u.ac.jp/WAT/
http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?JUMAN

System ID
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT XRBMT XRBMT XRBMT X
Online X
Online X

SystemMoses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
ATLAS V14 (Commercial system)
PAT-Transer 2009 (Commercial system)
J-Beijing 7 (Commercial system)
Hohrai 2011 (Commercial system)
Google translate (July, 2014)
Bing translator (July, 2014)

Type
SMT
SMT
SMT
SMT
RBMT
RBMT
RBMT
RBMT
RBMT
(SMT)
(SMT)

JE
✓
✓
✓

EJ
✓
✓

✓
✓
✓

✓
✓
✓
✓

✓
✓

✓
✓

JC
✓
✓
✓

CJ
✓
✓

✓
✓
✓
✓

✓
✓
✓
✓

✓

Table 2: Baseline Systems• Phrase extraction options: MaxSpan = 1000,
MinHoleSource = 1, and NonTermConsecSource.

• Stanford Word Segmenter version 2014-01045 (Chinese Penn Treebank (CTB) model)
for Chinese segmentation.
• The Moses toolkit for English tokenization.

The default values were used for the other systemparameters.

To obtain word alignments, GIZA++ and growdiag-final-and heuristics were used. We used 5gram language models with modified Kneser-Ney
smoothing, which were built using a tool in theMoses toolkit (Heafield et al., 2013).

3.6 Tree-to-String Syntax-based SMT
We used Berkeley parser to obtain source language
syntax. We used the following Moses’ configuration for the baseline tree-to-string syntax-based
SMT system.

3.3 Phrase-based SMT
We used the following Moses’ configuration forthe phrase-based SMT system.

• max-chart-span = 1000
• Phrase score option: GoodTuring• Phrase extraction options: MaxSpan = 1000,
MinHoleSource = 1, MinWords = 0,
NonTermConsecSource, and AllowOnlyUnalignedWords.

• distortion-limit = 20
• msd-bidirectional-fe lexicalized reordering• Phrase score option: GoodTuring
The default values were used for the other systemparameters.

The default values were used for the other systemparameters.

3.4 Hierarchical Phrase-based SMT

4 Automatic Evaluation

We used the following Moses’ configuration forthe hierarchical phrase-based SMT system.

4.1 Procedure of Calculating Automatic
Evaluation Score• max-chart-span = 1000
• Phrase score option: GoodTuring

We calculated automatic evaluation scores of thetranslation results applying two popular metrics:
BLEU (Papineni et al., 2002) and RIBES (Isozaki
et al., 2010). BLEU scores were calculated withmulti-bleu.perl distributed with the Moses toolkit(Koehn et al., 2007). RIBES scores were calculated with RIBES.py version 1.02.4 6 . All scoresof each task were calculated using one reference.
Before the calculation of the automatic evaluation
scores, the translation results have been tokenizedwith word segmentation tools on each language.

The default values were used for the other systemparameters.
3.5 String-to-Tree Syntax-based SMT
We used Berkeley parser to obtain target language
syntax. We used the following Moses’ configuration for the string-to-tree syntax-based SMT system.
• max-chart-span = 1000
• Phrase score option: GoodTuring56

http://nlp.stanford.edu/software/segmenter.shtml
3

http://www.kecl.ntt.co.jp/icl/lirg/ribes/index.html

4Figure 1: The submission web page for participants

For Japanese segmentation we use three different tools, which are Juman version 7.0 (Kurohashi
et al., 1994), KyTea 0.4.6 (Neubig et al., 2011)
with Full SVM model 7 and MeCab 0.996 (Kudo,
2005) with IPA dictionary 2.7.0 8 . For Chinesesegmentation we use two different tools, which are
KyTea 0.4.6 with Full SVM Model in MSR modeland Stanford Word Segmenter version 2014-06-16
with Chinese Penn Treebank (CTB) and Peking
University (PKU) model 9 (Tseng, 2005). For English segmentation we use tokenizer.perl 10 in theMoses toolkit.
The detailed procedures for the automatic evaluation are shown at WAT2014 evaluation web
page 11 .

after WAT2014. Everybody can use the system byregistering on the registration web page 12 .

5 Human Evaluation
5.1 Using Crowdsourcing
As all the MT researchers know, the human evaluation costs a lot of time and money. One of thesolutions to reduce them is using crowdsourcing.
Other machine translation evaluation campaigns
such as IWSLT (2011, 2012) and WMT (2012,2013) also used crowdsourcing for the humanevaluation. Recently, there are so many crowdsourcing services in the world: Amazon Mechanical Turk13 , CrowdFlower14 , Yahoo Crowdsourcing15 , Lancers16 and so on. Among these services, we used Lancers for the human evaluationof WAT2014.
There are two reasons of choosing Lancers. One
is that we can set the category of the crowdsourcing task (’Translation’ for this case). We can reachthe appropriate workers by setting the appropriate
categories. The other reason is that we can ask thetask to the identity-verified workers. This function guarantee the quality of the workers. These
two advantages can keep the evaluation quality athigher level.

4.2 Automatic Evaluation System
The participants submit the translation results
via an automatic evaluation system deployed at
WAT2014 web page, which give them automatic
evaluation scores of the results they upload. Figure 1 shows the submitting interface for participants. The system requires the participants to provide the following information when they upload
the translation results:• Subtask (J ↔ E, J ↔ C)
• Method (SMT, RBMT, SMT and RBMT,EBMT, Other)

5.2 Human Evaluation Method
For the human evaluation, we randomly chose
documents from the Test set of ASPEC data, in
total 400 sentence pairs for JE and JC. We excluded the documents which contains a sentencewith longer than 100 Japanese characters. Eachsubmission is compared with the baseline translation (Phrase-based SMT, described in Section 3)
and given HUMAN score.

• Existence of the use of other resources in addition to ASPEC• Permission of publishing the automatic evaluation scores on WAT2014 web pageThe server of the system keeps all submitted information including translation results or scoresand participants can confirm the only information
they uploaded. The information of translation results which the participant permits to publish isdisclosed on the web page. In addition to submitting the translation results for automatic evaluation, participants submit the results for humanevaluation with the same web interface. This automatic evaluation system will be available even

5.2.1 Pairwise Evaluation of Sentences7

http://www.phontron.com/kytea/model.html
http://code.google.com/p/mecab/downloads/detail?
name=mecab-ipadic-2.7.0-20070801.tar.gz
9
http://nlp.stanford.edu/software/segmenter.shtml
10
https://github.com/moses-smt/mosesdecoder/tree/
RELEASE-2.1.1/scripts/tokenizer/tokenizer.perl
11
http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/index.html

We conducted pairwise evaluation of each test sentence of the 400 sentences. The input sentence
and two translations (the baseline and a submission) are shown to the workers, and the workers
are asked to judge which translation is better thanthe other, or they are of the same quality. The order of the two translations are at random. Figure 2
shows the illustration of the evaluation.

8

5

12

http://lotus.kuee.kyoto-u.ac.jp/WAT/registration/index.html
https://www.mturk.com
14
http://www.crowdflower.com
15
http://crowdsourcing.yahoo.co.jp (Japanese service)16
http://www.lancers.jp (Japanese service)
13

Figure 2: The illustration of the crowdsourcing evaluation. The workers are asked to judge which translation is better, or the same.
Worker 1
Worker 2
Worker 3
Decision

A
A
A
A

A
A
Tie
A

A
A
B
A

A
Tie
Tie
A

A
Tie
B
Tie

A
B
B
B

Tie
Tie
Tie
Tie

Tie
Tie
B
B

Tie
B
B
B

B
B
B
B

Table 3: The combinations of human judgements and the final decision of each sentence pairs fromsystem A and B.
1. randomly select 300 sentences from the 400
human evaluation sentences, and calculate
the HUMAN score on the selected sentences

5.2.2 Voting
The crowdsourcing workers are not specialists,
thus the quality of the judgements are not necessarily precise. To guarantee the quality of theevaluation, each sentence is evaluated by 3 different workers and the final decision is made by thevoting of the judgements. Table 3 shows all the
combinations of the worker judgements and the final decision.

2. iterate the previous step 1000 times and get1000 HUMAN scores
3. sort the 1000 scores and estimate the 95%
confidence interval by discarding top and bottom 25 scores

5.2.3 HUMAN Score Calculation
5.3 Cost of Evaluation

Suppose W to be the number of wins compared tothe baseline, L to be the number of losses and T to
be the number of ties, the HUMAN score, which is
the official human evaluation score of WAT2014,
can be calculated by the following formula:
HU M AN = 100 ×One big benefit of using crowdsourcing is that we
can reduce the cost of evaluations. In WAT2014,
one judgement costs 5 JPY. The evaluation of a
submission requires 3 (judgements) × 400 (sentence pairs) = 1,200 judgements and it costs 5 ×
1,200 = 6,000 JPY. The time for the evaluationdiffers depending on the translation direction. On
the average, one evaluation finished in a couple of
days.

W −L
W +L+T

From the definition, the HUMAN score ranges between -100 and 100.
5.2.4 Confidence Interval Estimation

6 Participants List

As there are several ways to estimate the confidence interval, we chose the bootstrap resampling(Koehn, 2004) to estimated 95% confidence interval. The procedure is as follows:

Table 4 shows the list of participants to WAT2014.
There are not only the Japanese organizations, butsome organizations came from outside Japan. 12
teams submitted one or more translation results to6the automatic evaluation server, and 11 teams submitted one or more translation results to the humanevaluation.

7

ings are also done by the bootstrap resampling asfollows:
1. randomly select 300 sentences from the 400
human evaluation sentences, and calculate
the HUMAN scores on the selected sentences
for both systems

Evaluation Results

In this section, the evaluation results of WAT2014
are reported from several perspectives. Parts of theresults of both automatic and human evaluations
are also accessible at WAT2014 website17 .

2. iterate the previous step 1000 times and countthe number of wins (W ), losses (L) and ties
(T )7.1 Official Automatic Evaluation Results

3. calculate p = WL+L

Figure 3 shows the official automatic evaluation
results of the representative submissions and baseline systems. The automatic evaluation results ofall the submissions are shown in Section Appendix
A.

Inter-annotator Agreement
To assess the reliability of agreement between thecrowdsourcing workers, we calculated the Fleiss’
kappa (Fleiss and others, 1971) values. The results are shown in Table 9. We can see that the
Kappa values are larger for X → J translations
than J → X translations. This may be because
we used Japanese crowdsourcing service for theevaluation and the majority of the crowdsourcing
workers are Japanese. The MT evaluation of theirmother tongue is much easier than the others ingeneral.

7.2 Official Human Evaluation Results
HUMAN scores
Figure 4 shows the official human evaluation results. The error bars in the figures show the 95%
confidence interval (see Section 5.2.4). Note thatoverlapping the error bars between two submissions does not necessarily mean that there is no
significant difference. If an error bar crosses thex-axis (HUMAN score = 0), it means that there isno significant difference between the submission
and the baseline (SMT Phrase).
From the results, the followings can be observed:

Case Study: Direct Comparison and Relative
Comparison
Looking at evaluation results of WEBLIO-EJ1
1 and 2 submissions (see Table 12), the automatic and human evaluations are inconsistent:
the WEBLIO-EJ1 2 is consistently better than
• The best SMT system achieved better quality
WEBLIO-EJ1 1 in the automatic evaluation, howthan RBMT system.
ever it is much worse in the human evaluation. According to the descriptions of the two submissions,• The translation quality of the widely usedthe difference of the two is whether it uses the forsystems was Phrase-based SMT < Hierarchiest input or not. It is natural that using the forcal PBSMT < Syntax-based SMT (S2T and
est input improves the translation quality, thus we
T2S).
conducted the human evaluation of WEBLIO-EJ1• Forest-to-String Syntax-based SMT system
2 compared to WEBLIO-EJ1 1, which means we(Neubig, 2014) achieved the best quality forused WEBLIO-EJ1 1 as the baseline for the huall the translation directions.
man evaluation.
The HUMAN score was 2.50 ± 4.17 which
Statistical Significance Testing between
means there is no significant difference between
Submissions
the two, and this result is far from the results ofTables 5, 6, 7 and 8 show the results of statisticalthe official results. Actually, taking the confidence
significance testings of JE, EJ, JC and CJ transintervals into consideration, this conclusion can belations respectively where all the pairs of submisderived under some probability.
sions are tested. ≫, ≫ and > mean that the sysThe Fleiss’ kappa value was 0.528 and it is
tem in the row is better than the system in the colmuch higher than the other E→J human evaluaumn by p < 0.01, 0.05, 0.1 respectively. The testtions. This may be because the outputs of the two
17
http://lotus.kuee.kyoto-u.ac.jp/WAT/evaluation/index.html systems are quite similar and it is very easy for the7

JE BLEUJE RIBESEJ BLEUEJ RIBESJC BLEUJC RIBESCJ BLEUCJ RIBES

non-removal
0.46489
0.78255
0.415240.75105
0.49240
0.38695
0.787130.70081

removal0.95098
0.836910.844180.85730
0.079370.10198
0.82592
0.83209

Also, we are very happy to include other languages if there are available resources.

Appendix A

Tables 11, 12, 13 and 14 summarize all the submissions listed in the automatic evaluation serverat the point of WAT2014 workshop (4th, October,2014). The OTHER RESOURCES column shows
the use of other resources such as parallel corpora,monolingual corpora and parallel dictionaries in
addition to ASPEC.

Table 10: The changes of correlations (R2 ) before
and after removing RBMT and online systems.

workers to judge which translation is better. If two
translations have both better and worse parts thanthe other, the workers would evaluate differently
from person to person.
7.3 Correlation between Automatic andHuman Evaluations
Figure 5 shows the correlations between automatic
evaluation measures (BLEU/RIBES) and the HUMAN score. It is well known that the automaticand human evaluations do not have good correlations for RBMT and online systems. Removing
these systems from the graph changes the correlation values (R2 ) like in Table 10. The correlation
becomes much better after removing the RBMTand online systems for all the translation directions
other than J→C.

8 Submitted Data
The number of published automatic evaluaition
results of 12 teams exceeded 100 by the day of
WAT2014 workshop and 37 translation results forhuman evaluation was submitted by 11 teams. We
will organize the all submitted data for humanevaluation and make it public.

9

Submissions

Conclusion and Future Perspective

This paper summarized the WAT2014 machinetranslation evaluation campaing. We had 12 participants worldwide, and collected a large numberof submissions which are useful to improve thecurrent machine translation systems by analyzing
the submissions and finding the issues.
For the next WAT workshop, we are planning toconduct context-aware MT evaluations. The test
data of WAT is prepared using the paragraph as aunit, while almost all other evaluation campaigns
use the sentence as a unit. Therefore, it is suitable
to investigate the importance of the context for thetranslation.
8

Team ID
NAIST (Neubig, 2014)
EIWA (Ehara, 2014)
Kyoto-U (Richardson et al., 2014)
WEBLIO-EJ1 (Zhu, 2014)
TMU (Ohwada et al., 2014)
BJTUNLP (Cai et al., 2014)
NII (Hoshino et al., 2014)
SAS MT (Wang et al., 2014)
Sense (Tan and Bond, 2014)
NICT (Ding et al., 2014)
TOSHIBA (Sonoh et al., 2014)
WASUIPS (Yang and Lepage, 2014)

Organization
Nara Institute of Science and Technology
Yamanashi Eiwa College
Kyoto University
Weblio, Inc.
Tokyo Metropolitan University
Beijing Jiaotong University
National Institute of Informatics
SAS Research and Development Co., Ltd
Saarland University & Nanyang Technological University
National Institute of Information and Communication Technology
Toshiba Corporation
Waseda University

JE
✓
✓
✓
✓
✓
✓

EJ
✓

JC
✓

✓
✓

✓

CJ
✓
✓
✓

✓
✓
✓

✓
✓
✓
✓*

✓

✓
✓
✓*

SMT Hiero
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

TMU 2

≫
≫
≫
≫
≫
≫
≫
≫
≫

TMU 1

≫
≫
>
>
-

Online D

TOSHIBA 2

≫
≫
>
-

NII 2

≫
≫
-

NII 1

≫
≫
-

Sense

≫
≫
-

Kyoto-U 2

≫
≫
-

EIWA

≫
≫

RBMT D

SMT S2T-TOSHIBA 1

Kyoto-U 1

NAIST 1NAIST 2
Kyoto-U 1
SMT S2TTOSHIBA 1
RBMT D
EIWA
Kyoto-U 2
TOSHIBA 2
Online D
SMT Hiero
Sense
NII 1
NII 2
TMU 1

NAIST 2

Table 4: The list of participants which submitted translation results to WAT2014 and their participations
to each subtasks. (*Only submitted to automatic evaluations.)

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
-

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
-

≫
≫
≫
>

≫
≫
≫
≫
-

≫
≫
≫
≫
≫
>
-

≫
≫
≫
≫
≫
≫
≫
≫
>

Table 6: Statistical significance testing of EJ results.

9

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

RBMT B

≫
≫
≫
≫
≫
-

Sense

≫
≫
≫
≫
>
-

SAS MTWEBLIO-EJ1 2

Kyoto-U 1

Online A≫≫-

SMT Hiero≫≫

Kyoto-U 2≫

SMT T2S

NAIST 1NAIST 2WEBLIO-EJ1 1Online A
Kyoto-U 1
WEBLIO-EJ1 2
SMT T2S
Kyoto-U 2
SMT Hiero
SAS MT
Sense

WEBLIO-EJ1 1

NAIST 2

Table 5: Statistical significance testing of JE results.

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
-

Figure 3: The official automatic evaluation results.
10

$"#""%%+'#''%%!"#$%&'(#)*+,-!

!"#$"%%&'#$"%%!"#""%%&"#""%%!"#$%&'(#)*+,-!

"'#''%% !"#$!%%!&#!'%%!'#''%%()#$!%%($#!'%%)*#''%%('#''%%)"#''%%)(#$!%%))#+!%%)&#!'%%$+#!'%%)'#''%%($#$"%%($#""%%(&#($%%(&#""%%((#$"%%()#($%%("#($%%("#""%%)&#'$%%'#'$%%)"#""%%$'#''%%)#($%% "#""%%
(%

5%
)%

5%

/0

&'#''%%)#+!%%

/0

%

%
-%(
+-

%
B@

-%)
+-

=B
@%

AE

<=
>

?>
,@
34
%
5/6 &%
,3
7&
%$
%
0A
1%
1$
0%
<=
>?
>,
@%
$%
0A
1%
B:
;C
>%
0.
0D
A
1%
0;
8E
;
F4 %
A
1
0A
%4
%
1%
GH
CI
E;
%

9:8
68

2

2

34

5/6

,3
7%&

%&

;%
.%

%

%

%

%$

%&

01

01

,&'#''%%-.
/*("#""%%*&"#""%%)'#''%%'#+!%% '#''%%'#''%%*)!#($%%*)'#""%%*)'#($%%

-.
/

.@

*$#'$%%

.0

/%
CD

%

3%

@%
:

@A
7?

>?=

/%

6=

.0

(%

,%
(%
-8

.7

,%

5%
3*

34
12

/6

:%

;-

/%

<

,%
)%
-8

0

.7
/6

98

/%

%)
%

.(

*5

12

34
3

/%

-.

.0

*)"#""%%

+,

+,
-

./

%)

/%
(%

%

"#""%%*"#""%%!"#$%&'(#)*+,-!

!"#$%&'(#)*+,-!

!"#$!%%!"#""%%*'#''%% !"#"$%%&'#""%%+"#""%%!&#''%%&"#""%%!'#''%%!'#''%%(#$'%%((#!"%%)*#""%% )!#""%%("#""%%)#"$%%!#*$%% '#"$%%$#!"%% *#""%%+#$!%%)"#""%%'#''%%'#''%%

.%
1%

=%
.%

FG
3

=%

E<C
DC

%

,)#""%%

0=
CB

B=

1%
03

,("#""%%+!&#$'%%

?@
>A

=>

8%

(%

;<

03
1%

67
8

98

,:
%

)%

.%
67
8

4/

98

5

,:
%

0%
1(

1%

03
1%

0.

02

01

3

%(

%

%
-.
/

%)
01

,)"#""%%

-.
/

3%

1%

03
45

1%
0*

02 /61%
1%
78
39
:
-. %
/0
1; 1%*
07 %/02 <.%
!%
1%
=>
9?
5
3%
@A
:B
:+
C%
!%
<D
1C
-E
1;
=
07 %
/<
.%
*%
@A
:B
:+
C%
;4 *%
F84
3%
G
H< %
2
1%
<%

+!'#''%%

02

-.
/

01

%!

%

"#""%%"#""%%+'#"$%%+)#"$%%+$#*$%%+,#"$%%,()#$!%%,&"#""%%+*'#''%%+*'#''%%,+"#""%%,&$#$!%%,!"#""%%+)'#''%%

≫
≫
≫
≫
≫
≫
≫
>
-

≫
≫
≫
≫
≫
≫
≫
≫
>
-

Table 7: Statistical significance testing of JC results.

11

RBMT B

≫
≫
≫
≫
≫
>
>
-

Kyoto-U 2

≫
≫
≫
≫
-

TOSHIBA 2TOSHIBA 1
≫
≫
≫
≫
-

Online D

≫
≫
≫
>
-

BJTUNLP≫≫≫-NAIST 2≫≫-

SMT Hiero≫-

Kyoto-U 1-

NICT

NAIST 1SMT S2T
Sense
NICT
SMT Hiero
NAIST 2TOSHIBA 1
Kyoto-U 1
BJTUNLPTOSHIBA 2
Kyoto-U 2
Online D

Sense

SMT S2T

Figure 4: The official human evaluation results.

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
≫
>

≫
≫
≫
≫
≫
-

≫
≫
≫
≫
≫
≫
≫
≫

RBMT A

≫
≫
≫
≫
≫
-

Sense

≫
≫
≫
≫
≫

Online A≫≫≫-

SMT Hiero≫≫≫

Kyoto-U 2

SMT T2S≫≫

Kyoto-U 1

SAS MT≫

EIWA

NAIST 2
NAIST 1NAIST 2
SAS MT
SMT T2S
EIWA
Kyoto-U 1
Kyoto-U 2
SMT Hiero
Sense
Online A

≫
≫
≫
≫
≫
≫
≫
≫
≫

≫
≫
≫
≫
≫
≫
≫
≫
≫
≫

Table 8: Statistical significance testing of CJ results.

JE
System ID
NAIST 1NAIST 2
SMT S2T
Kyoto-U 1
TOSHIBA 1
RBMT D
EIWA
Kyoto-U 2
TOSHIBA 2
Online D
SMT Hiero
Sense
NII 1
NII 2
TMU 1
TMU 2
ave.

Kappa
0.162
0.047
0.099
0.070
0.098
0.075
0.083
0.139
0.078
0.055
0.119
0.245
0.119
0.086
0.091
0.136
0.106

EJ
System ID
NAIST 1NAIST 2WEBLIO-EJ1 1Online A
Kyoto-U 1
WEBLIO-EJ1 2
SMT T2S
Kyoto-U 2
SMT Hiero
SAS MT
Sense
RBMT Bave.

JC
System ID
NAIST 1SMT S2T
Sense
NICT
SMT Hiero
NAIST 2TOSHIBA 1
Kyoto-U 1
BJTUNLPTOSHIBA 2
Kyoto-U 2
Online D
RBMT B
ave.

Kappa
0.280
0.250
0.238
0.219
0.216
0.240
0.240
0.229
0.277
0.248
0.395
0.217
0.254

Kappa
0.077
0.069
0.087
0.066
0.202
0.093
0.089
0.091
0.198
0.066
0.163
0.035
0.083
0.101

CJ
System ID
NAIST 1NAIST 2
SAS MT
SMT T2S
EIWA
Kyoto-U 1
Kyoto-U 2
SMT Hiero
Sense
Online A
RBMT Aave.

Table 9: The Fleiss’ kappa values of human evaluation results.

12

Kappa
0.168
0.203
0.167
0.236
0.175
0.199
0.180
0.274
0.228
0.239
0.130
0.200

)(*+!"#$%,&'("-*)(*+!"#$%,-.&(/0*!"#$%&'&!"#$%&(&

A?&

D?&

E?&

3=&>&?@ABACD&
07897:&4&'?&$)%&19:;,&$:7<:&

?&
'F&

'B&

'G&

'C&
!##&'&'D&$)%&$(%&*+,-,./&'&%0$1#2"&'&32)%&4&
5#6"&*+,-,./&(&%0$1#2"&(&(?&!"#$%!

!"#$%!

$)%&$(%&*+,-,./&'&%0$1#2"&'&
5#6"& *+,-,./&(&

32)%&4&%0$1#2"&(&(?&07897:&4&

('&

((&

(E&

(A&

3=&>&?@AB(CC&'?&$)%&19:;,&$:7<:&

?&
(?&

(F&

F'&

FD&

FC&

FA&

FG&

A'&

AD&

!##&'&.'?&.'?&!##&(&!##&(&%)/&'&%)/&(&%)/&'&%)/&(&

.(?&

.(?&

&'("!

FA&

&'("!

GA&

()*+!"#$%,&'("-!

)+,-!"#$%.&'()*/!

!"#$%&'&!"#$%&(&DA&!"#$%&(&!"#$%!

$"$<9%&>?&@&ABC'D(C&)*+,#-.*/'&'&-01203&"&FA&45676.8&'&)*+,#-.*/'&(&$9%&%($&
45676.8&(&$9%&:23;6&

EA&!"#$%&'&DA&)*+,#-.*/'&'&-01203&"&
CA&!"#$%!

!"#$%&'&!"#$%&(&

E?&

45676.8&'&)*+,#-.*/'&(&$9%&%($&
45676.8&(&$9%&:23;6&

EA&$"$<9%&>?&@&ABCD'AD&

(A&

(A&

'A&

'A&
$30=3&$30=3&>+9%&+&A&'A&>+9%&+&

A&

'D&

(A&

)A&

(D&
&'("!

EA&

ED&

CA&

GG&

CA&

)?&

)*+,!"#$%-&'(".!

C(&

CF&
&'()*!

CG&'E&$(%&$)%&$*+,*&

'A&

'E&

)A&

7E&

)E&
39%8!:;&

E&

FE&

4516178&)&2+</+*&=&>3(%&3&

7)E&

FE&

FC&

7E&

FF&

FB&

FD&

4516178&'&B?&
39%8!:;&)*+,!"#$%-&'(".!

!"#$%&'&BA&

BG&<3(%&3&&'()*!

+,-.!"#$%/&'()*0!

E?&!"#$%&'&C?&!"#$%&(&!"#$%&(&

D?&$"$)*%&(?&-./0/12&'&-./0/12&(&$*%&3456/&;=&>&?@ABA'C&
?&
E&

'?&

'E&

(?&

(E&

!"#$%!

'?&

C?&$5785&
CE&$"$)*%&(?&+#,"& $*%&%($&+#,"&$*%&%($&'?&-./0/12&'&-./0/12&(&$*%&3456/&

?&
D?&

DE&

EF&

GC&

GF&

AC&

1'?&

1'?&

;=&>&?@A??B'&1(?&

97:475&"&

97:475&"&

1D?&

1C?&
1D?&

B)&

4516178&)&

7)E&

&'("!

C?&

1(?&

B'&

%2$.#3"&)&

7)?&

D?&

!"#$%!

FG&

7'E&

7)A&

E?&

!"#$%&)&%2$.#3"&'&

?&

7'?&

>?&@&ABCD)C&

7'E&$(%&./*01&<=&>&?@ABCDE&%2$.#3"&)&

7'A&!"#$%&'&!#-%&$(%&./*01&!"#$%&)&%2$.#3"&'&4516178&'&
FA&!"#$%!

!"#$%!

E&

H(&$*+,*&

'?&

A&

HA&

$(%&$)%&

!#-%&
E&

CH&

+,-.!"#$%/&'()*0!

!"#$%&'&

'E&
'A&

GH&

;<*%&"&

1C?&

&'("!

;<*%&"&&'()*!

Figure 5: The correlations between BLEU/RIBES and HUMAN scores.
13$5785&AF&BC&

SYSTEM ID ID
METHOD
OTHER RESOURCES BLEU
SMT Hiero 2
SMT
NO
18.72
SMT Phrase 6
SMT
NO
18.45
SMT S2T
9
SMT
NO
20.36
Online D
35
Other
YES
15.08
76
Other
YES
14.82
RBMT ERBMT F
79
Other
YES
13.86
Online C
87
Other
YES
10.64
RBMT D
96
Other
YES
15.29
NAIST 1
46
SMT
YES
23.82
NAIST 2
119
SMT
NO
23.29
125
SMT
NO
23.47
NAIST 3EIWA
116 SMT and RBMT
YES
19.86
Kyoto-U 3 136
EBMT
NO
20.02
Kyoto-U 2 256
EBMT
NO
20.60
Kyoto-U 1 262
EBMT
NO
21.07
TMU 2
300
SMT
NO
15.55
TMU 1
301
SMT
NO
15.95
TMU 3
307
SMT
NO
15.40
NII 1
271
SMT
NO
17.47
NII 2
272
SMT
NO
17.01
Sense 1
164
SMT
NO
18.82
Sense 2
185
SMT
NO
18.57
Sense 3
191
SMT
NO
18.00
Sense 4
205
SMT
NO
18.87
Sense 5
206
SMT
NO
18.91
TOSHIBA 2 240
RBMT
YES
15.69
TOSHIBA 1 241 SMT and RBMT
YES
20.61

14

RIBES
0.651066
0.645137
0.678253
0.643588
0.663851
0.661387
0.624827
0.683378
0.722599
0.723541
0.723670
0.706686
0.689829
0.701154
0.698953
0.644698
0.648879
0.6131190.630825
0.610833
0.6462040.640393
0.641377
0.646133
0.637375
0.687122
0.707936

Table 11: JE submissions

HUMAN SYSTEM DESCRIPTION+7.75 Hierarchical Phrase-based SMT—– Phrase-based SMT+25.50 String-to-Tree SMT+13.75 Online D—– RBMT E—– RBMT F—– Online C+23.00 RBMT D+40.50 Travatar-based Forest-to-String SMT System with Extra Dictionaries+37.50 Travatar-based Forest-to-String SMT System—– Travatar-based Forest-to-String SMT System (Tuned BLEU+RIBES)+22.50 Combination of RBMT and SPE(statistical post editing)—– Our baseline system using 3M parallel sentences+21.25 Our new baseline system after several modifications+25.00 Our new baseline system after several modifications + 20-best parses, KN7, RNNLM reranking-17.25 Our baseline system with preordering method-17.20 Our baseline system with another preordering method—– Our baseline system-5.75 Our Baseline-14.25 Our Baseline with Preordering+1.25 Paraphrase max10—– Baseline SMT—– Context sensitive SMT—– SMT with lexicon—– SMT with lexicon X5+20.25 RBMT system+23.25 RBMT with SPE(Statistical Post Editing) system

EBMT
EBMT
EBMT
EBMT

SMT
SMT
SMT
SMT
SMT
SMT
SMT
SMT

5
12
34
66
68
91
95
367
118
126

134
186
253
267

SMT Phrase
SMT T2S
Online A
RBMT B
RBMT A
Online B
RBMT CSMT Hiero
NAIST 1NAIST 2

Kyoto-U 3
Kyoto-U 4
Kyoto-U 2
Kyoto-U 1

WEBLIO-EJ1 1 132
WEBLIO-EJ1 2 202
SAS MT
264
Sense 2
163
Sense 1
184
Sense 3190
Sense 4265
Sense 5
274

SMT
SMT
Other
Other
Other
Other
Other
SMT
SMT
SMT

ID METHODSYSTEM ID

15

NO
NO
NO
NO
NO
NO
NO
NO

NO
NO
NO
NO

32.53
32.69
30.47
27.88
27.92
26.59
27.00
27.33

28.93
30.25
29.76
31.09

OTHER
RESOURCES juman
NO
27.48
NO
31.05
YES
19.66
YES
13.18
YES
12.86
YES
17.04
YES
12.19
NO
30.19
NO
35.03
NO
34.84

34.87
35.04
33.00
30.27
30.18
28.46
29.15
29.54

31.61
32.78
32.46
33.55

BLEU
kytea
29.80
33.44
21.63
14.85
14.43
18.67
13.32
32.56
37.16
37.15

33.26
33.40
31.47
28.72
28.66
27.15
27.81
28.16

29.59
30.84
30.46
31.73

mecab
28.27
32.10
20.17
13.48
13.16
17.36
12.14
30.94
35.81
35.67

0.786902
0.790066
0.770948
0.699334
0.700583
0.694678
0.689623
0.688801

0.755744
0.765251
0.764049
0.770908

RIBES
kytea
0.691926
0.758031
0.723486
0.680748
0.676464
0.693390
0.672645
0.7469780.801520
0.807010

0.792616
0.795027
0.771605
0.703139
0.703049
0.697257
0.693560
0.691011

0.756545
0.766495
0.766435
0.771545

mecab
0.695390
0.760516
0.725848
0.682683
0.678934
0.698126
0.676018
0.747722
0.806581
0.811081

Table 12: EJ submissions

0.782066
0.785015
0.759415
0.690718
0.690464
0.684467
0.681194
0.679666

0.743969
0.755629
0.752058
0.766435

juman
0.683735
0.748883
0.718019
0.671958
0.670167
0.687797
0.668372
0.734705
0.796079
0.801742—– Phrase-based SMT+34.25 Tree-to-String SMT+42.50 Online A+0.75 RBMT B—– RBMT A—– Online B—– RBMT C+31.50 Hierarchical Phrase-based SMT+56.25 Travatar-based Forest-to-String SMT System+51.50 Travatar-based Forest-to-String SMT System (Tuned
BLEU+RIBES)—– Our baseline system using 3M parallel sentences—– Using n-best parses and RNNLM+33.75 Our new baseline system after several modifications+38.00 Our new baseline system after several modifications + 20-bestparses, KN7, RNNLM reranking+43.25 Weblio Pre-reordering SMT System Baseline+36.00 Weblio Pre-reordering SMT System (with forest inputs)+27.50 Syntactic reordering Hierarchical SMT (using part of data)—– Paraphrase max10+3.75 Baseline SMT—– Context sensitive SMT—– SMT with 20x lexicon—– SMT with lexicon X5

HUMAN SYSTEM DESCRIPTION

3
7
10
37
216
243
244
122
123

18
257
259

224
175
201
260

SMT Hiero
SMT Phrase
SMT S2T
Online D
Online C
RBMT B
RBMT CNAIST 1NAIST 2

Kyoto-U 3
Kyoto-U 1
Kyoto-U 2

BJTUNLP
Sense 2
Sense 1NICTSYSTEM ID ID

SMT

SMT

SMT
SMT

SMT
SMT

SMT

SMT

WASUIPS 3 376

WASUIPS 4 377

WASUIPS 5 381
WASUIPS 6 382

WASUIPS 7 385
WASUIPS 8 386

WASUIPS 9 389

WASUIPS 10 390

YES

NO

NO
YES

NO
YES

YES

NO

YES

SMT

WASUIPS 2 373

NO
NO
NO
NO

NO
NO
NO

NO
NO
NO
YES
YES
NO
NO
NO
NO

25.63

25.08

25.45
25.68

22.01
22.20

25.60

25.44

24.70

19.28
27.42
22.71

24.12
27.92
23.09
27.98

26.69
27.21
27.67

27.71
27.96
28.65
9.37
7.26
17.86
9.62
30.53
29.83

OTHER
RESOURCES kytea

YES
YES
NO

SMT
SMT
SMT
SMT

EBMT
EBMT
EBMT

SMT
SMT
SMT
Other
Other
RBMT
RBMT
SMT
SMT

METHODTOSHIBA 2 236
RBMT
TOSHIBA 1 238 SMT and RBMTWASUIPS 1 371
SMT

16

25.30

24.81

25.10
25.01

21.81
22.02

25.10

25.04

24.25

18.93
26.82
22.49

23.76
28.03
22.94
28.18

26.48
27.02
27.44

BLEU
stanford
(ctb)
27.70
28.01
28.65
8.93
7.01
17.75
9.96
30.46
29.77

25.18

24.64

25.01
25.11

21.61
21.91

25.07

24.98

24.28

18.82
26.79
22.39

23.55
27.67
23.04
27.84

26.30
26.83
27.34

stanford
(pku)
27.35
27.68
28.35
8.84
6.72
17.49
9.59
30.25
29.54

0.795307

0.791430

0.793308
0.795504

0.767414
0.773341

0.795786

0.793945

0.790460

0.765346
0.803302
0.777615

0.796186
0.796589
0.779502
0.808684

0.798084
0.792166
0.789069

RIBES
stanford(ctb)
0.809561
0.790263
0.809457
0.606328
0.613075
0.745885
0.648758
0.819406
0.830839

0.794024

0.790142

0.793029
0.795129

0.766092
0.772107

0.795594

0.794823

0.790898

0.763931
0.803980
0.777327

0.793054
0.797332
0.780262
0.807809

0.798383
0.790743
0.788206

stanford(pku)
0.811394
0.790937
0.808417
0.604149
0.611563
0.743794
0.645385
0.819492
0.830529

Table 13: JC submissions

0.794646

0.790498

0.793819
0.795721

0.767418
0.771952

0.794716

0.794244

0.790030

0.764491
0.804444
0.776323

0.794834
0.793876
0.779495
0.806070

0.796402
0.791270
0.788321

0.809128
0.788961
0.807606
0.606905
0.612808
0.744818
0.642278
0.818040
0.829627

kytea+3.75 Hierarchical Phrase-based SMT—– Phrase-based SMT+14.00 String-to-Tree SMT-14.50 Online D—– Online C-20.00 RBMT B—– RBMT C+17.75 Travatar-based Forest-to-String SMT System+1.25 Travatar-based Forest-to-String SMT System (Tuned
BLEU+RIBES)—– Our baseline system-0.75 Our new baseline system after several modifications-8.75 Our new baseline system after several modifications + 20-bestparses, KN7, RNNLM reranking-3.75 SMT—– SMT+10.00 Character based SMT+6.50 Pre-reordering for phrase-based SMT (dependency parsing +manual rules)-5.25 RBMT system+0.75 RBMT with SPE(Statistical Post Editing) system—– Our baseline system (segmentation tools: urheen and mecab,
moses: 1.0).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
—– Our baseline system (segmentation tools: urheen and mecab,
moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus segmentation tools: urheen and mecab, moses: 2.1.1).
—– Our baseline system (segmentation tools: kytea, moses: 1.0).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
—– Our baseline system (segmentation tools: kytea, moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
—– Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).

HUMAN SYSTEM DESCRIPTION

137
RBMT
138 SMT and RBMT
133
EBMT
135
EBMT
258
EBMT
268
EBMT

SMT
SMT
SMT
SMT
SMT

SMT

SMT

SMT

SMT
SMT

SMT
SMT

SMT

SMT

232
263
174
200
369

EIWA 2EIWA 1
Kyoto-U 3
Kyoto-U 4
Kyoto-U 2
Kyoto-U 1

SAS MT 2
SAS MT 1
Sense 2
Sense 1WASUIPS 1

WASUIPS 2 370

WASUIPS 3 374

WASUIPS 4 375

WASUIPS 5 379
WASUIPS 6 380

WASUIPS 7 383
WASUIPS 8 384

WASUIPS 9 387

WASUIPS 10 388

SMT
SMT
SMT
Other
Other
RBMT
RBMT
SMT
SMT

4
8
13
36
215
239
242
120
124

METHOD

SMT Hiero
SMT Phrase
SMT T2S
Online A
Online B
RBMT A
RBMT DNAIST 1NAIST 2SYSTEM ID ID

17

YES

NO

NO
YES

NO
YES

YES

NO

YES

NO
NO
NO
NO
NO

YES
YES
NO
NO
NO
NO

32.65

32.52

32.08
32.43

27.37
27.86

32.19

31.87

30.44

36.58
37.42
34.56
33.66
27.66

18.69
33.53
33.26
32.68
33.57
34.75

OTHER
RESOURCES juman
NO
35.43
NO
34.65
NO
36.52
YES
11.63
YES
10.48
NO
9.37
NO
8.39
NO
40.11
NO
40.21

32.81

32.69

33.09
33.36

28.28
28.89

32.55

32.26

30.92

36.22
37.65
35.08
33.86
28.09

18.33
33.74
35.09
33.30
34.43
35.89

BLEU
kytea
35.91
35.16
37.07
13.21
11.26
9.87
8.70
41.29
40.82

32.59

32.47

32.18
32.48

27.43
28.00

32.54

32.26

30.86

36.10
37.07
34.64
33.46
28.20

18.32
33.87
33.62
32.45
33.45
34.83

mecab
35.64
34.77
36.64
11.87
10.47
9.35
8.30
40.30
40.15

0.781733

0.780402

0.775168
0.778075

0.753749
0.756721

0.780027

0.777876

0.773142

0.807535
0.825551
0.766470
0.774338
0.762949

0.720281
0.800506
0.787105
0.783016
0.795390
0.798631

RIBES
kytea
0.798726
0.766384
0.820490
0.598172
0.596006
0.652402
0.626400
0.834824
0.838092

0.791219

0.790107

0.787665
0.789657

0.767073
0.769409

0.787591

0.786422

0.781475

0.817368
0.833048
0.771081
0.784012
0.770846

0.732466
0.808504
0.791269
0.786352
0.800986
0.802930

mecab
0.807665
0.771005
0.825025
0.598573
0.600706
0.661730
0.633319
0.842235
0.845625

Table 14: CJ submissions

0.796777

0.796059

0.793230
0.7962200.774423
0.776550

0.795838

0.794303

0.789824

0.8221800.834170
0.771975
0.789495
0.779183

0.740183
0.811350
0.791680
0.786229
0.800949
0.802629

juman
0.810406
0.772498
0.825292
0.595925
0.600733
0.666277
0.641189
0.842477
0.845486+4.75 Hierarchical Phrase-based SMT—– Phrase-based SMT+16.00 Tree-to-String SMT-21.75 Online A—– Online B-37.75 RBMT A—– RBMT D+50.75 Travatar-based Forest-to-String SMT System+38.00 Travatar-based Forest-to-String SMT System (Tuned
BLEU+RIBES)—– RBMT plus user dictionary+15.00 RBMT with user dictionary plus SPE(statistical post editing)—– Using n-best parses and RNNLM—– Our baseline system+6.00 Our new baseline system after several modifications+7.50 Our new baseline system after several modifications + 20-bestparses, KN7, RNNLM reranking—– Syntactic reordering phrase-based SMT (SAS token tool)+22.50 Syntactic reordering Hierarchical SMT (using SAS token tool)—– SMT-1.00 Character based SMT—– Our baseline system (segmentation tools: urheen and mecab,
moses: 1.0).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 1.0).
—– Our baseline system (segmentation tools: urheen and mecab,
moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: urheen and mecab, moses: 2.1.1).
—– Our baseline system (segmentation tools: kytea, moses: 1.0).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 1.0).
—– Our baseline system (segmentation tools: kytea, moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: kytea, moses: 2.1.1).
—– Our baseline system (segmentation tools: stanford-ctb and juman, moses: 2.1.1).
—– Our baseline system + additional quasi-parallel corpus (segmentation tools: stanford-ctb and juman, moses: 2.1.1).

HUMAN SYSTEM DESCRIPTION

References

T. Kudo.
2005.
Mecab :
Yet anotherpart-of-speech and morphological analyzer.
http://mecab.sourceforge.net/.

Jingsheng Cai, Yujie Zhang, Hua Shan, and Jinan Xu.
2014. System Description: Dependency-based Preordering for Japanese-Chinese Machine Translation.
In Proceedings of the 1st Workshop on Asian Translation (WAT2014).

Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements of Japanese morphological analyzer JUMAN.
In Proceedings of The International Workshop on
Sharable Natural Language, pages 22–28.

Chenchen Ding, Masao Utiyama, Eiichiro Sumita, andMikio Yamamoto. 2014. Word Order Does NOT
Differ Significantly Between Chinese and Japanese.
In Proceedings of the 1st Workshop on Asian Translation (WAT2014).

Graham Neubig, Yosuke Nakata, and Shinsuke Mori.
2011. Pointwise prediction for robust, adaptablejapanese morphological analysis. In Proceedings ofthe 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 529–
533, Stroudsburg, PA, USA. Association for Computational Linguistics.

Terumasa Ehara. 2014. A machine translation system
combining rule-based machine translation and statistical post-editing. In Proceedings of the 1st Workshop on Asian Translation (WAT2014).
J.L. Fleiss et al. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin,
76(5):378–382.

Graham Neubig. 2014. Forest-to-String SMT for
Asian Language Translation: NAIST at WAT 2014.
In Proceedings of the 1st Workshop on Asian Translation (WAT2014).

Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H.
Clark, and Philipp Koehn. 2013. Scalable modified
kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers), pages 690–696, Sofia, Bulgaria, August. Association for Computational Linguistics.

Kenichi Ohwada, Ryosuke Miyazaki, and Mamoru
Komachi. 2014. Predicate-Argument Structurebased Preordering for Japanese-English Statistical
Machine Translation of Scientific Papers. In Proceedings of the 1st Workshop on Asian Translation(WAT2014).

Hieu Hoang, Philipp Koehn, and Adam Lopez. 2009.
A unified framework for phrase-based, hierarchical,
and syntax-based statistical machine translation. In
Proceedings of the International Workshop on Spoken Language Translation, pages 152–159.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–
318.
Slav Petrov, Leon Barrett, Romain Thibaux, and Dan
Klein. 2006. Learning accurate, compact, andinterpretable tree annotation. In Proceedings ofthe 21st International Conference on Computational
Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440,
Sydney, Australia, July. Association for Computational Linguistics.

Sho Hoshino, Hubert Soyer, Yusuke Miyao, and Akiko
Aizawa. 2014. Japanese to English Machine Translation using Preordering and Compositional Distributed Semantics. In Proceedings of the 1st Workshop on Asian Translation (WAT2014).
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic
evaluation of translation quality for distant language
pairs. In Proceedings of the 2010 Conference on
Empirical Methods in Natural Language Processing, EMNLP ’10, pages 944–952, Stroudsburg, PA,
USA. Association for Computational Linguistics.

John Richardson, Fabien Cromières, Toshiaki
Nakazawa, and Sadao Kurohashi. 2014. KyotoEBMT System Description for the 1st Workshop
on Asian Translation. In Proceedings of the 1st
Workshop on Asian Translation (WAT2014).

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session.

Satoshi Sonoh, Satoshi Kinoshita, Hiroyuki Tanaka,
and Satoshi Kamatani. 2014. Toshiba MT System
Description for the WAT2014 Workshop. In Proceedings of the 1st Workshop on Asian Translation(WAT2014).
Liling Tan and Francis Bond. 2014. Manipulating Input Data in Machine Translation. In Proceedings ofthe 1st Workshop on Asian Translation (WAT2014).

Philipp Koehn. 2004. Statistical significance tests formachine translation evaluation. In Dekang Lin andDekai Wu, editors, Proceedings of EMNLP 2004,
pages 388–395, Barcelona, Spain, July. Association
for Computational Linguistics.

Huihsin Tseng. 2005. A conditional random field word
segmenter. In In Fourth SIGHAN Workshop on Chinese Language Processing.
18

Masao Utiyama and Hitoshi Isahara. 2007. Ajapanese-english patent parallel corpus. In MT summit XI, pages 475–482.
Rui Wang, Xu Yang, and Yan Gao. 2014. The SAS
Statistical Machine Translation System for WAT
2014. In Proceedings of the 1st Workshop on AsianTranslation (WAT2014).
Wei Yang and Yves Lepage. 2014. Consistent
Improvement in Translation Quality of Chinese–
Japanese Technical Texts by Adding Additional
Quasi-parallel Training Data. In Proceedings of the
1st Workshop on Asian Translation (WAT2014).
Zhongyuan Zhu. 2014. Weblio Pre-reordering Statistical Machine Translation System. In Proceedings ofthe 1st Workshop on Asian Translation (WAT2014).

19