PACLIC 32 - WAT 2018

Overview of the 5th Workshop on Asian Translation
Toshiaki Nakazawa
Katsuhito Sudoh
The University of Tokyo
Nara Institute of Science and Technology
nakazawa@logos.t.u-tokyo.ac.jp
sudoh@is.naist.jp
Shohei Higashiyama and Chenchen Ding and Raj Dabre
National Institute ofInformation and Communications Technology{shohei.higashiyama, chenchen.ding, raj.dabre}@nict.go.jp
Hideya Mino and Isao Goto
Win Pa Pa
NHK
University of Conputer Study, Yangon{mino.h-gq, goto.i-es}@nhk.or.jp
winpapa@ucsy.edu.mm
Anoop Kunchukuttan
Microsoft AI and Researchanoop.kunchukuttan@gmail.com
Abstract
This paper presents the results of the sharedtasks from the 5th workshop on Asian translation (WAT2018) including Ja↔En, Ja↔Zhscientific paper translation subtasks, Zh↔Ja,
K↔Ja, En↔Ja patent translation subtasks,
Hi↔En, My↔En mixed domain subtasks and
Bn/Hi/Ml/Ta/Te/Ur/Si↔En Indic languages
multilingual subtasks. For the WAT2018, 17
teams participated in the shared tasks. About
500 translation results were submitted to theautomatic evaluation server, and selected submissions were manually evaluated.

1 Introduction
The Workshop on Asian Translation (WAT) is a
new open evaluation campaign focusing on Asian
languages. Following the success of the previous
workshops WAT2014-WAT2017 (Nakazawa et al.,2014; Nakazawa et al., 2015; Nakazawa et al., 2016;
Nakazawa et al., 2017), WAT2018 brings together
machine translation researchers and users to try,evaluate, share and discuss brand-new ideas of machine translation. We have been working towardpractical use of machine translation among all Asian
countries.
For the 5th WAT, we adopted new translation subtasks with Myanmar ↔ En-

Sadao Kurohashi
Kyoto University
kuro@i.kyoto-u.ac.jp
glish mixed domain corpus1 and Bengali/Hindi/Malayalam/Tamil/Telugu/Urdu/Sinhalese↔ English OpenSubtitles corpus2 in addition to thesubtasks at WAT2017.
WAT is the uniq workshop on Asian languagetransration with the following characteristics:• Open innovation platform
Due to the fixed and open test data, we canrepeatedly evaluate translation systems on thesame dataset over years. WAT receives submissions at any time; i.e., there is no submission
deadline of translation results w.r.t automatic
evaluation of translation quality.
• Domain and language pairs
WAT is the world’s first workshop thattargets scientific paper domain,
andChinese↔Japanese and Korean↔Japanese
language pairs. In the future, we will add more
Asian languages such as Vietnamese, Thai andso on.
• Evaluation method
Evaluation is done both automatically and manually. Firstly, all submitted translation results1

http://lotus.kuee.kyoto-u.ac.jp/WAT/
my-en-data/2
http://lotus.kuee.kyoto-u.ac.jp/WAT/
indic-multilingual/90432nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Lang
JE
JC

Train
3,008,500
672,315

Dev
1,790
2,090

DevTest
1,784
2,148

Test
1,812
2,107

Lang
zh-ja
ko-ja
en-ja

Train
1,000,000
1,000,000
1,000,000

Dev
2,000
2,000
2,000

DevTest
2,000
2,000
2,000

Test-N
5,204
5,230
5,668

Lang
zh-ja
ko-ja
en-ja

Test-N1
2,000
2,000
2,000

Test-N2
3,000
3,000
3,000

Test-N3
204
230
668

Test-EP
1,151––

Table 1: Statistics for ASPEC

are automatically evaluated using three metrics:
BLEU, RIBES and AMFM. Among them, selected translation results are assessed by two
kinds of human evaluation: pairwise evaluation
and JPO adequacy evaluation.

Table 2: Statistics for JPC

similarity score.

2

Dataset2.1.2 ASPEC-JC

2.1 ASPECASPEC was constructed by the Japan Science andTechnology Agency (JST) in collaboration with the
National Institute of Information and Communications Technology (NICT). The corpus consists ofa Japanese-English scientific paper abstract corpus(ASPEC-JE), which is used for ja↔en subtasks, anda Japanese-Chinese scientific paper excerpt corpus(ASPEC-JC), which is used for ja↔zh subtasks.
The statistics for each corpus are shown in Table 1.
2.1.1 ASPEC-JE
The training data for ASPEC-JE was constructed
by NICT from approximately two million JapaneseEnglish scientific paper abstracts owned by JST. The
data is a comparable corpus and sentence correspondences are found automatically using the methodfrom (Utiyama and Isahara, 2007). Each sentencepair is accompanied by a similarity score that arecalculated by the method and a field ID that indicates a scientific field. The correspondence between
field IDs and field names, along with the frequency
and occurrence ratios for the training data, are described in the README file of ASPEC-JE.
The development, development-test and test data
were extracted from parallel sentences from theJapanese-English paper abstracts that exclude thesentences in the training data. Each dataset consistsof 400 documents and contains sentences in eachfield at the same rate. The document alignment was
conducted automatically and only documents with a
1-to-1 alignment are included. It is therefore possible to restore the original documents. The format is
the same as the training data except that there is no

ASPEC-JC is a parallel corpus consisting ofJapanese scientific papers, which come from the literature database and electronic journal site J-STAGE
by JST, and their translation to Chinese with permission from the necessary academic associations.
Abstracts and paragraph units are selected from thebody text so as to contain the highest overall vocabulary coverage.
The development, development-test and test data
are extracted at random from documents containing
single paragraphs across the entire corpus. Each set
contains 400 paragraphs (documents). There are no
documents sharing the same data across the training,development, development-test and test sets.
2.2 JPC
JPO Patent Corpus (JPC) for the patent tasks wasconstructed by the Japan Patent Office (JPO) in
collaboration with NICT. The corpus consists ofChinese-Japanese, Korean-Japanese and EnglishJapanese patent descriptions whose International
Patent Classification (IPC) sections are chemistry,electricity, mechanical engineering, and physics.
At WAT2018, the patent tasks has two subtasks:
normal subtask and expression pattern subtask. Bothsubtasks uses common training, development anddevelopment-test data for each language pair. The
normal subtask for three language pairs uses four
test data with different characteristics:• test-N: union of the following three sets;• test-N1: patent documents from patent families
published between 2011 and 2013;905
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Lang
en-ja

Train
200,000

Dev
2,000

DevTest
2,000

Test
2,000

Table 3: Statistics for JIJI Corpus• test-N2: patent documents from patent families
published between 2016 and 2017; and• test-N3: patent documents published between
2016 and 2017 where target sentences are manually created by translating source sentences.

Lang
hi-en
hi-ja
hi

Train
1,492,827
152,692
–

Dev
520
1,566
–

Test
2,507
2,000
–

Mono
–
–
45,075,279

Table 4: Statistics for IITB Corpus. “Mono” indicatesmonolingual Hindi corpus.
Lang
en-ja

TextType
Title
Ingredient
Step

Train
14,779127,244108,993

Dev
500
4,274
3,303

DevTest
500
4,188
3,086

Test
500
3,935
2,804

Table 5: Statistics for Recipe Corpus

The expression pattern subtask for zh→ja pair uses
test-EP data. The test-EP data consists of sentences
annotated with expression pattern categories: title
of invention (TIT), abstract (ABS), scope of claim(CLM) or description (DES). The corpus statistics
are shown in Table 2. Note that training, development, development-test and test-N1 data are thesame as those used in WAT2017.
2.3 JIJI Corpus
JIJI Corpus was constructed by Jiji Press Ltd. in collaboration with NICT. The corpus consists of news
text that comes from Jiji Press news of various categories including politics, economy, nation, business, markets, sports and so on. The corpus is partitioned into training, development, development-test
and test data, which consists of Japanese-English
sentence pairs. The statistics for each corpus areshown in Table 3.
The sentence pairs in each data are identified
in the same manner as that for ASPEC using themethod from (Utiyama and Isahara, 2007).
2.4 IITB Corpus
IIT Bombay English-Hindi Corpus contains
English-Hindi parallel corpus as well as monolingual Hindi corpus collected from a variety ofsources and corpora. This corpus had been developed at the Center for Indian Language Technology,
IIT Bombay over the years. The corpus is used formixed domain tasks hi↔en. The statistics for the
corpus are shown in Table 4.
2.5 Recipe Corpus
Recipe Corpus was constructed by Cookpad Inc.
Each recipe consists of a title, ingredients, steps, adescription and a history. Every text in titles, ingredients and steps consists of a parallel sentence while
one in descriptions and histories is not always a parallel sentence. Although all of the texts in the training set can be used for training, only titles, ingredients and steps in the test set is used for evaluation.
The statistics for each corpus are described in Table5.
2.6 ALT and UCSY Corpus
The parallel data for Myanmar-English translation
tasks at WAT2018 consists of two corpora, the ALT
corpus and UCSY corpus.
• The ALT corpus is one part from the
Asian Language Treebank (ALT) project (Rizaet al., 2016), consisting of twenty thousand Myanmar-English parallel sentences fromnews articles.
• The UCSY corpus (Yi Mon Shwe Sin and Khin
Mar Soe, 2018) is constructed by the NLPLab, University of Computer Studies, Yangon(UCSY), Myanmar. The corpus consists of
200 thousand Myanmar-English parallel sentences collected from different domains, including news articles and textbooks.
The released Myanmar textual data have been tokenized into writing units and Romanized. The script
for tokenization and recovery is also provided forparticipants,3 so that they can make use of their own
data and tools for further processing. The automatic3

http://www2.nict.go.jp/astrec-att/
member/mutiyama/ALT/myan2roma.py

906
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Corpus
ALT
UCSY
All

Train
17,965
208,638
226,603

Dev
993
–
993

Test
1,007
–
1,007

Table 6: Statistics for the data used in Myanmar-English
translation tasks
Lang

Train

Dev

Test

bn-en
hi-en
ml-en
ta-en
te-en
ur-en
si-en
en

337,428
84,557359,423
26,21722,165
26,619521,726
–

500
500
500
500
500
500
500
–

1,000
1,000
1,000
1,000
1,000
1,000
1,000
–

Mono
(src)
453,859
104,967
402,761
30,26824,75029,086
705,793
2,891,079

Table 7: Statistics for Indic Languages Corpus

evaluation of Myanmar translation results is basedon the tokenized writing units, and the human evaluation is based on the recovered Myanmar text.
The detailed composition of training, development, and test data of the Myanmar-English translation tasks are listed in Table 6.
2.7 Indic Languages Corpus
The Indic Languages Corpus covers 8 languages,
namely: Bengali, Hindi, Malayalam, Tamil, Telugu, Sinhalese, Urdu and English. The corpus has
been collected from OPUS4 and belongs to the spoken language (OpenSubtitles) domain. This corpus is used for the pilot as well as multilingualEnglish↔Indic Languages sub-tasks. The corpus isa collection of 7 bilingual parallel corpora of varying
sizes, one for each Indic language and English. The
parallel corpora are also accompanied by monolingual corpora from the same domain. The statisticsof the parallel and monolingual corpora are given in
Table 7.

3

each participant’s system. That is, the specific baseline system was the standard for human evaluation.
At WAT 2018, we adopted a neural machine translation (NMT) with attention mechanism as a baseline
system except for the IITB tasks. We used a phrasebased statistical machine translation (SMT) system,
which is the same system as that at WAT 2017, as
the baseline system for the IITB tasks.
The NMT baseline systems consisted of publicly
available software, and the procedures for building
the systems and for translating using the systems
were published on the WAT web page.5 We used
OpenNMT (Klein et al., 2017) as the implementation of the baseline NMT systems. In addition to the
NMT baseline systems, we have SMT baseline systems for the tasks that started at last year or before
last year. The baseline systems are shown in Tables
8, 9, and 10.
SMT baseline systems are described in the previous WAT overview paper (Nakazawa et al., 2017).
The commercial RBMT systems and the onlinetranslation systems were operated by the organizers. We note that these RBMT companies and onlinetranslation companies did not submit themselves.
Because our objective is not to compare commercial
RBMT systems or online translation systems fromcompanies that did not themselves participate, thesystem IDs of these systems are anonymous in thispaper.

Baseline Systems

Human evaluations were conducted as pairwise
comparisons between the translation results for a
specific baseline system and translation results for4

http://opus.nlpl.eu

5

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2018/baseline/baselineSystems.html

907
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT X
Online X
Online X
AIAYN

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
Other
Other
NMT
✓
✓
✓

✓
✓
✓

ja-en
✓
✓
✓
✓

✓
✓
✓

✓
✓

✓
✓

ASPEC
en-ja ja-zh
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

Table 8: Baseline Systems I

SystemOpenNMT’s attention-based NMT
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
ATLAS V14 (Commercial system)
PAT-Transer 2009 (Commercial system)
PC-Transer V13 (Commercial system)
J-Beijing 7 (Commercial system)
Hohrai 2011 (Commercial system)
J Soul 9 (Commercial system)
Korai 2011 (Commercial system)
Google translate
Bing translator
Google’s implementation of “Attention Is All You Need”
✓
✓

✓
✓

✓

zh-ja
✓
✓
✓

✓
✓

✓
✓
✓

ja-en
✓
✓
✓
✓

✓
✓

✓
✓
✓
✓

en-ja
✓
✓
✓

✓
✓

✓

✓
✓

✓
✓

JPC
ja-zh zh-ja
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓
✓
✓

ja-ko
✓
✓
✓

✓
✓
✓
✓

ko-ja
✓
✓
✓

PACLIC 32 - WAT 2018908
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
Other
Other

JIJI
ja-en en-ja
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

Type
NMT
NMT
NMT
NMT

Table 10: Baseline Systems III

SystemOpenNMT’s NMT with attention
OpenNMT’s NMT with attention and multilingual tags (many to one)
OpenNMT’s NMT with attention and multilingual tags (one to many)
OpenNMT’s NMT with attention and multilingual tags (many to many)

Table 9: Baseline Systems II

SystemOpenNMT’s NMT with attention
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
PC-Transer V13 (Commercial system)
Google translate
Bing translator

System ID
NMT
NMT M2ONMT O2M
NMT M2M

System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT X
Online X
Online X
✓
✓

✓
✓

IITB
en-hi hi-ja
✓
✓
✓
✓

✓
✓

ja-hi
✓
✓

✓
✓
✓
✓

✓
✓
✓
✓

Recipeja-en en-ja
✓
✓
✓
✓

✓

✓

ALT
my-en en-my
✓
✓

Indic
{bn,hi,ml,ta,te,ur,si}-en en-{bn,hi,ml,ta,te,ur,si}
✓
✓
✓
✓
✓
✓

✓
✓

hi-en
✓
✓

PACLIC 32 - WAT 2018909
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

3.1 Training Data
We used the following data for training the NMT
baseline systems.
• All of the training data for each task were used
for training except for the ASPEC Japanese–
English task. For the ASPEC Japanese–English
task, we only used train-1.txt, which consistsof one million parallel sentence pairs with highsimilarity scores.
• All of the development data for each task wasused for validation.
3.2 Tokenization
We used the following tools for tokenization.
• Juman version 7.06 for Japanese segmentation.
• Stanford Word Segmenter version 2014-01047 (Chinese Penn Treebank (CTB) model) for
Chinese segmentation.
• The Moses toolkit for English and Indonesian
tokenization.
• Mecab-ko8 for Korean segmentation.
• Indic NLP Library9 for Indic language segmentation.
• subword-nmt10 for all languages.
When we built BPE-codes, we merged source andtarget sentences and we used 100,000 for -s option. We used 10 for vocabulary-threshold whensubword-nmt applied BPE.
3.3 NMT with attention
We used the following OpenNMT configuration for
the NMT with attention system.
• encoder type = brnn
• brnn merge = concat• src seq length = 150• tgt seq length = 150• src vocab size = 100000
6

http://nlp.ist.i.kyoto-u.ac.jp/EN/
index.php?JUMAN
7
http://nlp.stanford.edu/software/segmenter.shtml
8
https://bitbucket.org/eunjeon/mecab-ko/
9https://bitbucket.org/anoopk/indic_nlp_
library10
https://github.com/rsennrich/
subword-nmt• tgt vocab size = 100000• src words min frequency = 1• tgt words min frequency = 1
The default values were used for the other systemparameters.
For many to one, one to many, and many to many
multilingual NMT (Johnson et al., 2017), we add<2XX> tags, which indicate the target language(XX is replaced by the language code), to the headof the source language sentences.

4 Automatic Evaluation
4.1 Procedure for Calculating Automatic
Evaluation Score
We evaluated translation results by three metrics:
BLEU (Papineni et al., 2002), RIBES (Isozaki etal., 2010) and AMFM (Banchs et al., 2015). BLEU
scores were calculated using multi-bleu.perlin the Moses toolkit (Koehn et al., 2007). RIBES
scores were calculated using RIBES.py version
1.02.4.11 AMFM scores were calculated using
scripts created by the technical collaborators listed
in the WAT2018 web page.12 All scores for each task
were calculated using the corresponding referencetranslations.
Before the calculation of the automatic evaluation scores, the translation results were tokenized or
segmented with tokenization/segmentation tools foreach language. For Japanese segmentation, we used
three different tools: Juman version 7.0 (Kurohashi
et al., 1994), KyTea 0.4.6 (Neubig et al., 2011) withfull SVM model13 and MeCab 0.996 (Kudo, 2005)
with IPA dictionary 2.7.0.14 For Chinese segmentation, we used two different tools: KyTea 0.4.6 withfull SVM Model in MSR model and Stanford Word
Segmenter (Tseng, 2005) version 2014-06-16 with
Chinese Penn Treebank (CTB) and Peking University (PKU) model.15 For Korean segmentation, we
11
http://www.kecl.ntt.co.jp/icl/lirg/
ribes/index.html
12
lotus.kuee.kyoto-u.ac.jp/WAT/WAT2018/13
http://www.phontron.com/kytea/model.
html
14
http://code.google.com/p/mecab/
downloads/detail?name=mecab-ipadic-2.7.
0-20070801.tar.gz
15
http://nlp.stanford.edu/software/segmenter.shtml

910
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018used mecab-ko.16 For English tokenization, we usedtokenizer.perl17 in the Moses toolkit. For
Hindi, Bengali, Malayalam, Tamil, Telugu, Urdu
and Sinhalese tokenization, we used Indic NLP Library.18 The detailed procedures for the automatic
evaluation are shown on the WAT2018 evaluation
web page.19
4.2 Automatic Evaluation System
The automatic evaluation system receives translation results by participants and automatically gives
evaluation scores to the uploaded results. As shownin Figure 1, the system requires participants to provide the following information for each submission:• Human Evaluation: whether or not they submit
the results for human evaluation;• Publish the results of the evaluation: whether or
not they permit to publish automatic evaluation
scores on the WAT2018 web page.
• Task: the task you submit the results for;• Used Other Resources: whether or not theyused additional resources; and• Method: the type of the method including
SMT, RBMT, SMT and RBMT, EBMT, NMT
and Other.
Evaluation scores of translation results that participants permit to be published are disclosed via the
WAT2018 evaluation web page.20 Participants canalso submit the results for human evaluation using
the same web interface.
This automatic evaluation system will remain
available even after WAT2018. Anybody can register an account for the system by the procedures described in the registration web page. 21
16https://bitbucket.org/eunjeon/mecab-ko/
https://github.com/moses-smt/
mosesdecoder/tree/RELEASE-2.1.1/scripts/
tokenizer/tokenizer.perl
18
https://bitbucket.org/anoopk/indic_nlp_
library19
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/index.html
20
lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/index.html
21
http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2018/registration/index.html
17

5 Human Evaluation
In WAT2018, we conducted two kinds of humanevaluations: pairwise evaluation and JPO adequacy
evaluation.
5.1 Pairwise Evaluation
We conducted pairwise evaluation for participants’systems submitted for human evaluation. The submitted translations were evaluated by a professionaltranslation company and Pairwise scores were given
to the submissions by comparing with baselinetranslations (described in section 3).
5.1.1 Sentence Selection and Evaluation
For the pairwise evaluation, we randomly selected
400 sentences from the test set of each task. We used
the same sentences as the last year for the continuous
subtasks. Baseline and submitted translations wereshown to annotators in random order with the input
source sentence. The annotators were asked to judge
which of the translations is better, or whether they
are on par.
5.1.2 Voting
To guarantee the quality of the evaluations, eachsentence is evaluated by 5 different annotators andthe final decision is made depending on the 5 judgements. We define each judgement ji (i = 1, · · · , 5)as:

if better than the baseline 1−1 if worse than the baselineji =
0if the quality is the sameThe∑ final decision D is defined as follows using S =ji : win (S ≥ 2)
loss (S ≤ −2)D=
tie (otherwise)
5.1.3 Pairwise Score Calculation
Suppose that W is the number of wins comparedto the baseline, L is the number of losses and T is thenumber of ties. The Pairwise score can be calculatedby the following formula:
P airwise = 100 ×

W −L
W +L+T

From the definition, the Pairwise score ranges between -100 and 100.

911
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

WAT

The Workshop on Asian Translation
Submission
SUBMISSION
Logged in as: ORGANIZER
LogoutSubmission:
Human

human evaluation

Evaluation:
Publish theresults of the

publishevaluation:
Team Name:

ORGANIZER

Task:

en-ja

Submission

ファイルを選択 選択されていません

File:Used Other

used other resources such as parallel corpora, monolingual corpora andResources:

parallel dictionaries in addition to oﬃcial corpora

Method:SMT

SystemDescription(public):

100
charactersor less

SystemDescription(private):100
charactersor less

SubmitGuidelines for submission:
System requirements:
The latest versions of Chrome, Firefox, Internet Explorer and Safari are supported for this site.
Before you submit ﬁles, you need to enable JavaScript in your browser.
File format:
Submitted ﬁles should NOT be tokenized/segmented. Please check the automatic evaluation procedures.
Submitted ﬁles should be encoded in UTF-8 format.
Translated sentences in submitted ﬁles should have one sentence per line, corresponding to each test sentence. The number of lines in the submitted ﬁle and that of thecorresponding test ﬁle should be the same.
Tasks:
en-ja, ja-en, zh-ja, ja-zh indicate the scientiﬁc paper tasks with ASPEC.
HINDENen-hi, HINDENhi-en, HINDENja-hi, and HINDENhi-ja indicate the mixed domain tasks with IITB Corpus.
JIJIen-ja and JIJIja-en are the newswire tasks with JIJI Corpus.
RECIPE{ALL,TTL,STE,ING}en-ja and RECIPE{ALL,TTL,STE,ING}ja-en indicate the recipe tasks with Recipe Corpus.
ALTen-my and ALTmy-en indicate the mixed domain tasks with UCSY and ALT Corpus.
INDICen-{bn,hi,ml,ta,te,ur,si} and INDIC{bn,hi,ml,ta,te,ur,si}-en indicate the Indic languages multilingual tasks with Indic Languages Multilingual Parallel Corpus.
JPC{N,N1,N2,N3,EP}zh-ja ,JPC{N,N1,N2,N3}ja-zh, JPC{N,N1,N2,N3}ko-ja, JPC{N,N1,N2,N3}ja-ko, JPC{N,N1,N2,N3}en-ja, and JPC{N,N1,N2,N3}ja-en indicate the patent tasks
with JPO Patent Corpus. JPCN1{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} are the same tasks as JPC{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} in WAT2015-WAT2017. AMFM is not calculated
for JPC{N,N2,N3} tasks.
Human evaluation:
If you want to submit the ﬁle for human evaluation, check the box "Human Evaluation". Once you upload a ﬁle with checking "Human Evaluation" you cannot change the ﬁle used
for human evaluation.
When you submit the translation results for human evaluation, please check the checkbox of "Publish" too.
You can submit two ﬁles for human evaluation per task.
One of the ﬁles for human evaluation is recommended not to use other resources, but it is not compulsory.
Other:
Team Name, Task, Used Other Resources, Method, System Description (public) , Date and Time(JST), BLEU, RIBES and AMFM will be disclosed on the Evaluation Site when you
upload a ﬁle checking "Publish the results of the evaluation".
You can modify some ﬁelds of submitted data. Read "Guidelines for submitted data" at the bottom of this page.
Back to top

Figure 1: The interface for translation results submission

912
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

5.1.4 Confidence Interval Estimation

5

There are several ways to estimate a confidence
interval. We chose to use bootstrap resampling(Koehn, 2004) to estimate the 95% confidence interval. The procedure is as follows:432

1. randomly select 300 sentences from the 400
human evaluation sentences, and calculate the
Pairwise score of the selected sentences

1

All important information is transmitted correctly. (100%)
Almost all important information is transmitted correctly. (80%–)
More than half of important information istransmitted correctly. (50%–)
Some of important information is transmitted
correctly. (20%–)
Almost all important information is NOTtransmitted correctly. (–20%)
Table 11: The JPO adequacy criterion

2. iterate the previous step 1000 times and get1000 Pairwise scores
3. sort the 1000 scores and estimate the 95% confidence interval by discarding the top 25 scoresand the bottom 25 scores
5.2 JPO Adequacy Evaluation
We conducted JPO adequacy evaluation for the top
two or three participants’ systems of pairwise evalution for each subtask.22 The evaluation was carried out by translation experts based on the JPO adequacy evaluation criterion, which is originally defined by JPO to assess the quality of translated patentdocuments.
5.2.1 Sentence Selection and Evaluation
For the JPO adequacy evaluation, the 200 test sentences were randomly selected from the 400 test sentences used for the pairwise evaluation. For eachtest sentence, input source sentence, translation by
participants’ system, and reference translation wereshown to the annotators. To guarantee the quality ofthe evaluation, each sentence was evaluated by twoannotators. Note that the selected sentences are thesame as those used in the previous workshops except
for the new subtasks at WAT2018.
5.2.2 Evaluation Criterion
Table 11 shows the JPO adequacy criterion from 5
to 1. The evaluation is performed subjectively. “Important information” represents the technical factors
and their relationships. The degree of importance ofeach element is also considered to evaluate. The percentages in each grade are rough indications for the22

The number of systems varies depending on the subtasks.

transmission degree of the source sentence meanings. The detailed criterion is described in the JPO
document (in Japanese). 23

6 Participants
Table 12 shows the participants in WAT2018. The
table lists 17 organizations from various countries,
including Japan, China, India, Myanmar, Czech andIreland.
More than 500 translation results by 17 teams
were submitted for automatic evaluation and about
70 translation results by 16 teams were submitted
for pairwise evaluation. We selected about 40 translation results for JPO adequacy evaluation according
to the pairwise evaluation scores. Table 13 shows
tasks for which each team submitted results by thesubmission deadline. Unfortunately, there were no
submissions to Recipe and JIJI tasks this year.

7 Evaluation Results
In this section, the evaluation results for WAT2018
are reported from several perspectives. Some of theresults for both automatic and human evaluations arealso accessible at the WAT2018 website.24
7.1 Official Evaluation Results
Figures 2, 3, 4 and 5 show the official evaluation
results of ASPEC subtasks, Figures 6, 7, 8, 9, 10,
11, 12 and 13 show those of JPC subtasks, Figures
14 and 15 show those of IITB subtasks, Figures 16
and 17 show those of ALT subtasks and Figures 18,
23

http://www.jpo.go.jp/shiryou/toushin/chousa/tokkyohonyaku_hyouka.htm
24
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/

913
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 201819, 20 and 21 show those of INDIC subtasks. Eachfigure contains automatic evaluation results (BLEU,
RIBES, AM-FM), the pairwise evaluation resultswith confidence intervals, correlation between automatic evaluations and the pairwise evaluation, the
JPO adequacy evaluation result and evaluation summary of top systems. Some of the figures for somesubtasks are omitted because the pairwise evaluation
was not conducted or none of the human evaluation
was conducted.
The detailed automatic evaluation results areshown in Appendix A. The detailed JPO adequacy
evaluation results for the selected submissions areshown in Table 14. The weights for the weighted
κ (Cohen, 1968) is defined as |Evaluation1 −Evaluation2|/4.
7.2 Statistical Significance Testing of Pairwise
Evaluation between Submissions
Tables 15 and 16 show the results of statisticalsignificance testing of ASPEC subtasks, Table 17
shows that of IITB subtasks, Table 18 shows that of
ALT subtasks and Tables 19 and 20 show those of
INDIC subtasks. ≫, ≫ and > mean that the system in the row is better than the system in the column at a significance level of p < 0.01, 0.05 and 0.1
respectively. Testing is also done by the bootstrap
resampling as follows:
1. randomly select 300 sentences from the 400
pairwise evaluation sentences, and calculate the
Pairwise scores on the selected sentences forboth systems
2. iterate the previous step 1000 times and countthe number of wins (W ), losses (L) and ties (T )languages in general. The κ values for Hindi languages are relatively higt. This might be because theoverall translation quality of the Hindi languages are
low, and the evaluators can easily distinguish bettertranslations from worse ones.

8 Conclusion and Future Perspective
This paper summarizes the shared tasks of
WAT2018. We had 17 participants worldwide, andcollected a large number of useful submissions for
improving the current machine translation systems
by analyzing the submissions and identifying the issues.
For the next WAT workshop, we plan to conduct
documen-level evaluation using the new dataset with
context for some translation subtasks and we would
like to consider how to realize context-aware evaluation in WAT. Also, we are planning to do extrinsic
evaluation of the translations.

Appendix A

Submissions

Tables 23 to 37 summarize translation results submitted for WAT2018 human evaluation. Type,
RSRC, Pair, and Adeq columns indicate type of
method, use of other resources, pairwise evaluation
score, and JPO adequacy evaluation score, respectively.
The tables also include results by the organizers’
baselines, which are listed in Table 10. For ALT
tasks, we also evaluated outputs of Online-A system
and its post-processed version where the westerncomma (,) is replaced into Myanmar native comma(0x104a). We conducted the post-processing because Myanmar native punctuation marks are consistently used in the WAT 2018 dataset.

3. calculate p = WL+L
Inter-annotator Agreement
To assess the reliability of agreement between theworkers, we calculated the Fleiss’ κ (Fleiss and others, 1971) values. The results are shown in Table
21. We can see that the κ values are larger for X→ J translations than for J → X translations. Thismay be because the majority of the workers for theselanguage pairs are Japanese, and the evaluation of
one’s mother tongue is much easier than for other914
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Team IDsrcb (Li et al., 2018)
Osaka-U (Kawara et al., 2018)
RGNLP (Ojha et al., 2018)
TMU (Zhang et al., 2018),(Matsumura et al., 2018)
EHR (Ehara, 2018)
NICT (Wang et al., 2018b)
NICT-4 (Marie et al., 2018)
NICT-5 (Dabre et al., 2018)
XMUNLP (Wang et al., 2018a)
UCSYNLP (Mo et al., 2018)
UCSMNLP (Thida et al., 2018)
kmust88USTC
CUNI (Kocmi et al., 2018)
Anuvaad (Banerjee et al., 2018)
IITP-MT (Sen et al., 2018)cvit-mt (Philip et al., 2018)

Organization
RICOH Software Research Center Beijing Co.,Ltd
Osaka University
Jawaharlal Nehru University / Dublin City University

Country
China
Japan
India, IrelandTokyo Metropolitan University

Japan

Ehara NLP Research Laboratory
NICT
NICT
NICT
Xiamen University
University of Computer Studies, Yangon
University of Computer Studies, Mandalay
Kunming University of Science and Technology
University of Science and Technology of China
Charles University, Prague
IIT Bombay / Microsft AI and Research, India
Indian Institute of Technology Patna
International Institute of Information Technology, Hyderabad

Japan
Japan
Japan
Japan
China
Myanmar
Myanmar
China
China
Czech
India
India
India

Table 12: List of participants in WAT2018

Team ID
srcb
Osaka-U
TMU
EHR
NICT
NICT-4
NICT-5XMUNLPUCSYNLPUCSMNLP
kmust88
USTC
CUNI
cvit-mt

Team ID
RGNLP
NICT-5
AnuvaadIITP-MT

EB
✓
✓
✓
✓

ASPEC
JE CJ
✓
✓
✓ ✓

EJ
✓
✓
✓
✓
✓

✓

✓

JC
✓

JPC (N/N1/N2/N3)
EJ CJ JC KJ

✓

✓

EH
✓
✓
✓
✓

HE
✓
✓
✓
✓

✓

IITB
EH HE

ALT
E-My My-E

✓

✓

✓

BE
✓
✓
✓
✓

JPC (EP)
CJ

E-Ml
✓
✓
✓
✓

Ml-E
✓
✓
✓
✓

✓

✓
✓

✓
✓
✓
✓
✓
✓

✓
✓
✓
✓

✓

Indic
E-Ta Ta-E
✓
✓
✓
✓
✓
✓
✓
✓

✓

✓
✓

E-Te
✓
✓
✓
✓

Te-E
✓
✓
✓
✓

✓
✓

EU
✓
✓
✓
✓

UE
✓
✓
✓
✓

ES
✓
✓
✓
✓

SE
✓
✓
✓
✓

Table 13: Submissions for each task by each team. E, J, C, K, H, B, U, and S denote English, Japanese, Chinese,
Korean, Hindi, Bengali, Urdu, and Sinhalese language, respectively.

915
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 2: Official evaluation results of aspec-ja-en.

916
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 3: Official evaluation results of aspec-en-ja.

917
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 4: Official evaluation results of aspec-ja-zh.

918
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 5: Official evaluation results of aspec-zh-ja.

919
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 6: Official evaluation results of jpcn1-en-ja.

Figure 7: Official evaluation results of jpcn1-ja-zh.

Figure 8: Official evaluation results of jpcn1-zh-ja.

920
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 9: Official evaluation results of jpcn1-ko-ja.

Figure 10: Official evaluation results of jpcn2-en-ja.

Figure 11: Official evaluation results of jpcn2-ja-zh.

921
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 12: Official evaluation results of jpcn2-zh-ja.

Figure 13: Official evaluation results of jpcn2-ko-ja.

922
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 14: Official evaluation results of iitb-hi-en.

923
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 15: Official evaluation results of iitb-en-hi.

924
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 16: Official evaluation results of alt-en-my.

925
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 17: Official evaluation results of alt-my-en.

926
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 18: Official evaluation results of indic-en-hi.

927
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 19: Official evaluation results of indic-hi-en.

928
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 20: Official evaluation results of indic-en-ta.

929
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Figure 21: Official evaluation results of indic-ta-en.

930
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

SYSTEM DATA Annotator A
Annotator B
allweightedκID
ID average varianceaverage varianceaverage κ
srcb
2474 4.37
0.49
4.63
0.44
4.50 0.15 0.25
NICT-5 2174 4.37
0.61
4.60
0.51
4.49 0.26 0.32
aspec-ja-en
0.91
3.92
1.41
3.94 0.34 0.48
TMU2464 3.94
2017 best 1681 4.15
0.58
4.13
0.52
4.14 0.29 0.41
NICT-5 2219 4.16
0.90
4.57
0.57
4.36 0.17 0.30
srcb
2479 4.04
1.07
4.30
1.00
4.17 0.22 0.38
aspec-en-ja
1.34
4.17
0.88
3.95 0.25 0.42
Osaka-U 2439 3.74
2017 best 1729 4.54
0.56
4.28
0.49
4.41 0.33 0.43
NICT-5 2266 4.67
0.32
4.27
0.90
4.47 0.28 0.36
aspec-ja-zh
srcb
2473 4.69
0.30
4.16
0.98
4.42 0.19 0.24
0.73
3.71
0.98
3.98 0.10 0.18
2017 best 1483 4.25
NICT-5 2267 4.78
0.26
4.48
0.67
4.63 0.31 0.33
aspec-zh-ja
2017 best 1481 4.63
0.47
3.99
0.98
4.31 0.17 0.23
0.35
4.62
0.45
4.64 0.36 0.44
EHR2476 4.66
jpcn1-en-ja
2017 best 1454 4.74
0.45
4.76
0.38
4.75 0.32 0.48
0.44
4.46
0.65
4.55 0.38 0.48
USTC2202 4.66
jpcn1-ja-zh
2017 best 1465 3.99
1.12
4.19
0.94
4.09 0.22 0.32
USTC2206 4.60
0.43
4.43
0.68
4.51 0.34 0.43
jpcn1-zh-ja
0.71
4.14
0.92
4.22 0.46 0.57
EHR
2210 4.29
2017 best 1484 4.41
0.68
4.51
0.64
4.46 0.26 0.34
EHR
2215 4.88
0.13
4.89
0.11
4.88 0.53 0.56
jpcn1-ko-ja
2017 best 1448 4.82
0.24
4.87
0.11
4.84 0.55 0.55
jpcn2-en-ja
EHR
2477 4.32
0.72
4.40
0.73
4.36 0.35 0.50
jpcn2-ja-zh USTC
2203 4.71
0.33
4.52
0.52
4.61 0.38 0.45
USTC2207 4.54
0.48
4.38
0.82
4.46 0.42 0.56
jpcn2-zh-ja
EHR
2211 4.37
0.65
4.08
1.07
4.22 0.33 0.46
2216 4.77
jpcn2-ko-ja
EHR
0.36
4.68
0.48
4.72 0.62 0.72
CUNI
2381 2.96
2.55
2.96
2.52
2.96 0.48 0.76
iitb-hi-en
cvit-mt 2331 2.87
2.54
2.88
2.68
2.88 0.53 0.76
2017 best 1511 3.43
1.64
3.60
1.74
3.51 0.22 0.45
CUNI2362 3.58
2.71
3.40
2.52
3.49 0.52 0.74
iitb-en-hi
cvit-mt 2254 3.21
2.58
3.18
2.56
3.20 0.64 0.81
2017 best 1576 3.95
1.18
3.76
1.85
3.86 0.17 0.36
NICT2345 3.51
0.65
4.04
0.54
3.77 0.03 0.08
alt-en-my NICT-4 2087 3.69
0.70
3.62
0.73
3.65 0.09 0.12
UCSYNLP 2339 2.04
0.60
2.90
0.56
2.47 0.01 0.07
0.61
3.88
0.73
4.04 0.13 0.23
NICT2329 4.19
alt-my-en NICT-4 2069 4.13
0.35
3.65
0.84
3.89 -0.00 0.07
UCSYNLP 2332 2.09
0.66
2.71
0.71
2.40 0.06 0.18
IITP-MT 2354 1.92
1.35
1.63
1.03
1.77 0.27 0.48
indic-en-hi NICT-5 2128 1.88
1.23
1.57
1.02
1.73 0.29 0.57
RGNLP 2417 1.46
0.82
1.44
0.86
1.45 0.48 0.67
NICT-5 2129 2.12
1.86
1.89
1.79
2.00 0.43 0.71
indic-hi-en IITP-MT 2347 2.02
1.91
1.94
1.75
1.98 0.43 0.67
0.92
1.47
0.86
1.46 0.54 0.75
RGNLP 2367 1.46
IITP-MT 2356 1.36
0.42
1.28
0.29
1.32 0.33 0.43
indic-en-ta
Anuvaad 2443 1.20
0.27
1.11
0.23
1.16 0.21 0.37
NICT-5 2132 1.10
0.17
1.14
0.27
1.12 0.39 0.51
IITP-MT 2349 1.84
1.32
1.78
0.89
1.81 0.23 0.39
indic-ta-en
NICT-5 2133 1.74
0.72
1.58
0.55
1.66 0.16 0.23
Anuvaad 2400 1.15
0.33
1.03
0.05
1.09 0.10 0.14
Subtask

Table 14: JPO adequacy evaluation results in detail.
931
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

NICT-5 (2174)≫≫≫≫≫≫
NICT-5 (2273)≫≫≫≫≫
srcb (2474)≫≫≫≫≫≫≫
TMU (2464)
Osaka-U (2440)≫≫≫
ORGANIZER (0006)

Osaka-U (2470)

ORGANIZER (0005)

TMU (2469)

EHR (2245)

Osaka-U (2439)

srcb (2479)
NICT-5 (2048)

Osaka-U (2472)

ORGANIZER (0006)

Osaka-U (2440)

TMU (2464)

NICT-5 (2273)
srcb (2474)

PACLIC 32 - WAT 2018

NICT-5 (2219)>≫≫≫≫≫≫
srcb (2479)≫≫≫≫≫≫
NICT-5 (2048)≫≫≫≫≫
Osaka-U (2439)≫ ≫≫≫EHR (2245)≫≫≫≫≫
TMU (2469)
ORGANIZER (0005)≫

ORGANIZER (0008)

srcb (2473)≫≫≫
NICT-5 (2266)- ≫NICT-5 (2175)≫

NICT-5 (2052)

ORGANIZER (0007)

NICT-5 (2175)

NICT-5 (2266)

Table 15: Statistical significance testing of the aspec-ja-en (left) and aspec-en-ja (right) Pairwise scores.

NICT-5 (2267) ≫ ≫NICT-5 (2052)≫

CUNI (2362) ≫ ≫ ≫cvit-mt (2254)≫≫
CUNI (2365)≫

CUNI (2381)

cvit-mt (2251)

CUNI (2365)

cvit-mt (2254)

Table 16: Statistical significance testing of the aspec-ja-zh (left) and aspec-zh-ja (right) Pairwise scores.

cvit-mt (2331) ≫Table 17: Statistical significance testing of the iitb-en-hi (left) and iitb-hi-en (right) Pairwise scores.

932
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

UCSMNLP (2338)

Osaka-U (2463)

Osaka-U (2438)

UCSYNLP (2332)

NICT-5 (2056)

NICT (2345)≫≫≫≫≫≫≫≫≫≫≫≫≫≫≫≫≫
NICT-4 (2087)
NICT (2282)- ≫≫≫≫≫≫
NICT-4 (2287)≫≫≫≫≫≫UCSYNLP (2339)- ≫≫≫≫
kmust88 (2360)≫≫≫≫
Osaka-U (2437)- ≫≫UCSYNLP (2340)≫≫≫
Osaka-U (2471)

NICT (2329)
NICT-4 (2290)
NICT (2281)

UCSMNLP (2337)

Osaka-U (2471)

UCSYNLP (2340)

Osaka-U (2437)

kmust88 (2360)

UCSYNLP (2339)

NICT-4 (2287)

NICT (2282)

NICT-4 (2087)

PACLIC 32 - WAT 2018

NICT-4 (2069)- ≫≫≫≫≫≫≫NICT (2329)> ≫≫≫≫≫≫
NICT-4 (2290)≫≫≫≫≫≫NICT (2281)≫≫≫≫≫
NICT-5 (2056)≫≫≫≫≫≫≫
UCSYNLP (2332)
Osaka-U (2438)≫ ≫≫Osaka-U (2463)

NICT-5 (2128) ≫ ≫ ≫ ≫ ≫IITP-MT (2354)≫ ≫≫≫RGNLP (2417)- ≫ ≫NICT-5 (2067)> ≫≫Anuvaad (2445)

RGNLP (2383)

Anuvaad (2403)

Anuvaad (2406)

NICT-5 (2066)

NICT-5 (2129)

RGNLP (2367)

RGNLP (2422)

Anuvaad (2445)

NICT-5 (2067)

RGNLP (2417)

IITP-MT (2354)

Table 18: Statistical significance testing of the alt-en-my (left) and alt-my-en (right) Pairwise scores.

IITP-MT (2347) ≫ ≫ ≫ ≫ ≫ ≫NICT-5 (2129)≫≫≫≫≫
RGNLP (2367)≫≫≫≫
NICT-5 (2066)≫≫≫
Anuvaad (2406)≫ ≫Anuvaad (2403)-

Anuvaad (2443) ≫ ≫ ≫IITP-MT (2356)≫≫
NICT-5 (2132)≫

Anuvaad (2408)

Anuvaad (2400)

NICT-5 (2111)

NICT-5 (2133)

NICT-5 (2109)

NICT-5 (2132)

IITP-MT (2356)

Table 19: Statistical significance testing of the indic-en-hi (left) and indic-hi-en (right) Pairwise scores.

IITP-MT (2349) ≫ ≫ ≫ ≫NICT-5 (2133)≫≫≫
NICT-5 (2111)≫≫
Anuvaad (2400)-

Table 20: Statistical significance testing of the indic-en-ta (left) and indic-ta-en (right) Pairwise scores.

933
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

aspec-ja-en
SYSTEMDATA κORGANIZER 0006 0.216
TMU
2464 0.201
2474 0.183
srcb
Osaka-U
2440 0.128
2472 0.130
Osaka-U
NICT-52174 0.182
2273 0.145
NICT-5ave.
0.169

aspec-en-ja
SYSTEMDATA κORGANIZER 0005 0.394
TMU2469 0.4502245 0.314
EHR
srcb
2479 0.325
2439 0.302
Osaka-U
Osaka-U
2470 0.305
2048 0.324
NICT-5
NICT-52219 0.256
ave.
0.334

aspec-ja-zh
SYSTEMDATA κORGANIZER 0007 0.254
srcb
2473 0.150
NICT-52175 0.162
NICT-52266 0.174
ave.
0.185

aspec-zh-ja
SYSTEMDATA κORGANIZER 0008 0.389
NICT-52052 0.266
NICT-52267 0.282
ave.
0.312

iitb-en-hi
SYSTEMDATA κ
CUNI
2362 0.358
CUNI
2365 0.454
cvit-mt 2251 0.447
cvit-mt 2254 0.356
ave.
0.404

iitb-hi-en
SYSTEMDATA κ
CUNI
2381 0.404
cvit-mt 2331 0.381
0.393
ave.

alt-en-my
alt-my-en
SYSTEM DATA κ SYSTEM DATA κ
NICT2282 0.181 NICT
2281 0.107
NICT2345 0.091 NICT2329 0.202
Osaka-U 2437 0.061 Osaka-U 2438 0.153
Osaka-U 2471 0.187 Osaka-U 2463 0.161
2087 0.205 NICT-4
2069 0.284
NICT-4
NICT-42287 0.262 NICT-4
2290 0.122
UCSYNLP 2339 0.268 NICT-5
2056 0.072
UCSYNLP 2340 0.303 UCSYNLP 2332 0.068
UCSMNLP 2337 0.212 UCSMNLP 2338 0.087
kmust882360 0.275 ave.
0.140
ave.
0.205

indic-en-hi
SYSTEMDATA κ
IITP-MT 2354 0.330
RGNLP 2417 0.386
RGNLP 2422 0.417
NICT-5 2067 0.447
NICT-5 2128 0.341
Anuvaad 2445 0.437
ave.
0.393

indic-hi-en
SYSTEMDATA κ
IITP-MT 2347 0.204
RGNLP 2367 0.252
RGNLP 2383 0.411
NICT-5 2066 0.327
NICT-5 2129 0.263
Anuvaad 2403 0.441
Anuvaad 2406 0.281
ave.
0.311

indic-en-ta
SYSTEMDATA κ
IITP-MT 2356 0.209
NICT-5 2109 0.373
NICT-5 2132 0.443
Anuvaad 2443 0.308
ave.
0.333

indic-ta-en
SYSTEMDATA κ
IITP-MT 2349 0.299
NICT-5 2111 0.256
NICT-5 2133 0.185
Anuvaad 2400 0.159
Anuvaad 2408 0.139
ave.
0.208

Table 21: The Fleiss’ kappa values for the pairwise evaluation results.

934
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

NMT
NICT-5 (1)srcb
NICT-5 (2)Osaka-U (1)EHR
TMUOsaka-U (2)System

1900
2219
2479
2048
2439
2245
2469
2470

ID

NMT
NMT
NMT
NMT
NMT
NMT
NMT
SMT

Type
juman
36.37
42.87
42.49
41.91
38.01
37.97
35.08
23.24

SystemNMT
NICT-5 (1)NICT-5 (2)srcb
TMU
Osaka-U (1)Osaka-U (2)

NO
NO
NO
NO
YES
NO
NO
NO

RSRC

ID
1901
2174
2273
2474
2464
2440
2472

BLEU
kytea
38.48
44.42
44.11
43.50
40.00
40.00
37.69
25.50
juman
0.824985
0.8471340.850318
0.840776
0.825061
0.828746
0.823653
0.716889

RIBES
kytea
0.831183
0.849399
0.8522090.845042
0.829328
0.833333
0.829156
0.726469

935
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

RSRC
NO
NO
NO
NO
NO
YES
NO

BLEU
26.91
28.63
29.65
30.59
25.85
26.19
13.97

RIBES
0.764968
0.765933
0.774788
0.777896
0.761450
0.749825
0.665391

Table 23: ASPEC ja-en submissions

Type
NMT
NMT
NMT
NMT
NMT
NMT
SMT

Pair
–
+15.75+11.50+5.75-20.00-37.00-95.75juman
0.759910
0.779560
0.781000
0.771400
0.763140
0.758750
0.753040
0.705050

AMFM
0.595370
0.608070
0.612060
0.619390
0.600730
0.588290
0.571400

mecab
0.833207
0.853634
0.857017
0.849326
0.8332000.837806
0.831219
0.729323

Table 22: ASPEC en-ja submissions

mecab
37.15
43.49
43.20
42.60
39.10
38.66
36.14
24.26

Adeq
–
4.49
–
4.50
3.94
–
–

AMFM
kytea
0.759910
0.779560
0.781000
0.771400
0.763140
0.758750
0.753040
0.705050
mecab
0.759910
0.779560
0.781000
0.771400
0.763140
0.758750
0.753040
0.705050–+28.50+25.00+20.25+4.50-0.50-12.00-82.25

Pair

–
4.36
4.17
–
3.95
–
–
–

Adeq

PACLIC 32 - WAT 2018

NMT
srcb
NICT-5 (1)NICT-5 (2)SystemNMT
NICT-5 (1)NICT-5 (2)System

1903
2473
2266
2175

ID

1902
2267
2052

ID

NMT
NMT
NMT
NMT

Type

NMT
NMT
NMT

Type

NO
NO
NO
NO

RSRC

NO
NO
NO

RSRC

33.26
37.60
35.99
35.71

kytea

juman
43.31
49.79
48.43

BLEU
stanford
(ctb)
33.33
37.34
35.89
35.67

BLEU
kytea
43.53
50.66
48.78
juman
0.870734
0.889674
0.884426

RIBES
kytea
0.866281
0.886490
0.879456
mecab
0.870886
0.889853
0.884782

0.844322
0.859132
0.851382
0.851890

kytea

RIBES
stanford(ctb)
0.844572
0.858042
0.851416
0.850699

0.777600
0.7911200.781410
0.785440

Pair–+14.00
+7.00
+5.25

–
4.42
4.47
–

Adeq

–
4.63
–

Adeq

Pair

–
+22.75+11.00stanford(pku)
0.777600
0.7911200.781410
0.785440

mecab
0.782100
0.804920
0.800670

AMFM
stanford(ctb)
0.777600
0.7911200.781410
0.785440

AMFM
kytea
0.782100
0.804920
0.800670

kytea

juman
0.782100
0.804920
0.800670

stanford(pku)
0.844959
0.8581620.850944
0.850580

Table 25: ASPEC ja-zh submissions

stanford
(pku)
33.14
37.35
35.87
35.55

Table 24: ASPEC zh-ja submissions

mecab
43.34
49.89
48.52

PACLIC 32 - WAT 2018936
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

System

NMT
EHR
NMT
EHR

Task

N1

N2

N2

NMT
EHR
NMT
EHR

System

N1

Task

1966
2215
1947
2216

ID

1964
2476
1936
2477

ID

NMT
NMT
NMT
NMT

Type

NMT
NMT
NMT
NMT

Type

NO
NO
NO
NO

RSRC

NO
YES
NO
YES

RSRC

juman
71.42
71.45
70.65
73.29

juman
43.84
48.03
38.51
42.12

mecab
43.70
47.86
38.47
42.06

juman
0.860702
0.872828
0.825565
0.840713

RIBES
kytea
0.8574220.870332
0.824420
0.839052

937
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

mecab
71.68
71.74
70.94
73.61

juman
0.945593
0.945771
0.942943
0.953524

RIBES
kytea
0.944644
0.944284
0.942543
0.952935

Table 27: JPC N1/N2 ko-ja submissions

BLEU
kytea
72.21
72.35
71.46
74.25

Table 26: JPC N1/N2 en-ja submissions

BLEU
kytea
45.28
49.24
40.60
43.76

mecab
0.945126
0.945497
0.943101
0.953640

mecab
0.859818
0.872442
0.824770
0.841133juman
0.868760
0.869540––

juman
0.744270
0.759120
–
–

AMFM
kytea
0.868760
0.869540
–
–

AMFM
kytea
0.744270
0.759120––mecab
0.868760
0.869540––

mecab
0.744270
0.759120
–
–

–
4.89
–
4.73

Adeq

–
4.76
–
4.36

Adeq

PACLIC 32 - WAT 2018

NMT
USTC
EHR
NMT
USTC
EHR

System

NMT
USTC
NMT
USTC

N1

Task

N1

N2

N2

System

Task

1960
2202
1961
2203

ID

1963
2206
2210
1941
2207
2211

ID

NMT
NMT
NMT
NMT

Type

NMT
NMT
NMT
NMT
NMT
NMT

Type

NO
NO
NO
NO

RSRC

NO
NO
NO
NO
NO
NO

RSRC

39.07
39.71
39.14
39.91

kytea

juman
46.32
48.37
48.10
45.33
47.72
47.12

mecab
46.11
48.57
47.96
45.49
48.24
47.14

juman
0.857318
0.866232
0.858259
0.857120
0.866873
0.861697

RIBES
kytea
0.855085
0.864284
0.855649
0.854593
0.865270
0.859213

938
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

stanford
(pku)
39.75
40.05
39.77
40.32

0.847112
0.846472
0.847486
0.853978

kytea

RIBES
stanford(ctb)
0.850851
0.850750
0.852161
0.859330

Table 29: JPC N1/N2 ja-zh submissions

BLEU
stanford
(ctb)
40.32
40.54
40.28
40.53

Table 28: JPC N1/N2 zh-ja submissions

BLEU
kytea
46.73
49.78
48.51
46.05
49.45
47.71

stanford(pku)
0.850913
0.849818
0.850707
0.857456

mecab
0.856442
0.865423
0.858142
0.857052
0.866705
0.861437

0.752360
0.757690
–
–

kytea

juman
0.761820
0.771310
0.764670
–
–
–

AMFM
stanford(ctb)
0.752360
0.757690
–
–

AMFM
kytea
0.761820
0.771310
0.764670–––

stanford(pku)
0.752360
0.757690––

mecab
0.761820
0.771310
0.764670
–
–
–

–
4.56
–
4.61

Adeq

4.52
4.22
–
4.46
4.22

Adeq

PACLIC 32 - WAT 2018

SystemNMT
cvit-mt
CUNI

SystemNMT
CUNI (1)cvit-mt (1)
CUNI (2)cvit-mt (2)

ID
2567
2331
2381

ID
2566
2362
2254
2365
2251

Type
NMT
NMT
NMT

Type
NMT
NMT
NMT
NMT
NMT

BLEU
13.76
17.63
19.69
20.07
16.77

RIBES
0.710210
0.753895
0.758365
0.761582
0.714197

BLEU
15.44
20.63
17.80

RIBES
0.718751
0.751883
0.731727

AMFM
0.644860
0.693830
0.699810
0.701300
0.664330

AMFM
0.586360
0.623240
0.611090

Table 31: IITB hi-en submissions

RSRC
NO
YES
NO

Table 30: IITB en-hi submissions

RSRC
NO
NO
YES
NO
NO

Pair
–
+72.25+67.25Pair–+77.00+69.50+60.00+50.50

Adeq
–
2.88
2.96

Adeq
–
3.49
3.20
–
–

PACLIC 32 - WAT 2018939
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

Type
Other
NMT
NMT
NMT
Other
NMT
NMT
NMT
NMT
NMT
SMT

SystemOnline A
NMT
NICT-4 (1)NICT (1)
NICT-4 (2)NICT (2)NICT-5UCSYNLP
Osaka-U (1)Osaka-U (2)UCSMNLP

ID
2141
2228
2069
2329
2290
2281
2056
2332
2438
2463
2338

ID
2142
2143
2227
2345
2087
2282
2287
2339
2360
2437
2340
2471
2337

SystemOnline-A
Online-A (comma→0x104a)
NMT
NICT (1)
NICT-4 (1)NICT (2)NICT-4 (2)UCSYNLP (1)kmust88
Osaka-U (1)UCSYNLP (2)Osaka-U (2)UCSMNLP

RSRC
YES
YES
NO
NO
NO
NO
NO
NO
NO
YES
NO
NO
NO

BLEU
20.31
20.83
22.42
29.89
29.57
26.02
30.52
21.19
19.34
22.33
19.19
20.88
8.16

940
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

BLEU
14.24
14.44
21.97
20.82
22.53
16.31
15.44
9.56
11.38
9.99
2.22

RIBES
0.598345
0.696861
0.753209
0.740819
0.753767
0.710528
0.717430
0.642309
0.655643
0.648923
0.470280
Table 33: ALT my-en submissions

RSRC
YES
NO
NO
NO
NO
NO
NO
NO
YES
NO
NO

Pair
–
–
+22.25+20.50+16.00+7.25-6.50-37.50-57.00-61.00-99.50AMFM
0.5871200.594230
0.7455500.800230
0.803810
0.785920
0.809750
0.756710
0.721280
0.740760
0.717480
0.774750
0.222510

AMFM
0.576780
0.525950
0.586770
0.580690
0.582230
0.589020
0.579520
0.518990
0.510900
0.552040
0.354550

RIBES
0.678360
0.679968
0.667437
0.726922
0.738538
0.694652
0.733501
0.679800
0.650796
0.668596
0.671461
0.639517
0.470758

Table 32: ALT en-my submissions

Type
Other
Other
NMT
NMT
NMT
NMT
Other
NMT
NMT
NMT
NMT
SMT
SMT

Adeq
–
–
3.89
4.04
–
–
–
2.40
–
–
–

Pair
–
–
–
+61.00+53.00+42.50+39.75+10.50
+9.75
+3.00
+0.75
-23.50-96.75

Adeq
–
–
–
3.78
3.65
–
–
2.47
–
–
–
–
–

PACLIC 32 - WAT 2018

ID
2003
2146
2188
2128
2354
2417
2067
2445
2422

ID
2004
2099
2189
2347
2129
2367
2066
2406
2403
2383

SystemNMT
NMT O2M
NMT M2M
NICT-5 (1)IITP-MTRGNLP (1)
NICT-5 (2)AnuvaadRGNLP (2)SystemNMT
NMT M2ONMT M2M
IITP-MT
NICT-5 (1)RGNLP (1)
NICT-5 (2)Anuvaad (1)Anuvaad (2)RGNLP (2)

Type
NMT
NMT
NMT
NMT
NMT
SMT
NMT
SMT
SMT
NMT

Type
NMT
NMT
NMT
NMT
NMT
SMT
NMT
SMT
NMT

BLEU
20.78
23.27
22.24
26.59
26.60
44.08
29.65
26.49
22.50

RIBES
0.682944
0.709586
0.705747
0.734027
0.722756
0.751187
0.721379
0.692385
0.678207

941
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

BLEU
21.15
26.71
26.55
32.95
30.21
21.54
31.06
22.45
25.57
21.86

RIBES
0.752783
0.787645
0.784968
0.803497
0.801291
0.697379
0.786655
0.709235
0.720866
0.751517
Table 35: Indic hi-en submissions

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

Table 34: Indic en-hi submissions

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO

AMFM
0.559420
0.586760
0.577010
0.629890
0.617860
0.599760
0.599000
0.558850
0.599880
0.573160

AMFM
0.574960
0.601800
0.604040
0.636900
0.615620
0.698550
0.636500
0.6571800.584720

Pair
–
–
–
+40.75+32.00+22.25+13.50
+5.75
+0.75
-1.25

Pair
–
–
–
+30.75+20.50+15.50+13.75+11.00
-0.25

Adeq
–
–
–
1.98
2.00
1.46
–
–
–
–

Adeq
–
–
–
1.73
1.77
1.45
–
–
–

PACLIC 32 - WAT 2018

ID
2007
2148
2192
2443
2356
2132
2109

ID
2008
2101
2193
2349
2133
2111
2400
2408

SystemNMT
NMT O2M
NMT M2M
AnuvaadIITP-MT
NICT-5 (1)NICT-5 (2)SystemNMT
NMT M2ONMT M2M
IITP-MT
NICT-5 (1)NICT-5 (2)Anuvaad (1)Anuvaad (2)

Type
NMT
NMT
NMT
NMT
NMT
SMT
SMT
SMT

Type
NMT
NMT
NMT
SMT
NMT
NMT
NMT

BLEU
7.12
16.05
15.41
15.87
18.81
20.39
18.60

RIBES
0.457948
0.651935
0.664354
0.668548
0.658740
0.690652
0.649454

942
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

BLEU
9.14
19.71
18.59
22.42
24.31
21.37
14.34
14.09

RIBES
0.649417
0.751277
0.744884
0.757610
0.768865
0.744744
0.671535
0.673058
Table 37: Indic ta-en submissions

RSRC
NO
NO
NO
NO
NO
NO
NO
NO

Table 36: Indic en-ta submissions

RSRC
NO
NO
NO
NO
NO
NO
NO

AMFM
0.488060
0.568020
0.561550
0.604300
0.593410
0.552630
0.5111300.487250

AMFM
0.545370
0.706760
0.711080
0.756890
0.710610
0.736930
0.700960

Pair
–
–
–
+75.25+63.25+47.75+29.75+29.25Pair–––+73.75+66.50+60.50+45.25

Adeq
–
–
–
1.82
1.66
–
1.09
–

Adeq
–
–
–
1.16
1.33
1.12
–

PACLIC 32 - WAT 2018

PACLIC 32 - WAT 2018

References
Rafael E. Banchs, Luis F. D’Haro, and Haizhou Li.
2015. Adequacy-fluency metrics: Evaluating mt in
the continuous space model framework. IEEE/ACM
Trans. Audio, Speech and Lang. Proc., 23(3):472–482,
March.
Tamali Banerjee, Anoop Kunchukuttan, and Pushpak
Bhattacharyya. 2018. Multilingual Indian Language Translation System at WAT 2018: Many-to-one
Phrase-based SMT. In Proceedings of the 5th Workshop on Asian Translation (WAT2018), Hong Kong,
China, December.
Jacob Cohen. 1968. Weighted kappa: Nominal scaleagreement with provision for scaled disagreement or
partial credit. Psychological Bulletin, 70(4):213 – 220.
Raj Dabre, Anoop Kunchukuttan, Atsushi Fujita, and Eiichiro Sumita. 2018. NICT’s Participation in WAT
2018: Approaches Using Multilingualism and Recurrently. In Proceedings of the 5th Workshop on AsianTranslation (WAT2018), Hong Kong, China, December.
Terumasa Ehara. 2018. SMT reranked NMT (2). In
Proceedings of the 5th Workshop on Asian Translation(WAT2018), Hong Kong, China, December.
J.L. Fleiss et al. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin,
76(5):378–382.
Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs.
In Proceedings of the 2010 Conference on Empirical
Methods in Natural Language Processing, EMNLP’10, pages 944–952, Stroudsburg, PA, USA. Association for Computational Linguistics.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2017. Google’s
multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.
Yuki Kawara, Yuto Takebayashi, Chenhui Chu, and Yuki
Arase. 2018. Osaka University MT Systems for WAT
2018: Rewarding, Preordering, and Domain Adaptation. In Proceedings of the 5th Workshop on AsianTranslation (WAT2018), Hong Kong, China, December.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Opensource toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages
67–72. Association for Computational Linguistics.

Tom Kocmi, Shantipriya Parida, and Ondrej Bojar. 2018.
CUNI NMT System for WAT 2018 Translation Tasks.
In Proceedings of the 5th Workshop on Asian Translation (WAT2018), Hong Kong, China, December.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran, Richard
Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source
toolkit for statistical machine translation. In Annual
Meeting of the Association for Computational Linguistics (ACL), demonstration session.
Philipp Koehn. 2004. Statistical significance tests formachine translation evaluation. In Dekang Lin andDekai Wu, editors, Proceedings of EMNLP 2004,
pages 388–395, Barcelona, Spain, July. Association
for Computational Linguistics.
T. Kudo.
2005.
Mecab :
Yet anotherpart-of-speechandmorphologicalanalyzer.
http://mecab.sourceforge.net/.
Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto,and Makoto Nagao. 1994. Improvements of Japanesemorphological analyzer JUMAN. In Proceedings of
The International Workshop on Sharable Natural Language, pages 22–28.
Yihan Li, Boyan Liu, Yixuan Tong, Shanshan Jiang, and
Bin Dong. 2018. SRCB Neural Machine Translation Systems in WAT 2018. In Proceedings of the
5th Workshop on Asian Translation (WAT2018), Hong
Kong, China, December.
Benjamin Marie, Atsushi Fujita, and Eiichiro Sumita.
2018. Combination of Statistical and Neural MachineTranslation for Myanmar-English. In Proceedings of
the 5th Workshop on Asian Translation (WAT2018),
Hong Kong, China, December.
Yukio Matsumura, Satoru Katsumata, and Mamoru Komachi. 2018. TMU Japanese-English Neural MachineTranslation System using Generative Adversarial Network for WAT 2018. In Proceedings of the 5th Workshop on Asian Translation (WAT2018), Hong Kong,
China, December.
Hsu Myat Mo, Yi Mon Shwe Sin, Thazin Myint Oo,
Win Pa Pa, Khin Mar Soe, and Ye Kyaw Thu.
2018. UCSYNLP-Lab Machine Translation Systems
for WAT 2018. In Proceedings of the 5th Workshop
on Asian Translation (WAT2018), Hong Kong, China,
December.
Toshiaki Nakazawa, Hideya Mino, Isao Goto, Sadao
Kurohashi, and Eiichiro Sumita. 2014. Overview
of the 1st Workshop on Asian Translation. In Proceedings of the 1st Workshop on Asian Translation(WAT2014), pages 1–19, Tokyo, Japan, October.
Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham
Neubig, Sadao Kurohashi, and Eiichiro Sumita. 2015.

943
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors

PACLIC 32 - WAT 2018

Overview of the 2nd Workshop on Asian Translation.
In Proceedings of the 2nd Workshop on Asian Translation (WAT2015), pages 1–28, Kyoto, Japan, October.
Toshiaki Nakazawa, Chenchen Ding, Hideya MINO, Isao
Goto, Graham Neubig, and Sadao Kurohashi. 2016.
Overview of the 3rd workshop on asian translation. In
Proceedings of the 3rd Workshop on Asian Translation(WAT2016), pages 1–46, Osaka, Japan, December. The
COLING 2016 Organizing Committee.
Toshiaki Nakazawa, Shohei Higashiyama, Chenchen
Ding, Hideya Mino, Isao Goto, Hideto Kazawa,
Yusuke Oda, Graham Neubig, and Sadao Kurohashi.
2017. Overview of the 4th workshop on asian translation. In Proceedings of the 4th Workshop on AsianTranslation (WAT2017), pages 1–54. Asian Federationof Natural Language Processing.
Graham Neubig, Yosuke Nakata, and Shinsuke Mori.
2011. Pointwise prediction for robust, adaptablejapanese morphological analysis. In Proceedings ofthe 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 529–
533, Stroudsburg, PA, USA. Association for Computational Linguistics.
Atul Kr. Ojha, Koel Dutta Chowdhury, Chao-Hong Liu,and Karan Saxena. 2018. The RGNLP MachineTranslation Systems for WAT 2018. In Proceedingsof the 5th Workshop on Asian Translation (WAT2018),
Hong Kong, China, December.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318.
Jerin Philip, Vinay P. Namboodiri, and C V Jawahar.
2018. CVIT-MT Systems for WAT-2018. In Proceedings of the 5th Workshop on Asian Translation(WAT2018), Hong Kong, China, December.
Hammam Riza, Michael Purwoadi, Teduh Uliniansyah,
Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai,
Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea,
Sethserey Sam, Sopheap Seng, Khin Mar Soe, Khin
Thandar Nwet, Masao Utiyama, and Chenchen Ding.
2016. Introduction of the asian language treebank. In
In Proc. of O-COCOSDA, pages 1–6.
Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, and Pushpak Bhattacharyya. 2018. IITP-MT at WAT2018:
Transformer-based Multilingual Indic-English Neural
Machine Translation System. In Proceedings of the
5th Workshop on Asian Translation (WAT2018), Hong
Kong, China, December.
Aye Thida, Nway Nway Han, and Sheinn Thawtar Oo.
2018. Statistical Machine Translation Using 5-grams
Word Segmentation in Decoding. In Proceedings of
the 5th Workshop on Asian Translation (WAT2018),
Hong Kong, China, December.

Huihsin Tseng. 2005. A conditional random field word
segmenter. In In Fourth SIGHAN Workshop on Chinese Language Processing.
Masao Utiyama and Hitoshi Isahara. 2007. A japaneseenglish patent parallel corpus. In MT summit XI, pages
475–482.
Boli Wang, Jinming Hu, Yidong Chen, and Xiaodong
Shi. 2018a. XMU Neural Machine Translation
Systems for WAT2018 Myanmar-English Translation
Task. In Proceedings of the 5th Workshop on AsianTranslation (WAT2018), Hong Kong, China, December.
Rui Wang, Chenchen Ding, Masao Utiyama, and Eiichiro
Sumita. 2018b. English-Myanmar NMT and SMT
with Pre-ordering: NICT’s machine translation systems at WAT-2018. In Proceedings of the 5th Workshop on Asian Translation (WAT2018), Hong Kong,
China, December.
Yi Mon Shwe Sin and Khin Mar Soe. 2018. Syllablebased myanmar-english neural machine translation. In
In Proc. of ICCA, pages 228–233.
Longtu Zhang, Yuting Zhao, and Mamoru Komachi.
2018. TMU Japanese-Chinese Unsupervised NMT
System for WAT 2018 Translation Task. In Proceedings of the 5th Workshop on Asian Translation(WAT2018), Hong Kong, China, December.

944
32nd Pacific Asia Conference on Language, Information and Computation
The 5th Workshop on Asian Translation
Hong Kong, 1-3 December 2018
Copyright 2018 by the authors