Overview of the 6th Workshop on Asian Translation
Toshiaki Nakazawa
The University of Tokyo
nakazawa@logos.t.u-tokyo.ac.jp

Nobushige Doi
Japan Exchange Groupn-doi@jpx.co.jp

Shohei Higashiyama and Chenchen Ding and Raj Dabre
National Institute ofInformation and Communications Technology{shohei.higashiyama, chenchen.ding, raj.dabre}@nict.go.jp
Hideya Mino and Isao GotoNHK
{mino.h-gq, goto.i-es}@nhk.or.jp

Win Pa Pa
University of Conputer Study, Yangon
winpapa@ucsy.edu.mm

Anoop Kunchukuttan
Microsoft AI and Research

Shantipriya Parida
Idiap Research Instituteanoop.kunchukuttan@microsoft.com

shantipriya.parida@idiap.chOndřej Bojar
Charles University, MFF, ÚFALbojar@ufal.mff.cuni.cz

Sadao Kurohashi
Kyoto University
kuro@i.kyoto-u.ac.jp

Abstracttranslation researchers and users to try, evaluate, share and discuss brand-new ideas formachine translation. We have been working
toward practical use of machine translation
among all Asian countries.
For the 6th WAT, we adopted new translation subtasks with Khmer↔English andTamil↔English mixed domain corpora,2 Russian↔Japanese news commentary corpus andEnglish→Hindi multi-modal corpus3 in addition to most of the subtasks of WAT2018.
WAT is a unique workshop on Asian language translation with the following characteristics:

This paper presents the results of theshared tasks from the 6th workshop on
Asian translation (WAT2019) including
Ja↔En, Ja↔Zh scientific paper translation subtasks, Ja↔En, Ja↔Ko, Ja↔En
patent translation subtasks, Hi↔En,
My↔En, Km↔En, Ta↔En mixed domain
subtasks, Ru↔Ja news commentarytranslation task, and En→Hi multi-modaltranslation task. For the WAT2019, 25
teams participated in the shared tasks.
We also received 10 research paper submissions out of which 71 were accepted.
About 400 translation results were submitted to the automatic evaluation server,and selected submissions were manually
evaluated.

1• Open innovation platform
Due to the fixed and open test data, we
can repeatedly evaluate translation systems on the same dataset over years.
WAT receives submissions at any time;i.e., there is no submission deadline oftranslation results w.r.t automatic evaluation of translation quality.

Introduction

The Workshop on Asian Translation (WAT)
is an open evaluation campaign focusing on
Asian languages. Following the success ofthe previous workshops WAT2014-WAT2018(Nakazawa et al., 2014, 2015, 2016, 2017,2018), WAT2019 brings together machine2
http://lotus.kuee.kyoto-u.ac.jp/WAT/
my-en-data/3
https://ufal.mff.cuni.cz/hindi-visual-genome/
wat-2019-multimodal-task

1
One paper was withdrawn post acceptance andhence only 6 papers will be in the proceedings.

1
Proceedings of the 6th Workshop on Asian Translation, pages 1–35
Hong Kong, China, November 4, 2019. c 2019 Association for Computational Linguistics

Lang
JE
JC

Train
3,008,500
672,315

Dev
1,790
2,090

DevTest
1,784
2,148

Test
1,812
2,107

Table 1: Statistics for ASPEC

2.1

ASPEC

DevTest
2,000
2,000
2,000

Test-N
5,204
5,230
5,668

Lang
zh-ja
ko-ja
en-ja

Test-N1
2,000
2,000
2,000

Test-N2
3,000
3,000
3,000

Test-N3
204
230
668

Test-EP
1,151––

2.1.2 ASPEC-JCASPEC-JC is a parallel corpus consisting ofJapanese scientific papers, which come fromthe literature database and electronic journalsite J-STAGE by JST, and their translation to
Chinese with permission from the necessary
academic associations. Abstracts and paragraph units are selected from the body text
so as to contain the highest overall vocabulary
coverage.
The development, development-test andtest data are extracted at random from documents containing single paragraphs across theentire corpus. Each set contains 400 paragraphs (documents). There are no documents
sharing the same data across the training, development, development-test and test sets.

ASPEC was constructed by the Japan Scienceand Technology Agency (JST) in collaboration
with the National Institute of Information andCommunications Technology (NICT). The
corpus consists of a Japanese-English scientific paper abstract corpus (ASPEC-JE),
which is used for ja↔en subtasks, and aJapanese-Chinese scientific paper excerpt corpus (ASPEC-JC), which is used for ja↔zhsubtasks. The statistics for each corpus areshown in Table 1.
2.1.1

Dev
2,000
2,000
2,000

frequency and occurrence ratios for the training data, are described in the README fileof ASPEC-JE.
The development, development-test andtest data were extracted from parallel sentences from the Japanese-English paper abstracts that exclude the sentences in the training data. Each dataset consists of 400 documents and contains sentences in each field at
the same rate. The document alignment was
conducted automatically and only documentswith a 1-to-1 alignment are included. It istherefore possible to restore the original documents. The format is the same as the training
data except that there is no similarity score.

• Evaluation method
Evaluation is done both automaticallyand manually. Firstly, all submitted
translation results are automatically evaluated using three metrics: BLEU, RIBES
and AMFM. Among them, selected translation results are assessed by two kindsof human evaluation: pairwise evaluation
and JPO adequacy evaluation.

Datasets

Train
1,000,000
1,000,000
1,000,000

Table 2: Statistics for JPC• Domain and language pairs
WAT is the world’s first workshop thattargets scientific paper domain, and Chinese↔Japanese and Korean↔Japanese
language pairs. In the future, we will
add more Asian languages such as Vietnamese, Thai and so on.

2

Lang
zh-ja
ko-ja
en-ja

ASPEC-JE

The training data for ASPEC-JE was constructed by NICT from approximately two
million Japanese-English scientific paper abstracts owned by JST. The data is a comparable corpus and sentence correspondences arefound automatically using the method from
Utiyama and Isahara (2007). Each sentencepair is accompanied by a similarity score calculated by the method and a field ID that indicates a scientific field. The correspondence between field IDs and field names, along with the

2.2 JPC
JPO Patent Corpus (JPC) for the patenttasks was constructed by the Japan Patent Office (JPO) in collaboration with NICT. The
corpus consists of Chinese-Japanese, KoreanJapanese and English-Japanese patent descriptions whose International Patent Classi2

Disclosure
Period2016-01-01 to
2017-12-31
2018-01-01 to
2018-06-30

Train
1,089,346(614,817)
314,649(218,495)

Dev
Texts
Items
1,153
2,845
(1,148)(2,650)DevTest
Texts
Items
1,114
2,900
(1,111)(2,671)

Test
Texts
Items
1,153
2,129
(1,135)(1,763)

Table 3: Statistics for TDDC (The number of unique sentences)

fication (IPC) sections are chemistry, electricity, mechanical engineering, and physics.
At WAT2019, the patent tasks has two subtasks: normal subtask and expression pattern
subtask. Both subtasks use common training, development and development-test data
for each language pair. The normal subtask forthree language pairs uses four test data withdifferent characteristics:

to avoid mistranslations that would confuse
investors.
TSE is one of the largest capital markets
in the world that has over 3,600 companies
listed as of the end of 2018. Companies
are required to disclose material information
including financial statements, corporate actions, and corporate governance policies tothe public in a timely manner. These timely
disclosure documents form an important basis for investment decisions, containing important figures (e.g., sales, profits, significant
dates) and proper nouns (e.g., names of persons, places, companies, business and product). Since such information is critical for investors, mistranslations should be avoided andtranslations should be of a high quality.

• test-N: union of the following three sets;• test-N1: patent documents from patentfamilies published between 2011 and2013;• test-N2: patent documents from patentfamilies published between 2016 and2017; andThe corpus consists of Japanese-English
sentence pairs, document hashes, and sentencehashes. A document hash is a hash of the Document ID, which is a unique identifier of thesource document. A sentence hash is a hash of
the Document ID and the Sentence ID, which
is a unique identifier of the sentence in eachsource document.

• test-N3: patent documents published between 2016 and 2017 where target sentences are manually created by translating source sentences.
The expression pattern subtask for zh→ja pairuses test-EP data. The test-EP data consistsof sentences annotated with expression pattern categories: title of invention (TIT), abstract (ABS), scope of claim (CLM) or description (DES). The corpus statistics are shownin Table 2. Note that training, development,
development-test and test-N1 data are thesame as those used in WAT2017.
2.3

The corpus is partitioned into training, development, development-test, and test data.
The training data is split into two (2) setsof data from different periods. The first data
set was created based on documents disclosed
from January 1, 2016 to December 31, 2017,
and the second data set was based on documents from January 1, 2018 to June 30, 2018.
The development, development-test, and test
data set were extracted from timely disclosure
documents disclosed from January 1, 2018
to June 30, 2018, excluding documents thatwere used to create the training data. The
documents for the period were randomly selected, and the sentences were extracted fromeach randomly selected, discrete document set
so that the sources extracted are not biased.
Therefore, the set of source documents fortraining, development, development-test and

TDDC

Timely Disclosure Documents Corpus(TDDC) was constructed by Japan ExchangeGroup (JPX). The corpus was made byaligning the sentences manually from past
Japanese and English timely disclosure documents in PDF format published by companies
listed on Tokyo Stock Exchange (TSE).
Timely Disclosure tasks focus on Japanese
to English translation of sentences extracted
from timely disclosure documents in order
3

Lang
en-ja

Train
200,000

Dev
2,000

DevTest
2,000

Test
2,000

Corpus
ALT
UCSY
All

Table 4: Statistics for JIJI Corpus
Lang
hi-en
hi-ja
hi

Train
1,492,827
152,692
–

Dev
520
1,566
–

Test
2,507
2,000
–

Mono
–
–
45,075,279• The ALT corpus is one part from the
Asian Language Treebank (ALT) project(Riza et al., 2016), consisting of twenty
thousand Myanmar-English parallel sentences from news articles.
• The UCSY corpus (Yi Mon Shwe Sin andKhin Mar Soe, 2018) is constructed by the
NLP Lab, University of Computer Studies, Yangon (UCSY), Myanmar. The corpus consists of 200 thousand MyanmarEnglish parallel sentences collected fromdifferent domains, including news articles
and textbooks.

JIJI Corpus

JIJI Corpus was constructed by Jiji Press Ltd.
in collaboration with NICT. The corpus consists of news text that comes from Jiji Press
news of various categories including politics,
economy, nation, business, markets, sports
and so on. The corpus is partitioned intotraining, development, development-test andtest data, which consists of Japanese-English
sentence pairs. The statistics for each corpus
are shown in Table 4.
The sentence pairs in each data are identified in the same manner as that for ASPEC
using the method from (Utiyama and Isahara,2007).

The ALT corpus has been manually segmented into words (Ding et al., 2018, 2019),
and the UCSY corpus is unsegmented. A
script to tokenize the Myanmar data into writing units is released with the data. The automatic evaluation of Myanmar translation results is based on the tokenized writing units,
regardless to the segmented words in the ALT
data. However, participants can make a useof the segmentation in ALT data in their own
manner.
The detailed composition of training, development, and test data of the MyanmarEnglish translation tasks are listed in Table 6.
Notice that both of the corpora have been
modified from the data used in WAT2018.

IITB Corpus

IIT Bombay English-Hindi Corpus contains
English-Hindi parallel corpus as well as monolingual Hindi corpus collected from a variety
of sources and corpora. This corpus had beendeveloped at the Center for Indian Language
Technology, IIT Bombay over the years. The
corpus is used for mixed domain tasks hi↔en.
The statistics for the corpus are shown in Table 5.
2.6

Test
1,018
–
1,018

pora, the ALT corpus and UCSY corpus.

test data are independent of each other. Furthermore, each data set of the development,
development-test, and test is further split into
two (2) sets of data: sentences that end with aJapanese period (。: U+3002) are classified as’Texts’, which has various sentences, and others are classified as ’Items’, which has many
duplicates and similar expressions. The statistics for each corpus are shown in Table 3.

2.5

Dev
1,000
–
1,000

Table 6: Statistics for the data used in MyanmarEnglish translation tasks

Table 5: Statistics for IITB Corpus. “Mono” indicates monolingual Hindi corpus.

2.4

Train
18,088204,539
222,627

2.7 ALT and ECCC Corpus
The parallel data for Khmer-English translation tasks at WAT2019 consists of two corpora,
the ALT corpus and ECCC corpus.
• The ALT corpus is one part from the
Asian Language Treebank (ALT) project(Riza et al., 2016), consisting of twenty
thousand Khmer-English parallel sentences from news articles.

ALT and UCSY Corpus

The parallel data for Myanmar-English translation tasks at WAT2019 consists of two cor4

Corpus
ALT
ECCC
All

Train
18,088104,660
122,748

Dev
1,000
–
1,000

Test
1,018
–
1,018

DatasetTraining Set
D-Test
E-Test (EV)C-Test (CH)

Table 7: Statistics for the data used in KhmerEnglish translation tasks

TokensEnglishHindi
143,178 136,722
4,922
4,695
7,852
7,535
8,185
8,665

Table 8: Data for the English→Hindi multi-modaltranslation task. One item consists of source English sentence, target Hindi sentence, and a rectangular region within an image. The total number ofEnglish and Hindi tokens in the dataset also listed.
The abbreviations EV and CH are used in the official task names in WAT scoring tables.

• The ECCC corpus consists of 100 thousand Khmer-English parallel sentences
extracted from document pairs of KhmerEnglish bi-lingual records in Extraordinary Chambers in the Court of Cambodia, collected by National Institute ofPosts, Telecoms & ICT, Cambodia.

was distributed to task participants and thetarget side was published only after output
submission deadline.
Note that the original Visual Genome suffers from a considerable level of noise. Someobserved English grammar errors are illustrated in Figure 1. We also took the chanceand used our manual evaluation for validating
the quality of the captions given the picture,see 8.4.1 below.
The multi-modal task includes three tracks
as illustrated in Figure 1:

The ALT corpus has been manually segmented into words (Ding et al., 2018), and the
ECCC corpus is unsegmented. A script to tokenize the Khmer data into writing units is released with with the data. The automatic evaluation of Khmer translation results is based on
the tokenized writing units, regardless to thesegmented words in the ALT data. However,participants can make a use of the segmentation in ALT data in their own manner.
The detailed composition of training, development, and test data of the Khmer-English
translation tasks are listed in Table 7.
2.8

Items
28,932
998
1,595
1,400

2.8.1 Multi-Modal Task Tracks
1. Text-Only Translation (labeled “TEXT”
in WAT official tables): The participants
are asked to translate short English captions (text) into Hindi. No visual information can be used. On the other hand,additional text resources are permitted(but they need to be specified in the corresponding system description paper).

Multi-Modal Task Corpus

For English→Hindi multi-modal translation
task we asked the participants to use the Hindi
Visual Genome corpus (HVG, Parida et al.,2019a,b). The statistics of the corpus are given
in Table 8. One “item” in the original HVG
consists of an image with a rectangular region
highlighting a part of the image, the original
English caption of this region and the Hindi
reference translation. Depending on the track(see 2.8.1 below), some of these item components are available as the source and someserve as the reference or play the role of a competing candidate solution.
HVG Training, D-Test and E-Test sections
were accessible to the participants in advance.
The participants were explicitly instructed not
to consult E-Test in any way but strictly
speaking, they could have used the referencetranslation (which would mean cheating fromthe evaluation point of view). C-Test was provided only for the task itself: the source side

2. Hindi Captioning (labeled “HI”): The
participants are asked to generate captions in Hindi for the given rectangularregion in an input image.
3. Multi-Modal
Translation(labeled“MM”): Given an image, a rectangular region in it and an English caption
for the rectangular region, the participants are asked to translate the English
text into Hindi. Both textual and visual
information can be used.

5

Text-Only MT

Hindi Captioning

Multi-Modal MT

Image
Source Text–
the bird is stand on a tree
branch–man stand on skateboard

System Output
Glossच ड़या एक पेड़ शाखा पर है
Bird on a branch of treeलाल और सफेद च
Red and white signकेटबोड पर मनु य टे पग
Man stepping on skateboard

Reference Solutionप ी एक पेड़ क शाखा पर खड़ा हैस केत अं ेजी और वदेशी भाषा म लखेगये हA sign is written in English
and a foreign languageआदमी केटबोड पर खड़ा हैGloss

A bird standing on the
branch of a tree

A man is standing on askateboard

Figure 1: An illustration of the three tracks of WAT 2019 Multi-Modal Task. Note the grammaticalerrors in the English source. The correct sentences would be “The bird is standing on a tree branch.”and “A man is standing on a skateboard.”
DatasetSentences

train
test
development
total
Domain166,871
2,000
1,000
169,871
Sentencesbible
cinemanews
total

26,792 (15.77%)
30,242 (17.80%)
112,837 (66.43%)
169,871

Englishtokens3,913,541
47,14423,353
3,984,038
Englishtokens703,838
445,230
2,834,970
3,984,038

Lang.pairTamiltokens2,727,174
32,847
16,376
2,776,397
Tamil
tokens373,082
298,419
2,104,896
2,776,397

Ja↔Ru

Ja↔En

Ru↔En

#tokens
341k / 229k
16k / 11k
22k / 15k
1.27M / 1.01M
21k / 16k
22k / 17k
1.61M / 1.83M
7.8k / 8.4k
15k / 17k

#types
22k / 42k
2.9k / 4.3k
3.5k / 5.6k
48k / 55k
3.5k / 3.8k
3.5k / 3.8k
144k / 74k
3.2k / 2.3k
5.6k / 3.8ktranslation quality as well but this is beyond
the scope of this years sub-task. Refer toTable 10 for the statistics of the in-domain
parallel corpora. In addition we encouraged
the participants to use out-of-domain parallel
corpora from various sources such as KFTT,6
JESC,7 TED,8 ASPEC,9 UN,10 Yandex11 andRussian↔English news-commentary corpus.12

EnTam Corpus

For Tamil↔English translation task we asked
the participants to use the publicly available
EnTam mixed domain corpus4 (Ramasamy
et al., 2012). This corpus contains training,development and test sentences mostly fromthe news-domain. The other domains are
Bible and Cinema. The statistics of the corpus
are given in Table 9.
2.10#sent.
12,356486
600
47,082589
600
82,072313
600

Table 10:
In-Domain data for the Russian–
Japanese task.

Table 9: Data for the Tamil↔English task.

2.9

Partition
train
development
test
train
development
test
train
development
test

3 Baseline Systems
Human evaluations of most of WAT tasks wereconducted as pairwise comparisons between
the translation results for a specific baseline
system and translation results for each partic-

JaRuNC Corpus

For the Russian↔Japanese task we asked
participants to use the JaRuNC corpus5(Imankulova et al., 2019) which belongs to thenews commentary domain. This dataset was
manually aligned and cleaned and is trilingual.
It can be used to evaluate Russian↔English

6

http://www.phontron.com/kftt/
https://datarepository.wolframcloud.com/
resources/Japanese-English-Subtitle-Corpus
8
https://wit3.fbk.eu/
9
http://lotus.kuee.kyoto-u.ac.jp/ASPEC/10https://cms.unov.org/UNCorpus/
11
https://translate.yandex.ru/corpus?lang=en12
http://lotus.kuee.kyoto-u.ac.jp/WAT/
News-Commentary/news-commentary-v14.en-ru.
filtered.tar.gz
74

http://ufal.mff.cuni.cz/~ramasamy/parallel/
html/
5
https://github.com/aizhanti/JaRuNC

6

Training Data

✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓

✓
✓

✓
✓

✓
✓

✓
✓

✓
✓
✓

✓
✓
✓
✓
✓
✓
✓

✓
✓
✓
✓

SystemOpenNMT’s NMT with attention
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
ATLAS V14 (Commercial system)
PAT-Transer 2009 (Commercial system)
PC-Transer V13 (Commercial system)
J-Beijing 7 (Commercial system)
Hohrai 2011 (Commercial system)
J Soul 9 (Commercial system)
Korai 2011 (Commercial system)
Google translate
Bing translator
Google’s implementation of “Attention Is All You Need”

We used the following data for training the
NMT baseline systems.
• All of the training data for each task
were used for training except for the ASPEC Japanese–English task. For the ASPEC Japanese–English task, we only usedtrain-1.txt, which consists of one million
parallel sentence pairs with high similarity scores.
• All of the development data for each task
was used for validation.
3.2

✓
✓
✓
✓

ja-ko
✓
✓
✓
JPC
ja-zh zh-ja
✓
✓
✓
✓
✓
✓
✓
✓
en-ja
✓
✓
✓

ja-en
✓
✓
✓
✓
zh-ja
✓
✓
✓

ASPEC
en-ja ja-zh
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
ja-en
✓
✓
✓
✓

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
Other
Other
NMT

3.1

ko-ja
✓
✓
✓

ipant’s system. That is, the specific baseline
system was the standard for human evaluation. At WAT 2019, we adopted a neural machine translation (NMT) with attention mechanism as a baseline system.
The NMT baseline systems consisted of
publicly available software, and the proceduresfor building the systems and for translating using the systems were published on the WAT
web page.13 We also have SMT baseline systems for the tasks that started at WAT 2017 or
before 2017. The baseline systems are shownin Tables 11, 12, and 13. SMT baseline systems are described in the WAT 2017 overview
paper (Nakazawa et al., 2017). The commercial RBMT systems and the online translation
systems were operated by the organizers. We
note that these RBMT companies and onlinetranslation companies did not submit themselves. Because our objective is not to comparecommercial RBMT systems or online translation systems from companies that did notthemselves participate, the system IDs of thesesystems are anonymous in this paper.

Tokenization

We used the following tools for tokenization.
For ASPEC, JPC, TDDC, JIJI, ALT,UCSY, ECCC, and IITB
• Juman version 7.014 for Japanese segmentation.
• Stanford Word Segmenter version 201401-0415 (Chinese Penn Treebank (CTB)

System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT X
Online X
Online X
AIAYN

3.2.1

13

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2019/baseline/baselineSystems.html
14
http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?
JUMAN
15
http://nlp.stanford.edu/software/segmenter.
shtml

Table 11: Baseline Systems I

7

Table 12: Baseline Systems II

System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT X
Online X
Online X

System ID
NMT
NMT T2T
NMT OT
Online X

SystemOpenNMT’s NMT with attention
Tensor2Tensor’s Transformer
OpenNMT-py’s Transformer
Azure Custom Translator

SystemOpenNMT’s NMT with attention
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
PC-Transer V13 (Commercial system)
Google translate
Bing translator

Table 13: Baseline Systems III

8
Type
NMT
NMT
NMT
Other

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
Other
Other

✓

✓

✓
✓

✓
✓

✓

✓
✓

ja-hi
✓
✓

✓

✓

EnTam
ta-en en-ta

✓
✓

IITB
en-hi hi-ja
✓
✓
✓
✓

TDDC
ja-en
✓

hi-en
✓
✓

NewsCommentary
ru-ja
ja-ru

JIJI
ja-en en-ja
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓

✓

✓

ALT
en-my km-en✓✓

Multimodal
en-hi

✓

my-en
✓

✓

en-km
✓

model) for Chinese segmentation.
• The Moses toolkit for English and Indonesian tokenization.
• Mecab-ko16 for Korean segmentation.
• Indic NLP Library17 for Indic language
segmentation.
• The tools included in the ALT corpus for
Myanmar and Khmer segmentation.
• subword-nmt18 for all languages.

3.3.1 NMT with Attention
We used OpenNMT (Klein et al., 2017) as theimplementation of the baseline NMT systemsof NMT with attention (System ID: NMT).
We used the following OpenNMT configuration.
• encoder_type = brnn
• brnn_merge = concat• src_seq_length = 150• tgt_seq_length = 150• src_vocab_size = 100000• tgt_vocab_size = 100000• src_words_min_frequency = 1• tgt_words_min_frequency = 1

When we built BPE-codes, we merged source
and target sentences and we used 100,000 for s option. We used 10 for vocabulary-thresholdwhen subword-nmt applied BPE.
3.2.2

The default values were used for the other system parameters.

For EnTam, News Commentary• The Moses toolkit for English and Russian only for the News Commentary data.

3.3.2 Transformer (Tensor2Tensor)
For the News Commentary and English↔Tamil tasks, we used tensor2tensor’s20
implementation of the Transformer (Vaswani
et al., 2017) and use default hyperparametersettings corresponding to the “base” model forall baseline models. The baseline for the News
Commentary task is a multilingual model as
described in Imankulova et al. (2019) which
is trained using only the in-domain parallel
corpora. We use the token trick proposed by(Johnson et al., 2017) to train the multilingual model. As for the English↔Tamil task,
we train separate baseline models for eachtranslation direction with 32,000 separate
sub-word vocabularies.

• Mecab19 for Japanese segmentation.
• The EnTam corpus is not tokenized byany external toolkits.
• Both corpora are further processedby tensor2tensor’s internal pre/postprocessing which includes sub-word segmentation.
3.2.3

For Multi-Modal Task

• Hindi Visual Genome comes untokenized
and we did not use or recommend any specific external tokenizer.

3.3.3 Transformer (OpenNMT-py)
For the Multimodal task, we used the Transformer model (Vaswani et al., 2018) as implemented in OpenNMT-py (Klein et al., 2017)and used the “base” model with default parameters for the multi-modal task baseline.
We have generated the vocabulary of 32k subword types jointly for both the source and target languages. The vocabulary is shared between the encoder and decoder.

• The standard OpenNMT-py sub-wordsegmentation was used for pre/postprocessing for the baseline system andeach participant used what they wanted.
3.3

Baseline NMT Methods

We used the following NMT with attention
for most of the tasks. We used Transformer(Vaswani et al., 2017) (Tensor2Tensor)) for theNews Commentary and English↔Tamil tasks
and Transformer (OpenNMT-py) for the Multimodal task.

4 Automatic Evaluation
4.1 Procedure for Calculating Automatic
Evaluation Score

16

We evaluated translation results by three metrics: BLEU (Papineni et al., 2002), RIBES

https://bitbucket.org/eunjeon/mecab-ko/
https://bitbucket.org/anoopk/indic_nlp_library18
https://github.com/rsennrich/subword-nmt
19
https://taku910.github.io/mecab/
17

20

9

https://github.com/tensorflow/tensor2tensor

(Isozaki et al., 2010) and AMFM (Banchs
et al., 2015). BLEU scores were calculatedusing multi-bleu.perl in the Moses toolkit(Koehn et al., 2007). RIBES scores were
calculated using RIBES.py version 1.02.4.21
AMFM scores were calculated using scripts
created by the technical collaborators listed in
the WAT2019 web page.22 All scores for eachtask were calculated using the correspondingreference translations.

4.2 Automatic Evaluation System
The automatic evaluation system receives
translation results by participants and automatically gives evaluation scores to the uploaded results. As shown in Figure 2, the system requires participants to provide the following information for each submission:• Human Evaluation: whether or not theysubmit the results for human evaluation;

Before the calculation of the automatic
evaluation scores, the translation results
were tokenized or segmented with tokenization/segmentation tools for each language.
For Japanese segmentation, we used three different tools: Juman version 7.0 (Kurohashi
et al., 1994), KyTea 0.4.6 (Neubig et al., 2011)
with full SVM model23 and MeCab 0.996(Kudo, 2005) with IPA dictionary 2.7.0.24 For
Chinese segmentation, we used two different
tools: KyTea 0.4.6 with full SVM Model in
MSR model and Stanford Word Segmenter(Tseng, 2005) version 2014-06-16 with Chinese Penn Treebank (CTB) and Peking University (PKU) model.25 For Korean segmentation, we used mecab-ko.26 For Myanmar and
Khmer segmentations, we used myseg.py27
and kmseg.py28 . For English and Russian tokenizations, we used tokenizer.perl29 in theMoses toolkit. For Hindi and Tamil tokenizations, we used Indic NLP Library.30 The detailed procedures for the automatic evaluation
are shown on the WAT2019 evaluation web
page.31• Publish the results of the evaluation:
whether or not they permit to publish automatic evaluation scores on the
WAT2019 web page.
• Task: the task you submit the results for;• Used Other Resources: whether or notthey used additional resources; and• Method: the type of the method including
SMT, RBMT, SMT and RBMT, EBMT,NMT and Other.
Evaluation scores of translation results thatparticipants permit to be published are disclosed via the WAT2019 evaluation web page.
Participants can also submit the results for human evaluation using the same web interface.
This automatic evaluation system will remain available even after WAT2019. Anybody
can register an account for the system by theprocedures described in the registration web
page.32
4.3 Additional Automatic Scores inMulti-Modal Task
For the multi-modal task, several additional
automatic metrics were run aside from the
WAT evaluation server, namely: BLEU (nowcalculated by Moses scorer33 ), characTER(Wang et al., 2016), chrF3 (Popović, 2015),
TER (Snover et al., 2006), WER, PER and
CDER (Leusch et al., 2006). Except for chrF3
and characTER, we ran Moses tokenizer34 on
the candidate and reference before scoring.
For all error metrics, i.e. metrics where better21

http://www.kecl.ntt.co.jp/icl/lirg/ribes/index.
html
22
lotus.kuee.kyoto-u.ac.jp/WAT/WAT2019/23
http://www.phontron.com/kytea/model.html
24
http://code.google.com/p/mecab/downloads/
detail?name=mecab-ipadic-2.7.0-20070801.tar.gz
25
http://nlp.stanford.edu/software/segmenter.
shtml
26
https://bitbucket.org/eunjeon/mecab-ko/
27
http://lotus.kuee.kyoto-u.ac.jp/WAT/
my-en-data/wat2019.my-en.zip
28
http://lotus.kuee.kyoto-u.ac.jp/WAT/
km-en-data/km-en.zip
29
https://github.com/moses-smt/mosesdecoder/tree/RELEASE-2.1.1/scripts/tokenizer/tokenizer.perl
30
https://bitbucket.org/anoopk/indic_nlp_library31
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/index.html

32

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2019/registration/index.html
33
https://github.com/moses-smt/mosesdecoder/
blob/master/mert/evaluator.cpp
34
https://github.com/moses-smt/mosesdecoder/
blob/master/scripts/tokenizer/tokenizer.perl

10

WAT

The Workshop on Asian Translation
Submission
SUBMISSION
Logged in as: ORGANIZER
LogoutSubmission:
Human
Evaluation:

human evaluation

Publish theresults of theevaluation:

publishTeam Name:

ORGANIZER

Task:

en-ja

Submission
File:

ファイルを選択 選択されていません

Used Other
Resources:used other resources such as parallel corpora, monolingual corpora andparallel dictionaries in addition to oﬃcial corpora

Method:SMT

SystemDescription(public):

100
charactersor less

SystemDescription(private):100
charactersor less

SubmitGuidelines for submission:
System requirements:
The latest versions of Chrome, Firefox, Internet Explorer and Safari are supported for this site.
Before you submit ﬁles, you need to enable JavaScript in your browser.
File format:
Submitted ﬁles should NOT be tokenized/segmented. Please check the automatic evaluation procedures.
Submitted ﬁles should be encoded in UTF-8 format.
Translated sentences in submitted ﬁles should have one sentence per line, corresponding to each test sentence. The number of lines in the submitted ﬁle and that of thecorresponding test ﬁle should be the same.
Tasks:
en-ja, ja-en, zh-ja, ja-zh indicate the scientiﬁc paper tasks with ASPEC.
HINDENen-hi, HINDENhi-en, HINDENja-hi, and HINDENhi-ja indicate the mixed domain tasks with IITB Corpus.
JIJIen-ja and JIJIja-en are the newswire tasks with JIJI Corpus.
RECIPE{ALL,TTL,STE,ING}en-ja and RECIPE{ALL,TTL,STE,ING}ja-en indicate the recipe tasks with Recipe Corpus.
ALTen-my and ALTmy-en indicate the mixed domain tasks with UCSY and ALT Corpus.
INDICen-{bn,hi,ml,ta,te,ur,si} and INDIC{bn,hi,ml,ta,te,ur,si}-en indicate the Indic languages multilingual tasks with Indic Languages Multilingual Parallel Corpus.
JPC{N,N1,N2,N3,EP}zh-ja ,JPC{N,N1,N2,N3}ja-zh, JPC{N,N1,N2,N3}ko-ja, JPC{N,N1,N2,N3}ja-ko, JPC{N,N1,N2,N3}en-ja, and JPC{N,N1,N2,N3}ja-en indicate the patent tasks
with JPO Patent Corpus. JPCN1{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} are the same tasks as JPC{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} in WAT2015-WAT2017. AMFM is not calculated
for JPC{N,N2,N3} tasks.
Human evaluation:
If you want to submit the ﬁle for human evaluation, check the box "Human Evaluation". Once you upload a ﬁle with checking "Human Evaluation" you cannot change the ﬁle used
for human evaluation.
When you submit the translation results for human evaluation, please check the checkbox of "Publish" too.
You can submit two ﬁles for human evaluation per task.
One of the ﬁles for human evaluation is recommended not to use other resources, but it is not compulsory.
Other:
Team Name, Task, Used Other Resources, Method, System Description (public) , Date and Time(JST), BLEU, RIBES and AMFM will be disclosed on the Evaluation Site when you
upload a ﬁle checking "Publish the results of the evaluation".
You can modify some ﬁelds of submitted data. Read "Guidelines for submitted data" at the bottom of this page.
Back to top

Figure 2: The interface for translation results submission

5 Human Evaluation

scores are lower, we reverse the score by taking 1 − x and indicate this by prepending “n”
to the metric name. With this modification,higher scores always indicate a better translation result. Also, we multiply all metric scoresby 100 for better readability.

In WAT2019, we conducted three kinds ofhuman evaluations: pairwise evaluation (Section 5.1) and JPO adequacy evaluation (Section 5.2) for text-only language pairs and apairwise variation of direct assessment (Section 5.3) for the multi-modal task.
11

5.1

5

Pairwise Evaluation

We conducted pairwise evaluation for participants’ systems submitted for human evaluation. The submitted translations were evaluated by a professional translation company
and Pairwise scores were given to the submissions by comparing with baseline translations(described in Section 3).

4
3
2
1

Table 14: The JPO adequacy criterion

5.1.1 Sentence Selection and Evaluation
For the pairwise evaluation, we randomly selected 400 sentences from the test set of eachtask. We used the same sentences as the lastyear for the continuous subtasks. Baseline andsubmitted translations were shown to annotators in random order with the input source
sentence. The annotators were asked to judge
which of the translations is better, or whether
they are on par.

confidence interval. The procedure is as follows:
1. randomly select 300 sentences from the
400 human evaluation sentences, and calculate the Pairwise score of the selected
sentences
2. iterate the previous step 1000 times andget 1000 Pairwise scores

5.1.2 Voting
To guarantee the quality of the evaluations,each sentence is evaluated by 5 different annotators and the final decision is made dependingon the 5 judgements. We define each judgement ji (i = 1, · · · , 5) as:
if better than the baseline 1−1 if worse than the baselineji =
0if the quality is the same

3. sort the 1000 scores and estimate the 95%
confidence interval by discarding the top
25 scores and the bottom 25 scores
5.2 JPO Adequacy Evaluation
We conducted JPO adequacy evaluation for
the top two or three participants’ systems ofpairwise evalution for each subtask.35 The
evaluation was carried out by translation experts based on the JPO adequacy evaluation
criterion, which is originally defined by JPO
to assess the quality of translated patent documents.

The final∑ decision D is defined as follows using
S=
ji :

 win (S ≥ 2)
loss (S ≤ −2)D=
tie (otherwise)

5.2.1 Sentence Selection and Evaluation
For the JPO adequacy evaluation, the 200 test
sentences were randomly selected from the 400
test sentences used for the pairwise evaluation.
For each test sentence, input source sentence,translation by participants’ system, and reference translation were shown to the annotators.
To guarantee the quality of the evaluation,
each sentence was evaluated by two annotators. Note that the selected sentences are thesame as those used in the previous workshops
except for the new subtasks at WAT2019.

5.1.3 Pairwise Score Calculation
Suppose that W is the number of wins compared to the baseline, L is the number of losses
and T is the number of ties. The Pairwisescore can be calculated by the following formula:
P airwise = 100 ×

All important information is transmitted correctly. (100%)
Almost all important information is transmitted correctly. (80%–)
More than half of important information istransmitted correctly. (50%–)
Some of important information is transmitted
correctly. (20%–)
Almost all important information is NOTtransmitted correctly. (–20%)

W −L
W +L+T

From the definition, the Pairwise score ranges
between -100 and 100.

5.2.2 Evaluation Criterion
Table 14 shows the JPO adequacy criterion
from 5 to 1. The evaluation is performed

5.1.4 Confidence Interval Estimation
There are several ways to estimate a confidence interval. We chose to use bootstrap resampling (Koehn, 2004) to estimate the 95%

35
The number of systems varies depending on thesubtasks.

12

Figure 3: Manual evaluation of text-only translation in the multi-modal task.

subjectively. “Important information” represents the technical factors and their relationships. The degree of importance of each element is also considered to evaluate. The percentages in each grade are rough indications
for the transmission degree of the source sentence meanings. The detailed criterion is described in the JPO document (in Japanese).36
5.3

Figure 4: Manual evaluation of multi-modal translation.

Manual Evaluation for the Multi-Modal
Task

ation. Instead, the reference can be included
among other candidates and the manual evaluation allows us to directly compare the performance of MT to human translators.
The dual judgment (scoring two candidates
at once) was added experimentally. The advantage is saving some of the annotators’ time(they do not need to read the source or examine the picture again) and the chance to evaluate candidates also in terms of direct pairwise
comparisons. In the history of WMT (Bojaret al., 2016), 5-way relative ranking was used
for many years. With 5 candidates, the individual pairs may not be compared very precisely. With the single-candidate DA, pairwise
comparisons cannot be used as the basis for
system ranking. We believe that two candidates on one screen could be a good balance.
For the full statistical soundness, the judgments should be independent of each other.
This is not the case in our dual scoring, evenif we explicitly ask people to score the candidates independent of each other. The fullindependence is however not assured even in
the original approach because annotators willremember their past judgments. This year,
WMT even ran DA with document context
available to the annotators by scoring all segments from a given document one after another in their natural order. We thus dare topretend independence of judgments when interpreting DA scores.

The evaluations of the three tracks of themulti-modal task follow the Direct Assessment(DA, Graham et al., 2016) technique by asking annotators to assign a score from 0 to 100
to each candidate. The score is assigned using
a slider with no numeric feedback, the scale istherefore effectively continuous. After a certain number of scored items, each of the annotators stabilizes in their predictions.
The collected DA scores can be either directly averaged for each system and track (denoted “Ave”), or first standardized per annotator and then averaged (“Ave Z”). The standardization removes the effect of individual
differences in the range of scores assigned: thescores are scaled so that the average score ofeach annotator is 0 and the standard deviation
is 1.
Our evaluation differs from the basic DA in
the following respects: (1) we run the evaluation bilingually, i.e. we require the annotators
to understand the source English sufficiently
to be able to assess the adequacy of the Hindi
translation, (2) we ask the annotators to score
two distinct segments at once, while the original DA displays only one candidate at a time.
The main benefit of bilingual evaluation isthat the reference is not needed for the evalu36
http://www.jpo.go.jp/shiryou/toushin/chousa/
tokkyohonyaku_hyouka.htm

13evaluation. Table 16 shows tasks for which
each team submitted results by the deadline.

7 Evaluation Results
In this section, the evaluation results for
WAT2019 are reported from several perspectives. Some of the results for both automaticand human evaluations are also accessible atthe WAT2019 website.37
7.1 Official Evaluation Results
Figures 6, 7, 8 and 9 show the official evaluation results of ASPEC subtasks, Figures 10,
11, 12, 13, 14 and 15 show those of JPC subtasks, Figures 16 and 17 show those of JIJI
subtasks, Figures 18 and 19 show those of
NCPD subtasks, Figures 20 and 21 show thoseof IITB subtasks, Figures 22, 23, 24 and 25
show those of ALT subtasks, Figures 26 and27 show those of TDDC subtasks and Figures
28 and 29 show those of UFAL subtasks. Eachfigure contains the JPO adequacy evaluation
result and evaluation summary of top systems.
The detailed automatic evaluation results
are shown in Appendix A. The detailed JPO
adequacy evaluation results for the selectedsubmissions are shown in Tables 17 and 18.
The weights for the weighted κ (Cohen, 1968)
is defined as |Evaluation1 − Evaluation2|/4.
The automatic scores for the multi-modal
task along with the WAT evaluation server
BLEU scores are provided in Table 20. For
each of the test sets (E-Test and C-Test), thescores are comparable across all the tracks(text-only, captioning or multi-modal translation) because of the underlying set of referencetranslations is the same. The scores for thecaptioning task will be however very low because captions generated independently of the
English source caption are very likely to differ
from the reference translation.
For multi-modal task, Table 19 shows the
manual evaluation scores for all valid system
submissions. As mentioned above, we used
the reference translation as if it was one of
the competing systems, see the rows “Reference” in the table. The annotation was fully
anonymized, so the annotators had no chanceof knowing if they are scoring human translation or MT output.

Figure 5: Manual evaluation of Hindi captioning.

The user interface for our annotation foreach of the tracks is illustrated in Figure 3,
Figure 4, and Figure 5.
In the “text-only” evaluation, one English
text (source) and two Hindi translations (candidate 1 and 2) are shown to the annotators.
In the “multi-modal” evaluation, the annotators are shown both the image and the source
English text. The first question is to validate
if the source English text is a good caption for
the indicated area. For two translation candidates, the annotators are asked to independently indicate to what extent the meaning ispreserved. The “Hindi captioning” evaluationshows only the image and two Hindi candidates. The annotators are reminded that thetwo captions should be treated independently
and that each of them can consider a very different aspect of the region.

6

Participants

Table 15 shows the participants in WAT2019.
The table lists 25 organizations from various
countries, including Japan, India, Myanmar,
USA, Korea, China, France, and Switzerland.
About 400 translation results by 25 teams
were submitted for automatic evaluation andabout 30 translation results by 8 teams were
submitted for pairwise evaluation. We selected
about 50 translation results for JPO adequacy

37
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/

14

Figure 6: Official evaluation results of aspec-ja-en.

Figure 7: Official evaluation results of aspec-en-ja.

Figure 8: Official evaluation results of aspec-ja-zh.

15

Figure 9: Official evaluation results of aspec-zh-ja.

Figure 10: Official evaluation results of jpcn-ja-en.

Figure 11: Official evaluation results of jpcn-en-ja.

16

Figure 12: Official evaluation results of jpcn-ja-zh.

Figure 13: Official evaluation results of jpcn-zh-ja.

Figure 14: Official evaluation results of jpcn-ja-ko.

17

Figure 15: Official evaluation results of jpcn-ko-ja.

Figure 16: Official evaluation results of jiji-ja-en.

Figure 17: Official evaluation results of jiji-en-ja.

18

Figure 18: Official evaluation results of ncpd-ja-ru.

Figure 19: Official evaluation results of ncpd-ru-ja.

Figure 20: Official evaluation results of iitb-en-hi.

19Figure 21: Official evaluation results of iitb-hi-en.

Figure 22: Official evaluation results of alt2-en-my.

Figure 23: Official evaluation results of alt2-my-en.

20Figure 24: Official evaluation results of alt2-en-km.

Figure 25: Official evaluation results of alt2-km-en.

Figure 26: Official evaluation results of tddc-itm-ja-en.

21

Figure 27: Official evaluation results of tddc-txt-ja-en.

Figure 28: Official evaluation results of ufal-en-ta.

Figure 29: Official evaluation results of ufal-ta-en.

22

Team ID
TMU
NICT
NTT
NICT-2
NICT-4
NICT-5UCSYNLPUCSMNLP
cvit
srcb
sarah
683
KNU_Hyundai
NITSNLPryan
PUP-INDFBAIAISTAISYSTRAN
NHK-NES
geoduck
LTRC-MT
ykkd
IDIAP
NLPRL

Organization
Tokyo Metropolitan University
NICT
NTT Corporation
NICT
NICT
NICT
University of Computer Studies, Yangon
University of Computer Studies, Mandalay
IIIT Hyderabad
RICOH Software Research Center Beijing Co.,Ltd
Rakuten Institute of Technology
National Institute of Technology Silchar
Kangwon National University
National Institute of Technology Silchar
Kakao Brain
Punjabi University Patiala
Facebook AI Research
National Institute of Advanced Industrial Science and Technology
SYSTRANNHK & NHK Engineering System
Microsoft Research
IIIT Hyderabad
The University of Tokyo
Idiap Research Institute
Indian Institute of Technology (BHU) Varanasi

Country
Japan
Japan
Japan
Japan
Japan
Japan
Myanmar
Myanmar
India
China
Japan
India
Korea
India
Korea
India
USA
Japan
France
Japan
USA
India
Japan
Switzerland
India

Table 15: List of participants in WAT2019
Team ID
TMU
NTT
NICT-2
NICT-5
srcb
sarah
KNU_Hyundairyan
AISTAISYSTRAN
NHK-NES
geoduck
ykkd

EJ
✓
✓

ASPEC
JE CJ

JC

✓

✓
✓
✓
✓

✓

✓

✓

✓

✓

✓

JE

CJ

JPC
JC

Ko-J

J-Ko

TDDC
JE

✓
✓
✓

✓
✓
✓

✓
✓
✓

✓
✓
✓

✓
✓
✓

✓

✓

EM
✓
✓

ALT
ME E-Kh
✓
✓
✓

✓
✓

✓
✓

✓

✓

✓

Mixed-domain tasks
IITB
Kh-E EH HE
✓

✓

✓

✓

✓

✓

NCPD
RJ JR
✓
✓
✓

✓

✓

✓

✓

✓

Team ID

JIJI
EJ JE

✓
✓

✓

NICT
NICT-4
NICT-5UCSYNLPUCSMNLP
cvit
sarah
683
NITSNLP
PUP-INDFBAILTRC-MT
IDIAP
NLPRL

EJ

UFAL (EnTam)
ET
TE

✓

✓

✓

✓

✓

✓

Mutimodal task
EV/CH
EH

✓
✓
✓

✓

✓
✓

✓

✓

Table 16: Submissions for each task by each team. E, J, C, Ko, R, M, Kh, H, and T denote English,
Japanese, Chinese, Korean, Russian, Myanmar, Khmer, Hindi, and Tamil language, respectively.

23

SYSTEMDATA Annotator A
Annotator B
allweightedID
ID average varianceaverage varianceaverage κκ
KNU_Hyundai 3173 4.63
0.40
4.38
0.41
4.51 0.23 0.31
ntt3225 4.62
0.39
4.34
0.42
4.49 0.26 0.32
aspec-ja-en
NICT-23086 4.51
0.74
4.30
0.68
4.41 0.22 0.39
2018 best
2474 4.37
0.49
4.63
0.44
4.50 0.15 0.25
ntt3236 4.60
0.51
4.39
0.43
4.50 0.07 0.15
srcb
3212 4.54
0.62
4.32
0.52
4.43 0.17 0.31
aspec-en-ja
NICT-23182 4.54
0.53
4.28
0.46
4.41 0.22 0.31
2018 best
2219 4.16
0.90
4.57
0.57
4.36 0.17 0.30
KNU_Hyundai 3170 4.44
0.47
4.29
0.85
4.36 0.15 0.15
aspec-ja-zh
0.74
4.37
0.86
4.25 0.16 0.26
srcb
3208 4.14
2018 best
2266 4.67
0.32
4.27
0.90
4.47 0.28 0.36
srcb
3210 4.80
0.24
4.46
0.61
4.63 0.27 0.31
aspec-zh-ja KNU_Hyundai 3179 4.76
0.26
4.42
0.71
4.59 0.14 0.16
2018 best
2267 4.78
0.26
4.48
0.67
4.63 0.31 0.33
0.40
4.83
0.22
4.78 0.36 0.46
KNU_Hyundai 3188 4.73
sarah
2927 4.63
jpcn-ja-en
0.53
4.78
0.29
4.71 0.44 0.55
0.50
4.77
0.27
4.70 0.33 0.38
ryan2962 4.62
KNU_Hyundai 3192 4.43
0.81
4.57
0.77
4.50 0.36 0.49
jpcn-en-jasarah2926 4.38
0.83
4.40
0.98
4.39 0.35 0.51
ryan
2961 4.30
0.90
4.44
0.94
4.37 0.36 0.53
KNU_Hyundai 3157 4.53
0.45
4.56
0.54
4.54 0.29 0.35
jpcn-ja-zh
0.49
4.41
0.74
4.42 0.29 0.39
ryan2948 4.43
0.50
4.41
0.75
4.40 0.39 0.48
sarah2921 4.39
KNU_Hyundai 3152 4.72
0.26
4.57
0.55
4.65 0.26 0.35
2949 4.42
jpcn-zh-ja
ryan
0.58
4.42
0.81
4.42 0.35 0.48
sarah
2920 4.45
0.56
4.37
0.80
4.41 0.36 0.51
ryan
2850 4.82
0.27
4.73
0.34
4.77 0.56 0.65
jpcn-ja-ko
0.27
4.71
0.36
4.77 0.31 0.46
sarah2925 4.83
sarah2924 4.72
0.39
4.58
0.68
4.65 0.59 0.69
jpcn-ko-ja KNU_Hyundai 2998 4.70
0.35
4.59
0.58
4.65 0.68 0.76
ryan2890 4.68
0.35
4.55
0.63
4.62 0.71 0.74
NHK-NES2884 4.50
0.68
4.61
0.72
4.55 0.26 0.38
jiji-ja-en
sarah2793 3.27
1.13
3.73
1.39
3.50 0.23 0.39
NHK-NES
2886 4.04
1.02
4.18
1.37
4.11 0.21 0.42
jiji-en-jasarah2814 3.00
1.35
2.89
1.99
2.95 0.19 0.42
Subtask

Table 17: JPO adequacy evaluation results in detail (1).

24

SYSTEM DATA Annotator A
Annotator B
allweightedID
ID average varianceaverage varianceaverage κ
κ
2.56
2.62
2.12
2.64 0.26 0.47
SYSTRAN 3076 2.65
NICT-53026 1.70
ncpd-ja-ru
1.39
1.54
1.19
1.62 0.29 0.52
TMU
3095 1.32
0.65
1.23
0.57
1.28 0.30 0.43
2.16
2.42
1.99
2.32 0.24 0.48
SYSTRAN 2912 2.23
ncpd-ru-ja
NICT-53027 1.41
0.95
1.44
0.74
1.42 0.34 0.57
TMU3097 1.21
0.50
1.29
0.65
1.25 0.36 0.56
cvit-mt
2680 3.94
1.02
3.58
0.82
3.76 0.53 0.58
iitb-en-hi
2018 best 2362 3.58
2.71
3.40
2.52
3.49 0.52 0.74
cvit-mt
2681 4.53
0.53
3.74
1.18
4.13 0.05 0.13
NICT-52865 4.26
0.90
3.39
1.48
3.83 0.10 0.22
iitb-hi-en
0.91
2.94
1.15
3.43 0.05 0.16
LTRC3119 3.92
2018 best 2381 2.96
2.55
2.96
2.52
2.96 0.48 0.76
fbai
3203 4.36
0.67
3.36
1.02
3.86 0.02 0.19
NICT2818 2.74
1.34
2.71
1.25
2.73 0.97 0.98
alt2-en-my
NICT-42979 2.40
1.21
2.38
1.13
2.39 0.97 0.98
0.05
1.06
0.06
1.06 0.59 0.59
UCSYNLP 2858 1.05
fbai
3201 4.49
0.53
4.24
0.47
4.36 0.13 0.18
NICT2816 3.88
0.78
4.03
0.46
3.96 0.07 0.18
alt2-my-en
NICT-42977 2.56
0.78
3.01
0.85
2.79 0.26 0.42
UCSYNLP 3252 1.25
0.27
1.34
0.22
1.30 0.62 0.60
organizer 2898 2.54
1.54
3.00
1.19
2.77 0.24 0.49
alt2-en-km
NICT-42929 2.43
1.35
2.25
1.20
2.34 0.67 0.80
1.52
3.35
0.92
2.97 0.08 0.31
organizer 2897 2.60
alt2-km-en
NICT-42915 2.91
1.33
2.73
1.25
2.82 0.61 0.76
ntt3002 4.48
0.83
4.46
1.05
4.47 0.17 0.30
tddc-itm-ja-en
0.77
4.40
1.04
4.46 0.29 0.47
sarah2807 4.52
0.94
4.30
1.19
4.36 0.40 0.52
NICT-23081 4.42
ntt3005 4.34
0.65
4.58
0.65
4.46 0.16 0.27
NICT-23084 4.26
0.82
4.58
0.72
4.42 0.22 0.33
tddc-txt-ja-en
sarah2808 4.20
0.83
4.49
0.87
4.34 0.26 0.39
geoduck 3200 4.07
0.99
4.17
1.64
4.12 0.30 0.48
0.56
3.29
0.82
3.50 0.17 0.34
NLPRL018 3015 3.71
cvit-mt
2830 3.46
ufal-en-ta
1.10
3.50
1.34
3.48 0.84 0.90
NICT-53046 3.15
0.92
3.19
1.16
3.17 0.62 0.74
NICT-53054 3.88
0.46
3.75
0.90
3.81 0.39 0.48
ufal-ta-en
cvit-mt
2833 3.58
0.89
3.70
1.19
3.64 0.75 0.83
NLPRL018 3014 3.67
0.47
3.31
0.91
3.49 0.10 0.22
Subtask

Table 18: JPO adequacy evaluation results in detail (2).

25

EV
TEXT
CH
TEXT
EV
MM
CH
MM
CH EV
HI HI

Team ID
IDIAP
Reference
683
683
NITSNLP
Reference
IDIAP
IDIAP
683
683
NITSNLP
Reference
683
PUP-IND
PUP-IND
NITSNLP
Reference
683
NITSNLP
PUP-IND
PUP-IND
Reference
NITSNLP
Reference
NITSNLP
683

Data ID
2956
3285
3286
3299
3277
3267
3284
3287
3300
3271
3296
3295
3288
3270
3298
3281
3280
3289
3297
3304

Ave
72.85
71.34
68.89
61.64
52.53
79.23
60.81
60.17
45.69
45.52
28.48
70.04
69.17
62.42
60.22
58.98
75.96
54.51
48.45
48.06
47.06
68.80
51.78
72.60
44.46
26.54

Ave Z
0.70
0.66
0.57
0.36
0.00
0.94
0.25
0.25
-0.28
-0.24
-0.81
0.60
0.61
0.35
0.28
0.25
0.76
0.08
-0.20
-0.13
-0.17
0.52
-0.05
0.61
-0.35
-0.94

CH EV
HI HI

CH
MM

EV
MM

CH
TEXT

EV
TEXT

Table 19: Manual evaluation result for WAT Multi-Modal Tasks.

System
IDIAP
683
683
NITSNLP
IDIAP
IDIAP
683
683
NITSNLP
683
PUP-IND
NITSNLP
PUP-IND
683
NITSNLP
PUP-IND
PUP-IND
NITSNLPNITSNLP
683

Run
2956
3285
3286
3299
3277
3267
3284
3287
3300
3271
3296
3288
3295
3270
3298
3281
3280
3289
3297
3304

BLEU
52.18
48.29
33.47
30.05
40.40
39.08
21.56
21.50
10.50
51.46
39.67
39.13
38.50
28.62
19.68
18.32
16.15
8.68
2.28
1.07

chrF3
58.81
54.66
40.37
34.49
50.18
49.30
30.90
30.27
17.91
57.63
47.76
45.50
45.35
37.86
27.99
27.79
25.78
14.45
8.88
8.63

nCDER
62.18
58.18
45.36
41.36
52.58
51.78
33.92
≀ 34.66
23.04
61.51
51.98
49.45
≀ 50.33
41.60
31.84
30.08
28.57
14.27
8.00
6.65

nCharacTER
57.95
54.12
00.11
≀ 10.92
44.32
41.72
13.69
-65.00≀ -60.87
52.61
46.84
27.92
≀ 41.40
20.10-24.40≀ 19.6306.31-15.81-50.33-19.81

nPER
69.32
65.34
50.54
48.23
60.19
59.49
41.14
38.98
28.05
68.52
59.50
57.43
≀ 58.82
48.64
38.61≀ 40.51
37.34
22.51
12.97
-32.82

nTER
56.87
52.52
43.11
36.42
49.11
48.42
30.53
≀ 32.91
20.87
55.99
43.47
≀ 43.91
41.84
38.38
29.38
23.51
23.38
06.85
06.05
-52.44

nWER
55.07
51.00
42.13
35.10
46.02
45.51
28.40
≀ 31.47
19.90
54.28
41.92
≀ 42.17
40.65
36.44
27.16
21.12
≀ 21.2806.19
05.62-52.59

BLEUw
41.32
38.19
25.34
20.13
30.94
30.34
14.69
≀ 15.85
5.56
40.55
28.27
≀ 28.45
27.39
20.37
12.58
11.77
10.19
2.59
0.00
0.00

Table 20: Multi-Modal Task automatic evaluation results. For each test set (EV and CH) and each track(TEXT, MM and HI), we sort the entries by our BLEU scores. The symbol “≀” in subsequent columns
indicates fields where the other metric ranks candidates in a different order. BLEUw denotes the WAT
official BLEU scores.

26

7.2

Statistical Significance Testing of
Pairwise Evaluation between
Submissions

these high-rated pairs consist of typical terms
and sentences from timely disclosure documents. This tasks focus on the accurate translation of figures, so the evaluation criteria confirmed there are no mistranslation in the typical sentences containing figures, such as unitof money and dates.
However, uncommon sentences used intimely disclosure documents tend to be mistranslated. For example, uncommon proper
nouns tended to be omitted or mistranslated
to other meaning words, besides sentences
which has complex and uncommon structures,
generally long sentences, caused errors at dependency of subordinate clauses.
In addition, some systems translated sentences without subjects into sentences withincorrect subjects. Japanese sentences often
omit subjects and objects, which would normally be included in English. For example, aJapanese sentence, “当社普通株式 27,000 株を上限とする。
”(Common shares of the Company, limited to a maximum of 27,000 shares),
was translated to“(Unrelated company name)common stocks up to 27,000 shares”.
Moreover, there are some incorrect modifiers or determiners. In Japanese timely disclosure documents, there are many variable prefix
for dates, such as“本”(this),“当”(this),“次”(next), and “前”(last). Some systems translated sentences containing these words withincorrect year. For example, a Japanese sentence contains “当第 3 四半期連結会計期間末”(the end of third quarter of this fiscal year)
was translated to“the end of the third quarterof FY 2016”.
In summary, the causes of these mistranslations are considered as follows:

Table 21 shows the results of statistical significance testing of aspec-ja-en subtasks, Table 22shows that of JIJI subtasks, Table 23 showsthat of TDDC subtasks. ≫, ≫ and > meanthat the system in the row is better than thesystem in the column at a significance level ofp < 0.01, 0.05 and 0.1 respectively. Testingis also done by the bootstrap resampling asfollows:
1. randomly select 300 sentences from the
400 pairwise evaluation sentences, andcalculate the Pairwise scores on the selected sentences for both systems
2. iterate the previous step 1000 times andcount the number of wins (W ), losses (L)and ties (T )3. calculate p = WL+L
Inter-annotator Agreement
To assess the reliability of agreement between
the workers, we calculated the Fleiss’ κ (Fleiss
et al., 1971) values. The results are shownin Table 24. We can see that the κ values
are larger for X→J translations than for J→Xtranslations. This may be because the majority of the workers for these language pairs are
Japanese, and the evaluation of one’s mothertongue is much easier than for other languages
in general. The κ values for Hindi languages
are relatively higt. This might be because theoverall translation quality of the Hindi languages are low, and the evaluators can easily distinguish better translations from worse
ones.

8

• It is difficult for the systems to translate
long sentence and proper nouns which
TDDC does not contain.

Findings• Some source sentences are unclear due tolack of subjects and/or objects, so these
are not suitable for English translation.

In this section, we will show findings of someof the translation tasks.
8.1

TDDC

• TDDC contains not semantically balanced pairs and the systems might be affected strongly by either of source pairsentences.

In the results of both the automatic evaluation and the human evaluation, every systemtranslated most sentences correctly. According to the human evaluation of the subtasks of’Items’ and ’Texts’, all evaluators rated morethan 70% of all the pairs at 4 or 5. Most of

On the other hand, some translations seemto be fitted to sentences of TDDC which are27

KNU_Hyundai (3172)

srcb (3212)

AISTAI (3251)

NICT-2 (3182)

srcb (3205)

KNU_Hyundai (3173)
NICT-2 (3086)
NTT (3225)
KNU_Hyundai (3173)
NICT-2 (3086)

NTT (3236)≫≫≫≫
NICT-2 (3182)- ≫≫srcb (3212)> ≫AISTAI (3251)
-

- ≫≫
- ≫
>

NHK-NES (2884) ≫ ≫ ≫NHK-NES (2883)≫≫
sarah (2793)≫

sarah (2815)

sarah (2814)

NHK-NES (2885)

sarah (2813)

sarah (2793)

NHK-NES (2883)

Table 21: Statistical significance testing of the aspec-ja-en (left) and aspec-en-ja (right) Pairwise scores.

NHK-NES (2886) ≫ ≫ ≫NHK-NES (2885)≫≫
sarah (2814)≫

ORGANIZER (3264) ≫ ≫ ≫ ≫ ≫ ≫≫ ≫≫≫≫
sarah (2807)
NTT (3002)- ≫≫≫NICT-2 (3081)≫≫≫
sarah (2811)- ≫geoduck (3197)≫

ORGANIZER (3265)
NTT (3005)
sarah (2808)
NICT-2 (3084)
sarah (2812)
geoduck (3200)

geoduck (3217)

geoduck (3200)

sarah (2812)

NICT-2 (3084)

sarah (2808)

NTT (3005)

geoduck (3216)

geoduck (3197)

sarah (2811)

NICT-2 (3081)

sarah (2807)

NTT (3002)

Table 22: Statistical significance testing of the jiji-ja-en (left) and jiji-en-ja (right) Pairwise scores.

- ≫≫≫≫≫
≫≫≫≫≫
- ≫≫≫
≫≫≫
≫≫
≫

Table 23: Statistical significance testing of the tddc-itm-ja-en (left) and tddc-txt-ja-en (right) Pairwisescores.

freely and omitted redundant expressions, but
evaluators mark them as low scores, probably
because they are not literal translations. This
result implies that it is necessary to create another evaluation criterion, which evaluates thecorrectness of transmitting information to investors correctly.

the main reason is the lack of helping parallel
corpora. We expect that utilization of large
in-domain monolingual corpora for backtranslation should help alleviate this problem.
We will provide such corpora for next year’stask.
8.3 News Commentary Task

8.2

English↔Tamil Task

We only received 3 submissions for Russian↔Japanese translation and all submissions leveraged multilingualism and multi-stepfine-tuning proposed by Imankulova et al.
(2019) and showed that carefully choosing corpora and robust training can dramatically enhance the quality of NMT for language pairs

We observed that most participants
used transfer learning techniques suchas fine-tuning and mixed fine-tuning for
Tamil→English translation leading to reasonably high quality translations. However,English→Tamil translation is still poor and28

aspec-ja-en
SYSTEMDATA κ
NTT
3225 0.157
NICT-23086 0.187
srcb
3205 0.220
KNU_Hyundai 3173 0.156
ave.
0.180

aspec-en-ja
SYSTEMDATA κ
NTT
3236 0.298
NICT-23182 0.319
srcb
3212 0.305
KNU_Hyundai 3172 0.302
AISTAI
3251 0.303
ave.
0.305

jiji-ja-en
SYSTEM DATA κ
sarah2793 0.146
sarah2813 0.158
NHK-NES 2883 0.273
NHK-NES 2884 0.159
ave.
0.184

jiji-en-ja
SYSTEM DATA κ
sarah2814 0.357
sarah2815 0.299
NHK-NES 2885 0.354
NHK-NES 2886 0.390
ave.
0.350

tddc-itm-ja-en
SYSTEMDATA κORGANIZER 3264 0.382
NTT3002 0.403
NICT-23081 0.423
sarah2807 0.408
sarah2811 0.391
geoduck
3197 0.404
geoduck
3216 0.493
ave.
0.415

tddc-txt-ja-en
SYSTEMDATA κORGANIZER 3265 0.135
NTT3005 0.163
NICT-23084 0.175
sarah2808 0.163
sarah2812 0.167
geoduck
3200 0.172
geoduck
3217 0.321
ave.
0.185

Table 24: The Fleiss’ kappa values for the pairwise evaluation results.

Source Good?
Yes
No
No Answer
Total

C-Test1586 (78.7 %)
20 (1.0 %)
410 (20.3 %)2016 (100.0 %)E-Test1348 (66.9 %)46 (2.3 %)622 (30.9 %)2016 (100.0 %)receive any answer. This confirms that evennon-anonymous annotators can easily provide
sloppy evaluations. It is possible that part ofthese omissions can be attributed to our annotation interface which was showing all items
on one page and relying on scrolling. Next
time, we will show only one annotation itemon each page and also consider highlighting
unanswered questions. Strictly requiring ananswer would not be always appropriate but
we need to ensure that annotators are awarethat they are skipping a question.
Luckily, the bad source captions are not afrequent case, amounting to 1 or 2% of assessed examples.

Table 25: Appropriateness of source English captions in the 4032 assessments collected for themulti-modal track.

that have very small in-domain parallel corpora. For next year’s task we expect more
submissions where participants will leverage
additional larger helping monolingual as wellas bilingual corpora.
8.4

Multi-Modal Task

8.4.2 Relation to Human Translation

8.4.1 Validation of Source English Captions
In the manual evaluation of multimodal track,
our annotators saw both the picture and thesource text (and the two scored candidates).
We took this opportunity to double check thequality of the original HVG data. Prior toscoring the candidates, we asked our annotators to confirm that the source English text isa good caption for the indicated region of theimage.
The results in Table 25 indicate that for asurprisingly high number of items we did not

The bilingual style of evaluation of the multimodal task allowed us to evaluate the referencetranslations as if they were yet another competing MT system. Table 19 thus lists also the“Reference”.
Across the tracks and test sets (EV vs. CH),
humans surpass MT candidates. One single
exception is IDIAP run 2956 winning in textonly translation of the E-Test, but this is notconfirmed on the C-Test (CH). The score of
the anonymized system 683 on E-Test in multi29

modal track (MM) has also almost reached human performance. These are not the first casesof MT performing on par with humans and weare happy to see this when targetting an Indian language.

prove the automatic speech recognitionand natural language understanding technologies for German (Title: “SM2: Extracting Semantic Meaning from Spoken
Material”funding application no. 29814.1
IP-ICT). The work was also supportedby the EU H2020 project “Real-time network, text, and speaker analytics for combating organized crime” (ROXANNE),
grant agreement: 833635.

8.4.3 Evaluating Captioning
While the automatic scores are comparable
across tasks, the Hindi-only captioning (“HI”)
must be considered separately. Without asource sentence, both humans and machines
are very likely to come up with highly varying textual captions. The same image can be
described in many different aspects. All our
automatic metrics compare the candidate caption with the reference one generally on thebasis of the presence of the same character
sequences, words or n-grams. Candidates diverging from the reference will get a low scoreregardless of their actual quality.
The automatic evaluation score for the“Hindi caption” is very very low as comparedto other sub-tasks (“text-only” and “multimodal” translations) as can be seen in the Table 20. Even the human annotators couldn’t
give any score for most of the segments submitted from the “Hindi caption” entries dueto the wrong caption generation.

9• At Charles University,
the work
was supported by the grants 1926934X (NEUREM3) of the Czech
Science Foundation and “Progress”
Q18+Q48 of Charles University, andusing language resources distributedby the LINDAT/CLARIN project of
the Ministry of Education, Youthand Sports of the Czech Republic(projects LM2015071 and OP VVV VI
CZ.02.1.01/0.0/0.0/16013/0001781).
Some of the human evaluations for the othertasks were supported by“Research and Development of Deep Learning Technology for Advanced Multilingual Speech Translation”, theCommissioned Research of National Instituteof Information and Communications Technology (NICT), JAPAN.

Conclusion and Future Perspective

This paper summarizes the shared tasks of
WAT2019. We had 25 participants worldwide,and collected a large number of useful submissions for improving the current machine translation systems by analyzing the submissions
and identifying the issues.
For the next WAT workshop, we plan to
conduct document-level evaluation using the
new dataset with context for some translationsubtasks and we would like to consider how
to realize context-aware machine translationin WAT. Also, we are planning to do extrinsic
evaluation of the translations.

References
Rafael E. Banchs, Luis F. D’Haro, and Haizhou
Li. 2015. Adequacy-fluency metrics: Evaluating mt in the continuous space model framework. IEEE/ACM Trans. Audio, Speech andLang. Proc., 23(3):472–482.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow,
Matthias Huck, Antonio Yepes, Philipp Koehn,
Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Névéol, Mariana Neves, Martin
Popel, Matt Post, Raphael Rubino, Carolina
Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findingsof the 2016 Conference on Machine Translation(WMT16). In Proceedings of the First Conference on Machine Translation (WMT). Volume 2:
Shared Task Papers, volume 2, pages 131–198,
Stroudsburg, PA, USA. Association for Computational Linguistics, Association for Computational Linguistics.

Acknowledgement
The multi-modal shared task was supportedby the following grants at Idiap Research Institute and Charles University.
• At Idiap Research Institute, the work was
supported by an innovation project (under an InnoSuisse grant) oriented to im-

Jacob Cohen. 1968. Weighted kappa: Nominal
scale agreement with provision for scaled dis-30

agreement or partial credit. Psychological Bulletin, 70(4):213 – 220.

Philipp Koehn, Hieu Hoang, Alexandra Birch,
Chris Callison-Burch, Marcello Federico, Nicola
Bertoldi, Brooke Cowan, Wade Shen, Christine
Moran, Richard Zens, Chris Dyer, Ondrej Bojar,
Alexandra Constantin, and Evan Herbst. 2007.
Moses: Open source toolkit for statistical machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL),
demonstration session.

Chenchen Ding, Hnin Thu Zar Aye, Win Pa Pa,
Khin Thandar Nwet, Khin Mar Soe, Masao
Utiyama, and Eiichiro Sumita. 2019.
Towards Burmese (Myanmar) morphological analysis: Syllable-based tokenization and part-ofspeech tagging. ACM Transactions on Asianand Low-Resource Language Information Processing (TALLIP), 19(1):5.

T. Kudo. 2005.
Mecab :
Yet anotherpart-of-speech and morphological analyzer.
http://mecab.sourceforge.net/.

Chenchen Ding, Masao Utiyama, and Eiichiro
Sumita. 2018. NOVA: A feasible and flexible annotation system for joint tokenization and partof-speech tagging. ACM Transactions on Asianand Low-Resource Language Information Processing (TALLIP), 18(2):17.

Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements of Japanese morphological analyzer JUMAN. In Proceedings of The International
Workshop on Sharable Natural Language, pages
22–28.

J.L. Fleiss et al. 1971. Measuring nominal scaleagreement among many raters. Psychological
Bulletin, 76(5):378–382.

Gregor Leusch, Nicola Ueffing, and Hermann Ney.
2006. CDER: Efficient MT Evaluation Using
Block Movements. In In Proceedings of EACL,
pages 241–248.

Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone.
Natural Language Engineering, FirstView:1–28.

Toshiaki Nakazawa, Chenchen Ding, Hideya
MINO, Isao Goto, Graham Neubig, and Sadao
Kurohashi. 2016. Overview of the 3rd workshop on asian translation. In Proceedings of the
3rd Workshop on Asian Translation (WAT2016),
pages 1–46, Osaka, Japan. The COLING 2016
Organizing Committee.

Aizhan Imankulova, Raj Dabre, Atsushi Fujita,
and Kenji Imamura. 2019. Exploiting out-ofdomain parallel data through multilingual transfer learning for low-resource neural machinetranslation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track,
pages 128–139, Dublin, Ireland.

ToshiakiNakazawa,
Shohei
Higashiyama,
Chenchen Ding, Hideya Mino, Isao Goto,
Hideto Kazawa, Yusuke Oda, Graham Neubig,
and Sadao Kurohashi. 2017. Overview of the
4th workshop on asian translation. In Proceedings of the 4th Workshop on Asian Translation(WAT2017), pages 1–54. Asian Federation of
Natural Language Processing.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages
944–952, Stroudsburg, PA, USA. Association for
Computational Linguistics.

Toshiaki Nakazawa, Hideya Mino, Isao Goto,
Sadao Kurohashi, and Eiichiro Sumita. 2014.
Overview of the 1st Workshop on Asian Translation. In Proceedings of the 1st Workshop
on Asian Translation (WAT2014), pages 1–19,
Tokyo, Japan.

Melvin Johnson, Mike Schuster, Quoc V. Le,
Maxim Krikun, Yonghui Wu, Zhifeng Chen,
Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, andJeffrey Dean. 2017. Google’s multilingual neuralmachine translation system: Enabling zero-shottranslation. Transactions of the Association for
Computational Linguistics, 5:339–351.

Toshiaki Nakazawa, Hideya Mino, Isao Goto, Graham Neubig, Sadao Kurohashi, and Eiichiro
Sumita. 2015. Overview of the 2nd Workshop
on Asian Translation. In Proceedings of the 2nd
Workshop on Asian Translation (WAT2015),
pages 1–28, Kyoto, Japan.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean
Senellart, and Alexander Rush. 2017. Opennmt: Open-source toolkit for neural machinetranslation. In Proceedings of ACL 2017, System Demonstrations, pages 67–72. Association
for Computational Linguistics.

Toshiaki Nakazawa, Katsuhito Sudoh, Shohei
Higashiyama, Chenchen Ding, Raj Dabre,
Hideya Mino, Isao Goto, Win Pa Pa, Anoop
Kunchukuttan, and Sadao Kurohashi. 2018.
Overview of the 5th workshop on Asian translation. In Proceedings of the 32nd PacificAsia Conference on Language, Information andComputation: 5th Workshop on Asian Translation: 5th Workshop on Asian Translation, Hong

Philipp Koehn. 2004. Statistical significance tests
for machine translation evaluation. In Proceedings of EMNLP 2004, pages 388–395, Barcelona,
Spain. Association for Computational Linguistics.

31

Kong. Association for Computational Linguistics.

Ashish Vaswani, Samy Bengio, Eugene Brevdo,
Francois Chollet, Aidan Gomez, Stephan
Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam
Shazeer, and Jakob Uszkoreit. 2018.
Tensor2tensor for neural machine translation. In
Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 193–
199. Association for Machine Translation in theAmericas.

Graham Neubig, Yosuke Nakata, and Shinsuke
Mori. 2011. Pointwise prediction for robust,
adaptable japanese morphological analysis. In
Proceedings of the 49th Annual Meeting of the
Association for Computational Linguistics: Human Language Technologies: Short Papers Volume 2, HLT ’11, pages 529–533, Stroudsburg, PA, USA. Association for Computational
Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar,
Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,
Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.

Kishore Papineni, Salim Roukos, Todd Ward, andWei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In
ACL, pages 311–318.

Weiyue Wang, Jan-Thorsten Peter, Hendrik
Rosendahl, and Hermann Ney. 2016. Character:Translation edit rate on character level. In ACL
2016 First Conference on Machine Translation,
pages 505–510, Berlin, Germany.

Shantipriya Parida, Ondřej Bojar, and Satya Ranjan Dash. 2019a. Hindi visual genome: A
dataset for multimodal english-to-hindi machinetranslation. arXiv preprint arXiv:1907.08948.
Shantipriya Parida, Ondřej Bojar, and Satya Ranjan Dash. 2019b. Hindi Visual Genome: A
Dataset for Multimodal English-to-Hindi Machine Translation. Computación y Sistemas. In
print. Presented at CICLing 2019, La Rochelle,
France.

Yi Mon Shwe Sin and Khin Mar Soe. 2018.
Syllable-based myanmar-english neural machinetranslation. In In Proc. of ICCA, pages 228–233.

Appendix A

Submissions

Tables 26 to 31 summarize translation results
submitted for WAT2019 pairwise evaluation.
Type, RSRC, and Pair columns indicate typeof method, use of other resources, and pairwise evaluation score, respectively. The tables
also include results by the organizers’ baselines, which are listed in Table 13.

Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisboa, Portugal. Association
for Computational Linguistics.
Loganathan Ramasamy, Ondřej Bojar, and Zdeněk
Žabokrtský. 2012. Morphological processing forenglish-tamil statistical machine translation. In
Proceedings of the Workshop on Machine Translation and Parsing in Indian Languages (MTPIL-2012), pages 113–122.
Hammam Riza, Michael Purwoadi, Teduh Uliniansyah, Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai, Vu Tat Thang, Nguyen Phuong
Thai, Vichet Chea, Sethserey Sam, Sopheap
Seng, Khin Mar Soe, Khin Thandar Nwet,
Masao Utiyama, and Chenchen Ding. 2016. Introduction of the asian language treebank. In In
Proc. of O-COCOSDA, pages 1–6.
Matthew Snover, Bonnie Dorr, Richard Schwartz,
Linnea Micciulla, and John Makhoul. 2006. A
study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas,
pages 223–231.
Huihsin Tseng. 2005. A conditional random field
word segmenter. In In Fourth SIGHAN Workshop on Chinese Language Processing.
Masao Utiyama and Hitoshi Isahara. 2007. Ajapanese-english patent parallel corpus. In MT
summit XI, pages 475–482.

32

33

NMT
NTT
NICT-2srcb
AISTAIKNU_Hyundai

System

1900
3236
3182
3212
3251
3172

ID

NMT
NMT
NMT
NMT
NMT
NMT

Type
juman
36.37
45.83
44.61
45.71
42.64
44.08

SystemNMTNTT
KNU_Hyundai
srcb

NO
NO
NO
NO
NO
NO

RSRC
mecab
37.15
46.57
45.66
46.29
43.34
44.78

juman
0.824985
0.861994
0.852970
0.857506
0.8491290.857060

RIBES
kytea
0.831183
0.865640
0.856755
0.860642
0.851400
0.859885

Type
NMT
SMT
NMT
NMT

RSRC
NO
NO
NO
NO

BLEU
26.91
30.56
30.88
30.92

RIBES
0.764968
0.773281
0.774653
0.778832

Table 27: ASPEC ja-en submissions

ID
1901
3225
3173
3205

Table 26: ASPEC en-ja submissions

BLEU
kytea
38.48
47.63
46.59
47.55
44.17
45.88

AMFM
0.595370
0.626880
0.622070
0.630150

mecab
0.833207
0.868175
0.859059
0.862819
0.856177
0.863400

Pair–
14.00011.7506.500

juman
0.759910
0.774950
0.7613400.770440
0.757590
0.760050

AMFM
kytea
0.759910
0.774950
0.7613400.770440
0.757590
0.760050

mecab
0.759910
0.774950
0.7613400.770440
0.757590
0.760050

–
47.75
42.25
40.00
36.75
36.00

Pair

34
System
NMT
NTT
sarah
NICT-2sarah
geoduck
geoduck

System
NMT
sarah
NTT
NICT-2sarah
geoduck
geoduck

ID
3263
3005
2808
3084
2812
3200
3217

ID
3262
2807
3002
3081
2811
3197
3216

RSRC
NO
NO
NO
NO
NO
YES
YES

BLEU
38.55
54.25
57.34
53.87
52.77
54.27
46.21

RIBES
0.758265
0.832909
0.844086
0.834898
0.823336
0.828493
0.703675

RSRC
NO
NO
NO
NO
NO
YES
YES

BLEU
24.11
61.19
58.38
56.95
54.84
61.38
51.08

RIBES
0.701005
0.861346
0.865364
0.848530
0.845921
0.850551
0.659190

Table 29: TDDC TXT ja-en submissions

Type
NMT
NMT
NMT
NMT
NMT
Other
Other

Table 28: TDDC ITM ja-en submissions

Type
NMT
NMT
SMT
NMT
NMT
Other
Other

AMFM
0.583850
0.753630
0.7451100.744390
0.732790
0.743820
0.623770

AMFM
0.712040
0.807250
0.816660
0.805460
0.794050
0.800360
0.710630

Pair–
55.50049.50049.00037.75030.000-28.250

Pair–
36.75034.00033.50029.25027.250-32.500

35

NMT
NHK-NESNHK-NES
sarah
sarah

System

1904
2886
2885
2814
2815

ID

NMT
NMT
NMT
NMT
NMT

Type
juman
16.63
27.22
28.25
22.65
21.80

SystemNMT
NHK-NESNHK-NES
sarah
sarah

NO
YES
YES
NO
NO

RSRC

ID
1905
2884
2883
2793
2813

mecab
16.36
27.63
28.63
22.76
21.91

juman
0.653766
0.730816
0.739036
0.705044
0.698609

RIBES
kytea
0.657279
0.731653
0.7411640.707209
0.699981

RSRC
NO
YES
YES
NO
NO

BLEU
16.48
14.23
26.38
21.84
21.34

RIBES
0.640558
0.612351
0.703808
0.675386
0.676723

Table 31: JIJI ja-en submissions

Type
NMT
NMT
NMT
NMT
NMT

Table 30: JIJI en-ja submissions

BLEU
kytea
17.01
28.75
29.76
23.48
22.67

AMFM
0.459080
0.526010
0.554110
0.526530
0.524530

mecab
0.658830
0.734618
0.743977
0.710111
0.701739

Pair–
89.00072.00050.75044.750juman
0.514530
0.660570
0.654260
0.614840
0.613770

AMFM
kytea
0.514530
0.660570
0.654260
0.614840
0.613770
mecab
0.514530
0.660570
0.654260
0.614840
0.613770–
87.75081.25063.25055.250Pair