Overview of the 7th Workshop on Asian Translation
Toshiaki Nakazawa
The University of Tokyo

Hideki Nakayama
The University of Tokyo

nakazawa@logos.t.u-tokyo.ac.jp

nakayama@ci.i.u-tokyo.ac.jp

Chenchen Ding and Raj Dabre and Shohei Higashiyama
National Institute ofInformation and Communications Technology{chenchen.ding, raj.dabre, shohei.higashiyama}@nict.go.jp

Hideya Mino and Isao GotoNHK

Win Pa Pa
University of Conputer Study, Yangon{mino.h-gq, goto.i-es}@nhk.or.jp

winpapa@ucsy.edu.mm

Anoop Kunchukuttan
Microsoft AI and Research

Shantipriya Parida
Idiap Research Instituteanoop.kunchukuttan@microsoft.com

shantipriya.parida@idiap.chOndřej Bojar
Charles University, MFF, ÚFALSadao Kurohashi
Kyoto University

bojar@ufal.mff.cuni.cz

kuro@i.kyoto-u.ac.jp• Odia ↔ English task

Abstract
This paper presents the results of the sharedtasks from the 7th workshop on Asian transla­tion (WAT2020). For the WAT2020, 20 teams
participated in the shared tasks and 14 teams
submitted their translation results for the hu­
man evaluation. We also received 12 research
paper submissions out of which 7 were ac­
cepted. About 500 translation results were
submitted to the automatic evaluation server,and selected submissions were manually eval­uated.

1• Bengali / Hindi / Malayalam / Tamil / Telugu/ Marathi / Gujarati ↔ English Indic multilin­
gual tasks• English ↔ Japanese multimodal tasks• English ↔ Japanese document­level transla­tion tasks
All the tasks are explained in Section 2.
WAT is a unique workshop on Asian languagetranslation with the following characteristics:

Introduction

The Workshop on Asian Translation (WAT) is an
open evaluation campaign focusing on Asian lan­
guages. Following the success of the previous
workshops WAT2014­WAT2019(Nakazawa et al.,2019), WAT2020 brings together machine transla­tion researchers and users to try, evaluate, shareand discuss brand­new ideas for machine transla­tion. We have been working toward practical use
of machine translation among all Asian countries.
For the 7th WAT, we included the following newtasks:• Open innovation platform
Due to the fixed and open test data, we can re­peatedly evaluate translation systems on thesame dataset over years. WAT receives sub­missions at any time; i.e., there is no submis­
sion deadline of translation results w.r.t auto­
matic evaluation of translation quality.
• Domain and language pairs
WAT is the world’s first workshop thattargets scientific paper domain, andChinese↔Japanese and Korean↔Japanese
language pairs. In the future, we will add• Hindi / Thai / Malay / Indonesian ↔ English
IT and Wikinews tasks
1

Proceedings of the 7th Workshop on Asian Translation, pages 1–44
December 4, 2020. c 2020 Association for Computational Linguistics

Lang
JE
JC

Train
3,008,500
672,315

Dev
1,790
2,090

DevTest
1,784
2,148

Test
1,812
2,107

Lang
zh­ja
ko­ja
en­ja

Train
1,000,000
1,000,000
1,000,000

Dev
2,000
2,000
2,000

DevTest
2,000
2,000
2,000

Test­N
5,204
5,230
5,668

Lang
zh­ja
ko­ja
en­ja

Test­N1
2,000
2,000
2,000

Test­N2
3,000
3,000
3,000

Test­N3
204
230
668

Test­EP
1,151––

Table 1: Statistics for ASPEC

more Asian languages such as Vietnamese,
Lao and so on.

Table 2: Statistics for JPC• Evaluation method
Evaluation is done both automatically andmanually. Firstly, all submitted translation re­sults are automatically evaluated using three
metrics: BLEU, RIBES and AMFM. Amongthem, selected translation results are assessedby two kinds of human evaluation: pairwise
evaluation and JPO adequacy evaluation.

2

Tasks

2.1

ASPEC Task

uments with a 1­to­1 alignment are included. It
is therefore possible to restore the original docu­
ments. The format is the same as the training data
except that there is no similarity score.
2.1.2 ASPEC­JCASPEC­JC is a parallel corpus consisting ofJapanese scientific papers, which come from theliterature database and electronic journal site J­
STAGE by JST, and their translation to Chinesewith permission from the necessary academic as­sociations. Abstracts and paragraph units are se­lected from the body text so as to contain the high­est overall vocabulary coverage.
The development, development­test and test
data are extracted at random from documents con­taining single paragraphs across the entire corpus.
Each set contains 400 paragraphs (documents).
There are no documents sharing the same data
across the training, development, development­
test and test sets.

ASPEC was constructed by the Japan Science andTechnology Agency (JST) in collaboration with
the National Institute of Information and Com­
munications Technology (NICT). The corpus con­sists of a Japanese­English scientific paper abstract
corpus (ASPEC­JE), which is used for ja↔ensubtasks, and a Japanese­Chinese scientific paper
excerpt corpus (ASPEC­JC), which is used forja↔zh subtasks. The statistics for each corpus areshown in Table 1.
2.1.1 ASPEC­JE
The training data for ASPEC­JE was constructed
by NICT from approximately two million
Japanese­English scientific paper abstracts owned
by JST. The data is a comparable corpus andsentence correspondences are found automaticallyusing the method from Utiyama and Isahara(2007). Each sentence pair is accompanied bya similarity score calculated by the method anda field ID that indicates a scientific field. The
correspondence between field IDs and field names,
along with the frequency and occurrence ratios forthe training data, are described in the README
file of ASPEC­JE.
The development, development­test and test
data were extracted from parallel sentences fromthe Japanese­English paper abstracts that exclude
the sentences in the training data. Each dataset con­sists of 400 documents and contains sentences ineach field at the same rate. The document align­
ment was conducted automatically and only doc­

2.2 JPC Task
JPO Patent Corpus (JPC) for the patent tasks wasconstructed by the Japan Patent Office (JPO) in
collaboration with NICT. The corpus consists ofChinese­Japanese, Korean­Japanese and English­
Japanese patent descriptions whose International
Patent Classification (IPC) sections are chemistry,electricity, mechanical engineering, and physics.
At WAT2020, the patent tasks has two sub­tasks: normal subtask and expression pattern sub­task. Both subtasks use common training, develop­
ment and development­test data for each language
pair. The normal subtask for three language pairs
uses four test data with different characteristics:• test­N: union of the following three sets;• test­N1: patent documents from patent fami­lies published between 2011 and 2013;
2• test­N2: patent documents from patent fami­lies published between 2016 and 2017; andTraining
Test set I• test­N3: patent documents published be­
tween 2016 and 2017 where target sentences
are manually created by translating source
sentences.

Test set II

The expression pattern subtask for zh→ja pairuses test­EP data. The test­EP data consists ofsentences annotated with expression pattern cate­gories: title of invention (TIT), abstract (ABS),
scope of claim (CLM) or description (DES). The
corpus statistics are shown in Table 2. Note thattraining, development, development­test and test­
N1 data are the same as those used in WAT2017.
2.3

Test
DevTest
Dev
Test­2Dev­2Context for Test­2
Context for Dev­2

0.2 M sentence pairs
2,000 sentence pairs
2,000 sentence pairs
2,000 sentence pairs
1,912 sentence pairs
497 sentence pairs
567 article pairs
135 article pairs

Table 3: Statistics for JIJI Corpus

2.4 Mixed­domain Task
2.4.1 ALT and UCSY Corpus
The parallel data for Myanmar­English translation
tasks at WAT2020 consists of two corpora, the ALT
corpus and UCSY corpus.

Newswire (JIJI) Task

• The ALT corpus is one part from the Asian
Language Treebank (ALT) project (Rizaet al., 2016), consisting of twenty thousand
Myanmar­English parallel sentences fromnews articles.

The Japanese ↔ English newswire task uses JIJI
Corpus which was constructed by Jiji Press Ltd.
in collaboration with NICT and NHK. The corpus
consists of news text that comes from Jiji Press
news of various categories including politics, econ­omy, nation, business, markets, sports and so on.
The corpus is partitioned into training, develop­
ment, development­test and test data, which con­sists of Japanese­English sentence pairs. In addi­
tion to the test set (test set I) that has been pro­vided from WAT 2017, we added a new test set(test set II) with document­level context at WAT
2020. These test sets are as follows.

• The UCSY corpus (Yi Mon Shwe Sin andKhin Mar Soe, 2018) is constructed by the
NLP Lab, University of Computer Studies,
Yangon (UCSY), Myanmar. The corpus con­sists of 200 thousand Myanmar­English par­allel sentences collected from different do­mains, including news articles and textbooks.
The ALT corpus has been manually segmented
into words (Ding et al., 2018, 2019), and the UCSY
corpus is unsegmented. A script to tokenize the
Myanmar data into writing units is released withthe data. The automatic evaluation of Myanmartranslation results is based on the tokenized writ­ing units, regardless to the segmented words in the
ALT data. However, participants can make a useof the segmentation in ALT data in their own man­ner.
The detailed composition of training, develop­
ment, and test data of the Myanmar­English trans­
lation tasks are listed in Table 5. Notice that bothof the corpora have been modified from the data
used in WAT2018.

Test set I : A pair of test and reference sentences.
The references were automatically extracted
from English newswire sentences and not
manually checked. There are no context data.
Test set II : New test set added at WAT 2020. A
pair of test and reference sentences and con­text data that are articles including test sen­tences. The references were automatically ex­
tracted from English newswire sentences andmanually selected. Therefore, the quality of
the references of test set II is better than thatof test set I.
The statistics of JIJI Corpus are shown in Ta­
ble 3.
The definition of data use is shown in Table 4.
Participants submit the translation results of one
or more of the test data.
The sentence pairs in each data are identified
in the same manner as that for ASPEC using themethod from (Utiyama and Isahara, 2007).

2.4.2 ALT and ECCC Corpus
The parallel data for Khmer­English translation
tasks at WAT2020 consists of two corpora, the ALT
corpus and ECCC corpus.
• The ALT corpus is one part from the Asian
Language Treebank (ALT) project (Riza et al.,
3

Task

Use
Training
Test set I

Japanese to English

Test set II
Training
Test set I

English to Japanese

Test set II

Content
Training, DevTest, Dev, Dev­2, context for Dev2
Test in Japanese
Test in English
Test­2 in JapaneseContext in Japanese for Test­2
Test­2 in English
Training, DevTest, Dev, Dev­2, context for Dev2
Test in English
Test in JapaneseTest­2 in English
Context in English for Test­2
Test­2 in Japanese

To be translated
Reference
Test­2
Context
Reference
To be translated
Reference
To be translated
Context in English for Test­2
Reference

Table 4: Definition of data use in the Japanese ↔ English newswire task
Corpus
ALT
UCSY
All

Train
18,088204,539
222,627

Dev
1,000
–
1,000

Test
1,018
–
1,018

Split
Train
Dev

Table 5: Statistics for the data used in Myanmar­
English translation tasks
Corpus
ALT
ECCC
All

Train
18,088104,660
122,748

Dev
1,000
–
1,000

Test

Domain
ALT
IT
ALT
IT
ALT
IT

Hi
254,242
2,016
2,073

Language Pair
Id
Ms
18,088
158,472 506,739
1,000
2,023
2,050
1,018
2,037
2,050

Th
74,4972,049
2,050

Table 7: The NICT­SAP task corpora splits. The cor­pora belong to two domains: wikinews (ALT) and soft­
ware documentation (IT). The Wikinews corpora are N­
way parallel.

Test
1,018
–
1,018

Table 6: Statistics for the data used in Khmer­English
translation tasks

2.5 NICT­SAP Task
This year, we created a new task for joint multi­
domain multilingual neural machine translation in­volving 4 low­resource Asian languages: Thai(Th), Hindi (Hi), Malay (Ms), Indonesian (Id). En­
glish (En) is the source or the target language forthe translation directions being evaluated. The pur­pose of this task was to test the feasibility of multi­
domain multilingual solutions for extremely low­
resource language pairs and domains. Naturally
the solutions could be one­to­many, many­to­one
or many­to­many NMT models. The domains inquestion are Wikinews and IT (specifically, Soft­ware Documentation). The total number of evalu­
ation directions are 16 (8 for each domain). Thereis very little clean and publicly available data forthese domains and language pairs and thus we en­couraged participants to not only utilize the small
Asian Language Treebank (ALT) parallel corpora(Thu et al., 2016) but also the parallel corpora
from OPUS1 . The ALT dataset contains 18,088,
1,000 and 1,018 training, development and test­
ing sentences. As for corpora for the IT domain
we only provided evaluation (dev and test sets)2016), consisting of twenty thousand Khmer­
English parallel sentences from news articles.
• The ECCC corpus consists of 100 thousand
Khmer­English parallel sentences extracted
from document pairs of Khmer­English bi­
lingual records in Extraordinary Chambers in
the Court of Cambodia, collected by National
Institute of Posts, Telecoms & ICT, Cambo­
dia.
The ALT corpus has been manually segmented
into words (Ding et al., 2018), and the ECCC
corpus is unsegmented. A script to tokenize the
Khmer data into writing units is released with withthe data. The automatic evaluation of Khmer trans­
lation results is based on the tokenized writing
units, regardless to the segmented words in the
ALT data. However, participants can make a useof the segmentation in ALT data in their own man­ner.
The detailed composition of training, develop­
ment, and test data of the Khmer­English transla­tion tasks are listed in Table 6.

14

http://opus.nlpl.eu/

Lang.pair
Ja↔Ru

Ja↔En

Ru↔En

Table 8:task.

Partition
train
development
test
train
development
test
train
development
test#sent.
12,356486
600
47,082589
600
82,072313
600#tokens
341k / 229k
16k / 11k
22k / 15k
1.27M / 1.01M
21k / 16k
22k / 17k
1.61M / 1.83M
7.8k / 8.4k
15k / 17k

#types
22k / 42k
2.9k / 4.3k
3.5k / 5.6k
48k / 55k
3.5k / 3.8k
3.5k / 3.8k
144k / 74k
3.2k / 2.3k
5.6k / 3.8k

2.7 Indic Multilingual Task
In 2018, we had organized an Indic languages task
(Nakazawa et al., 2018) but due to lack of reli­able evaluation corpora we discontinued it in WAT
2019. However, in 2020, high quality publicly
available evaluation (and training) corpora became
available which motivated us to relaunch the task.
The Indic task involves mixed domain corpora forevaluation consisting of various articles composed
by Indian Prime Minister. The languages involved
are Hindi (Hi), Marathi (Mr), Tamil (Ta), Telugu(Te), Gujarati (Gu), Malayalam (Ml), Bengali (Bg)and English (En). English is either the sourceor the target language during evaluation leading
to a total of 14 translation directions. The objec­
tive of this task, like the Indic languages task in
2018, was to evaluate the performance of multilin­
gual NMT models. The desired solution could beone­to­many, many­to­one or many­to­many NMT
models. We provided a filtered version of the PM
India dataset13 and further encouraged the use of
the CVIT­PIB dataset14 . Our organizer’s baselines
used the PMI and PIB corpora for training. De­
tailed statistics for the aforementioned corpora canbe found in Table 9. We also listed additionalsources of corpora for participants to use. See Ap­
pendix B for details.

In­Domain data for the Russian–Japanese

corpora2 (Buschbeck and Exel, 2020) and encour­
aged participants to consider GNOME, UBUNTU
and KDE corpora from OPUS. In Table 7 we give
statistics of the aforementioned corpora which weused for the organizer’s baselines. Note that we do
not list3 all available corpora here and participants
were not restricted from using any corpora as longas they are freely available.
2.6

News Commentary Task

For the Russian↔Japanese task we asked partic­
ipants to use the JaRuNC corpus4 (Imankulova
et al., 2019) which belongs to the news commen­tary domain. This dataset was manually aligned
and cleaned and is trilingual. It can be used
to evaluate Russian↔English translation quality
as well but this is beyond the scope of thisyears sub­task. Refer to Table 8 for the statis­
tics of the in­domain parallel corpora. In addi­
tion we encouraged the participants to use out­of­domain parallel corpora from various sources suchas KFTT,5 JESC,6 TED,7 ASPEC,8 UN,9 Yan­
dex10 and Russian↔English news­commentary
corpus11 . This year we also encouraged partici­
pants to use any corpora from WMT 2020 12 in­volving Japanese, Russian and English as long as
it did not belong to the news commentary domain
to prevent any test set sentences from being inten­
tionally seen during training.

2.8 UFAL (EnOdia) Task
This task introduced this year at WAT2020
and the first Odia↔English machine translation
shared task running in any conference. For
Odia↔English translation task we asked the partic­
ipants to use OdiEnCorp 2.0 (Parida et al., 2020).15
The statistics of the corpus are given in Table 10.
2.9 English→Hindi Multi­Modal Task
For English→Hindi multi­modal translation task
we asked the participants to use the updated ver­sion 1.1 of Hindi Visual Genome corpus (HVG,
Parida et al., 2019a,b).16 The update consisted incorrecting primarily the Hindi, i.e. the target sideof the corpus.
The statistics of HVG 1.1 are given in Table 11.
One “item” in HVG consists of an image with a2Software Domain Evaluation Splits
http://lotus.kuee.kyoto­u.ac.jp/WAT/NICT­SAP­Task
4
https://github.com/aizhanti/JaRuNC
5
http://www.phontron.com/kftt/
6
https://datarepository.wolframcloud.com/
resources/Japanese-English-Subtitle-Corpus
7
https://wit3.fbk.eu/
8
http://lotus.kuee.kyoto-u.ac.jp/ASPEC/9https://cms.unov.org/UNCorpus/
10
https://translate.yandex.ru/corpus?lang=en11
http://lotus.kuee.kyoto-u.ac.jp/WAT/
News-Commentary/news-commentary-v14.en-ru.
filtered.tar.gz
12
http://www.statmt.org/wmt20/translation­task.html
313

http://lotus.kuee.kyoto-u.ac.jp/WAT/
indic-multilingual/cvit-pmindia-mono-bi.zip
14
http://preon.iiit.ac.in/~jerin/resources/
datasets/pib_v0.2.tar
15
https://lindat.mff.cuni.cz/repository/
xmlui/handle/11234/1-3211
16
https://lindat.mff.cuni.cz/repository/
xmlui/handle/11234/1-3267

5

Split
Train
Dev
Test

Bn
74,593

Gu
73,504

Hi
247,926

3,522

4,463

3,169

LanguageMlMr
61,678 112,429
2,000
2,886
3,760

Ta
122,337

Te
41,741

3,637

3,049

Table 9: The Indic task corpora splits. The training corpora statistics are the result of combining the PIB and PMI
corpora. While the number of development set sentences are the same, they are not N­way parallel as in the caseof the Wikinews corpora.

Dataset
Train
Dev
Test

Items
69,37013,544
14,344

TokensEnglish
Odia
1.34M
1.16M
158,188 140,726
186,320 165,274

DatasetTraining Set
D­TestE­Test (EV)C­Test (CH)Items
28,930
998
1,595
1,400

TokensEnglishHindi
143,164 145,448
4,922
4,978
7,853
7,852
8,186
8,639

Table 10: Statistics of OdiEnCorp 2.0 Corpus.
Table 11: Statistics of Hindi Visual Genome 1.1 used
for the English→Hindi Multi­Modal translation task.
One item consists of a source English sentence, target
Hindi sentence, and a rectangular region within an im­age. The total number of English and Hindi tokens in
the dataset also listed. The abbreviations EV and CH
are used in the official task names in WAT scoring ta­
bles.

rectangular region highlighting a part of the im­age, the original English caption of this region andthe Hindi reference translation. Depending on the
track (see 2.9.1 below), some of these item compo­
nents are available as the source and some serve as
the reference or play the role of a competing can­didate solution.
Since HVG 1.0 was used already in WAT 2019,
all the data were publicly available before WAT
2020. We instructed the participants to use only
the Training and D­Test sections and avoid using
E­Test and C­Test which are the official test sets
for the task this year.
The English→Hindi multi­modal task includes
three tracks as illustrated in Figure 1:

Data
Train
Dev
Test

Images29,7831,000
1,000

Sentences/Tokens
EnglishJapanese
59,566/1.03M 59,562∗ /0.99M
2,000/34,670
2,000/33,022
1,000/10,876
1,000/17,731

Table 12:
Statistics of the dataset used forJapanese↔English multi­modal tasks.
Here weuse the MeCab tokenizer to count Japanese tokens.
∗
Four of the original English sentences are actuallybroken so we did not provide their translations.

2.9.1

English→Hindi Multi­Modal Task
Tracks
1. Text­Only Translation (labeled “TEXT” in
WAT official tables): The participants are
asked to translate short English captions (text)
into Hindi. No visual information can be used.
On the other hand, additional text resources
are permitted (but they need to be specified in
the corresponding system description paper).

2.10 Japanese↔English Multi­Modal Tasks
The goal of Japanese↔English multi­modal task17
is to improve translation performance with the helpof another modality (images) associated with in­put sentences. For both English→Japanese andJapanese→English tasks, we use the Flickr30k En­
tities Japanese (F30kEnt­Jp) dataset (Nakayama
et al., 2020). This is an extended dataset of
the Flickr30k18 and Flickr30k Entities19 datasets
where manual Japanese translations are added. No­
tably, it has the annotations of many­to­manyphrase­to­region correspondences in both English
and Japanese captions, which are expected tostrongly supervise multimodal grounding and pro­vide new research directions.
We summarize the statistics of our dataset in Ta­

2. Hindi Captioning (labeled “HI”): The partici­
pants are asked to generate captions in Hindi
for the given rectangular region in an input
image.
3. Multi­Modal Translation (labeled “MM”):
Given an image, a rectangular region in it andan English caption for the rectangular region,
the participants are asked to translate the En­
glish text into Hindi. Both textual and visual
information can be used.

17

https://nlab­mpg.github.io/wat2020­mmt­jp/
http://shannon.cs.illinois.edu/DenotationGraph/19
http://bryanplummer.com/Flickr30kEntities/
18

6

Text­Only MT

Hindi Captioning

Image
Source Text–
man in the middle of tennis –court

System Output
Glossआकाश में सफेद बादलटे िनस कोटर् के बीच में आदमीMan in the middle of a tennis White cloud on the sky
court

Reference Solution टे िनस कोटर् के बीच में मनुष्य
Gloss

Multi­Modal MT

man in a white hat on the tennis
courtफोटो पर तारीख की मोहर।

Man in the middle of a tennis Date stamp on the photocourtटे िनस कोटर् पर व्यिक्त
Man on the tennis courtटे िनस कोटर् पर एक सफेद टोपी मेंआदमीMan in a white cap on the tennis
court

Figure 1: An illustration of the three tracks of WAT 2020 English→Hindi Multi­Modal Task. Note the missing
articles in the English source. The correct sentence would be “A man in the middle of a tennis court”. The system
outputs are not correct for the Hindi captioning and multimodal with respect to the reference solution, although theoutput of the captioning system is understandable.

ble 12. We use the same splits of training, valida­tion and test data specified in Flickr30k Entities.
For the training and the validation data, we use the
F30kEnt­Jp version 1.0 which are publicly avail­able.20 While the original Flickr30k has five En­
glish sentences for each image, our Japanese set
has the translations of the first two sentences of
each. Therefore, we have two parallel sentences
for each image. For the test data, we use theJapanese sentences not included in the version 1.0
dataset (i.e., one of the other three sentences foreach image) which are not publicly available at thetime of WAT 2020. Note that phrase­to­region an­
notation is not included in the test data.
There are two settings of submission: with andwithout resource constraints. In the constrained
setting, external resources such as additional dataand pre­trained models (with external data) are
not allowed to use, except for pre­trained convo­
lutional neural networks (for visual analysis) andbasic linguistic tools such as taggers, parsers, andmorphological analyzers. As the baseline system
to compute the Pairwise score, we implement the
text­only model in (Nishihara et al., 2020) under
the constrained setting.

2.11.1 Document­level Scientific Paper
Translation
Traditional ASPEC translation tasks are sentence­level and the translation quality of them seem tobe saturated. We think it’s high time to move on to
document­level evaluation. For the first year, we
use ParaNatCom 21 (Parallel English­Japanese ab­stract corpus made from Nature Communications
articles) for the development and test sets of theDocument­level Scientific Paper Translation sub­task. We cannot provide document­level training
corpus, but you can use ASPEC and any other ex­
tra resources.
2.11.2 Document­level Business Scene
Dialogue Translation
There are a lot of ready­to­use parallel corpora
for training machine translation systems, however,most of them are in written languages such as
web crawl, news­commentary, patents, scientificpapers and so on. Even though some of the paral­
lel corpora are in spoken language, they are mostly
spoken by only one person (TED talks) or containa lot of noise (OpenSubtitle). Most of other MT
evaluation campaigns adopt the written language,
monologue or noisy dialogue parallel corpora for
their translation tasks. Traditional ASPEC trans­
lation tasks are sentence­level and the translation
quality of them seem to be saturated. We think it’shigh time to move on to document­level evaluation.

2.11 Document­level Translation Task
In WAT2020, we set up 2 document­level transla­tion tasks: ParaNatCom and BSD.
20

21
http://www2.nict.go.jp/astrec-att/member/mutiyama/paranatcom/

https://github.com/nlab­mpg/Flickr30kEnt­JP

7

Lang
hi­en
hi

Train
1,609,682
–

Dev
520
–

Test
2,507
–

Mono
–
45,075,279

systems were published on the WAT web page.23
We also have SMT baseline systems for the tasks
that started at WAT 2017 or before 2017. The base­line systems are shown in Tables 16, 17, and 18.
SMT baseline systems are described in the WAT
2017 overview paper (Nakazawa et al., 2017). The
commercial RBMT systems and the online transla­tion systems were operated by the organizers. We
note that these RBMT companies and online trans­
lation companies did not submit themselves. Be­
cause our objective is not to compare commercial
RBMT systems or online translation systems fromcompanies that did not themselves participate, thesystem IDs of these systems are anonymous in thispaper.

Table 13: Statistics for IITB Corpus. “Mono” indicatesmonolingual Hindi corpus.

For the first year, WAT uses BSD Corpus 22 (TheBusiness Scene Dialogue corpus) for the dataset
including training, development and test data. Par­
ticipants of this taks must get a copy of BSD corpus
by themselves.
2.12

IITB Hindi–English task

In this task we use IIT Bombay English­Hindi Cor­pus (Kunchukuttan et al., 2018) which contains
English­Hindi parallel corpus as well as mono­
lingual Hindi corpus collected from a variety ofsources and corpora (Bojar et al., 2014). This cor­pus had been developed at the Center for Indian
Language Technology, IIT Bombay over the years.
The corpus is used for mixed domain tasks hi↔en.
The statistics for the corpus are shown in Table 13.

3

4.1 Tokenization
We used the following tools for tokenization.
4.1.1 For ASPEC, JPC, TDDC, JIJI, ALT,UCSY, ECCC, and IITB
• Juman version 7.024 for Japanese segmenta­tion.
• Stanford Word Segmenter version 2014­01­0425 (Chinese Penn Treebank (CTB) model)
for Chinese segmentation.
• The Moses toolkit for English and Indonesian
tokenization.
• Mecab­ko26 for Korean segmentation.
• Indic NLP Library27 (Kunchukuttan, 2020)
for Indic language segmentation.
• The tools included in the ALT corpus for
Myanmar and Khmer segmentation.
• subword­nmt28 for all languages.

Participants

Table 14 shows the participants in WAT2020. The
table lists 14 organizations from various countries,
including Japan, India, Singapore, China, Ireland,
and Switzerland.
493 translation results by 20 teams were submit­
ted for automatic evaluation and about 121 trans­
lation results by 14 teams were submitted for thehuman evaluation. Table 15 shows tasks for which
each team submitted results by the deadline. The
human evaluation was conducted only for the tasks
with the check marks in ”human eval” line.

4

When we built BPE­codes, we merged source andtarget sentences and we used 100,000 for ­s op­
tion. We used 10 for vocabulary­threshold whensubword­nmt applied BPE.

Baseline Systems

Human evaluations of most of WAT tasks wereconducted as pairwise comparisons between thetranslation results for a specific baseline systemand translation results for each participant’s sys­tem. That is, the specific baseline system was thestandard for human evaluation. At WAT 2020, we
adopted a neural machine translation (NMT) withattention mechanism as a baseline system.
The NMT baseline systems consisted of pub­
licly available software, and the procedures forbuilding the systems and for translating using the22

4.1.2 For News Commentary• The Moses toolkit for English and Russian
only for the News Commentary data.
23
http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2020/baseline/baselineSystems.html
24
http://nlp.ist.i.kyoto-u.ac.jp/EN/index.
php?JUMAN
25
http://nlp.stanford.edu/software/segmenter.
shtml
26
https://bitbucket.org/eunjeon/mecab-ko/
27
https://github.com/anoopkunchukuttan/indic_
nlp_library28
https://github.com/rsennrich/subword-nmt

https://github.com/tsuruoka-lab/BSD

8

Team ID
TMU
NICT­5*cvit­mt
NHK­NESODIANLP
Kyoto­U+ECNU
goku20WT
HW­TSCut­mrt*iiitscadapt­dcu
DEEPNLP
CNLP­NITS

Organization
Tokyo Metropolitan University
NICT
IIIT Hyderabad
NHK & NHK Engineering System
Idiap Research InstituteKyoto University and East China Normal University
Rakuten Institute of Technology Singapore, Rakuten Asia.
Wipro
Huawei Translation Services Center
The University of Tokyo
Indian Institute of Information Technology
Dublin City University
TATA CONSULTANCY SERVICES
National Institute of Technology Silchar

Country
Japan
Japan
India
Japan
Switzerland
Japan and China
Singapore
India
China
Japan
India
Ireland
India
India

Table 14: List of participants who submitted translations for the human evaluation in WAT2020 (Note: teams
with ’*’ marks did not submit their system description papers, therefore the evaluation results are UNOFFICIAL
according to our policy)

Team ID
TMU
NHK­NESODIANLP
Kyoto­U+ECNU
goku20*HW­TSC*iiitsc
CNLP­NITS
human eval

ASPEC
CJ JC

✓

✓

✓

✓

EJ

✓

✓

JE

✓

✓

JPC
CJ JC

✓

✓

✓

✓

KJ
✓

✓

✓

JK

JIJI
JE
✓

Multimodal
En­Hi
En­Ja
TX HI MM EJ JE
✓
✓
✓

✓

✓

✓

✓

✓
✓
✓

✓

✓
✓

✓

✓

✓

✓

Indic Multilingual
En­BnBn­EnEn­HiHi­EnEn­MlMl­EnEn­TaTa­EnEn­TeTe­EnEn­GuGu­EnEn­MrMr­En
Team ID
NICT­5
✓
✓
✓
✓
✓
✓
✓
*cvit­mt
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
ODIANLP
HW­TSC
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
human eval✓✓✓✓Team ID
NICT­5*cvit­mt
ODIANLP
goku20WT
ut­mrtadapt­dcu
DEEPNLP
human eval

UFAL EnOdia
En­Od Od­En
✓
✓

✓

✓
✓

✓

BSD
HindenMultilingual Multi­domain (IT/Wikinews)
EJ JE En­HiHi­EnEn­HiHi­EnEn­ThTh­EnEn­MsMs­EnEn­InIn­En✓/✓ ✓/✓ ✓/✓ ­/✓ ✓/✓ ✓/✓ ✓/✓ ✓/✓
✓

✓

✓

✓
✓
✓
✓

✓
✓

✓

✓

✓

✓

Table 15: Submissions for each task by each team. E, J, C, and K denote English, Japanese, Chinese, and Korean
respectively. The human evaluation was conducted only for the tasks with the check marks in ”human eval” line.

9System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT X
Online X
Online X
AIAYN

Table 17: Baseline Systems II

Table 16: Baseline Systems I

10System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT X
Online X
Online X

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
Other
Other
NMT
✓
✓
✓

✓
✓
✓

ja­en
✓
✓
✓
✓

SystemOpenNMT’s NMT with attention
Moses’ Phrase­based SMT
Moses’ Hierarchical Phrase­based SMT
Moses’ String­to­Tree Syntax­based SMT and Berkeley parser
Moses’ Tree­to­String Syntax­based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
PC­Transer V13 (Commercial system)
Google translate
Bing translator

SystemOpenNMT’s NMT with attention
Moses’ Phrase­based SMT
Moses’ Hierarchical Phrase­based SMT
Moses’ String­to­Tree Syntax­based SMT and Berkeley parser
Moses’ Tree­to­String Syntax­based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
ATLAS V14 (Commercial system)
PAT­Transer 2009 (Commercial system)
PC­Transer V13 (Commercial system)
J­Beijing 7 (Commercial system)
Hohrai 2011 (Commercial system)
J Soul 9 (Commercial system)
Korai 2011 (Commercial system)
Google translate
Bing translator
Google’s implementation of “Attention Is All You Need”

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
Other
Other

✓
✓
✓

✓
✓

✓
✓

✓

zh­ja
✓
✓
✓

JIJI
ja­en en­ja
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓

✓
✓

ASPEC
en­ja ja­zh
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓

✓
✓
✓
✓

en­ja
✓
✓
✓

✓
✓

✓

JPC
ja­zh
✓
✓
✓
✓

✓

✓

ALT
my↔en km↔en
✓
✓

✓
✓

✓
✓
✓

ja­en
✓
✓
✓
✓

✓
✓

✓
✓

✓

zh­ja
✓
✓
✓

✓
✓
✓
✓

ja↔ko
✓
✓
✓

Table 18: Baseline Systems III

11

System ID
NMT
NMT T2T
NMT OT
MNMT

SystemOpenNMT’s NMT with attention
Tensor2Tensor’s Transformer
OpenNMT­py’s Transformer
Multimodal NMT

Type
NMT
NMT
NMT
NMT

IT&Wikinews{hi,id,ms,th}↔en✓

NewsCommentary
ru↔ja
✓
✓

od↔en
✓

Indic
{bn,hi,gu,ml,mr,ta,te}­en✓✓Multimodalen­hi ja↔en• Mecab29 for Japanese segmentation.

• encoder_type = brnn
• brnn_merge = concat• src_seq_length = 150• tgt_seq_length = 150• src_vocab_size = 100000• tgt_vocab_size = 100000• src_words_min_frequency = 1• tgt_words_min_frequency = 1• Corpora are further processed by ten­sor2tensor’s internal pre/post­processing
which includes sub­word segmentation.
4.1.3 Indic and NICT­SAP Tasks• For the Indic task we did not perform any ex­plicit tokenization of the raw data.

The default values were used for the other systemparameters.
We used the following data for training the NMT
baseline systems of NMT with attention.

• For the NICT­SAP task we only character seg­
mented the Thai corpora as it was the only lan­
guage for which character level BLEU was tobe computed. Other languages corpora were
not preprocessed in any way.

• All of the training data mentioned in Section 2
were used for training except for the AS­
PEC Japanese–English task. For the ASPEC
Japanese–English task, we only used train­
1.txt, which consists of one million parallel
sentence pairs with high similarity scores.
• All of the development data for each task wasused for validation.

• Any subword segmentation or tokenization
was handled by the internal mechanisms of
tensor2tensor.
4.1.4

For English→Hindi Multi­Modal and
UFAL EnOdia Tasks• Hindi Visual Genome 1.1 and OdiEnCorp 2.0
comes untokenized and we did not use or rec­ommend any specific external tokenizer.

4.2.2 Transformer (Tensor2Tensor)
For the News Commentary task, we used ten­sor2tensor’s30 implementation of the Transformer(Vaswani et al., 2017) and used default hyperpa­
rameter settings corresponding to the “base” model
for all baseline models. The baseline for theNews Commentary task is a multilingual model
as described in Imankulova et al. (2019) which istrained using only the in­domain parallel corpora.
We use the token trick proposed by (Johnson et al.,2017) to train the multilingual model.
As for the Indic and NICT­SAP tasks, we used
tensor2tensor to train many­to­one and one­to­many models where the latter were trained withthe aforementioned token trick. We used default
hyperparameter settings corresponding to the “big”
model. Since the NICT­SAP task involves two do­mains for evaluation (Wikinews and IT) we useda modification of the token trick technique for do­main adaptation to distinguish between corpora fordifferent domains. In our case we used tokens
such as 2alt and 2it to indicate whether the sen­tences belonged to the Wikinews or IT domain, re­spectively. For both tasks we used 32,000 sepa­
rate sub­word vocabularies. We trained our mod­els on 1 GPU till convergence on the development
set BLEU scores, averaged the last 10 checkpoints• The standard OpenNMT­py sub­word seg­mentation was used for pre/post­processing
for the baseline system and each participant
used what they wanted.
4.1.5

For English↔Japanese Multi­Modal
Tasks• For English sentences, we applied lowercase,punctuation normalization, and the Moses to­
kenizer.
• For Japanese sentences, we used KyTea forword segmentation.

4.2

Baseline NMT Methods

We used the following NMT with attention for
most of the tasks. We used Transformer (Ten­sor2Tensor, Vaswani et al., 2017) for the News
Commentary and English↔Tamil tasks and Trans­
former (OpenNMT­py) for the Multimodal task.
4.2.1 NMT with Attention
We used OpenNMT (Klein et al., 2017) as theimplementation of the baseline NMT systems of
NMT with attention (System ID: NMT). We used
the following OpenNMT configuration.
29

30
https://github.com/tensorflow/
tensor2tensor

https://taku910.github.io/mecab/

12

(separated by 1000 batches) and performed decod­ing with a beam of size 4 and a length penalty of
0.6.
4.2.3

Before the calculation of the automatic evalua­tion scores, the translation results were tokenizedor segmented with tokenization/segmentation
tools for each language. For Japanese segmenta­
tion, we used three different tools: Juman version
7.0 (Kurohashi et al., 1994), KyTea 0.4.6 (Neubig
et al., 2011) with full SVM model33 and MeCab
0.996 (Kudo, 2005) with IPA dictionary 2.7.0.34
For Chinese segmentation, we used two different
tools: KyTea 0.4.6 with full SVM Model in MSR
model and Stanford Word Segmenter (Tseng,2005) version 2014­06­16 with Chinese Penn
Treebank (CTB) and Peking University (PKU)
model.35 For Korean segmentation, we usedmecab­ko.36 For Myanmar and Khmer segmen­
tations, we used myseg.py37 and kmseg.py38 .
For English and Russian tokenizations, we usedtokenizer.perl39 in the Moses toolkit. For
Indonesian and Malay tokenizations, we usedtokenizer.perl as same as the English tok­
enization. For Thai tokenization, we segmented
the whole character separately. For Bengali,
Gujarati, Hindi, Marathi, Malayalam, Odia, Tamil,
and Telugu tokenizations, we used Indic NLP
Library40 (Kunchukuttan, 2020). The detailed
procedures for the automatic evaluation are shownon the WAT2020 evaluation web page.41

Transformer (OpenNMT­py)

For the English→Hindi Multimodal and UFAL
EnOdia tasks, we used the Transformer model(Vaswani et al., 2018) as implemented in
OpenNMT­py (Klein et al., 2017) and used the“base” model with default parameters for themulti­modal task baseline. We have generated thevocabulary of 32k sub­word types jointly for boththe source and target languages. The vocabulary
is shared between the encoder and decoder.
4.2.4

Multimodal Transformer with
Supervised Attention

As the baselines for the English↔Japanese Mul­
timodal tasks, we implement five models de­scribed in (Nishihara et al., 2020): (i) the text­
only Transformer NMT model, (ii) the multi­
modal Transformer NMT model (MNMT) which
incorporates the ResNet50 convolutional neu­
ral network to extract visual features, (iii) the
MNMT model with the supervised visual atten­
tion mechanism (MNMT+SVA), (iv) the MNMT
model with the supervised cross­lingual atten­
tion mechanisms (MNMT+SCA), and (v) the
MNMT model with the both supervised attentions(MNMT+SVA+SCA). For training with the super­
vised attention mechanisms, we utilize the phrase­to­phrase and phrase­to­region annotations avail­able in the training dataset.

5

Automatic Evaluation

5.1

Procedure for Calculating Automatic
Evaluation Score

5.2 Automatic Evaluation System
The automatic evaluation system receives transla­tion results by participants and automatically gives
evaluation scores to the uploaded results. As
shown in Figure 2, the system requires participants
to provide the following information for each sub­mission:• Human Evaluation: whether or not they sub­mit the results for human evaluation;

We evaluated translation results by three met­
rics: BLEU (Papineni et al., 2002), RIBES(Isozaki et al., 2010) and AMFM (Banchs et al.,2015).
BLEU scores were calculated using
multi-bleu.perl in the Moses toolkit (Koehn
et al., 2007). RIBES scores were calculated using
RIBES.py version 1.02.4.31 AMFM scores were
calculated using scripts created by the technicalcollaborators listed in the WAT2020 web page.32
All scores for each task were calculated using thecorresponding reference translations.

33

http://www.phontron.com/kytea/model.html
http://code.google.com/p/mecab/downloads/
detail?name=mecab-ipadic-2.7.0-20070801.tar.gz
35
http://nlp.stanford.edu/software/segmenter.
shtml
36
https://bitbucket.org/eunjeon/mecab-ko/37
http://lotus.kuee.kyoto-u.ac.jp/WAT/
my-en-data/wat2020.my-en.zip38
http://lotus.kuee.kyoto-u.ac.jp/WAT/
km-en-data/km-en.zip
39
https://github.com/moses-smt/mosesdecoder/tree/RELEASE-2.1.1/scripts/tokenizer/
tokenizer.perl
40
https://github.com/anoopkunchukuttan/indic_
nlp_library41
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/index.html
34

31

http://www.kecl.ntt.co.jp/icl/lirg/ribes/
index.html
32
lotus.kuee.kyoto-u.ac.jp/WAT/WAT2020/

13

WAT

The Workshop on Asian Translation
Submission
SUBMISSION
Logged in as: ORGANIZER
LogoutSubmission:
Human
Evaluation:

human evaluation

Publish theresults of theevaluation:

publishTeam Name:

ORGANIZER

Task:

en-ja

Submission
File:

ファイルを選択 選択されていません

Used Other
Resources:used other resources such as parallel corpora, monolingual corpora andparallel dictionaries in addition to oﬃcial corpora

Method:SMT

SystemDescription(public):

100
charactersor less

SystemDescription(private):100
charactersor less

SubmitGuidelines for submission:
System requirements:
The latest versions of Chrome, Firefox, Internet Explorer and Safari are supported for this site.
Before you submit ﬁles, you need to enable JavaScript in your browser.
File format:
Submitted ﬁles should NOT be tokenized/segmented. Please check the automatic evaluation procedures.
Submitted ﬁles should be encoded in UTF-8 format.
Translated sentences in submitted ﬁles should have one sentence per line, corresponding to each test sentence. The number of lines in the submitted ﬁle and that of thecorresponding test ﬁle should be the same.
Tasks:
en-ja, ja-en, zh-ja, ja-zh indicate the scientiﬁc paper tasks with ASPEC.
HINDENen-hi, HINDENhi-en, HINDENja-hi, and HINDENhi-ja indicate the mixed domain tasks with IITB Corpus.
JIJIen-ja and JIJIja-en are the newswire tasks with JIJI Corpus.
RECIPE{ALL,TTL,STE,ING}en-ja and RECIPE{ALL,TTL,STE,ING}ja-en indicate the recipe tasks with Recipe Corpus.
ALTen-my and ALTmy-en indicate the mixed domain tasks with UCSY and ALT Corpus.
INDICen-{bn,hi,ml,ta,te,ur,si} and INDIC{bn,hi,ml,ta,te,ur,si}-en indicate the Indic languages multilingual tasks with Indic Languages Multilingual Parallel Corpus.
JPC{N,N1,N2,N3,EP}zh-ja ,JPC{N,N1,N2,N3}ja-zh, JPC{N,N1,N2,N3}ko-ja, JPC{N,N1,N2,N3}ja-ko, JPC{N,N1,N2,N3}en-ja, and JPC{N,N1,N2,N3}ja-en indicate the patent tasks
with JPO Patent Corpus. JPCN1{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} are the same tasks as JPC{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} in WAT2015-WAT2017. AMFM is not calculated
for JPC{N,N2,N3} tasks.
Human evaluation:
If you want to submit the ﬁle for human evaluation, check the box "Human Evaluation". Once you upload a ﬁle with checking "Human Evaluation" you cannot change the ﬁle used
for human evaluation.
When you submit the translation results for human evaluation, please check the checkbox of "Publish" too.
You can submit two ﬁles for human evaluation per task.
One of the ﬁles for human evaluation is recommended not to use other resources, but it is not compulsory.
Other:
Team Name, Task, Used Other Resources, Method, System Description (public) , Date and Time(JST), BLEU, RIBES and AMFM will be disclosed on the Evaluation Site when you
upload a ﬁle checking "Publish the results of the evaluation".
You can modify some ﬁelds of submitted data. Read "Guidelines for submitted data" at the bottom of this page.
Back to top

Figure 2: The interface for translation results submission• Publish the results of the evaluation: whether
or not they permit to publish automatic evalu­
ation scores on the WAT2020 web page;

ing SMT, RBMT, SMT and RBMT, EBMT,NMT and Other.
Evaluation scores of translation results that partic­
ipants permit to be published are disclosed via the
WAT2020 evaluation web page. Participants canalso submit the results for human evaluation using
the same web interface.
This automatic evaluation system will remain
available even after WAT2020. Anybody can reg­

• Task: the task you submit the results for;
• Used Other Resources: whether or not theyused additional resources; and• Method:

the type of the method includ­14ister an account for the system by the procedures
described in the registration web page.42
5.3

6.1 Pairwise Evaluation
We conducted pairwise evaluation for participants’systems submitted for human evaluation. The sub­mitted translations were evaluated by a profes­
sional translation company and Pairwise scores
were given to the submissions by comparing withbaseline translations (described in Section 4).

Additional Automatic Scores inMulti­Modal and UFAL EnOdia Tasks

For the multi­modal task and UFAL EnOdia task,
several additional automatic metrics were run
aside from the WAT evaluation server, namely:
BLEU (this time calculated by Moses scorer43 ),characTER (Wang et al., 2016), chrF3 (Popović,
2015), TER (Snover et al., 2006), WER, PER and
CDER (Leusch et al., 2006). Except for chrF3 andcharacTER, we ran Moses tokenizer44 on the can­didate and reference before scoring. For all error
metrics, i.e. metrics where better scores are lower,
we reverse the score by taking 1 − x and indicate
this by prepending “n” to the metric name. With
this modification, higher scores always indicate abetter translation result. Also, we multiply all met­
ric scores by 100 for better readability.
These additional scores document again, that
BLEU implementations (and the underlying tok­
enization schemes) heavily vary in their outcomes.
The scores are thus comparable only within eachof the metric variation, even if it is supposed to be
the same “BLEU”. In Table 22, we highlight witha special symbol whenever the ranking in one of
the metrics differs from the top­to­bottom sortingof the scores. Last year, a number of these met­
ric, including our BLEU vs. official WAT BLEU
(BLEUw in Table 22) lead to varying rankings.
This year, the system differences are probably suf­
ficiently big in the optics of these metrics that only
nCharacTER in the E­Test text­only (“EV TEXT”)
scoring differs.

6

6.1.1 Sentence Selection and Evaluation
For the pairwise evaluation, we randomly selected
400 sentences from the test set of each task. We
used the same sentences as the last year for thecontinuous subtasks. Baseline and submitted trans­
lations were shown to annotators in random orderwith the input source sentence. The annotators
were asked to judge which of the translations is bet­ter, or whether they are on par.
6.1.2 Voting
To guarantee the quality of the evaluations, eachsentence is evaluated by 5 different annotators andthe final decision is made depending on the 5 judge­
ments. We define each judgement ji (i = 1, · · · , 5)as:

if better than the baseline 1−1 if worse than the baselineji =
0if the quality is the same
The final∑ decision D is defined as follows using
S=
ji :

 win (S ≥ 2)
loss (S ≤ −2)D=
tie (otherwise)
6.1.3 Pairwise Score Calculation
Suppose that W is the number of wins comparedto the baseline, L is the number of losses and T
is the number of ties. The Pairwise score can becalculated by the following formula:

Human Evaluation

In WAT2020, we conducted 2 kinds of human eval­uations: pairwise evaluation (only for JaEn multi­
modal translation task, Section 6.1) and JPO ad­equacy evaluation (other than HiEn multi­modaltranslation task, Section 6.2) and a pairwise varia­
tion of direct assessment (Section 6.4) for the HiEn
multi­modal task.

P airwise = 100 ×

W −L
W +L+T

From the definition, the Pairwise score ranges be­
tween ­100 and 100.
6.2 JPO Adequacy Evaluation
We conducted JPO adequacy evaluation for the top
two or three participants’ systems of pairwise eval­uation for each subtask.45 The evaluation was car­ried out by translation experts based on the JPO

42

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2020/registration/index.html
43
https://github.com/moses-smt/mosesdecoder/
blob/master/mert/evaluator.cpp
44
https://github.com/moses-smt/mosesdecoder/
blob/master/scripts/tokenizer/tokenizer.perl

45

15

The number of systems varies depending on the subtasks.

5
4
3
2
1

All important information is transmitted correctly.
(100%)
Almost all important information is transmitted cor­
rectly. (80%–)
More than half of important information is transmit­
ted correctly. (50%–)
Some of important information is transmitted cor­
rectly. (20%–)
Almost all important information is NOT transmit­
ted correctly. (–20%)

6.3 Manual Evaluation for the UFAL
(EnOdia) Task
The user interface for our annotation for each of
the tracks is illustrated in Figure 3, and Figure 4.
The interpretation of these judgements is carriedout as described in the following Section 6.4.
6.4 Manual Evaluation for the
English→Hindi Multi­Modal Task

Table 19: The JPO adequacy criterion

The evaluations of the three tracks of the multi­
modal task and also the UFAL EnOdia task follow
the Direct Assessment (DA, Graham et al., 2016)
technique by asking annotators to assign a scorefrom 0 to 100 to each candidate. The score is as­
signed using a slider with no numeric feedback, thescale is therefore effectively continuous. After acertain number of scored items, it is assumed thateach of the annotators stabilizes in their scoring cri­
teria.
The collected DA scores can be either directly
averaged for each system and track (denoted“Ave”), or first standardized per annotator acrossall annotation tasks and then averaged (“Ave Z”).
The standardization removes the effect of individ­
ual differences in the range of scores assigned: thescores are scaled so that the average score of eachindividual annotator across all tasks he or she an­notated is 0 and the standard deviation is 1.
Our evaluation differs from the basic DA in thefollowing respects: (1) we run the evaluation bilin­
gually, i.e. we require the annotators to understandthe source English sufficiently to be able to assess
the adequacy of the Hindi translation, (2) we ask
the annotators to score two distinct segments at
once, while the original DA displays only one can­didate at a time.
The main benefit of bilingual evaluation is thatthe reference is not needed for the evaluation. In­
stead, the reference can be included among othercandidates and the manual evaluation allows us todirectly compare the performance of MT to humantranslators.
The dual judgment (scoring two candidates atonce) was added experimentally last year. The
advantage is saving some of the annotators’ time(they do not need to read the source or examine
the picture again) and the chance to better stabi­lize their score assignments by seeing another can­didate output. This essentially direct pairwise com­
parison could be useful especially for systems very

adequacy evaluation criterion, which is originallydefined by JPO to assess the quality of translated
patent documents.
6.2.1

Sentence Selection and Evaluation

For the JPO adequacy evaluation, the 200 test sen­tences were randomly selected from the test sen­tences. For the Newswire (JIJI) task test set II, ar­ticles were randomly selected from the context of
test set II until the number of the test sentences thatwere contained in the selected articles became 200.
For each test sentence, input source sentence,translation by participants’ system, and referencetranslation were shown to the annotators. For the
Newswire (JIJI) task test set II, input source sen­tences were shown in articles, which means that
not only input source sentences but also their con­text were shown to the evaluators. The evaluators
considered the context of the input sentences toevaluate the translations. To guarantee the qual­
ity of the evaluation, each sentence was evaluatedby two annotators. Note that the selected sentences
are basically the same as those used in the previous
workshop (WAT2019).
6.2.2

Evaluation Criterion

Table 19 shows the JPO adequacy criterion from
5 to 1. The evaluation is performed subjectively.
“Important information” represents the technicalfactors and their relationships. The degree of im­
portance of each element is also considered to eval­
uate. The percentages in each grade are rough in­dications for the transmission degree of the source
sentence meanings. The detailed criterion is de­scribed in the JPO document (in Japanese).46
46
http://www.jpo.go.jp/shiryou/toushin/chousa/tokkyohonyaku_hyouka.htm

16

Figure 3: Manual evaluation of English to Odia translation task.

Figure 4: Manual evaluation of Odia to English translation task.
Value Chosen [% of Cases]
Unscored Min Other Max
H0 25.8
0.2
52.8 21.3
H1 13.5
0.0
41.4 45.1
H2 45.6
0.3
23.4 30.7
H3 18.6
1.9
78.9
0.6
H4 21.6
0.2
58.5 19.7
H5 11.7
0.1
58.8 29.4
O0
0.2
0.7
90.8
8.3
O1
0.3
0.5
87.5 11.6
O2
0.0
0.3
90.8
8.9
O3
0.5
8.5
67.7 23.3
O4
0.0
8.5
70.0 21.5
O5
0.0 10.965.3 23.8

close in their performance.47
The user interface for our annotation for eachof the tracks is illustrated in Figures 5 to 7. Bydefault, the position of the slider appears to be at
the “worst” score but technically, the user inter­face is capable of distinguishing if the annotator
has touched the slider at all or not. The default
score is ­1 while the lowest score that the annota­
tor can assign is 0.
Table 20 provides an overview of the usage ofsliders in annotation for the 6 Hindi annotators (H∗,each scoring 1256 outputs) and 6 Odia annotators(O∗, each scoring 576 outputs). The Hindi anno­tation was carried out first and we observed thatthe default value was left untouched (“Unscored”)
rather often, in up to 45.6% of cases for the an­notator H2. Given this large proportion, we de­cided to consider two interpretations of the default
score ­1: we either disregard these scorings, assum­
ing that the annotator forgot about that particularslider, or we interpret the scoring as “Worst”, i.e.
merging the “Unscored” and “Min” cases. For the

Total Cases
1256 (100.0%)1256 (100.0%)1256 (100.0%)1256 (100.0%)1256 (100.0%)1256 (100.0%)576 (100.0%)576 (100.0%)576 (100.0%)576 (100.0%)576 (100.0%)576 (100.0%)

Table 20: Usage of DA sliders in English→Hindi Multi­
Modal and UFAL EnOdia Tasks.

Odia task, we urged the annotators to touch every
slider. The low “Unscored” rates indicate that thisreminder helped and only very few items were for­gotten.
It may seem surprising that many annotators
used the very highest score (“Max”). This is possi­
ble because the sentences are often short and sim­ple and also because human translations are in­cluded in the scoring. The big differences in the us­age of the extreme values however justify the needfor score standardization.
In the “text­only” evaluation, one English text
(source) and two Hindi translations (candidate 1

47

For the full statistical soundness of the subsequent inter­
pretation of DA scores, the judgments should be independentof each other. We explicitly ask our annotators to judge thecandidates independently but the dependence cannot be de­nied. Whether the violation of the independence assumption
is offset by the benefit of obtaining more stable judgements is
yet to be analyzed

17

Figure 5: Manual evaluation of text­only translation in the multi­modal task.

Figure 6: Manual evaluation of multi­modal transla­
tion.

Figure 7: Manual evaluation of Hindi captioning.

and 2) are shown to the annotators. In the “multi­
modal” evaluation, the annotators are shown boththe image and the source English text. The firstquestion is to validate if the source English text
is a good caption for the indicated area. For two
translation candidates, the annotators are asked toindependently indicate to what extent the meaning
is preserved. The “Hindi captioning” evaluationshows only the image and two Hindi candidates.
The annotators are reminded that the two captions
should be treated independently and that each ofthem can consider a very different aspect of the re­
gion.

7

7.1 Official Evaluation Results
Figures 8 and 9 show the official evaluation resultsof ASPEC subtasks, Figures 10, 11, 12, 13, 14 and
15 show those of JPC subtasks, Figure 16 showsthat of JIJI­c subtask, Figures 17 and 18 show thoseof MMT subtasks, Figures 19, 20, 21 and 22 show
those of Indic Multilingual subtasks, Figures 23
and 24 show those of BSD subtasks and Figures
25 and 26 show those of Hinden subtasks. Eachfigure contains the JPO adequacy evaluation resultand evaluation summary of top systems.
The detailed automatic evaluation results areshown in Appendix A. The detailed JPO adequacy
evaluation results for the selected submissions areshown in Table 21. The weights for the weighted
κ (Cohen, 1968) is defined as |Evaluation1 −Evaluation2|/4.
The automatic scores for the multi­modal and
UFAL EnOdia tasks along with the WAT evalua­tion server BLEU scores are provided in Table 22.
For each of the test sets of the multi­modal task
(E­Test, C­Test), the scores are comparable acrossEvaluation Results

In this section, the evaluation results for WAT2020
are reported from several perspectives. Some of
the results for both automatic and human evalu­
ations are also accessible at the WAT2020 web­site.48
48
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/18

all the tracks (text­only, captioning or multi­modaltranslation) because of the underlying set of refer­ence translations is the same. The scores for thecaptioning task will be however very low because
captions generated independently of the English
source caption are very likely to differ from the ref­erence translation.
For multi­modal task, Table 23 shows the man­ual evaluation scores for all valid system submis­
sions. As mentioned above, we used the referencetranslation as if it was one of the competing sys­tems, see the rows “Reference” in the table. The
annotation was fully anonymized, so the annota­
tors had no chance of knowing if they are scoring
human translation or MT output.
The UFAL EnOdia task has its official manualscores listed in Table 24.

8

Findings

8.1

ASPEC Task

affect the human evaluation scores.
8.2 JPC Task
Two teams participated in the JPC task; goku20
submitted their systems for all language pairs(J↔E, J↔C, and J↔K) and TMU submitted theirsystems for the K→J pair. goku20 used baseline
Transformer models and mBART models, whichwere pre­trained on large­scale monolingual cor­pus in 25 languages, fine­tuned on the JPO cor­pus. TMU used baseline Transformer models, en­hanced models with Hanja loss to obtain close
embeddings of Sino­Korean (Hanja) and Sino­
Japanese (Kanji) words, and domain adaptation
models fine­tuned for each domain on source test
sentences and target sentences translated by their
domain­specific model. Domain indicates section
based on IPC: chemistry, electricity, mechanicalengineering, or physics.
We discuss results on test­N data as follows. For
J→C and J→E, goku20’s ensemble Transformer
models achieved higher BLEU scores than the bestsystems in previous years’ WAT except for sys­
tems using additional resources. According to
goku20’s experiments, single mBART models out­
performed single Transformer models for several
language pairs but ensemble mBART models un­derperformed ensemble Transformer models for
all language pairs. For K→J, TMU’s domain adap­
tation model achieved the highest BLEU score,
while their Hanja loss+domain adaptation models(and their unofficial baseline Transformer model)
achieved similar scores. According to TMU’s ex­
periments, single Hanja loss model slightly im­
proved single Transformer model but no difference
was observed between ensemble versions of bothtypes of models. As for JPO adequacy evaluation,
goku20 submitted ensemble mBART models for
all language pairs and TMU submitted ensemble
domain adaptation model and ensemble domain
adaptation + Hanja loss model for K→J. All sys­
tems by both teams achieved high adequacy scores
close to or better than 4.5.

There is only one team (Kyoto­U+ECNU) who par­ticipated ASPEC task this year. Kyoto­U+ECNU
team participated Japanese ↔ Chinese translationsubtasks. They achieved the state­of­the­art auto­
matic evaluation scores, however, the human eval­uation scores are below those of last year’s (see Fig­ure 8 and 9. Strictly speaking, the human evalua­tion scores of this year and last year are not directlycomparable because the evaluators might be differ­
ent.
They trained a lot of different NMT models
which exploit 1) different training data (out­of­domain external data, back/forward­translation ofJapanese­side of ASPEC­JE), 2) different S2S
frameworks (LSTM, ConvS2S, Transformer and
Ligntconv) and 3) different model capacities (dif­ferent hyperparameter settings). They also tried
to use mBART. Among all the models, the ones
which exploit data augmentation by back/forward­translation performed the best. They also tried to
combine various NMT models trained above. The
BLEU score improves a lot (about 0.5 to 1.5 points)
by adding first 2 or 3 models, however, the impact
is getting smaller (about 0.1 to 0.2 points improve­ment) after that.
From the results, we can say that using exter­
nal resources and combining various models both
improves the automatic evaluation scores becauseof the generalization effect or improvement of flu­
ency, but they might have a bad effect on lexical
choice of technical term translations which directly

Thus, evaluation results in this year demon­
strated high translation accuracy by Transformer
models similarly to WAT2019’s results. Althoughboth teams reported improvements by their ad­
ditional techniques, their enhanced models per­formed similarly or worse than strong Transformer
baselines if ensemble models were used.
19

Figure 8: Official evaluation results of aspec­ja­zh.

Figure 9: Official evaluation results of aspec­zh­ja.

Figure 10: Official evaluation results of jpcn­ja­en.

20

Figure 11: Official evaluation results of jpcn­en­ja.

Figure 12: Official evaluation results of jpcn­ja­zh.

Figure 13: Official evaluation results of jpcn­zh­ja.

21

Figure 14: Official evaluation results of jpcn­ja­ko.

Figure 15: Official evaluation results of jpcn­ko­ja.

Figure 16: Official evaluation results of jijic­ja­en.

22

Figure 17: Official evaluation results of mmt­en­ja.

Figure 18: Official evaluation results of mmt­ja­en.

Figure 19: Official evaluation results of indic20­en­bn.

23

Figure 20: Official evaluation results of indic20­bn­en.

Figure 21: Official evaluation results of indic20­en­hi.

Figure 22: Official evaluation results of indic20­hi­en.

24Figure 23: Official evaluation results of bsd­en­ja.

Figure 24: Official evaluation results of bsd­ja­en.

25

Figure 25: Official evaluation results of iitb­en­hi.

Figure 26: Official evaluation results of iitb­hi­en.

26

8.3

hibits some properties of Dravidian languages suchas agglutination making it morphologically richer
than the other Indo­Aryan languages. This causes
translation quality to and from Marathi to be lower
than the translation quality to and from the otherIndo­Aryan languages.

Newswire (JIJI) Task

There were two submissions to the Japanese­to­English task from the NHK­NES team. The two
submissions were translations without using con­text and translations using context. The team ad­
dressed the problem of translating zero subject sen­tences in Japanese into English by extracted sub­jects and topics from source context using deep
analysis and added them to the input sentences ascontext. The automatic evaluation scores and thehuman evaluation scores of JPO Adequacy were
improved by using context. Although official train­
ing data does not contain contextual information,
the team used external training data that contained
contextual information.
8.4

Human evaluation was done for Hindi–English
and Bengali–English (both directions) which re­vealed that higher BLEU scores often did notcorrelate with what humans considered as higher
quality translations. A deeper look showed thatwhile “HW­TSC” had significantly higher BLEU
scores, the percentage of sentences that were per­fectly translated (a rating of 5 by human evalu­
ators) were substantially lower than the percent­age of sentences that were perfectly translated by
the team “cvit­mt”. For all human evaluated di­rections, translations by “cvit­mt” was rated to be
the best despite lagging behind in terms of BLEU
when compared with “HW­TSC”. This indicatesthat human evaluation and automatic evaluation fo­
cus on different aspects of translation quality andthus both types of evaluation should be performed
in order to better evaluate the quality of transla­
tions.

NICT­SAP Task

Despite the novelty of the task and the availabil­
ity of clean evaluation data for Wikinews and Soft­ware Documentation domain, we had only 1 sub­mission from the “NICT­5” team. They submitted
a many­to­many model which in most cases, sig­
nificantly outperformed the organizers baselines
which were one­to­many and many­to­one models.
Thai to English translation was significantly lower(around 10 BLEU) compared to all other transla­tion directions. Human evaluation was not per­formed and at present it is difficult to draw any
conclusions on the translation quality due to lackof participation.
8.5

8.6 UFAL (EnOdia) Task
This year, four teams participated in this newtask. For the English→Odia translation task, we
received 10 submissions from four teams (ex­cluding organizers) which includes five submis­
sions from the team “cvit”, two submissions fromthe teams “ODIANLP” and “ADAPT” and a sin­gle submission from the team “NLPRL”. For
the Odia→English translation task, we received
6 submissions from three teams (excluding orga­
nizers) which includes three submisions from theteam “cvit”, two submissions from the team “ODI­
ANLP”, and a single submission from the team
“NLPRL”.

Indic Multilingual Task

Of the several submissions we collected from 4
teams (excluding organizers), the best translations,as measured by BLEU, were submitted by “HW­
TSC”. The BLEU scores for most translation direc­
tions for this team were significantly higher com­pared to the other participants. With regards to au­
tomatic evaluation scores, translation into English
was observed to be substantially better than trans­
lation into the Indic languages. This is understand­able because Indic languages are morphologicallyricher than English. Furthermore, translation qual­
ity to and from Dravidian languages such as Tamil,
Telugu and Malayalam was observed to be the least
when compared to the translation quality to andfrom the Indo­Aryan languages Bengali, Marathi,
Hindi and Gujarati. Given that the Dravidian lan­
guages are morphologically richer than the Indo­
Aryan ones, making them hard to translate or trans­late into. Marathi is a special language which ex­

The team “ODIANLP” obtained the high­est BLEU score for both English→Odia andOdia→English translation tasks.
Manual evaluation of EnOdia Task is provided
in Table 24. Regardless the exact interpretation of
the rankings (see the discussion in the following
section on Hindi Multi­Modal task), manual trans­
lation is the best, followed by “cvit” in the transla­tion into Odia and by “ODIANLP” in the transla­tion into English.
27

8.7

English→Hindi Multi­Modal Task

8.8 Japanese↔English Multi­Modal Tasks

This year three teams participated in the differentsub­tasks (TEXT, MM, and HI) of the English→
Multi­Modal task. The team “ODIANLP” ob­tained the highest BLEU score for the text­onlytranslation (TEXT) for both the evaluation (E­Test)and challenge (C­Test) test set. For the caption­
ing and multimodal sub­tasks (HI and MM), we re­ceived only one submission from the teams “ODI­
ANLP” and “CNLP­NITS”, respectively.
In order to make the evaluation better grounded,
we included also the outputs of the best system ineach of the sub­tasks in the annotation. In manual
ranking, the 2020 systems thus compete not onlywith the reference translation but also with the win­ner from 2019.
It should be noted that a revision of E­Test andC­Test files was carried out between the years but
the source English did not really change.49 The
2019 system outputs could be thus directly used inthis year’s evaluation.
Table 23 in the appendix presents the results ofthe manual annotation. We compare the two inter­
pretations, either ignoring items where the slider
was not touched by the annotator, or interpreting it
as the lowest value. The final ranking of the sys­
tems do not change (despite substantial changes in
the actual average scores). Similarly, the standard­
ization of the scores (“Ave” vs. “Ave Z”) do notchange the overall ranking of the systems.
Across the tasks, 2020 systems perform better
than the best system from 2019. The referencetranslation generally scores much better than thebest system in each task, except for text­only trans­
lation of the E­Test. Same as last year, the best sys­
tem comes out marginally better than the reference.
Interestingly, IDIAP system ID 2956, which sur­passed the reference in 2019 ended up fourth thisyear. This can be explained by the different ran­
dom choice of evaluated sentences this year, but isstill clearly illustrates that text­only competition istight.
Across the various tasks, each of the three par­ticipating teams got its medal.

This year two teams participated in both
Japanese→English
andEnglish→Japanesetasks. Overall, we found that the positive effectof image information was not very significant. In
English→Japanese task, MMT systems slightly
outperform text­only NMT baselines, but inEnglish→Japanese task, NMT achieved equal or
better performance. This is perhaps because of thelack of sufficient training data which could make it
difficult to properly optimized additional network
parameters for visual inputs. We are planning
to increase the size of the dataset so that we canhopefully mitigate this issue. Another observa­
tion is that no systems except for the baselines
provided by organizers used phrase­to­phrase orphrase­to­region annotations in the dataset. We
believe strong text–image grounding is the key tomore successful MMT, and there is much room
for research to utilize this information in future.
8.9 Document­level Translation Task
WAT2020 has 2 kinds of document­level transla­tion tasks for the first time: English −→ Japanese
Scientific Paper (ParaNatCom) and English ⇐⇒Japanese Business Scene Dialogue (BSD). Unfor­
tunately there was no participants for ParaNatCom(we think this is because there is no document­
level training data available), and 4 participants for
BSD.
The evaluation results of BSD tasks are in Fig­
ures 23 and 24. The ut­mrt team did not selectthe best­BLEU submissions for the human evalu­
ation, we (organizers) additionally evaluate themand show the results in the figure with the ’*’ markwith the team ID. From the results, the top 3 sub­missions (goku20 1, DEEPNLP 1 and *ut­mrt)
are competitive while their systems are different.
goku20 used mBART which considers previous 3
sentences together with the source sentence. The
other two did not consider context in their system,
but they used a lot of external resources with finetuning on the provided training data.
Interestingly, goku20 reported that context­
aware models fine­tuned only on the provided data
did not improve the translation quality, but it has
a good effect when it is fine­tuned on a larger data
(they used JESC corpus). adapt­dcu also reported
very similar results: mixed­fine tuning using JESC
and OpenSubtiles along with BSD is much better
than using only BSD. ut­mrt tried to use context­49
The revision affected 906 out of 1595 lines of E­Test but
the changes were only in the Hindi (i.e. target) texts, with
one exception, correcting the typo “mam” to “man” in oneof the English source sentences. C­Test was corrected in 376
out of 1400 lines, again mainly in the target Hindi. The only
two changes in the English side were a correction of a typo(“wres”→“wires”) and a rather unfortunate removal of quota­
tion marks around a “press here” sticker text.

28

aware model (ut­mrt 2), but it did not improve thetranslation quality compared to the model without
context (ut­mrt 1). From these results, fine­tuning
also requires a substantial amount of training datawith context.
Another interesting point is that smaller (6,000)
sub­word units leads to better translation quality,
which is reported by adapt­dcu. We currently do
not have any idea about the reason of this result.
It is worth investigating the results deeper in the
future.
8.10

ful submissions for improving the current machinetranslation systems by analyzing the submissions
and identifying the issues.
For the next WAT workshop, we will include
several new datasets and new languages (Arabic
and additional Indic languages). We are also plan­ning to provide document­level test sets for sometranslation tasks because sentence­level translation
quality is almost saturated for some tasks. We alsoplan to have a new shared task named “RestrictedTranslation” task where we will investigate trans­
lation with restricted target vocabularies.

IITB Hindi–English Task

Acknowledgement

This year we received two submissions for English
to Hindi translation and one submission for Hindi
to English translation, all from one team (“WT”).
For English to Hindi translation, the submissions
had about 1 to 2 BLEU points higher than thebest submissions of 2018 and 2019. The AM­FM
scores were substantially higher by approximately
16 to 17 points compared to previous years. How­ever the human evaluation for adequacy showedthat the translation quality was slightly worse thanthe best translation quality in 2019 and comparable
to the best translation quality in 2018. For the re­verse direction, Hindi to English, there was an ex­
plosive growth in translation quality as measured
by BLEU. Where the best submission of 2019 had
a BLEU score of 22.91, the best (and only) sub­mission of 2020 had a BLEU score of 29.59. Hu­
man evaluation (adequacy), revealed that the over­
all adequacy score is less than the best adequacy
score in 2019. Deeper investigation showed that
this is due to most translations in 2020 being of
average quality compared to the best translationsin 2019 (42.00% in 2020 versus 17.17% in 2019). On the other hand the number of perfectly rated
translations dropped substantially from 48.00% in2019 to 30% in 2020. This explains why the over­
all human evaluation score for the best 2019 trans­
lations is higher than the one for the 2020 transla­
tions. This discrepancy between BLEU score andhuman evaluation shows the importance of man­ual investigations of translation instead of blindly
relying on automatic scores.

9

The English→Hindi Multi­Modal and UFAL En­
Odia shared tasks were supported by the following
grants at Idiap Research Institute and Charles Uni­
versity.
• At Idiap Research Institute, the work was sup­ported by the EU H2020 project “Real­time
network, text, and speaker analytics for com­bating organized crime” (ROXANNE), grant
agreement: 833635.
• At Charles University, the work was sup­ported by the grant 19­26934X (NEUREM3)of the Czech Science Foundation and using
language resources distributed by the LIN­
DAT/CLARIN project of the Ministry of Ed­
ucation, Youth and Sports of the Czech Re­
public (projects LM2015071 and OP VVV VI
CZ.02.1.01/0.0/0.0/16013/0001781).
Some of the human evaluations for the othertasks were supported by “Research and Develop­
ment of Deep Learning Technology for Advanced
Multilingual Speech Translation”, the Commis­
sioned Research of National Institute of Infor­
mation and Communications Technology (NICT),JAPAN.

ReferencesŽeljko Agić and Ivan Vulić. 2019. JW300: A wide­
coverage parallel corpus for low­resource languages.
In Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics, pages
3204–3210, Florence, Italy. Association for Compu­
tational Linguistics.

Conclusion and Future Perspective

This paper summarizes the shared tasks of
WAT2020. We had 14 participants worldwide who
submitted their translation results for the humanevaluation, and collected a large number of use­

Rafael E. Banchs, Luis F. D’Haro, and Haizhou Li.
2015. Adequacy­fluency metrics: Evaluating mt in
the continuous space model framework. IEEE/ACM

29

Trans. Audio, Speech and Lang. Proc., 23(3):472–
482.

Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senel­lart, and Alexander Rush. 2017. Opennmt: Open­
source toolkit for neural machine translation. In
Proceedings of ACL 2017, System Demonstrations,pages 67–72. Association for Computational Lin­
guistics.

Ondrej Bojar, Vojtech Diatka, Pavel Rychlỳ, Pavel
Stranák, Vít Suchomel, Ales Tamchyna, and Daniel
Zeman. 2014.
HindEnCorp­Hindi­English andHindi­only Corpus for Machine Translation. In Lan­guage Resources and Evaluation Conference.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison­Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra
Constantin, and Evan Herbst. 2007. Moses: Open
source toolkit for statistical machine translation. In
Annual Meeting of the Association for Computa­
tional Linguistics (ACL), demonstration session.

Bianka Buschbeck and Miriam Exel. 2020. A parallel
evaluation data set of software documentation with
document structure annotation.
Jacob Cohen. 1968. Weighted kappa: Nominal scaleagreement with provision for scaled disagreement or
partial credit. Psychological Bulletin, 70(4):213 –220.

T. Kudo. 2005.
Mecab : Yet another part­of­speechandmorphologicalanalyzer.
http://mecab.sourceforge.net/.

Chenchen Ding, Hnin Thu Zar Aye, Win Pa Pa, Khin
Thandar Nwet, Khin Mar Soe, Masao Utiyama, andEiichiro Sumita. 2019. Towards Burmese (Myan­mar) morphological analysis: Syllable­based tok­
enization and part­of­speech tagging. ACM Trans­
actions on Asian and Low­Resource Language Infor­
mation Processing (TALLIP), 19(1):5.

Anoop Kunchukuttan. 2020. The IndicNLP Library.
https://github.com/anoopkunchukuttan/indic_
nlp_library/blob/master/docs/indicnlp.pdf.

Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhat­
tacharyya. 2018. The IIT Bombay English­Hindi
parallel corpus. In Proceedings of the Eleventh In­
ternational Conference on Language Resources and
Evaluation (LREC 2018), Miyazaki, Japan. Euro­pean Language Resources Association (ELRA).

Chenchen Ding, Masao Utiyama, and Eiichiro Sumita.
2018. NOVA: A feasible and flexible annotation sys­
tem for joint tokenization and part­of­speech tagging.
ACM Transactions on Asian and Low­Resource Lan­guage Information Processing (TALLIP), 18(2):17.

Sadao Kurohashi, Toshihisa Nakamura, Yuji Mat­sumoto, and Makoto Nagao. 1994. Improvements ofJapanese morphological analyzer JUMAN. In Pro­
ceedings of The International Workshop on Sharable
Natural Language, pages 22–28.

Yvette Graham, Timothy Baldwin, Alistair Moffat, andJustin Zobel. 2016. Can machine translation systems
be evaluated by the crowd alone. Natural Language
Engineering, FirstView:1–28.

Gregor Leusch, Nicola Ueffing, and Hermann Ney.
2006. CDER: Efficient MT Evaluation Using Block
Movements. In In Proceedings of EACL, pages 241–
248.

Barry Haddow and Faheem Kirefu. 2020. Pmindia – acollection of parallel corpora of languages of india.
arxiv 2001.09907.
Aizhan Imankulova, Raj Dabre, Atsushi Fujita, andKenji Imamura. 2019. Exploiting out­of­domain par­allel data through multilingual transfer learning forlow­resource neural machine translation. In Pro­
ceedings of Machine Translation Summit XVII Vol­ume 1: Research Track, pages 128–139, Dublin, Ire­
land.

Hideki Nakayama, Akihiro Tamura, and Takashi Ni­
nomiya. 2020. A visually­grounded parallel corpus
with phrase­to­region linking. In Proceedings of the
12th Language Resources and Evaluation Confer­
ence.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic evalu­
ation of translation quality for distant language pairs.
In Proceedings of the 2010 Conference on Empirical
Methods in Natural Language Processing, EMNLP’10, pages 944–952, Stroudsburg, PA, USA. Associ­ation for Computational Linguistics.

Toshiaki Nakazawa, Nobushige Doi, Shohei Hi­
gashiyama, Chenchen Ding, Raj Dabre, Hideya
Mino, Isao Goto, Win Pa Pa, Anoop Kunchukut­tan, Yusuke Oda, Shantipriya Parida, Ondřej Bojar,
and Sadao Kurohashi. 2019. Overview of the 6thworkshop on Asian translation. In Proceedings of
the 6th Workshop on Asian Translation, pages 1–35,
Hong Kong, China. Association for Computational
Linguistics.

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2017. Google’s
multilingual neural machine translation system: En­
abling zero­shot translation. Transactions of the As­
sociation for Computational Linguistics, 5:339–351.

Toshiaki Nakazawa, Shohei Higashiyama, Chenchen
Ding, Hideya Mino, Isao Goto, Hideto Kazawa,
Yusuke Oda, Graham Neubig, and Sadao Kurohashi.
2017. Overview of the 4th workshop on asian trans­
lation. In Proceedings of the 4th Workshop on AsianTranslation (WAT2017), pages 1–54. Asian Federa­tion of Natural Language Processing.

30

Seventh Workshop on Statistical Machine Transla­tion, pages 401–409, Montréal, Canada. Association
for Computational Linguistics.

Toshiaki Nakazawa, Katsuhito Sudoh, Shohei Hi­
gashiyama, Chenchen Ding, Raj Dabre, Hideya
Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan,
and Sadao Kurohashi. 2018. Overview of the 5thworkshop on Asian translation. In Proceedings ofthe 32nd Pacific Asia Conference on Language, In­formation and Computation: 5th Workshop on AsianTranslation: 5th Workshop on Asian Translation,
Hong Kong. Association for Computational Linguis­
tics.

Loganathan Ramasamy, Ondřej Bojar, and Zdeněk
Žabokrtský. 2012. Morphological Processing forEnglish­Tamil Statistical Machine Translation. In
Proceedings of the Workshop on Machine Transla­tion and Parsing in Indian Languages.
Hammam Riza, Michael Purwoadi, Teduh Uliniansyah,
Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai,
Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea,
Sethserey Sam, Sopheap Seng, Khin Mar Soe, Khin
Thandar Nwet, Masao Utiyama, and Chenchen Ding.
2016. Introduction of the asian language treebank.
In In Proc. of O­COCOSDA, pages 1–6.

Graham Neubig, Yosuke Nakata, and Shinsuke Mori.
2011. Pointwise prediction for robust, adaptablejapanese morphological analysis. In Proceedings ofthe 49th Annual Meeting of the Association for Com­
putational Linguistics: Human Language Technolo­
gies: Short Papers ­ Volume 2, HLT ’11, pages 529–
533, Stroudsburg, PA, USA. Association for Compu­
tational Linguistics.

Holger Schwenk, Vishrav Chaudhary, Shuo Sun,
Hongyu Gong, and Francisco Guzmán. 2019. Wiki­matrix: Mining 135m parallel sentences in 1620 lan­
guage pairs from wikipedia. arxiv 1907.05791.

Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya,
Yutaro Omote, and Hideki Nakayama. 2020. A
visually­grounded parallel corpus with phrase­to­region linking. In Proceedings of the 28th Interna­
tional Conference on Computational Linguistics.

Parth Shah and Vishvajit Bakrola. 2019. Neural ma­chine translation system of indic languages ­ an atten­
tion based approach. In 2019 Second International
Conference on Advanced Computational and Com­
munication Paradigms (ICACCP). IEEE.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei­
Jing Zhu. 2002. Bleu: a method for automatic eval­uation of machine translation. In ACL, pages 311–
318.

Shashank Siripragrada, Jerin Philip, Vinay P. Nam­boodiri, and C V Jawahar. 2020. A multilingual par­allel corpora collection effort for Indian languages.
In Proceedings of The 12th Language Resources
and Evaluation Conference, pages 3743–3751, Mar­
seille, France. European Language Resources Asso­
ciation.

Shantipriya Parida, Ondrej Bojar, and Satya Ranjan
Dash. 2018. Odiencorp: Odia­english and odia­only
corpus for machine translation. In PProceedings of
the Third International Conference on Smart Com­puting and Informatics.

Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin­nea Micciulla, and John Makhoul. 2006. A study oftranslation edit rate with targeted human annotation.
In In Proceedings of Association for Machine Trans­
lation in the Americas, pages 223–231.

Shantipriya Parida, Ondřej Bojar, and Satya Ranjan
Dash. 2019a. Hindi visual genome: A dataset
for multimodal english­to­hindi machine translation.
arXiv preprint arXiv:1907.08948.
Shantipriya Parida, Ondřej Bojar, and Satya Ranjan
Dash. 2019b. Hindi Visual Genome: A Dataset forMultimodal English­to­Hindi Machine Translation.
Computación y Sistemas. In print. Presented at CI­
CLing 2019, La Rochelle, France.

Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew
Finch, and Eiichiro Sumita. 2016. Introducing the
Asian language treebank (ALT). In Proceedingsof the Tenth International Conference on Language
Resources and Evaluation (LREC’16), pages 1574–
1578, Portorož, Slovenia. European Language Re­
sources Association (ELRA).

Shantipriya Parida, Satya Ranjan Dash, Ondřej Bojar,
Petr Motlicek, Priyanka Pattnaik, and Debasish Ku­
mar Mallick. 2020. Odiencorp 2.0: Odia­english
parallel corpus for machine translation. In Proceed­
ings of the WILDRE5–5th Workshop on Indian Lan­guage Data: Resources and Evaluation, pages 14–
19.

Jörg Tiedemann. 2012. Parallel data, tools and inter­faces in OPUS. In Proceedings of the Eighth In­
ternational Conference on Language Resources and
Evaluation (LREC’12), pages 2214–2218, Istanbul,
Turkey. European Language Resources Association(ELRA).

Maja Popović. 2015. chrF: character n­gram F­score
for automatic MT evaluation. In Proceedings of
the Tenth Workshop on Statistical Machine Trans­
lation, Lisboa, Portugal. Association for Computa­
tional Linguistics.

Huihsin Tseng. 2005. A conditional random field word
segmenter. In In Fourth SIGHAN Workshop on Chi­nese Language Processing.

Matt Post, Chris Callison­Burch, and Miles Osborne.
2012. Constructing parallel corpora for six Indian
languages via crowdsourcing. In Proceedings of the

Masao Utiyama and Hitoshi Isahara. 2007. A japanese­english patent parallel corpus. In MT summit XI,
pages 475–482.

31

Ashish Vaswani, Samy Bengio, Eugene Brevdo, Fran­cois Chollet, Aidan Gomez, Stephan Gouws, Llion
Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Par­mar, Ryan Sepassi, Noam Shazeer, and Jakob Uszko­
reit. 2018. Tensor2tensor for neural machine trans­
lation. In Proceedings of the 13th Conference of the
Association for Machine Translation in the Americas(Volume 1: Research Papers), pages 193–199. Asso­
ciation for Machine Translation in the Americas.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. CoRR, abs/1706.03762.
Weiyue Wang, Jan­Thorsten Peter, Hendrik Rosendahl,
and Hermann Ney. 2016. Character: Translation
edit rate on character level. In ACL 2016 First
Conference on Machine Translation, pages 505–510,
Berlin, Germany.
Yi Mon Shwe Sin and Khin Mar Soe. 2018. Syllable­
based myanmar­english neural machine translation.
In In Proc. of ICCA, pages 228–233.

Appendix A

Submissions

Tables 25 to 52 summarize translation results sub­mitted for WAT2020 pairwise evaluation. Type,
RSRC, and Pair columns indicate type of method,use of other resources, and pairwise evaluation
score, respectively. The tables also include resultsby the organizers’ baselines, which are listed in Ta­
bles 16, 17, and 18.

32

SYSTEMDATA Annotator A
Annotator B
allweightedID
ID average varianceaverage varianceaverage κκKyoto­U+ECNU 3814 4.00
0.75
4.35
0.60
4.17 0.07 0.31
aspec­ja­zh Kyoto­U+ECNU 4053 4.03
0.73
4.31
0.69
4.17 0.08 0.32
0.47
4.29
0.85
4.36 0.15 0.15
2019 best
3170 4.44
Kyoto­U+ECNU 3813 3.81
0.83
4.61
0.47
4.21 0.16 0.24
aspec­zh­ja Kyoto­U+ECNU 3933 3.80
0.85
4.61
0.47
4.20 0.14 0.23
2019 best
3210 4.80
0.24
4.46
0.61
4.63 0.27 0.31
goku203930 4.48
0.68
4.69
0.33
4.58 0.27 0.39
jpcn­ja­en
2019 best
3188 4.73
0.40
4.83
0.22
4.78 0.36 0.46
goku203929 4.46
0.90
4.37
0.99
4.42 0.39 0.60
jpcn­en­ja
2019 best
3192 4.43
0.81
4.57
0.77
4.50 0.36 0.49
goku203926 4.61
0.49
4.66
0.47
4.63 0.35 0.51
jpcn­ja­zh2019 best
3157 4.53
0.45
4.56
0.54
4.54 0.29 0.35
goku203925 4.60
0.61
4.42
0.58
4.51 0.21 0.31
jpcn­zh­ja
2019 best
3152 4.72
0.26
4.57
0.55
4.65 0.26 0.35
goku203928 4.78
0.37
4.68
0.60
4.73 0.48 0.65
jpcn­ja­ko
0.27
4.73
0.34
4.77 0.56 0.65
2019 best
2850 4.82
TMU
3829 4.74
0.33
4.67
0.48
4.71 0.70 0.77
TMU3830 4.75
0.33
4.66
0.61
4.70 0.72 0.78
jpcn­ko­ja
goku203927 4.68
0.46
4.60
0.72
4.64 0.67 0.77
2019 best
2924 4.72
0.39
4.58
0.68
4.65 0.59 0.69
NHK­NES
3820 4.22
0.81
3.69
1.03
3.96 0.13 0.22
organizer
3893 4.21
0.64
3.70
1.18
3.95 0.14 0.23
jijic­ja­en
0.88
3.60
1.10
3.89 0.16 0.24
NHK­NES3818 4.18
organizer
3895 4.09
0.73
3.48
1.05
3.78 0.13 0.25
0.83
1.90
1.01
1.83 0.22 0.40
organizer
3642 1.76
0.53
3.67
0.73
3.90 0.25 0.36
cvit­mt
3853 4.13
HW­TSC4032 3.91
indic20­en­bn
0.85
3.61
0.86
3.76 0.20 0.44
0.80
2.98
1.07
3.17 0.27 0.50
ODIANLP
3780 3.35
cvit­mt
3837 3.94
0.66
4.03
0.85
3.98 0.15 0.35
HW­TSC4039 2.92
1.31
2.97
2.18
2.94 0.24 0.58
indic20­bn­en
0.94
2.27
1.97
2.58 0.10 0.42
NICT­53986 2.88
ODIANLP
3822 1.79
1.41
1.76
1.95
1.78 0.41 0.72
cvit­mt
3854 4.12
0.75
3.49
0.65
3.81 0.05 0.32
indic20­en­hi
1.09
2.48
0.96
2.89 ­0.03 0.29
HW­TSC4033 3.29
ODIANLP
3786 2.94
0.92
2.46
1.02
2.70 0.14 0.42
0.70
3.52
0.57
3.79 0.05 0.26
cvit­mt
3838 4.05
NICT­54000 2.54
1.45
2.13
0.81
2.34 0.18 0.46
indic20­hi­en
HW­TSC
4040 2.41
1.18
1.82
0.78
2.12 0.11 0.41
1.24
1.55
0.82
1.71 0.26 0.56
ODIANLP
3823 1.87
goku203756 4.14
0.84
4.27
1.08
4.20 0.32 0.47
DEEPNLP
4050 4.08
0.87
4.17
1.30
4.13 0.31 0.49
ut­mrt4074 3.57
1.74
3.55
2.33
3.56 0.33 0.59
bsd­en­ja
goku203753 3.67
1.18
3.44
2.02
3.55 0.35 0.53
ut­mrt4071 3.56
1.59
3.48
2.49
3.52 0.37 0.60
2.54
2.63
2.90
2.60 0.49 0.74
DEEPNLP
4049 2.56
goku203747 4.47
0.70
3.92
1.37
4.19 0.29 0.43
DEEPNLP
4048 4.43
0.72
3.76
1.54
4.10 0.25 0.41
adapt­dcu
3836 4.22
0.79
3.65
1.64
3.93 0.37 0.49
bsd­ja­enut­mrt4073 3.90
1.37
3.34
2.03
3.62 0.31 0.53
goku203748 3.92
1.30
3.23
2.08
3.57 0.25 0.47
1.43
3.25
2.15
3.55 0.30 0.54
ut­mrt4072 3.86
DEEPNLP
4047 2.60
2.51
2.19
2.40
2.40 0.42 0.68
WT3640 3.71
1.89
3.40
1.15
3.56 0.34 0.60
iitb­en­hi
WT3639 3.69
1.99
3.29
1.78
3.49 0.37 0.64
2019 best
2680 3.94
1.02
3.58
0.82
3.76 0.53 0.58
1.49
3.65
0.65
3.71 0.26 0.44
WT3638 3.78
iitb­hi­en2019 best
2681 4.53
0.53
3.74
1.18
4.13 0.05 0.13
Subtask

Table 21: JPO adequacy evaluation results in detail.

33

EV
TEXT
CH
TEXT
CH EV CH EV
HI HI MM MM
ODIA
en­od
ODIA
od­en

System2019:IDIAPODIANLP
CNLP­NITS
iiitsc
ODIANLP
2019:IDIAP
CNLP­NITS
iiitsc
CNLP­NITS
2019:638
2019:NITSNLP
CNLP­NITS
2019:638
ODIANLP
2019:NITSNLPODIANLPODIANLP
cvit
cvit
ODIANLP

Run
2956
3711
3897
4030
3713
3277
3898
4031
3896
3271
3288
3894
3270
3779
3297
3759
3788
3874
3872
3772

BLEU
0.51420.50240.48480.42910.47280.40370.36910.28590.51010.49340.38010.42640.2866
0.03480.02290.01790.11600.10380.03230.0315chrF30.58140.59220.58160.51880.55630.49980.45610.37300.60980.55760.44420.51280.37840.10480.08890.08940.18150.16930.20890.2072nCDER
0.62010.61410.6051
0.54950.58750.52550.49840.41240.62160.6012
0.48740.55240.4171
0.06870.08020.05950.25250.23360.13080.1425nCharacTER
0.58250.6079≀ 0.6174
0.5186
0.48150.44370.42680.26290.64850.50630.26520.50130.2036­0.3779­0.5016­0.4421­1.3391­1.1605­0.9498­1.1965

nPER
0.69260.68510.6796
0.62170.65040.60100.5667
0.48130.70380.67310.56700.61990.48730.16360.12970.12800.28950.27180.16460.1746nTER
0.57580.58520.57230.5126
0.56720.49050.46920.39030.58670.55450.43980.5226
0.3851­0.1298
0.0615­0.08290.21710.19190.11730.1290nWER
0.55480.56330.55300.49660.5404
0.46140.44750.37150.56480.53560.42050.49730.3658­0.1349
0.0566­0.0860
0.20580.18090.11390.1253

BLEUw
40.21
40.85
38.84
33.83
38.50
30.72
27.75
20.52
40.51
38.63
27.41
33.57
20.34
0.78
0.00
0.00
11.07
7.86
17.89
18.31

CH EV
HI HI

CH
MM

EV
MM

CH
TEXT

EV
TEXT

Table 22: Multi­Modal and UFAL EnOdia Task automatic evaluation results. For each test set (EV, CH or Odia) andeach track (TEXT, MM and HI; and Odia), we sort the entries by our BLEU scores. The symbol “≀” in subsequentcolumns indicates fields where the other metric ranks candidates in a different order. BLEUw denotes the WAT
official BLEU scores.

Team ID
ODIANLP
Reference
CNLP­NITS
2019:IDIAP
iiitscReference
ODIANLP
2019:IDIAP
CNLP­NITS
iiitscReference
CNLP­NITS
2019:638
2019:NITSNLP
Reference
CNLP­NITS
2019:638
Reference
ODIANLP
Reference
2019:NITSNLPODIANLP

Data ID
3711­
3897
2019:2956
4030
­
3713
2019:3277
3898
4031
­
3896
2019:3271
2019:3288­
3894
2019:3270­
3779­
2019:3297
3759

Ignoring Unscored
Ave
Ave Z
83.38
0.34
82.19
0.29
80.01
0.23
76.94
0.15
74.27
0.07
88.07
0.47
75.21
0.08
67.79
­0.10
59.61
­0.38
54.85
­0.53
86.82
0.45
81.75
0.28
74.82
0.07
59.31
­0.39
90.66
0.53
68.72
­0.11
57.03
­0.45
90.26
0.53
47.16
­0.73
88.94
0.51
58.56
­0.37
52.10
­0.57

Unscored = Worst
Ave
Ave Z
78.25
0.53
75.14
0.47
70.46
0.37
67.64
0.30
58.60
0.07
85.44
0.71
63.60
0.18
56.29
0.03
40.40
­0.36
36.78
­0.47
83.82
0.68
73.78
0.43
63.47
0.18
42.88
­0.31
88.53
0.78
55.80
0.01
41.79
­0.33
80.45
0.58
10.69
­1.10
78.53
0.53
21.29
­0.84
10.47
­1.11

OD
EN

EN
OD

Table 23: Manual evaluation result for WAT2020 English→Hindi Multi­Modal Tasks.

Team ID
Reference
cvit
ODIANLP
Reference
ODIANLP
cvit

Data ID
­
3874
3788
­
3772
3872

Ignoring Unscored
Ave
Ave Z
66.98
0.12
63.45
0.04
63.35
0.01
70.40
0.21
61.75
­0.05
59.44
­0.09

Unscored = Worst
Ave
Ave Z
66.74
0.11
63.45
0.04
63.24
0.01
70.40
0.21
61.55
­0.05
59.25
­0.09

Table 24: Manual evaluation result for WAT2020 UFAL (EnOdia) Tasks.

34

35

baseline
Kyoto­U+ECNU
Kyoto­U+ECNU

System

1738
3814
4053

ID

1996
3926

baselinegoku201995
3925

baselinegoku20

NMT
NMT

Type

ID

kytea
34.97
38.66
38.52

NO
YES
NO

NO
NO

RSRC

NMT
NMT

mecab
47.00
52.92
52.80

juman
0.880815
0.897053
0.896551

RIBES
kytea
0.875511
0.894441
0.894073

kytea
0.850199
0.858491
0.858229

RIBES
stanford­ctb
0.850052
0.857645
0.857229

juman
43.72
48.09

BLEU
kytea
44.38
49.06
mecab
43.72
48.34

BLEU
stanford­ctb
39.18
42.59

stanford­pku
38.65
42.27

Table 28: JPCN ja­zh submissions

kytea
38.07
41.52

stanford­pku
0.848360
0.857898

mecab
0.851797
0.861534

RIBES
stanford­ctb
0.849118
0.859118

RIBES
kytea
0.849783
0.859846

stanford­pku
0.787250
0.787730
0.786870

mecab
0.7981100.823390
0.821660

AMFM
stanford­ctb
0.787250
0.787730
0.786870

AMFM
kytea
0.7981100.823390
0.821660

kytea
0.787250
0.787730
0.786870

juman
0.7981100.823390
0.821660

stanford­pku
0.848394
0.8581030.857722

mecab
0.880368
0.897199
0.896743

kytea
0.845075
0.853135

juman
0.852623
0.861858

Table 27: JPCN zh­ja submissions

NO
NO

RSRC

Table 26: ASPEC ja­zh submissions

stanford­pku
34.72
38.43
38.30

Table 25: ASPEC zh­ja submissions

BLEU
kytea
47.30
53.64
53.48

BLEU
stanford­ctb
34.96
38.56
38.43

juman
46.87
52.80
52.65

Type

RSRC

System

NO
YES
NO

RSRC

NMT
NMT
NMT

Type

ID

NMT
NMT
NMT

Type

1740
3813
3933

ID

Systembaseline
Kyoto­U+ECNU
Kyoto­U+ECNU

System

36
ID
1999
3929

baselinegoku20

1997
3829
3830
3927

baseline
TMU
TMU
goku20

SystemID

System

BLEU
kytea
70.97
74.03
73.99
70.41
mecab
70.39
73.45
73.41
69.68

Type
NMT
NMT

RSRC
NO
NO

BLEU
71.62
70.48

BLEU
kytea
43.13
46.63
mecab
41.16
44.57

Type
NMT
NMT

RSRC
NO
NO

BLEU
39.39
43.51

Table 32: JPCN ja­en submissions

ID
2000
3930

RIBES
kytea
0.838359
0.854173

RIBES
0.837932
0.857091

juman
0.840117
0.855454

Table 31: JPCN en­ja submissions

NO
NO

juman
41.26
44.52

RIBES
kytea
0.941336
0.9498160.950247
0.946878

RIBES
0.944406
0.942321

juman
0.941950
0.950506
0.950791
0.947397

Table 30: JPCN ja­ko submissions

ID
2026
3928

RSRC

Systembaselinegoku20

NMT
NMT

Type

juman
70.13
73.13
73.10
69.37

Table 29: JPCN ko­ja submissions

NO
NO
NO
NO

RSRC

Systembaselinegoku20

NMT
NMT
NMT
NMT

Type

mecab
0.839361
0.855317

mecab
0.941905
0.950269
0.950528
0.947266

37

1943
3924
3932

baselinegoku20goku20

NMT
NMT
NMT

Type

Systembaseline
WT

Systembaseline
WT
WTID

System
juman
32.41
40.60
38.54

BLEU
kytea
34.37
42.34
40.55
mecab
32.53
40.66
38.99

juman
0.796011
0.813915
0.816579

Type
NMT
NMT
NMT

RSRC
NO
NO
YES

BLEU
13.76
22.08
22.80

RIBES
0.710210
0.765340
0.769138

Type
NMT
NMT

RSRC
NO
NO

BLEU
15.44
29.59

RIBES
0.718751
0.792065

Table 35: HINDEN hi­en submissions

ID
2567
3638

Table 34: HINDEN en­hi submissions

ID
2566
3639
3640

Table 33: JPCEP zh­ja submissions

NO
NO
NO

RSRC

AMFM
0.586360
0.637410

AMFM
0.644860
0.869400
0.873830

RIBES
kytea
0.796321
0.812992
0.817017
mecab
0.796411
0.813073
0.816112

38

ID
3586
3711
3897
4030

Type
NMT
NMT
NMT
NMT

RSRC
NO
YES
YES
NO

BLEU
39.78
40.85
38.84
33.83

RIBES
0.776892
0.790908
0.793416
0.753529

AMFM
0.796570
0.808340
0.804250
0.767900

ID
3779

Type
OTHER

RSRC
NO

BLEU
0.78

RIBES
0.064960

ID
3896

Type
NMT

RSRC
YES

BLEU
40.51

RIBES
0.803208

AMFM
0.820980

AMFM
0.351940

Table 38: HINDENMMEVMM20en en­hi submissions

System
CNLP­NITS

Table 37: HINDENMMEVHI20 en­hi submissions

SystemODIANLP

Table 36: HINDENMMEVTEXT20 en­hi submissions

Systembaseline
ODIANLP
CNLP­NITS
iiitsc

39

ID
3587
3713
3898
4031

Type
NMT
NMT
NMT
NMT

RSRC
NO
YES
YES
NO

BLEU
28.35
38.50
27.75
20.52

RIBES
0.691315
0.785252
0.714980
0.623644

AMFM
0.727010
0.804880
0.750320
0.698600

ID
3759

Type
OTHER

RSRC
NO

BLEU
0.00

RIBES
0.040071

ID
3894

Type
NMT

RSRC
YES

BLEU
33.57

RIBES
0.754141

AMFM
0.787320

AMFM
0.374450

Table 41: HINDENMMCHMM20en en­hi submissions

System
CNLP­NITS

Table 40: HINDENMMCHHI20 en­hi submissions

SystemODIANLP

Table 39: HINDENMMCHTEXT20 en­hi submissions

Systembaseline
ODIANLP
CNLP­NITS
iiitsc

40

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
YES
YES
YES
YES
NO
NO
YES
NO

BLEU
5.49
7.93
11.07
7.86
9.85
9.48
8.17
8.17
3.53
4.94

RIBES
0.326116
0.398153
0.502642
0.425505
0.489535
0.487873
0.458266
0.458266
0.283107
0.319955

ID
3585
3593
3772
3872
3873
4064

Type
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
YES
YES
YES
YES

BLEU
8.92
12.54
18.31
17.89
15.06
13.89

Type
NMT
NMT
NMT

RSRC
NO
YES
YES

BLEU
13.95
28.02
28.41

RIBES
0.646680
0.736564
0.736924

Table 44: JIJIC ja­en submissions

ID
3642
3818
3820

AMFM
0.470790
0.613710
0.624530

RIBES
0.349459
0.464391
0.558949
0.494172
0.487789
0.508536

Table 43: ODIAEN od­en submissions

Systembaseline
ODIANLPODIANLP
cvit
cvit
cvit

Systembaseline
NHK­NESNHK­NES

ID
3584
3592
3788
3874
4022
4052
4062
4063
4118
4119

Table 42: ODIAEN en­od submissions

Systembaseline
ODIANLPODIANLP
cvit
cvit
cvit
cvit
cvit
ADAPT
ADAPT

41

ID
3651
3914
4046

SystemTMU
HW­TSC
TMU

NO
YES
NO

BLEU
kytea
49.81
49.79
48.94
mecab
44.57
44.76
43.78

juman
0.869433
0.870792
0.871343

RSRC
NO
NO
YES

BLEU
48.38
48.33
52.28

Pair
−1.50
1.25
34.00

RIBES
kytea
0.874742
0.875362
0.876282

RIBES
0.899610
0.900684
0.911053

Table 46: MMT ja­en submissions

Type
NMT
NMT
NMT

Table 45: MMT en­ja submissions

juman
40.71
41.26
39.92

ID
3692
3706
3725

RSRC

SystemTMU
TMU
HW­TSC

NMT
NMT
NMT

Type
mecab
0.869886
0.870791
0.871542
−4.00
14.25
−11.50Pair42
Systembaseline
ODIANLP
cvit
NICT­5
HW­TSC

Systembaseline
ODIANLP
cvit
HW­TSC

Systembaseline
ODIANLP
cvit
NICT­5
HW­TSC

Systembaseline
ODIANLP
cvit
HW­TSC

Type
NMT
NMT
NMT
NMT

RSRC
NO
NO
YES
NO

BLEU
15.03
16.38
7.93
19.64

RIBES
0.627104
0.633248
0.669877
0.680257

Type
NMT
NMT
NMT
NMT
NMT

RSRC
NO
YES
NO
NO
NO

BLEU
21.80
19.71
16.50
16.14
23.38

RIBES
0.611506
0.588298
0.729215
0.654700
0.638013

Type
NMT
NMT
NMT
NMT

RSRC
NO
NO
YES
NO

BLEU
13.96
21.05
16.23
24.48

RIBES
0.700555
0.703507
0.709665
0.731919

Type
NMT
NMT
NMT
NMT
NMT

RSRC
NO
YES
NO
NO
NO

BLEU
25.68
21.88
20.95
21.25
28, 51RIBES
0.668709
0.620309
0.759791
0.707683
0.687383

Table 50: INDIC20 hi­en submissions

ID
3632
3823
3838
4000
4040

Table 49: INDIC20 en­hi submissions

ID
3625
3786
3854
4033

Table 48: INDIC20 bn­en submissions

ID
3631
3822
3837
3986
4039

Table 47: INDIC20 en­bn submissions

ID
3624
3780
3853
4032

AMFM
0.699228
0.670681
0.756972
0.711637
0.736056

AMFM
0.636668
0.652905
0.685356
0.691416

AMFM
0.677976
0.652696
0.733984
0.669477
0.717710

AMFM
0.667009
0.672342
0.679612
0.688318

43

goku20goku20DEEPNLPDEEPNLPut­mrtut­mrtSystem
3753
3756
4049
4050
4071
4074

ID
juman
14.19
19.43
8.66
19.39
14.59
14.19

BLEU
kytea
20.89
26.04
15.10
26.59
21.09
20.32
mecab
15.75
20.75
10.10
20.95
16.03
15.49

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
YES
NO
YES
NO
YES
YES
YES

BLEU
23.15
17.02
18.70
10.19
22.83
18.05
18.57

Table 52: BSD ja­en submissions

ID
3747
3748
3836
4047
4048
4072
4073

RIBES
kytea
0.739468
0.771588
0.686233
0.770217
0.744652
0.737332

RIBES
0.755099
0.688501
0.730460
0.615230
0.752619
0.723202
0.720809juman
0.700058
0.745493
0.640669
0.7442090.720883
0.698327

Table 51: BSD en­ja submissions

NO
YES
NO
YES
YES
YES

RSRC

Systemgoku20goku20
adapt­dcu
DEEPNLPDEEPNLPut­mrtut­mrt

NMT
NMT
NMT
NMT
NMT
NMT

Type
mecab
0.715917
0.749985
0.658636
0.753233
0.731101
0.719519

Source
CVIT­Mann ki Baat
CVIT­PIBIITB en­hi v2.0
MTurk Corpora
JW300
MTEnglish2Odia
NLPC­Uom Corpus
OdiEnCorp 1.0
OPUS
PMIndia
UFAL­en­ta­v2
Urs Tarsadia Corpus
Wikimatrix
Wikititles

Citation(Siripragrada et al., 2020)(Siripragrada et al., 2020)(Kunchukuttan et al., 2018)(Post et al., 2012)(Agić and Vulić, 2019)(Parida et al., 2018)(Tiedemann, 2012)(Haddow and Kirefu, 2020)(Ramasamy et al., 2012)(Shah and Bakrola, 2019)(Schwenk et al., 2019)

Table 53: Recommended Parallel Corpora for the
Indic multilingual translation task. All download
URLs can be obtained from https://github.com/
AI4Bharat/indicnlp_catalog

Appendix B

Parallel Corpora Sources
for the Indic Multilingual
Task

Table 53 lists the parallel corpora recommended
for the Indic multilingual translation task.

44