Overview of the 8th Workshop on Asian Translation
Toshiaki Nakazawa
The University of Tokyo

Hideki Nakayama
The University of Tokyo

nakazawa@logos.t.u-tokyo.ac.jp

nakayama@ci.i.u-tokyo.ac.jp

Chenchen Ding and Raj Dabre and Shohei Higashiyama
National Institute ofInformation and Communications Technology{chenchen.ding, raj.dabre, shohei.higashiyama}@nict.go.jp

Hideya Mino and Isao GotoNHK

Win Pa Pa
University of Conputer Study, Yangon{mino.h-gq, goto.i-es}@nhk.or.jp

winpapa@ucsy.edu.mm

Anoop Kunchukuttan
Microsoft AI and Research

Shantipriya Parida
Idiap Research Instituteanoop.kunchukuttan@microsoft.com

shantipriya.parida@idiap.chOndřej Bojar
Charles University, MFF, ÚFALChenhui Chu
Kyoto University

Akiko Eriguchi
Microsoft

bojar@ufal.mff.cuni.cz

chu@i.kyoto-u.ac.jp

akikoe@microsoft.com

Kaori Abe
Tohoku University

Yusuke Oda
Tohoku University, LegalForce

Sadao Kurohashi
Kyoto University

abe-k@ecei.tohoku.ac.jp

yusuke.oda.c1@tohoku.ac.jp

kuro@i.kyoto-u.ac.jp

Abstract• MultiIndicMT: Bengali / Gujarati / Hindi /Kannada / Malayalam / Marathi / Odia / Punjabi / Tamil / Telugu ↔ English translation

This paper presents the results of the sharedtasks from the 8th workshop on Asian translation (WAT2021). For the WAT2021, 28 teams
participated in the shared tasks and 24 teams
submitted their translation results for the human evaluation. We also accepted 5 research
papers. About 2,100 translation results were
submitted to the automatic evaluation server,and selected submissions were manually evaluated.

1• Restricted Translation Task: Japanese ↔ English translation• Ambiguous MSCOCO Task: Japanese ↔ English multi-modal translation
All the tasks are explained in Section 2.
WAT is a unique workshop on Asian languagetranslation with the following characteristics:

Introduction

The Workshop on Asian Translation (WAT) is an
open evaluation campaign focusing on Asian languages. Following the success of the previous
workshops WAT2014-WAT2020 (Nakazawa et al.,2020), WAT2021 brings together machine translation researchers and users to try, evaluate, shareand discuss brand-new ideas for machine translation. We have been working toward practical use
of machine translation among all Asian countries.
For the 8th WAT, we included the following newtasks:• Open innovation platform
Due to the fixed and open test data, we can repeatedly evaluate translation systems on thesame dataset over years. WAT receives submissions at any time; i.e., there is no submission deadline of translation results w.r.t automatic evaluation of translation quality.
• Domain and language pairs
WAT is the world’s first workshop thattargets scientific paper domain, andChinese↔Japanese and Korean↔Japanese
language pairs.

• Malayalam Visual Genome Task: English →Malayalam multi-modal translation
1

Proceedings of the 8th Workshop on Asian Translation, pages 1–45
Bangkok, Thailand (online), August 5-6, 2021. ©2021 Association for Computational Linguistics

• Evaluation method
Evaluation is done both automatically andmanually. Firstly, all submitted translation results are automatically evaluated using three
metrics: BLEU, RIBES and AMFM. Amongthem, selected translation results are assessedby two kinds of human evaluation: pairwise
evaluation and JPO adequacy evaluation.

2

Tasks

2.1

ASPEC+ParaNatCom Task

Dev
2,000
2,000
2,000

DevTest
2,000
2,000
2,000

Test-N
5,204
5,230
5,668

Lang
zh-ja
ko-ja
en-ja

Test-N1
2,000
2,000
2,000

Test-N2
3,000
3,000
3,000

Test-N3
204
230
668

Test-EP
1,151––

collaboration with NICT. The corpus consists ofChinese-Japanese, Korean-Japanese and EnglishJapanese patent descriptions whose International
Patent Classification (IPC) sections are chemistry,electricity, mechanical engineering, and physics.
At WAT2021, the patent task has two subtasks:
normal subtask and expression pattern subtask.
Both subtasks use common training, development
and development-test data for each language pair.
The normal subtask for three language pairs uses
four test datasets with different characteristics:• test-N: union of the following three sets;

Document-level Business Scene Dialogue
Translation• test-N1: patent documents from patent families published between 2011 and 2013;

There are a lot of ready-to-use parallel corpora
for training machine translation systems, however,most of them are in written languages such as web
crawl, news-commentary, patents, scientific papers
and so on. Even though some of the parallel corpora are in spoken language, they are mostly spoken by only one person (TED talks) or contain a lotof noise (OpenSubtitle). Most of other MT evaluation campaigns adopt the written language, monologue or noisy dialogue parallel corpora for theirtranslation tasks. Traditional ASPEC translation
tasks are sentence-level and the translation qualityof them seem to be saturated. To move to a highly
topical setting of translation of dialogues evaluatedat the level of documents, WAT uses BSD Corpus2 (The Business Scene Dialogue corpus) for thedataset including training, development and test
data for the first time this year. Participants of thistask must get a copy of BSD corpus by themselves.
2.3

Train
1,000,000
1,000,000
1,000,000

Table 1: Statistics for JPC

Traditional ASPEC translation tasks are sentencelevel and the translation quality of them seem tobe saturated. We think it’s high time to move on to
document-level evaluation. For the first year, we
use ParaNatCom1 (Parallel English-Japanese abstract corpus made from Nature Communications
articles) for the development and test sets of theDocument-level Scientific Paper Translation subtask. We cannot provide document-level training
corpus, but you can use ASPEC and any other extra resources.
2.2

Lang
zh-ja
ko-ja
en-ja

• test-N2: patent documents from patent families published between 2016 and 2017; and• test-N3: patent documents published between
2016 and 2017 where target sentences aremanually created by translating source sentences.
The expression pattern subtask for zh→ja pair uses
test-EP data. The test-EP data consists of sentences
annotated with expression pattern categories: title
of invention (TIT), abstract (ABS), scope of claim(CLM) or description (DES). The corpus statistics
are shown in Table 1. Note that training, development, development-test and test-N1 data are thesame as those used in WAT2017.
2.4 Newswire (JIJI) Task
The Japanese ↔ English newswire task uses JIJI
Corpus which was constructed by Jiji Press Ltd.
in collaboration with NICT and NHK. The corpus
consists of news text that comes from Jiji Press
news of various categories including politics, economy, nation, business, markets, sports and so on.
The corpus is partitioned into training, development, development-test and test data, which con-

JPC Task

JPO Patent Corpus (JPC) for the patent tasks wasconstructed by the Japan Patent Oﬀice (JPO) in1

http://www2.nict.go.jp/astrec-att/member/mutiyama/paranatcom/
2
https://github.com/tsuruoka-lab/BSD

2

Training
Test set I

Test set II

Test
DevTestDev
Test-2Dev-2
Context for Test-2
Context for Dev-2

Yangon (UCSY), Myanmar. The corpus consists of 200 thousand Myanmar-English parallel sentences collected from different domains, including news articles and textbooks.

0.2 M sentence pairs
2,000 sentence pairs
2,000 sentence pairs
2,000 sentence pairs
1,912 sentence pairs
497 sentence pairs
567 article pairs
135 article pairs

The ALT corpus has been manually segmented
into words (Ding et al., 2018, 2019), and the UCSY
corpus is unsegmented. A script to tokenize the
Myanmar data into writing units is released withthe data. The automatic evaluation of Myanmartranslation results is based on the tokenized writing units, regardless to the segmented words in the
ALT data. However, participants can make a use of
the segmentation in ALT data in their own manner.
The detailed composition of training, development, and test data of the Myanmar-English translation tasks are listed in Table 4. Notice that bothof the corpora have been modified from the data
used in WAT2018.

Table 2: Statistics for JIJI Corpus

sists of Japanese-English sentence pairs. In addition to the test set (test set I) that has been provided from WAT 2017, a test set (test set II) with
document-level context has also been provided
from WAT 2020. These test sets are as follows.
Test set I : A pair of test and reference sentences.
The references were automatically extracted
from English newswire sentences and not
manually checked. There are no context data.

2.6 NICT-SAP Task
In WAT2021, we decided to continue the
WAT2020 task for joint multi-domain multilingual neural machine translation involving 4
low-resource Asian languages: Thai (Th), Hindi(Hi), Malay (Ms), Indonesian (Id). English (En) is
the source or the target language for the translation
directions being evaluated. The purpose of thistask was to test the feasibility of multi-domain
multilingual solutions for extremely low-resource
language pairs and domains.
Naturally thesolutions could be one-to-many, many-to-one
or many-to-many NMT models. The domains
in question are Wikinews and IT (specifically,
Software Documentation). The total number of
evaluation directions are 16 (8 for each domain).
There is very little clean and publicly available
data for these domains and language pairs and thus
we encouraged participants to not only utilize thesmall Asian Language Treebank (ALT) parallel
corpora (Thu et al., 2016) but also the parallel
corpora from OPUS3 , other WAT tasks (past andpresent) and WMT4 . The ALT dataset contains
18,088, 1,000 and 1,018 training, development
and testing sentences. As for corpora for the IT
domain we only provided evaluation (dev andtest sets) corpora5 (Buschbeck and Exel, 2020)and encouraged participants to consider GNOME,UBUNTU and KDE corpora from OPUS. We

Test set II : A pair of test and reference sentences
and context data that are articles including test
sentences. The references were automaticallyextracted from English newswire sentences
and manually selected. Therefore, the quality of the references of test set II is better thanthat of test set I.
The statistics of JIJI Corpus are shown in Table 2.
The definition of data use is shown in Table 3.
Participants submit the translation results of one
or more of the test data.
The sentence pairs in each data are identified
in the same manner as that for ASPEC using themethod from (Utiyama and Isahara, 2007).
2.5

ALT and UCSY Corpus

The parallel data for Myanmar-English translation
tasks at WAT2021 consists of two corpora, the ALT
corpus and UCSY corpus.
• The ALT corpus is one part from the Asian
Language Treebank (ALT) project (Rizaet al., 2016), consisting of twenty thousand
Myanmar-English parallel sentences fromnews articles.
• The UCSY corpus (Yi Mon Shwe Sin andKhin Mar Soe, 2018) is constructed by the
NLP Lab, University of Computer Studies,3

http://opus.nlpl.eu/
http://www.statmt.org/wmt20/
5
Software Domain Evaluation Splits
4

3

Task

Use
Training
Test set I

Japanese to English
Test set II

To be translated
Reference
Test-2
Context
Reference

Training
Test set I
English to Japanese
Test set II

To be translated
Reference
To be translated
Context in English for Test-2
ReferenceContent
Training, DevTest, Dev, Dev-2, context for Dev2
Test in Japanese
Test in English
Test-2 in JapaneseContext in Japanese for Test-2
Test-2 in English
Training, DevTest, Dev, Dev-2, context for Dev2
Test in English
Test in JapaneseTest-2 in English
Context in English for Test-2
Test-2 in Japanese

Table 3: Definition of data use in the Japanese ↔ English newswire task
Corpus
ALT
UCSY
All

Train
18,088204,539
222,627

Dev
1,000
–
1,000

Test
1,018
–
1,018

Lang.pair
Ja↔Ru

Ja↔En

Table 4: Statistics for the data used in Myanmar-English
translation tasks
Split
Train
Dev
Test

Domain
ALT
IT
ALT
IT
ALT
IT

Hi
254,242
2,016
2,073

Language Pair
Id
Ms
18,088
158,472 506,739
1,000
2,023
2,050
1,018
2,037
2,050

Ru↔En

#sent.
12,356486
600
47,082589
600
82,072313
600#tokens
341k / 229k
16k / 11k
22k / 15k
1.27M / 1.01M
21k / 16k
22k / 17k
1.61M / 1.83M
7.8k / 8.4k
15k / 17k

#types
22k / 42k
2.9k / 4.3k
3.5k / 5.6k
48k / 55k
3.5k / 3.8k
3.5k / 3.8k
144k / 74k
3.2k / 2.3k
5.6k / 3.8k

Table 6: In-Domain data for the Russian–Japanese task.

Th
74,497et al., 2019) which belongs to the news commentary domain. This dataset was manually aligned
and cleaned and is trilingual. It can be used toevaluate Russian↔English translation quality as
well but this is beyond the scope of this years
sub-task. Refer to Table 6 for the statistics of
the in-domain parallel corpora. In addition, we
encouraged the participants to use out-of-domain
parallel corpora from various sources such as
KFTT,8 JESC,9 TED,10 ASPEC,11 UN,12 Yandex13 and Russian↔English news-commentary
corpus.14 This year we also encouraged participants to use any corpora from WMT 202015 andWMT 202116 involving Japanese, Russian, and English as long as it did not belong to the news commentary domain to prevent any test set sentencesfrom being unintentionally seen during training.

2,049
2,050

Table 5: The NICT-SAP task corpora splits. The corpora belong to two domains: wikinews (ALT) and software documentation (IT). The Wikinews corpora are Nway parallel.

also encouraged the use of monolingual corpora
expecting that it would be for pre-trained NMT
models such as BART/MBART (Lewis et al.,2020; Liu et al., 2020). In Table 5 we give
statistics of the aforementioned corpora which weused for the organizer’s baselines. Note that theevaluation corpora for both domains are createdfrom documents and thus contain document levelmeta-data. Participants were encouraged to use
document level approaches. Note that we do not
exhaustively list6 all available corpora here andparticipants were not restricted from using any
corpora as long as they are freely available.
2.7

Partition
train
development
test
train
development
test
train
developmenttest8

http://www.phontron.com/kftt/
https://datarepository.wolframcloud.com/
resources/Japanese-English-Subtitle-Corpus
10
https://wit3.fbk.eu/
11
http://lotus.kuee.kyoto-u.ac.jp/ASPEC/12https://cms.unov.org/UNCorpus/
13
https://translate.yandex.ru/corpus?lang=en
14
http://lotus.kuee.kyoto-u.ac.jp/WAT/
News-Commentary/news-commentary-v14.en-ru.
filtered.tar.gz
15
http://www.statmt.org/wmt20/
translation-task.html
16
http://www.statmt.org/wmt21/
translation-task.html
9News Commentary Task

For the Russian↔Japanese task we asked participants to use the JaRuNC corpus7 (Imankulova
6

http://lotus.kuee.kyoto-u.ac.jp/WAT/
NICT-SAP-Task
7https://github.com/aizhanti/JaRuNC

4

sourcealtbibleuedin
cvit-pib
iitb
jw
mtenglish2odia
nlpc
odiencorp
opensubtitles
pmi
tanzilted2020
ufal
urst
wikimatrix
wikititles
Grand Totalbn
20,106

gu

91,985

15,60958,264

278,307

310,094hi
20,106
62,073
266,545
1,603,080
509,594

kn

ml

mr

or

pa

61,70761,30043,08760,876114,220

94,494

101,092

115,968

62,191
44,720362,816

270,346

388,364

673,232

192,904

303,991ta

te

34,846
31,37390,854
411,097
23,306
187,052
10,31892,31950,349187,080
46,75941,57815,691

28,901
2,253

383,313
26,916
187,081
5,990

28,97431,966

22,60828,294749

32,14032,63893,540
11,105
166,866

27,06333,38095,159102,1311,354,152

91,9085,236

65,000280,566
1,302,737

231,459
11,665517,901

3,069,364

396,852

71,508124,304

1,142,011

621,328

252,160

518,499

457,402

Grand Total
40,212
323,756
930,375
1,603,080
3,289,648
34,846
31,37390,854
945,932
326,302654,753
120,709
166,866
65,000894,904
113,796
9,632,406

Table 7: Statistics of the filtered parallel corpora provided by the organizers. The target language is English.

Language
as
bn
en
gu
hi
kn
,
ml
mr
or
pa
ta
te

#Lines
1.39M
39.9M
54.3M
41.1M
63.1M
53.3M
50.2M
34.0M
6.94M
29.2M
31.5M
47.9M

composed by the Prime Minister of India. The languages involved are Hindi (Hi), Marathi (Mr), Kannada (Kn), Tamil (Ta), Telugu (Te), Gujarati (Gu),
Malayalam (Ml), Bengali (Bn), Oriya (Or), Punjabi (Pa) and English (En). Compared to 2020, we
have 3 additional languages leading to a total of 10
Indian languages, 4 of which are Dravidian and therest are Indo-Aryan. English is either the source or
the target language during evaluation leading to a
total of 20 translation directions. Due to the N-way
nature of the evaluation corpus we can also evaluate 90 Indian language to Indian language translation pairs but this may be the focus in future workshops.
The objective of this task, like the Indic languages tasks in 2018 and 2020, was to evaluate theperformance of multilingual NMT models. The
desired solution could be one-to-many, many-toone or many-to-many NMT models. We provided
a filtered parallel corpus collection spanning all
languages19 which was split into training, development and test sets. This dataset was created byfirst creating an evaluation set of 3,390 11-way
sentences (1,000 for development and 2,390 for
testing) and then filtering them out from all parallel
corpora we could obtain at the time. Furthermore,
we made sure to filter out sentences from the 2020
evaluation set. This way the provided parallel
corpus can be safely used for benchmarking the2020 evaluation set as well. The filtered training
parallel corpora came from a variety of sources
such as: CVIT-PIB, PMIndia, IITB 3.0,20 JW,21

Table 8: Monolingual corpora statistics.

2.8

Indic Multilingual Task

Owing to the increasing interest in Indian languagetranslation and the success of the multilingual Indian languages tasks in 2018 (Nakazawa et al.,2018) and 2020 (Nakazawa et al., 2020), we decided to enlarge the scope of the 2020 task byadding new languages, scouring new data and creating an N-way parallel evaluation set. In 2020, theevaluation data came from the CVIT-PIB dataset17
but it did not contain suﬀicient N-way parallel sentences to evaluate on additional languages. To thisend, we decided to obtain evaluation corpora fromthe PMI dataset18 which contains suﬀicient N-way
parallel corpora spanning 10 Indian languages andEnglish and is similar (domain wise) to the CVITPIB dataset.
The evaluation data consists of various articles

19

http://lotus.kuee.kyoto-u.ac.jp/WAT/
indic-multilingual/indic_wat_2021.tar.gz
20
http://www.cfilt.iitb.ac.in/iitb_parallel/21
http://opus.nlpl.eu/JW300.php

17

http://preon.iiit.ac.in/~jerin/resources/
datasets/pib_v1.3.tar
18
http://data.statmt.org/pmindia

5

NLPC,22 UFAL EnTam,23 Uka Tarsadia,24 Wiki
Titles (ta,25 gu,26 )ALT,27 OpenSubtitles,28 Bibleuedin,29 MTEnglish2Odia,30 OdiEnCorp 2.0,31
TED,32 and WikiMatrix33 . Additionally we listed
the CCAligned corpus34 to be used despite its
poor quality which applies to WikiMatrix as well.
We also provided filtered monolingual corpora35
sourced from PMI and we also encouraged the useof monolingual corpora from the IndicCorp.36 The
statistics of this corpus are given in table 8. We
expected that this year, the novel way of using themonolingual corpora would be to pre-train NMT
models such as BART/MBART (Lewis et al.,2020; Liu et al., 2020). In general we encouraged
participants to focus on multilingual NMT (Dabreet al., 2020) solutions.
Detailed statistics for the aforementioned corpora can be found in Table 7. We also listed additional sources of corpora for participants to use.
Our organizer’s baselines used the PMI corpora fortraining as it is the in-domain corpus.
2.9

DatasetTraining Set
D-Test
E-Test (EV)C-Test (CH)Items
28,930
998
1,595
1,400

TokensEnglishHindi
143,164 145,448
4,922
4,978
7,853
7,852
8,186
8,639

Table 9: Statistics of Hindi Visual Genome 1.1 used for
the English→Hindi Multi-Modal translation task. One
item consists of a source English sentence, target Hindi
sentence, and a rectangular region within an image. The
total number of English and Hindi tokens in the dataset
also listed. The abbreviations EV and CH are used in
the oﬀicial task names in WAT scoring tables.

2019a,b).37
The statistics of HVG 1.1 are given in Table 9.
One “item” in HVG consists of an image with arectangular region highlighting a part of the image, the original English caption of this region andthe Hindi reference translation. Depending on the
track (see 2.9.1 below), some of these item components are available as the source and some serve as
the reference or play the role of a competing candidate solution.

English→Hindi Multi-Modal Task

This task is running successfully in WAT since2019 and attracted many teams working on multimodal machine translation and image captioningin Indian languages (Nakazawa et al., 2019, 2020).
For English→Hindi multi-modal translationtask, we asked the participants to use Hindi Visual Genome 1.1 corpus (HVG, Parida et al.,2.9.1 English→Hindi Multi-Modal Task
Tracks
1. Text-Only Translation (labeled “TEXT” in
WAT oﬀicial tables): The participants are
asked to translate short English captions (text)
into Hindi. No visual information can be used.
On the other hand, additional text resources
are permitted (but they need to be specified in
the corresponding system description paper).

22
https://github.com/nlpc-uom/
English-Tamil-Parallel-Corpus
23
http://ufal.mff.cuni.cz/~ramasamy/parallel/
html/
24
https://github.com/shahparth123/eng_guj_
parallel_corpus
25
http://data.statmt.org/wikititles/v2/
wikititles-v2.ta-en.tsv.gz
26
http://data.statmt.org/wikititles/v1/
wikititles-v1.gu-en.tsv.gz
27
http://www2.nict.go.jp/astrec-att/member/mutiyama/ALT28
http://opus.nlpl.eu/OpenSubtitles-v2018.
php
29
http://opus.nlpl.eu/bible-uedin.php
30
https://github.com/soumendrak/
MTEnglish2Odia
31
https://ufal.mff.cuni.cz/odiencorp
32
http://lotus.kuee.kyoto-u.ac.jp/WAT/
indic-multilingual/33
https://github.com/facebookresearch/LASER/
tree/master/tasks/WikiMatrix34
http://www.statmt.org/cc-aligned/35
http://lotus.kuee.kyoto-u.ac.jp/WAT/
indic-multilingual/filteredmono.tar.gz
36
https://indicnlp.ai4bharat.org/corpora

2. Hindi Captioning (labeled “HI”): The participants are asked to generate captions in Hindi
for the given rectangular region in an input image.
3. Multi-Modal Translation (labeled “MM”):
Given an image, a rectangular region in it andan English caption for the rectangular region,
the participants are asked to translate the English text into Hindi. Both textual and visual
information can be used.
The English→Hindi multi-modal task includes
three tracks as illustrated in Figure 1.
37
https://lindat.mff.cuni.cz/repository/
xmlui/handle/11234/1-3267

6

Text-Only MT

Hindi Captioning

Image
Source Text–
The woman is waiting to cross –the street

System Output

मिहला सड़क पार करने का इं तजार कर सड़क पर काररही है
Woman waiting to cross the Car on the roadstreetGlossMulti-Modal MT

A blue wall beside tennis courtटे िनस कोटर् के बगल में एक नीली दीवार
a blue wall next to the tennis
court

Reference Solution एक मिहला सड़क पार करने के िलए इं - सड़क के िकनारे खड़ी कारेंटे िनस कोटर् के बगल में एक नीली दीवार
तजार कर रही है
the woman is waiting to cross Cars parked along the side of the A blue wall beside the tennis
Gloss
the street
road
court

Figure 1: An illustration of the three tracks of WAT 2021 English→Hindi Multi-Modal Task.

2.10.1 English→Malayalam Multi-Modal
Task Tracks
1. Text-Only Translation (labeled “TEXT” in
WAT oﬀicial tables): The participants are
asked to translate short English captions (text)
into Malayalam. No visual information canbe used. On the other hand, additional text
resources are permitted (but they need to be
specified in the corresponding system description paper).

English Text: Two elephants standing in the water.
Malayalam Text: െവള്ളത്തിൽ നിൽന്ന ര് ആനകൾ

Figure 2: Sample item from Malayalam Visual Genome(MVG), Image with specific region and its description.

2.10

2. Malayalam Captioning (labeled “ML”): The
participants are asked to generate captions in
Malayalam for the given rectangular region in
an input image.

English→Malayalam Multi-Modal Task

3. Multi-Modal Translation (labeled “MM”):
Given an image, a rectangular region in it andan English caption for the rectangular region,
the participants are asked to translate the English text into Malayalam. Both textual andvisual information can be used.

This task is introduced this year using the first multimodal machine translation dataset in Malayalam
language. For English→Malayalam multi-modaltranslation task we asked the participants to use the
Malayalam Visual Genome corpus (MVG for short
Parida and Bojar, 2021)38 .
The statistics of MVG are given in Table 10.
One “item” in MVG consists of an image with arectangular region highlighting a part of the image,the original English caption of this region and the
Malayalam reference translation as shown in Figure 2. Depending on the track (see 2.10.1 below),some of these item components are available as thesource and some serve as the reference or play therole of a competing candidate solution.

2.11 Flickr30kEnt-JP Japanese↔English
Multi-Modal Tasks
The goal of Flickr30kEnt-JP Japanese↔English
multi-modal task39 is to improve translation performance with the help of another modality (images) associated with input sentences. For both
English→Japanese and Japanese→English tasks,
we use the Flickr30k Entities Japanese (F30kEntJp) dataset (Nakayama et al., 2020). This is an38
https://lindat.mff.cuni.cz/repository/
xmlui/handle/11234/1-3533

39

7

https://nlab-mpg.github.io/wat2021-mmt-jp/

Dataset
Training Set
D-Test
E-Test (EV)C-Test (CH)Items
28,930
998
1,595
1,400

TokensEnglish Malayalam
143,112
107,126
4,922
3,619
7,853
6,689
8,186
6,044

without resource constraints. In the constrained
setting, external resources such as additional dataand pre-trained models (with external data) are notallowed, except for pre-trained convolutional neural networks (for visual analysis) and basic linguistic tools such as taggers, parsers, and morphological analyzers.

Table 10: Statistics of Malayalam Visual Genome used
for the English→Malayalam Multi-Modal translation
task. One item consists of a source English sentence,target Hindi sentence, and a rectangular region within
an image. The total number of English and Malayalam
tokens in the dataset also listed. The abbreviations EV
and CH are used in the oﬀicial task names in WAT scoring tables.
Data
Train
Dev
Test

Images29,7831,000
1,000

2.12 Ambiguous MS COCO
Japanese↔English Multimodal Task
This is another Japanese–English multimodal machine translation task. We provide the Japanese–
English Ambiguous MS COCO dataset (Merritt
et al., 2020) for validation and testing, whichcontains ambiguous verbs that may require visual information in images for disambiguation.
The validation and testing sets contain 230 and
231 Japanese–English sentence pairs, respectively.
The Japanese sentences are translated from theEnglish sentences in the original Ambiguous MS
COCO dataset.43
Participants can use the constrained and unconstrained training data to train their multimodal machine translation system. In the constrained setting,
only the Flickr30kEntities Japanese (F30kEnt-Jp)
dataset44 can be used as training data. In the unconstrained setting, the MS COCO English data45
and STAIR Japanese image captions46 can be usedas additional training data.
We prepare a baseline using the double attention on image region method following (Zhaoet al., 2020) for both Japanese→English andEnglish→Japanese directions.

Sentences/Tokens
EnglishJapanese
148,915/1.99M 148,910∗ /2.50M
5,000/67,288
5,000/84,017
1,000/10,876
1,000/16,113

Table 11:
Statistics of the dataset used forJapanese↔English multi-modal tasks.
Here weuse the MeCab tokenizer to count Japanese tokens.
∗
Some of the original English sentences are actuallybroken so we did not provide their translations.

extended dataset of the Flickr30k40 and Flickr30k
Entities41 datasets where manual Japanese translations are added. Notably, it has the annotations of
many-to-many phrase-to-region correspondences
in both English and Japanese captions, which are
expected to strongly supervise multimodal grounding and provide new research directions.
This year, from the same shared tasks in WAT
2020, we increased the number of parallel sentences for training and validation. We summarize
the statistics of the dataset for this year in Table
11. We use the same splits of training, validation and test data specified in Flickr30k Entities.
For the training and the validation data, we use
the F30kEnt-Jp version 2.0 which is publicly available.42 The original Flickr30k has five English sentences for each image. While the Japanese set for
WAT 2020 had the translations of only the first two
sentences, this year we have all five translations foreach image. Therefore, we can use five parallel sentences for each image to train and validate the systems. The test data remain exactly the same as in
WAT 2020, where phrase-to-region annotation is
not included.
There are two settings of submission: with and2.13 Restricted Translation Task
Despite recent success of NMT, the MT systems
still struggle to generate translation with a consistent terminology. Consistency is the key to clear
and accurate translation, especially when translating documents in a specific field, for instance, science or business and marketing contexts, requiring
technical terms and proper nouns to get translated
into the corresponding unique expressions continuously in the entire documents. To tackle this inconsistent translation issue, we have designed Restricted Translation task at WAT 2021.
In the restricted translation task, participants are
required to submit a system that translates source43

http://www.statmt.org/wmt17/multimodal-task.html
https://github.com/nlab-mpg/Flickr30kEnt-JP
45
https://cocodataset.org/#captions-201546
https://stair-lab-cit.github.io/STAIR-captions-web/4440

http://shannon.cs.illinois.edu/DenotationGraph/41
http://bryanplummer.com/Flickr30kEntities/
42
https://github.com/nlab-mpg/Flickr30kEnt-JP

8

Dev.
DevtestTest

En-Ja(# phrase, # char)(2.8, 164)
(3.2, 18.2)(3.3, 18.1)

Ja-En(# phrase, # word)(2.8, 6.6)
(3.2, 7.3)
(3.2, 7.4)

participation of teams across WAT2021 tasks andindicates which tasks included manual evaluation.
The human evaluation was conducted only for thetasks with the check marks in “human eval” line.
There were no participants in the Newswire(JIJI) task, BSD task and JaRuNC task.

Table 12: Statistics of the restricted vocabulary in theevaluation data. We report average number of phrases
and characters/words per source sentence.

4 Baseline Systems
Human evaluations of most of WAT tasks wereconducted as pairwise comparisons between thetranslation results for a specific baseline systemand translation results for each participant’s system. That is, the specific baseline system served
as the standard for human evaluation. At WAT
2021, we adopted some of neural machine translation (NMT) as baseline systems. The details of
the NMT baseline systems are described in this section.
The NMT baseline systems consisted of publicly
available software, and the procedures for building the systems and for translating using the systems were published on the WAT web page.47 We
also have SMT baseline systems for the tasks thatstarted at WAT 2017 or before 2017. The baseline
systems are shown in Tables 15, 16, and 17. SMT
baseline systems are described in the WAT 2017
overview paper (Nakazawa et al., 2017). The commercial RBMT systems and the online translation
systems were operated by the organizers. We notethat these RBMT companies and online translationcompanies did not submit their systems. Because
our objective is not to compare commercial RBMT
systems or online translation systems from companies that did not themselves participate, the system
IDs of these systems are anonymous in this paper.

texts under target vocabulary constraints. At inference time, such a restricted vocabulary is provided as a list of target words, consisting of scientific technical terms in the target language, and thesystem outputs must contain all these target words.
For the English↔Japanese translation tasks, weemploy the ASPEC corpus and allow to use otherexternal data source. We built the restricted vocabulary lists by asking 10 bilingual speakers to manually extract the scientific technical terms from theevaluation data sets (“dev/devtest/test”). Table 12
reports the data statistics of the restricted vocabulary in the evaluation data.
We evaluate systems with two distinct metrics: 1) BLEU score as a conventional translation accuracy and 2) a consistency score: the ratio of the number of sentences satisfying exact
match of given constraints over the whole test corpus. For the “exact match” evaluation, we conduct the following process. In English, we simply lowercase hypotheses and constraints, thenjudge character-level sequence matching (including whitespaces) for each constraint. In Japanese,
we judge character-level sequence matching (including whitespaces) for each constraint without
preprocessing. For the final ranking, we also calculate the combined score of both: calculating BLEU
with only the exact match sentences. We note that,in this scenario, the brevity score in BLEU does
not carry its usual meaning, but the n-gram scores
maintain their consistency.

3

4.1 Tokenization
We used the following tools for tokenization.
4.1.1 For ASPEC, JPC, JIJI, and
ALT+UCSY

Participants• Juman version 7.048 for Japanese segmentation.
• Stanford Word Segmenter version 2014-010449 (Chinese Penn Treebank (CTB) model)
for Chinese segmentation.

Table 13 shows the participants in WAT2021. The
table lists 24 organizations from various countries,
including Japan, India, USA, Singapore, Myanmar,
Thailand, Korea, Poland, Denmark and Switzerland.
2,100 translation results by 28 teams were submitted for automatic evaluation and about 360
translation results by 24 teams were submitted forthe human evaluation. Table 14 summarizes the47

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2021/baseline/baselineSystems.html
48
http://nlp.ist.i.kyoto-u.ac.jp/EN/index.
php?JUMAN
49
http://nlp.stanford.edu/software/segmenter.
shtml

9

Team ID
TMU
NTT
NICT-2
NICT-5
NLPHutTMEKU*goodjob
YCC-MT1
YCC-MT2
NECTECmcairtnictrbsakuraIIT-H*gauvar*JBJBJB
SRPOL
NHK
CFILT
iitp
Volta
coastal
CFILT-IITB
CNLP-NITS-PP
Bering Lab
tpt_watOrganization
Tokyo Metropolitan University
NTT Corporation
NICT
NICT
Idiap Research Institute Switzerland, IIT BHU, BITS Pilani India, KIIT University India, Silicon Techlab pvt. Ltd India, University of Chicago
Tokyo Metropolitan University, Ehime University, Kyoto University
Dalian University of Technology
University of Technology (Yatanarpon Cyber City)
University of Technology (Yatanarpon Cyber City)
National Electronics and Computer Technology Center (NECTEC)
CAIR
NICT
Rakuten Institute of Technology Singapore, Rakuten Asia.
International Institue of Information Technology
AmazonIndivisual participantSamsung R&D Poland
NHK
Computing for Indian Language Technology
IIT PatnaInternational Institute of Information Technology Hyderabad
University of Copenhagen
Indian Institute of Technology Bombay
NIT Silchar
Bering Lab
Transperfect Translations

Country
Japan
Japan
Japan
Japan
Switzerland, India, USA
Japan
China
Myanmar
MyanmarThailand
India
Japan
Singapore
India
Singapore
Korea
Poland
Japan
India
India
India
Denmark
India
India
South Korea
USA

Table 13: List of participants who submitted translations for the human evaluation in WAT2021 (Note: teams
with ‘*’ marks did not submit their system description papers, therefore the evaluation results are UNOFFICIAL
according to our policy)

10Team ID
TMU
NTT
NICT-2
goodjob
YCC-MT1
YCC-MT2
NECTEC
nictrbsakuraNHK
human evalTeam ID
TMU
NLPHutTMEKU
sakuraiitp
Volta
CNLP-NITS-PP
Bering Lab
tpt_wathuman eval

ASPEC +
ParaNatCom
EJ

ASPEC
Restricted
EJ
JE
✓
✓
✓

ALT +
UCSY
En-My My-En

✓

✓
✓

✓

EJ
✓

✓
✓
✓

JE
✓

✓
✓
✓

✓

✓

✓
✓

✓
✓

JPC
CJ JC

✓
✓

✓
✓

KJ
✓

✓
✓

NICT-SAP
En-Hi/Id/Ms/Th Hi/Id/Ms/Th-En
IT Wikinews IT Wikinews
✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

JK
✓

✓
✓

✓

En-Hi
TX HI MM

Multimodal
En-Ml
FlickrTX HI EJ JE

✓

✓

✓
✓
✓
✓

✓
✓
✓

✓

✓

✓

✓

✓

✓
✓

✓

✓

✓

MS COCO
EJ✓✓

Indic21
En-X
X-En
Team ID
Bn Kn Ml Mr Or Hi Gu Pa Ta Te Bn Kn Ml Mr Or Hi Gu Pa Ta Te
NICT-5✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓NLPHut✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓mcairtsakura✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓IIT-Hgauvar✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓JBJBJB✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓SRPOL✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓CFILT✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓coastal
CFILT-IITB✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓human eval ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓Table 14: Submissions for each task by each team. E and J denote English and Japanese respectively. The humanevaluation was conducted only for the tasks with the check marks in ”human eval” line.

11

System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT XRBMT X
Online X
Online X
AIAYN

Table 16: Baseline Systems II

Table 15: Baseline Systems I

12System ID
NMT
SMT Phrase
SMT Hiero
SMT S2T
SMT T2S
RBMT XRBMT X
Online X
Online X

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
RBMT
Other
Other
NMT
✓
✓
✓

✓
✓
✓

ja-en
✓
✓
✓
✓

SystemOpenNMT’s NMT with attention
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
PC-Transer V13 (Commercial system)
Google translate
Bing translator

SystemOpenNMT’s NMT with attention
Moses’ Phrase-based SMT
Moses’ Hierarchical Phrase-based SMT
Moses’ String-to-Tree Syntax-based SMT and Berkeley parser
Moses’ Tree-to-String Syntax-based SMT and Berkeley parser
The Honyaku V15 (Commercial system)
ATLAS V14 (Commercial system)
PAT-Transer 2009 (Commercial system)
PC-Transer V13 (Commercial system)
J-Beijing 7 (Commercial system)
Hohrai 2011 (Commercial system)
J Soul 9 (Commercial system)
Korai 2011 (Commercial system)
Google translate
Bing translator
Google’s implementation of “Attention Is All You Need”

Type
NMT
SMT
SMT
SMT
SMT
RBMT
RBMT
Other
Other

✓
✓
✓

✓
✓

✓
✓

✓

zh-ja
✓
✓
✓

JIJI
ja-en en-ja
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓

✓
✓

ASPEC
en-ja ja-zh
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓

✓
✓

✓
✓
✓
✓

en-ja
✓
✓
✓

✓
✓

✓

JPC
ja-zh
✓
✓
✓
✓

✓

✓

ALT+UCSY
my↔en km↔en
✓
✓

✓
✓

✓
✓
✓

ja-en
✓
✓
✓
✓

✓
✓

✓
✓

✓

zh-ja
✓
✓
✓

✓
✓
✓
✓

ja↔ko
✓
✓
✓

Table 17: Baseline Systems III

13

System ID
NMT
NMT T2T
NMT OT
MNMT
MNMT2

SystemOpenNMT’s NMT with attention
Tensor2Tensor’s Transformer
OpenNMT-py’s Transformer
Multimodal NMT
Double-attention based Multimodal NMT

Type
NMT
NMT
NMT
NMT
NMT
✓

✓

ru↔ja

NICT+SAP
IT&Wikinews{hi,id,ms,th}↔en

NewsCommentary✓{bn,gu,hi,kn,ml,mr,or,pa,ta,te}↔en

Indic

✓

en-hi
✓

✓

✓

Multimodal
Flickr MS COCO
en-ml ja↔en
ja↔en

4.1.5 For English↔Japanese Multi-Modal
Tasks• The Moses toolkit for English and Indonesian
tokenization.
• Mecab-ko50 for Korean segmentation.
• Indic NLP Library51 (Kunchukuttan, 2020)
for Indic language segmentation.
• The tools included in the ALT corpus for
Myanmar and Khmer segmentation.
• subword-nmt52 for all languages.

• For English sentences, we applied lowercase,punctuation normalization, and the Moses tokenizer.
• For Japanese sentences, we used KyTea forword segmentation.

When we built BPE-codes, we merged source andtarget sentences and we used 100,000 for -s option. We used 10 for vocabulary-threshold whensubword-nmt applied BPE.
4.1.2

4.2 Baseline NMT Methods
We used the NMT models for all tasks. Unless mentioned otherwise we use the Transformer
model (Vaswani et al., 2017). We used OpenNMT (Klein et al., 2017) (RNN-model) for ASPEC, JPC, JIJI, and ALT tasks, tensor2tensor54 for
the News Commentary (JaRuNC), NICT-SAP andMultiIndicMT tasks and OpenNMT-py55 for othertasks.

For News Commentary• The Moses toolkit for English and Russian
only for the News Commentary data.
• Mecab53 for Japanese segmentation.
• Corpora are further processed by tensor2tensor’s internal pre/post-processing
which includes sub-word segmentation.
4.1.3

4.2.1 NMT with Attention (OpenNMT)For ASPEC, JPC, JIJI, and ALT tasks, we used
OpenNMT (Klein et al., 2017) as the implementation of the baseline NMT systems of NMT withattention (System ID: NMT). We used the following OpenNMT configuration.

For Indic and NICT-SAP Tasks• For the Indic task we did not perform any explicit tokenization of the raw data.

• encoder_type = brnn
• brnn_merge = concat• src_seq_length = 150• tgt_seq_length = 150• src_vocab_size = 100000• tgt_vocab_size = 100000• src_words_min_frequency = 1• tgt_words_min_frequency = 1• For the NICT-SAP task we only character segmented the Thai corpora as it was the only language for which character level BLEU was tobe computed. Other languages corpora were
not preprocessed in any way.
• Any subword segmentation or tokenization
was handled by the internal mechanisms of
tensor2tensor.
4.1.4

The default values were used for the other systemparameters.
We used the following data for training the NMT
baseline systems of NMT with attention.

For English→Hindi Multi-Modal andEnglish→Malayalam Tasks• Hindi Visual Genome 1.1 and Malayalam Visual Genome comes untokenized and we did
not use or recommend any specific externaltokenizer.

• All of the training data mentioned in Section 2 were used for training except for the ASPEC Japanese–English task. For the ASPEC
Japanese–English task, we only used train1.txt, which consists of one million parallel
sentence pairs with high similarity scores.
• All of the development data for each task wasused for validation.

• The standard OpenNMT-py sub-word segmentation was used for pre/post-processing
for the baseline system and each participant
used what they wanted.
50

https://bitbucket.org/eunjeon/mecab-ko/
https://github.com/anoopkunchukuttan/indic_
nlp_library52
https://github.com/rsennrich/subword-nmt
53
https://taku910.github.io/mecab/
5154

https://github.com/tensorflow/
tensor2tensor
55
https://github.com/OpenNMT/OpenNMT-py

14

4.2.2

Transformer (Tensor2Tensor)

parameters for the multi-modal task baseline. We
have generated the vocabulary of 32k sub-word
types jointly for both the source and target languages. The vocabulary is shared between theencoder and decoder.

For the News Commentary task, we used tensor2tensor’s56 implementation of the Transformer(Vaswani et al., 2017) and used default hyperparameter settings corresponding to the “base” model
for all baseline models. The baseline for the News
Commentary task is a multilingual model as described in Imankulova et al. (2019) which is trainedusing only the in-domain parallel corpora. We use
the token trick proposed by Johnson et al. (2017)
to train the multilingual model.
For the NICT-SAP task, we used tensor2tensor
to train many-to-one and one-to-many models
where the latter were trained with the aforementioned token trick. We used default hyperparameter settings corresponding to the “big” model.
Since the NICT-SAP task involves two domains forevaluation (Wikinews and IT) we used a modification of the token trick technique for domain adaptation to distinguish between corpora for different domains. In our case we used tokens such as 2alt and2it to indicate whether the sentences belonged to
the Wikinews or IT domain, respectively. For bothtasks we used 32,000 separate sub-word vocabularies. We trained our models on 1 GPU till convergence on the development set BLEU scores, averaged the last 10 checkpoints (separated by 1000
batches) and performed decoding with a beam ofsize 4 and a length penalty of 0.6.
For the MultiIndicMT task we trained unidirectional models using only the PMI corpus insteadof the entire training data. We intentionally used
the PMI corpus because its domain is the sameas that of the evaluation set. Due to lack of timeand resources we did not train multilingual models nor did we use additional data. We trained“transformer_base” models with shared vocabularies of 8,000 subwords. We trained our models on
1 GPU till convergence on the development set
BLEU scores, chose the model with the best development set BLEU and performed decoding with abeam of size 4 and a length penalty of 0.6.
4.2.3

5 Automatic Evaluation
5.1 Procedure for Calculating Automatic
Evaluation Score
We evaluated translation results by three metrics: BLEU (Papineni et al., 2002), RIBES(Isozaki et al., 2010) and AMFM (Banchs et al.,2015a). BLEU scores were calculated using
multi-bleu.perl in the Moses toolkit (Koehn
et al., 2007). RIBES scores were calculated using
RIBES.py version 1.02.4.57 AMFM scores were
calculated using scripts created by the technicalcollaborators listed in the WAT2021 web page.58
All scores for each task were calculated using thecorresponding reference translations.
Before the calculation of the automatic evaluation scores, the translation results were tokenizedor segmented with tokenization/segmentation
tools for each language. For Japanese segmentation, we used three different tools: Juman version
7.0 (Kurohashi et al., 1994), KyTea 0.4.6 (Neubig
et al., 2011) with full SVM model59 and MeCab
0.996 (Kudo, 2005) with IPA dictionary 2.7.0.60
For Chinese segmentation, we used two different
tools: KyTea 0.4.6 with full SVM Model in MSR
model and Stanford Word Segmenter (Tseng,2005) version 2014-06-16 with Chinese Penn
Treebank (CTB) and Peking University (PKU)
model.61 For Korean segmentation, we usedmecab-ko.62 For Myanmar and Khmer segmentations, we used myseg.py63 and kmseg.py64 .
For English and Russian tokenizations, we usedtokenizer.perl65 in the Moses toolkit. For57

http://www.kecl.ntt.co.jp/icl/lirg/ribes/
index.html
58
lotus.kuee.kyoto-u.ac.jp/WAT/WAT2021/
59
http://www.phontron.com/kytea/model.html
60
http://code.google.com/p/mecab/downloads/
detail?name=mecab-ipadic-2.7.0-20070801.tar.gz
61
http://nlp.stanford.edu/software/segmenter.
shtml
62
https://bitbucket.org/eunjeon/mecab-ko/63
http://lotus.kuee.kyoto-u.ac.jp/WAT/
my-en-data/wat2020.my-en.zip
64
http://lotus.kuee.kyoto-u.ac.jp/WAT/
km-en-data/km-en.zip
65
https://github.com/moses-smt/mosesdecoder/tree/RELEASE-2.1.1/scripts/tokenizer/
tokenizer.perl

Transformer (OpenNMT-py)

For the English→Hindi Multimodal andEnglish→Malayalam Multimodal tasks, weused the Transformer model (Vaswani et al., 2018)as implemented in OpenNMT-py (Klein et al.,2017) and used the “base” model with default
56
https://github.com/tensorflow/
tensor2tensor

15

WAT

The Workshop on Asian Translation
Submission
SUBMISSION
Logged in as: ORGANIZER
LogoutSubmission:
Human
Evaluation:

human evaluation

Publish theresults of theevaluation:

publishTeam Name:

ORGANIZER

Task:

en-ja

Submission
File:

ファイルを選択 選択されていません

Used Other
Resources:used other resources such as parallel corpora, monolingual corpora andparallel dictionaries in addition to oﬃcial corpora

Method:SMT

SystemDescription(public):

100
charactersor less

SystemDescription(private):100
charactersor less

SubmitGuidelines for submission:
System requirements:
The latest versions of Chrome, Firefox, Internet Explorer and Safari are supported for this site.
Before you submit ﬁles, you need to enable JavaScript in your browser.
File format:
Submitted ﬁles should NOT be tokenized/segmented. Please check the automatic evaluation procedures.
Submitted ﬁles should be encoded in UTF-8 format.
Translated sentences in submitted ﬁles should have one sentence per line, corresponding to each test sentence. The number of lines in the submitted ﬁle and that of thecorresponding test ﬁle should be the same.
Tasks:
en-ja, ja-en, zh-ja, ja-zh indicate the scientiﬁc paper tasks with ASPEC.
HINDENen-hi, HINDENhi-en, HINDENja-hi, and HINDENhi-ja indicate the mixed domain tasks with IITB Corpus.
JIJIen-ja and JIJIja-en are the newswire tasks with JIJI Corpus.
RECIPE{ALL,TTL,STE,ING}en-ja and RECIPE{ALL,TTL,STE,ING}ja-en indicate the recipe tasks with Recipe Corpus.
ALTen-my and ALTmy-en indicate the mixed domain tasks with UCSY and ALT Corpus.
INDICen-{bn,hi,ml,ta,te,ur,si} and INDIC{bn,hi,ml,ta,te,ur,si}-en indicate the Indic languages multilingual tasks with Indic Languages Multilingual Parallel Corpus.
JPC{N,N1,N2,N3,EP}zh-ja ,JPC{N,N1,N2,N3}ja-zh, JPC{N,N1,N2,N3}ko-ja, JPC{N,N1,N2,N3}ja-ko, JPC{N,N1,N2,N3}en-ja, and JPC{N,N1,N2,N3}ja-en indicate the patent tasks
with JPO Patent Corpus. JPCN1{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} are the same tasks as JPC{zh-ja,ja-zh,ko-ja,ja-ko,en-ja,ja-en} in WAT2015-WAT2017. AMFM is not calculated
for JPC{N,N2,N3} tasks.
Human evaluation:
If you want to submit the ﬁle for human evaluation, check the box "Human Evaluation". Once you upload a ﬁle with checking "Human Evaluation" you cannot change the ﬁle used
for human evaluation.
When you submit the translation results for human evaluation, please check the checkbox of "Publish" too.
You can submit two ﬁles for human evaluation per task.
One of the ﬁles for human evaluation is recommended not to use other resources, but it is not compulsory.
Other:
Team Name, Task, Used Other Resources, Method, System Description (public) , Date and Time(JST), BLEU, RIBES and AMFM will be disclosed on the Evaluation Site when you
upload a ﬁle checking "Publish the results of the evaluation".
You can modify some ﬁelds of submitted data. Read "Guidelines for submitted data" at the bottom of this page.
Back to top

Figure 3: The interface for translation results submission

Indonesian and Malay tokenizations, we usedtokenizer.perl actually sticking to the English
tokenization settings. For Thai tokenization, we
segmented the text at each individual character.
For Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu
tokenizations, we used Indic NLP Library66(Kunchukuttan, 2020). The detailed proceduresfor the automatic evaluation are shown on the
WAT evaluation web page.67

66
https://github.com/anoopkunchukuttan/indic_
nlp_library67
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/index.html

5.2 Automatic Evaluation System
The automatic evaluation system receives translation results by participants and automatically gives

16

evaluation scores to the uploaded results. As
shown in Figure 3, the system requires participants
to provide the following information for each submission:

5• Human Evaluation: whether or not they submit the results for human evaluation;

2

4
3

1

• Publish the results of the evaluation: whether
or not they permit to publish automatic evaluation scores on the WAT2021 web page;

Table 18: The JPO adequacy criterion• Task: the task you submit the results for;

6.1 JPO Adequacy Evaluation• Used Other Resources: whether or not theyused additional resources; andWe conducted JPO adequacy evaluation for the top
two or three participants’ systems of pairwise evaluation for each subtask.69 The evaluation was carried out by translation experts based on the JPO
adequacy evaluation criterion, which is originallydefined by JPO to assess the quality of translated
patent documents.

• Method: the type of the method including SMT, RBMT, SMT and RBMT, EBMT,NMT and Other.
Evaluation scores of translation results that participants permit to be published are disclosed via the
WAT2021 evaluation web page. Participants canalso submit the results for human evaluation using
the same web interface.
This automatic evaluation system will remain
available even after WAT2021. Anybody can register an account for the system by the procedures
described in the application site.68
5.3

6.1.1 Sentence Selection and Evaluation
For the JPO adequacy evaluation, the 200 test sentences were randomly selected from the test sentences.
For each test sentence, input source sentence,translation by participants’ system, and referencetranslation were shown to the annotators. To guarantee the quality of the evaluation, each sentence
was evaluated by two annotators. Note that the
selected sentences are basically the same as thoseused in the previous workshop.

A Note on AMFM Scores

Up until WAT 2020, we used an older generation AMFM evaluation approach which did not use
deep neural networks. Given the advances in multilingual pre-trained models, this year, our collaborators provided us with deep AMFM models. With
the exception of ASPEC and restricted translation
tasks we used the provided deep AMFM models
to compute AMFM scores. Given that these deepmodels need GPUs to run quickly, we have notyet integrated it into our evaluation server as it is
not equipped with GPUs. Instead, we compute the
AMFM scores offline and add them to the evaluation scoreboard. For readers interested in AMFM
and recent advances we refer readers to the following literature: Zhang et al. (2021b,a); D’Haro et al.
(2019); Banchs et al. (2015b).

6

All important information is transmitted correctly.
(100%)
Almost all important information is transmitted correctly. (80%–)
More than half of important information is transmitted correctly. (50%–)
Some of important information is transmitted correctly. (20%–)
Almost all important information is NOT transmitted correctly. (–20%)

6.1.2 Evaluation Criterion
Table 18 shows the JPO adequacy criterion from
5 to 1. The evaluation is performed subjectively.
“Important information” represents the technicalfactors and their relationships. The degree of importance of each element is also considered to evaluate. The percentages in each grade are rough indications for the transmission degree of the source
sentence meanings. The detailed criterion is described in the JPO document (in Japanese).70

7 Evaluation Results
In this section, the evaluation results for WAT2021
are reported from several perspectives. Some of
the results for both automatic and human evaluations are also accessible at the WAT2021 web-

Human Evaluation

In WAT2021, we conducted JPO adequacy evaluation (other than En-Hi and En-Ml multi-modal
task, Section 6.1).

69

The number of systems varies depending on the subtasks.
http://www.jpo.go.jp/shiryou/toushin/chousa/tokkyohonyaku_hyouka.htm
7068

http://lotus.kuee.kyoto-u.ac.jp/WAT/
WAT2021/application/index.html

17

site.71

8.2 NICT-SAP Task

7.1

In contrast to 2020 where we had only 1 submission, this year we received submissions from 5
teams, 4 of which submitted system description
papers. The submitted models were trained using a variety of techniques such as domain adaptation, corpora selection and weighing, MBART pretraining and multilingual NMT training. All submissions significantly outperformed the organizers
baselines as well as the best submission in 2020.
The gains showed by this year’s submissions range
from approximately 14 to 30 BLEU (depending on
the task) compared to the baselines. The main reason was that this year’s submission rely on highquality data selection as well as on massively multilingual pre-trained models. Out of the 4 teams that
submitted system description papers, only one relied on data selection and surprisingly obtained thebest results for some language pairs. For other language pairs, this team obtained cometitive results.
Regardless, is is clear that models like MBART
are extremely useful in extremely low-resource domains such as Wikinews and software documentation.

Oﬀicial Evaluation Results

Figures 4 and 5 show those of JPC subtasks, Figures 6 and 7 show those of MMT subtasks, Figures
8, 9, 10, 11, 12, 13, 14, 15, 16, and 17 show thoseof Indic Multilingual subtasks and Figures 18 and19 show those of . Each figure contains the JPO adequacy evaluation result and evaluation summaryof top systems.
The detailed automatic evaluation results areshown in Appendix A. The detailed JPO adequacy
evaluation results for the selected submissions areshown in Table 19. The weights for the weighted
κ (Cohen, 1968) is defined as |Evaluation1 −Evaluation2|/4.

8

Findings

8.1

JPC Task

Three teams participated in JPC task. Bering
Lab and tpt_wat submitted results for all language
pairs and TMU submitted results for J↔K andJ↔E pairs. Similarly to WAT 2020, participants’systems were transformer-based or BART-based.
Bering Lab trained Transformer models with additional corpora, which were crawled patent document pairs aligned by a sentence encoding method
and contained more than 13M sentences for eachlanguage pair. Their system achieved the best
BLEU, RIBES, and AMFM scores for J→C/K/E
and the best BLEU and RIBES scores for K→Jamong the past and this year’s systems. tpt_wat
used Transformer and back-translation with a single setting for six language pairs. TMU used finetuned Japanese BART models and achieved thebest AMFM score for K→J. As for human adequacy evaluation, the evaluated system TMU did
not show superior performance to past years’ systems for J↔E, while the results cannot be directlycompared.
Among the top-performing systems, BeringLab’s systems obtained large BLEU improvements
around two points over the past years’ systems
for J↔K. The improvements were probably due
to their additional corpora, because their model
without the additional corpus ranked second for
J→K. Another finding by TMU was that pretrained
Japanese BART brought gains for all J↔K/E directions.

Regarding, human evaluation we did JPO adequacy evaluation for English to Indonesian and English to Malay for the Software Documentation domain. Kindly refer to Figure 18 and 19 for the results of human evaluation. For both translation directions, team “sakura” had the highest JPO as well
as BLEU scores but the scores for team “NICT-2”
were not that far behind. They were certainly significantly better than the organizer scores who only
developed models using parallel corpora without
any pre-training. We can certainly say that at highenough BLEU score levels (higher than 40), thelarge differences in BLEU do not necessarily correlate with large differences in human evaluation
scores. To be specific, the gap between “sakura”
and “NICT-2” in terms of BLEU for English to Indonesian is 2.14 and for English to Malay is 1.5
BLEU. However, the corresponding gaps in humanevaluation are 0.08 and 0.15 which is not significant. Human evaluation on a larger scale might beneeded but we were unable to do so due to budgetary limitations.
8.2.1 News Commentary (JaRuNC) task
Unfortunately we did not receive any submissions
this year.

71
http://lotus.kuee.kyoto-u.ac.jp/WAT/
evaluation/

18

Figure 4: Oﬀicial evaluation results of jpcn-ja-en.

Figure 5: Oﬀicial evaluation results of jpcn-en-ja.

19

Figure 6: Oﬀicial evaluation results of mmt-en-ja.

Figure 7: Oﬀicial evaluation results of mmt-ja-en.

20

Figure 8: Oﬀicial evaluation results of indic21-en-bn.

Figure 9: Oﬀicial evaluation results of indic21-bn-en.

Figure 10: Oﬀicial evaluation results of indic21-en-kn.

21

Figure 11: Oﬀicial evaluation results of indic21-kn-en.

Figure 12: Oﬀicial evaluation results of indic21-en-ml.

Figure 13: Oﬀicial evaluation results of indic21-ml-en.

22

Figure 14: Oﬀicial evaluation results of indic21-en-mr.

Figure 15: Oﬀicial evaluation results of indic21-mr-en.

Figure 16: Oﬀicial evaluation results of indic21-en-or.

23

Figure 17: Oﬀicial evaluation results of indic21-or-en.

Figure 18: Oﬀicial evaluation results of software-en-id.

Figure 19: Oﬀicial evaluation results of software-en-ms.

24

SYSTEM DATA Annotator A
Annotator B
allweightedID
ID average varianceaverage varianceaverage κκTMU 5187 4.34
0.52 4.74
0.30 4.54 0.09 0.20
TMU 5347 4.21
1.34 4.34
1.14 4.27 0.33 0.53
SRPOL 6232 4.57
0.71 4.74
0.36 4.65 0.30 0.36
indic21-en-bn sakura 6150 4.32
1.25 4.46
0.60 4.38 0.20 0.34
IIIT-H 6005 3.89
2.01 4.00
1.55 3.94 0.27 0.52
SRPOL 6242 4.79
0.31 4.80
0.18 4.80 0.14 0.18
indic21-bn-en IIIT-H 6015 3.67
2.38 3.96
1.49 3.81 0.31 0.54
mcairt 6332 3.33
1.82 3.85
1.26 3.59 0.19 0.34
SRPOL 6235 4.70
0.28 4.74
0.41 4.71 0.23 0.29
indic21-en-kn sakura 6153 4.73
0.20 4.41
0.64 4.57 0.15 0.22
IIIT-H 6008 4.11
0.63 3.90
1.35 4.00 0.33 0.48
SRPOL 6245 4.63
0.29 4.81
0.25 4.72 0.25 0.30
indic21-kn-en sakura 5873 4.62
0.38 4.36
1.23 4.49 0.21 0.32
IIIT-H 6018 4.17
0.77 3.70
2.23 3.94 0.21 0.40
SRPOL 6236 4.26
1.09 4.56
0.37 4.41 0.20 0.30
indic21-en-ml CFILT 6046 3.46
1.30 3.60
1.26 3.54 0.16 0.30
IIIT-H 6009 2.24
1.99 3.19
0.58 2.71 0.04 0.11
SRPOL 6246 3.27
0.90 4.78
0.43 4.03 0.05 0.05
indic21-ml-en sakura 5874 3.57
0.86 4.42
1.33 3.99 0.03 0.10
IITP-MT 6289 3.31
1.23 4.12
1.41 3.71 0.11 0.21
0.34 4.42
0.44 4.34 0.05 0.04
SRPOL 6237 4.26
indic21-en-mr CFILT 6047 4.08
0.44 4.20
0.65 4.14 0.01 0.01
IIIT-H 6010 3.63
0.71 4.05
0.93 3.84 0.09 0.18
SRPOL 6247 4.34
0.55 4.79
0.31 4.57 0.07 0.11
indic21-mr-en sakura 5875 4.14
0.70 4.56
0.53 4.35 0.12 0.18
IIIT-H 6021 3.86
1.26 4.15
0.99 4.00 0.05 0.15
SRPOL 6238 4.12
0.65 4.38
0.69 4.25 0.31 0.49
indic21-en-or IIIT-H 6011 3.80
0.77 3.83
1.08 3.82 0.63 0.75
CFILT 6048 3.75
0.90 3.77
0.97 3.76 0.77 0.85
SRPOL 6248 4.36
0.85 4.38
0.56 4.37 0.14 0.35
indic21-or-en sakura 5876 4.24
1.06 4.26
0.75 4.25 0.26 0.49
2.08 3.50
1.32 3.42 0.32 0.63
IIIT-H 6022 3.34
0.20 3.62
1.37 4.24 0.02 0.07
sakura 5799 4.86
software-en-id NICT-2 5902 4.74
0.45 3.58
1.56 4.16 0.07 0.15
1.66 2.73
2.04 3.45 0.13 0.25
organizer 3609 4.17
sakura 5818 3.44
0.76 4.66
0.38 4.05 0.01 0.08
software-en-ms NICT-2 5904 3.25
0.93 4.54
0.61 3.90 -0.03 0.09
organizer 3610 2.88
1.18 4.05
1.34 3.46 0.06 0.27
Subtaskjpcn-ja-en
jpcn-en-ja

Table 19: JPO adequacy evaluation results in detail.

8.3

Indic Multilingual Tasktion. Following “SRPOL” teams such as “sakura”,“CFILT”, “IIIT-H”, “IITP-MT” and “mcairt” performed the best with ranks varying depending on
the translation direction. One important observation we made was that “SRPOL” results for Indian to English translation were far higher thanthose of the other teams. In general their submission were 2 to 5 BLEU higher than the second
best team. We suppose that this is due to theirdetailed experimentation with data selection andback-translation. On the other hand, for English
to Indian language translation, although “SRPOL”
had the highest BLEU for most directions, the gap
between “SRPOL” and other participants was notthat high. In a number of cases the differences wereless than 0.5 BLEU which is not significant.

In WAT 2021, we received an overwhelming participation from 11 teams, 10 of which submitted
system description papers. In contrast, in WAT
2020 there were only 4 system description papers.
All participants trained multilingual NMT models. Some teams focused on leveraging monolingual corpora for pre-training MBART models orfor backtranslation whereas other teams focusedon script mapping to increase the similarity between the Indian languages and other teams focused on language family specific (Indo-Aryan vsDravidian) models. Compared to the previous
years, it is clear that backtranslation needs to besupplemented with pre-training as well as data selection for the best translation quality. The best
performing team, “SRPOL”, used back-translation,
pre-training, data selection and domain adapta-

In general, we observed that translation into
English had substantially high BLEU scores with25

most participants obtaining higher than 25 BLEU
for most directions. This makes sense because Indian languages are similar to each other and when
the target language is the same, the increase in
the target language data and transfer learning on
the source side will lead to a large improvementin translation quality. In most cases, the scoresfor Indo-Aryan (Hindi, Marathi, Oriya, Punjabi,
Gujarati and Bengali) to English translation weremuch higher than the scores for Dravidian (Tamil
Telugu, Kannada and Malayalam) to English translation.

and “CFILT” in terms of BLEU is 2.7 and in termsof JPO scores is 0.87. For the same teams in caseof English to Marathi, the gap in BLEU and JPO
scores are 1.95 and 0.2 respectively. The difference between a gap of 2.7 and 1.95 is not very large
as it is on a scale of 10072 but the difference between 0.87 and 0.2 on a scale of 573 is quite large.
In previous editions of this workshop we have always insisted that BLEU scores should not always
be trusted in order to decide if translations truly are
the best and this year’s human evaluation results
show that this is still the case. Multi-metric evaluation helps us better understand different aspectsof translation and we recommend readers to adopt
the same even if automatic metrics are used. Although we are limited by budgetary constraints wehope to conduct larger scale human evaluation in
the future.

On the other hand, for translation into Indian languages, BLEU scores were relatively lower. Thisis due to the morphological richness of Indian languages as well as the fact that multilingual English to Indian language translation does not benefit from the abundance of target language corpus like multilingual Indian language to English
translation does. The BLEU scores for translation
into Indo-Aryan languages such as Hindi and Punjabi showed the best translation quality exceeding
30 BLEU. This makes sense because Hindi and
Punjabi are very similar and Hindi is the most resource rich among all Indian languages. It is certain that Punjabi benefits from the Hindi parallel
data via transfer learning despite not sharing thesame script. Script sharing, a technique used bysome participants, could help enhance the amountof transfer learning taking place even further. For
other Indo-Aryan languages the translation quality
was a bit lower where English to Bengali exhibitedthe least translation quality compared to the otherIndo-Aryan languages. This shows that linguistic
similarity is not enough to lead to a high amount oftransfer. In the case of translation into Dravidian
languages we observed the lowest BLEU scores,
usually around 15 BLEU or lower, with the exception of English to Kannada. Despite having larger
corpora than some Indo-Aryan languages, translation into Dravidian languages is very hard as they
are significantly morphologically richer than IndoAryan languages. Simply leveraging large monolingual corpora may not be enough and methods
that take Dravidian linguistics into account may benecessary.

8.4 English→Hindi Multi-Modal Task
This year four teams participated in the different sub-tasks (TEXT, MM, and HI) of the
English→Hindi Multi-Modal task. The WAT2021
automatic evaluation scores for the participatingteams are shown in Tables 63, 60, 62, 58, 55, 57.
The team “Volta” obtained the highest BLEU score
for the text-only translation (TEXT) for both theevaluation (E-Test) and challenge (C-Test) test set.
The best performance is obtained by fine-tuned
mBART using IITB Corpus as an additional resource. For the captioning sub-task (HI) one team
“NLPHut” participated and able to obtained better results compared to previous years’ best results
based on region-specific image caption generation.
For the multimodal sub-task (MM), we received
three submissions from the teams “Volta”, “iitp”
and “CNLP-NITS-PP”, respectively. The team“Volta” obtained the highest BLEU score for themultimodal translation (MM) for both the evaluation (E-Test) and challenge (C-Test) test set. They
extracted object tags from images using visual information to enhance the textual input and achieve
the BLEU score of 51.60 on the challenge test set,also the translation output able to resolve ambiguity as compared with text-only translation.
Due to constraints, no human evaluation wasmade this year for the English→Hindi MultiModal Task.

With regards to human evaluation, we observedthat differences in BLEU scores do not always correspond to differences in human evaluation scores.
For example, take the case of English to Malayalam translation where the gap between “SRPOL”

72
73

26

BLEU scores go from 0 to 100.
Human evaluation scores go from 1 to 5.

8.5

English→Malayalam Multi-Modal Task

8.7 Ambiguous MS COCO
Japanese↔English Multimodal Task

This year one team “NLPHut” participated in the
different sub-tasks text-only translation (TEXT)
and Malayalam captioning (ML) sub-tasks of the
English→ Multi-Modal task. The WAT2021 automatic evaluation scores are shown in the Table 64,
61, 59, 56.

This year only one team participated in the
English→Japanese task. Their system was basedon a word-region alignment method to enhance theinteraction between source tokens and image regions and then integrating aligned information to
the visual features during decoding (Zhao et al.,2021). We observe that their system outperformed
the organizer’s system, which is based on double
attention to both source tokens and image regions.
It verified that it is important to integrate visual information in a proper way for this task and multimodal MT in general that text is a strong clue fortranslation, but visual information can further improve translation if it is used properly.
Unfortunately, there is no team participating theJapanese→English task. We hope that we can havemore participates next year for the tasks in both directions.

For English to Malayalam text-only translation
the team “NLPHut” using the Transformer modelobtained a BLEU score of 34.83 as compared tobaseline of 30.49 on the evaluation test set and forthe challenge test set obtained 12.15 compared tothe baseline 12.98. For Malayalam image captioning, the team “NLPHut” used the region-specificapproach by extracting image features for the given
specific region (bounding box) along with thewhole image features and concatenating both topass into an LSTM decoder to obtained the captions.
Due to constraints, no human evaluation wasmade this year for the English→Malayalam MultiModal Task.

8.6

8.8 Restricted Translation Task
We received 3 systems for the English→ Japanesetranslation task and 4 systems for the Japanese→
English.74 On the whole, all the submitted systems
are basically lexical-constraint-aware NMT models with lexically constrained decoding method,where the restricted target vocabulary is concatenated into source sentences and, during the beam
search at inference time, the models generate translation outputs containing the target vocabulary.
We observed that these techniques boost the final
translation performance of the NMT models in the
restricted translation task.
For human evaluation, we conducted the sourcebased direct assessment (Cettolo et al., 2017; Federmann, 2018) and source-based contrastive assessment (Sakaguchi and Van Durme, 2018; Federmann, 2018), to have the top-ranked systems ofeach team appraised by bilingual human annotators. In the human evaluation campaign, we also
include the human reference data. Table 20 reports
the final automatic evaluation score and the humanevaluation results. In both tasks, the systems fromthe team “NTT” are the most highly evaluated in
all the submitted systems in the final score and thehuman evaluation, consistently. We also note thatour designed automation metric is well correlated

Flickr30kEnt-JP Japanese↔English
Multi-Modal Tasks

This year, two teams participated in the
English→Japanese task, and one team participated in the Japanese→English task, respectively.
It is notable that all submissions outperformed thebest scores in WAT 2020, probably because of theincreased size of the training dataset as well as thenovel techniques introduced by the participants.
Overall, we observe the similar trend as in thelast year. In the English→Japanese task, MMT
systems constantly outperformed text-only NMT
models including unconstrained ones, while in
the Japanese→English task, unconstrained NMT
model achieved the best performance. This is perhaps because the Flickr30kEnt-JP dataset itself is
indeed constructed by English to Japanese humantranslation where images were actually referred toresolve ambiguity. One team developed an elegant
method for soft alignment of word–region to realize better grounding of multimodal information,
which is shown to achieve a favorable performancegain. This result again indicates the importance oftext–image grounding in MMT, and we believe that
we still have much room for improvements.

74

We discuss 3 submitted systems from the teams “NTT”,“NHK”, and “NICTRB” teams, as we do not have a systemdescription paper from the team “TMU”.

27

En-JaTeam
NTT
NHK
NICTRB(human ref.)
Ja-En
Team
NTT
NHK
NICTRB
TMU(human ref.)

final
57.2
33.9
28.8
—
final
44.1
37.5
31.8
22.6
—

Human Eval.
src-based DA src-based CA
77.5
79.7
74.1
77.2
73.6
77.1
73.4
76.4
Human Eval.
src-based DA src-based CA
75.6
74.4
73.9
73.5
72.1
71.8
50.2
48.3
74.1
72.9

• At Idiap Research Institute, the work was supported by the EU H2020 project “Real-time
network, text, and speaker analytics for combating organized crime” (ROXANNE), grant
agreement: 833635.
• At Charles University, the work was supported by the grant 19-26934X (NEUREM3)of the Czech Science Foundation and using
language resources distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (projects LM2015071 and OP VVV VI
CZ.02.1.01/0.0/0.0/16013/0001781).

Table 20: Human evaluation results of source-based direct assessment (src-based DA) and source-based contrastive assessment (src-based CA), ranging 0 to 100.
TThe column of “final” reports the final score of the automatic evaluation metric described in Section 2.13.

Referenceswith the human evaluation results. Besides that, wefound that the ASPEC human reference data might
have a quality issue, consisting of low-quality examples that are annotated with a score of [0, 50],
with the ratio of (En-Ja, Ja-En)=(13.30%, 12.43%).
This is why a few systems are shown to surpass theoriginal human reference data in the human evaluation.

9

Rafael E. Banchs, Luis F. D’Haro, and Haizhou Li.
2015a. Adequacy-fluency metrics: Evaluating mt in
the continuous space model framework. IEEE/ACM
Trans. Audio, Speech and Lang. Proc., 23(3):472–
482.
Rafael E. Banchs, Luis F. D’Haro, and Haizhou Li.
2015b. Adequacy–fluency metrics: Evaluating mt in
the continuous space model framework. IEEE/ACM
Transactions on Audio, Speech, and Language Processing, 23(3):472–482.

Conclusion and Future Perspective

Bianka Buschbeck and Miriam Exel. 2020. A parallel
evaluation data set of software documentation with
document structure annotation.

This paper summarizes the shared tasks of
WAT2021. We had 24 participants worldwide who
submitted their translation results for the humanevaluation, and collected a large number of useful submissions for improving the current machinetranslation systems by analyzing the submissions
and identifying the issues.
For the next WAT workshop, we will try to add
more Indic languages to our MultiIndicMT task
along with newer evaluation sets. Also, we will
add a new English→Bengali Multi-Modal task into
the Multimodal translation tasks.

M. Cettolo, Marcello Federico, L. Bentivogli, Niehues
Jan, Stüker Sebastian, Sudoh Katsuitho, Yoshino
Koichiro, and Federmann Christian. 2017. Overview
of the iwslt 2017 evaluation campaign.
Jacob Cohen. 1968. Weighted kappa: Nominal scaleagreement with provision for scaled disagreement or
partial credit. Psychological Bulletin, 70(4):213 –220.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan.
2020. A survey of multilingual neural machine translation. ACM Comput. Surv., 53(5).

Acknowledgement

Luis Fernando D’Haro, Rafael E. Banchs, Chiori Hori,and Haizhou Li. 2019. Automatic evaluation of endto-end dialog systems with adequacy-fluency metrics. Computer Speech and Language, 55:200–215.

The English→Hindi and English→Malayalam
Multi-Modal shared tasks were supported by thefollowing grants at Idiap Research Institute andCharles University. The authors do not see any significant ethical or privacy concerns that would prevent the processing of the data used in the study.
The datasets do contain personal data, and theseare processed in compliance with the GDPR andnational law.

Chenchen Ding, Hnin Thu Zar Aye, Win Pa Pa, Khin
Thandar Nwet, Khin Mar Soe, Masao Utiyama, andEiichiro Sumita. 2019. Towards Burmese (Myanmar) morphological analysis: Syllable-based tokenization and part-of-speech tagging. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(1):5.

28

Chenchen Ding, Masao Utiyama, and Eiichiro Sumita.
2018. NOVA: A feasible and flexible annotation system for joint tokenization and part-of-speech tagging.
ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(2):17.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan
Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation,and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 7871–7880, Online. Association
for Computational Linguistics.

Christian Federmann. 2018. Appraise evaluation framework for machine translation. In Proceedings ofthe 27th International Conference on Computational
Linguistics: System Demonstrations, pages 86–88,
Santa Fe, New Mexico. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey
Edunov, Marjan Ghazvininejad, Mike Lewis, andLuke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Aizhan Imankulova, Raj Dabre, Atsushi Fujita, andKenji Imamura. 2019. Exploiting out-of-domain parallel data through multilingual transfer learning for
low-resource neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 128–139, Dublin, Ireland. European Association for Machine Translation.

Andrew Merritt, Chenhui Chu, and Yuki Arase. 2020.
A corpus for english-japanese multimodal neural machine translation with comparable sentences.

Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito
Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs.
In Proceedings of the 2010 Conference on Empirical
Methods in Natural Language Processing, EMNLP’10, pages 944–952, Stroudsburg, PA, USA. Association for Computational Linguistics.

Hideki Nakayama, Akihiro Tamura, and Takashi Ninomiya. 2020. A visually-grounded parallel corpus
with phrase-to-region linking. In Proceedings ofthe 12th Language Resources and Evaluation Conference.
Toshiaki Nakazawa, Nobushige Doi, Shohei Higashiyama, Chenchen Ding, Raj Dabre, Hideya
Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Yusuke Oda, Shantipriya Parida, Ondřej Bojar,
and Sadao Kurohashi. 2019. Overview of the 6thworkshop on Asian translation. In Proceedings of
the 6th Workshop on Asian Translation, pages 1–35,
Hong Kong, China. Association for Computational
Linguistics.

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2017. Google’s
multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. Opennmt: Opensource toolkit for neural machine translation. In
Proceedings of ACL 2017, System Demonstrations,pages 67–72. Association for Computational Linguistics.

Toshiaki Nakazawa, Shohei Higashiyama, Chenchen
Ding, Hideya Mino, Isao Goto, Hideto Kazawa,
Yusuke Oda, Graham Neubig, and Sadao Kurohashi.
2017. Overview of the 4th workshop on asian translation. In Proceedings of the 4th Workshop on AsianTranslation (WAT2017), pages 1–54. Asian Federation of Natural Language Processing.

Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris
Callison-Burch, Marcello Federico, Nicola Bertoldi,
Brooke Cowan, Wade Shen, Christine Moran,
Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses:
Open source toolkit for statistical machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session.

Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding,
Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao
Goto, Win Pa Pa, Anoop Kunchukuttan, ShantipriyaParida, Ondřej Bojar, and Sadao Kurohashi. 2020.
Overview of the 7th workshop on Asian translation.
In Proceedings of the 7th Workshop on Asian Translation, pages 1–44, Suzhou, China. Association for
Computational Linguistics.

T. Kudo. 2005.
Mecab : Yet another partof-speechandmorphologicalanalyzer.
http://mecab.sourceforge.net/.

Toshiaki Nakazawa, Katsuhito Sudoh, Shohei Higashiyama, Chenchen Ding, Raj Dabre, Hideya
Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan,
and Sadao Kurohashi. 2018. Overview of the 5thworkshop on Asian translation. In Proceedings ofthe 32nd Pacific Asia Conference on Language, Information and Computation: 5th Workshop on AsianTranslation: 5th Workshop on Asian Translation,
Hong Kong. Association for Computational Linguistics.

Anoop Kunchukuttan. 2020. The IndicNLP Library.
https://github.com/anoopkunchukuttan/indic_
nlp_library/blob/master/docs/indicnlp.pdf.

Sadao Kurohashi, Toshihisa Nakamura, Yuji Matsumoto, and Makoto Nagao. 1994. Improvements ofJapanese morphological analyzer JUMAN. In Proceedings of The International Workshop on Sharable
Natural Language, pages 22–28.

29

Graham Neubig, Yosuke Nakata, and Shinsuke Mori.
2011. Pointwise prediction for robust, adaptablejapanese morphological analysis. In Proceedings ofthe 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 529–
533, Stroudsburg, PA, USA. Association for Computational Linguistics.

Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the
Association for Machine Translation in the Americas(Volume 1: Research Papers), pages 193–199. Association for Machine Translation in the Americas.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. CoRR, abs/1706.03762.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311–318.

Yi Mon Shwe Sin and Khin Mar Soe. 2018. Syllablebased myanmar-english neural machine translation.
In In Proc. of ICCA, pages 228–233.

Shantipriya Parida and Ondřej Bojar. 2021. Malayalam visual genome 1.0. LINDAT/CLARIAH-CZ
digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics
and Physics, Charles University.

Chen Zhang, Luis Fernando D’Haro, Rafael E. Banchs,
Thomas Friedrichs, and Haizhou Li. 2021a. Deep
AM-FM: Toolkit for Automatic Dialogue Evaluation,
pages 53–69. Springer Singapore, Singapore.

Shantipriya Parida, Ondřej Bojar, and Satya Ranjan
Dash. 2019a. Hindi visual genome: A dataset
for multimodal english-to-hindi machine translation.
arXiv preprint arXiv:1907.08948.

Chen Zhang, Grandee Lee, Luis Fernando D’Haro, andHaizhou Li. 2021b. D-score: Holistic dialogue evaluation without reference. IEEE/ACM Transactions
on Audio, Speech, and Language Processing, pages
1–1.

Shantipriya Parida, Ondřej Bojar, and Satya Ranjan
Dash. 2019b. Hindi Visual Genome: A Dataset forMultimodal English-to-Hindi Machine Translation.
Computación y Sistemas. In print. Presented at CICLing 2019, La Rochelle, France.

Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara,
and Chenhui Chu. 2020. Double attention-based
multimodal neural machine translation with semantic image regions. In EAMT, pages 105–114.

Hammam Riza, Michael Purwoadi, Teduh Uliniansyah,
Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai,
Vu Tat Thang, Nguyen Phuong Thai, Vichet Chea,
Sethserey Sam, Sopheap Seng, Khin Mar Soe, Khin
Thandar Nwet, Masao Utiyama, and Chenchen Ding.
2016. Introduction of the asian language treebank.
In In Proc. of O-COCOSDA, pages 1–6.

Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara,
and Chenhui Chu. 2021. Neural machine translation
with semantically relevant image regions. In NLP.

Appendix A

Submissions

Tables 21 to 76 summarize translation results submitted to WAT2021. Type and RSRC columns indicate type of method and use of other resources.

Keisuke Sakaguchi and Benjamin Van Durme. 2018.
Eﬀicient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of
the Association for Computational Linguistics (Volume 1: Long Papers), pages 208–218, Melbourne,
Australia. Association for Computational Linguistics.
Ye Kyaw Thu, Win Pa Pa, Masao Utiyama, Andrew
Finch, and Eiichiro Sumita. 2016. Introducing the
Asian language treebank (ALT). In Proceedingsof the Tenth International Conference on Language
Resources and Evaluation (LREC’16), pages 1574–
1578, Portorož, Slovenia. European Language Resources Association (ELRA).
Huihsin Tseng. 2005. A conditional random field word
segmenter. In In Fourth SIGHAN Workshop on Chinese Language Processing.
Masao Utiyama and Hitoshi Isahara. 2007. A japaneseenglish patent parallel corpus. In MT summit XI,
pages 475–482.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion

30

System
NICT-2sakura

ID
5916
5791

Type
NMT
NMT

RSRC
YES
NO

BLEU
34.970000
34.250000

RIBES
0.822350
0.820590

AMFM
0.839182
0.849202

Table 21: ALT20 en-hi submissions
System
NICT-2sakura

ID
5918
5798

Type
NMT
NMT

RSRC
YES
NO

BLEU
41.15
41.57

RIBES
0.901974
0.901977

AMFM
0.867678
0.868025

Table 22: ALT20 en-id submissions
System
NICT-2sakura

ID
5920
5816

Type
NMT
NMT

RSRC
YES
NO

BLEU
45.17
44.01

RIBES
0.912195
0.908439

AMFM
0.873476
0.871875

Table 23: ALT20 en-ms submissions
System
NICT-2sakura

ID
5922
5843

Type
NMT
NMT

RSRC
YES
NO

BLEU
55.690000
55.980000

RIBES
0.815863
0.818307

AMFM
0.832513
0.837062

Table 24: ALT20 en-th submissions
System
NICT-2sakura

ID
5917
5793

Type
NMT
NMT

RSRC
YES
NO

BLEU
35.21
36.17

RIBES
0.834649
0.835220

AMFM
0.814594
0.832895

Table 25: ALT20 hi-en submissions
System
NICT-2sakura

ID
5919
5800

Type
NMT
NMT

RSRC
YES
NO

BLEU
43.90
44.72

RIBES
0.898700
0.897314

AMFM
0.844199
0.850998

Table 26: ALT20 id-en submissions
System
NICT-2sakura

ID
5921
5821

Type
NMT
NMT

RSRC
YES
NO

BLEU
44.53
45.70

RIBES
0.904478
0.901696

AMFM
0.841632
0.851471

Table 27: ALT20 ms-en submissions
System
NICT-2sakura

ID
5923
5845

Type
NMT
NMT

RSRC
YES
NO

BLEU
28.96
30.10

RIBES
0.829525
0.832399

AMFM
0.817972
0.822585

Table 28: ALT20 th-en submissions
SystemYCC-MT1
YCC-MT1
YCC-MT2
YCC-MT2
sakura

ID
6195
6201
6175
6178
6031

Type
SMT
SMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO

BLEU
20.880000
20.130000
14.820000
14.020000
29.620000

RIBES
0.553319
0.545962
0.659582
0.639593
0.739320

Table 29: ALT2 en-my submissions

31

AMFM
0.655310
0.654820
0.663840
0.645470
0.752340

System
NECTECNECTEC
sakurasakura

ID
6188
6192
5230
5990

Type
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO

BLEU
6.24
4.62
19.75
18.70

RIBES
0.620840
0.587155
0.742698
0.736523

Table 30: ALT2 my-en submissions

32

AMFM
0.424640
0.391710
0.562680
0.550430

33

ID
5368
5616
5591
5502

SystemNTT
NTT
nictrb
NHK

NO
YES
YES
NO

BLEU
kytea
56.87
56.67
51.32
52.69
mecab
56.57
56.47
51.36
52.33

juman
0.882454
0.885308
0.836874
0.815612

RSRC
NO
NO
YES
YES
NO

BLEU
25.29
44.34
44.28
37.01
42.94

mecab
0.887104
0.889831
0.8441410.827300

AMFM
0.612290
0.672320
0.676670
0.651570
0.661560

RIBES
kytea
0.882322
0.885612
0.839934
0.823084

RIBES
0.653597
0.811700
0.813155
0.753823
0.801015

Table 32: ASPECRT ja-en submissions

Type
NMT
NMT
NMT
NMT
NMT

Table 31: ASPECRT en-ja submissions

juman
56.39
56.20
51.07
52.07

ID
5994
5209
5615
5592
5505

RSRC

SystemTMU
NTT
NTT
nictrb
NHK

NMT
NMT
NMT
NMT

Type
0.817290
0.818190
0.799950
0.801660

AMFM

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4789
5274
5349
4583
6026
6332
5870
6015
5556
5565
6280
6242
6268
6052
6162
6112
6124

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
11.27
21.37
23.89
13.88
25.22
29.96
26.69
28.28
11.33
11.83
25.77
31.87
31.82
25.98
24.39
18.48
20.18

RIBES
0.638781
0.747435
0.754772
0.669588
0.773387
0.798326
0.776808
0.773574
0.634088
0.629932
0.774004
0.800501
0.800145
0.760268
0.772190
0.721176
0.732342

AMFM
0.613093
0.744400
0.758921
0.6571190.778620
0.786717
0.772365
0.773292
0.673457
0.674034
0.777377
0.789735
0.792364
0.766461
0.778356
0.730379
0.734491

Table 33: HINDEN21 bn-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4788
5273
5348
4582
6000
6150
6005
5588
5938
6278
6232
6258
6041
6074

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
5.580000
10.590000
12.840000
8.130000
13.020000
13.830000
14.730000
3.230000
2.950000
11.040000
15.970000
15.580000
13.240000
11.090000

RIBES
0.573377
0.677858
0.704620
0.645895
0.715490
0.716347
0.724245
0.452631
0.465755
0.703372
0.733646
0.732792
0.710664
0.694142

AMFM
0.701527
0.755363
0.767497
0.735005
0.779592
0.764714
0.759513
0.628707
0.641712
0.731181
0.771033
0.772309
0.777074
0.763665

Table 34: HINDEN21 en-bn submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4790
5275
5350
4585
6003
6151
6006
5580
5927
6281
6233
6259
6042
6078

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
16.380000
23.040000
24.260000
17.760000
23.210000
25.270000
26.970000
6.810000
6.920000
20.460000
27.800000
27.310000
24.560000
20.420000

RIBES
0.748273
0.797371
0.806181
0.763222
0.809389
0.814798
0.820249
0.586360
0.599337
0.750935
0.824866
0.822329
0.806649
0.795314

Table 35: HINDEN21 en-gu submissions

34

AMFM
0.757069
0.801466
0.811717
0.768177
0.816739
0.813350
0.820127
0.628529
0.645669
0.808824
0.821221
0.819923
0.817681
0.809795

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4792
5277
5352
5987
6004
6152
6007
5578
5928
6283
6254
6260
6043
6079

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
23.310000
29.590000
30.180000
25.370000
35.850000
36.920000
38.250000
17.020000
15.860000
34.480000
38.650000
38.040000
35.390000
31.750000

RIBES
0.778841
0.817892
0.820984
0.788001
0.846656
0.848042
0.854192
0.681760
0.647511
0.844721
0.855879
0.852496
0.843969
0.829731

AMFM
0.759679
0.800234
0.801680
0.747598
0.822626
0.816999
0.822836
0.676601
0.681511
0.820543
0.824649
0.822371
0.821713
0.801179

Table 36: HINDEN21 en-hi submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4794
5279
5354
4591
5998
6153
6008
5581
5929
6285
6235
6261
6044
6113

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
10.110000
16.130000
18.220000
11.840000
14.580000
18.830000
19.570000
4.350000
3.900000
13.220000
21.300000
20.910000
17.980000
16.110000

RIBES
0.651048
0.732794
0.746230
0.689612
0.726259
0.7601000.756613
0.477922
0.469815
0.6352880.770110
0.771246
0.747233
0.736528

AMFM
0.741873
0.798654
0.813658
0.762931
0.805963
0.817831
0.812490
0.658271
0.657091
0.791821
0.821941
0.821329
0.816981
0.809687

Table 37: HINDEN21 en-kn submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4796
5281
5356
4590
6002
5886
6009
5582
5930
6287
6236
6262
6046
6081

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
3.340000
5.980000
6.510000
4.570000
6.170000
10.940000
12.760000
1.790000
1.480000
3.790000
15.490000
15.430000
12.790000
6.270000

RIBES
0.475441
0.605053
0.623301
0.554478
0.622598
0.686534
0.672331
0.338533
0.306966
0.437679
0.736915
0.734111
0.707437
0.619774

Table 38: HINDEN21 en-ml submissions

35

AMFM
0.706782
0.764924
0.789337
0.740136
0.793308
0.794481
0.745043
0.666547
0.656847
0.758960
0.807998
0.808089
0.805291
0.784292

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4798
5283
5358
4594
5999
6156
6010
5583
5931
6291
6237
6263
6047
6082

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
8.820000
14.690000
16.380000
10.410000
14.900000
17.870000
19.480000
5.100000
4.490000
13.950000
20.420000
19.930000
18.470000
14.480000

RIBES
0.6521340.720677
0.7391710.684554
0.740079
0.752439
0.760009
0.482727
0.467281
0.665934
0.771845
0.766897
0.759182
0.727647

AMFM
0.730656
0.785952
0.800357
0.745915
0.791850
0.803566
0.807758
0.654698
0.658104
0.798673
0.809721
0.810757
0.811499
0.799538

Table 39: HINDEN21 en-mr submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4800
5285
5360
4596
5996
6157
6011
5584
5932
6293
6238
6264
6048
6084

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
9.080000
15.010000
16.690000
12.810000
17.710000
17.880000
20.150000
2.200000
2.600000
12.570000
19.940000
19.150000
18.220000
15.660000

RIBES
0.638520
0.716665
0.734028
0.693696
0.743984
0.740263
0.750260
0.380253
0.431373
0.714731
0.751086
0.749740
0.738397
0.727477

AMFM
0.714530
0.748319
0.757804
0.736638
0.763064
0.769884
0.735718
0.591864
0.611704
0.737576
0.771831
0.771493
0.768399
0.758199

Table 40: HINDEN21 en-or submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4802
5287
5362
4598
6001
6158
6012
5585
5933
6298
6239
6265
6049
6085

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
21.770000
26.940000
29.150000
22.600000
30.560000
30.930000
33.350000
9.350000
10.020000
16.810000
33.430000
32.880000
31.160000
27.250000

RIBES
0.765216
0.808173
0.820085
0.785047
0.830405
0.829019
0.837603
0.633937
0.632319
0.785680
0.837542
0.835465
0.826367
0.816792

Table 41: HINDEN21 en-pa submissions

36

AMFM
0.762364
0.794023
0.803326
0.778215
0.8101060.802223
0.810972
0.620318
0.643473
0.663206
0.814115
0.813158
0.813658
0.803382

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4804
5289
5364
4616
5995
6159
6013
5586
5934
6303
6240
6266
6050
6086

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
6.380000
10.330000
11.420000
7.680000
11.980000
13.250000
14.430000
4.090000
3.600000
8.510000
14.150000
13.890000
12.990000
9.990000

RIBES
0.588286
0.675039
0.701210
0.630830
0.707054
0.721520
0.711995
0.452271
0.431281
0.578195
0.730705
0.728770
0.715699
0.682220

AMFM
0.723160
0.776138
0.792622
0.739011
0.801632
0.795712
0.778991
0.694376
0.684232
0.756693
0.798837
0.799382
0.802920
0.788022

Table 42: HINDEN21 en-ta submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal

ID
4806
5291
5366
5986
5997
6160
6014
5587
5935
6305
6241
6267
6051
6088

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
2.800000
4.590000
4.200000
4.880000
11.170000
15.480000
15.610000
2.310000
2.310000
6.250000
16.850000
16.820000
15.520000
12.860000

RIBES
0.479896
0.569735
0.576863
0.570112
0.702337
0.725543
0.728432
0.414016
0.389727
0.530898
0.739835
0.734483
0.725496
0.707817

AMFM
0.708086
0.754015
0.752068
0.713960
0.783647
0.785055
0.780218
0.634376
0.642502
0.764977
0.791085
0.792970
0.789820
0.778251

Table 43: HINDEN21 en-te submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4791
5276
5351
4633
6334
5871
6016
5557
5566
6282
6243
6269
6053
6163
6114
6125

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
26.21
33.65
33.53
23.10
36.77
38.73
39.39
16.79
17.50
36.49
43.98
42.87
35.31
34.60
28.79
31.02

RIBES
0.764569
0.810918
0.811609
0.755101
0.829389
0.834934
0.830158
0.715044
0.712002
0.827301
0.853263
0.849734
0.807849
0.824060
0.786408
0.795199

Table 44: HINDEN21 gu-en submissions

37

AMFM
0.726576
0.793874
0.796604
0.713984
0.819546
0.820654
0.806061
0.696879
0.698257
0.814556
0.835789
0.833146
0.797069
0.814168
0.765441
0.776935

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4793
5278
5353
5985
6333
5872
6017
5532
5567
6284
6244
6270
6054
6164
6115
6126

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
28.21
35.80
36.20
24.55
40.05
41.58
43.23
20.90
21.33
40.08
46.93
45.61
39.71
36.47
30.90
33.70

RIBES
0.782146
0.828390
0.832916
0.785027
0.850322
0.856469
0.853267
0.729188
0.759034
0.851601
0.872874
0.867712
0.837668
0.840014
0.807304
0.820716

AMFM
0.736131
0.8081800.805716
0.721805
0.8321190.834172
0.823007
0.714649
0.722822
0.831265
0.847064
0.843456
0.822034
0.824040
0.775032
0.791408

Table 45: HINDEN21 hi-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4795
5280
5355
4593
6374
5873
6018
5558
5568
6286
6245
6271
6055
6165
6121
6131

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
20.33
29.29
30.87
17.72
31.16
34.11
34.69
13.45
13.86
31.24
40.34
39.01
30.23
31.04
24.01
24.18

RIBES
0.717654
0.793521
0.7961190.710551
0.803525
0.815837
0.804694
0.683906
0.674282
0.806170
0.840458
0.837287
0.772913
0.811950
0.758489
0.759045

AMFM
0.692019
0.782087
0.792622
0.679617
0.799216
0.805112
0.790977
0.687726
0.687810
0.798540
0.823730
0.820355
0.778602
0.806951
0.751223
0.744802

Table 46: HINDEN21 kn-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4797
5282
5357
4634
6344
5874
6020
5559
5569
6289
6246
6272
6056
6166
6117
6130

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
13.64
26.55
28.23
15.47
28.07
32.23
29.19
12.99
13.64
29.37
38.38
37.04
29.28
28.55
22.10
22.84

RIBES
0.6731090.780019
0.786269
0.700957
0.792884
0.810429
0.780463
0.678961
0.6574400.802153
0.835444
0.830449
0.784424
0.803090
0.751437
0.763162

Table 47: HINDEN21 ml-en submissions

38

AMFM
0.646559
0.772691
0.786909
0.668778
0.794932
0.805450
0.748518
0.684370
0.684483
0.798550
0.823006
0.820716
0.789095
0.805091
0.744459
0.745908

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4799
5284
5359
5983
6335
5875
6021
5560
5570
6292
6247
6273
6057
6167
6118
6127

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
15.10
25.45
27.88
17.07
27.29
31.76
34.02
13.38
13.96
29.96
36.64
35.68
29.71
27.71
23.57
25.40

RIBES
0.676716
0.771352
0.783012
0.706399
0.785579
0.804834
0.803479
0.679550
0.669879
0.799383
0.824831
0.821164
0.786570
0.795729
0.752476
0.765200

AMFM
0.6581300.764852
0.779746
0.696839
0.780231
0.795844
0.792878
0.692897
0.693109
0.797333
0.812258
0.810290
0.789075
0.791157
0.751917
0.767347

Table 48: HINDEN21 mr-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4801
5286
5361
4597
6338
5876
6022
5550
5571
6294
6248
6274
6058
6107
6119
6128

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
16.35
25.81
27.93
18.92
29.96
32.67
34.11
13.71
13.69
31.19
37.06
36.04
30.46
19.61
25.05
26.34

RIBES
0.679781
0.762604
0.769634
0.720916
0.798326
0.801734
0.795132
0.634313
0.662493
0.794791
0.816956
0.812816
0.772850
0.737380
0.754313
0.761082

AMFM
0.730819
0.780431
0.782917
0.740606
0.795586
0.808239
0.804930
0.725121
0.721531
0.803226
0.817318
0.814871
0.793769
0.727657
0.770941
0.780009

Table 49: HINDEN21 or-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4803
5288
5363
4615
6342
5877
6023
5551
5572
6301
6249
6275
6059
6168
6123
6129

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
23.66
34.34
35.81
24.35
38.42
40.38
41.24
18.61
18.59
38.41
46.39
44.87
38.01
35.90
29.87
32.34

RIBES
0.749459
0.816975
0.827528
0.766047
0.840360
0.844351
0.837608
0.703876
0.730487
0.839598
0.865765
0.861389
0.818396
0.835327
0.795413
0.805722

Table 50: HINDEN21 pa-en submissions

39

AMFM
0.701483
0.792541
0.800753
0.717322
0.818332
0.823464
0.811169
0.693631
0.694658
0.815989
0.841641
0.836440
0.804561
0.814440
0.772655
0.782112

System
ORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4805
5290
5365
5984
6346
5878
6024
5563
5573
6304
6250
6276
6060
6169
6122
6132

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
16.07
24.72
26.90
15.40
28.04
31.09
29.61
13.36
13.77
27.76
36.13
35.06
29.34
26.69
21.37
22.75

RIBES
0.690144
0.766631
0.7801200.702428
0.793839
0.806993
0.785332
0.677433
0.660037
0.788181
0.822312
0.815951
0.784291
0.794380
0.747748
0.756364

AMFM
0.675969
0.758282
0.772249
0.669984
0.790184
0.7960740.750297
0.687892
0.688325
0.786587
0.806540
0.803595
0.785098
0.786098
0.742311
0.745090

Table 51: HINDEN21 ta-en submissions

SystemORGANIZER
NICT-5
NICT-5
NLPHutmcairtsakura
IIIT-HgaurvargaurvarIITP-MT
SRPOL
SRPOL
CFILT
coastal
CFILT-IITB
CFILT-IITB

ID
4807
5292
5367
4619
6348
5879
6025
5564
5574
6306
6251
6277
6061
6170
6120
6133

Type
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT
NMT

RSRC
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO

BLEU
14.70
27.76
28.77
16.48
29.26
33.87
30.44
12.14
12.44
28.13
39.80
38.57
30.10
30.50
22.37
24.02

RIBES
0.665774
0.777383
0.782427
0.695348
0.790319
0.810630
0.783709
0.652408
0.629617
0.784897
0.836433
0.831502
0.778981
0.806646
0.746368
0.757702

Table 52: HINDEN21 te-en submissions

40

AMFM
0.636031
0.7711090.779053
0.674821
0.786396
0.802030
0.754690
0.668328
0.666143
0.776964
0.820889
0.820360
0.783349
0.799696
0.743435
0.745885

41

ID
5347

System

TMU

NO

BLEU
kytea
47.12
mecab
45.27

juman
0.854558

RSRC
NO

BLEU
43.78

mecab
0.854298

AMFM
0.578009

RIBES
kytea
0.853335

RIBES
0.857054

Table 54: JPCN ja-en submissions

Type
NMT

Table 53: JPCN en-ja submissions

juman
45.24

ID
5187

RSRC

System
TMU

NMT

Type
0.876323

AMFM

System
NLPHut

ID
5231

Type
NMT

RSRC
NO

BLEU
1.690000

RIBES
0.095373

AMFM
0.385495

Table 55: MMCHHI21 en-hi submissions
SystemNLPHut

ID
5439

Type
OTHER

RSRC
NO

BLEU
0.990000

RIBES
0.024940

AMFM
0.383880

Table 56: MMCHHI21 en-ml submissions
SystemVolta
CNLP-NITS-PP
iitp

ID
6430
5730
5942

Type
NMT
NMT
NMT

RSRC
YES
YES
NO

BLEU
51.600000
39.280000
37.500000

RIBES
0.859645
0.792097
0.790809

AMFM
0.877000
0.817356
0.823429

Table 57: MMCHMM21 en-hi submissions
SystemVolta
NLPHut
CNLP-NITS-PP

ID
6429
4623
5732

Type
NMT
NMT
NMT

RSRC
YES
YES
YES

BLEU
51.660000
43.290000
37.160000

RIBES
0.855410
0.824521
0.770621

AMFM
0.876300
0.841544
0.797409

Table 58: MMCHTEXT21 en-hi submissions
SystemORGANIZERNLPHut

ID
6146
4621

Type
NMT
NMT

RSRC
NO
NO

BLEU
12.980000
12.150000

RIBES
0.378045
0.373986

AMFM
0.603143
0.649550

Table 59: MMCHTEXT21 en-ml submissions
SystemNLPHut

ID
5400

Type
OTHER

RSRC
NO

BLEU
1.300000

RIBES
0.093243

AMFM
0.333490

Table 60: MMEVHI21 en-hi submissions
SystemNLPHut

ID
5438

Type
OTHER

RSRC
NO

BLEU
0.970000

RIBES
0.047566

AMFM
0.405275

Table 61: MMEVHI21 en-ml submissions
SystemVolta
iitp
CNLP-NITS-PP

ID
6428
5941
5731

Type
NMT
NMT
NMT

RSRC
YES
NO
YES

BLEU
44.640000
42.470000
39.460000

RIBES
0.823319
0.807123
0.802055

AMFM
0.8391000.629444
0.641430

Table 62: MMEVMM21 en-hi submissions
SystemVolta
NLPHut
CNLP-NITS-PP

ID
6427
4622
5733

Type
NMT
NMT
NMT

RSRC
YES
YES
YES

BLEU
44.120000
42.110000
37.010000

RIBES
0.821469
0.813837
0.795302

AMFM
0.8381800.634481
0.642785

Table 63: MMEVTEXT21 en-hi submissions
SystemORGANIZERNLPHut

ID
6145
4620

Type
NMT
NMT

RSRC
NO
NO

BLEU
30.490000
34.830000

RIBES
0.580807
0.636404

Table 64: MMEVTEXT21 en-ml submissions

42

AMFM
0.726976
0.798859

43

ID
4730
5452
6313

SystemTMEKU
TMEKU
sakura

NO
NO
NO

ID
6349

RSRC

Systemsakura

NMT
NMT
NMT

Type

BLEU
kytea
53.50
51.81
51.17
mecab
48.57
47.02
46.32

juman
0.886046
0.874392
0.875110

RSRC
NO

BLEU
52.20

RIBES
0.909991

Table 66: MMT ja-en submissions

Type
NMT

Table 65: MMT en-ja submissions

juman
44.95
43.40
43.09

AMFM
0.577316

RIBES
kytea
0.890507
0.880350
0.879799
mecab
0.886316
0.874700
0.875825
0.644124
0.644113
0.644507

AMFM

44

ORGANIZERORGANIZER
TMEKU
TMEKU

System
NMT
NMT
NMT
NMT

Type
NO
NO
NO
NO

juman
22.65
25.73
28.79
28.23

BLEU
kytea
28.16
31.46
34.33
33.71
mecab
24.75
27.77
31.04
30.23

juman
0.781797
0.804437
0.809852
0.806312

Type
NMT
NMT

RSRC
NO
NO

BLEU
30.04
30.70

mecab
0.778157
0.801715
0.8102930.800428

AMFM
0.757189
0.755753

RIBES
kytea
0.784997
0.806973
0.813066
0.808009

RIBES
0.800134
0.798426

Table 68: MSCOCO ja-en submissions

ID
4404
4422

Table 67: MSCOCO en-ja submissions

RSRC

SystemORGANIZERORGANIZER

4403
4423
4731
5451

ID
0.790050
0.806910
0.821745
0.815016

AMFM

System
NICT-2sakura

ID
5900
5792

Type
NMT
NMT

RSRC
YES
NO

BLEU
29.050000
28.500000

RIBES
0.651775
0.663932

AMFM
0.821077
0.826771

Table 69: SOFTWARE en-hi submissions

System
NICT-2sakura

ID
5902
5799

Type
NMT
NMT

RSRC
YES
NO

BLEU
43.25
45.39

RIBES
0.767124
0.759304

AMFM
0.863589
0.863010

Table 70: SOFTWARE en-id submissions

System
NICT-2sakura

ID
5904
5818

Type
NMT
NMT

RSRC
YES
NO

BLEU
40.76
42.26

RIBES
0.823552
0.838933

AMFM
0.866766
0.873296

Table 71: SOFTWARE en-ms submissions

System
NICT-2sakura

ID
5906
5844

Type
NMT
NMT

RSRC
YES
NO

BLEU
50.910000
55.640000

RIBES
0.770522
0.813347

AMFM
0.809907
0.829860

Table 72: SOFTWARE en-th submissions

System
NICT-2sakura

ID
5901
5795

Type
NMT
NMT

RSRC
YES
NO

BLEU
35.32
40.17

RIBES
0.712675
0.726708

AMFM
0.843388
0.861348

Table 73: SOFTWARE hi-en submissions

System
NICT-2sakura

ID
5903
5810

Type
NMT
NMT

RSRC
YES
NO

BLEU
40.69
44.70

RIBES
0.745225
0.759751

AMFM
0.852173
0.862999

Table 74: SOFTWARE id-en submissions

System
NICT-2sakura

ID
5905
5823

Type
NMT
NMT

RSRC
YES
NO

BLEU
38.42
40.97

RIBES
0.818175
0.819980

AMFM
0.843418
0.849354

Table 75: SOFTWARE ms-en submissions

System
NICT-2sakura

ID
5907
5846

Type
NMT
NMT

RSRC
YES
NO

BLEU
21.89
26.30

RIBES
0.673464
0.694253

Table 76: SOFTWARE th-en submissions

45

AMFM
0.787909
0.809105