FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for
Low-Resource NMT
Abhisek Chakrabarty∗, Raj Dabre∗, Chenchen Ding,
Hideki Tanaka, Masao Utiyama and Eiichiro Sumita
National Institute of Information and Communications Technology
3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan{abhisek.chakra,raj.dabre,chenchen.ding,hideki.tanaka,mutiyama,eiichiro.sumita}@nict.go.jp
Abstract

to a significant improvement in translation quality (Sennrich and Haddow, 2016; Hoang et al.,2016; Li et al., 2018; Pan et al., 2020; Chakrabarty
et al., 2020) in both low- and high-resource settings. Most recently, in an extremely low-resource
setting, Chakrabarty et al. (2020) show the effectiveness of using the aforementioned linguistic features, especially when their influence on the model
is controlled via relevance mechanisms that appropriately scale feature embeddings before theyaugment word embeddings. We hypothesize thatincorporating linguistic features, into the denoisingbased pre-training framework, should improve thequality of pre-training, which should then have apositive impact on the translation quality via finetuning. To this end, we propose FeatureBART, afeature based sequence-to-sequence pre-training
method.

In this paper we present FeatureBART, alinguistically motivated sequence-to-sequencemonolingual pre-training strategy in which syntactic features such as lemma, part-of-speechand dependency labels are incorporated into
the span prediction based pre-training framework (BART). These automatically extracted
features are incorporated via approaches suchas concatenation and relevance mechanisms,
among which the latter is known to be better
than the former. When used for low-resource
NMT as a downstream task, we show that thesefeature based models give large improvements
in bilingual settings and modest ones in multilingual settings over their counterparts that do
not use features.

1

Introduction

Sequence-to-sequence (S2S) pre-training done via
denoising objectives on monolingual corpora is
known to improve generation quality in lowresource settings (Lewis et al., 2020). This has been
extensively explored for neural machine translation,
however most works show that the improvements
for translation into English are more pronounced
than those for translation into a non-English language (Liu et al., 2020; Tang et al., 2020). One
reason for this is that pre-training leads to a strong
decoder that learns to de-noise masked inputs,
whereas the knowledge retained in the encoder israther limited. Thus far, there has been no explicit
effort towards improving the contribution of theencoder during pre-training.
Most pre-training methods rely on the powerof large models and large corpora, but ignore thepossibility of incorporating linguistic knowledge
into the pre-trained model. On the other hand,
there are several works which show that incorporating linguistic knowledge in the form of lemma,
part-of-speech tags and dependency labels, lead∗

Equal contribution.

In FeatureBART, linguistic features are obtained
via automatic annotators and then converted into
embeddings, which are used to augment the wordembeddings of the encoder. Feature embeddings,
are incorporated by either naive concatenation withthe word embeddings or by first weighing themwith a relevance mechanism and then adding them
to word embeddings. The model itself is trainedusing a monolingual corpus via either the textinfilling or the mask prediction approaches, theformer used in BART and the latter used in BERT.
This FeatureBART model is then fine-tuned for
low-resource language pairs, where linguistic features are also used during fine-tuning. Experiments
on English to 8 Asian languages from the Asian
Language Treebank (ALT) dataset (Riza et al.,2016), using bilingual as well as multilingual finetuning, show that our feature based pre-training
and fine-tuning leads to significant improvements
in translation quality indicating the complementary
nature of denoising pre-training and features. Analyses of training curves show that when compared to
non-feature based pre-training, feature based pre5014

Proceedings of the 29th International Conference on Computational Linguistics, pages 5014–5020
October 12–17, 2022.

training leads to significantly lower perplexities
during the initial stages of fine-tuning.

2

Methodology

We first give some background knowledge about
feature based NMT modeling, followed by an explanation of FeatureBART.
2.1

Background: Use of Source Side
Morphological Features into NMT

Sennrich and Haddow (2016) proposed the concatenation of embeddings of features of a token(word or sub-word) to the token embedding. In
case a word is split into sub-words, the featureis duplicated for each sub-word. For K featuresof a source token denoted by si = (si1 , . . . , siK ),let Vk , Ek , and dk denote the vocabulary, embedding matrix and dimension of the k th feature. So,
Ek ∈ Rdk ×|Vk | , si1 is the word or sub-word featureand si2 , . . . , siK are the linguistic features. The embedding of si , say ei , is formulated as eik = Ek sik ,and ei = ∥K
k=1 eik . eik is the vector embedding of
sik where ∥ is the concatenation operation.
2.2

Relevance of Features

Chakrabarty et al. (2020) proved the effectivenessof the following two feature weighting strategies tobe applied to features prior to concatenation:
Self-Relevance: The relevance of a feature embedding is evaluated w.r.t itself. For k ∈ {1, . . . , K},
the self relevance is calculated as maskik =sigmoid(Wk eik ), and then e′ik = maskik ⊙ eik .
Wk ∈ Rdk ×dk is the learnable weight matrix forthe k th feature, and ⊙ is the element-wise multiplication operation. The vector maskik signifies theself relevance of eik and is multiplied element-wisewith eik to produce the modified feature embedding e′ik . Finally, e′i1 , . . . , e′iK are concatenated tomake the final embedding e′i for the source token′
si . Thus, e′i = ∥K
k=1 eik .
Word-Relevance: It uses the word/sub-word embedding (ei1 ) to determine the relevance of the remaining feature embeddings. Formally, for k ∈{2, 3, . . . , K}, maskik = sigmoid(Wk (ei1 ∥eik )),and then e′ik = maskik ⊙ eik . Wk ∈ Rdk ×(d1 +dk )is the learnable weight matrix and the final embedding e′i is obtained by concatenating e′i2 , . . . , e′iKwith ei1 .
2.3

FeatureBART

which are used to augment the word embeddings ofthe encoder. First, a large monolingual corpus is annotated via morphological and syntactic annotators
to obtain different types of features for each token(word or sub-word). This feature annotated monolingual corpus is used for self-supervised training
where during training, some tokens (token masking) or token spans (text infilling) in a sentence are
replaced with the “MASK” token just like in Lewis
et al. (2020). We use dummy mask features formasked tokens or spans. The sentence containing
masked content is fed to the encoder and the modelis trained so that the decoder can predict the original sentence. We hypothesize that features help in
better pre-training as they provide the model withadditional information for denoising.

3

Experiments

We describe the datasets for pre-training and finetuning, features used and model training details.
Datasets: For pre-training of our BART and FeatureBART models, we choose the English monolingual News Crawl articles1 of 2007, 2010, and2013 from WMT-2017 with varying sizes that contain 3.8, 6.8, and 21.7 million sentences respectively. For fine-tuning, we experiment with the multilingual, multi-parallel Asian language treebank(ALT) (Riza et al., 2016)2 for English to Asian language translation. Following (Chakrabarty et al.,2020), eight Asian languages - Bengali (bg), Filipino (fi), Hindi (hi), Indonesian (id), Khmer (khm),
Malay (ms), Myanmar (my) and Vietnamese (vi)
are set as the targets. So, bilingual experiments
cover eight language pairs from en-bg to en-vi. For
our multilingual experiments, we explore one-tomany multilingual translation setup keeping thesource side fixed to English (en) and the target
side to the eight Asian languages mentioned above.
The source side is fixed to English throughout because as the initial attempt on feature-based pretraining, we can rely on high quality automatic morphological analyzers available for English. Given
the potential of this work established by empiricalresults, other languages with moderate quality morphological annotation can also be tried in the future.
We use the official train/dev/test splits containing
18088, 1000, and 1018 sentences, respectively.
1

https://statmt.org/wmt17/
translation-task.html
2
http://www2.nict.go.jp/astrec-att/
member/mutiyama/ALT/

FeatureBART, is a feature based encoder-decoder
pre-trained model trained using linguistic features
5015

Pre-training
None

Noise
-

NC-07

I
I

NC-10

I
I

NC-13

I
I

Pre-training
None

Noise
-

NC-07

M
M

NC-10

I
I

NC-13

I
I

Config
Base+Self-Rel
Base+Self-Rel
Base+Word-Rel
Base+Concat

en-bg
7.5
8.4†
9.77
10
9.78
9.7
10.09
9.75

Config
Base+Self-Rel
Base+Self-Rel
Base+Self-Rel
Base+Self-Rel

en-bg
11.55
11.40
11.78
11.51
11.84
11.54
11.98
11.65

Bilingual Results
en-fi
en-hi
en-id
26.98
23.62
30.88
28.22† 26.13† 32.65†
31.28
25.63
33.86
31.53
26.93† 34.47†
32.07
26.21
35.25
32.57† 26.44
36.07†
32.62
26.3
35.56
33.12† 26.96† 35.32
Multilingual Results
en-fi
en-hi
en-id
31.04
27.29
34.78
31.14
27.94† 34.42
31.90
27.03
35.77
32.17
27.68† 36.11
32.66
26.77
35.71
32.57
27.76† 36.25†
32.67
26.69
36.11
32.38
27.77† 36.39

en-khm26.24
27.33†
29.0
29.02
27.5
29.4†
29.5
30.21†

en-ms
35.78
37.22†
38.59
39.01
38.79
40.34†
40.09
40.64†

en-my
16.48
18.13†
18.83
19.21
19.5
18.98
19.72
20.01

en-vi
29.05
29.91†
32.76
33.5†
32.32
34.78†
33.95
35.26†

Avg.
24.57
26
27.46
27.96
27.68
28.53
28.48
28.91

en-khm
30.27
30.09
30.47
30.94†
30.72
31.16†
30.41
30.98†

en-ms
39.37
39.84†
40.39
40.80
40.11
40.48
40.20
41.35†

en-my
20.93
20.99
20.65
21.13†
20.65
21.26†
20.60
20.95†

en-vi
34.58
33.85
34.94
35.28
34.56
35.44†
35.24
35.75†

Avg.
28.73
28.71
29.12
29.45
29.13
29.58
29.24
29.65

Table 1: BLEU scores of the bilingual and multilingual models. For a given pre-training corpus size, we only show
the results of the best feature and pre-training configuration due to lack of space. Highest scores are BOLD. † marks
scores significantly better (p < 0.05) than the corresponding non-feature (base) counterparts. The “Noise” column
indicates the pre-training approach, “I” and “M” for text-infilling and token masking. Self-Rel, Word-Rel, andConcat denote self-relevance, word-relevance, and concatenation of feature embedding configurations respectively.

Pre-Processing: The monolingual corpora for pretraining and the source side of fine-tuning corpora are tokenized and true-cased by Moses tokenizer (Papineni et al., 2002) and the target languages are tokenized to separate the delimiters andthe punctuation symbols. Following Johnson et al.
(2017), each source language sentence in multilingual translation setup is appended with a token
like <tgt-id> which indicates the target language.
Byte-pair-encoding (BPE) (Sennrich et al., 2016)is performed to obtain subword vocabularies. We
train a single BPE model of vocabulary size 32K
on the combined training corpora of all 9 languages
and use it during pre-training as well as fine-tuning.

dropouts, number of training epochs etc. (See A
for details). All training is done on a single 32
GB V-100 GPU. Pre-training is done for 3 epochs
for each monolingual corpora. During fine-tuning,validation is done after every 10000 steps and training stops if validation accuracy does not improvefor consecutive 5 evaluations. Test set decoding isdone using beam search with a beam size of 5 andlength penalty of 1.0. Translation performance ismeasured by BLEU score calculated using multibleu.perl.

4

Results

Table 1 contains our results for bilingual and multilingual fine-tuning of FeatureBART along with
their non-feature counterparts. Bilingual and multilingual results are divided into 4 groups: no pretraining, pre-training using News Crawl articles
of 2007 (NC-07), 2010 (NC-10), and 2013 (NC13) from WMT-2017. We analyze the results asfollows:
Bilingual vs. Multilingual Translation withHyperparameters and Training Details: We use
out and with Pre-Training: Comparing correvariations of the Transformer-base model (Vaswani
sponding rows between the bilingual and mulet al., 2017) for our experiments available fromtilingual blocks, shows that multilingual modOpenNMT PyTorch (Klein et al., 2017), whichels are significantly better than bilingual ones,
we modify for feature based experiments. Wher- observations which are in accordance with Ariever possible, we perform hyperparameter tuning
vazhagan et al. (2019); Johnson et al. (2017);
of layers, hidden sizes, number of attention heads, Dabre et al. (2020); Zhang et al. (2020). Without
5016
Features Used: Morphological annotation of the
English datasets is done using Stanford CoreNLP
toolkit (Manning et al., 2014). There are three
word-level linguistic features - lemma, part-ofspeech (POS), and dependency labels. All subwords of a word take the features of that word. We
use subword tags (Sennrich et al., 2016) to denote
beginning, middle and ending of a subword unit.

Figure 1: Perplexity plots for English–Bengali models.

Figure 2: Perplexity plots for multilingual models.

pre-training setup, adding features deteriorates thescores, whereas feature based pre-training gives
consistent improvements, indicating that feature
based modeling and pre-training are complementary.
Optimal Pre-Training and Feature Configurations: From Table 1, looking at the “Noise” column, it is clear that text infilling is the best pretraining objective in most cases. With regard to feature incorporation mechanism, self-relevance predominantly gives the best results. We therefore recommend the use of self-relevance and text-infillingbased FeatureBART in low-resource settings.
Studying Model Perplexities: Figure 1 shows the
perplexities3 of en-bg bilingual models. The perplexities with pre-training and features are lower
compared to when features are not used, duringinitial stages of training, but this changes towards
convergence. This explains why en-bg does notshow performance improvements from features.
Figure 2 contains plots of cumulative perplexityof all language pairs for multilingual models. Here,features seem to have larger impact than in bilingual settings, both at the beginning and later stagesof training. However, BLEU gains are not always
observed, indicating that it may not always be reliable motivating future multi-metric and humanevaluation (Marie et al., 2021).

5

Related Work

pre-training, improvements in maximum BLEU
Pre-Training: Pre-trained models reduce the needpoints comparing bilingual vs. multilingual setups
are (3.15, 2.92, 1.81, 2.13, 2.94, 2.62, 2.86, 4.67) for large fine-tuning data for a given downstreamfor (en → bg, fil, hi, id, khm, ms, my, vi) re- task, and in this paper we focus on extremely lowresource settings. In this context, T5 (Raffel et al.,spectively. Pre-training, although it improves2020), mT5 (Xue et al., 2021), BART (Lewis et al.,performance in both cases, naturally reduces thegap between bilingual and multilingual models. 2020), mBART-25 (Liu et al., 2020), mBART-50(Tang et al., 2020) and, most recently, IndicBARTNevertheless, we get maximum improvements(Dabre et al., 2021) are most commonly used forof (1.89, 0.98, 0.95, 0.71, 1.25, 0.49) BLEU points
fine-tuning. None of these works focus on linguisfor (en → bg, hi, khm, ms, my, vi) when comparing
tic features, which is the key focus of our paper.
bilingual and multilingual models.
Impact of Features: From Table 1 the follow- Feature Based NMT: Most works focusing on
linguistic features, experiment on low-resource
ing observations can be made: (1) In case of no
settings, and a majority of them focus on how
pre-training, features are very useful in bilingual
settings but not in multilingual settings as multi- to exploit syntactic/dependency structures of thesource language (Eriguchi et al., 2016; Shi et al.,lingual systems can utilize a multi-parallel corpora
2016; Chen et al., 2017; Li et al., 2017; Wu et al.,efficiently by acquiring supplementary knowledge
from other languages, thus making linguistic infor- 2018; Zhang et al., 2019; Bugliarello and Okazaki,2020). These works rely on various sophisticatedmation redundant. (2) However, when pre-training
is used, multilingual models tend to benefit more
3
Note that, adding features tends to make the training curves
from features. As an example, consider en-khm
smoother, especially in bilingual settings which are comparand en-vi multilingual scores in Table 1. Under noatively lower resource than multilingual settings.
5017

approaches but, Sennrich and Haddow (2016) showthat enriching encoder word embeddings with morphological features is a simple but nice techniqueto exploit the features. Chakrabarty et al. (2020)
improve upon this further by relevance mechanisms
on top of morphological and syntactic feature embeddings, to enable effective use of features, aninsight we adopt in this paper for feature based
pre-training.
Multilingual NMT: Where feature based NMT
focuses on utilizing linguistic information, multilingual NMT (Johnson et al., 2017; Dabre et al.,2020) focuses on leveraging training data for otherlanguages to improve translation quality. Incorporating linguistic features into multilingual models
has been neglected to the best of our knowledge,
and our work aims to fill in this gap.

6

Conclusion

We have presented FeatureBART, an encoderdecoder pre-trained model that augments the encoder’s embeddings with linguistic feature embeddings. Our experiments on English to Asian language translation in an extremely low-resource setting show that FeatureBART leads to better translation quality compares to its counterpart that does
not use features. Analyses of training curves revealthat compared to pre-training without features, feature based pre-training leads to significantly lower
perplexities during the initial stages of fine-tuning,which we think is responsible for improvement intranslation quality. Future work will focus on: (1)exploring the impact of individual feature categoryon feature based pre-training and (2) multilingualversion of FeatureBART which uses features in theencoder for languages other than English.

Abhisek Chakrabarty, Raj Dabre, Chenchen Ding,
Masao Utiyama, and Eiichiro Sumita. 2020. Improving low-resource NMT through relevance based
linguistic features incorporation. In Proceedings ofthe 28th International Conference on Computational
Linguistics, pages 4263–4274, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Huadong Chen, Shujian Huang, David Chiang, and Jiajun Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. CoRR,
abs/1707.05436.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan.
2020. A survey of multilingual neural machine translation. ACM Comput. Surv., 53(5).
Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan,
Ratish Puduppully, Mitesh M. Khapra, and Pratyush
Kumar. 2021. Indicbart: A pre-trained model for natural language generation of indic languages. CoRR,
abs/2109.02903.
Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa
Tsuruoka. 2016. Tree-to-sequence attentional neuralmachine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 823–833,
Berlin, Germany. Association for Computational Linguistics.
Cong Duy Vu Hoang, Reza Haffari, and Trevor Cohn.
2016. Improving neural translation models with linguistic factors. In Proceedings of the Australasian
Language Technology Association Workshop 2016,
pages 7–14, Melbourne, Australia.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim
Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,
Fernanda Viégas, Martin Wattenberg, Greg Corrado,
Macduff Hughes, and Jeffrey Dean. 2017. Google’s
multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. OpenNMT:Open-source toolkit for neural machine translation.
In Proc. ACL.

References
Naveen Arivazhagan, Ankur Bapna, Orhan Firat,
Dmitry Lepikhin, Melvin Johnson, Maxim Krikun,
Mia Xu Chen, Yuan Cao, George Foster, Colin
Cherry, Wolfgang Macherey, Zhifeng Chen, andYonghui Wu. 2019. Massively multilingual neuralmachine translation in the wild: Findings and challenges.
Emanuele Bugliarello and Naoaki Okazaki. 2020. Enhancing machine translation with dependency-aware
self-attention. In Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, pages 1618–1627, Online. Association for
Computational Linguistics.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan
Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Veselin Stoyanov, and Luke Zettlemoyer. 2020.
BART: Denoising sequence-to-sequence pre-training
for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pages 7871–7880, Online. Association for Computational Linguistics.
Junhui Li, Deyi Xiong, Zhaopeng Tu, Muhua Zhu,
Min Zhang, and Guodong Zhou. 2017. Modelingsource syntax for neural machine translation. CoRR,
abs/1705.01020.

5018

Qiang Li, Derek F. Wong, Lidia S. Chao, Muhua Zhu,
Tong Xiao, Jingbo Zhu, and Min Zhang. 2018. Linguistic knowledge-aware neural machine translation.
IEEE/ACM Trans. Audio, Speech and Lang. Proc.,26(12):2341–2354.

subword units. In Proceedings of the 54th Annual
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725,
Berlin, Germany. Association for Computational Linguistics.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey
Edunov, Marjan Ghazvininejad, Mike Lewis, andLuke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does
string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–
1534, Austin, Texas. Association for Computational
Linguistics.

Christopher D. Manning, Mihai Surdeanu, John Bauer,
Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations,pages 55–60.

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. CoRR,
abs/2008.00401.

Benjamin Marie, Atsushi Fujita, and Raphael Rubino.
2021. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing(Volume 1: Long Papers), pages 7297–7306, Online.
Association for Computational Linguistics.
Yirong Pan, Xiao Li, Yating Yang, and Rui Dong. 2020.
Dual-source transformer model for neural machinetranslation with linguistic knowledge.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia,
Pennsylvania, USA. Association for Computational
Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. 2020. Exploring thelimits of transfer learning with a unified text-to-texttransformer. Journal of Machine Learning Research,
21(140):1–67.
Hammam Riza, Michael Purwoadi, Gunarso, Teduh
Uliniansyah, Aw Ai Ti, Sharifah Mahani Aljunied,Luong Chi Mai, Vu Tat Thang, Nguyen Phuong
Thai, Rapid Sun, Vichet Chea, Khin Mar Soe,
Khin Thandar Nwet, Masao Utiyama, and Chenchen
Ding. 2016. Introduction of the Asian language treebank. In Proc. of O-COCOSDA, pages 1–6.
Rico Sennrich and Barry Haddow. 2016. Linguistic
input features improve neural machine translation.
In Proceedings of the First Conference on MachineTranslation: Volume 1, Research Papers, pages 83–
91, Berlin, Germany. Association for Computational
Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
S. Wu, D. Zhang, Z. Zhang, N. Yang, M. Li, and
M. Zhou. 2018. Dependency-to-dependency neuralmachine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(11):2132–
2141.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale,
Rami Al-Rfou, Aditya Siddhant, Aditya Barua, andColin Raffel. 2021. mT5: A massively multilingual
pre-trained text-to-text transformer. In Proceedingsof the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–
1639, Online. Association for Computational Linguistics.
Meishan Zhang, Zhenghua Li, Guohong Fu, and Min
Zhang. 2019. Syntax-enhanced neural machine translation with syntax-aware word representations. In
Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long and Short Papers), pages 1151–1161,
Minneapolis, Minnesota. Association for Computational Linguistics.

A

Hyperparameters and Training Details

We perform extensive hyperparameter tuning todetermine optimial settings for pre-training and5019

Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural machine translation of rare words with

fine-tuning. We use 6 encoder and decoder layers, 8 multi-attention heads, 2, 048 dimension sizeof fully-connected network. For models without
features, the token embedding and model hidden
dimension is set to 512. For feature based models,following Sennrich et al. (2016) the embedding
and hidden dimensions are 536 (250, 250, 15, 15, 6
corresponding to subword, lemma, POS, dependency label, and subword-tag), in order to makethe number of parameters comparable. We use
batch-sizes of 4, 096 tokens. The dropout rate is
set to 0.3 for fine tuning experiments.
We pre-train models using token masking as wellas text infilling, where span lengths are drawn froma Poisson distribution (λ = 3). We investigate theeffect of different noise percentage from 10% to
80% and find that 50% − 60% noising is the optimum to get the best performance in downstream
translation task. Pre-training is done for 3 epochs
for each monolingual corpora of 2007, 2010, and2013, covering 43200, 75600, and 237600 training
steps respectively. During fine-tuning, maximum
training steps are set as 200000 with validation
accuracy performed after every 10000 steps.

5020