言語処理学会 第29回年次大会 発表論文集 (2023年3月)

Preordering based Low Resource Translation Using Pretrained
Multilingual Model
Jingyi Zhu1 Takuya Tamura1 Fuzhu Zhu1 Xiaotian Wang1 Taiki Sakai1 Takehito Utsuro1 Masaaki Nagata2
1 Deg. Prog. Sys.&Inf. Eng., Grad. Sch. Sci.&Tech., University of Tsukuba
2 NTT Communication Science Laboratories, NTT Corporation, Japan

Abstract
In this paper, we propose to apply the pretrained seq2seq
model to conduct preordering process and translation. We
use the manual word alignment data to make preorder training data, and we compared the performance of diﬀerent
kinds of mT5 and mBART in preordering. For the translation procedure, we choose mBART as our baseline model.
We evaluated our approach on the Asian Language Treebank dataset (total of 20,000 parallel data), with directionsin Japanese and English, and in-house parallel data (total of 3,000 parallel data) of Japanese and Chinese fromNTT corporation. For the results, our proposed approachexceeds the baseline on the translation direction of Zh-Ja
pairs, and is close to level with the baseline on Ja-En pairs.

1

Introduction

In recent years, more and more researches have been conducted on sequence-to-sequence (seq2seq) models basedon pretraining [14, 6, 5]. Since the introduction of Transformer [13], the quality of translation has been greatly improved. However, in the task of low resource translation,
due to the dataset’s size limitation, this kind of parameter
randomization model often performs poorly [16]. In order to meet this challenge, many researches have proposedlarge-scale pretraining models, which have been widelyused in several tasks in NLP [1, 12].
In this paper, We propose applying the pretrained
seq2seq model to both preordering and translation. We
discussed diﬀerent sizes of mT5s [14] and mBART [6],to verify the performance that those models could makein preordering when using manual word alignment data.
For the translation process, we choose mBART as ourbaseline translation model. Translation results with theoriginal source language as input and generated preordering as input is veriﬁed respectively. In the validation of

the ALT Japanese-English dataset [9], the result of preordering as input is similar to that of the baseline. On
the other hand, the result of using preordering as input ishigher than the baseline in the veriﬁcation of the in-house
Chinese-Japanese parallel dataset provided by NTT.

2

Ralated Work

Kawara et al.[4] discussed the inﬂuence of word orderon the NMT model and concluded that it is important tomaintain the consistency between an input source word
order and the output target word orders, to improve thetranslation accuracy.
Murthy et al.[7] proposed a transfer learning approachfor NMT, that trains an NMT model on an assistinglanguage-target language pair, and improves the translation quality in extremely low-resource scenarios.
Nevertheless, both methods above rely on separately pretraining a translation model using a large-scale parallel
corpus, and handle the preordering based on the syntax
tree.
Zhu et al.[16] discussed a framework that focuses on
the translation task limited to a small-scale corpus using
preordering and highly accurate word alignment in lowresource translation. In their work, they used an SMT
model as a solution for translation and received a better
result compared with Transformer. But they did not explore
the use of the seq2seq large-scale pretrained model.
Our work focuses on the low-resource translation task
and uses the large-scale pretrained multilingual model for
ﬁne-tuning not only the preordering but also the translation
procedure.

3 Using Seq2seq Models for Preordering and Translation― 1818 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

3.1

tokens, because they were not aligned with any tokens

Seq2seq Models

on the target side. Speciﬁcally, according to the word

Seq2seq can be described as a sequence input of thealignment, the Japanese sentence ” 私 (I) は 黒い (black)source sequence 𝑆 = {𝑠1 , 𝑠2 , . . . , 𝑠 𝑘 } to a sequence output

猫 (cat) が 好き (like) ” can be easily preordered into theof the target 𝑇 = {𝑡1 , 𝑡 2 , . . . , 𝑡 𝑚 }, where 𝑠𝑖 (𝑖 = 1, . . . , 𝑘)English order of ” 私 (I) 好き (like) 黒い (black) 猫 (cat)”and 𝑡 𝑗 ( 𝑗 = 1, . . . , 𝑚) represent the tokens in the sourcewith the alignments of (私-I), (黒い-black), (猫-cat) andsequence and the target sequence, respectively.

(好き-like) based on the word alignment. Therefore, we

Seq2seq models are basically composed of an en-

ignore ” は” and ” が” in the preordered sequence, because

coder and a decoder [11, 2]. The encoder does a high-they were not aligned to any tokens. Thus, after removing

dimensional vector conversion of the input sequence, and” は” and ” が” in the output side of preordered sequence,the decoder maps the high-dimensional vectors into thethe training pair is ” 私 (I) は 黒い (black) 猫 (cat) が 好

output dictionary from the encoder’s output. This pro-き (like)” and ” 私 (I) 好き (like) 黒い (black) 猫 (cat)cess has applications in tasks such as machine summariza-”. We use such training pairs to train order transformation

tion [10], question-answering systems [15], and machinetranslation [11]. Therefore, we also tried to use the seq2seq

seq2seq neural networks.

model to conduct preordering and translation.

3.2

3.3

Seq2seq Model for Translation

For the translation process, we use mBART as the base

Seq2seq Models for Preorderingtranslation model. To compare diﬀerent input results, we
tried multiple input patterns, which are original input, pre-3.2.1

Preordering Processordered input, tagged input, concatenated input, and mixed

The preordering process transforms the orders of the to-

input. Original input uses the original source language se-

kens in a source sequence to those of the tokens in its target

quence as input, and outputs the target language sequence.

sequence before translation is performed. An example of

We see this pattern as the seq2seq translation baseline.

transferring a Japanese sentence is shown in Figure 1.
䙕2ULJLQDOVRXUFHVHTXHQFH
3UHRUGHU䙕3UHRUGHUHGVRXUFHVHTXHQFH• Original input: Original source sequence ⇒ Targetஎमహःქऋ஀ऌ

language sequence
Preordered input uses the preordered source language se-எ஀ऌహःქ

quence as input, and outputs the target language sequence.
• Preordered input: Preordered source sequence ⇒Target language sequence

7UDQVODWH
䙕7DUJHWODQJXDJHVHTXHQFH,OLNHEODFNFDWV

Transform the word order of the source Japanese
language to the target English language before translation.

Figure 1

For the preordering procedure, we use mT5 [14] andmBART [6], which now are kinds of state-of-the-art
seq2seq models. Both of them have similar self-attentionbased encoder-decoder structures, with a slight diﬀerencein the task of pretraining. The output of mT5 and mBART

Tagged input uses both original and preordered source language sequences as input but carries the sequence type tag
at the head of the sequence (for example, using [ord] and[pre] to represent the original sequence and preordered sequence). Note that the size of the training set is two of thebaseline, because of using original input and preordered
input separately.

are mapped to the entire dictionary rather than the input• Tagged input: [ord] Original source sequence ⇒dictionary.

Target language sequence[pre] Preordered source sequence ⇒ Target language

3.2.2

sequence

Reordered Training Data

We followed our previous work [16] for conducting thetraining data for preordering as in Figure 1.
For the model input, we use the original source sequence.
On the output side, we simply ignored those NULL-aligned

Concatenated input merges the original and preordered
source language sequence into one sequence but split by alearnable symbol.

― 1819 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Original source sequencelarge with a batch size of 324）. We generate the preordered[SEP] Preordered source sequence ⇒ Target language

sequence using the model of the maximum BLEU scoresequence

which is evaluated on the validation parts.

• Concatenated input:

Mixed input is a combination of the three types of input

RIBES result of seq2seq model trained by manual
word alignments of transferring Japanese order into English
order and opposite.

Table 1

above, with the addition of a learning process for preordering. To enable the model to distinguish the expected output
from the diﬀerent inputs, we put a type tag before each input
similar to Tagged input.

Model

Training Batch Size Ja⇒En En⇒Ja

mT5-small

16

0.876

0.872

mT5-base

16

0.895

0.889

• Mixed input: [ord2pre] Original source sequence ⇒mT5-large

16

0.901

0.905

Preordered source sequencemT5-large

32

0.904

0.909

[ord2tgt] Original source sequence ⇒ Target language

mBART-large

16

0.883

0.894

sequence[pre2tgt] Preordered source sequence ⇒ Target lan-

4.3

Translation Setting

guage sequence[concat2tgt] Original source sequence [SEP] Pre-We trained mBART models for translation usingordered source sequence ⇒ Target language sequence

Fairseq 5）, while for each input pattern, every model istrained for 40,000 steps, with a max input length of 1024

4

Experiments

4.1

Datasetand a learning rate of 3e-5 (the same number of update
steps and learning rate with the preordering process). For
the generation process, we use the model with minimum

We use ALT 1）Japanese-English and in-house ChineseJapanese parallel data as our base dataset in our seq2seq
experiments2）. For word alignment, we choose to use theword alignment data based on human annotations. The
data is split into the training, validation, and test parts.

label-smoothed cross-entropy loss on the validation set togenerate the target translation. For preordering inputs, we
use sequences generated by mT5-large, which is trainedwith a batch size of 32.
BLEU score of diﬀerent models when applying thepreorder input pattern for translating Ja-En pairs.

Table 2Each of them in ALT data includes parallel sequence pairsof 18K, 1K, and 1K. The in-house data includes parallel
sequence pairs of 2K, 0.5K, and 0.5K.

Preordering Model
Oracle4.2

Preordering Setting

Training data for seq2seq preordering is made by manual
word alignment as described in section 3.2. We compare
preordering results using RIBES [3] between mT5-small,
mT5-base, mT5-large and mBART-large3）. Every model isTraining

Ja⇒En

En⇒Ja

-

34.82

34.27

mT5-small

16

21.44

26.06

mT5-base

16

24.38

27.68

mT5-large

16

24.83

28.48

mT5-large

32

25.22

28.34

mBART-large

16

23.28

27.28

Batch Size

ensured to be trained for 40,000 steps, with a training batch
size of 16, and a learning rate of 3e-5. Furthermore, due to5

Result5.1

Preordering Performancethe mT5-large obtained the best preordering result with thecondition of a batch size of 16, we trained another mT5-

Table 1 shows the RIBES result between diﬀerent
seq2seq models. As we can see, the RIBES score of1） https://www2.nict.go.jp/astrec-att/member/mutiyama/
ALT/2） In ALT, for language pairs other than Ja-En, there is no supervised
word alignment data available.
3） All pretrained seq2seq models are downloaded from the public
Huggingface library.

mT5-large models have broken through 0.9, regardless of4） We also tried training mT5-large with a batch size of 64, but for
preordering process, the result is lower than training with a batch sizeof 32.
5） https://github.com/facebookresearch/fairseq― 1820 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

BLEU scores between the diﬀerent Model input types and input order. Oracle rearranges the test set according to the manual
word alignment data, which is the most perfect data. Rows with Tagged as model input represent the mixed input pattern result, while
the results in parentheses represent the BLEU score of tagging only on the original order and preordered sequence. † represents thesigniﬁcant diﬀerence (p < 0.05) with baseline using mT5-large.

Table 3Model Input

Input

Preorder Model

Ja-En

En-Ja

Ja-Zh

Zh-Ja

-

25.74

29.32

14.11

17.93

Oracle

34.82

34.27

17.19

22.73

mT5-large

25.22

28.34

14.25

18.46†

Original Order(Baseline)
NormalPreordered
Concatenation

Oracle

35.53

35.74

17.77

22.87

mT5-large

25.62

29.61

14.71†

19.22†-

25.78(25.81)

28.94(29.21)

13.96(14.16)

18.80† (17.95)

Oracle33.67(33.64)

33.48(33.61)

15.86(16.13)

20.81(20.86)

mT5-large
Oracle25.13(25.51)
35.01

28.18(28.45)
35.07

13.68(13.98)
16.14

18.33(17.83)
22.03

mT5-large

25.92

29.66

14.35

19.41†

Original Order
TaggedPreordered
Concatenation

RIBES score when transferring the original source
sequence to preordered source sequence using mT5-large.
Ja⇒En En⇒Ja Ja⇒Zh Zh⇒Ja

In addition, in the results of preordering using mT5-

Table 4

RIBES

0.904

0.909

0.927

0.919

Unigram Precision

0.91

0.92

0.89

0.83

Normalized Kendall’s Tau

0.93

0.94

0.96

0.97

Brevity Penalty

0.96

0.95

0.93

0.95

large, the translation quality of the concatenated input isbetter than that of the other inputs and baseline. By combining the original and preorder inputs into one sequence,
the models could learn more about their relative positions.
For Chinese and Japanese data, the proposed approach is
better than the baseline, which we believe is due to the size

32. Table 4 contains the RIBES score of transferring theof the dataset. For the pretrained seq2seq model, the 18,000
training data of ALT is suﬃcient for ﬁne-tuning with theoriginal source sequence to preordered source sequence us-

original source input. However, under the condition ofing mT5-large. Meanwhile, thanks to the pretraining task,

2,000 parallel data, the model could learn more translation

the preordering result fully demonstrates the feasibility ofrules through the preordered input.

using the seq2seq model in the preordering task.

6

whether the model is trained with a batch size of 16 or

5.2

Translation Performance

Conclusion
In this paper, we proposed to utilize seq2seq multilin-

As shown in Table 3, the concatenated inputs acquire thegual pretrained models for the process of preordering andhighest BLEU score [8] compared to other input patterns.
For comparison experiments, we conducted experiments

translation. We used mT5-large to generate preordering se-using oracle. Oracle means that the test set is also pre-estimated our approach on ALT Ja-En pairs and in-houseordered using manual word alignment data instead of being

Zh-Ja pairs. For the results, Our method is eﬀective in

generated by the seq2seq model. Based on the results, oracle outperformed the baseline by a wide margin. Although

the experiments using Chinese and Japanese parallel dataand is mostly equal to the baseline in Japanese and En-

this result is impractical, it still shows the possibility ofglish pairs. In our future work, we will try the method of

applying our method to the seq2seq model. Table 2 shows

data expansion on the pretraining model, and we will also

the BLEU score of diﬀerent models when applying thetransplant our method to mT5s to evaluate the translation

preorder input pattern for translating Ja-En pairs. It is ap-

accuracy.

quences and mBART to translate. In our experiments, weparent that the ﬁnal translation result corresponds with theperformance of preordering. The higher the quality of thepreordering, the better the ﬁnal translation will gain.

― 1821 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

tion for Natural Language Processing, pp. 1312–1316,

References2022.

[1] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, pp. 4171–4186,
2019.
[2] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, Vol. 9, No. 8, p. 1735–1780,
1997.
[3] H. Isozaki, T. Hirao, K. Duh, K. Sudoh, and H. Tsukada.
Automatic evaluation of translation quality for distant language pairs. In Proc. EMNLP, pp. 944–952, 2010.
[4] Y. Kawara, C. Chu, and Y. Arase. Recursive neuralnetwork-based preordering for statistical machine translation and its analysis. Journal of Natural Language Processing, Vol. 26, No. 1, pp. 155–178, 2019.
[5] Z. Lin, X. Pan, M. Wang, X. Qiu, J. Feng, H. Zhou, and
L. Li. Pre-training multilingual neural machine translationby leveraging alignment information. In Proc. EMNLP,
pp. 2649–2663, 2020.
[6] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer. Multilingual denoising
pre-training for neural machine translation. Transactionsof the ACL, pp. 726–742, 2020.
[7] R. Murthy, A. Kunchukuttan, and P. Bhattacharyya. Addressing word-order divergence in multilingual neural machine translation for extremely low resource languages. In
Proc. NAACL, pp. 3868–3873, 2019.
[8] K. Papineni, S. Roukos, T. Ward, and W. Zhu. Bleu: amethod for automatic evaluation of machine translation.
In Proc. 40th ACL, pp. 311–318, 2002.
[9] H. Riza, M. Purwoadi, Gunarso, T. Uliniansyah, A. Ti,
S. Aljunied, L. Mai, V. Thang, N. Thai, V. Chea, R. Sun,
S. Sam, S. Seng, K. Soe, K. Nwet, M. Utiyama, and
C. Ding. Introduction of the Asian language treebank.
In Proc. Oriental COCOSDA, pp. 1–6, 2016.
[10] T. Shi, Y. Keneshloo, N. Ramakrishnan, and Chandan K. Reddy. Neural abstractive text summarization withsequence-to-sequence models. Vol. 2, 2021.
[11] I. Sutskever, O. Vinyals, and Q. Le. Sequence to sequencelearning with neural networks. In NIPS, Vol. 27, 2014.
[12] B. Tom, M. Benjamin, and et al. Nick, R. Language models
are few-shot learners. In Advances in NIPS, Vol. 33, pp.
1877–1901, 2020.
[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,
A. Gomez, U. Kaiser, and I. Polosukhin. Attention is all
you need. In Proc. 30th NIPS, pp. 5998–6008, 2017.
[14] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou,
A. Siddhant, A. Barua, and C. Raﬀel. mT5: A massively
multilingual pre-trained text-to-text transformer. In Proc.
15th NAACL, pp. 483–498, 2021.
[15] J. Yin, X. Jiang, Z. Lu, L. Shang, H. Li, and X. Li. Neuralgenerative question answering. In Proceedings of theWorkshop on Human-Computer Question Answering,pp. 36–42, 2016.
[16] J. Zhu, Y. Wei, T. Tamura, T. Utsuro, and N Masaaki. A
framework for low resource language translation based on
smt and highly accurate word alignment. In The Associa-― 1822 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

A

Appendix
Table 5 shows our actual transformation results. In the ﬁrst result, although most of the preordered tokens are arranged

in relatively correct places, those have meaningful tokens such as “金曜日 夜” and “停車場” are not placed correctly.
However, in the second result, the generated preordered sequence is exactly the same as the reference preordered sequence.
Japanese original sequence
English target sequence

Table 5 Results for preordering generated by mT5-large.
彼 は 金曜日 の 夜 に サウス メルボルン の 停車場 から トラム を 盗ん だ こと でも 訴え られ て いる 。
He is also accused of stealing a tram on Friday night , from South Melbourne depot .

Reference preordered sequence

彼 て いる でも 訴え られ でも 盗ん だ こと トラム に 金曜日 夜 から サウス メルボルン 停車場 。

Generated preordered sequence

彼 られ て いる でも 訴え こと 盗ん だ トラム から 停車場 の サウス メルボルン に 金曜日 夜 。

Japanese original sequence

知事 の ジョン ・ コーザイン 氏 は 、 幹 細胞 研究 に 2 億 7000 万 ドル を 提供 する 法案 を 発表 し た 。

English target sequence

Governor Jon Corzine announced a bill that would provide $ 270 million to stem cell research .

Reference preordered sequence

知事 ジョン コーザイン 発表 し た 法案 提供 する ドル 2 億 7000 万 に 幹 細胞 研究 。

Generated preordered sequence

知事 ジョン コーザイン 発表 し た 法案 提供 する ドル 2 億 7000 万 に 幹 細胞 研究 。

― 1823 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).