高精度の単語アライメントを利用した
SMTベース低資源機械翻訳
Using Highly Accurate Word Alignment for Low Resource Translation based on SMT

Division of Intelligent Interaction Technologies, Faculty of Engineering, Information and
Systems, University of Tsukuba ／筑波大学システム情報系知能機能工学域教授

Takehito Utsuro ／宇津呂武仁

Takehito Utsuro is a professor at the Division of Intelligent Interaction Technologies, Faculty of Engineering, Informationand Systems, University of Tsukuba, since 2012. His professional interests in natural language processing, Web
intelligence, information retrieval, machine learning, spoken language processing, and artificial intelligence.

Master’s Program in Intelligent and Mechanical Interaction Systems, Degree Programs in Systems and Information Engineering, Graduate
School of Science and Technology, University of Tsukuba ／筑波大学大学院理工情報生命学術院システム情報工学研究群知能機能システム学位プログラム

Jingyi Zhu ／朱鏡伊Jingyi Zhu is a student of Master’s Program in Intelligent and Mechanical Interaction Systems,
Degree Programs in Systems and Information Engineering, Graduate School of Science andTechnology, University of Tsukuba.

Master’s Program in Intelligent and Mechanical Interaction Systems, Degree Programs in Systems and Information Engineering, Graduate
School of Science and Technology, University of Tsukuba ／筑波大学大学院理工情報生命学術院システム情報工学研究群知能機能システム学位プログラム

Takuya Tamura ／田村拓也

Takuya Tamura is a student of Master’s Program in Intelligent and Mechanical Interaction Systems,
Degree Programs in Systems and Information Engineering, Graduate School of Science andTechnology, University of Tsukuba.

NTT Communication Science Laboratories, NTT Corporation ／ NTT コミュニケーション科学基礎研究所

Masaaki Nagata ／永田昌明

Masaaki Nagata is a senior distinguished researcher at NTT Communication Science Laboratories,
NTT Corporation, since 2012. His professional interests in machine translation and word
segmentation.

1

Introduction

Neural Machine Translation (NMT), a data-294

embeddings into a common cross-lingual
embedding space[9][16]. However, these effective
methods need relatively high computation[10].

hungry technology, suﬀers from the lack ofIn this paper, we proposed a framework based

bilingual data in low resource constraints[17][20][4].

on SMT and highly accurate word alignment

To overcome the challenge that exists in small-

methods SpanAlign and AWESoME-align, toscale translation or low resource translation

explore the feasibility of low resource languagetasks, several kinds of research focus on thetranslation. We use both the original orderapproaches such as pre-training with large scalesentences and pre-ordered sentences as input

monolingual data and ﬁne-tuning with a small-because, with the help of high precision word

scale corpus, or using the unsupervisedalignment, it is hard to predict which side asmethods and mapping the monolingual vector

input will be translated in a better result. Since[10][19]

we focus on the limitation of the low resource

made word alignment data or parallel corpus.

language pairs, we use the Asian Language

Then we use the word alignment model toTree bank corpus, which contains 20,000align words in the training sentences, while

parallel sentences as the base corpus. We

the word alignment data is used to train thedo the experiments between the directions

Moses model, consisting of the phrase table1of the Japanese, Chinese, English, Bengali,

and statistical-based language model. At last,

Filipino, Hindi, Indonesian, Malay, and Thai, while

the original order test data or preordered test

Japanese is either the source side or the target

data is translated by the phrase-based SMT

side. The experimental results show that ourmodel. On the other hand, Figure 1(b) shows the

proposed framework signiﬁcantly outperforms

procedure to create preordered test data.

an NMT based on the Transformer-base, andThe word alignment of the training corpus is

except for Ja-Th, the best results of eachalso used to train the Pointer Network. Then thelanguage pairs outperformed Transformer-small.

trained Pointer Network transforms the originalorder test data into preordered test data.

Related Work

3.1

M ultilingual BERT-based Word
Aligners

Although preordering has often been used in
SMT related works, some studies have recently

The ﬁrst method is SpanAlign [12] , whichapplied preordering to NMT. Kawara et al. [7]extracts alignments with reading comprehension

discusses the inﬂuence of word order on thestyle, that inputs with the source language

NMT model and concludes that it is important tosentence and target language sentence,

keep the consistency between the input sourceand predicts a span in the target sentence

word order and the output target word order,

corresponding to the word in the source

to improve the translation accuracy. Murthy etsentence enclosed between the two boundaryal.

proposed a transfer learning approach forsymbols. This approach allows for high precision

NMT, that trains an NMT model on an assistingalignment even with less word alignment ﬁne-language-target language pair, and improves

tuning data in a supervised way.

[11]

the translation quality in extremely low-resource

The second method is AWESoME-align [3] ,scenarios. Nevertheless, those methods bothwhich, on the other side, can be ﬁne-tuned in an

rely on the neural network translation model or

unsupervised way by adjusting the embedding

separately pre-training a translation model by

distribution of mBERT output to achieve worda large-scale corpus. In contrast, our proposedalignments. The advantage is that this method

framework has no neural translation component

does not require manually made word alignment

and we focus on the translation task limited todata.

a small-scale corpus.

3

The Framework Based on SMT
and Word Alignment

As shown in Figure 1(a), for the beginningof the process, we ﬁne-tune the multilingual
BERT-based word aligner using the manually

In our experiments, we ﬁne-tune SpanAlign
1

Unlike the conventional pre-ordering translation, in
our case, the phrase table is made in the originalorder, which means, we did not use the pre-orderedsentences to learn the probability of the phrase
table. We also tried making the phrase table
after pre-ordering the training data, however, thetranslation BLEU score is lower than that made by
the original order data.

YEAR BOOK 2O22 295

4
機械翻訳技術の向上

2

Figure 1: Our proposed framework.

and AWESoME-align respectively and compare

inputs.

them with the unﬁne-tuned (directly using

The output of the attention mechanism is aparameters of pre-trained mBERT) AWESoME-

softmax distribution with a dictionary size equalalign. Because of the supervised attribute ofto the length of the input.

SpanAlign, we only leverage it on Ja-En and JaZh pairs.

3.2

Pre-ordering by Pointer Network

3.2.1

Architecture

The pre-ordering process transforms theorders of the tokens in a source sentence tothose of the tokens in its target sentence
before translation is performed. Figure 3
shows an example of transferring a Japanese
sentence.

296

Figure 2: A rchitecture of Pointer Network for sequenceorder transformation (the modified Pointer Network
accepts the original order sequence as input, andoutputs the pre-ordered sequence).

The original Pointer Network[18] is an LSTM[5]Inspired by this, we apply Pointer Network

based neural network, which aims at solving

to word order rearrangement like Figure 2.

graph theory problems such as the travelingSpeciﬁcally, we replace the input of Pointersalesman problem and convex hull. Structurally,

Network with a sequence of the token instead,an encoding RNN converts the input sequenceand then add an embedding layer to represent

to a vector that is fed to the generatingwords with vectors. At decoding time, thenetwork. And at each step, the generating

decoder predicts the next pointer probability

network produces a vector that modulatesp(Ci | C1, ..., Ci-1, P) relying on inputs and predicted

a content-based attention mechanism over

outputs :u ij = vT tanh(W1e j + W2 d i ) j = 1,..., np(Ci | C1,..., Ci 1, P) = softmax (ui )4Experiments

4.1

Datasetwhere softmax normalizes the vector (of length

W e u s e t h e A L T [15] ( A s i a n L a n g u a g en) to be an output distribution of inputs. P is theTreebank)3 as our main experiment corpus of

input sentence, and Ci is the token of the output

setting Japanese on either the source side orsentence, ui is the vector. Parameters v, W1, aretarget side, about 20K sentence pairs for eachlearnable parameters of the output model, andlanguage pair. Others are Chinese, English,

ej , di represent the encoder state and decoder

Bengali, Filipino, Hindi, Indonesian, Malay, andstate, respectively.

Thai. Parallel data are divided into the training

3.2.2

data (18K), dev data (1K), and test data (1K).

Phrase-based Translation

Phrase-based SMT (PSMT) is found more
eﬃcient than word-based SMT framework

4.2

Experimental Settings

thanks to the use of multi word translation units

4.2.1

Word Aligner Settings

For SpanAlign, we use the ALT Ja-En dev

plays a signiﬁcant role. Bilingual phrase tables

data of word alignment to ﬁne-tune for Ja-En

can be simply seen as lists of terms (words orpair. For Ja-Zh, we use about 3,000 sentences

phrases) in one language associated with theirof in-house word alignment data to ﬁne-tunetranslations in a second language. Therefore,

SpanAlign. Speciﬁc hyperparameters havePhrase-based translation is a process that, forfollowed as default, while the training batch

each token in the source sentence, retrieves

size is set to 8 and the training epoch is set

and outputs the most appropriate target tokens

to 10. The average extraction threshold onin the built-up phrase table.

bidirectional sides is 0.4. We did not conduct

In our approach, for the translation model
input, we use both original order sentences

experiments based on other hyperparameters,
so our choice may not be the most optimal.

and preordered sentences as the SMT input.

For AWESoME-align, unsupervised data allSpeciﬁcally, we replace the original SMT

comes from ALT dev data. The setting of thealignment method GIZA++ 2 with SpanAlignﬁne-tuning step of the training epoch as well as

and AWESoME-align, and follow the workﬂow ofbatch size is consistent with the SpanAlign.

normal SMT.

Figure 3: T ransform the word order of the source Japanese to
the target English before translation.

2

Table 1: P, R, and F1 score of each alignment approaches
of En-Ja. After ﬁne-tuning, the precision ofunsupervised AWESoME-align can reach the samelevel of supervised SpanAlign, while for the recall
there remains still a gap.
Alignment ApproachP

R

F1 score
0.54

GIZA++

0.54

0.55

AWESoME-align

0.71

0.46

0.56

SpanAlign

0.79

0.86

0.83

AWESoME-align (fine-tuned)

0.79

0.58

0.67

4.2.2

Pointer Network Settings

Training data for the Pointer network are thehttps://github.com/moses-smt/giza-pp3

https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/YEAR BOOK 2O22 297

4
機械翻訳技術の向上, and for the translation part, the phrase table[1]

training data of original order sentences andevaluation, we only use Japanese and English

pre-ordered sentences made by the alignments

data to verify the performance of the Pointergenerated by SpanAlign. We use a 2-layer

Network. Table 1 shows the F1 score between

bidirectional LSTM, with a hidden state of 512

SpanAlign and AWESoME-align, demonstrating

and an embedding state of 128. And we set thethe high alignment accuracy.

training batch size to 16, the learning rate toTable 2 shows the result of the score of the

3e-4, the training epoch is 10, max sequence

preordered test data for transferring Japaneselength to 120. After training, the weighted

order into English order veriﬁed by RIBES. Here,

Pointer Network is used to do the pre-ordering

we see ALT Japanese manual alignment data

operation for test data sentences. We exploit

as the reference. From the results, Pointer

RIBES , an eﬃcient measure for automatically

Network trained with tokens extracted from

evaluating machine translation qualities

SpanAlign and AWESoME-align (ﬁne-tuned) arebased on the order of words, to evaluate thenearly the same as that of manual alignment,

performance of the Pointer Network.

though the ﬁne-tuned AWESoME-align is left

[6]

Table 2: R IBES result of Pointer Network trained byword alignments extracted by each approach, oftransferring Japanese order into English order.

behind. It can be considered that PointerNetwork successfully learned certain language

Model

RIBES

order features which are eﬀective for the pre-Manual Word Alignment

0.761

ordering task.

GIZA++

0.631

AWESoME-align0.623

SpanAlign

0.751

AWESoME-align (fine-tuned)

0.722

4.3

Table 4: BLEU scores between the phrase-based translationof diﬀerent phrase table length.‘Pre’represents for‘Pre-order input’, while‘Ori’represents for‘Original
input’and‘FT’is short for‘fine-tuned’.
Model

Statistical Machine Translation

LPT = 3

LPT = 5

Pre

Ori

Pre

Ori

GIZA++

8.33

8.05

8.40

8.36

SpanAlign

11.61

9.83

8.85

8.47

AWESoMEalign FT

10.86

9.15

12.35

9.88

AWESoMEalign FT+ MERT

11.33

10.18

12.42

10.40

side sentences contained in the training part

SpanAlign

10.11

8.24

8.59

7.84

corpus, to ensure the ﬂuency of the output

AWESoMEalign FT

10.62

9.91

10.80

9.65

AWESoMEalign FT+ MERT

10.80

10.20

11.24

9.67

We use Moses[8]4 to make the phrase table,
and the maximum length of each phrase is set
to 3. We use a statistical-based trigram LM

En → Ja(Language Model), which is learned by target

sequence. Furthermore, we use dev data andZh → Ja

MERT[13], joint with batch MIRA[2] to adjust theweight of the translation model. Note that alldata used for SMT is token-based, we did notlearn the BPE to further split the tokens.

4.4.2

Translation Accuracy

As a criterion to verify the translation
accuracy, we use the BLEU[14] score. And we

4.4

Results

select Transformer-base and Transformer-

4.4.1

Pointer Network Performancesmall [17] as our baseline. Table 3 shows the

Because there is no ALT Chinese-Japanesetranslation accuracy of our proposed method

manual alignment data that exists forwith the alignment approach of SpanAlign and
AWESoME-align (FT), also the accuracy of the4298

https://www.statmt.org/moses/

baseline. From the two sets of the results, usingSpanAlign as an aligner is better than ﬁne-tuned

used for translation, the translation accuracy

AWESoME-align for EnJa, nevertheless, using

decreased compared with that of length 3. In

AWESoME-align as an aligner is better thancontrast, when the length of the phrase table

Spanalign for Ja-Zh. The factor that causes thisis changed from 3 to 5 using AWESOME-alignresult is that, for En-Ja, both aligners are ﬁne-in the En-Ja direction, the accuracy of thetuned by dev data in ALT, while for Ja-Zh, thetranslation is improved regardless of whether

ﬁne-tuning data of SpanAlign comes from in-pre-order or original order is used as the input.

house rather than ALT, so there are diﬀerences

For the translation of AWESoME align in the Zh-in speciﬁc domains. In addition, the results

Ja direction, the result with pre-order as input

under various experimental conditions of our

improved, while original order as input decreased

proposed framework are superior to Transformersmall.
4.4.3

5

Inﬂuence of the Length of Phrase Table

Conclusion and Future WorkIn this paper, we propose a framework for low

the phrase table on translation results in En-

resource translation using SMT joint with highly

Ja and Zh-Ja directions, which is shown inaccurate word alignment method SpanAlign

Table 4. For SpanAlign, we are surprised to

and AWESoME-align rather than a sequence-to-ﬁnd that when a phrase table with length 5 issequence neural translation model. We use both

Table 3: BLEU score between two types of Transformer baseline and proposed method with alignment approach of SpanAlignand AWESoME-align (FT). FT represents for‘fine-tuned’. † for significant (p < 0.05) difference with baseline, and ‡ forsignificant (p < 0.05) difference of higher of the result of original input or pre-ordered input with the other side.
Model

Ja → En

En → Ja

Ja → Zh

Zh → Ja

Pre

Ori

Pre

Ori

Pre

Ori

Pre

Ori

Transformer-base

-

8.12

-

5.91

-

4.08

-

6.14

Transformer-small

-

7.02

-

10.64

-

6.33

-

9.83

PSMT+SpanAlign

8.74 †

9.23 †‡

11.61†‡

9.83

7.17 †

8.36 †‡

10.11 †

8.24

PSMT+AWESoME FT

8.36

9.22 †‡

10.86 †‡

9.15

9.30 †

9.49 †

10.62 †‡

9.91

PSMT+AWESoME FT
+MERT

8.37

9.65 †‡

11.33 †‡

10.18

9.29 †

10.04 †‡

10.80 †‡

10.20

Model

Ja → Bg
Pre

Bg → Ja
Ori

Pre

Ja → Fil
Ori

Pre

Fil → Ja
Ori

Pre

Ori

Transformer-base

-

6.03

-

4.45

-

3.26

-

4.65

Transformer-small

-

11.25

-

7.63

-

7.32

-

7.76

PSMT+AWESoME FT

10.97

10.64

8.86 †

9.28 †

7.84 †‡

7.02

8.92 †‡

6.75

PSMT+AWESoME FT+MERT12.96 †

12.90 †

8.89 †

9.22 †

8.48 †

8.24 †

8.90 †‡

7.85

Model
Transformer-base

Ja → Hi

Hi → Ja

Ja → Id

Id → Ja

Pre

Ori

Pre

Ori

Pre

Ori

Pre

Ori

-

5.93

-

4.92

-

2.04

-

5.68

Transformer-small

-

13.11

-

8.44

-

5.23

-

9.03

PSMT + AWESoME FT

13.20

13.33

11.32 †‡

10.70 †

5.98 †

5.62

10.31†‡

7.19

PSMT+AWESoME FT+MERT15.85 †

16.17 †

11.32 †‡

10.84 †

6.83 †

6.71 †

10.35 †‡

8.73

Model
Transformer-base

Ja → Ms

Ms → Ja

Ja → Th

Th → Ja

Pre

Ori

Pre

Ori

Pre

Ori

Pre

Ori

-

2.32

-

6.00

-

4.87

-

5.56

Transformer-small

-

5.18

-

9.28

-

7.25

-

7.84

PSMT + AWESoME FT

6.62 †‡

6.07 †

9.93 †‡

7.34

4.64

4.70

6.35

5.99

PSMT+AWESoME FT+MERT6.54 †

7.54 †

9.86 †‡

8.83

6.33

6.36

7.96 †

7.25

YEAR BOOK 2O22 299

4
機械翻訳技術の向上

We explore the inﬂuence of the length ofpre-ordered sentences which are preordered by［8］P. Koehn, H. Hoang, A. Birch, C. Callison-

Pointer Network, and original order sentences as

Burch, M. Federico, N. Bertoldi, B. Cowan,

input and perform the phrase-based translation.

W. Shen, C. Moran, R. Zens, C. Dyer, O.

The results exceed the baseline of Transformer-

Bojar, A. Constantin, and E. Herbst. Moses:

base and Transformer-small except for Ja-Th

Open source toolkit for statistical machine

and Th-Ja for high precision alignment. In futuretranslation. In Proc. 45th ACL Volume

work, we will apply our approach with highly

Proceedings of the Demo and Poster

accurate word alignment to other language

Sessions, pp. 177–180, 2007.

pairs.

［9］Z. Lin, X. Pan, M. Wang, X. Qiu, J. Feng, H.
Zhou, and L. Li. Pre-training multilingual

References

neural machine translation by leveraging［1］A. Bisazza and M. Federico. Surveys: Aalignment information. In Proc. EMNLP, pp.

survey of word reordering in statistical

2649–2663, 2020.

machine translation: Computational models［10］ Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M.

and language phenomena. Computational

Ghazvininejad, M. Lewis, and L. Zettlemoyer.

Linguistics, pp. 163–205, 2016.

Multilingual denoising pre-training for neural［2］C. Cherry and G. Foster. Batch tuning
strategies for statistical machinetranslation. In Proc. 9th NAACL, pp. 427–
436, 2012.

machine translation. Transactions of the
ACL, pp. 726–742, 2020.
［11］ R. Murthy, A. Kunchukuttan, and P.
Bhattacharyya. Addressing word-order［3］Z. Dou and G. Neubig. Word alignment by

divergence in multilingual neural machine

ﬁne-tuning embeddings on parallel corpora.

translation for extremely low resource

In proc. 16th EACL, pp. 2112– 2128, 2021.

languages. In Proc. NAACL, pp. 3868–［4］K. Duh, P. McNamee, M. Post, and B.
Thompson. Benchmarking neural and［12］ M. Nagata, K. Chousa, and M. Nishino.

statistical machine translation on low-

A supervised word alignment method based

resource African languages. In Proc. 12th

on cross-language span prediction using

LREC, pp. 2667–2675, 2020.

multilingual BERT. In Proc. EMNLP, pp. 555–

［5］S. Hochreiter and J. Schmidhuber. Long
short-term memory. Neural Computation,
Vol. 9, No. 8, p. 1735–1780, 1997.
［6］H. Isozaki, T. Hirao, K. Duh, K. Sudoh, and H.

565, 2020.
［13］ F. Och. Minimum error rate training instatistical machine translation. In Proc.
41st ACL, pp. 160–167, 2003.

Tsukada. Automatic evaluation of translation［14］ K. Papineni, S. Roukos, T. Ward, andquality for distant language pairs. In Proc.

W. Zhu. Bleu: a method for automatic

EMNLP, pp. 944–952, 2010.

evaluation of machine translation. In Proc.

［7］Y. Kawara, C. Chu, and Y. Arase. Recursive

300

3873, 2019.

40th ACL, pp. 311–318, 2002.

neural network-based preordering for［15］ H . R i z a , M . P u r w o a d i , G u n a r s o , T .

statistical machine translation and its

Uliniansyah, A. Ti, S. Aljunied, L. Mai, V.

analysis. Journal of Natural Language

Thang, N. Thai, V. Chea, R. Sun, S. Sam, S.

Processing, Vol. 26, No. 1, pp. 155–178,

Seng, K. Soe, K. Nwet, M. Utiyama, and C.

2019.

Ding. Introduction of the asian language

treebank. In Proc. Oriental COCOSDA, pp.

とれるように共通の多言語埋め込み空間に対応させ、二1–6, 2016.

言語間で類似の意味を持つ語の埋め込みが近くなるよう［16］ S. Sen, K. Gupta, A. Ekbal, and P.

にするとともに、反復的に疑似の原言語文と目的言語文

Bhattacharyya. Multilingual unsupervisedを生成し、訓練の過程を行う。しかし、これらの方式に

NMT using shared encoder andは多くの計算量が必要な点が問題である。

languagespeciﬁc decoders. In Proc. 57th
ACL, pp. 3083–3089, 2019.

本論文では、低資源言語翻訳方式として、SMT、および、高精度単語アライメント手法である SpanAlign［17］ A. Vaswani, N. Shazeer, N. Parmar, J.

または AWESoME-align に基づく枠組みを提案する。

Uszkoreit, L. Jones, A. Gomez, U. Kaiser, and

この枠組みにおいては、NMT モデルではなく、フレー

I. Polosukhin. Attention is all you need. In

ズベースの SMT モデルを用いる。本論文では特に，高Proc. 30th NIPS, pp. 5998-6008, 2017.

精度単語アライメント手法である SpanAlign または［18］ O. Vinyals, M. Fortunato, and N. Jaitly.

AWESoME-align によって得られた単語アライメント

Pointer networks. In Proc. 28th NIPS, pp.1-

を入力として SMT モデルを訓練することにより、高精9, 2015.

度な SMT モデルを実現している。さらに、入力文の語
順を目的言語文の語順に変換するためのニューラルネッ

R. Al-Rfou, A. Siddhant, A. Barua, and C.

トワークとして Pointer Network を用いる。各言語対

Raﬀel. mT5: A massively multilingual pre-において、原言語文の語順のままで SMT モデルを適

trained text-to-text transformer. In Proc.

用した場合、および、目的言語文の語順へ変換後 SMT

15th NAACL, pp. 483–498, 2021.

モデルを適用した場合の二通りの評価を行った。SMT［20］ P. Zaremoodi and G. Haﬀari. Adaptively

モ デ ル の 訓 練 用 コ ー パ ス と し て、Asian Languagescheduled multitask learning: The case ofTreebank コーパスの対訳文 2 万文を用いた。言語対

low-resource neural machine translation. Inとして、日本語を原言語または目的言語として、中国

Proc. 3rd WNGT, pp. 177–186, 2019.

語、英語、ベンガル語、フィリピン語、ヒンディー語、
インドネシア語、マレー語、タイ語との間の翻訳におい高精度の単語アライメントを利用した SMT ベース低資源機械翻訳（抄録）て評価を行った。評価結果においては、提案手法は、全言語対において Transformer-base を上回り、言語対
Ja-Th を除いては、Transformer-small を上回った。

近年、従来の統計的機械翻訳（Statistical MachineTranslation；SMT）に代わって，ニューラルネットワー
ク 機 械 翻 訳（Neural Machine Translation；NMT）
モデルが盛んに研究されている．通常、NMT においては、大規模な訓練用資源が不可欠であるため、低資源言語翻訳においては、高い性能を出すことが難しい。
NMT におけるこれまでの低資源言語翻訳に関する研究としては、主に以下のアプローチの研究が行われてきた。
一つのアプローチとして、大規模な単言語コーパスを用
いて単語埋め込み表現を学習し、その後、小規模な二言語対訳コーパスを用いて、下流タスクにおける finetuning を行うものがある。一方、教師なしのアプロー
チでは、原言語と目的言語の間で単語埋め込みの対応がYEAR BOOK 2O22 301

4
機械翻訳技術の向上［19］ L. Xue, N. Constant, A. Roberts, M. Kale,