Unsupervised Extraction of Partial Translations
for Neural Machine Translation
Benjamin Marie and Atsushi Fujita{bmarie,atsushi.fujita}@nict.go.jp
National Institute of Information and Communications Technology (NICT), Kyoto, Japan

Key Points

Motivation

Assumptions

Generation vs. extraction of synthetic bilingual data

• Partial translations can be extracted from monolingual data

• Generation of back-translations (Sennrich+, 16) :synthetic bilingual data with a synthetic source side

• Prior translation knowledge can bias newly extracted knowledge• Extraction of partial translations:
synthetic bilingual data in which both sides are fluent sentences

Method
1. Extraction: only using an unsupervised phrase table

Example of a Partial Translation

2. Post-processing: masking untranslated tokens• Application: training data for NMT

Englishmonolingual data

Germanmonolingual data——————————————————————————————————————————————————————-——————————————————————————————————————————————————————-

1

Extraction ofpartial translations
with an unsupervised
phrase table

Using partial translations in NMT• Better translation quality for low-resource settingsa man was arrested at the scene .
der Mann wurde festgenommen .

2• Complementary to back-translations as NMT training data

UNKPP man was arrested UNKPP UNKPP UNKPP .

Masking of unaligned
source tokens

der Mann wurde festgenommen .

Approach: the Whole Framework

Evaluation: Experiments in NMT

1. Step by step partial translation extractionDatasource language
monolingual data

• Language pairs: English-German, English-Turkish, Bengali-Malay

target language
monolingual data

• Bilingual training data (low-resource conditions):
sourcephrases

source wordembeddings

target wordembeddings

targetphrases– en-de and en-tr: 100k sentence pairs from WMT18– bn-ms: 18k sentence pairs from ALT project (Riza+, 16)bilingual wordembeddings• Monolingual data for phrase table induction and partial translations
extraction:

unsupervised
phrase tableinduction

induced phrase tablepartial translation extraction– Resouch-rich languages: 239M, 237M, and 104M lines, for English, German, and Turkish

partial translations– Low-resource languages: 5.3M and 4.6M lines for Bengali andMalay• Induce a phrase table from monolingual data (Marie+, 17; Artetxe+, 18;Lample+, 18)• Make a set of sentence pairs through the Cartesian product of thesets of sentences in source and target monolingual data

NMT systems• baseline: trained only using the original bilingual training data

• Find the best target sentence for each source sentence• backtr: same data as baseline mixed with 100k back-translations– Filter sentence pairs given their coverage by the phrase table• partial: same data as baseline mixed with 100k partial translations– Score the remaining sentence pairs with phrase-based forced decoding (PBFD) (Zhang+, 17)

Results• Keep the n best sentence pairs with the highest PBFD score

Training data

2. For training NMT• Mask unaligned tokens found by PBFD on the source side of partial
translations• Resource-rich scenario: mix the resulting partial translations withback-translations and the original bilingual data

en→deen→trbn→msbaseline

7.1

9.3

6.1

backtrpartial

9.1
9.9

11.4
10.4

5.4
5.5

backtr+partial

11.5

11.6

4.5

• backtr and partial are complementary (best configuration)• Low-resource scenario: mix the resulting partial translations with thesmall original bilingual data

• Partial translations can be more useful than back-translations (en-de)• For bn-ms (low-resource), both backtr and partial failAnalysis
Phrase table
Training data

Masking vs. dropping
en→debn→msbaseline

7.1

6.1

unsupervised
surpervised

9.9
9.4

5.5
6.3

• More useful partial translations
with an unsupervised phrase table

Training databaselineoriginal
droppedpartialWith large parallel data

en→deen→tr

para.

7.1

9.3

all
all
all
all

6.2
8.8
9.9

7.7
10.0
10.4

• Masking unaligned tokens performs better
than dropping thembacktrpartialen→deen→tr

1M
1M

26.2
27.7
26.4
28.2

13.6
18.6
14.7
19.0

1M
1M

• Back-translations are more useful• backtr and partial are complementary