Efficient Extraction of Pseudo-Parallel Sentencesfrom Raw Monolingual Data Using Word Embeddings
Benjamin Marie and Atsushi Fujita
National Institute of Information and Communications Technology, Japan{bmarie, atsushi.fujita}@nict.go.jp
Highlights

Motivation

2-step method for sentence pair extraction:

Bilingual sentence pair

1. filtering with sentence embeddings• indispensable for MT, but costly to produce

2. refining with a classifier• can be extracted automatically from monolingual corpora

Resource-less: extraction from raw monolingual data
⇒ no need of document-level information⇒ no strong reliance on a lexical translation model

Weaknesses of the previous work
• relies on document pairs to reduce the search space and to use
cross-lingual IR methods (Abdul Rauf+, 11; S, tefan̆escu+, 12)⇒ but document pairs are rare and difficult to collect

Fast: 12x faster than the state-of-the-art method (Tillmann+, 09)• relies heavily on lexical translation models (Tillmann+, 09)

Useful: as training data for MT⇒ up to +1.7 BLEU in domain adaptation⇒ assume the availability of large parallel corpora⇒ significant reduction of out-of-vocabulary (OOV) words⇒ prone to collecting sentence pairs with less OOV words

Step 1: fast filtering with embeddings
Objective: reduce the size of the search space

SMT experiments in domain adaptation
Data

Requirement: score efficiently trillions of sentence pairs

Medical translation task: EMEA (Fr↔En) (Carpuat+, 12)

Sentence pair scoring with word embeddings:

General-domain parallel data: Europarl (1.99M sent.)

1. train source and target word embeddings on monolingual data
2. project them in the same space (Mikolov+, 13)

Medical-domain monolingual data: WMT’14 (Fr-En: 1M-5M sent.)Phrase table adaptation using extracted sentence pairs (PBSMT)

3. compute sentence embeddings by averaging the word embeddings
4. score all sentence pairs by computing their cosine similarity
Return: k-best target sentences for each source sentence

SystemNot adapted
Baseline (Tillmann+, 09)

Step 2: refining with a classifier
Objective: rerank the k-best sentence pairs with a classifier
Requirement: characterize each pair with informative features
Features:cov.
constraint

Proposed method

√√

Fr→En
BLEU #OOVEn→FrBLEU #OOV

#extracted
pairs

speed
(#pairs/sec)

25.9

3,134

23.1

3,099--

27.2

2,729

24.7

2,661

121k

1.22M

28.6
26.1

1,985
3,064

26.4
23.2

1,955
3,077

361k
11k

14.46M
19.21M⇒ +1.4 (Fr→En) and +1.7 (En→Fr) BLEU points of improvement⇒ faster extraction of more useful sentence pairs⇒ significant reduction of the number of OOV tokens

sntemb: cosine similarity between sentence embeddings (step 1)
maxalemb: max. alignment between word embeddings (Kajiwara+, 16)

Analysis|x|1 Xembemb
S(x, y) =max φ(xi , yj )|x| i=1 j
lexprob: lexical translation probability trained on truly parallel sentences, for both translation directions:
|x|

X 1

⇒ disposal of useful sentence pairs by the coverage constraint

|y|

1 X
tok tok
P (x|y) =log(p(xi |yj ))
|x|
|y|
i=1
j=1
length: ratio of the source and target sentence lengths, |x|/|y|Training data:
positive: small set of held-out parallel sentences (e.g., 5k sent.)
negative: randomly paired source and target sentences
Classifier: maximum entropy
Reranking:1. score each sentence pair with the classifier
2. rerank the sentence pairs given their score
Return: 1-best sentence pair for each remaining source sentence whosescore exceeds a given threshold

Most important features:⇒ lexprob⇒ maxalembClassifier accuracy (step 2):

Feature set
all-sntemb-maxalemb-maxalemb-lexprob-length

Fr→En

En→Fr

28.6
28.8
29.0
28.4
28.3
28.9

26.4
26.1
26.1
25.6
26.0
26.4

⇒ on truly in-domain parallel sentences: 89.6%

Conclusion & future workConclusion• faster than previous work and extract more useful sentence pairs• provide a better handling of OOV• useful in low-resource settings by leveraging monolingual data
Future work
• speed up the extraction: lower number of dimensions for word embeddings, search approximations (e.g., LSH)
• evaluate extracted sentence pairs in more downstream tasks (phrasepair extraction, NMT, ...)