Summary

Step 1. Seed Paraphrase Acquisition

Large multiple of # of seeds! Paraphrase acquisition! Pivot-based PA using generic SMT systems! (A) Europarl + GigaFrEn, (B) NTCIR Patent data

Step 1.
Seed paraphrases (PSeed)	
health issue ⇒ health problem
health problem ⇒ health issue
look like ⇒ resemble
regional issue ⇒ regional problem
regional problem ⇒ regional issue
resemble ⇒ look like

Monolingual
Non-parallel
Corpus	

Step 2.
Paraphrase patterns	
X issue ⇒ X problem
X ∋ {“health”, “regional”}
X problem ⇒ X issue
X ∋ {“health”, “regional”}

Step 3.
Novel paraphrases (PHvst)	
backlog issue ⇒ backlog problem
communal issue ⇒ communal problem
phishing issue ⇒ phishing problem
spatial issue ⇒ spatial problem
unrelated issue ⇒ unrelated problem
…

105
104

rp: control system

lp: control apparatus

rp: the control device.015
lp: control apparatus

lp: controller

.005
.004
.003

rp: control device of the.001

rp: control system of.001

rp: controlling device

lp: the control apparatus
lp: control apparatus of
lp: controlling unit
lp: control equipment

rp: a control system for an

lp: controller for arp: a controlling device

lp: to the control apparatus

106105.135103

8.010.008.004.002

rp: control device.001

18.1M

106

1.22M

105
10Ignore morphological variatione.g., number (sg./pl.), gender, case, person, tense! Related work

103

7106
105
PHvst (Patent)
PSeed (Patent)
PHvst (Europarl)
PSeed (Europarl)

4

103

1.41M

106
105
10

Pair in PHvst
LHS in PHvst
Pair in PSeed
LHS in PSeed

4

106
107
108
# of words in the English side of bilingual corpus

108

107

101028.7M

7103

8108

00.1

0.2

0.3 0.4 0.5 0.6 0.7
Probability threshold thp

0.8

0.9

107
10610510103

1

PHvst (Patent)
PSeed (Patent)
PHvst (Europarl)
PSeed (Europarl)

400.1

0.2

0.3 0.4 0.5 0.6 0.7
Similarity threshold ths

0.8

0.9

1

Good quality! Human evaluation of phrase substitutions

Europarl paraphrases on WMT “newstest” data Comparable to the state-of-the-art

Develop patterns manually [Jacquemin, 99][Fujita+, 07]
 Add contextual constraints [Callison-Burch, 08][Zhao+, 09]

5-pt	
Binary	n	G	M	 G≥4	 M≥3	 Both	PSeed	 55	 4.60	 4.35	 .85	 .93	 .78	PHvst	 295	 4.22	 3.35	 .74	 .67	 .55	Total	 350	 4.28	 3.50	 .76	 .71	 .58	Step 3. Paraphrase Instance Acquisition! Harvest novel instances of the patterns

1. Collect expressions that match both sides of the pattern
2. Score each instance by contextual similarity
Learn class-dependent patterns [De Saeger+, 09;11]
 (Pattern-dependent) set expansion

6101010# of words in the English side of bilingual corpus

! Related work

Pair in PHvst
LHS in PHvst
Pair in PSeed
LHS in PSeed

4.001

106
107
108
# of words in the English side of bilingual corpus

108

710

PRaw
PRaw (thp=0.01)
PSeed (thp=ε, ths=ε)
PSeed (thp=0.01, ths=ε)

104

108

! Identical words of LHS and RHS  Variable slots

7101010# of words in the English side of bilingual corpusp(lp|rp).153

Step 2. Paraphrase Pattern Induction
6107# of paraphrase pairs# of unique LHS phrases.032

rp: control device# of paraphrase pairs# of paraphrase pairs

106# of paraphrase pairs

BilingualParallelCorpus	

107

55

4.5

4.5
Avg. scorep(rp|lp).172

108

PRawPRaw (thp=0.01)
PSeed (thp=ε, ths=ε)
PSeed (thp=0.01, ths=ε)103

Translation table	
health issue ||| problème de santéhealth problem ||| problème de santélook like ||| ressemble
regional issue ||| problème régional
regional problem ||| problème régional
resemble ||| ressemble

108# of paraphrase pairs# of unique LHS phrases

Corpora (bilingual parallel and monolingual) Tokenizer
 SMT system
 Lists of stop words (optional) Morphological resources

One-variable patterns and single words
 High leverage rate (# pair):
1580%, 2140%
 Paraphrases for many novel phrases
# of paraphrase pairs! Resources

e.g., Phrase-based SMT system [Koehn, 03]
1. Clean up phrase table: sig. pruning [Johnson+, 07]
2. Pair phrases that get translated to the same phrases[Bannard and Callison-Burch, 05]
3. Filter paraphrase candidate pairs
3a. stop word differences, word super-sequences
3b. conditional probability and contextual similarity
Avg. scoreThrough generalization and instantiation
 Using both bilingual and monolingual data

4
Grammaticality (PSeed)
Grammaticality (PHvst)Meaning (PSeed)
Meaning (PHvst)

3.5
300.1

0.2

0.3 0.4 0.5 0.6 0.7
Probability threshold thp

0.8

0.94
Grammaticality (PSeed)
Grammaticality (PHvst)
Meaning (PSeed)
Meaning (PHvst)

3.5
3
1

00.1

0.2

0.3 0.4 0.5 0.6 0.7
Similarity threshold ths

0.8

0.9

1

Additional statistics

Recipe for contextual similarity

Human evaluation: details! Paraphrase patterns! Ingredients! Show 5 alternatives at the same time

Coverage depends on corpus/domain
 Mostly 1-var patterns
4053010Coverage [%]# of paraphrase patterns

106

104
All (Patent)
1-var (Patent)
All (Europarl)
1-var (Europarl)

103
102! Leverage rate

… There have been many approaches to compute the similarity
between words based on their distribution in a corpus. …

40
20PSeed grows slowly
 PHvst depends on patterns
3103

3102

2101

186105

5104

4103

3102

2101

17810545104103

3102

2101

13456
Length of LHS phrase

74103

3102

2101

18100

23456
7
Length of source language phrase810676105

5104

4103

3102

2101
23456
Length of LHS phrase

78105

645104103

3102

2101

1123456
Length of LHS phrase

78100! “Grammaticality” and “Meaning equiv.”

5-pt scales and binary prec. [Callison-Burch, 08]
 [Callison-Burch, 08]
 Europarl (10 langs-En) + CCG + LM
 WMT 2007 “newstest” data Binary prec. : .68 for G, .62 for M, .55 for both
 Ours (total): .76 for G, .71 for M, .58 for both [Chan+, 11] Europarl (10 langs-En) + CCG + Google N-gram
 Europarl (i.e., closed)
 Score for 1-best: 4.2 pts for G and 3.7 pts for M
 Ours (1-best): 4.57 pts for G and 3.96 pts for M
Limitations! Our method (current version)
Does not cover totally different expressions! Type-based approaches

Do not properly deal with polysemy Tend to miss rare expressions
! Corpus-based approaches

Do not acquire expressions that do not appear

Future work

! Any high-prec. set can be used as PSeed! In-depth analyses of the proposed methode.g., Multiple definition sentences [Hashimoto+, 11]... Osteoporosis is a disease that decreases the quantity of boneand makes bones fragile ... Osteoporosis is a disease that
reduces bone mass and increases the risk of bone fracture. ...

106

7To make results more consistent To reduce the human laborAlternative for seed paraphrases


100

107

Patent, PHvst

transitional {task, strategy, phase, costs, ...}
Europarl, PHvst⇒ {task, strategy, phase, costs, ...} of transitionin the course of the last few {weeks, years, decades}⇒ during recent {weeks, years, decades}
Patent, PSeed
overall structure ⇒ entire configurationin accordance with the structure mentioned above ⇒ due to such a constitution{bypass, chip} condensers ⇒ {bypass, chip} capacitors
will be described with reference to {drawings} ⇒ is explained based on the {drawings}{layer, ceramic, ferroelectric, solid, ...} condensers
Patent, PHvst⇒ {layer, ceramic, ferroelectric, solid, ...} capacitors
will be described with reference to {embodiments}⇒ is explained based on the {embodiments}

100107

Patent, PSeed810662104

1107

7151100

Length of RHS phrase

3456
Length of LHS phrase

Europarl, PHvst

8
Length of RHS phrase

2105

8
Length of RHS phrase

1071616106

7107

Europarl, PSeed

8
Length of RHS phrase

234
567
Length of source language phrase# of pairs

1100

Patent, Ttable

8# of pairs

4107# of pairs

104

R4:between:words:based:onmulti-lateral ⇒ multilateral
i would like to start by congratulating ⇒ let me first of all congratulatetransitional {process, year} ⇒ {process, year} of transitionin the course of the last few {months} ⇒ during recent {months}

106
107
108
# of words in the English side of bilingual corpus# of pairs

5

Length of target language phrase

105# of pairs

6# of pairs

Length of target language phrase

10

7

R2:between:words

Europarl, PSeed

Our filters tend to discard long phrases
 Setting: 1-var patterns & single-word fillers
6

L4:been:many:approaches:toExamples

2

Europarl, Ttable

R1:between

R3:between:words:based! Phrases tend to be short107

L2:approaches:to

L3:many:approaches:toPHvst (Patent)
PSeed (Patent)
PHvst (Europarl)
PSeed (Europarl)

4

1

106
107
108
# of words in the English side of bilingual corpus

8

L1:to



Avg. # of RHS phrases

Ratio of PHvst to PSeed

106
107
108
# of words in the English side of bilingual corpus

5

LHS (Patent)
Pair (Patent)
LHS (Europarl)
Pair (Europarl)

60

15%

! Relative yield

Small bilingual data High leverage

80

0

100

106
107
108
# of words in the English side of bilingual corpus40%

All (Patent)
1-var (Patent)
All (Europarl)
1-var (Europarl)

20

Extract contextual features: adjacent n-grams
 cf. Bag-of-words (cheap but noisy)
 cf. Dependency trees (accurate but expensive) Weight and filter features: nothing Aggregate into a single value: cosine of vectors




PSeed
decreases the quantity of bone ⇒ reduces bone mass
makes bones fragile ⇒ increases the risk of bone fracture
decreases the quantity of {body, tissue, fat, gas, tumor, muscle, ...} PHvst⇒ reduces {body, tissue, fat, gas, tumor, muscle, ...} mass

Similarity metrics
 Paraphrase patterns with more than one variable Size & type of monolingual corpora
! Sophisticated paraphrase patterns

Hierarchical pattern induction
 Deeper level of lexical correspondences! Use for NLP applications