FUN-­‐NRC:	  Paraphrase-­‐Augmented	  Phrase-­‐Based	  SMT	  Systems	  for	  NTCIR-­‐10	  PatentMT	  Atsushi	  Fujita	  	  	  	  	  	  	  	  	  Future	  University	  Hakodate	  hFp://paraphrasing.org/~fujita/	  Marine	  Carpuat	  	  	  	  	  	  	  	  	  NaJonal	  Research	  Council	  hFp://marinecarpuat.weebly.com/

Summary	  of	  our	  systems! Phrase-­‐based	  SMT	  +	  paraphrases	  State-­‐of-­‐the-­‐art	  non-­‐hierarchical	  system:	  PortageII	  @	  NRC	  Almost	  no	  language-­‐	  or	  domain-­‐	  speciﬁc	  knowledge	  Phrase	  table	  augmentaJon	  Paraphrases	  in	  both	  source	  &	  target	  languages	  (separately)	   Comparison	  of	  paraphrase	  collecJons	   AggregaJon	  of	  mulJple	  paths	  w/	  feature	  engineering	  Improved	  performance	  over	  a	  vanilla	  phrase-­‐based	  SMT	  at	  least	  BLEU,	  NIST,	  and	  RIBES	
0.735	

zooming	  operaJon	  0.709	

0.352	

0.228/0.136	

zoom	  operaJon	  zoom	  acJon	  zooming	  0.002/0.005	

ズーム動作
0.021/0.003	
2

MoJvaJon	  &	  proposed	  method

Modern	  SMT	  systems:	  LimitaJons! Principle	  Bilingual	  corpus

TranslaJon	  table! LimitaJons	   At	  source	  side	  Unseen	  expressions	  will	  never	  be	  translated	   They	  are	  either	  dropped	  or	  retained	  as	  is	  At	  target	  side	  Only	  seen	  expressions	  can	  be	  generated	  as	  hypotheses	   cf.	  Language	  models	  only	  ranks	  the	  given	  hypotheses	  	  

4Expressions	  that	  convey	  the	  same	  meaning! Paraphrase:	  monolingual	  Emma	  burst	  into	  tears	  and	  he	  tried	  to	  comfort	  her.
Emma	  cried,	  and	  he	  tried	  to	  console	  her.

! TranslaJon:	  cross-­‐lingual	  Désirez-­‐vous	  obtenir	  des	  conseils	  praJques	  sur	  le	  déménagement?	  
Are	  you	  looking	  for	  some	  helpful	  Jps	  for	  moving?	  

5

Paraphrases! LinguisJc	  expressions	  in	  the	  same	  language	  that	  convey	  the	  same	  meaning	  Word	  /	  word	  sequence	  burst	  into	  tears

resemblecried

look	  likeClause	  (simple	  sentence)	  The	  car	  collided	  with	  the	  bicycle
The	  car	  and	  the	  bicycle	  collided

Beyond	  single	  clause	  It	  was	  his	  best	  suit	  that	  John	  wore	  to	  the	  dance	  last	  night.
John	  wore	  his	  best	  suit	  to	  the	  dance	  last	  night.
6

Prior	  arts	  in	  integraJng	  paraphrases	  to	  MTRewrite	  input
Input[Shirai+,	  93][Doi+,	  04]	  [Xu+,	  05]	  [Nanjo+,	  12]

Post-­‐edit
OutputAugment	  input
Expand	  tuning	  data

[Onishi+,	  10]	  [Du+,	  10][Jiang+,	  11]

Augment	  translaJon	  table[Callison-­‐Burch,	  06]	  [Marton+,	  09]

Expand	  training	  corpus[Bond+,	  08][Nakov+,	  11][Madnani+,	  07]StaJsJcal	  models

Bilingual	  corpus
7AugmentaJon	  of	  translaJon	  table! Updates	  from	  [Callison-­‐burch,	  06][Marton+,	  09]	  Comparison	  of	  several	  paraphrase	  collecJons	   AggregaJon	  of	  mulJple	  paths	  (both	  sides)


Source	  side	  (Saug):	  translate	  more	  phrases	  0.735	
0.709	

zooming	  operaJon	  zoom	  acJon	  ズーム動作
0.021/0.003	

Target	  side	  (Taug):	  generate	  more	  hypotheses	  0.317/0.195	

zooming	  operaJon	  0.650	

ズーミング動作

0.146/0.108	

0.046/0.008	
0.002/0.005	

zooming	  0.352	


0.228/0.136	

zoom	  operaJon	  

ズーム操作
変倍動作

0.463	

ズーム動作
0.606	

Feature	  engineering	  for	  decoding	  8

Key	  issue:	  how	  to	  realize	  paraphrases?
! Large-­‐scale	  knowledge-­‐base	  is	  indispensable	  Handcraming	   AutomaJc	  paraphrase	  acquisiJon	  (PA)	  ! Pros.	  &	  cons.	  of	  prior	  arts	   PA	  from	  Monolingual	  non-­‐parallel	  corpora	  Pro.	  Large	   (potenJally)	  high	  recall	   Con.	  Only	  weak	  evidences	   low	  precision	  PA	  from	  Mono/Bi/MulJ-­‐lingual	  parallel	  corpora	  Pro.	  Sentence-­‐level	  equivalence	   high	  precision	   Con.	  Limited	  availability	  	  low	  recall	  

9PA	  from	  monolingual	  non-­‐parallel	  corpora! DistribuJonal	  Hypothesis	  [Harris,	  68]	  Expressions	  that	  appear	  frequently	  in	  similar	  contexts	  have	  similar	  meanings	   e.g.,	  “Tezgüno”	  [Pantel+,	  02]	  A	  boFle	  of	  tezgüno	  is	  on	  the	  table	  Everyone	  likes	  tezgüno	  Tezgüno	  makes	  you	  drunk	  We	  make	  tezgüno	  out	  of	  corn	  Similar	  to	  wine,	  cognac,	  whiskey	  	  alcoholic	  beverage	   Con.	  Not	  necessarily	  equivalent:	  e.g.,	  antonyms,	  hypernyms	  resembles	  looks	  like	  ≠tezgüno	  wine	  ≠decrease	  increase	  ≠liquor	  wine	  10

PA	  from	  bilingual	  parallel	  corpora! TranslaJons	  as	  pivot	  [Bannard+,	  05]	  A	  more	  reliable	  evidence	  than	  context	   Obtainable	  from	  bilingual	  parallel	  corpora	  

i.e.,	  word	  alignment	  +	  phrase	  extracJon	  Automa'cally	  learned	  transla'on	  table
health	  issue 	  	  health	  problem 	  regional	  issue 	  regional	  problem	  	  |||	  problème	  de	  santé	  	  |||	  problème	  de	  santé	  	  |||	  problème	  régional	  	  |||	  problème	  régional	  health	  issue	  health	  problem	  regional	  issue	  regional	  problem	  Polysemy	  would	  generate	  non-­‐paraphrases	   Con.	  Parallel	  corpora	  <<	  monolingual	  non-­‐parallel	  corpora	  

11Paraphrase	  collecJons	  examined[Fujita+,	  12]

! PSeed,	  PHvst,	  and	  POOPH

Bilingual	  corpus

Monolingual	  non-­‐parallel	  corpus

TranslaJon	  Table“health	  issue”	   	  “problème	  de	  santé”	  “health	  problem”	   	  “problème	  de	  santé”	  “look	  like”	   	  “ressemble”	  “regional	  issue”	   	  “problème	  régional”	  “regional	  problem”	   	  “problème	  régional”	  “resemble”	   	  “ressemble”	  PSeed:	  Seed	  Paraphrases“health	  issue”	   	  “health	  problem”	  “look	  like”	   	  “resemble”	  “regional	  issue”	   	  “regional	  problem”	  Paraphrase	  PaFerns“X	  issue”	   	  “X	  problem”;	  	  	  	  	  	  	  	  	  {health,	  regional,	  ...}

PHvst:	  Novel	  Paraphrases“backlog	  issue”	   	  “backlog	  problem”	  “communal	  issue”	   	  “communal	  problem”	  “phishing	  issue”	   	  “phishing	  problem”	  “spaJal	  issue”	   	  “spaJal	  problem”

12Paraphrase	  collecJons	  examined[Fujita+,	  12]

! PSeed,	  PHvst,	  and	  POOPH

Bilingual	  corpus

Monolingual	  non-­‐parallel	  corpus

TranslaJon	  Table“health	  issue”	   	  “problème	  de	  santé”	  “health	  problem”	   	  “problème	  de	  santé”	  “look	  like”	   	  “ressemble”	  “regional	  issue”	   	  “problème	  régional”	  “regional	  problem”	   	  “problème	  régional”	  “resemble”	   	  “ressemble”	  PSeed:	  Seed	  Paraphrases“health	  issue”	   	  “health	  problem”	  “look	  like”	   	  “resemble”	  “regional	  issue”	   	  “regional	  problem”	  Paraphrase	  
PaFerns
PHvst:	  Novel	  Paraphrases

POOPH:	  unseen	  phrases	  	  seen	  phrases“backlog	  issue”	   	  “backlog	  problem”	  “communal	  issue”	   	  “communal	  problem”	  “phishing	  issue”	   	  “phishing	  problem”	  “spaJal	  issue”	   	  “spaJal	  problem”

13AggregaJon	  of	  mulJple	  paths	  (1/2)! Source-­‐side	  augmentaJon	  0.735	

zooming	  operaJon	  0.709	

0.352	! TranslaJon	  scores	   Forward	  p(t|s0 ) =Backward	  p(s0 |t)

0.228/0.136	

zoom	  operaJon	  =zoom	  acJon	  0.002/0.005	

ズーム動作

zooming	  0.021/0.003	

⇣⌘

PP

s2S

Pp(t|s)Para(s0 ) s)s2S Para(s⇣0) s)0p(s|t)Para(s)s)
s2SP0Para(s)s)
s2S⌘

14AggregaJon	  of	  mulJple	  paths	  (2/2)! Target-­‐side	  augmentaJon	  0.317/0.195	

zooming	  operaJon	  0.146/0.108	

ズーム操作
変倍動作

0.046/0.008	

! TranslaJon	  scores	   Forward	  p(t0 |s) =Backward	  0.650	

ズーミング動作0p(s|t )

=

PP

t2T

⇣

P

t2T

ズーム動作
0.606	

p(t|s)Para(t ) t0 )⌘p(s|t)Para(t0 ) t)⌘0
t2T Para(t ) t )⇣P

0.463	

0Para(t) t)
t2T

15

Paraphrase-­‐related	  Features
Features	  in	  the	  translaJon	  model	

Original	

(a1)	  Forward	  translaJon	  score	(a2)	  Backward	  translaJon	  score	(b1)	  Obtained	  from	  IBM2	  alignment	(b2)	  Obtained	  from	  HMM	  alignment	(b3)	  Obtained	  from	  IBM4	  alignment	(c1)	  Fabricated	  using	  Seed	(c2)	  Fabricated	  using	  Hvst/OOPH	

Cond.Prob.	
Cond.Prob.	
True/False	(d1)	  Unseen	  in	  the	  phrase	  table	(d2)	  Unseen	  in	  the	  bilingual	  data	(e1)	  Paraphrase	  score	  (Saug/fwd)	
(e2)	  Paraphrase	  score	  (Saug/bwd)	
(e3)	  Paraphrase	  score	  (Taug/fwd)	
(e4)	  Paraphrase	  score	  (Taug/bwd)	

True/False	
True/False	
False	

Source-­‐side	  fabricated	

Target-­‐side	  fabricated	

[0,1]	
[0,1]	
False	
False	
False	

[0,1]	
[0,1]	
False	
False	
False	
True/False	

False	
False	

True/False	
True/False	
True/False	

False	

True/False	

True/False	
True/False	1	1	1	
1	[0,1]	
[0,1]	1	1	1	[0,1]	

1	[0,1]	

True/False	Score	  of	  each	  paraphrase	  pair	  (1/2)! PivProb:	  Pivot-­‐based	  paraphrase	  probability	  [Bannard+,	  05]	  For	  PSeed	  only	  stlook	  like	  	  resemble

p(s|t) p(t|s)

	  |||	  ressemble	  |||	  0.0177	  	  0.0061	  	  |||	  ressemble	  |||	  0.0074	  	  0.0181	  resemble
look	  like

Asymmetric	  score	  Para(s1 ) s2 )==p(s2 |s1 )X

t2tr (s1 )\tr (s2 )p(s2 |t)p(t|s1 )17

Score	  of	  each	  paraphrase	  pair	  (2/2)! CosSim:	  cosine	  similarity	  of	  “contexts”	  For	  all	  of	  PSeed,	  PHvst,	  and	  POOPH	  	   Contextual	  similarity	  in	  a	  monolingual	  corpus	   Adjacent	  1-­‐	  to	  4-­‐grams	  of	  each	  token	  	  feature	  vector	  cf.	  cheap	  but	  noisy	  features,	  e.g.,	  bag-­‐of-­‐words	   cf.	  accurate	  but	  expensive	  features,	  e.g.,	  dependency	  trees	  

There have been many approaches to compute the similarity
between words based on their distribution in a corpus.

L1:to
L2:approaches:to
L3:many:approaches:toL4:been:many:approaches:to

R1:between
R2:between:words
R3:between:words:based
R4:between:words:based:on18

Dev	  &	  TestOur	  base	  system! PortageII	  1.0	  [NaJonal	  Research	  Council,	  12]	  A	  state-­‐of-­‐the-­‐art	  phrase-­‐based	  SMT	  system	  Reasonably	  good	  results	  at	  NIST	  OpenMT	  2012	  [Foster,	  12]	  Advanced	  features	  (cf.	  Moses)	  Kneser-­‐Ney	  translaJon	  probability	  smoothing	  [Chen+,	  11]	   Hierarchical	  lexicalized	  reordering	  [Cherry+,	  12]	   Laxce-­‐batch-­‐MIRA	  opJmizaJon	  [Cherry	  &	  Foster,	  12]	   etc.	  
User-­‐friendly	  features	  Highly	  tuned	  libraries	  for	  using	  giganJc	  models	  [Germann+,	  09]	   High	  stability	  (cf.	  GIZA++)	   Fits	  well	  to	  cluster	  compuJng	  environment	  

20

Training	  component	  models! Provided	  data	  ! Component	  models	  Training	  bi-­‐text	  

3.2M	  sentence	  pairs	  Language	  models	  

SRILM-­‐5g	  TranslaJon	  models	  IBM2	   HMM	   IBM4	  Monolingual	  text	  Ja:	  594M	  sentences	  (27.3B	  words)	   En:	  413M	  sentences	  (13.4B	  words)	  

Lexical	  model	   Hierarchical	  lexical	  model	  Data	  for	  tuning	  2000	  sentence	  pairs	  Reordering	  models	  Paraphrase	  tables	  ! Parameter	  tuning	  21

#	  of	  learned	  phrasal	  equivalent	  pairs#	  of	  trans.	  pairs	#	  of	  paraphrase	  pairs	

Ja	  	  En	 En	   Ja	
extracJon	
IBM2	
9.4M	
9.1M	
ﬁltering	
HMM	 230.6M	 234.4M	
expansion	
IBM4	
80.6M	 81.8M	
Union	 260.4M	 264.8M	

thp	 ths	
En	
Ja	
0	 0	 7.2M	 5.1M	
PSeed	
PSeed	 0.01	 0.1	 1.1M	 0.8M	
PHvst	 0.01	 0	 272M	 143M	

dev&test data driven ﬁltering

thp	 ths	
PSeed	 0.01	 0.1	
PSeed	 0.01	 0.1	PHvst	 0.01	 0.1	

En	

Ja	

0.7M	 0.5M	
3.8M	 2.7M	
1.8M	 1.5M	

22Avg.	  BLEU	  score	  over	  held-­‐out	  data
! On	  two	  2006-­‐2007	  dev	  data	  (v7,	  v8)	  Ja	  	  En	En	   Ja	
Para	  #	  of	  trans.	  
#	  of	  trans.	  
System	score	
BLEU	
BLEU	
pairs	
pairs	-­‐	Base	  system	
18.0M	 33.30	
15.5M	 37.64	
24.6M	 37.98	 +0.34	
Saug-­‐PSeed	
PivProb	 27.3M	 33.65	 +0.35	
Cosine	Saug-­‐PSeed	
27.3M	 33.27	 -­‐0.03	
24.6M	 37.73	 +0.09	
Cosine	Saug-­‐PHvst	
23.6M	 33.22	 -­‐0.08	
22.0M	 37.89	 +0.25	
Cosine	Saug-­‐POOPH	
18.1M	 33.72	 +0.42	
15.6M	 38.16	 +0.52	
32.8M	 32.91	 -­‐0.39	
30.9M	 37.76	 +0.12	
Saug-­‐PSeed+PHvst	 Cosine	
Taug-­‐PSeed	
PivProb	 22.9M	 33.34	 +0.04	
19.6M	 37.64	 +0.00	
Taug-­‐PSeed	
Taug-­‐PHvst	
Taug-­‐POOPH	
Taug-­‐PSeed+PHvst	

Cosine	Cosine	Cosine	Cosine	22.9M	 33.56	 +0.26	
29.1M	 33.43	 +0.13	
23.4M	 33.21	 -­‐0.09	
33.9M	 32.99	 -­‐0.31	

19.6M	 38.19	 +0.55	
26.8M	 37.98	 +0.34	
21.5M	 38.08	 +0.44	
30.8M	 37.53	 -­‐0.11	
23

Oﬃcial	  results! Human	  evaluaJon	  (Saug-­‐POOPH)	  Ja	  	  En	En	   Ja	
Score	 Ranking	 Score	 Ranking	
Adequacy	
2.89/5.00	 10th/18	 2.67/5.00	 10th/14	
Acceptability	 0.43/1.00	 8th/9	 0.38/1.00	 8th/9	! AutomaJc	  evaluaJon
Ja	  	  En	System	
BLEU	 NIST	RIBES	
Saug-­‐POOPH	
31.56	 8.2507	 0.6955	
Taug-­‐PSeed	
31.65	 8.2198	 0.6929	
*Const-­‐Saug-­‐PHvst	
30.58	 8.1114	 0.6911	
*Const	  mixLM	
30.65	 8.1400	 0.6906	
*Systems	  built	  using	  only	  bilingual	  data.	

BLEU	
34.22	
34.05	
32.89	
22.59	

En	   Ja	
NIST	
RIBES	
8.2345	 0.7096	
8.2116	 0.7089	
8.0977	 0.7048	
7.1185	 0.6651	

33.03	 8.1101	 0.7051	
24

ImplicaJons! RelaJvely	  high	  BLEU	  and	  NIST	  scores	  

Useful	  n-­‐grams	  (~	  phrases)	  were	  generated	  and	  selected	  ! Low	  RIBES	  score	  and	  human	  evaluaJon	  score	   Reordering	  ability	  was	  poor	   Features	  of	  superior	  systems	  Structure-­‐aware	  SMT	   RBMT	  adapted	  to	  the	  patent	  domain	  25

We’ve	  used	  7	  for	  the	  distorJon	  limit	  ...
本/実施/形態/の/トレンチ/型/キャパシタ/１２０/を/含む/半導体/装置/の/製造/工程/の/一例/を/図/２/から/図/８/を/参照/し/て/説明/する/。	  
1	  2	  3	  4	  5	  6	  7	  8	  9	  10	  11	  12	  13	  14	  15	  16	  17	  18	  19	  20	  21	  22	  23	  24	  25	  26	  27	  28	  29	  30	  31	  	  adverbial	  phrase

main	  verb	  w/	  no	  subj.
subordinate	  clause1	  2	  3	  4	  5	  6	  7	  8	  9	  10	  11	  12	  13	  14	  15	  16	  17	  18	  19	  20	  21	  22	  23	  24	  25	  26	  27	  28	  29	  30	  31	  32	  33	  34	  35	  	  Referring	  to	  FIGS.	  2	  to	  8,	  descripJon	  will	  be	  given	  to	  an	  example	  of	  a	  manufacturing	  process	  of	  the	  semiconductor	  storage	  device	  which	  comprises	  the	  trench	  capacitor	  120	  according	  to	  the	  embodiment.	  
26

RelaxaJon	  of	  distorJon	  limit! Held-­‐out	  data	  same	  as	  development	  Obtained	  signiﬁcantly	  higher	  score	   PosiJve	  impact	  led	  by	  paraphrases	  was	  retained
0.355
0.35
0.345
0.34
Base system
Saug-OOPH
Taug-Seed

0.335
0.33
60.74

8101214Ja
En
Distortion limit

1618

20

0.73
0.72
0.71

Base system
Saug-OOPH
Taug-Seed

0.7
68101214
Distortion limit

161820BLEU (English to Japanese)

0.36

0.405

RIBES (English to Japanese)

RIBES (Japanese to English)

BLEU (Japanese to English)



0.77

0.4
0.395
0.390.385
Base system
Saug-OOPH
Taug-Seed

0.38
0.375
68101214En Jalimit
Distortion

161820

0.76
0.75
0.74
Base system
Saug-OOPH
Taug-Seed

0.73
0.72
68101214
Distortion limit

161820Conclusion! Phrase-­‐based	  SMT	  +	  paraphrases	  State-­‐of-­‐the-­‐art	  non-­‐hierarchical	  system:	  PortageII	  @	  NRC	  Almost	  no	  language-­‐	  or	  domain-­‐	  speciﬁc	  knowledge	  Phrase	  table	  augmentaJon	  Paraphrases	  in	  both	  source	  &	  target	  languages	  (separately)	   Comparison	  of	  paraphrase	  collecJons	   AggregaJon	  of	  mulJple	  paths	  w/	  feature	  engineering	  Improved	  performance	  over	  a	  vanilla	  phrase-­‐based	  SMT	  at	  least	  BLEU,	  NIST,	  and	  RIBES	
0.735	

zooming	  operaJon	  0.709	

0.352	

0.228/0.136	

zoom	  operaJon	  zoom	  acJon	  zooming	  0.002/0.005	

ズーム動作
0.021/0.003	
28

Greatest	  thanks	  go	  to! Supporters	  of	  the	  research	  program	  

NRC:	  NaJonal	  Research	  Council	  Canada	  esp.	  All	  members	  in	  the	  Portage	  team	  FUN:	  Future	  University	  Hakodate	   JSPS:	  Japan	  Society	  for	  the	  PromoJon	  of	  Science	  ! PatentMT	  task	  organizers	  29