Overview	  of	  the	  2nd	  Workshop	  on	  Asian	  Transla7on	
Toshiaki	  Nakazawa	  	  	  Hideya	  Mino	  	  	  	  	  	  	  	  	  	  Isao	  Goto	  	  Graham	  Neubig	  	  	  	  	  	  	  	  	  Sadao	  Kurohashi	  	  	  Eiichiro	  Sumita	  	WAT2015,	  16	  October,	  2015,	  Kyoto	The	  Features	  of	  WAT2015	
• MT	  evalua7on	  campaign	  focusing	  on	  Asian	  languages	  (Japanese,	  Chinese,	  Korean	  and	  English	  for	  this	  7me)	  • The	  ﬁrst	  evalua7on	  for	  Chinese-­‐Japanese	  and	  Korean-­‐Japanese	  patent	  transla7on.	  	  • Professional	  translators	  evaluate	  the	  outputs	  of	  the	  top	  systems	  based	  on	  JPO	  adequacy.	  	  1	Comparison	  of	  WAT	  2014	  and	  2015	
WAT2014	

Task	

Scien7ﬁc	  paper	  (ASPEC)	  • Japanese	  ↔	  English	  • Chinese	  ↔	  Japanese	  WAT2015	
Scien7ﬁc	  paper	  (ASPEC)	  • Japanese	  ↔	  English	  • Chinese	  ↔	  Japanese	  JPO	  Patent	  corpus	  (JPC)	  • Chinese	  →	  Japanese	  • Korean	  →	  Japanese	  New	

Automa7c	  Evalua7on	  Automa7c	  Evalua7on	  Human	  Evalua7on	  Human	  Evalua7on	  Evalua7on	• Pairwise	  Crowdsourcing	 • Pairwise	  Crowdsourcing	  • JPO	  Adequacy	  (Top	  3	  systems)	
Number	  of	  par7cipants	

12	12	New	

2	

Notable	  Findings	  at	  WAT2015	
• Neural	   Network	   based	   re-­‐ranking	   was	   eﬀec7ve	  for	  human	  evalua7on	  (NAIST,	  Kyoto-­‐U,	  naver)	  • The	   top	   SMT	   outperformed	   RBMT	   for	   Chinese-­‐Japanese	   and	   Korean-­‐Japanese	   patent	  transla7on	  • Korean-­‐Japanese	   patent	   transla7on	   achieved	  high	   scores	   for	   automa7c	   and	   human	  evalua7ons.	  	  • A	  problem	  of	  automaOc	  evaluaOon	  was	  found	  in	  the	  Korean-­‐Japanese	  transla7on.	  	
3	

Par7cipants	
company	

outside	  Japan	

ASPEC	JPC	JE	 EJ	 JC	 CJ	 CJ	 KJ	Team	  ID	

OrganizaOon	

NAIST	

Nara	  Ins7tute	  of	  Science	  and	  Technology	✓	 ✓ ✓ ✓Kyoto-­‐U	

Kyoto	  University	✓	 ✓ ✓ ✓ ✓WEBLIO_MT	 Weblio,	  Inc.	

✓TMU	

Tokyo	  Metropolitan	  University	

BJTUNLP	

Beijing	  Jiaotong	  University	

Sense	

Saarland	  University	  &	  Nanyang	  Technological	  University	 ✓	 ✓

✓

NICT	

Na7onal	  Ins7tute	  of	  Informa7on	  and	  Communica7on	  Technology	✓	✓

TOSHIBA	

Toshiba	  Corpora7on	✓	 ✓ ✓ ✓ ✓ ✓WASUIPS	

Waseda	  University	naver	

NAVER	  Corpora7on	

✓	✓

EHR	

Ehara	  NLP	  Research	  Laboratory	✓	 ✓✓	 ✓	nb	NTT	  Communica7on	  Science	  Laboratories	

✓	✓	✓

4		✓

Baseline	  Systems	
System	  ID	

System	Type	ASPEC	

JE	 EJ	 JC	 CJ	 CJ	 KJ	

✓	SMT	  Phrase	
Moses'	  Phrase-­‐based	  SMT	✓	SMT	  Hiero	 Moses'	  Hierarchical	  Phrase-­‐based	  SMT	SMT	✓	SMT	  S2T	 Moses'	  String-­‐to-­‐Tree	  Syntax-­‐based	  SMT	  and	  Berkeley	  parser		
SMT	  T2S	 Moses'	  Tree-­‐to-­‐String	  Syntax-­‐based	  SMT	  and	  Berkeley	  parser	
✓	RBMT	  X	 The	  Honyaku	  V15	  (Commercial	  system)	
✓	
RBMT	  X	 ATLAS	  V14	  (Commercial	  system)	
✓	
RBMT	  X	 PAT-­‐Transer	  2009	  (Commercial	  system)	
RBMT	  X	 J-­‐Beijing	  7	  (Commercial	  system)	
RBMT	 		RBMT	  X	 Hohrai	  2011	  (Commercial	  system)		
RBMT	  X	 J	  Soul	  9	  (Commercial	  system)		
RBMT	  X	 Korai	  2011	  (Commercial	  system)	
Online	  X	 Google	  translate	  (August,	  2015)	
(SMT)	 ✓	Online	  X	 Bing	  translator	  (August	  and	  September,	  2015)	
(SMT)	 ✓	•••

JPC	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓		✓				✓		✓	✓		✓					✓					✓						✓	✓	✓			✓	✓	✓						✓					✓	✓	✓	✓	✓	✓	

✓	✓	

✓	

✓	

✓	4	  types	  of	  SMT	  systems,	  7	  RBMT	  systems,	  and	  2	  online	  systems	  The	  SYSTEM-­‐IDs	  of	  the	  commercial	  RBMT	  and	  online	  systems	  are	  anonymized.	  
5	The	  transla7on	  procedures	  for	  the	  SMT	  systems	  were	  published	  on	  the	  WAT	  web	  site.	  Evalua7on	  Methods	

Automa7c	  Evalua7on	
• BLEU,	  RIBES	  • The	  automa7c	  evalua7on	  server	  for	  WAT	  accepts	  transla7ons	  any	  7me.	  

7	Pairwise	  Crowdsourcing	  Evalua7on	  	• Sentence-­‐level	  evalua7on	  comparing	  to	  the	  baseline	  Phrase-­‐based	  SMT	  output	  System	  output	
••••VS	Baseline	  output	

The	  Lancers	  crowdsourcing	  plajorm	  (the	  same	  as	  that	  at	  WAT2014)	  Evaluators	  judge	  win	  (1),	  loss	  (-­‐1),	  or	  7e	  (0)	5	  evaluators	  assessed	  for	  each	  sentence.	  
The	  ﬁnal	  judgment	  for	  each	  sentence	  is	  decided	  by	  vo7ng	  based	  on	  the	  sum	  of	  judgments:	  	  – Win:	  sum	  ≧	  2	  – Loss:	  	  sum	  ≦	  -­‐2	  – Tie:	  otherwise	This	  vo7ng	  criterion	  is	  diﬀerent	  from	  that	  of	  WAT2014.	  	  The	  Crowd	  scores	  between	  WAT2014	  and	  WAT2015	  cannot	  be	  compared.	  	
8	Crowd	  score	400	  sentences	  were	  evaluated	  for	  each	  system.	  	

W −L
Crowd	  score	  =	100 ×W + L +T
(The	  score	  range	  is	  -­‐100	  to	  100)	
W:	  the	  number	  of	  wins	  compared	  to	  the	  baseline	  L:	  	  	  the	  number	  of	  losses	  compared	  to	  the	  baseline	  T:	  	  the	  number	  of	  7es	9	

JPO	  Adequacy	• Sentence-­‐level	  5-­‐scale	  criterion	  deﬁned	  by	  Japan	  Patent	  Oﬃce	
5	  All	  important	  informa7on	  is	  transmibed	  correctly.	  (100%)	  4	  Almost	  all	  important	  informa7on	  is	  transmibed	  correctly.	  (80%〜)	  3	  More	  than	  half	  of	  important	  informa7on	  is	  transmibed	  correctly.	  
(50%〜)	  2	  Some	  of	  important	  informa7on	  is	  transmibed	  correctly.	  (20%〜)	  1	  Almost	  no	  important	  informa7on	  is	  transmibed	  correctly.	  (〜20%)	  ••••Professional	  translators	  assessed.	  
Top	  3	  systems	  of	  the	  Crowd	  scores	  were	  evaluated.	  
One	  sentence	  was	  evaluated	  by	  two	  evaluators.	  
200	  sentences	  were	  evaluated	  for	  each	  system.	  

10	Crowd	  Evalua7on	  Results	     

 



 
 

  
 







  

 










Baseline	

Many	  par7cipants	  outperformed	  SMT	  S2T,	  which	  ranked	  at	  the	  second	  at	  WAT2014.	  	
12	     


$ 



# 



" 



"   !
  ! " 

"
 
"
   !
  "
  !
 
 "  " 
!
% 



# 

$ 

Baselines	

Many	  par7cipants	  outperformed	  the	  baselines	  of	  Online	  A	  and	  SMT	  T2S	  	13	     


Baseline	

Two	  teams	  outperformed	  SMT	  S2T,	  which	  ranked	  at	  the	  second	  at	  WAT2014.	  	  14	 	  NAIST	  1	  ranked	  at	  middle.	  However	  this	  ranking	  was	  diﬀerent	  from	  JPO	  adequacy.	       


Baseline	


NAIST	  team	  achieved	  the	  best	  score.	  	

15	      

Baseline	  RBMT	


SMT	  systems	  outperformed	  the	  baseline	  RBMT	  for	  CJ	  patent	  transla7on.	

16	            



Baseline	  RBMT	

SMT	  systems	  outperformed	  the	  baseline	  RBMT	  for	  KJ	  patent	  transla7on.	  	
17	 	  Online	  A	  was	  top-­‐ranked.	  However	  this	  rank	  was	  diﬀerent	  from	  JPO	  adequacy.	  JPO	  Adequacy	  Evalua7on	  Results	  and	  Reliability	  of	  Crowd/BLEU/RIBES	

Comparison	  with	  JPO	  Adequacy	  (ASPEC-­‐JE)	

 

"'$

"'"

    

&#(%

&!#%

 & '$'"'&"&$ %

 "&"%$"%

"$$
!$# $"%$"!$!!'(#"%
!$
!!'(""$$

!"$

"#$  !   !

The	  ranking	  of	  Crowd	  is	  consistent.	  	  The	  ranking	  of	  BLEU	  is	  par7ally	  consistent.	  
The	  ranking	  of	  RIBES	  is	  consistent.	  	  	

19	Comparison	  with	  JPO	  Adequacy	  (ASPEC-­‐EJ)	
$! $ % $    (!!%$ $( $&""%#)%

%#"%

 ( $'( '  %#'%#) #(%) & % #%(##$& #(! ##"#$ #( # #'%" #' ! #&% !

"     !

The	  ranking	  of	  Crowd	  is	  par7ally	  consistent.	  	  The	  ranking	  of	  BLEU	  is	  consistent.	  
The	  ranking	  of	  RIBES	  is	  par7ally	  consistent.	  	  	

20	Comparison	  with	  JPO	  Adequacy	  (ASPEC-­‐JC)	
""
"!

    '"!''& " & " "" % !&!%

(' & %$!'&

!(!' ' &"!(((!&$ "&&%
&

#"   

!


!$   !

Diﬀerent	  ranking	   The	  ranking	  of	  Crowd	  is	  not	  consistent.	  	  JC	  is	  diﬃcult	  to	  The	  ranking	  of	  BLEU	  is	  par7ally	  consistent.	  
from	  Crowd	
21	evaluate!	
The	  ranking	  of	  RIBES	  is	  par7ally	  consistent.	  	  	Comparison	  with	  JPO	  Adequacy	  (ASPEC-­‐CJ)	
"
!'

!&&

    

&##

!&

&"%

!%"

 !%
!$
!#
!"

"%#
!#%#

'& %

 $#!&#!

!%'
! #
#%#

!!! & $&#

"
!


!
!

'      The	  ranking	  of	  Crowd	  scores	  is	  par7ally	  consistent.	  	  The	  ranking	  of	  BLEU	  scores	  is	  consistent.	  
The	  ranking	  of	  RIBES	  scores	  is	  consistent.	  	  	

22	Comparison	  with	  JPO	  Adequacy	  (JPC-­‐CJ)	
     & &#

!"

& %

 !!#! #

!"!!



& %#"$

& %

 $!! "!#'" ! #" #

#
"
!

   !"#! !#   The	  ranking	  of	  Crowd	  was	  par7ally	  consistent.	  	  The	  ranking	  of	  BLEU	  was	  par7ally	  consistent.	  
The	  ranking	  of	  RIBES	  was	  consistent.	  	  	

23	Comparison	  with	  JPO	  Adequacy	  (JPC-­‐KJ)	
   %)!

%(!

%()
("$)

*%$&

*!*#

%('(!)"

%'!

&&!&%&$

%&!

$)(&

%%!
%$!

*&%& "!!
)&#$

  )!'!%!

%$#
"%(&

#!"!&!

%#!

    *%$)%*!

The	  highest	  automaOc	  scores	  	!%"!

"((&Not	  high	  human	  scores	
#!

%!!
# #"

The	  Crowd	  score	  of	  Online	  A	  is	  very	  high.	  	  The	  best	  BLEU	  and	  RIBES	  scores.	  	
JPO	  adequacy	  of	  Online	  A	  was	  not	  top-­‐ranked.	
However,	  human	  evalua7on	  was	  not	  24	high.	  	
There	  might	  be	  a	  remaining	  problem	  in	  automa7c	  evalua7on.	  Comparison	  of	  Transla7on	  Quality	  	  between	  Languages	  	Graph	  Format	  of	  JPO	  Adequacy	

26	

ASPEC-­‐JE	ASPEC-­‐CJ	ASPEC-­‐EJ	ASPEC-­‐JC	The	  second	  high	  quality	  Low	  quality	  	  (Diﬃcult	  language	  pair)	

JPC-­‐CJ	

JPC-­‐KJ	

High	  quality	  was	  achieved!	  
27	

Summary	  of	  WAT2015	• 12	  par7cipants	  for	  the	  evalua7on	  tasks	  – Including	  4	  companies	  and	  3	  teams	  outside	  Japan	  • 2	  domains	  (scien7ﬁc	  paper	  and	  patent)	  and	  4	  languages	  (Japanese,	  Chinese,	  Korean,	  and	  English)	  	  • Human	  evalua7ons	  were	  performed	  – Pairwise	  Evalua7on	  by	  crowdsourcing	  – JPO	  Adequacy	  by	  professional	  translators	  • Empirically	  conﬁrmed	  that	  MT	  systems	  achieved	  high	  quality	  Korean-­‐Japanese	  patent	  transla7on.	  	  • Each	  idea	  used	  for	  the	  submissions	  will	  be	  presented	  by	  the	  par7cipants.	  
28	

Future	  Perspec7ve	
• Papers	  submibed	  to	  WAT	  will	  appear	  on	  the	  ACL	  Anthology.	  	  • Automa7c	  evalua7on	  server	  will	  keep	  running	  even	  aser	  the	  workshop	  – promote	  con7nuous	  evolu7on	  of	  MT	  research	  • WAT	  will	  be	  held	  annually	  – include	  more	  languages,	  domains...	  

• Need	  more	  inves7ga7on	  to	  acquire	  reliable	  human	  evaluaIon	  results	  at	  low	  cost	29	

Thank	  you	  very	  much	  	  for	  abending	  WAT2015