Overview	  of	  WAT2014	
Toshiaki Nakazawa
nakazawa@pa.jst.jp

Hideya Minohideya.mino@nict.go.jp

Sadao Kurohashi

Eiichiro Sumita

kuro@i.kyoto-u.ac.jp

eiichiro.sumita@nict.go.jp

HUMAN  score

Participants  List
Team	  ID	 Organiza8on	
J-­‐>E	
E-­‐>J	
J-­‐>C	C-­‐>J	
NAIST	Nara	  Ins+tute	  of	  Science	  and	  Technology	✓	 ✓	 ✓	 ✓	EIWA	
Yamanashi	  Eiwa	  College	✓	✓	Kyoto-­‐U	Kyoto	  University	✓	 ✓	 ✓	 ✓	
WEBLIO-­‐EJ1	
Weblio,	  Inc.	
✓	TMU	
Tokyo	  Metropolitan	  University	✓	BJTUNLP	 Beijing	  Jiaotong	  University	✓	NII	Na+onal	  Ins+tute	  of	  Informa+cs	✓	SAS_MT	 SAS	  Research	  and	  Development	  Co.,	  Ltd	✓	✓	Sense	
Saarland	  University	  &	  Nanyang	  Technological	  University	✓	 ✓	 ✓	 ✓	
NICT	
Na+onal	  Ins+tute	  of	  Informa+on	  and	  Communica+on	  Technology	
✓	TOSHIBA	 Toshiba	  Corpora+on	✓	✓	
WASUIPS	 Waseda	  University	✓*	 ✓*	company	 outside	  Japan	

Isao Goto

goto.i-es@nhk.or.jp*	  To	  guarantee	  the	  quality	  of	  the	  evalua+on,	  Wins − Losses
HUMAN = 100 ×each	  pair	  is	  evaluated	  3	  diﬀerent	  workers	  Wins + Losses + Ties
*	  The	  evalua+on	  result	  of	  each	  pair	  is	  decided	  by	  the	  vo+ng	  of	  3	  judgments	  Cost

hUp://www.lancers.jp	

*	  Pairwise	  evalua+on	  compared	  to	  the	  baseline	  using	  crowdsourcing	  	  -­‐	  reduce	  the	  cost,	  +me,	  number	  of	  sentences	  to	  be	  evaluated	  	  -­‐	  enable	  the	  evalua+on	  of	  new	  transla+on	  results	  aVer	  the	  workshop	  one	  judgment	  by	  a	  worker	  =	  5	  JPY	  ×	  3	  judgments	  for	  one	  decision	  ×	  400	  sentences	  ||	  6000	  JPY	  /	  submission	*	  Reference	  transla+ons	  are	  not	  shown	  *	  400	  sentences	  are	  randomly	  selected	  (paragraph-­‐based)	  from	  the	  test	  set	  Time
Depending	  on	  the	  transla+on	  direc+on	  On	  the	  average,	  one	  evalua+on	  ﬁnished	  in	  a	  couple	  of	  days	*	  Only	  submiUed	  to	  the	  automa+c	  evalua+ons	

Manual  (HUMAN)  and  Automatic  (BLEU)  Evaluation  Results  and  Their  Correlations

Case  Study

Inter  Annotator  Agreement
JE	

EJ	

JC	

CJ	

System	  ID	 Kappa	
NAIST	  1	 0.162	
NAIST	  2	 0.047	
SMT	  S2T	 0.099	
Kyoto-­‐U	  1	 0.070	
TOSHIBA	  1	 0.098	
RBMT	  D	 0.075	
EIWA	
0.083	
Kyoto-­‐U	  2	 0.139	
TOSHIBA	  2	 0.078	
Online	  D	 0.055	
SMT	  Hiero	 0.119	
Sense	
0.245	NII	  1	0.119	
NII	  2	0.086	
TMU	  1	0.091	
TMU	  2	0.136	

System	  ID	Kappa	NAIST	  1	0.280	
NAIST	  2	0.250	
WEBLIO-­‐EJ1	  1	 0.238	
Online	  A	0.219	
Kyoto-­‐U	  1	0.216	
WEBLIO-­‐EJ1	  2	 0.240	
SMT	  T2S	0.240	
Kyoto-­‐U	  2	0.229	
SMT	  Hiero	
0.277	
SAS_MT	
0.248	
Sense	
0.395	
RBMT	  B	0.217	

System	  ID	 Kappa	
NAIST	  1	 0.077	
SMT	  S2T	 0.069	
Sense	
0.087	NICT	0.066	SMT	  Hiero	 0.202	
NAIST	  2	 0.093	
TOSHIBA	  1	 0.089	
Kyoto-­‐U	  1	 0.091	
BJTUNLP	 0.198	
TOSHIBA	  2	 0.066	
Kyoto-­‐U	  2	 0.163	
Online	  D	 0.035	
RBMT	  B	 0.083	

System	  ID	 Kappa	
NAIST	  1	 0.168	
NAIST	  2	 0.203	
SAS_MT	 0.167	
SMT	  T2S	 0.236	
EIWA	
0.175	
Kyoto-­‐U	  1	 0.199	
Kyoto-­‐U	  2	 0.180	
SMT	  Hiero	 0.274	
Sense	
0.228	
Online	  A	 0.239	
RBMT	  A	 0.130	

Ave.	

Ave.	 0.106	

Ave.	 0.200	
0.254	
Ave.	 0.101	
*	  Fleiss’s	  Kappa	  value	

X-­‐>J	  evalua+ons	  are	  easier	  than	  J-­‐>X	  evalua+ons	  (because	  the	  workers	  are	  almost	  Japanese?)	  Submission	  ID	 BLEU	 RIBES	 HUMAN	
Forest	  Input	
WEBLIO-­‐EJ1	  1	 32.53	 0.782	 43.25	
No	WEBLIO-­‐EJ1	  2	 32.69	 0.785	 36.00	
Yes	

BASELINE	  (WEBLIO-­‐EJ1	  1)	WEBLIO-­‐EJ1	  2	HUMAN	 Kappa	
2.50	  ±	  4.17	
0.528	
*	  No	  signiﬁcant	  diﬀerence	  *	  Much	  higher	  Kappa	  value	  similar	  outputs	  can	  be	  easily	  and	  faithfully	  judged	

Conclusion  and  Future  Perspective• 12	  par+cipants	  for	  the	  evalua+on	  task	  • Automa+c	  evalua+on	  server	  will	  keep	  running	  even	  aVer	  the	  	  	  	  	  	  	  	  	  	  -­‐	  including	  3	  companies	  and	  3	  outside	  Japan	   workshop	  to	  promote	  con+nuous	  evolu+on	  of	  MT	  research	  	  • NAIST	  team	  achieved	  the	  best	  results	  for	  	  	  	  	  	  	  	  	  	  	  	  	  show:	  	  hUp://lotus.kuee.kyoto-­‐u.ac.jp/WAT/evalua+on/	  all	  the	  subtasks	  	  	  	  	  	  	  	  	  	  submit:	  	  hUp://lotus.kuee.kyoto-­‐u.ac.jp/WAT/submission/	  • Shared	  the	  ﬁndings	  of	  MT	  for	  scien+ﬁc	  papers	   	  	  	  	  	  	  	  	  	  	  	  	  KEEP	  ON	  UPDATING	  THE	  STATUS	  OF	  YOUR	  MT	  ENGINE!!	  
• WAT	  will	  be	  held	  annually	  • Need	  more	  inves+ga+on	  to	  acquire	  reliable	   • The	  importance	  of	  both	  sub/super	  senten+al	  context	  human	  evalua+on	  results	  at	  low	  cost	  • Let’s	  share	  your	  resource	  • Need	  to	  ﬁnd	  a	  beUer	  way	  to	  compare	  two	  	  	  	  	  	  monolingual/bilingual	  corpora,	  dic+onaries	  and	  so	  on	  systems	  eﬃciently	  and	  reliably	  Thank	  you	  very	  much	  for	  aUending	  WAT2014	  Stay	  tuned	  for	  the	  next	  WAT	  workshop!	
WAT2014, Tokyo Japan, 4th October 2014