Translation Using JAPIO Patent Corpora:
JAPIO at WAT2016
Satoshi Kinoshita, Tadaaki Oshio,
Tomoharu Mitsuhashi, Terumasa Ehara

Questions
1. Can PB-SMT with large corpora generatebetter translations than systems with anadvanced framework such as NMT?
2. Are large patent corpora useful fortranslation of scientific domain?

JAPIO Patent Corpora
・Parallel corpora of EJ/CJ/KJ, which areautomatically extracted from patent families
using an alignment tool by NICT・Corpus size:
EJ: 250 million
CJ: 100 million
KJ: 5 million

・Purpose- Train SMT- Search by human translators- Build translation dictionaries

Systems
・NICT PB-SMT toolkit for EJ & CJ (Preorderingincluded)
Moses for KJ (character-based tokenizer)
・Additional post-editing functions:
ASPEC-EJ/CJ: modify punctuations and patent
specific expressions
EJ: Recapitalization of OOVs
KJ: Resolving unbalanced parentheses

Results (#1)
#Subtask

1

SystemCorpus

Size(million)
5

BLEU

RIEBS

AMFM

HUMAN

45.57

0.851376 0.747910

17.750JAPIO-aJAPIO-test

JAPIO-b

JAPIO-test+JPC

647.790.859139 0.762850

26.750JAPIO-cJAPIO

550.280.859957 0.768690－4JAPIO-d

JPC

138.590.839141 0.733020－

5JAPIO-aJAPIO-test

343.870.833586 0.748330

43.500JAPIO-b

JAPIO-test

444.320.834959 0.751200

46.250JAPIO-cJAPIO

4958.66

0.868027 0.808090－

8JAPIO-d

JPC

139.290.820339 0.733300－

9JAPIO-a

JAPIO

568.620.938474 0.858190-9.000JAPIO-b

JAPIO+JPC

670.32

0.942137 0.863660

17.500JAPIO-c

JPC

169.100.940367 0.859790－

23671011

JPC-EJ

JPC-CJ

JPC-KJ

・SMTs with larger corpora are competitive but notsuperior than state-of-the-art systems like NMTs trainedwith 1M JPC corpora.

Results (#2)

SMT x

NMT y

NMT x

SMT y

SMT x
*Figures on this page are provided by Organizer.

Results (#3)
#Subtask

SystemCorpus
JAPIO

Size(million)
10

BLEU

RIEBS

AMFM

HUMAN

20.52

0.723467 0.660790

4.250

12JAPIO-a

13

ASPEC-EJ Online x

－－

18.28

0.706639 0.677020

49.75014

RBMT x－－13.180.671958－

15JAPIO-a4926.24

0.790553 0.696770

16.50016ASPEC-CJ Online x－－

11.56

0.589802 0.659540-51.250

17RBMT x－－

19.24

0.741665－JAPIO－－・SMTs with Patent Corpora are better than RBMT andOnline (except AMFM and HUMAN in EJ)
・SMTs with Patent Corpora could be used practically fortranslation in scientific-paper domain

Error Analysis
・Error analysis of patent subtask (#sent=200)
Error Type

EJCJ

KJ

Insertion

006Deletion

491

OOV

692

Mistranslation(content word)

444130

Mistranslation(functional word)

21510Pre-ordering

33

450Other672Total114

162

41

# of mistranslation of functional words and errors
related to pre-ordering suggests improvement of preordering is essential.

Conclusion
1. Can PB-SMT with large corpora generate bettertranslations than systems with an advanced
framework such as NMT?
→ No
Improvement of pre-ordering seems more effective thanincreasing corpus

2. Are large patent corpora useful for translationof scientific domain?
→ Yes
Could be used without domain adaptation

Suggestion
Suggestions to Workshop Organizer:
To make results of WAT more useful,・Make a new test set without using patentfamilies

If impossible, remove training examples which are extracted
from the same documents as test set examples are extracted.

・Balance sentence length of test sets of JPC-EJ/CJ- Sentences of EJ test set are much shorter
than CJ- Current results may mislead readers