Translation of Patent Sentenceswith a Large Vocabularyof Technical Terms
Using Neural Machine Translation
Zi Long, Takehito Utsuro
University of Tsukuba

Tomoharu Mitsuhashi
Japan Patent Information Organization

Mikio Yamamoto
University of Tsukuba
WAT2016, December 12, 2016 @ Osaka, Japan1

Vinyals et al. Grammar as a foreign
Neural Machine Translation
language. In Proc. NIPS, 2015(encoder-decoder Model and attention mechanism)(as shown in Figure … )

input

図に示す

……

。

<EOS>

……
encoderA large vector that
represents the entire
input sentenceVinyals et al. Grammar as a foreign
Neural Machine Translation
language. In Proc. NIPS, 2015(encoder-decoder Model and attention mechanism)(as shown in Figure … )

input

図に示す

……

。

<EOS>

。

<EOS>

……
encoderdecoder……
output如图所示……

Neural Machine Translation VS
Statistic Machine Translation
Neural MachineTranslation (NMT)

Statistic MachineTranslation (SMT)

fluencyhighlow

vocabulary

smalllarge

Neural Machine Translation VS
Statistic Machine Translation
Neural Machine(NMT)
Luong et al. AddressingTranslation
the Rare

Statistic MachineTranslation (SMT)

Word Problem in Neural MachineTranslation. In Proc. ACL, 2015

• word-level
fluency for• inappropriate
translating technical termshigh• phrase-level• store explicitlow
phrasetranslation tablevocabulary

smalllarge

NMT with a Large Vocabularyof Technical Terms
 Step 1. training NMT model with technicalterm tokens

 Step 2. applying NMT model with technicalterm tokens

Approach 1. NMT decoding and SMT technicalterm translation
Approach 2. NMT rescoring of 1,000-best SMT
translations (not as fluent as Approach 1)
6NMT with a Large Vocabularyof Technical Terms
 Step 1. training NMT model with technicalterm tokens

 Step 2. applying NMT model with technicalterm tokens

Approach 1. NMT decoding and SMT technicalterm translation
Approach 2. NMT rescoring of 1,000-best SMT
translations (not as fluent as Approach 1)
7

NMT training after replacing TechnicalTerms with Tokens
aligned patent sentence pairsaligned patentsentence pairs
SMT

Japanese sentence:

Chinese sentence:

cmac/ユニット/312/は/信号/を/ブリッジ/インタフェース/388/に/提供/する/。

cmac/单元/312/将/信号/提供/给/桥架/接口/388/。

(cmac unit 312 provides a signal to the bridge interface 388.)

SMT translation modelphrasewordtranslationalignment
table

1. aligning technicalterm pairs using SMT
phrase translation table
and word alignment
8

NMT training after replacing TechnicalTerms with Tokens
aligned patent sentence pairsaligned patentsentence pairs
SMT

Japanese sentence:

Chinese sentence:

cmac/ユニット/312/は/信号/を/ブリッジ/インタフェース/388/に/提供/する/。

cmac/单元/312/将/信号/提供/给/桥架/接口/388/。

(cmac unit 312 provides a signal to the bridge interface 388.)

SMT translation model

2. replacing each aligned technical term pair withan identical technical term token “TTi” (i = 1, 2, …)

phrasewordtranslationalignment
tablealigned patent sentence pairs(with technical term tokens)
Japanese sentence with
Chinese sentence withtechnical term tokens
technical term tokens“TT1”, “TT2” :“TT1”, “TT2” :TT1 /312/は/信号/を/TT2
TT1 /312/将/信号/提供/给/TT2 /388/。
/388/に/提供/する/。
(TT1 312 provides a signal to the TT2 388.) 9

NMT training after replacing TechnicalTerms with Tokens
aligned patent sentence pairsaligned patentsentence pairs
SMTJapanese sentence:

Chinese sentence:

cmac/ユニット/312/は/信号/を/ブリッジ/インタフェース/388/に/提供/する/。

cmac/单元/312/将/信号/提供/给/桥架/接口/388/。

(cmac unit 312 provides a signal to the bridge interface 388.)

SMT translation modelphrasewordtranslationalignment
table

NMT translation
NMT
model (with technicalterm tokens)aligned patent sentence pairs(with technical term tokens)
Japanese sentence with
Chinese sentence withtechnical term tokens
technical term tokens“TT1”, “TT2” :“TT1”, “TT2” :TT1 /312/は/信号/を/TT2
TT1 /312/将/信号/提供/给/TT2 /388/。
/388/に/提供/する/。
(TT1 312 provides a signal to the TT2 388.) 10

NMT with a Large Vocabularyof Technical Terms
 Step 1. training NMT model with technicalterm tokens

 Step 2. applying NMT model with technicalterm tokens

Approach 1. NMT decoding and SMT technicalterm translation
Approach 2. NMT rescoring of 1,000-best SMT
translations (not as fluent as Approach 1)
11

NMT Decoding and SMT technical Term Translation
extractedJapanesetechnical terms

inputJapanesesentence

replacingtechnicalterms withtechnicalterm tokens

decoding by NMTtranslation model (withtechnical term tokens)

phrasetranslation
table

NMT
translation
model (withtechnicalterm tokens)

Chinesetranslations oftechnical terms

outputChinesetranslation(withtechnicalterm tokens)replacingtechnical
term tokens
with technicaltermtranslation
by SMT

output Chinesetranslation
12

Training and Test Sets
• 2.8M parallel sentences extracted from JapaneseChinese patent families•

Randomly selected 1,000 sentence pairs as the test set
pair 1：

J： ・・・冷蔵庫および冷蔵庫扉閉鎖装置・・・
C: ・・・电冰箱及电冰箱门锁闭装置・・・。
(closing appliance of Refrigerator's door)

pair 2：

J: ・・・真空断熱材及びその製造方法・・・．
C: ・・・真空绝热材料及其制作方法・・・。
・・・・・・

pair n：(vacuum thermal insulation)

J: ・・・運動に関する3次元デカルト座標系を定める。
C: ・・・以限定用于描述运动的三维笛卡尔坐标系・・・。
(Cartesian coordinate system)

13

Experiments Settings
 Baseline SMT (PBMT) phrase-based SMT model trained with the same

training set using Moses.

 Baseline NMT
 uni-direction model with attention mechanism

 3 layer deep LSTMs with 512 cells in each layer and a512-dimensional word embedding.
 limit both the Japanese vocabulary and the Chinese
vocabulary to 40,000 most frequently used word
 more training details.

14Experiments Settings
 NMT with PosUnk model [Luong+ 2015]
 same training parameters with the baseline NMT training NMT model with PosUnk model NMT with technical term tokens
 same training parameters with the baseline NMT training NMT model after replacing technical termswith tokens.

150

Evaluation Results(automatic evaluation - BLEU)• BLEU score

55.3

50

50

53.5

50554.052.5

050

Baseline
SMT(PBMT)

Baseline
NMTNMT with
PosUnk model[Luong+ 2015]

NMT withtechnical term
tokens 16

0Evaluation Results(automatic evaluation - RIBES)• RIBES score

90.8
90.4

590.0

0

50

50

88.5

Baseline
SMT(PBMT)

Baseline
NMT

NMT with
PosUnk model[Luong+ 2015]

NMT withtechnical term
tokens 17

Evaluation Results(human evaluation – pairwise evaluation)• Pairwise Evaluation (scores range from -100 to 100)
40.0

36.5

35.0
30.0
25.0

20.0
15.0
10.0
5.0

5.0

0.0

Baseline
NMT

NMT withtechnical term
tokens18

Evaluation Results(human evaluation – JPO adequacy evaluation)• JPO adequacy evaluation (scores range from 1 to 5)
4.5
4.3

3.8

4.0
3.5

3.5

3.0
2.5
2.0

Baseline
SMT(PBMT)

Baseline
NMT

NMT withtechnical term
tokens19

Example of correct translations produced
by NMT decoding
NMT decoding
by the PROPOSED NMT

Japanese sentence
次に、酸化膜をhf洗浄により除去した後、貼り合わせウェーハの剥離面から酸素イオンを注入した。
(Next, after removing an oxide film by
hf washing, we inject oxygen ions fromthe peeled surface of the laminatedwafer.)The Chinese word
“晶片”(wafer) is out ofthe vocabulary

接着，通过hf洗涤除去氧化膜后，
从贴合晶片的剥离面注入氧离子。

Correct!

(TRANSLATION:
Subsequently, after the oxide film wasremoved by washing with hf, and oxygenions were injected from the peeled surfaceof the laminated wafer.)

NMT decoding
by the BASELINE NMT

接着，通过hf清洗除去氧化膜后，
从贴合的UNK的剥离面注入氧气。

Incorrect!

(TRANSLATION:
Then, after the oxide film was removed by
hf cleaning, oxygen was injected from thepeeled surface of the laminated UNK.)
20

Example of correct translations produced
by NMT decoding
NMT decoding
by the PROPOSED NMTJapanese sentence
次に、酸化膜をhf洗浄により除去した後、貼り合わせウェーハの剥離面から酸素イオンを注入した。
(Next, after removing an oxide film byhf washing, we inject oxygen ions fromthe peeled surface of the laminatedwafer.)

接着，通过hf洗涤除去氧化膜后，
从贴合晶片的剥离面注入氧离子。

Correct!

(TRANSLATION:
Subsequently, after the oxide film wasremoved by washing with hf, and oxygenions were injected from the peeled surfaceof the laminated wafer.)

NMT decoding
by the BASELINE NMT

接着，通过hf清洗除去氧化膜后，
Japanese technical term
从贴合的UNK的剥离面注入氧气。
“貼り合わせウェーハ”(laminatedIncorrect!
wafer) is translated as Chinese (TRANSLATION:Then, after the oxide film was removed bytechnical term “贴合晶片” by SMT hf cleaning, oxygen was injected from thepeeled surface of the bonded UNK.)
21

NMT with a Large Vocabularyof Technical Terms
 Step 1. training NMT model with technicalterm tokens

 Step 2. applying NMT model with technicalterm tokens

Approach 1. NMT decoding and SMT technicalterm translation
Approach 2. NMT rescoring of 1,000-best SMT
translations (not as fluent as Approach 1)
22

6.0

5.04.0Evaluation Results (2)(automatic evaluation)• BLEU score
55.6

55.3
Slightlyhigher• RIBES score
91.0
90.8
90.5

90.0

3.0
89.5

2.0

much higher

89.3

89.0

1.0

0.0

NMT decoding NMT rescoring ofand SMT technical SMT 1,000-best
term translationtranslation

88.5

NMT decoding
NMT rescoring ofand SMT technical SMT 1,000-best
23
term translationtranslation

0000000000

Evaluation Results (2)(human evaluation)• Pairwise Evaluation(scores range from -100 to 100)

• JPO adequacy evaluation(scores range from 1 to 5)
4.4

36.5

4.3

higher

4.3

4.3
higher

4.2

31.0

4.2
4.1

4.1

4.1
4.0

NMT decoding NMT rescoring ofand SMT technical SMT 1,000-best
term translationtranslation

NMT decoding NMT rescoring ofand SMT technical SMT 1,000-best24
term translationtranslation

Conclusion and Future Work Translating patent sentences with a large vocabulary oftechnical terms by training an NMT system on abilingual corpus, wherein technical terms are replacedwith tokens

 Evaluation experiments on Japanese-Chinese patentsentences proved the effectiveness of the proposed
method Future Work: Evaluate the proposed method with a bidirection NMT system(Bahdanau et al. 2015 ) Rescore 1,000-best NMT translations by using SMT system

25

Thank you for your attention!

26