Patent NMT integrated with Large
Vocabulary Phrase Translation by SMT
Division of Intelligent Interaction Technologies, Faculty of Engineering, Information and Systems, University of Tsukuba

Takehito Utsuro

Takehito Utsuro is a professor at the Division of Intelligent Interaction
Technologies, Faculty of Engineering, Information and Systems, University ofTsukuba, since 2012. His professional interests in natural language processing,Web intelligence, information retrieval, machine learning, spoken language
processing, and artificial intelligence.

Department of Intelligent Interaction Technologies, Graduate School of Systems and Information Engineering, University of Tsukuba

Zi Long

Zi Long is a student of doctor course in Department of Intelligent Interaction
Technologies, Graduate School of Systems and Information Engineering,
University of Tsukuba.

Department of Intelligent Interaction Technologies, Graduate School of Systems and Information Engineering, University of Tsukuba

Ryuichiro Kimura

Ryuichiro Kimura is a student of master course in Department of Intelligent
Interaction Technologies, Graduate School of Systems and Information Engineering,
University of Tsukuba.

Division of Information Engineering, Faculty of Engineering, Information and Systems, University of Tsukuba

Mikio Yamamoto

Mikio Yamamoto is a professor at the Division of Information Enginieering,
Faculty of Engineering, Information and Systems, University of Tsukuba, since2008. His professional interests in natural language processing and machinetranslation.

1 Introduction

when translating patent documents, which contain

Neural machine translation （NMT）, a new

There have been a number of related studies

approach to solving machine translation, hasthat address the vocabulary limitation of NMT. However, asystems. Among them, Luong et al.[5] proposed

conventional NMT is limited when it comesannotating the occurrences of a target unknown

to larger vocabularies. This is because theword token with positional information to track

training complexity and decoding complexity

its alignments, after which they replace theproportionally increase with the number oftokens with their translations using simple word

target words. Words that are out of vocabulary

dictionary lookup or identity copy. However,are represented by a single unknown token inthis previous approach has limitations whentranslations. The problem becomes more serioustranslating patent sentences. This is because

achieved promising results

290

several newly introduced technical terms.

[8][1]

their method only focuses on addressing theconsists of forward and backward LSTMs. The

problem of unknown words even though theforward LSTM reads the source sentence aswords are parts of technical terms. It is obvious

it is ordered（from x1 to xN）and calculates athat a technical term should be consideredsequence of forward hidden states, while theas one word that comprises components thatbackward LSTM reads the source sentence inalways have different meanings and translations

the reverse order（from xN to x1）, resulting inwhen they are used alone.

a sequence of backward hidden states. The

In this article, we present a method that enables

decoder then predicts target words using not

NMT to translate patent sentences with a large

only a recurrent hidden state and the previously

vocabulary of technical terms. We use an NMT

predicted word but also a context vector asmodel similar to that used by Bahdanau el al. ,followings:[1]

p（yz | y＜z, x）＝g（yz－1, sz－1, cz）

in which the technical terms are replaced with

where sz－1 is an LSTM hidden state of decoder,technical term tokens; this allows it to translate

and cz is a context vector computed from both

most of the source sentences except technicalof the forward hidden states and backward hidden

terms. Similar to Bahdanau et al.[1], we use it

states, for 1 z

as a decoder to translate source sentenceswith technical term tokens and replace thetokens with technical term translations using

3

M.

NMT with a Large Technical Term
Vocabulary

3.1 N MT Training after Replacing[3][4].
statistical machine translation（SMT）

Technical Term Pairs with Tokens
Figure 1 illustrates the procedure of the

2 Neural Machine Translation

training model with parallel patent sentence

NMT uses a single neural network trained jointly

pairs, wherein technical terms are replaced.

with technical term tokens “TT1”, “TT2”, ….1 In

Given a source sentence x（x＝x1, …xN）and target

the step 1 of Figure 1, we align the source

sentence y（y＝y1, …yM）, an NMT model uses atechnical terms, which are automaticallyneural network to parameterize the conditional

extracted from the source sentences, with their

to maximize the translation performance[8][1]

distributionsp（yz | y＜z, x）
for 1 zM. Consequently, it becomes possible

to compute and maximize the log probability ofthe target sentence given the source sentenceasMp（y | x）＝∑log（yz | y＜z, x）
z＝1

In this article, we use an NMT model similar tothat used by Bahdanau et al.[1], which consistsof an encoder of a bidirectional long short-term
memory（LSTM）
  and another LSTM as decoder.
In the model of Bahdanau et al.[1], the encoder

1

In this work, we approximately regard all theJapanese compound nouns as source technicalterms. These Japanese compound nouns areautomatically extracted by simply concatenating
a sequence of morphemes whose parts of speech
are either nouns, prefixes, suffixes, unknown
words, numbers, or alphabetical characters. Here,morpheme sequences starting or ending withcertain prefixes are inappropriate as Japanesetechnical terms and are excluded. The sequencesthat include symbols or numbers are also excluded.
In target side, on the other hand, we regard target
translations of extracted Japanese compoundnouns as target technical terms, where we do notregard other target phrases as technical terms.

YEAR BOOK 2O17 291

4

機械翻訳技術の向上

and train the NMT model on a bilingual corpus

sequence of forward hidden states, while the1 of Figure 1, we align the source technical terms,

2. replacing each aligned technicalterm pair with an identical technical‐
term token “TTi” (i = 1, 2, …)

1. aligning technical termpairs by SMT translation model
Japanese sentence:
cmac/ユニット/312/は/信号/を/ブリッジ/インタフェース/388/に/ 提供/する/。

Japanese sentence with technicalterm tokens “TT1”, “TT2” :TT1 /312/は/信号/を/TT2/388/に/提供/する/。

NMT translation
model (withtechnical term tokens

3. training
by NMT
Chinese sentence:

cmac/单元/312/将/信号/提供/给/桥架/接口/388/。

Chinese sentence with tec hnicalter m tokens “TT1”, “TT2” :TT1 /312/将/信号/提供/给/TT2 /388/。

(cmac unit 312 provides a signal to the bridge interface 388.)(TT1 312 provides a signal to the TT2 388.)

Figure 1“TT1”, “TT2”, …)

NMT training after replacing technical term pairs with tokens“TT ”,“TT ”, ⋯1
Figure 1 NMT training after replacing technical term pairs with tokens“��2� ”, “��� ”, ⋯target translations
thetargetsentences.
backward
LSTM readsinthe
sourcesentencein the2targettranslations
viaextracted
decoding
thethesourcewhich areautomaticallyfrom
sourceAs shown
in thestep
1, in each ofreverseorder(from�� 2toof�Figure� ) , resulting in a

sentence
presented ininthis
sentences,usingwith thetheirmethodtarget translations
thesource-target
parallelpatentsentencepairs,
sequenceof backward
hiddenstates.
The decoder

aritcle.
the step2 1Asof Figuregiven2anof
target In
sentences.
shown 2,in whenthe step

2
S〉,〈ta,
occurrences
technical
termusingpairs〈tthenpredicts oftarget
words
not , tonly

input
sentence,
we first automaticallyFiguresource
1, in eachof source-target
parallel patenttT 〉are thenwithpreviously
technical
tT 〉, …,〈tS ,hidden
recurrent
state replaced
and theextracttechnical
terms andreplace themsentencethepairs,
occurrencesof technicalterm, …,〈TT
term tokens〈TT
1 , TT1 〉, 〈TT2 , TT 2 〉
predicted
word
but
also
a context
vector
ask,

（i＝1,
2, …,
with
〈� �technical〉, ⋯, tokens〈� � � � � 〉 “TTpairsthe� � � 〉, 〈� � � � �termarei”thenreplaced

1
S

2

k

1
T

k

TTk〉. Technical term pairs〈t , tT 〉,〈t , t 〉, …,
followings:
1
S

1

2
S

2
T

numberedthe order〈t , t 〉are����� � � � �in���� � �of� �occurrence
k
S

k
T

�

��

���

���

�

…, k）inof source
technical
terms
tSi（i＝1,
where�
is an LSTM
hiddenstate 2,of decoder,���. Here, notethatin alleach� source
sentence
SScomputedandis a context
vector
from
both
of
�

, technicaltheforward
parallel hiddensentencepairs〈SS, S T〉
thestatesandbackward
hidden

�

�

�

�

�

�

k）
. Consequently,
we haveinput〉, 〈�� � ��in〉,with
technical term
tokensan〈��� ��sentence
�

�

�

�

� 〉, 〈� � 2,
which
term
tokens〉. Technical〈� � i�”�（i＝1,⋯, 〈��the� ��technical� � � 〉,
term
pairs “TT
�

�

�

�

�

�

� �〉
…,
represent
positions inof the⋯ ,k）〈���
arethenumbered
the technicalorder of
�

�

terms
and a oflistsourceof extracted
sourcetechnical
occurrencetechnicalterms� � �� ��terms.
shownin sentence
the step 2-Nof Figure���� ⋯ � Next,
�� in aseach
source�� . Here,note”, . “TT2”, … that are identicalterm tokens
states,for � � “TT� � 1�

2,
theinsourcewith technicalthat
all thesentenceparallel sentencepairs 〈�term� � �� 〉,all NMTthe source
patentSS, the source
3.1
Training
after sentences
Replacing TechnicalTermin Section 3.1, whereas the extracted source

throughout all the parallel sentence pairs arein withthis aprocedure.
Therefore,
forVocabulary
example, in
3.usedNMT
Large TechnicalTermtokenstranslatedthe� ”,NMT
model
technicalis term
tokens using“��� ”, “��⋯ thataretrainedto thedescribed
identicalaccording
throughout
all procedure
the parallel
sentence

which appears earlier thantechnical
terms
are oftranslated
using
anof SMT
concatenating
a sequencemorphemes whoseparts
speechothersourcetechnical terms
in SJ is replaced
Figure1 illustrates
the procedureof thephrasetranslation table in the step 2-S ofor alphabetical characters. Here, morpheme sequences

technical
term tPairs
with Tokens

1S

are either nouns, prefixes, suffixes, unknown words, numbers,

We thenthe NMTsystemon awith TT1.modeltraining
withtrainparallelpatentsentence

starting or 4ending with certain prefixes are inappropriate as
Figure2. Finally, in the step 3, we replace thebilingual
corpus,in whichthepairs,
whereintechnical
terms
aretechnicalreplaced termwiththat includeterm
symbolsor numbers
also excluded.
Inoftarget（i＝1,2, …, k）
thetechnical
tokens“TTi”are1 In
k）tokenspairs is replacedby “��
“TT�i””,（i＝1,technical
term tokens“��� ”, 2,
⋯.…,
the
step

extracted JapanesecompoundnounsSMT
as target
terms,
sentencetranslation
withthetechnicaltechnical

and obtain an NMT model in which the technical

1 In this work, we approximately regard all the Japaneseterms are represented as technical termcompound nouns as source technical terms. These Japanesecompound 3nouns are automatically extracted by simply

tokens.

Japanese technical terms and are excluded. The sequencesside, on the other hand, we regard target translations of

where we do not regard other target phrases as technicalterms.translations.
term

2 Details of the procedure of identifying technical term pairs
in the bilingual corpus can be found in the work of Long etal.[3].

3.2 NMT Decoding and SMT TechnicalTerm Translation〔3〕

Figure 2 illustrates the procedure for producing

292

2

Details of the procedure of identifying technicalterm pairs in the bilingual corpus can be found in
the work of Long et al.[3].

3We treat the NMT system as a black box, and thestrategy we present in this article could be applied
to any NMT system. [7][1]

4

We use the translation with the highest probability
in the phrase translation table. When an input
source technical term has multiple translations
with the same highest probability or has notranslation in the phrase translation table, weapply a compositional translation generation
approach, wherein target translation is generated
compositionally from the constituents of source
technical terms.

extracted Japanese technical terms
input Japanese sentence:
コンタクト/プラグ/9/d/と/素子/分離/絶縁/膜/2/の/間/に/は/金属/膜/14/が/介在/し/て/いる 。

TT2 :素子/分離/絶縁 /膜(element isolation insulating film)

TT3 :金属/膜(metal film)

input Japanese sentencewith technical term tokens“TT1”, “TT2”, … :TT1 /9/d/と/ TT2 /2/の/間/に/は/ TT3 /14/が/介在/し/て/いる 。

TT2 : 素子/分離/絶縁/膜TT3 : 金属膜

Chinese translation withTechnical term tokens“TT1”, “TT2”, … :在/TT1 /9/d/和/TT2 /2/之间/夹/着/TT3 /14/。

2-N. decoding by
NMT translation
model (withtechnical term tokens(The TT3 14 is interposed between the TT1
9d and the TT2 2.)

Figure 2

TT1 : 接触/插头

2-S. technicalterm translationby phrasetranslationtable of SMT(contact plug)(The metal film 14 is interposed
between the contact plug 9d and theelement isolation insulating film 2.)

1. replacing them withtechnical
term tokens“TT1”, “TT2”, …

Chinese translation of technical terms

TT1 : コンタクト/プラグ

3. replacing
technical term
tokens “TT1”,“TT2”, … withtechnical termtranslation by
SMT

output Chinese translation:
在/接触/插头/9/d/和/
元件/分离/绝缘/膜/2/之间/夹/着/金属膜/14/。

“TT1”, “TT2”, … )NMT decoding with technical term tokens“TTi”（i＝1, 2, ⋯, k）and SMT technical term

Figure 2 NMT decoding with technical term tokens “ࢀࢀ࢏ ” ሺ࢏ ൌ ૚ǡ ૛ǡ ‫ ڮ‬ǡ ࢑ሻ and SMT technical termpairs are used in this procedure. Therefore, forJapanesesentences
fewer than40
Consequently,
we have containan input sentencein whichௌmorphemes
andChinesethe technicaltermtokenssentences“ ܶܶ௜ ” ሺ݅ ൌcontainͳǡʹǡ ‫ ڮ‬ǡ ݇ሻ

4 Evaluation
example, in all the source patent sentences ܵ ,4.1
PatentDocuments
the source
technical
term

8
fewer
than 40
represent
the words.
positionsof the technical terms andJapanese-Chinese
parallelpatentdocuments
earlierthan other source
technical
terms
in ܵ௃ ispatentdocuments
areas
aJapanese-English
list of extracted source
technical
terms. Next,

shown ininthe
2-N ofworkshop
Figure 2,[11]thesource, whichprovided
thestep
NTCIR-7

werecollected
thetrainJapanese
replacedwith ܶܶଵ .fromWe thenthe NMTpatent
systemdocuments
published
the Japanese
Patenton a bilingual
corpus, inbywhich
the technicaltermsentence
withfrom
technical
term
tokensis translatedarecollected
the 10years
of unexamined

Office（JPO）
during2004-2012
andthepairs isreplacedby “ܶܶ
tokens௜ ” ሺ݅ ൌ ͳǡʹǡ ‫ ڮ‬ǡ ݇ሻusing thepatent
NMT model
trainedpublishedaccordingbytothe
theJapaneseapplications

and obtainan NMTmodel in which
the technicalChinesepatentdocuments
publishedby theprocedurePatentdescribed
in（JPO）
Sectionand
3.1,
whereastheJapanese
Officethe
10 years

3
terms Intellectual
are represented
as technicalState
Property
Office ofterm
the tokens.
People's

extracted
sourcetranslated
patentgrant
data technicalpublished termsby theare
U.S.
PatentRepublic of China（SIPO）during 2005-2010.

using an SMT
phrasetranslationin the step
Trademark
Office（USPTO）
in table
1993-2000.

3.2 NMT
Decoding documents,
and SMT Technical
Term
From
the collected
we extracted

2-S numbersof Figure of
2. documents
Finally, in the3, we replaceThe
arestep
approximately

Translation
312,492
patent families, and the method of

the technicaltokens and“ܶܶ௜ ”1,300,000ሺ݅ ൌ ͳǡʹǡ ‫ ڮ‬ǡ ݇ሻ
3,500,000
forterm
Japaneseforof

the procedure
for producing
Figure 2 illustrates
was applied
to theUchiyama
and Isahara
[9]

5

4

the sentencewith SMT
the technicalEnglish.
Fromtranslationthese document
sets,
patenttargettranslations
decoding
theto source
textof theextractedvia
patentfamilies
alignterm translations.
families
are automatically extracted and the fields

sentenceusing andthe Chinese
method presented
in ThethistheJapanesesentences.

of “Background of the Invention” and “Detailed

aritcle. In thestep 1 ofwere
Figure2, when given
Japanesesentencessegmented
into anainput source
sentence, wefirsttheautomaticallysequenceof morphemes
usingJapanese
6extract the technicalandreplacethem withwiththe morpheme
morphologicalanalyzerterms
MeCab

the technicaltokens“ ܶܶ௜sentences” ሺ݅ ൌ ͳǡʹǡ ‫ڮ‬wereǡ ݇ሻ .
and theChineselexicon
IPAdic,7term

segmented into a sequence of words using

treat the NMT
system as a blackbox, and Stanford
the strategy
the WeChinesemorphologicalanalyzer3

we present in this article could be applied to any NMT system.
[10]
trained using the ChineseWord[7][1] Segment

4. Evaluationof the Preferred Embodiments”
Description

4.1selected.
Patent Documents
areThen, the method of Uchiyama
and Isahara [9] is applied to the text of those4

We use the translation with the highest probability in thephrase translationtable. Whenan English
input source
technical term
fields,and Japaneseandsentences
has multiple translations with the same highest probability or

are
The
Japanesesentences
has aligned.
no translationin thephrase translationtable, we wereapply acompositional translation generation approach, wherein target
segmented
into a sequence of morphemes

translation is generated compositionally from the constituentsof source
technical
terms. morphological analyzer
using
the
JapanesePenn Treebank. In this study, Japanese-Chinese

MeCab with the morpheme lexicon IPAdic.

parallel patent sentence pairs were ordered in

Similar to the case of Japanese-Chinese patentdescending order of sentence-alignment score
and we used the topmost 2.8M pairs, whose

5Herein, we used a Japanese-Chinese translation
lexicon comprising around 170,000 Chinese
entries.

6

http://mecab.sourceforge.net/

7

http://sourceforge.jp/projects/ipadic/〔4〕
8

It is expected that our NMT model can improvethe baseline NMT without our technique whentranslating longer sentences that contain morethan 40 morphemes / words. It is because theapproach of replacing phrases with tokens also
shortens the input sentences, expected tocontribute to solving the weakness of NMT model
when translating long sentences.

YEAR BOOK 2O17 293

4

機械翻訳技術の向上

which appears‫ݐ‬ௌଵdocuments, in this study, out of the provided

was set to 512. We ensured that all sentences

1.8M Japanese-English parallel sentences,in a minibatch were roughly the same length.

1.1M parallel sentences whose Japanese

Further training details are given below:

sentences contain fewer than 40 morphemes（1）We set the size of a minibatch to 128.

and English sentences contain fewer than 40（2）All of the LSTMʼs parameter were initialized

words are used.

with a uniform distribution ranging between－0.06 and 0.06.

4.2

Training and Test Sets

（3）We used the stochastic gradient descent,

We evaluated the effectiveness of the NMT

beginning at a fixed learning rate of 1. We

model presented in this article at translating

trained our model for a total of 10 epochs,

parallel patent sentences described in Section

and we began to halve the learning rate

4.1. Among the selected parallel sentence

every epoch after the first seven epochs.

pairs, we randomly extracted 1,000 sentence（4）Similar to Sutskever et al.[8], we rescaled thepairs for the test set and 1,000 sentence pairs

normalized gradient to ensure that its norm

for the validation set; the remaining sentence

does not exceed 5.
We trained the NMT model on the training set.

pairs were used for the training set. Table 1

The training time was around two days whenshows statistics of the datasets.
Table 1

using the described parameters on a 1-GPU

Statistics of datasets

training
set

validation
set

test
set

ja ↔ ch 2,877,178

1,000 1,000

ja ↔ en 1,167,198

1,000 1,000

machine.

4.4

Evaluation Results

We calculated automatic evaluation scoresfor the translation results using a metric

4.3

Training Details

called BLEU[7]. As shown in Table 2, we report

For the training of the SMT model, including

the evaluation scores, on the basis of thethe word alignment and the phrase translationtranslations by Moses[2], as the baseline SMT.9

table, we used Moses [2], a toolkit for phrase-

and the scores based on translations producedb ased SM T mo d e l s . W e tra i n e d t h e SM T

by the equivalent NMT system without ourmodel on the training set and tuned it with theapproach as the baseline NMT. As shown invalidation set.

Table 2, our NMT systems clearly improve the

For the training of the NMT model, our training

translation quality when compared with theprocedure and hyperparameter choices werebaselines. When compared with the baseline. The

SMT, the performance gain of our system is

encoder consists of forward and backward deep

approximately 6.1 BLEU points when translating

LSTM neural networks each consisting of three

Japanese into Chinese and 8.4 BLEU whenlayers, with 512 cells in each layer. The decodertranslating Japanese into English. When comparedis a three-layer deep LSTM with 512 cells inwith the result of decoding with the baseline

each layer. Both the source vocabulary and

NMT, our NMT system achieved performancethe target vocabulary are limited to the 40K

gains of 2.1 BLEU points when translating

similar to those of Bahdanau et al.

[1]

most-frequently used morphemes/words in the
training set. The size of the word embedding294

9

We train the SMT system on the same training set
and tune it with the validation set.

Table 2

Automatic evaluation results（BLEU）

ja → ch ja → en

5 Conclusion

Baseline SMT[2]

52.5

32.3

Baseline NMT

56.5

39.9

In this article, we presented an NMT method

NMT with PosUnk model[5]

56.9

40.1

NMT with technical termtranslation by SMT

58.6

40.7

SystemTable 3 H uman evaluation results [PE: Pairwise Evaluation（scores range from －100 to 100）and JAE: JPO
Adequacy Evaluation（scores range from 1 to 5）]

SystemBaseline SMT[2]Baseline NMT

ja → chja → en

PE

JAE

PE

JAE

－

3.5

－

3.1

23.0 4.2 21.0 3.9

capable of translating patent sentences with alarge vocabulary of technical terms by training
an NMT system on a bilingual corpus, whereintechnical terms are replaced with technical term
tokens. For the translation of Japanese patentsentences, we observed that our NMT system
performs better than the phrase-based SMT
system as well as the equivalent NMT system
without our approach.

NMT with technical term
30.5 4.3 29.5 4.0translation by SMT

4

NMT with PosUnk model[5] 37.0 4.5 33.5 4.1

機械翻訳技術の向上

Japanese into Chinese, and 0.8 BLEU points
when translating Japanese into English.
Furthermore, we quantitatively compared ourstudy with the work of Luong et al. [5]. As theresult shown in Table 2, compared with the NMT
system with PosUnk model that is proposedas the best model by Luong et al.[5], our NMT
system achieves performance gains of 1.7
BLEU points when translating Japanese into
Chinese and 0.6 BLEU points when translating
Japanese into English.
In this study, we also conducted two typesof human evaluation according to Nakazawa
et al.[6]: pairwise evaluation and JPO adequacy
evaluation.10 Table 3 shows the results of thehuman evaluation for the baseline SMT, thebaseline NMT, the NMT system with PosUnk[5]and our NMT system. We observed that oursystems achieved the best performance forboth pairwise evaluation and JPO adequacy
evaluation.

10 https://www.jpo.go.jp/shiryou/toushin/chousa/
pdf/tokkyohonyaku_hyouka/01.pdf  （in Japanese）

YEAR BOOK 2O17 295

Reference[1] Bahdanau, D., Cho, K., and Bengio, Y.: Neural[10] Tseng, H., Chang, P., Andrew, G., Jurafsky,

machine translation by jointly learning

D., and Manning, C.（2005）. A conditional

to align and translate, in Proc. 3rd ICLR

random field word segmenter for Sighan（2015）

bakeoff 2005. In Proc. 4th SIGHAN Workshop[2] Koehn, P., Hoang, H., Birch, A., CallisonBurch, C., Federico, M., Bertoldi, N., Cowan,

on Chinese Language Processing, pages
168–171.

B., Shen, W., Moran, C., Zens, R., Dyer, C.,[11] Fujii, A., Utiyama, M., Yamamoto, M., andBojar, O., Constantin, A., and Herbst, E.:

Utsuro, T.（2008）. Toward the evaluation ofMoses: Open Source Toolkit for Statisticalmachine translation using patent information.

Machine Translation, in Proc. 45th ACL,

In Proc. 8th AMTA, pages 97–106.

Companion Volume, pp. 177–180（2007）[3] Long, Z., Utsuro, T., Mitsuhashi, T., andYamamoto, M.: Translation of Patent
Sentences with a Large Vocabulary ofTechnical Terms Using Neural MachineTranslation, in Proc. 3rd WAT, pp. 47–57（2016）[4] Long, Z., Kimura, R., Utsuro, T., Mitsuhashi,T., and Yamamoto, M.: Neural MachineTranslation Model with a Large Vocabulary
Selected by Branching Entropy, in Proc. MT
Summit XVI,（2017）[5] Luong, M., Sutskever, I., Vinyals, O., Le, Q. V.,
and Zaremba, W.: Addressing the rare word
problem in neural machine translation, in
Proc. 53rd ACL, pp. 11–19（2015）[6] Nakazawa, T., Mino, H., Goto, I., Neubig, G.,
Kurohashi, S., and Sumita, E.: Overview ofthe 2nd Workshop on Asian Translation, in
Proc. 2nd WAT, pp. 1–28（2015）[7] Papineni, K., Roukos, S., Ward, T., andZhu, W.-J.: BLEU: a Method for Automatic
Evaluation of Machine Translation, in Proc.
40th ACL, pp. 311–318（2002）[8] Sutskever, I., Vinyals, O., and Le, Q. V.:
Sequence to sequence learning with neuralmachine translation, in Proc. 28th NIPS
（2014）[9] Utiyama, M. and Isahara, H. （2007）. A
Japanese- English patent parallel corpus. In

296

Proc. MT Summit XI, pages 475–482.

SMT による大語彙フレーズ翻訳との併用によるニューラルネットワーク機械翻訳（抄録）Takehito Utsuro, Zi Long, Ryuichiro Kimura, Mikio Yamamoto
近年，従来の統計的機械翻訳（Statistic MachineTranslation；SMT） に 代 わ っ て， ニ ュ ー ラ ル ネ ッ

よって従来型の NMT モデルの翻訳精度が改善することができた．

ト ワ ー ク 機 械 翻 訳（Nerual Machine Translation；NMT）モデルが盛んに研究されている．NMT は，原言

謝辞

語文を固定長ベクトルへ写像し，その固定長ベクトルか

本稿で述べた研究においては，日本特許情報機構

ら目的言語文を生成するため，意味的要素の翻訳に非常（Japio） より提供して頂いたパテントファミリーのに優れており，SMT を上回る翻訳精度を達成している．

データを利用させて頂いた．関係各位に感謝の意を表す．

しかしながら，NMT の弱点の一つとして，扱える語彙に限りがある点が知られている．具体的には，扱う語彙のサイズの増加に伴い，NMT モデルの訓練および翻訳

4

機械翻訳技術の向上に要する時間が増す点が課題となっている．
NMT においては，語彙辞書に含まれていない単語は未知語トークンとして出力されるため，これが誤訳となる．そこで，これまでにも，NMT が扱える語彙の規模を拡大する方式について研究が行われてきた．文献 [5]では，訓練用対訳文における単語対応の情報に基づいて，語彙辞書に含まれていない未知語単語を，単語間の対応関係を特定できるトークンに置き換えた後，NMT の訓練を行う方式を提案した．この方式では，出力文に含まれたトークンから未知語が対応する原言語の単語を推定し，その訳語に置き換えることによって，NMT の出力
文において出力可能となる語彙の規模を拡大した．しかし，文献 [5] の方式は，単語単位での語彙規模の拡大にとどまる点が弱点であった．この弱点のため，複合語によって構成される専門用語が多数含まれた特許文の翻訳精度の改善においては限界があった．
以上の背景のもとで，本稿では，特許文を対象とした
ニューラルネットワーク翻訳において，大規模専門用語
語彙に対応する方式 [3][4] について述べる．本方式においては，訓練用対訳文において専門用語間の二言語対応の情報を収集し，二言語間で対応済みの専門用語対訳対を同一のトークンに置き換えた後，NMT の訓練を行う．
本方式による特許文の翻訳時には，専門用語以外の部分に対しては，NMT モデルによる訳文生成がなされ，一方，専門用語部分に対しては，SMT モデルによる翻訳がなされる．本方式を用いない従来型の NMT モデルと，本方式との間で翻訳精度の比較を行った結果，本方式にYEAR BOOK 2O17 297