International Journal on Digital Libraries manus ript No.
(will be inserted by the editor)

Hidetsugu Nanba  Natsumi Anzen  Manabu Okumura

Automati Extra tion of Citation Information in Japanese
Patent Appli ations

Abstra t The need for a ademi resear hers to retrieve
patents and resear h papers is in reasing, be ause applying for patents is now onsidered an important resear ha tivity. However, retrieving patents using keywords isa laborious task for resear hers, be ause the terms usedin patents for the purpose of enlarging the s ope of thelaims are generally more abstra t than those used in resear h papers. Therefore, we have onstru ted a framework that fa ilitates patent retrieval for resear hers, andhave integrated resear h papers and patents by analysing
the itation relationships between them. We obtained
ited resear h papers in patents using two steps: (1)dete tion of senten es ontaining bibliographi information, and (2) extra tion of bibliographi information fromthose senten es. To investigate the e e tiveness of our
method, we ondu ted two experiments. In the experiment involving Step 1, we prepared 42,073 senten es,
among whi h a human subje t manually identi ed 1,476
senten es ontaining itations of papers. For Step 2, we
prepared 3,000 senten es, in whi h the titles, authors,and other bibliographi information were manually identi ed. We obtained a pre ision of 91.6%, and a re all of
86.9% in Step 1, and a pre ision of 86.2% and a re all of
85.1% in Step 2. Finally, we onstru ted an information
retrieval system that provided two methods of retrieving
resear h papers and patents. One method was retrieval

by query, and another was from the itation relationships
between resear h papers and patents.

Keywords Citation relationships  Information re-trieval  Invalidity sear h  S ientometri s  Resear hpaper  patent

1 Introdu tion

To appre iate the s ope of a parti ular resear h eld, retrieving both resear h papers and patents has be ome
important for resear hers in resear h elds with a highindustrial relevan e, su h as bios ien e, medi al s ien e,
omputer s ien e, and materials s ien e. However, retrieving patents using keywords is a laborious task forresear hers, be ause the terms used in patents for the
purpose of enlarging the s ope of the laims are generally more abstra t and more reative than those usedin resear h papers. As a result, di erent patents tend
to ontain di erent terms, even though these terms refer to the same things. Moreover, it is often ne essary
for resear hers to use patent lassi ation odes, su h
as International Patent Classi ation (IPC) odes andF terms, for e e tive patent retrieval, but professional
skills and abundant experien e are also required. Therefore, we propose a method that enables resear hers toretrieve patents without the need for any professional
Hidetsugu Nanba
skills.
Fa ulty of Information S ien es, Hiroshima City University
We have integrated resear h papers and patents by
3-4-1 Ozukahigashi, Asaminamiku, Hiroshima 731-3194 extra ting the itation information from patents. For aJAPAN
given retrieved resear h paper, any related patents an
Tel. & Fax: +81-82-830-1584
E-mail: nanbaits.hiroshima- u.a .jp
also be found by tra ing the itation relationships between
the paper and any patents. Therefore, it is possiNatsumi Anzenble for resear hers to retrieve patents, even though they
NEC System Te hnologies
1-40-1 Tomo-minami, Asaminamiku, Hiroshima 731-3168 may not have any spe ial skills for retrieving patents.
JAPAN
Our integrated database is also useful for examiners
E-mail: anzen-nxane st.ne . o.jp
in government patent oÆ es, and for sear hers in the inManabu Okumura
telle tual property divisions of private ompanies. Their
Pre ision and Intelligen e Laboratory, Tokyo Institute of parti ular purpose is to arry out an \invalidity sear h"
Te hnology
on existing patents or on resear h papers that an in4259 Nagatsuta, Yokohama 226-8503 JAPANE-mail: okupi.tite h.a .jp
validate the patents of rival ompanies or patents under2

Hidetsugu Nanba et al.

appli ation in a national patent oÆ e. Patents that have
many itations of papers are onsidered relevant to invalidity sear h, espe ially in dete ting resear h papers thatan invalidate the other patents.
Our system is also useful for resear hers in s ientometri s. In this resear h eld, the itations between
patents and resear h papers are used to analyse the in uen e of s ien e to te hnology [13℄[18℄. In general, if basi
resear h in a domain strongly a e ts a te hnology, thenit is assumed that the patents in that spe i domain
will ite many papers. Therefore, resear hers measure
the relevan e of basi resear h to an industry by ounting the number of ited resear h papers in patents in agiven domain. Manually onstru ted itation databases
are used in this type of analysis, but our automati ally
integrated database allows this type of analysis to bearried out eÆ iently.
The following four points need to be onsidered tointegrate a patent and a resear h paper database,
1. Extra tion of ited papers in a resear h paper.
2. Extra tion of ited patents in a resear h paper.
3. Extra tion of ited patents in a patent.
4. Extra tion of ited papers in a patent.
In this work, we have integrated a Japanese patent
database and a multilingual resear h paper database,\PRESRI"(Paper REtrieval System using Referen eInformation)1 [14{16℄, whi h was onstru ted by olle ting more than 78,000 PostS ript and PDF les found on
the Internet and extra ting the bibliographi information from these les. This database also ontains bibliographi from more than 346,000 ited papers that was
extra ted from the les. For ea h ited paper, one of thefollowing itation types were automati ally determinedby analysing the ontext of the itations in the resear hpapers using several ue phrases [14,15℄.
{ Type B: Citations to other resear hers' theories or
methods.
{ Type C: Citations omparing related work or to pointout problems.
{ Type O: Citations other than Types B and C above.
Using this database, Points 1 and 2 above are alreadyresolved. However, it is impossible to apply their method
to extra t ited patents and papers in a patent, be ause
itation styles in patents are mu h di erent from those inresear h papers, whi h we will des ribe in Se tion 3.2.2
in detail. Therefore, we fo used on Points 3 and 4.
In general, Japanese patent spe i ations have thefollowing stru ture:{ Invention title.
{ Claims.
{ Detailed des ription.
{ Field of the invention.
{ Prior art.
{ Means of solving the problems.
1

http://www.presri. om{ Embodiments of the invention.
{ E e ts of the invention.

{ Brief explanation of drawings.

In the \prior art" eld, the author of the patent ites
other related patents and/or papers. Therefore, we extra ted itation information from the \prior art" eld.
The remainder of this paper is organized as follows.
Se tion 2 des ribes some related works. Se tion 3 explains the pro edure used to integrate the resear h paper
and patent database, and dis usses how we investigatedthe e e tiveness of our method by ondu ting some examinations. Se tion 4 dis usses our experimental results,
and Se tion 5 dis usses the behaviour of our system. Finally, we provide our on lusions in Se tion 6.

2 Related WorkIn this se tion, we des ribe some related works in \invalidity sear hes", \ ross-genre information retrieval", and\s ientometri s".

Invalidity sear hes

An invalidity sear h task was performed in the Patent
Retrieval Task of the Fourth [3℄, the Fifth [4℄, and the
Sixth [5℄ NII Test Colle tion for Information Retrieval(NTCIR) workshops. The goal of this task was to retrieve
patents that ould invalidate existing laims. Five groups
with 21 systems parti ipated in the Japanese retrieval
subtask in the Sixth NTCIR, and the systems were evaluated using the Mean Average Pre ision (MAP). The
best system obtained a MAP of 0.0815 [12℄. The systemanalysed the stru ture of queries, and weighted terms
in parti ular essential parts of the queries, using several weighted methods, su h as the inverse do ument
frequen y (IDF) without a term frequen y (TF) method.
In ontrast to the task at the NTCIR, we aimed toretrieve both patents and resear h papers that ould invalidate existing laims. By integrating both patent andresear h paper databases, we were able to onstru t aross-genre retrieval environment.

Cross-genre information retrieval

There has been mu h resear h in the eld of rossgenre information retrieval, su h as that dis ussed in
the te hni al survey task of the Patent Retrieval Taskof the third NTCIR workshop [11℄. This task aimed toretrieve patents relevant to a given newspaper arti le. In
this task, Itoh et al. fo used on \Term Distillation" [10℄.
The distribution of the frequen y of the o urren e ofwords was onsidered to be di erent between heterogeneous databases. Therefore, unimportant words were assigned high s ores when using TFIDF to weight words.
Term Distillation is a te hnique that an prevent su hases by ltering out words that an be assigned in orre t weights. This idea was also used to link news artiles and blog entries [9℄. This is onsidered to be useful for integrating Japanese patent and resear h paper

Automati Extra tion of Citation Information in Japanese Patent Appli ations

databases. However, a ma hine translation te hnique isalso required in our ase, be ause most of the papers
in PRESRI are written in English. Therefore, we integrated our patent database and PRESRI by analysing
the itation relationships between these databases.
There have been several reports on the quality of itations between resear h papers and patents [1℄[21℄. A ording to S hmo h [21℄, patent itations an be divided
into two types: (1) do uments of parti ular relevan e,and (2) referen es on erning the general ba kground.
S hmo h reported that 29% of all itations an be lassi ed as Type 1. He also showed in another paper that alarge number of referen es an be linked to iting patents
in the eld of spa e te hnology in a very broad sense [20℄.
Although the ratio of itations in Type 1 is di erent forea h domain, sear hers are required to sele t relevant itations to some extent, when they olle t do uments by
tra ing itations. As it is diÆ ult to identify the ategoryof ea h itation automati ally, the problem of ross-genre
information retrieval is not fully resolved by analysing itation relationships alone. However, we still take an optimisti view of using itations, be ause they have beenused su essfully in s ientometri s resear h, whi h is des ribed below.

S ientometri s

S ientometri s is \the study of the measurement ofs ienti and te hnologi al progress". One of the typial te hniques used in s ientometri s is to evaluate theprodu tivity of individual resear hers, organizations, andountries using bibliometri te hniques, su h as an impa t fa tor. Another typi al method used in s ientometri s is to measure how basi s ien e a e ts te hnologies
in a given resear h eld using the itation relationships
between resear h papers and patents. These results an
be used by governments or by private ompanies for allo ating resear h funds to resear h areas that have highindustrial relevan e.
Many studies fo using on itations between papers
and patents have been arried out. To observe the links
between industry and basi s ien e, Narin et al. tra ed
the itations between patents and resear h papers, published in the USA, UK, Germany, Japan, and Fran e,and showed both the domesti and international e e tof s ien e on te hnology [18℄. Other related works in thiseld has been summarized by Meyer in a review arti le[13℄. The itation databases in these analyses were onstru ted manually, but we have integrated patent andresear h paper databases automati ally, whi h makes it
possible to ondu t this type of analysis more e e tively.

3 Integration of Patent and Resear h Paper
Databases
3.1 Pro edure for the Integration of Patent andResear h Paper Databases
We integrated the patent and resear h paper databases
by itation, as resear h papers are usually ited in the

3prior art elds. Figure 1 shows an example of a itationof a resear h paper in the prior art eld. Here, the serial
number is shown at the beginning of ea h senten e asa referen e. Among the three senten es shown in Figure
1, the paper is only ited in Senten e 3. To extra t any
bibliographi information from the ited papers in theprior art eld, we must rst extra t su h senten es (Step1), and then identify any bibliographi information fromthem, su h as the title or author (Step 2).
In the nal step, we identi ed any dupli ate publiation number of the patent that had been extra tedin the previous step. Then, the data from PRESRI and
the extra ted bibliographi information of the resear hpapers arried out in the rst step were gathered andidenti ed as dupli ate bibliographi information. Unlike
patents, a resear h paper does not have an identi ation(i.e., publi ation) number. For this reason, it is ne essary to ompare titles, publi ation year, and other elds
between two papers.
There are many related works for this task [6,8,7℄,
but we used a method based on Nanba's work to identify
dupli ate bibliographi information in papers [16℄. The
pro edure used was as follows.
1. Eliminate parti ular marks, su h as pun tuation andhyphens, from both titles.
2. Compare the year of publi ation.
3. Cal ulate how similar two strings are using a te hnique alled dynami programming (also known as\edit distan e") [19℄.
4. Identify bibliographi information if the ratio ex eedsa threshold value, and if the publi ation years mat h(if one or both publi ation years are not extra ted in||||||||||||||||||||||||||>h$3NoN*si$s8z'1uVO$b:YK8(1)
zr'19k?aK$~OQ?sH8`Q?sHNVG9H
m</^?OC'@VNP~U1rT&,W,"k%(2) 3
NP~U1GO$jq-NQ0KP7FbhrK'19k?aK$.gdhtN)sr]7?j$8`Q?sr~O
Q?sXNEJj,GgHJkh&K,~*0ArC(?
j7F$k%(3) >hN*si$s8z'1}!Nlc,$/$c60>$9Hm</1LN"U#sQ9rQ$?*
1995
si$sjq-Az'1JERpsL.Xq;Qspq PRU95KK-\5lF$k%
111 pp. 49-54[original℄[translation℄(1) Traditionally, this type of online hara ter re ognition
system requires the assignment of strokes or feature points
between an input pattern and a standard pattern for higha ura y hara ter re ognition. (2) In this assignment, strokeorder and stroke ount are used as the onstraints for a robust re ognition of various types of handwriting, and standard patterns are transformed so as to mat h an input
pattern. (3) An example of a traditional online hara ter re ognition method is des ribed in the following paper.
1995, Wakahara, et al. Online handwriting Kanjihara ter re ognition using aÆne transformation for ea hstroke (IEICE Te hni al Report PRU95-111, pp. 49-54).
||||||||||||||||||||||||||
Fig. 1 An example of a itation of a resear h paper in a prior
art eld4

Hidetsugu Nanba et al.

the extra tion stage, then we onsidered that both
years mat hed).
In the following subse tion, we des ribe the rst two
steps.
3.2 Extra tion of Cited Resear h Papers
3.2.1 Dete tion of Senten es Containing Bibliographi
Information

In general, this type of senten e ontains several usefulue phrases. For example, the itation of a paper often(one example)", su h as in Senfollows the phrase, \(des ribed in)" follows
ten e 3 in Figure 1, while \a itation of a paper. The terms \Vol. (volume)", \No.
(number)", and \pp. (page range)" are also useful. On(newspaper
the other hand, some phrases, su h as \(patent)", seldom appear in this typearti le)" or \of senten es. Therefore, we used su h ue phrases to dete t senten es that ontained bibliographi information.
We manually analysed hundreds of prior art eldsthat were randomly sele ted from a patent database, andfound that there were three types of ue phrases:

lc

K-\

79

Cv

Three orpora for olle ting ue phrases
Name
Genre Des ription
PRESRI
Paper 346,000 bibliographi
information
PATENT Patent 3,747,000 full text
appli ations publishedin the 10 years between
1993 and 2002 (100GB)
PRIOR
Patent 42,073 senten es with
ART 1
manually annotated tags.
PRIOR
Patent 3,000 senten es withART 2
manually annotated tags

Table 1

Language
English /JapaneseJapaneseJapaneseJapanese- PRIOR ART 1 is the orpus for dete tion of senten es
ontaining biliographi information.
- PRIOR ART 2 is the orpus for extra tion of biliographi
information.

[1. Initial sele tion step℄
We extra ted arbitrary length hara ter strings in a\sour e of paper" item (usually a onferen e name or
the title of a journal) in the PRESRI orpus to obtainandidates for the ue phrases to be in luded. Then, we
manually sele ted the in luded ue phrases, su h as \Proeedings of" and \Workshop of", from these andidates.

{ Positive ues{ External ues appeared
before, or after a itac(Pl [2. ML step℄(for example)", \tion of a paper, e.g., \c (one su h example)", and \K-\ (des ribedin)".

{ Internal ues appeared in the itation of a paper, e.g., \pp.", \Vol", and \No."{ Negative ues did not appear in senten es that

79

Cv

We applied the Support Ve tor Ma hine (SVM)
method to the \PRIOR ART 1" orpus. We extra ted
features from ea h senten e in the PRIOR ART 1 orpus, whi h in luded information on the existen e of ea hue phrase. We used a polynomial kernel with a degree= 2, and obtained a lassi er that identi ed senten es
ontaining ited papers. Here, we used the existen e ofue phrases as a feature of our ma hine learning.

ited resear h papers,(newspaper arti le)" and \(patent)".
e.g., \
However, it is ostly to form an exhaustive list of ue
phrases manually, be ause the ratio of ited papers ina prior art eld is not as high as that of ited patents. [3. Iterative sele tion step℄
Nevertheless, the style of itations is di erent in patents
We extra ted senten es from the PATENT orpus usompared with resear h papers. For example, the ex- ing the lassi er. Then, we looked for other ue phrases
pressions, \pp.", \p.", and \pages" are used when the from the senten es. When we found new ue phrases, wepages of ited patents are expressed in resear h papers. returned to the ML step dis ussed above.
In addition to these expressions, \ " (page) and \By repeating the above steps until no more ue phrases" (page) are also used in patents. Unfortunatelly, we were obtained, we nally obtained 14 external ues, 22
ould not over enough ue phrases to dete t senten es internal ues, and two negative ues. We also obtained an
ontaining bibliographi information from hundreds of SVM-based senten e extra tor using these ue phrases.
prior art elds. Therefore, we olle ted ue phrases semi- The Appendix shows a list of these ue phrases.
automati ally using three orpora: \PRESRI", \PATENT",
and \PRIOR ART 1."2 The detail of these orpora are 3.2.2 Extra tion of Bibliographi Information
shown in Table 1.
To olle t ue phrases eÆ iently, we arried out the We used information extra tion based on ma hine learning to extra t any bibliographi information, su h as thefollowing pro edure using these orpora.
title or authors, from the senten es extra ted in the pre2
We released these data to the resear h groups that had vious step dis ussed above. In the following subse tion,entered into a li ense agreement with the NTCIR-6 test
olle tion of the National Institute of Informati s (NII), we de ne the tags used in our examination, and thenJapan. Please refer to the NTCIR Web Page for details des ribe our extra tion method.

8

G

Z<

(http://nt ir.nii.a .jp). In this data set, the tags were annotated by a single human analyst, and were he ked a di erent
analyst.

Automati Extra tion of Citation Information in Japanese Patent Appli ations

Tag De nition

5

those in papers, we applied the state transition model
for papers [16℄ shown in Figure 3 to itations in patents.
We de ned a tag set for the bibliographi information Here, we ignored DATE and PAGE tags, be ause thesein the prior art eld as follows.
were not ontained in the transition model. As a result,in patents were not a { E-AUTHORS and J-AUTHORS in luded the au- we found that 62.2% of itations
3
thors' names written in English and Japanese, respe - epted using this model.
tively. If a paper had multiple authors, then the AUTHORS tags were also added before the rst author AUTHORS-TITLE-SOURCE
and after the last author.
AUTHORS-SOURCE{ E-TITLE and J-TITLE in luded the title of the AUTHORS-TITLE
paper written in English and Japanese, respe tively. TITLE-SOURCESOURCEThe TITLE tag ex luded any quotation marks.
{ E-SOURCE and J-SOURCE in luded the sour e TITLE
of the paper written in English and Japanese, respe - Fig. 3 State transition models for resear h papers [16℄tively. The tag in luded the title of any onferen e
pro eedings, volume, number, publisher, or URL.
{ DATE in luded the publi ation year. The DATE
Nanba et al. [16℄ proposed a method for extra tingtags ould also in lude the month or day (e.g., \Septem- bibliographi information from lists of referen es in reber 2003").
sear h papers based on the SVM method. They om{ PAGE in luded the page range of a resear h paper pared the SVM-based method with the HMM-based method,(e.g., \pp. 1{8", \p. 23", \2138{2152").
and on rmed experimentally that the SVM-based method{ OTHER in luded other letter strings (e.g., \to ap- was superior to the HMM-based method. In the samepear").
way, we also used an SVM method. As a ma hine learning method, we also examined the Conditional Random
A tagged example is given below.
Fields (CRF) method, whose empiri al su ess has been|||||||||||||||||||||||||| reported re ently in the eld of natural language proessing.
[original℄
3N@}^/m8+}!O
Both the SVM- and CRF-based methods assign a</OTHER><J<OTHER> h23sG6$s *<Ha<7gs+sU!lslassh word. Features and tags are given in the
SOURCE>
9=F8 </J-SOURCE><PAGE> h594G+ih600G SVM toandeaCRF
methods as follows: (1) the k tags o ur
J
1986/</PAGE><OTHER>K</OTHER><DATE>
before a target word, (2) k features o ur before a target</DATEOTHER> (</OTHER><E-SOURCE>Pro . word, and (3) k features follow a target word (Figure 4).
23 rd><DAC</E-SOURCE><OTHER>, </OTHER>
of594]600<F-\5lF$k#
PAGE>pp.
</PAGE><OTHER>KK*$ We used values of k = 2 and k = 5 for the CRF and
SVM methods, respe tively, whi h were determined in a</OTHER>[translation℄pilot study. Here, we use the following features for ma<OTHER>This logi ma ro expansion method was
des ribed in </OTHER> <J-SOURCE>pro eedings hine learning: a word, its part of spee h, and whetherof the 23th Design Automation Conferen e </J- the word was an internal ue, as des ribed in Se tion
SOURCE><PAGE> pages 594-600</PAGE> 3.2.1.

<OTHER>(</OTHER><DATE>1986</DATE><OTHER>) ( </OTHER> <E-SOURCE> Pro . of 23rd
DAC </E-SOURCE> <OTHER>, </OTHER> <PAGE>
pp. 594-600 </PAGE> <OTHER> ).</OTHER>||||||||||||||||||||||||||
Fig. 2 An example of a manually tagged prior art eld

Information extra tion based on ma hinelearning

There have been several studies on extra ting bibliographi information from lists of referen es in resear hpapers. Most of these were based on Hidden Markov
Models (HMMs) [2℄ [22℄. However, we did not use HMMs
in our study, be ause the state transition model of HMMs
for patent appli ations is more omplex than that for resear h papers. For example, a title or an author name(s)is usually written at the beginning of ea h bibliographi
information string. On the other hand, patent appli ations do not follow this pattern, as shown in Figure 1. To
on rm that itations in patents are more omplex thanFig. 4

3

Features and tags given to the SVM and the CRF

There were 65 state transition patterns in patents.

6

Hidetsugu Nanba et al.
||||||||||||||||||||||||||

4 Experiments>h$+i<h|h}uVK*1k$,4==KQ$ilkm9H[original℄

Khk8% \An Adaptive
To investigate the e e tiveness of our method, we on- 6h}NcH7FO$Floyd-Steinberg,NilF$k%
du ted the following two examinations: (1) dete tion of for Spatial Grey S ale"senten es ontaining bibliographi information, and (2) [translation℄
An arti le \An Adaptive for Spatial Grey S ale" by Floydextra tion of bibliographi information.
Steinberg is known as an example of Error Di usion Methodin olour image pro essing devi es.
||||||||||||||||||||||||||
Fig. 5 An example senten e that our method ould not dete t

4.1 Dete tion of Senten es Containing Bibliographi
Information

Experimental results and dis ussion

Data sets

We used the \PIRIOR ART 1" orpus, whi h was des ribed in Table 1 in Se tion 3.2.1. We manually assigned\PAPER" tags to 42,073 senten es that ontained bibli- Table 2 Experimental results for dete ting senten es onographi information. Among these senten es, we used taining bibliographi information
32,537 senten es for training (among these, the \PAPre ision
Re allPER" tags were assigned to 1,186 senten es), and 9,536
External ue
0.079
0.972
(baseline)(282/3555) (282/290)
senten es for testing (among these, the \PAPER" tags
Internal ue
0.239
0.921
were assigned to 290 senten es).
(baseline)
External and internalues (baseline)
Our method

Alternatives

We ondu ted experiments using the following threemethods:{ External ue (a baseline method), for extra ting(267/1119)
0.261(260/998)
0.916(252/275)(267/290)0.897(260/290)0.869(252/290)

all senten es ontaining an external ue.

Our results are shown in Table 2. The re all s ores

all senten es ontaining an internal ue.

pre ision s ore of the \Internal ue" was better than that{ Internal ue (a baseline method), for extra ting were almost the same among the three methods. The{ Internal and external ues (a baseline method), of the \External ue". This indi ates that internal ues

for extra ting all senten es ontaining both internal are more useful than external ues. A pre ision s ore
of \Internal and external ues" is better than \Internaland external ues.
{ Our method, for extra ting senten es based on the ue" and \External ue". Our method obtained the best
SVM, whi h used external and internal ues as fea- pre ision s ore. It is onsidered that the SVM methodtures.
an be used to optimize the ombination of internal andexternal ues.
In Figure 5, we show a typi al example of a senten e
Ma hine learningthat
our method ould not dete t. Most of su h senten es
We used the TinySVM4 software pa kage as our SVM did not ontain any sour es of papers, be ause most oflearning pa kage. We employed a polynomial kernel with the internal ues were related to the sour es of the paa degree = 2, whi h was de ned by the following equa- pers, su h as \Vol.", \No.", and \pp." However, this istion:
not a serious problem, be ause su h bibliographi inford
K (xi ; xj ) = (xi  xj + 1)
(1) mation is not integrated in PRESRI due to the la k ofbibliographi information.

Evaluation method

We used the following measures for evaluation.
T he number ofRe all=T he number of senten es

T he number of=orre tly

extra ted senten es
T he number of senten es
that the system extra ts

4

Ma hine learning

orre tly

extra ted senten es

that should be extra ted

P re ision

4.2 Extra tion of Bibliographi Information

http:// hasen.org/~taku/software/TinySVM

As shown in Figure 4, both the SVM- and CRF-based
methods assign one of the nine tags ( lasses) des ribed in
Se tion 3.2.2 to ea h word. We used the pairwise lassi ation method and a polynomial kernel with a degree = 2
for lassifying multi lasses using the SVM binary lassier. We used the YamCha5 software pa kage, whi h spe(3) ializes in text hunking based on TinySVM software. As
another example of a ma hine learning method, we also
used the CRF++6 software as a CRF learning pa kage.

(2)

5
6

http:// l.aist-nara.a .jp/~taku/software/yam ha/
http://www. hasen.org/~taku/software/CRF++/Automati Extra tion of Citation Information in Japanese Patent Appli ations

7

Data sets||||||||||||||||||||||||||

Evaluation methodtion of LSI manufa turing lines (\ECS Fall Meeting",
Vol. 88-2, p. 566(1988)) in the international onferen e\Automated IC Manufa turing" hosted by the Ele tro hemi al So iety (Ele tro hemi al So iety)||||||||||||||||||||||||||
Fig. 6 An example senten e where our method mistakenly
extra ted a \Sour e" eld as a \Title" eld (Type 1)c(P$LSIN=$i$sN+0=KD$F$FqE[original℄

We used the \PIRIOR ART 2" orpus shown in Table $ = X qJ
Kg E N q ] q D
Ele tro hemi al WK
So iety*$
Fsp5lF$k
1 des ribed in Se tion 3.2.1. We manually assigned tags, V
Kusing the pro ess des ribed in Se tion 3.2.2, and obtained JAutomated IC Manufa turing\ECS Fall Meeting", Vol. 88-2, p. 566(1988)
3,000 tagged senten es. Then, we performed a ten-fold [translation℄ross validation test.
For example, there is an report on the automaWe used Re all and Pre ision for evaluation. We onsidered an entry orre t if the alphabet, number, andhiragana, katakana, and kanji hara ters (Japanese text
only) in the system output mat hed those in the orre t
data. An example of our evaluation is shown in Table
3. In this example, both E-SOURCE and DATE data in ||||||||||||||||||||||||||[original℄
the system output are orre t, be ause the di eren e be- w
3 s Te< ? G R d w Ke< m G R , & f +tween the orre t data and the system output were the / 5 l F * j$= N \ Y K D $ F O < - K ( 91989/)|$h50s~Q*}XqXQViq 7 s ]separators ` ', ` ', and `)'.

H I

8&`@$8'9H$Vw3sTe<F#s0WJJSA cat]no' AP891232KKRYilF$k

Experimental results and dis ussion[translation℄

Experimental results obtained by extra ting bibliographi information from itations of prior art

Table 4

EnglishJapanese

AuthorSour eTitle
AuthorSour e
Title

Page
Date
Average

YamChaPre ision
Re all
0.720
0.787
0.752
0.787
0.763
0.901
0.742
0.715
0.733
0.662
0.868
0.880
0.941
0.932
0.897
0.897
0.802

0.822

CRF++
Pre ision
Re all
0.872
0.857
0.805
0.799
0.746
0.903
0.885
0.765
0.834
0.736
0.848
0.881
0.973
0.973
0.932
0.921
0.862

0.854

The results obtained are shown in Table 4. As an beseen from the data in Table 4, the performan e of the
CRF++ pa kage was better than that of the YamCha
pa kage. Among the eight types of eld, \Sour e" and\Title" performed worse using the CRF++ method.
Among the results of the CRF++ te hnique, the extra tion of English titles performed the worst. An example of su h ases is shown in Figure 6. In this example,
the title of the paper is not shown. Instead, both the onferen e name (Automated IC Manufa turing) and thesour e (\ECS Fall Meeting", Vol. 88-2, p. 566(1988)) are
shown in di erent parts. In this ase, our method mistook \Automated IC Manufa turing" as a title. We observed many other similar examples, when expressions,su h as \So iety of" or \Asso iation" were not ontainedin the sour es.
We also show other typi al errors in Figure 7. In this(opti alexample, both the title (\omputing)") and the sour e (\(the 50th Annual Meetingof Japan So iety of Applied Physi s, 1989 in the fall)"and \") areshown, but our method mistook the \Sour e" eld (underlined in Figure 6) for the \Title" eld. This is beause both the \Sour e" and \Title" elds onsisted oflong hara ter strings without pun tuation and symbols.
As su h, both elds super ially resembled ea h other.

w3sTe<F#s0
1989/)|$h5
0s~Q*}XqXQViq
JSA cat]no' AP891232

There
has
been
mu h
resear h
intooptialomputer devi es and opti al neuro devi es, the details of whi h will be reported in
the 50th Annual Meeting of Japan So iety of Applied
Physi s, 1989 in the fall symposium digest \opti al omputing" (JSA at-no: AP891232))||||||||||||||||||||||||||
Fig. 7 An example senten e where our method mistakenly
extra ted the \Sour e" eld as the \Title" eld (Type 2)

To improve these errors, we are now onsidering using
lists of journal titles and onferen e names as one of thefeatures for ma hine learning.
4.3 Dis ussion

E e ts of two-steps bibliographi information
extra tion

We dete ted ited resear h papers in patents in twostages: (1) dete tion of senten es ontaining bibliographi
information, and (2) extra tion of bibliographi information from those senten es. To on rm the e e tiveness of
our two step extra tion method, we ompared it with amethod that did not ontain Step 1 (a one-step method).
In the two-step method, we applied two ma hine learning methods, SVM and CRF, to the senten es ontaining
ited papers. In the one-step method, instead of using
only these senten es ontaining ited papers, we applied
the ma hine learning methods to all the senten es in the
PRIOR ART 1 orpus, whi h we des ribed in Se tion
3.2.1.
The results are shown in Table 5. As an be seenfrom the data in Table 5, the results using our method
were mu h better than those obtained using the onestep method. This indi ates that our two-step method ise e tive for dete ting ited resear h papers in patents.

Pro essing speed

To extra t the full bibliographi information from the
100 GB of the PATENT orpus, we needed to operate

8Hidetsugu Nanba et al.

Table 3

Table 5methods

Example of evaluation
Corre tc(P$String(Translation) (For example)
System\E-SOURCE
Solid state te hnology

E-SOURCE

Comparison of the two-step and one-step extra tion

EnglishJapanese

AuthorSour eTitle
AuthorSour e
Title

Page
Date
Averageour method(Step 1 and Step 2)
Pre ision
Re all
0.872
0.857
0.805
0.799
0.746
0.903
0.885
0.765
0.834
0.736
0.848
0.881
0.973
0.973
0.932
0.921
0.862

0.854

"

One-step method(Step 2)Pre ision
Re all
0.539
0.333
0.522
0.354
0.571
0.500
0.350
0.389
0.488
0.406
0.474
0.450
0.760
0.702
0.606
0.664
0.539

0.475

DATE
(May 1990
DATE

Table 6
IPC
ode

)

PAGE
P149-154
PAGE,"k#(exists)

The number of ited papers for ea h domain

C12N
H01L
A61K
C12R
C12P
G06F
C07D
C07C
G01N
H04N

des ription

the number of
ited papers

enzyme, biogeneti s
semi ondu tor, super ondu tion
medi ine, organi
ompoundal ohol, mi roorganism
mi robial produ tion of ompounds
omputer, memory, programming
peni illin ompoundlow-mole ular-weight ompoundmaterial s ien e
TV, fax, video tape

12160
10442
8834
8656
8282
7604
7434
6825
5553
4841

using the full senten es in the orpus. To ondu t thispro ess qui kly, we divided the orpus into 10 parts, and 5.2 System Behaviour
ran the assessment using 10 pro essors. As a result, we
extra ted the entire bibliographi information within a We integrated ited papers des ribed in Se tion 5.1 withperiod of one hour.
PRESRI using the pro edure outlined in Se tion 3.1, andonstru ted a system that enables us to retrieve both reExperimental results of our bibliographi
sear h papers and patents by key phrases and by itainformation extra tion
tions.
On onsidering the omplexity and the diÆ ulty of
Figure 8 shows the sear h results using the key phrase
our task, our experimental results are en ouraging. In a \(ma hine translation)". The he ked boxes areprevious stidy, Nanba et al. [16℄ obtained an a ura y of shown at the head of ea h result. If the user he ked the
87.4% in experiments extra ting bibliographi informa- boxes of relevant papers and sele ted the \display a ition from lists of referen es. Compared with their task, tation graph" ommand at the bottom of the page, thenour task was more omplex, be ause the bibliographi in- PRESRI showed the sele ted papers along with some reformation was embedded in natural language senten es lated papers and patents as a visual output, as shown inin our task. Nevertheless, our results are lose to Nanba's Figure 9.
gures.
In Figure 9, the dots, squares, and arrows denote thepapers, patents, and itation relationship between do uments, respe tively. The title of the paper was shown in
5 System Overview
a pop-up window [17℄ if the user pla ed the ursor overIn this se tion, we will introdu e the overview of our a parti ular dot (i.e., paper) or a square (i.e., patent). If
system.
the user paused the ursor for a period of more than ase ond, then the author(s) and an abstra t of the paper
were shown along with the title. The itation area was
5.1 Data of the Citation Relationships between Patents shown in a pop-up window if the user pla ed the ursor
and Resear h Papers
over an arrow. From the itation graph of resear h paUsing our method, we extra ted 86,415 ited papers from pers and patents shown in Figure 9, the user is able toeld
3,496,253 Japanese patent appli ations published in the understand the progress and transition of spe i
studies
at
a
glan
e.
This
type
of
informationis
helpful
10 years between 1993 and 2002. To grasp the outlineof the data, we lassi ed these ited papers using IPC for an eÆ ient survey of a eld of study.
odes of ea h iting patent. The IPC system is a global
Our system may also be useful for various s ientostandard hierar hi al patent lassi ation system thatmetri
s studies. For example, the data in the \number
ontains 7,314 main groups, and at least one IPC odeofitedpapers for ea h domain" shown in Table 8 inis manually assigned to ea h patent appli ation.
di
ate
the
resear h elds that have high industrial relBy ounting the number of ited papers for ea h IPC evan e. These
data an be used when a government orode, we an measure the relevan e of basi resear h to a private ompany
de ision for the allo ation of
the te hnology in ea h domain. The numbers of ited resear h funds to a makes
parti
ular
eld. Another example ispapers for ea h IPC ode are shown in Table 6. Our in identifying important fundamental
h from ansystem was used e e tively in the elds shown in Table industrial viewpoint using bibliometri researtehniques.
6.

!#]uAutomati Extra tion of Citation Information in Japanese Patent Appli ations

Fig. 8Sear h results using a key phrase

Fig. 9Citation relationships between resear h papers and patents

6 Con lusions

9

Our task was more omplex than Nanba's task of extra ting bibliographi information from lists of referen es[16℄, be ause the bibliographi information was embedded in natural language senten es in our task. Nevertheless, our experimental results almost rea hed the 87.4%,
a ura y s ore obtained in Nanba's experiments. Therefore, we onsider that our results are en ouraging.

We have integrated resear h papers and patents byanalysing the itation relationships between them. In
this work, we fo used on the dete tion of ited resear hpapers in patents in two ways: (1) dete tion of senten es
ontaining bibliographi information, and (2) extra tionof bibliographi information from those senten es. We
obtained a pre ision of 91.6% and a re all of 86.9% in
In future work, we need to improve our method of
Step 1, and a pre ision of 86.2% and a re all of 85.1% in identifying dupli ate bibliographi information between
Step 2.
resear h papers and ited papers in patents. As we de-10Hidetsugu Nanba et al.

s ribed in Se tion 3, there are many related works or
systems for this task [6,8,7℄.
We will also need to study identi ation of the typesof patent itations. As S hmo h reported [21℄, there areat least two types of patent itations: (1) do uments ofparti ular relevan e, and (2) referen es on erning thegeneral ba kground. Automati ally identifying the former type of itations is required if our system is to be
used for invalidity sear hes and s ientometri resear h.
There has been related resear h into identifying types ofitations in resear h papers [14℄[23℄. This resear h may
provide lues to identify types of patent itations.

AppendixHere, we show the list of ue phrases used for the dete tion of senten es ontaining bibliographi information.

Positive ues{ External ues appearedc(P before, or after a8%itation

H7F-\
RYilF$k
-R
/=
Rp
sF
k
(5lF$k
lc
8
\
@8o
gq

+(
2H
NilF$
@
K-

of a paper, e.g., \(for example)", \(arti le)", \(as)", \(written)", \(dis(des ribed)", \(relosed)", \ferred)", \(des ribed)", \(published)", \(introdu ed)", \(proposed)", \(is known)", \(is presented)", \(arti le)" \(one su h example)", and \(des ribed in)".
{ Internal ues appeared in the itation of a paper,e.g., \
(journal)", \(meeting)", \(asso iation)", \pro eedings of", \journal", \workshop",\ onferen e", \university", \international", \symposium", \transa tion", \letters", \pp.", \p.", \vol.", \(vol.)", \ (no.)", \no.", \(pages)",\(19j20)** (four digits of numbers)", \ (year)", \", \ ", \ ", \[", \℄", and \)"

,
f
H V W

Xq

Z<8

/

Negative ues did not appeared in senten es that ited79

Cv

resear h papers,(newspaper arti le)" and \(patent)".
e.g., \
In other words, senten es that ontained negative ues
should not be dete ted.

Referen es

1. Bare, R.: Results of a Statisti al Study of the Referen es
Cited in the Sear h Reports Established by the EPO.
World Patent Information, Vol. 3, No. 2, 56{60 (1981)
2. Borkar, V., Deshmukh, K., Sarawagi, S.: Automati Segmentation of Text into Stru tured Re ords. In Pro eedingsof the 2001 ACM SIGMOD International Conferen e on
Management of Data, 175{186 (2001)
3. Fujii, A., Iwayama, M., Kando, N.: Overview of Patent
Retrieval Task at NTCIR-4. In Working Notes of the 4th
NTCIR Workshop, 225{232 (2004)
4. Fujii, A., Iwayama, M., Kando, N.: Overview of Patent
Retrieval Task at NTCIR-5. In Pro eedings of the 5th
NTCIR Workshop Meeting on Evaluation of Information
A ess Te hnologies: Information Retrieval, Question Answering and Cross-Lingual Information A ess, 269{277(2005)

5. Fujii, A., Iwayama, M., Kando, N.: Overview of the Patent
Retrieval Task at NTCIR-6 Workshop. In Pro eedings of
the 6th NTCIR Workshop Meeting, 359{365 (2007)
6. Galhardas, D., Flores u, D., Shasha, D.: AJAX: An Extensible Data Cleaning Tool. In Pro eedings of the 2000
ACM SIGMOD International Conferen e on Managementof Data, 590 (2000)
7. Giles, C.L., Bolla ker, K., Lawren e, S.: An Automati
Citation Indexing System. In Pro eedings of the 3rd
ACM International Conferen e on Digital Libraries, 89{
98 (1998)
8. Hit h o k, S., Carr, L., Harris, S. Hey, J.M.N., Hall, W.:
Citation Linking: Improving A ess to Online Journals. In
Pro eedings of the 2nd ACM International Conferen e on
Digital Libraries, 115{122 (1997)
9. Ikeda, D., Fujiki, T., Okumura, M.: Automati ally Linking
News Arti les to Blog Entries. In Pro eedings of AAAI
Spring Symposium Series Computational Approa hes toAnalyzing Weblogs, 78{82 (2006)
10. Itoh, H., Mano, H., Ogawa, Y.: Term Distillation for
Cross-db Retrieval. In Pro eedings of Working Notes of
the 3rd NTCIR Workshop Meeting, Part III: Patent Retrieval Task, 11{14 (2002)
11. Iwayama, M., Fujii, A., Kando, N., Takano, A.: Overview
of Patent Retrieval Task at NTCIR-3. In Pro eedings ofWorking Notes of the 3rd NTCIR Workshop Meeting, Part
III: Patent Retrieval Task, 1{10 (2002)
12. Mase, H., Iwayama, H.: NTCIR-6 Patent Retrieval Experiments at Hita hi. In Pro eedings of the 6th NTCIR
Workshop, 403{406 (2007)
13. Mayer, M.: Does S ien e Push Te hnology? Patents Citing S ienti Literature. Resear h Poli y, Vol. 29, 409{
434 (2000)
14. Nanba, H., Okumura, M.: Towards Multi-paper Summarization Using Referen e Information. In Pro eedings ofthe 16th IJCAI, 926{931 (1999)
15. Nanba, H., Kando, N., Okumura, M.: Classi ation of
Resear h Papers using Citation Links and Citation Types:
Towards Automati Review Arti le Generation. In Proeedings of the Ameri an So iety for Information S ien e/ the 11th SIG Classi ation Resear h Workshop, Classiation for User Support and Learning, 117{134 (2000)
16. Nanba, H., Abekawa, T., Okumura, M., Saito, S.: Bilingual PRESRI: Integration of Multiple Resear h Paper
Databases. In Pro eedings of RIAO 2004, 195{211 (2004)
17. Nanno, T., Saito, S., Okumura, M.: Zero-Cli k: a System
to Support Web Browsing. The 11th International World
Wide Web Conferen e (2002)
18. Narin, F., Olivastro, D., Stevens, K.A., Bibliometri s/Theory, Pra ti e and Problems, Evaluation Review,Vol. 18, No. 1, 65{76 (1994)
19. Needleman, S.B., Wuns h, C.D.: A General Method Appli able to the Sear h for Similarities in the Amino A id
Sequen e of Two Proteins. Journal of Mole ular Biology,
Vol. 48, 443{453, (1970)
20. S hmo h, U., Kirs h, N., Lay, W., Ples her, E., Jung,
K.O.: Analysis of Te hni al Spin-o E e ts of Spa erelated R&D by Means of Patent Indi ators. A ta Astronauti a, Vol. 24, 353{362 (1991)
21. S hmo h, U.: Tra ing the Knowledge Transfer from S ien e to Te hnology as Re e ted in Patent Indi ators. S ientometri s, Vol.26, No. 1, 193{211 (1993)
22. Takasu, A.: Bibliographi Attribute Extra tion from Erroneous Referen e based on a Statisti al Model. In Proeedings of the Third ACM/IEEE-CS Joint Conferen e on
Digital Libraries 2003, 49{60 (2003)
23. Teufel, S., Moens, M.: Summarizing S ienti Arti les{Experiments with Relevan e and Rhetori al Status. Computational Linguisti s, Vol. 28, No. 4, 409{445 (2002)