The Patent Mining Task in the Seventh NTCIR Workshop
Hidetsugu NANBA

Atsushi FUJII

Graduate School of Information Sciences
Hiroshima City University
3-4-1 Ozuka-higashi, Hiroshima, 731-3194, Japan

Graduate School of Library, Information and Media
Studies, University of Tsukuba
1-2 Kasuga, Tsukuba, 305-8550, Japannanba@hiroshima-cu.ac.jp

fujii@slis.tsukuba.ac.jp

Makoto IWAYAMA

Taiichi HASHIMOTO

Hitachi, Ltd.
1-280 Hitashi-Kougakubo, Kokubunji, 185-8601, Japan

Integrated Research Institute, Tokyo Institute ofTechnology
4259 Nagatsuta, Yokohama, 226-8503, Japan

makoto.iwayama.nw@hitachi.com

ABSTRACTThis paper introduces the Patent Mining Task of the Seventh
NTCIR Workshop and the test collections produced in this task.
The task’s goal was the classification of research papers written ineither Japanese or English in terms of the International PatentClassification (IPC) system, which is a global standard. For thistask, 12 participant groups submitted 49 runs. In this paper, we
also report the evaluation results of the task.

Categories and Subject Descriptors
H.3.3 [Information Search and Retrieval]: Search process
H.3.4 [Systems and Software]: Performance evaluation
H.3.5 [Online Information Services]: Data sharing

Measurement, Performance, Experimentation

Keywords

1.

For a researcher in a field with high industrial relevance,
retrieving research papers and patents has become an important
aspect of assessing the scope of the field. Examples of these fields
are bioscience, medical science, computer science, and materials
science. In fact, the development of an information retrieval
system of research papers and patents for academic researchers iscentral to the Intellectual Property Strategic Programs for 20061
and 20072 of the Intellectual Property Strategy Headquarters in
the Cabinet Office, Japan.
In addition, research paper searches and patent searches are
required by examiners in government Patent Offices, and by theintellectual property divisions of private companies. An example
is the execution of an invalidity search among existing patents or
research papers, which could invalidate a rival company’s patents
or patents under application in a Patent Office.
However, the terms used in patents are often more abstract or
creative than those used in research papers, to try to widen thescope of the claims. Therefore, the Patent Mining Task aims todevelop fundamental techniques for retrieving and classifying
both research papers and patents.

General Terms

test collection, classification of researchInternational Patent Classification (IPC)hashimoto@iri.titech.ac.jp

papers, patent,

I&TRODUCTIO&

The Patent Mining Task in the Seventh NTCIR Workshop(NTCIR-7) investigated the effective retrieval of necessary
information from research papers and patent databases. In thispaper, we introduce the task and report the evaluation results.
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.
PaIR’08, October 30, 2008, Napa Valley, California, USA.
Copyright 2008 ACM 978-1-60558- 256-6 /08/10...$5.00.

In previous NTCIR Workshops, Patent Classification Subtasks
were conducted [4][5]. In these subtasks, participants were asked
to classify Japanese patent applications in terms of the File
Forming Term (F-term) system, which is a classification system
for Japanese patent documents. Here, we are focusing on theclassification of research papers in addition to patents. The aim of
the Patent Mining Task in NTCIR-7 was the classification ofresearch papers written in either Japanese or English in terms of
the International Patent Classification (IPC) system, an alternative
patent classification system.
The remainder of this paper is organized as follows. In Section 2,
we describe some related works. In Section 3, we explain the task
description. In Section 4, we describe the participants of the task.

12

http://www.kantei.go.jp/jp/singi/titeki2/keikaku2006_e.pdf
http://www.kantei.go.jp/jp/singi/titeki2/keikaku2007_e.pdf

In Section 5, we report the evaluation results. Finally, we
conclude in Section 6.

2.

RELATED WORKS

The task of research paper classification into the IPC system isconsidered a cross-genre text classification study. Although patentclassification tasks were conducted in the Fifth and Sixth NTCIR
Workshops [4][5], and in Falls' study [1], they did not focus oncross-genre text classification. In other evaluation workshops,
such as the Text REtrieval Conference (TREC) 3 and the CrossLanguage Evaluation Forum (CLEF)4 , no tasks that focused oncross-genre information access have been conducted.
Although the goal of the task was not text classification, a related
subtask was conducted in the Patent Retrieval Task in the Third
NTCIR workshop [6]. This subtask aimed to retrieve patents
relevant to a given newspaper article. In this task, Itoh et al.
focused on "Term Distillation" [3]. The distribution of thefrequency of the occurrence of words was considered to differ
between heterogeneous databases. For example, the word"president" often appears in newspaper articles but seldom
appears in patents. Therefore, unimportant words were assigned
high scores when using the Term Frequency Inverse Document
Frequency (TFIDF) method to weight words. Term Distillation isa technique that can prevent such cases by filtering out words that
could be assigned incorrect weights
There is another approach for cross-genre information retrieval.
Nanba et al. proposed a method to integrate a research paper
database and a patent database by analysing citation relations
between research papers and patents [11]. For the integration, they
extracted bibliographic information of cited literatures in "priorart" fields in Japanese patent applications. Using this integrated
database, users can retrieve patents that relate to a particularresearch paper by tracing citation relations between research
papers and patents. However, the number of cited papers among
patent applications is not enough to retrieve related papers or
patents, even though the number of opportunities for citing papers
in patents or for citing patents in papers has been increasing
recently.
To solve this problem, Kamaya et al. proposed a method toparaphrase scholarly terms into patent terms (e.g., paraphrase"floppy disc" into "magnetic recording medium") [7]. They usedhypernym-hyponymy relations between terms for theparaphrasing [10]. Some patent terms (e.g., "magnetic recording
medium") are the hyponyms of scholarly terms (e.g. "floppy
disc"). Therefore, a paraphrase of some scholarly terms can berealized by finding their hypernyms. However, some scholarly
terms do not have such a relationship with their corresponding
patent terms. For example, a hypernym of the scholarly term"machine translation" is "natural language processing," but theterms "automatic translation" or "language translation" are used inpatents instead of "natural language processing". Therefore, theyalso used citation relationships between research papers andpatents for paraphrasing. Generally, a research paper and a patentthat have citation relationships with each other, tend to be in thesame research field. Using this idea, a paraphrase of a scholarly
term can be realised by using the following procedure:3

http://trec.nist.gov/
4
http://www.clef-campaign.org/

1.

Retrieve research papers that contain a given scholarly termin their titles.

2.

Collect patents that have citation relationships with thepapers retrieved in Step 1.

3.

Extract patent terms from patents collected in Step 2.

4.

Output patent terms extracted in Step3.

They combined the hypernym-hyponym-based and the citationbased methods, and finally obtained the patent terms as theparaphrase of the given scholarly term.
These related works are a part of solutions for cross-genre
information access. The organizers of the Patent Mining Task
expected participants to propose other fundamental techniques forcross-genre information access.

3.

THE PATE&T MI&I&G TASK

3.1. Task Overview
As we described in Section 1, the goal of the Patent Mining Task
was the classification of research papers into the IPC system,
which is a global standard hierarchical patent classification system.
One or more IPC codes are manually assigned to each patent,
aiming for effective patent retrieval.
The sixth edition of the IPC system contains more than 50,000
classes at the most detailed level. The goal of this task was toassign one or more of these 50,000 classes to a given topic, as
expressed in terms of the title and abstract of a research paper. An
example of a topic is shown in Figure 1. Here, <TOPIC-ID>
specifies the topic identification number, and <TITLE> and<ABSTRACT> specify the title and abstract of the research paper
to be classified.

<TOPIC><TOPIC-ID> 100 </TOPIC-ID><TITLE> DTMF (Dual Tone Multi-Frequency) transmission
method for a mobile communication system </TITLE><ABSTRACT> A highly efficient speech-encoding scheme called
VSELP is adopted for Japanese digital mobile communication
systems. However, DTMP (Dual Tone Multi-Frequency) signals
are distorted by using this encoding scheme. This paper presents a
DTMF signal transmission scheme. DTMF signals are transmitted
in the form of call control messages from mobile stations (MS) to
the mobile control centre (MCC). In addition, necessary control
capabilities in MS and MCC are described. </ABSTRACT></TOPIC>
Figure 1. An example of a topic in the English subtask
Within the overall task, the following subtasks were conducted.
•

Japanese subtask: classification of Japanese research papers
using patent data written in Japanese.

•

English subtask: classification of English research papers
using patent data written in English.

In addition to these subtasks, we also conducted the following
more challenging subtasks, which require both cross-genre andcross-lingual information access techniques.

•Cross-lingual subtask (J2E): classification of Japanese
research papers using patent data written in English.

(2) United States Patent and Trademark Office (USPTO)
Patent Data•

Cross-lingual subtask (E2J): classification of English research
papers using patent data written in Japanese.

The data comprised the full texts of USPTO patents, without
images or formulas. A primary IPC code was manually assigned
to each patent. These data were distributed to the groups
participating in the English subtask and the cross-lingual subtask(J2E).

These four subtasks are summarized in Figure 2.
Research Paper written in
Japanese (Japanese / J2E
subtasks)

Research Paper written in
English (English / E2J
subtasks)

A Participant System

Machine Translation
Module (E2J / J2E)
Patent Data
- written in Japanese(Japanese / J2E)
- written in English(English / E2J)

Text Classification
ModuleList of IPC Codes
Figure 2. Summary of subtasks
In the next subsection, we describe in detail the patent data usedin these subtasks.

3.2. Patent Data
An overview of the patent data used in each subtask is shown in
Table 1. In the following, we describe details of the data.
Table 1. Document Sets
Data
(1) UnexaminedJapanese patentapplications(2) USPTO patent
data(3) Patent Abstractsof Japan (translated
into English)(4) NTCIR-1 andNTCIR-2 CLIR Task
test collection(Abstracts of research
papers)

Year
1993–
2002

Size
100
GB

&umber3.50M

Language
Japanese

1993–
2000
1993–
2002

33
GB
4.2
GB

0.99M

English3.50M

English

1988–
1999

1.4
GB

0.26M

Japanese/
English(1) Unexamined Japanese Patent Applications
The data comprised the full texts of Japanese unexamined patents.
Each patent was text only, without images or formulas. One ormore IPC codes were manually assigned to each patent. These
data were distributed to the teams participating in the Japanesesubtask and the cross-lingual subtask (E2J).

(3) Patent Abstracts of Japan (PAJs)
The data comprised English abstracts translated from Japan Patent
Information Organization (JAPIO) Patent Abstracts. One or more
IPC codes were manually assigned to each abstract. These data
were distributed to the groups participating in the English and J2E
subtasks.
(4) &TCIR-1 and &TCIR-2 CLIR Task Test Collection
This database was distributed to all participant groups, and they
were allowed to use it for any purposes. The database wasoriginally used in the Cross-lingual Information Retrieval (CLIR)
tasks in the first and second NTCIR Workshops (NTCIR-1 andNTCIR-2) [8][9]. It contains 255,960 records of Japanese-English
paired documents, with each record comprising a title, theauthor(s), an abstract, keywords, a publication year, and aconference name.

3.3. Relevance Judgements
Sets of topics with manually assigned IPC codes are necessary forthe evaluation. However, it is very costly and time consuming tocreate such data sets. Therefore, we have produced the data setsusing the following idea.
Essentially, an invention is not patentable if it was already known
before the date of filing. However, Article 30 in the Japanese
patent law provides a six-month grace period for disclosures madevia a publication or a presentation at a conference or exhibition. In
this case, the applicants must mention the proceedings’ title (orthe conference name) and the date it was published in an"Indication of exceptions to lack of novelty" field (or exception
field) in the patent. Figure 3 gives an example of an exception
field.
(original)【新規性喪失の例外の表示】特許法第３０条第１項適用申請有り２０００年３月１４日 社団法人情報処理学会発行の「第６０回（平成１２年前期）全国大会講演論文集（４）」に発表(English translation)[Indication of exceptions to lack of novelty] The provisions set
forth in Article 30, Paragraph 1 in Japanese patent law.
Proceedings (Volume 4) of the 60th Annual Meeting of theInformation Processing Society of Japan, published in March 14,
2000.
Figure 3. An example of the "Indication of exceptions to lackof novelty" field
We can assume that most of the content of the paper mentioned in
the exception field overlaps with the patent. Therefore, if weregard the IPC codes that were assigned to the patent as the codesthat should be assigned to the research paper mentioned in theexception field, it becomes possible to create a large-scale data set
at low cost. In fact, there are more than 9,000 applications withexception fields in the 3,496,253 Japanese patent applications
published in the 10-year period 1993–2002.
The procedure used to create the data set was as follows. Firstly,
we extracted publication years and proceedings titles from theexception fields in the 9,000 applications. Although the title andauthors of a paper are not mentioned in the exception field, theauthors are usually the same as the inventors of the patent. We
therefore extracted and used the inventors of the patent instead of
the authors’ names.
Secondly, we compared these extracted data with records in a
research paper database using a simple string matching method.
From this automatic matching, we obtained, on average, six
candidate records for each exception field.
Thirdly, we manually identified the correct match from among thecandidate records. This produced 976 pairs of matching patents
and research papers.
From these pairs, we created English and Japanese topics (titles
and abstracts) and their correct classifications (IPC codes
extracted from patents). For each topic, an average of 2.3 IPC
codes was assigned.
We then randomly assigned 97 topics to the "dry run" and theremaining 879 topics to the "formal run". The dry run data were
provided to the participant teams as training data for the formal
run. A list of pairs of a patent ID and one or more IPC codes werealso provided as additional training data5. These IPC codes wereextracted from each patent in the data sets (1), (2), and (3).
Participant teams were asked to submit one or more ranked lists6
of IPC codes for each topic, to be evaluated using Mean Average
Precision (MAP), Recall, and Precision measurements. To
calculate these measurements for each submitted run, theorganizers produced a Perl program that was compatible with thetrec_eval program7. The values for MAP, Recall, and Precision
can potentially be different depending on the version of trec_eval
used.

4.

PARTICIPA&TS

We had 24 participating systems for the Japanese subtask, 20 for
the English subtask, and five for the Cross-lingual subtask. As far
as the number of groups is concerned, we had 12 participatinggroups of universities and companies. Table 2 shows thebreakdown of the groups.

5
From the results of the dry run, we found that detecting patents that are
the counterparts of given research papers (topics) and using them for theclassification of papers was more effective than using all the data in thelist. We therefore removed the data for the 976 patents from the list anddistributed the list to the participant teams before the formal run, because
the approach of detecting the counterpart patents is not our intended
purpose here.
6
7

The maximum number of IPC codes for a single topic is 1,000.
http://trec.nist.gov/trec_eval/trec_eval_latest.tar.gz

Table 2. Breakdown of Participants (Please note that one
group consists of universities in &orth America and AsianCountries)
Japan

University
Company

3
2

Other
Asian
Countries
40

Europe&orthAmerica01

20The number of runs for each subtask was as follows.
Japanese subtask: 24 runs from five groupsEnglish subtask: 20 runs from nine groups

J2E: five runs from two groups

There were no runs submitted to E2J.

5.

RESULTS5.1. Evaluation Results of the Patent Mining
Task
We show the evaluation results for the Japanese, English, andCross-lingual subtasks in Tables 3, 4, and 5, respectively. We also
show the recall-precision curves for each subtask in Figures 4, 5,
and 6. The systems "A13", "F1", and "G5" obtained the bestscores in Japanese, English, and Cross-lingual subtasks,
respectively. These systems employed the k-Nearest Neighbours(k-NN) method in common, and exceeded 0.4 of MAP scores. In
addition to these, many other systems, such as C1 and C2 in theJapanese subtask and "G1", "G2", "G3", and "G4" in the English
subtask, also employed the k-NN method.
On the other hand, some systems using machine learning
approaches also obtained remarkable results. The "B4" system in
the Japanese subtask applied logistic regression to the dry run data
for tuning of the model, and obtained 39.64 of the MAP score forthe formal run data. In the English subtask, "B6" obtained 34.79
of the MAP score using the hybrid model of logistic regression
and Naïve Bayes.

5.2. Comparison with the Results of PatentClassification
To investigate the effectiveness of a cross-genre text classification
system, we compared it with a patent classification system. For
the cross-genre text classification system, we used "C1" for theJapanese subtask. The "C1" system comprises a patent retrieval
engine [10] developed for the Patent Retrieval Task in NTCIR-6[2]. The engine introduced the Vector Space Model as a retrieval
model and SMART [12] for term weighting. The "C1" system
obtained a list of IPC codes using the follow procedure.
1.

Retrieve top 170 results using the patent retrieval engine fora given query (research paper).

2.

Extract IPC codes with relevance scores for the query fromeach retrieved patent in step 1.

3.

Rank IPC codes using the following equation.

୬

Table 4. MAP for English Subtask

ScoreሺXሻ = ෍ Relevance score of each patent

Run ID୰ୀଵ

Here, X and n indicate an IPC code and the number of patents that
X is assigned to within the top 170 retrieved patents, respectively.
We also used this system for patent classification. As we
described in Section 3.3, each query has its counterpart in a patent
database. We regarded these counterparts as queries andconducted patent classification using the "C1" system. We used
the full text of patents as queries. Instead of using the top 170
retrieved patents, we used the top 20 for the calculation of each
IPC code in Step 3, which was determined using the dry run data.
From these experiments, we obtained 37.06 for the MAP score,
which is almost the same as the 39.13 obtained for the "C1"system, as shown in Table 3. This result indicates that theperformance of "C1" as a cross-genre text classification reachedalmost the same level as an intra-genre text classification.

Run IDMAP

Run ID

MAP

48.86

H2

14.37

F2

47.21

I1

14.36

F3

44.53

H1

14.23

G1

42.45

I2

13.39

G2

42.09

J2

12.65

G3

41.83

K4

10.45

G4

41.49

J4

9.90

B6

34.79

K1

9.78

B5

33.74

J3

9.37

D2

29.03

L1

3.79

Table 5. MAP for Cross-lingual Subtask (J2E)
Run ID

6.
MAP

Run ID

F1

Although the patent classification system used full texts in patents,
the MAP score was rather lower than that for the cross-genre textclassification system, which used titles and abstracts of research
papers as queries. One of the reasons for this result could be theway terms are used in claims. As we mentioned in Section 1, more
abstract and creative terms are used in patents than those used inresearch papers, to try to widen the scope of the claims. In
particular, the terms in claims are more abstract than those used in
specifications. It is therefore considered that using terms inspecifications may obtain a better MAP score than using wholeterms in patents for patent classification. This may also indicatethat using specifications instead of full texts can improve theperformance of cross-genre text classification.
Table 3. MAP for Japanese Subtask

MAP

MAP

G5

43.80

K5

10.70

K6

10.41

K2

9.41

K3

9.34

CO&CLUSIO&

We have given an overview of the evaluation and design of the
Patent Mining Task in NTCIR-7. We focused on the "Indicationof exceptions to lack of novelty" field in Japanese patentapplications and thereby created 976 English and Japanese topics
and their correct classifications (IPC codes). Forty-nine runs from
12 participant groups were submitted to the formal run, and thesystems using the k-NN method obtained the best MAP scores ineach subtask. Details of each system will be reported in the
NTCIR-7 final meeting to be held in December 2008. After the
NTCIR-7 meeting, the test collection described in this paper will
be available to the public for research purposes.

A13

44.02

A4

41.65

A11

43.71

B4

39.64

A12

43.61

*C1

39.13

A7

43.60

*C2

39.06

A1

43.34

A14

38.62

A6

43.29

B3

35.72

A5

43.26

B2

34.35

A8

43.23

B1

33.03

7.

A10

43.18

D1

27.27

A3

42.68

*C3

14.12

[1] Fall, C.J., Töresvári, A., Benzineb, K., and Karetka, G. 2003.
Automated Categorization in the International PatentClassification. ACM SIGIR Forum, Vol.37, No.1, pp.10-25.

A2

42.36

E1

6.98

A9

42.27

E2

4.06

(C1, C2, and C3 are the task organizer's systems)

REFERE&CES[2] Fujii, A., Iwayama, M., and Kando. N. 2007. Overview of the
Patent Retrieval Task at the NTCIR-6 Workshop. Proceedingsof the 6th NTCIR Workshop Meeting.
[3] Itoh, H., Mano, H., Ogawa, Y. 2002. Term Distillation for
Cross-db Retrieval, Proceedings of Working Notes of the 3rd
NTCIR Workshop Meeting, Part III: Patent Retrieval Task.
[4] Iwayama, M., Fujii, A., and Kando, N. 2007. Overview ofClassification Subtask at NTCIR-6 Patent Retrieval Task.
Proceedings of the 6th NTCIR Workshop Meeting.

[5] Iwayama, M., Fujii, A., and Kando, N. 2005. Overview ofClassification Subtask at NTCIR-5 Patent Retrieval Task.
Proceedings of the Fifth NTCIR Workshop Meeting on
Evaluation of Information Access Technologies: Information
Retrieval,
Question
Answering andCross-Lingual
Information Access.
[6] Iwayama, M., Fujii, A., Kando, N., Takano, A. 2002.
Overview of Patent Retrieval Task at NTCIR-3. ProceedingsofWorking Notes of the 3rd NTCIR Workshop Meeting,Part III: Patent Retrieval Task.
[7] Kamaya, H., Nanba, H., Okumura, M., Shinmori, A.,
Tanigawa, H., and Suzuki, T. 2007. Paraphrasing Scholarly
Terms into Patent Terms using Citation Relations between
Research Papers and Patents. IPSJ SIG Notes NL-178, pp.97102. (in Japanese)[9] Kando, N., Kuriyama, K., and Yoshioka, M. 2001. Overview
of Japanese and English Information Retrieval Tasks (JEIR) at
the Second NTCIR Workshop. Proceedings of the 2nd NTCIR
Workshop Meeting on Evaluation of Chinese & Japanese Text
Retrieval and Text Summarization, pp.4-37 – 4-60.
[10]
Nanba, H. 2007. Query Expansion using an
Automatically Constructed Thesaurus. Proceedings of the 6th
NTCIR Workshop, pp.414-419.
[11]
Nanba, H., Anzen, N., and Okumura, M. Automatic
Extraction of Citation Information in Japanese Patent
Applications. Special Issue of International Journal on Digital
Libraries on Very Large Digital Libraries (to appear).
[12]
Salton, G. 1971. The SMART Retrieval System –Experiments in Automatic Document Processing. PrenticeHall, Inc., Upper Saddle River, NJ.

[8] Kando, N., Kuriyama, K., Nozue, T., Eguchi, K., Kato, H.,
and Hidaka, S. 1999. Overview of IR Tasks at the FirstNTCIR Workshop. Proceedings of the 1st NTCIR Workshop
on Research in Japanese Text Retrieval and Term Recognition,
pp.11-44.

0.45

0.4

0.35

Precision

0.3

0.25

0.2

A13

A11

A12

A7

A1

A6

A5

A8

A10

A3

A2

A9

A4

B4

C1

C2

A14

B3

B2

B1

D1

C3

E1

E2

0.15

0.1

0.05

0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RecallFigure 4. Recall-precision Curves for All Topics (Japanese Subtask)

0.9

1

0.5
F1

F2

F3

G1

G2

G3

G4

B6

B5

D2

H2

I1

H1

I2

0.2

J2

K4

0.15

J4

K1

J3

L1

0.45
0.4

Precision

0.35
0.3
0.25

0.1
0.05
0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 5. Recall-precision Curves for All Topics (English Subtask)

0.45
0.4
0.35

G5
K5
K6
K2
K3

Precision

0.3
0.25
0.2
0.15
0.1
0.05
0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

RecallFigure 6. Recall-precision Curves for All Topics (Cross-lingual Subtask: J2E)

0.8

0.9

1