Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

Overview of the Patent Retrieval Task at the NTCIR-6 Workshop
Atsushi Fujii∗ , Makoto Iwayama† , Noriko Kando‡∗

Graduate School of Library, Information and Media Studies
University of Tsukuba
1-2 Kasuga, Tsukuba, 305-8550, Japanfujii@slis.tsukuba.ac.jp†

Hitachi, Ltd.
1-280 Higashi-Kougakubo, Kokubunji, 185-8601, Japan
makoto.iwayama.nw@hitachi.com/ Tokyo Institute of Technology‡

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-ku, 101-8430, Japankando@nii.ac.jp

Abstract
In the Sixth NTCIR Workshop, we organized the
Patent Retrieval Task and performed three subtasks;
Japanese Retrieval, English Retrieval, and Classification. This paper describes the Japanese Retrieval
Subtask and English Retrieval Subtask, both of whichwere intended for patent-to-patent invalidity searchtask. We report the evaluation results of the groups
participating in those subtasks.
Keywords: Patent retrieval, Invalidity search,
Patent classification, NTCIR

1 Introduction
Processes of patent retrieval differ significantly, depending on the purpose of retrieval. One process isthe “technology survey”, in which patents related to a
specific technology, e.g., “blue light-emitting diode”,
are searched. This process is similar to ad hoc retrieval
tasks targeting non-patent documents.
Another process is the “invalidity search”, in whichprior arts related to a patent application are searched.
Away from academic research, invalidity searches are
performed by examiners in government patent offices
and searchers in the intellectual property divisions ofprivate companies.
In the Third NTCIR Workshop (NTCIR-3), theauthors of this paper organized the Patent Retrieval
Task [8]. In NTCIR-3, we performed the technology
survey task, in which patents were used as technicalpublications.
Given a success in NTCIR-3, we also performed
the Patent Retrieval Task in the Fourth NTCIR Work-

shop (NTCIR-4) focusing on the invalidity search andpatent map generation subtasks [1, 2, 3].
In NTCIR-5, we performed three subtasks; Document Retrieval Subtask, Passage Retrieval Subtask,
and Classification Subtask [4, 5, 6]. The Document
Retrieval Subtask was the same as the invalidity searchin NTCIR-4. However, the numbers of search topics
and target documents were increased. The purpose of
the Passage Retrieval Subtask was to sort passages ina document according to the relevance to a topic. The
purpose of the Classification Subtask was to categorize
patent documents based on a multi-dimensional classification system called “F-term (File Forming Term)”.
In NTCIR-6, we performed the following subtasks:• Japanese Retrieval Subtask
The purpose was the same as the Document Retrieval Subtask in NTCIR-5. However, we introduced a new relevance degree.
• English Retrieval Subtask
The purpose was the invalidity search as in theJapanese Retrieval Subtask. However, we usedpatent grant data published by the U.S. Patent &Trademark Office (USPTO) as target documents.
• Classification Subtask
The purpose was the same as the Classification
Subtask in NTCIR-5. However, the number of
topics was increased. We also explored evaluation methods using the hierarchy of the F-term.
After the NTCIR-6 Workshop meeting, the test collections for these subtasks will be available to the public for research purposes1 .
1 http://if-lab.slis.tsukuba.ac.jp/fujii/ntc6pat/cfp-en.html

Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

This paper describes the Japanese Retrieval Subtask
and the English Retrieval Subtask. The Classification
Subtask is described by Iwayama et al. [7].

2 Japanese Retrieval Subtask
2.1 Overview
The purpose of the invalidity search is to find one
or more patents that can invalidate the demand in anexisting claim. This is a patent-to-patent associative
retrieval task. Away from academic research, invalidity searches are usually performed by examiners in agovernment patent office and searchers of the intellectual property division in private companies.
The Japanese Retrieval Subtask was performed asfollows. First, the task organizers (i.e., the authors of
this paper) provided each participating group with adocument collection and search topics.
Second, each group submitted retrieval results obtained by the topics. Each group was allowed to submit one or more retrieval results. In a single retrieval
result, up to the top 1000 retrieved documents must be
sorted by the relevance score.
Finally, the organizers evaluated the submitted results using relevant documents. The evaluation results
were sent to each group, who was also encouraged toanalyze the results of their methods and report the obtained knowledge at the workshop meeting.
The difference from the Document Retrieval Subtask in NTCIR-5 is three fold.
First, the definition of the relevance was changed.
During NTCIR-5, we identified that the retrieval task
was difficult when a topic patent application and thecitation used to reject the application do not share thesame IPC codes [4]. Thus, we defined relevant documents and partially relevant document according to
the IPC. See Section 2.5 for details.
Second, to compare the evaluation results across thepast test collections, the participating groups were requested to submit retrieval results for the four different
topic sets; NTCIR-4, NTCIR-5, the dry run of NTCIR6, and the formal run of NTCIR-6.
Finally, in each retrieval result (i.e., run), the number of top documents to be examined must be specified. This number is 1000 or less than 1000. This
rule is intended to improve the utility of best-match
retrieval systems. Unlike exact-match retrieval systems, which identify the exact number of documents tobe examined, in best-match retrieval systems it is not
obvious how many top documents must be examined.
This is one reason why professional patent searchers
still use exact-match retrieval systems. However, because most of the participating groups did not address
this problem, we did not apply this rule in the evaluation of the formal run.

2.2

Document Collection

In NTCIR-4, the document collection consisted offive years of unexamined Japanese patent applications published in 1993–1997. In NTCIR-5, the document collection consisted of ten years of unexamined Japanese patent applications published in 1993–
2002. The number of documents in the collection was
3,496,252. In NTCIR-6, we used the same document
collection as NTCIR-5.
To standardize the format of the documents, theorganizers provided an official tool, which inserts
SGML-style tags into each document. Table 1 shows
the tags inserted by that tool. Although <PASSAGE>
and <PNUM> were intended for the Passage Retrieval
Subtask in NTCIR-5, the participating groups wereallowed to use passage information for document retrieval purposes in NTCIR-6.

Table 1. Tags for Japanese patent applications.
<DOC><DOCNO><TEXT><PASSAGE><PNUM>

2.3

document
document identifier
text body
passagepassage identifier

Search Topics

Each search topic is a Japanese patent application
rejected by the Japanese Patent Office (JPO). For each
topic, one or more citations (i.e., prior art) were identified by examiners of the JPO to invalidate the demandin the topic patent.
To increase the number of topics, we decreased thecost required for producing search topics and relevance judgements as much as possible. We automatically extracted patent applications rejected by the JPO
and the citations used for the rejection. For this purpose, we used the citation information in the “Seirihyoujunka (Standardized)” Data, which had been produced from the master database in the JPO. We used
only the citations as relevant or partially relevant documents and did not perform relevance judgement byhuman assessors.
An application used as a search topic must satisfy
all of the following four criteria.
• the application was not used as a search topic in
NTCIR-4 and NTCIR-5.
• the texts of both the application and the corresponding citation are included in the document
collection.

Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan• the citation had been published before the application was filed because to invalidate the invention in a topic patent, relevant documents must
be prior art.
• the application does not claim the priority; otherwise it is difficult to identify the filing date of theapplication automatically.
From applications satisfying the above criteria, we selected 1685 applications as search topics. Although
the number of topics was determined with no particular reason, we intended to produce more than one
thousand topics. To produce topics with a large number of relevant documents, we selected applications forwhich at least five citations are provided.
Because candidates of relevant documents for an
application can be limited by the filing date of the application, for old applications the retrieval of relevant
documents can be a trivial task. Thus, we selected
search topics from recently filed applications.
The citation information we used did not include
the information as to which claim was the target of therejection. From each application, we systematicallyextracted the first claim, which is usually the target, asa topic.
Each search topic file includes a number of additional SGML-style tags. The claim used as the target
of invalidation is specified by <CLAIM>. The date offiling is specified by <FDATE> and only the patents
published before this date can potentially be relevant.
Table 2 shows the tags inserted into each topic. Because each topic is a Japanese patent application, thetags in Table 1 are also inserted in each topic.
Table 2. Tags for Japanese topics.
<TOPIC><NUM><LANG><PURPOSE><FDATE><CLAIM>

topic
topic identifier
topic language
purpose of searchdate of filingtarget claim

For the dry run, instead of the “Seirihyoujunka(Standardized)” Data, we produced the topics from acollection of search reports produced by professional
patent search intermediaries. The source data set wasreference data for examiners at the JPO. We used 1000
reports produced in 2001–2003, from which we produced 349 topics.

2.4 Submissions
Each participating group was allowed to submit one
or more retrieval results, in which at least one result must be obtained using only the <CLAIM> and<FDATE> fields. For the remaining results, any information in a topic file, such as IPC codes, can be used.
The participating groups were requested to submit
retrieval results for the following topic sets:• NTCIR-4: 34 topics• NTCIR-5: 1189 topics• Search report data: 349 topics for the dry run• NTCIR-6: 1685 topics
In NTCIR-4, although a small number of topics were
produced, for each topic professional searchers performed relevance judgement and collected as many
relevant documents as possible. In NTCIR-5 and
NTCIR-6, although a larger number of topics were
produced, for each topic only the citations provided
by examiners in the JPO, which are not exhaustively
listed, were used as the relevant documents. In the
Search report data, the number of topics and the exhaustiveness for the relevant documents are between
those for NTCIR-4 and NTCIR-5/6.

2.5

Evaluation Method

For the Document Retrieval Subtask in NTCIR-5,
the relevance degree of the citation for to a topic wasdetermined based on the following two ranks:• the citation used to reject an application was regarded as a “relevant document (A)” because thedecision was made confidently,• a citation used to reject an application with another citation was regarded as a “partially relevant document (B)”, because each citation is partially related to the claim in the application.
We call this definition “Def0”.
In NTCIR-6, we used an alternative definition for
the relevance degree. If a topic patent and its relevant
document are assigned to the same IPC code, the document can usually be retrieved with a high accuracy.
We define the following two sets for explanation purposes.
• TS: a set of IPC subclasses assigned to topic T• DS: a set of IPC subclasses assigned to relevant
document D
The degree of the relevance of D was classified into
the following three ranks.
• H: the intersection of TS and DS is empty.
• A: the intersection of TS and DS is not empty, but
TS and DS are not identical.
• B: TS and DS are identical.

Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

We call this definition “Def1”.
We used Mean Average Precision (MAP), which
has commonly been used in past IR literature, to evaluate the submitted runs for the Japanese Retrieval Subtask. In Section 4, we show the MAP values for Def0
and Def1, respectively.

Table 3. Tags for English patents.
<DOC><DOCNO><APP-NO><APP-DATE><PUB-NO><PUB-TYPE><PAT-NO><PAT-TYPE><PUB-DATE><PRI-IPC><IPC-VER><PRI-USPC><PRIORITY><CITATION><INVENTOR><ASSIGNEE><TITLE><ABST><SPEC><CLAIM>

2.6 Policy for Resource Usage
Because the citations provided by the examinersof the JPO are available to the public, participatinggroups can obtain relevant documents for the topics
before the formal run.
However, participating groups were not allowed touse the citations corresponding to a topic application
for training purposes. In addition, because we usedsearch topics in the past test collections for Patent Retrieval Tasks, the relevance judgements for these testcollections must not be used for training purposes.
These rules were not applied if their system had already been trained by a large number of citations and it
was not easy to remove information, such as statistics,
related to specific citations.
Except for the above cases, participating groups
were allowed to use any resources for the task according to the “reasonable use policy”.

3 English Retrieval Subtask
3.1 Overview
In the English Retrieval Subtask, we used patent
grant data published from the USPTO as target documents. Each topic is also a USPTO patent and the
purpose is to invalidate the demand in the topic. In
each patent, the applicant and the examiner list one ormore citations as prior arts. We used the citations in a
topic patent as relevant documents for the topic. As in
the Japanese Retrieval Subtask, we used the “reasonable use policy” for resource usage.

3.2 Document Collection
The document collection consists of eight years ofpatent grant data published in 1993–2000. The number of documents in the collection is 981,948. Whilein the Japanese Retrieval Subtask we used patent applications as target documents, in the English Retrieval
Subtask we used only patents that have been grantedby the USPTO.
To standardize the format of the documents, theorganizers provided an official tool, which inserts
SGML-style tags into each document. Table 3 shows
the tags inserted by that tool. Because the format of
the source data was more complicated than that for the

document
document identifier
application number
application datepublication number
publication typepatent number
patent type
publication dateprimary IPC
IPC version
primary USPC
priority information
citation(s)inventor(s)assignee(s)title
abstract
specification
claim(s)Japanese patent applications, we inserted a large number of tags to enhance the readability of the USPTO
patent data.

3.3

Search Topics

Each search topic is a patent grant published in
2000–2001. We selected patents that satisfy the following criteria:• at least 20 citations are listed.
• at least 90% of the citations are included in thetarget document collection.
As a result, we collected 3221 topics, in which we used1000 topics for training purposes and 2221 topics forthe formal run. The format of the topic is the same asthat for the target documents. In addition to the tagsin Table 3, we inserted the tags in Table 2 except for<FDATE> into each topic.

3.4

Submissions

Each participating group was allowed to submit one
or more retrieval results, in which at least one resultmust be obtained using only the <CLAIM> field. For
the remaining results, any information in a topic filecan be used.

3.5

Evaluation Method

The relevant patents for a search topic is one ormore citations provided by the applicant and the ex-Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

aminer in the topic patent.
If a topic patent and its relevant document are assigned to the same IPC class, the document can usually
be retrieved with a high accuracy. Thus, the degree ofthe relevance of each citation was classified into thefollowing two ranks.
• A: The IPC subclasses assigned to the topic
patent and the target document are not identical.
• B: The IPC subclasses assigned to the topic
patent and the target document are identical.
Unlike Japanese patent applications, only a single IPC
code is assigned to each USPTO patent. Thus, we did
not consider TS and DS, which was used in Section 2.5
for explanation purposes.
We used Mean Average Precision (MAP) to evaluate the submitted runs for the English Retrieval Subtask.

4.3

English Retrieval Subtask

Table 10 shows the submitted runs for the English
Retrieval Subtask, which also shows the submission
type (Mandatory or Optional) for each run. From fivegroups, 11 runs were submitted. However, JSPAT0–3,
which were received after the submission deadline, are
not regarded as the official results.
Table 10 also shows the MAP values of the submitted runs for different relevance degrees. While in“A” only “A” relevant documents were used as the correct answers, in “AB” both “A” and “B” relevant documents were used as the correct answers.
Looking at Table 10, the MAP values are generally
small. Because this is mainly due to the selection of
the search topics, we need to explore the process of
producing search topics.

5

Conclusion

4 Evaluation
4.1 Overview
For the Japanese Retrieval Subtask and the English
Retrieval Subtask, we used MAP as the evaluation
measure. To calculate MAP for each submitted run,the organizers produced a Perl program that is compatible with trec eval. This Perl program was also used
for the evaluation in the Classification Subtask. The
value of MAP can potentially be different dependingon the version of trec eval.

4.2 Japanese Retrieval Subtask
Table 4 shows the submitted runs for the Japanese
Retrieval Subtask, which also shows the submission
type (Mandatory or Optional) for each run. From fivegroups, 22 runs were submitted.
Tables 5–9 show the MAP values of the submitted
runs for different combinations of different conditions;
relevance and relevance degree. The condition can be
identified with the caption of each table. “Def0” and“Def1” correspond to the definitions described in Section 2.5. “Rel” denotes the relevance degree.
In Tables 5–9, the column “NTCX” denotes MAP
for NTCIR-X topics, for which X is “4”, “5”, or “6”.
the column “SR” denotes MAP for the search reports.
In the column “Run ID”, run IDs are sorted according
the MAP values in “Total”.
For “Def0”, “H” relevant documents do not exist.
By definition, “A” relevant documents do not exist for“Def0” in “NTC6”.
Looking at Tables 5–9, the relative superiority between groups in MAP was almost the same, irrespective of the condition. However, the relative superiority
between runs of the same group in MAP was salient
depending on the condition.

In the Sixth NTCIR Workshop, we organized the
Patent Retrieval Task and performed three subtasks.
This paper described the Japanese Retrieval Subtask
and English Retrieval Subtask. Both subtasks were intended for the patent-to-patent invalidity search.
During the evaluation in the Japanese Retrieval
Subtask, we compared the results obtained by the topics with exhaustively listed relevance judgement andthe topics with limited relevance judgement. However, the exhaustiveness for relevance judgement did
not affect the relative superiority between participating groups in the retrieval accuracy.

Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

Table 4. Submitted runs for Japanese Retrieval Subtask.
Run ID
AFLAB1
BETA6-1
hcu1
hcu2
HTC01
HTC02
HTC03
HTC04
HTC05
HTC06
HTC07
HTC08
HTC09
HTC10
JSPAT0
JSPAT1
JSPAT2
JSPAT3
JSPAT4
JSPAT5
JSPAT6
JSPAT7

Type
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Optional
Optional
Optional
Optional
Optional
Optional
Optional
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory

Table 6. MAP for Japanese Retrieval Subtask: Def0, Rel=AB
Run ID
HTC10
HTC09
HTC08
HTC07
HTC06
HTC05
HTC04
HTC02
HTC03
AFLAB1
HTC01
hcu1
hcu2
JSPAT3
JSPAT0
JSPAT7
JSPAT1
JSPAT4
JSPAT5
JSPAT2
JSPAT6
BETA6-1

Total
15.41
15.29
14.45
14.41
13.96
12.96
12.12
11.72
11.66
11.46
11.13
6.49
6.37
6.26
6.23
6.22
6.21
6.18
6.17
5.61
5.58
4.89

NTC4
26.36
26.27
25.31
24.85
22.66
24.18
25.96
24.09
24.11
16.15
21.23
15.67
15.30
12.06
10.95
11.12
10.73
10.08
9.87
11.51
11.17
11.31

NTC5
19.73
19.67
18.56
18.47
17.95
16.60
16.30
15.77
15.48
15.39
15.03
8.22
8.06
8.77
8.54
8.75
8.55
8.50
8.52
8.06
8.04
5.86

SR
18.45
18.34
17.17
17.44
16.63
15.99
14.38
14.05
14.43
13.27
12.76
7.52
7.47
6.47
6.65
6.45
6.66
6.63
6.64
5.30
5.27
5.98

NTC6
11.51
11.34
10.77
10.71
10.41
9.53
8.43
8.12
8.15
8.21
7.83
4.88
4.77
4.31
4.41
4.28
4.38
4.37
4.34
3.81
3.79
3.85

Table 5. MAP for Japanese Retrieval Subtask: Def0, Rel=A

Table 7. MAP for Japanese Retrieval Subtask: Def1, Rel=H

Run ID
HTC09
HTC10
HTC08
HTC07
HTC06
HTC05
HTC04
HTC02
HTC03
AFLAB1
HTC01
JSPAT3
JSPAT7
JSPAT0
JSPAT4
JSPAT1
JSPAT5hcu1
hcu2
JSPAT2
JSPAT6
BETA6-1

Run ID
HTC06
HTC08
HTC07
HTC05
HTC04
HTC10
HTC03
HTC02
HTC01
AFLAB1
HTC09
hcu2
hcu1
JSPAT1
JSPAT5
JSPAT3
JSPAT7
JSPAT0
JSPAT4
JSPAT2
JSPAT6
BETA6-1

Total
23.23
23.22
21.79
21.72
21.00
19.90
19.83
19.55
19.14
18.38
18.37
10.47
10.44
10.39
10.36
10.21
10.19
9.78
9.62
9.56
9.54
6.92

NTC4
28.28
28.34
28.12
28.56
24.22
30.14
34.09
29.93
29.73
21.37
27.79
12.36
12.24
12.11
11.98
11.80
11.69
20.52
19.38
13.92
13.91
14.39

NTC5
23.91
23.91
22.43
22.24
21.65
20.11
19.86
19.67
19.17
19.16
18.52
10.92
10.90
10.83
10.81
10.63
10.61
9.69
9.54
9.99
9.97
6.94

SR NTC6
18.32
—
18.25
—
16.80
—
17.19
—
16.74
—
16.12
—
15.91
—
16.20
—
16.25
—
13.50
—
15.10
—
7.58
—
7.57
—
7.60
—
7.59
—
7.60
—
7.59
—
7.39
—
7.47
—
6.10
—
6.07
—
4.86
—

Total
5.98
5.83
5.74
4.98
4.54
4.34
4.22
3.93
3.78
3.66
3.01
2.56
2.54
1.96
1.95
1.95
1.94
1.87
1.86
1.76
1.75
0.14

NTC4 NTC5 SR NTC6
7.39 10.13 11.32 4.29
9.17
9.09 12.58 4.25
9.23
9.03 11.88 4.20
6.45
7.87 10.16 3.67
10.32 8.25 8.46 2.99
8.40
6.32 6.33 3.48
9.35
7.25 9.01 2.82
8.69
6.99 5.51 2.77
7.37
6.15 6.65 2.76
10.03 5.36 7.45 2.71
8.15
4.71 3.69 2.31
3.17
4.41 6.75 1.67
3.19
4.34 6.23 1.70
5.49
2.97 2.54 1.52
5.45
2.96 2.48 1.51
6.06
2.44 1.22 1.76
6.04
2.43 1.16 1.75
5.66
2.64 2.50 1.51
5.63
2.63 2.43 1.49
13.08 2.18 0.95 1.42
13.06 2.17 0.91 1.41
1.37
0.13 0.00 0.13

Proceedings of NTCIR-6 Workshop Meeting, May 15-18, 2007, Tokyo, Japan

Table 8. MAP for Japanese Retrieval Subtask: Def1, Rel=HA
Run ID
HTC10
HTC09
HTC08
HTC07
HTC06
HTC05
HTC04
HTC02
AFLAB1
HTC03
HTC01
hcu1
hcu2
JSPAT3
JSPAT7
JSPAT1
JSPAT0
JSPAT5
JSPAT4
JSPAT2
JSPAT6
BETA6-1

Total
12.63
12.46
11.87
11.84
11.50
10.60
9.88
9.27
9.25
9.19
9.02
5.33
5.26
4.89
4.87
4.81
4.79
4.78
4.76
4.36
4.34
3.61

NTC4
19.84
19.70
18.12
17.39
15.32
17.34
19.50
17.82
12.59
18.47
15.13
10.77
10.35
6.56
5.71
4.92
5.02
4.16
4.23
8.32
8.22
7.51

NTC5
16.66
16.52
15.74
15.82
15.41
14.39
13.92
12.88
13.24
12.52
12.54
7.01
7.03
6.96
6.95
6.63
6.62
6.60
6.59
6.36
6.35
3.86

SR NTC6
17.33 9.65
17.14 9.46
16.02 9.06
16.12 8.97
15.61 8.73
14.24 7.91
13.12 7.09
12.65 6.68
11.77 6.69
13.01 6.68
11.64 6.65
6.58 4.16
6.44 4.06
4.90 3.78
4.88 3.76
5.45 3.75
5.40 3.73
5.43 3.73
5.39 3.71
4.32 3.25
4.30 3.23
4.72 3.24

Table 9. MAP for Japanese Retrieval Subtask: Def1, Rel=HAB
Run ID
HTC10
HTC09
HTC08
HTC07
HTC06
HTC05
HTC04
HTC02
HTC03
AFLAB1
HTC01
hcu1
hcu2
JSPAT3
JSPAT0
JSPAT7
JSPAT1
JSPAT4
JSPAT5
JSPAT2
JSPAT6
BETA6-1

Total
15.41
15.29
14.45
14.41
13.96
12.96
12.12
11.72
11.66
11.46
11.13
6.49
6.37
6.26
6.23
6.22
6.21
6.18
6.17
5.61
5.58
4.89

NTC4
26.36
26.27
25.31
24.85
22.66
24.18
25.96
24.09
24.11
16.15
21.23
15.67
15.30
12.06
10.95
11.12
10.73
10.08
9.87
11.51
11.17
11.31

NTC5
19.73
19.67
18.56
18.47
17.95
16.60
16.30
15.77
15.48
15.39
15.03
8.22
8.06
8.77
8.54
8.75
8.55
8.50
8.52
8.06
8.04
5.86

SR
18.45
18.34
17.17
17.44
16.63
15.99
14.38
14.05
14.43
13.27
12.76
7.52
7.47
6.47
6.65
6.45
6.66
6.63
6.64
5.30
5.27
5.98

NTC6
11.51
11.34
10.77
10.71
10.41
9.53
8.43
8.12
8.15
8.21
7.83
4.88
4.77
4.31
4.41
4.28
4.38
4.37
4.34
3.81
3.79
3.85

Table 10. MAP for English Retrieval Subtask.
Run ID
AFLAB1
AFLAB2
AFLAB3
hcu1
hcu2
JSPAT0
JSPAT1
JSPAT2
JSPAT3
KLE1
NTNU

Type
Mandatory
Optional
Optional
Mandatory
Mandatory
Mandatory
Mandatory
Optional
Optional
Mandatory
Mandatory

A
3.65
4.17
3.81
3.37
3.37
1.27
1.26
1.23
1.22
2.82
2.30

AB
7.12
8.11
7.48
2.12
2.10
1.85
1.84
5.72
4.50
6.94
6.94

References[1] A. Fujii, M. Iwayama, and N. Kando. Overview ofpatent retrieval task at NTCIR-4. In Proceedings ofthe Fourth NTCIR Workshop on Research in Information Access Technologies Information Retrieval, Question Answering and Summarization, 2004.
[2] A. Fujii, M. Iwayama, and N. Kando. The patent retrieval task in the fourth NTCIR workshop. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information
Retrieval, pages 560–561, 2004.
[3] A. Fujii, M. Iwayama, and N. Kando. Test collections
for patent-to-patent retrieval and patent map generationin NTCIR-4 workshop. In Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 1643–1646, 2004.
[4] A. Fujii, M. Iwayama, and N. Kando. Overview ofpatent retrieval task at NTCIR-5. In Proceedings ofthe Fifth NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval,
Question Answering and Cross-lingual Information Access, pages 269–277, 2005.
[5] A. Fujii, M. Iwayama, and N. Kando. Test collections
for patent retrieval and patent classification in the fifth
NTCIR workshop. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 671–674, 2006.
[6] M. Iwayama, A. Fujii, and N. Kando. Overview ofclassification subtask at NTCIR-5 patent retrieval task.
In Proceedings of the Fifth NTCIR Workshop Meetingon Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Crosslingual Information Access, pages 278–286, 2005.
[7] M. Iwayama, A. Fujii, and N. Kando. Overview ofclassification subtask at NTCIR-6 patent retrieval task.
In Proceedings of the Sixth NTCIR Workshop Meeting,
2007.
[8] M. Iwayama, A. Fujii, N. Kando, and Y. Marukawa.
Evaluating patent retrieval in the third NTCIR workshop.
Information Processing & Management,
42(1):207–221, 2006.