Proceedings of the Third NTCIR Workshop

NTCIR-3 Patent Retrieval Experiments at ULIS
Atsushi Fujii†,†† Tetsuya Ishikawa†
University of Library and Information Science
1-2 Kasuga, Tsukuba, 305-8550, Japan††
CREST, Japan Science and Technology Corporation
fujii@ulis.ac.jp†

Abstract
Given the growing number of patents filed in multiple countries, users are interested in retrieving patents
across languages. We propose a multi-lingual patent
retrieval system, which translates a user query into thetarget language, searches a multilingual database forpatents relevant to the query, and improves the browsing efficiency by way of machine translation and clustering. Our system also extracts new translations frompatent families consisting of comparable patents, toenhance the translation dictionary.
Keywords: multi-lingual patent retrieval, machinetranslation, document clustering, translation extraction, patent families

1Introduction

Given the growing number of patents filed in multiple countries, it is feasible that users are interested in
retrieving patent information across languages. However, many users find it difficult to perform patent retrieval (i.e., formulating queries, searching databases
for relevant patents, and browsing retrieved patents) inforeign languages.
To counter this problem, cross-language information retrieval (CLIR), where queries in one language
are submitted to retrieve documents in another language, can be an effective solution. CLIR has of latebecome one of the major topics within the information retrieval and natural language processing communities. In fact, a number of methods/systems for CLIR
have been proposed.
Since by definition queries and documents are indifferent languages, queries and documents need to
be standardized into a common representation, so thatmonolingual retrieval techniques can be applied. From
this point of view, existing CLIR methods are classified into the following three fundamental categories.
The first method translates queries into the document language [1, 8, 17], and the second method translates documents into the query language [16, 18]. The© 2003 National Institute of Informatics

third method projects both queries and documents intoa language-independent space by way of thesaurus
classes [10, 21] and latent semantic indexing [3, 14].
Among those above methods, the first method (i.e.,query translation method) is preferable, because thisapproach can simply be combined with existing monolingual retrieval systems.
Following a query translation method [6, 8],
we previously proposed a Japanese/English crosslanguage patent retrieval system [9], where users submit queries in either Japanese or English to retrieve
patents in the other language. In either case, the target
database is monolingual.
However, since users are not always sure as towhich language database contains patents relevant totheir information need, it is effective to retrieve patents
in multiple languages simultaneously. This process,which we shall call “multi-lingual information retrieval (MLIR)”, is an extension of CLIR. For this purpose, we proposed a Japanese/English multi-lingual
patent retrieval system called “PRIME” (Patent Retrieval In Multi-lingual Environment) [11].
The design of our system is based on that for technical documents [7], which combines query translation,
document retrieval, document translation and clustering modules.
Additionally, we introduced a module for enhancing a dictionary used for the query translation module. For this purpose, we proposed a method to extract Japanese/English translations from patent families consisting of comparable patents filed in Japan andthe United States.

2 System Description
2.1 Overview
Figure 1 depicts the overall design of PRIME,
which retrieves documents in response to user queries
in either Japanese or English. However, unlike the caseof CLIR, retrieved documents can potentially be in either a combination of Japanese and English or either

The Third NTCIR Workshop, Sep.2001 - Oct. 2002of the languages individually. We briefly explain theentire on-line process based on this figure.
First, a user query is translated into the foreign language (i.e., either Japanese or English) by way of aquery translation module.
Second, a document retrieval module uses boththe source (user) and translated queries to search aJapanese/English bilingual patent collection for relevant documents.
In real world usage, Japanese and English patents
are not comparable in the collection (this is the major reason why cross/multi-lingual retrieval is needed).
However, for the purpose of research and development, we currently target a comparable collection.
To put it more precisely, the collection contains approximately 1,750,000 pairs of Japanese abstracts andtheir English translations, which were provided on PAJ(Patent Abstract of Japan) CD-ROMs in 1995-19991 .
Third, among retrieved documents, only those thatare in the foreign language are translated into the user
language through a document translation module.
In principle, we need only above three modules torealize multi-lingual patent retrieval in the sense thatusers can retrieve/browse foreign documents throughtheir native language. However, to improve the browsing efficiency, a clustering module finally divides retrieved documents into a specific number of groups.
Additionally, in the off-line process, a translation
extraction module identifies Japanese/English translations in the database, to enhance the query translation
module.

2.2

Query Translation

The query translation module is based on themethod proposed by Fujii and Ishikawa [6, 8], which
has been applied to Japanese/English CLIR for the NTCIR collection consisting of technical abstracts [13].
This method translates words and phrases (compound words) in a given query, maintaining the word
order in the source language. A preliminary study
showed that approximately 95% of compound technical terms defined in a bilingual dictionary [5] maintainthe same word order in both Japanese and English.
Then, the Nova dictionary2 is used to derive possible word/phrase translations, and a probabilistic
method is used to resolve translation ambiguity.
The Nova dictionary includes approximately one
million Japanese-English translations related to 19
technical fields as listed below:
aeronautics, biotechnology, business, chemistry, computers, construction, defense,
ecology, electricity, energy, finance, law,mathematics, mechanics, medicine, metals,
oceanography, plants, trade.
1 Copyright by Japan Patent Office.
2 Developed by NOVA, Inc. http://www.nova.co.jp/

Query

Query translation

Dictionary

Translation
model

Language
model

Document retrieval

Document translation

Japanese/English
patent collection

Translation Extraction

Clustering

Clusters

Figure 1. The design of PRIME: our multilingual patent retrieval system (dashedarrows denote the off-line process).

In addition, for words unlisted in the Nova dictionary, transliteration is performed to identify phonetic
equivalents in the target language. Since Japaneseoften represents loanwords (i.e., technical terms andproper nouns imported from foreign languages) using
its special phonetic alphabet (or phonogram) called“katakana”, with which new words can be spelled out,transliteration is effective to improve the translation
quality.
We represent the user query and one translation
candidate in the document language by U and D, respectively. From the viewpoint of probability theory,
our task here is to select D’s with greater probability,
P (D|U ), which can be transformed as in Equation (1)through the Bayesian theorem.
P (D|U ) =P (U |D) · P (D)P (U )(1)

In practice, P (U ) can be omitted because this factor is a constant with respect to the given query, andthus does not affect the relative probability for different translation candidates.
P (D) is estimated by a word-based bi-gram language model produced from the target collection.
P (U |D) is estimated based on the word frequency obtained from the Nova dictionary. Those two factors
are commonly termed language and translation models, respectively (see Figure 1).

Proceedings of the Third NTCIR Workshop

2.3

Document Retrieval

The retrieval module is based on an existing probabilistic retrieval method [20], which computes the relevance score between the translated query and eachdocument in the collection. The relevance score fordocument i is computed based on Equation (2).



T
F
N
t,i


(2)
· log
DLi + T F
DF
t
t
t,i
avglenHere, T Ft,i denotes the frequency that term t appears
in document i. DFt and N denote the number of documents containing term t and the total number of documents in the collection. DLi denotes the length of
document i (i.e., the number of characters containedin i), and avglen denotes the average length of documents in the collection.
For both Japanese and English collections, we use
content words extracted from documents as terms, andperform a word-based indexing. For the Japanese collection, we use the ChaSen morphological analyzer3
to extract content words. However, for the English
collection, we extract content words based on partsof-speech as defined in WordNet [4].

2.4Document Translation

The document translation module consists of thethe PatTranser Japanese/English MT system, whichuses the same dictionary used for the query translation
module.
In practice, since machine translation is computationally expensive and degrades the time efficiency, we
perform machine translation on a phrase-by-phrase basis. In brief, phrases are sequences of content words indocuments, for which we developed rules to generate
phrases based on the part-of-speech information. Thismethod is practical because even a word/phrase-basedtranslation can potentially improve on the efficiency
for users to find relevant foreign documents from thewhole retrieval result [19].

2.5Document Clustering

For the purpose of clustering retrieved documents,
we use the Hierarchical Bayesian Clustering (HBC)
method [12], which merges similar items (i.e., documents in our case) in a bottom-up manner, until all theitems are merged into a single cluster. Thus, a specificnumber of clusters can be obtained by splitting the resultant hierarchy at a predetermined level.
The HBC method also determines the most representative item (centroid) for each cluster. Thus, we can
3 http://chasen.aist-nara.ac.jp/

enhance the browsing efficiency by presenting only
those centroids to users.
The similarity between documents is computed
based on feature vectors that characterize each document. In our case, vectors for each document consistof frequencies of content words appearing in the document. We extract content words from documents asperformed in word-based indexing (see Section 2.3).
Given the clustering module, the system can facilitate an interactive retrieval. To put it more precisely, through the interface, users can discard irrelevant clusters determined by browsing representative
documents, and re-cluster the remaining documents.
By performing this process recursively, relevant documents are eventually remained.

2.6 Extracting Translations Using Patent
Families
Since patents are usually associated with newwords, it is crucial to translate out-of-dictionary
words. The transliteration method used in the querytranslation module is one solution for this problem(see Section 2.2).
On the other hand, it is also effective to update thetranslation dictionary. For this purpose, a number ofmethods to extract translations from bilingual (parallel/comparable) corpora [22, 23] are applicable. However, it is considerably expensive to obtain bilingual
corpora with sufficient volume of alignment information.
To resolve this problem, we use patent families,
which are patent sets filed for the same/related contents in multiple countries, as comparable corpora.
Thus, patents contained in the same family are not necessarily parallel, but quite comparable.
Among a number of ways to apply for patents in
multiple countries, we focus solely on patents claiming priority under the Paris Convention, because we
can easily identify patent families by the identification
number assigned to each patent.
In addition, the number of patent families is still
increasing. Thus, we can easily update a large-scalebilingual comparable corpus based on patent families.
To the best of our knowledge no research has utilized
patent families for extracting translations.
Since patents are structured with a number of fields(e.g., titles, abstracts, and claims), our method first
identifies corresponding fragments based on the document structure, to improve the extraction accuracy.
However, structures of paired patents are not always
the same. For example, the number of fields claimed
in a single patent family often varies depending on
the language. Thus, we use only the title and abstract
fields, which usually parallel in Japanese and English
patents. In other words, unlike the case of most existing extraction methods, our method does not needThe Third NTCIR Workshop, Sep.2001 - Oct. 2002

sentence-aligned corpora.
We use the ChaSen morphological analyzer [15]and Brill tagger [2] to extract content words fromJapanese and English fragments, respectively. In addition, we combine more than one word into phrases, forwhich we developed rules to generate phrases based on
the part-of-speech information.
We then compute the association score for all thepossible combinations of Japanese/English phrases cooccurring in the same fragment, and select those withgreater score as the final translations. For this purpose,
we use the weighted Dice coefficient [23] as shown inEquation (3).
score(Wj , We ) = log Fje ·2Fje
Fj + Fe(3)

Here, Wj and We are Japanese and English phrases,
respectively. Fj and Fe denote the frequency that Wj
and We appear in the entire corpus, respectively. Fje
denotes the frequency that Wj and We co-occur in thesame fragment. The logarithm factor is effective to
discard infrequent co-occurrences, which usually decrease the extraction accuracy.

3

Experimentation

3.1

Overview

We evaluated our patent retrieval system with respect to the following two different perspectives. First,
we used the NTCIR-3 Patent Retrieval test collection, which consists of 31 topics and 697,262 Japanese
patents filed in 1998-1999, and evaluated our system for Japanese monolingual IR. Second, we used
Japanese-US patent families to evaluate the performance of our translation extraction method.

3.2

Results in The NTCIR-3 Formal Run

In the NTCIR-3 Patent Retrieval test collection,
topics contain a number of fields, such as article, supplement, title, description, narrative and concept, irrespective of the language. In the mandatory run, eachsystem participated in the Patent task must submit a
result obtained with a combination of the article andsupplement fields. However, in the optional run, any
fields could be used as queries.
Relevance assessment was performed based on
three ranks of relevance, that is, relevant, partially relevant and irrelevant.
Since patent documents are fairly long, we used
only abstracts and claims to produce an index. We
used words and bi-words (i.e., word-based bigrams)
as index terms.
Table 1 shows non-interpolated average precision
and R-precision values, averaged over the 31 queries,

for different methods. Although all the methods in Table 1 was fully-automated, topic fields used as queries
were different depending on the method. While in thecase of “Rigid” documents judged “relevant” were regarded as correct answers, in the case of “Relax” documents judged “partial relevant” were also regarded
as correct answers. In Table 1, while “words” is thecase where only words were used as index terms, “biwords” is the case where both words and bi-words
were used as index terms. Suggestions which can bederived from Table 1 are as follows.
First, in the mandatory run, where the article (A)and supplement (S) fields were used as queries, average precision values were fairly low. In fact, thesevalues were relatively low, compared with results obtained with other participating systems. The rationale
behind this observation includes that we did not preprocess topic fields, and thus a large number of irrelevant words in article fields (which are newspaper articles) decreased the retrieval accuracy.
Second, in the optional run, topic fields used as
queries were more influential for the retrieval accuracy
than the indexing method. For example, the average
precision and R-precision values obtained with only
the description fields were greater than those obtainedwith other fields, irrespective of the indexing method.
Third, in the case where only description fields
were used as queries, the best result was obtained
throughout this experiment.
Finally, an indexing method relying on bi-words
was more effective than a word-based indexing
method only in the mandatory run. In other words,
the effect of noisy query terms in article fields wasovershadowed by way of bi-words. However, in theoptional run, where topic fields used as queries wererelatively well-organized, the contribution of bi-words
was not observable. In addition, the computational
cost required for bi-word-based indexing was expensive.
Table 1. Non-interpolated average precision and R-precision values, averaged
over the 31 queries, for different topic
fields (A: article, S: supplement, T: title,
D: description, N: narrative).
Field
A, S
A, S
D
D
D, N
D, N
T
T

Index
word
bi-wordword
bi-wordword
bi-wordword
bi-word

Avg. Precision
Rigid Relax.0682 .0615.0762 .0778.1384 .1683.1325 .1559.1318 .1440.1316 .1337.0869 .1214.0841 .1050

R-Precision
Rigid Relax.0985 .0997.1108 .1140.1571 .2045.1498 .1904.1746 .1815.1696 .1901.1105 .1562.1162 .1379

Proceedings of the Third NTCIR Workshop

3.3

Evaluating Translation Extraction

A preliminary study showed that out of approximately 1,750,000 patents filed in Japan (1995-1999),
approximately 32,000 patents were paired with those
filed in the United States as patent families. Thus,in practice we obtained a bilingual comparable corpus
consisting of 32,000 Japanese/English pairs. From this
corpus, our method extracted 1,234,347 phrase-basedtranslations, which were judged it correct or incorrect.
However, we selected translations whose score wasabove 1.5, and manually judged their correctness, because a) the judgment can be considerably expensive
for the entire translations, and b) translations withsmall association scores are usually incorrect. The total number of selected translations was 37,669.
We then evaluated the accuracy of our extraction
method. The accuracy is the ratio between the number of correct translations, and the number of cases
where the association score of the translation is abovea specific threshold. By raising the value of the threshold, the accuracy also increased, while the number ofextracted translations decreased, as shown in Table 2.
According to this table, we could achieve a high accuracy by limiting the number of translations extracted.
We spent only four man-days in judging the 37,669
translations and identifying 5,879 correct translations.
In other words, our method facilitated to produce bilingual lexicons semi-automatically with a trivial cost.

Table 2. Accuracy of translation extraction.
Threshold for Score
1.5
2.0 3.0 4.0 5.0# of Translations
37,669 24,869 4,419 962 356# of Correct Translations 5,879 4,129 1,399 564 240
Accuracy (%)
15.6 16.6 31.7 58.6 67.4

4

Summary

In this paper, we described our multi-lingual system
for Japanese/English patent retrieval. For this purpose,
we used a query translation method explored in crosslanguage information retrieval (CLIR). However, unlike the case of CLIR, our system retrieves bilingual
patents simultaneously in response to a monolingual
query. Our system also summarizes retrieved patents
by way of machine translation and clustering to improve the browsing efficiency. In addition, our system
includes an extraction module which produces newtranslations from patent families consisting of comparable patents, and updates the translation dictionary.
Future work would include improving existing
modules in our system, and the application of ourframework to other languages.

References[1] L. Ballesteros and W. B. Croft. Resolving ambiguity for cross-language retrieval. In Proceedings of the
21st Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval,
pages 64–71, 1998.
[2] E. Brill. Transformation-based error-driven learningand natural language processing: A case study inpart-of-speech tagging. Computational Linguistics,
21(4):543–565, 1995.
[3] J. G. Carbonell, Y. Yang, R. E. Frederking, R. D.
Brown, Y. Geng, and D. Lee. Translingual information
retrieval: A comparative evaluation. In Proceedings ofthe 15th International Joint Conference on Artificial
Intelligence, pages 708–714, 1997.
[4] C. Fellbaum, editor. WordNet: An Electronic Lexical
Database. MIT Press, 1998.
[5] G. Ferber. English-Japanese, Japanese-English Dictionary of Computer and Data-Processing Terms. MIT
Press, 1989.
[6] A. Fujii and T. Ishikawa. Cross-language information retrieval for technical documents. In Proceedings of the Joint ACL SIGDAT Conference on Empirical Methods in Natural Language Processing and Very
Large Corpora, pages 29–37, 1999.
[7] A. Fujii and T. Ishikawa. Evaluating multi-lingual information retrieval and clustering at ULIS. In Proceedings of the 2nd NTCIR Workshop Meeting on
Evaluation of Chinese & Japanese Text Retrieval andText Summarization, 2001.
[8] A. Fujii and T. Ishikawa. Japanese/English crosslanguage information retrieval: Exploration of querytranslation and transliteration. Computers and the Humanities, 35(4):389–420, 2001.
[9] M. Fukui, S. Higuchi, Y. Nakatani, M. Tanaka, A. Fujii, and T. Ishikawa. Applying a hybrid query translation method to Japanese/English cross-language patent
retrieval. In ACM SIGIR Workshop on Patent Retrieval, 2000.
[10] J. Gonzalo, F. Verdejo, C. Peters, and N. Calzolari. Applying EuroWordNet to cross-language text
retrieval. Computers and the Humanities, 32:185–207,
1998.
[11] S. Higuchi, M. Fukui, A. Fujii, and T. Ishikawa.
Prime: A system for multi-lingual patent retrieval. In
Proceedings of MT Summit VIII, pages 163–167, 2001.
[12] M. Iwayama and T. Tokunaga. Hierarchical Bayesianclustering for automatic text classification. In Proceedings of the 14th International Joint Conference on
Artificial Intelligence, pages 1322–1327, 1995.
[13] N. Kando, K. Kuriyama, and T. Nozue. NACSIS testcollection workshop (NTCIR-1). In Proceedings of the
22nd Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval,
pages 299–300, 1999.
[14] M. L. Littman, S. T. Dumais, and T. K. Landauer.
Automatic cross-language information retrieval using
latent semantic indexing. In G. Grefenstette, editor, Cross-Language Information Retrieval, chapter 5,
pages 51–62. Kluwer Academic Publishers, 1998.

The Third NTCIR Workshop, Sep.2001 - Oct. 2002[15] Y. Matsumoto, A. Kitauchi, T. Yamashita, Y. Hirano,
H. Matsuda, and M. Asahara. Japanese morphologicalanalysis system ChaSen version 2.0 manual 2nd edition. Technical Report NAIST-IS-TR99009, NAIST,
1999.
[16] J. S. McCarley. Should we translate the documents or
the queries in cross-language information retrieval? In
Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 208–214,
1999.
[17] J.-Y. Nie, M. Simard, P. Isabelle, and R. Durand.
Cross-language information retrieval based on paralleltexts and automatic mining of parallel texts from theWeb. In Proceedings of the 22nd Annual International
ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74–81, 1999.
[18] D. W. Oard. A comparative study of query and document translation for cross-language information retrieval. In Proceedings of the 3rd Conference of the
Association for Machine Translation in the Americas,
pages 472–483, 1998.
[19] D. W. Oard and P. Resnik. Support for interactive document selection in cross-language information retrieval. Information Processing & Management,
35(3):363–379, 1999.
[20] S. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the
17th Annual International ACM SIGIR Conference on
Research and Development in Information Retrieval,
pages 232–241, 1994.
[21] G. Salton. Automatic processing of foreign language
documents. Journal of the American Society for Information Science, 21(3):187–194, 1970.
[22] F. Smadja, K. R. McKeown, and V. Hatzivassiloglou.
Translating collocations for bilingual lexicons: A statistical approach. Computational Linguistics, 22(1):1–
38, 1996.
[23] K. Yamamoto and Y. Matsumoto. Acquisition ofphrase-level bilingual correspondence using dependency structure. In Proceedings of the 18th International Conference on Computational Linguistics,
pages 933–939, 2000.