BMIR-J2: A Test Collection for Evaluation of Japanese
Information Retrieval Systems
Tetsuya Sakai (Toshiba) 3
Tsuyoshi Kitani (NTT DATA)
Yasushi Ogawa (Ricoh)
Tetsuya Ishikawa (Univ. of Library and Information Science)
Haruo Kimoto (NTT)
Ikuo Keshi (SHARP)
Jun Toyoura (Mitsubishi Electric)
Toshikazu Fukushima (NEC)Kunio Matsui (Fujitsu Laboratories)
Yoshihiro Ueda (Fuji Xerox)
Takenobu Tokunaga (Tokyo Institute of Technology)
Hiroshi Tsuruoka (ERI, Univ. of Tokyo)
Hidekazu Nakawatase (NTT)
Teru Agata (Keio Univ.)
Noriko Kando (NACSIS)

BMIR-J2 is the rst complete test collection generally available for evaluating Japanese information retrieval systems. BMIR-J2 features include anovel division of search requests based on various functions required to perform successful retrieval. BMIR-J2
and its smaller predecessor BMIR-J1 were constructed by
a volunteer-based working group under the Information
Processing Society of Japan. We hope that BMIR-J2 will
come into wide use and that it will foster the developmentof Japanese IR systems.
Abstract

1

Introduction

In Europe and the US, there are a variety of standard
test collections available for objective evaluation of information retrieval(IR) systems [2]. However, in spiteof the growing interest in IR research in Japan, no testcollection for Japanese existed until 1993. Unlike European languages, there are no explicit word boundaries inJapanese text. Moreover, Japanese has several characterclasses, each of which ful ll di erent linguistic functions.
Because of these language speci c problems, Japanese
test collections are indispensable for fostering researchand development of Japanese IR systems.
In April 1993, we formed a working group under theSpecial Interest Group on Database Systems of the Information Processing Society of Japan(IPSJ SIGDBS), to
design and construct Japanese test collections. In March
1996, with kind understanding from Nihon Keizai Shimbun, Inc., we released BMIR-J1, the rst ever Japanese
test collection [8] [11] . Although BMIR-J1 was a small
preliminary collection, we collected many useful comments from the monitor users. This positive response en3 Human Interface Laboratory, Toshiba R&D Center: email:tetsuya.sakai@toshiba.co.jp

couraged the members of the working group to constructa much larger collection BMIR-J2, with revised search requests and criteria for relevance assessments. With kindunderstanding from Mainichi Shimbun and in collaboration with the government-funded Real World Computing
Partnership(RWCP) [5], we released BMIR-J2 in March
1998 [10].
This paper introduces BMIR-J2 to the general IR
community. It describes the issues we addressed and ourexperiences in building and using test collections. We
hope that this paper encourages other researchers to experiment with BMIR-J2. Section 2 describes the featuresof BMIR-J2. Section 3 summarizes what we have learnt
from building our preliminary collection BMIR-J1. Section 4 describes the construction process of BMIR-J2.
Section 5 provides a discussion on the unique feature of
BMIR-J2, namely request complexity groups. Section 6
gives conclusions.
2
2.1

Features of BMIR-J2
Overview

BMIR-J2 consists of 5,080 newspaper article IDs and
60 search requests, each with a list of relevant article
IDs. The article IDs are as de ned by Mainichi Shimbun,
Inc., one of the major newspaper publishing companies
in Japan. For copyright reasons, BMIR-J2 contains only
the article IDs, so users must buy a copy of the Mainichi
Shimbun CD-ROM'94 in order to use the actual contents of the articles for retrieval experiments. More details, including contact information for BMIR-J2 and theMainichi Shimbun CD-ROM'94, can be found at http://www.ulis.ac.jp/~ishikawa/bmir-j2/eindex.html.

2.2

Search Requests

A BMIR-J2 search request is a noun phrase describing auser's needs, which is augmented by a narrative, in TREC
parlance. A narrative typically consists of a few sentencesclarifying the meaning of the noun phrase and criteria
for relevance assessments. For example, the narrative forthe request \ðç (satellite broadcasting)", in roughtranslation, is provided as follows:
Satellite broadcasting includes broadcasting bybroadcast satellites and that by communication
satellites. Even if a document contains keywords
such as ``satellite'' or ``TV with a BS tuner,''
it is considered nonrelevant unless it mentions
satellite broadcasting itself.

Compared to existing standard test collections for English, BMIR-J2 is unique in that the search requests are
explicitly categorized into six request complexity groups
according to various functions which the evaluated system might use to retrieve the relevant documents. Table
1 provides, for each complexity group, a brief description and an example of a request in translation. It is
supposed that as we move along from group A to F, therequests become harder to satisfy and thus the average
retrieval performance is likely to degrade. This feature
was inherited from BMIR-J1 [11].
2.3

Relevance Levels

We provide three levels of relevance, namely, A, B and
C. In an A-relevant document, the content of the searchrequest is discussed as the main topic. Whereas in a Brelevant document, the content is only mentioned brie y
at some point within the document. In evaluation of retrieval performance, users can choose whether to use Arelevant documents only or to use both A-relevant andB-relevant documents as the relevant sets, depending on
their tasks and needs. In addition, we provide documents with relevance level C, which typically containsome terms that are associated with the search request
but was eventually judged nonrelevant by the relevanceassessors.
2.4

Statistical Soundness

Search requests with few relevant documents tend to inuence the overall retrieval performance greatly. Therefore, in average-based evaluation using BMIR-J2, we encourage the users to use the standard set, that contain
50 requests with at least 5 relevant documents. The additional set, that contain the remaining 10 requests withfew relevant documents, is provided as a separate le,and may be useful for close per-request analysis.
Table 2 summarizes and supplements the aforementioned information about BMIR-J2, together with information on the preliminary BMIR-J1 and TREC-4 collections for comparison [2]. It can be observed that
BMIR-J2 is a small collection by current TREC standards. However, we believe that the relevance assessment
data it contains are consistent and reliable. Although it
is dicult to derive statistically sound results from sucha small test collection, we argue that looking at general trends in the results can still be useful for exploring
di erent retrieval techniques.
3

Lessons from BMIR-J1

The main purpose of releasing BMIR-J1 was to collectopinions from monitor users and set directions for constructing BMIR-J2. The 600 articles (See Table 2) wererandomly chosen from the economics sections of Nihon
Keizai Shimbun newspapers from 1993, and the searchrequests were devised by the working group members bybrowsing through the small document collection. The
limited size of the collection enabled us to perform exhaustive relevance assessments for each search request
without adopting a pooling method. Two relevance assessors were assigned to each request, who consulted thewhole working group whenever there was disagreementin their assessments.
BMIR-J1 was released to 50 sites, 32 of which lled
out and returned a questionnaire by e-mail to the working group. This included 11 sites to which each of theworking group members belonged. An analysis of thequestionnaire showed that the users found BMIR-J1 to
be a useful test collection with reliable relevance assessments. However, as expected, many users said that weshould enlarge the size of the collection for the sake ofstatistical soundness. Some also said that many of thesearch requests of BMIR-J1 are too hard for existing retrieval systems to deal with. There were also some otherinteresting opinions, some of which are summarized inTable 3.
4

Construction of BMIR-J2

The overall success of BMIR-J1 encouraged us to startworking on the complete collection. Although other kindsof text sources such as patents and technical papers werealso considered to be of importance by many monitor
users, lack of funding and limited availability of text resources made us nally settle on newspaper articles again.
With kind understanding from Mainichi Shimbun we decided to use a subset of Mainichi Shimbun newspaper
articles from 1994. This enabled us to reuse many of thesearch requests developed for BMIR-J1. Table 4 presents
information on the search requests inherited from BMIRJ1 and those newly developed for BMIR-J2. For someof the reused requests, the noun phrases and narratives
were modi ed in order to disambiguate them and also toadjust the number of relevant documents.
Most of the requests newly developed for BMIR-J2
are relatively easy from the viewpoint of request complexity. This was intended to be a feedback from thequestionnaire which suggested that BMIR-J1 contained
too many dicult requests by current standards. Table
5 provides information on the distribution of requests
over the complexity groups for BMIR-J1 and J2. It canbe observed that, compared to BMIR-J1, BMIR-J2 contains more requests in the easier groups(A through C)
and consequently fewer requests in the harder groups(Dthrough F).
The relevance assessment process of BMIR-J2 wasquite di erent from that of BMIR-J1 because of theGroupA
B
C
D
E
F

Table 1: Request Complexity Groups with Examples
Description of Functions Required Example Search Request
keyword search\liquid crystal"
query expansion using thesauri
numerical value/range comparison \companies that plan to lay omore than 1,000 employees"
syntactic analysis\construction of highways"
semantic/context analysis using\price reduction due to the rise of yen"linguistic knowledge
knowledge processing using\export from countries in Southeast Asia to Japan"
common sense/world knowledge
using both D and E\employment problems of women"Collection #Texts
TREC-4
567,529

Collection
BMIR-J1
BMIR-J2#Texts600
5,080

Table 2: Comparison of BMIR-J1, J2 and TREC-4
Average
Average
Average#Terms#Terms per#Relevantper Text#Requests
per Request
per Request
842.0
50
10
130
Average
Average
Average#Characters#A-relevant
Average#Characters
per Request
Averageplus#Characters
per Request(including#A-relevant #B-relevant
per Text#Requests (Noun Phrase) Narrative) per Request per Request
733.8
60
10.9
94.5
5.5
10.1
621.8
50+10(*1)
9.7(*2)
102.2
10.6
28.4

(*1) standard set + additional set.
(*2) The shortest noun phrase consists of 2 characters, and the longest consists of 26 characters.
enlarged size of the document collection. We had togive up the idea of exhaustive assessments and adopt aless resource-consuming method. Although the working
group itself had no funding, RWCP agreed to assign the
preliminary relevance assessment task to a subcontract
company and cover the expenses. BMIR-J2 was completed as follows:
1. The full texts of 5,080 articles were selected fromthe economics, engineering, and industrial technologies sections of Mainichi Shimbun newspapers
from 1994 as the document collection. Note that
BMIR-J2 has a broader domain than J1.
2. Some working group members had their own information retrieval systems that allowed simple
searches such as those using Boolean operators. For
each candidate request, some recall-oriented search
queries were devised by a relevance assessor anda working group member, and a set of candidate
documents were obtained using one of the systems.
3. Preliminary relevance assessments were performedby the assessor on the candidate documents only.
4. Each request was assigned to a working groupmember, who checked the relevance assessments returned by the assessors and corrected them whenever necessary. For some requests, the working
group asked the assessors to redo the relevance assessments after revising the recall-oriented query or
clarifying the criteria for relevance assessments.

5. Finally, for some of the requests, a new working
group member was assigned to cross-check the relevance assessments. As in the case of BMIR-J1,
if there was disagreement between members, they
consulted the whole working group.
5

Use of Request Complexity Groups

Because the search requests of BMIR-J2 are explicitly
categorized into six complexity groups, the user can focus his attention on a complexity group with a particularfunction or compare results across di erent complexity
groups. For example, if a system that performs fact retrieval using numerical comparisons is to be developed,group B can be used on its own for evaluation. Moreover, if the retrieval performance averaged over group D
is considerably lower than that over group C, for example, this may imply that the basic retrieval methodologyof the system is inadequate for advanced retrieval that
go beyond syntactic analysis.
Using the preliminary BMIR-J1 collection, [9] and[14] showed that requests in groups D through F are actually signi cantly harder than those in groups A throughC in terms of the traditional recall and precision measures. Recent work such as [15] showed that this is also
the case for BMIR-J2. If BMIR-J2 comes into wide use,cases may arise in which, for instance, system X outperforms system Y for a particular complexity group but notfor others.

Table 4: Reused/New Requests in BMIR-J2
Query Origin
Standard Additional Total
inherited from J126
632(not modi ed)
inherited from J1
11415(modi ed)
new
13
0
13
total
50
10
60

Table 3: Excerpt from the BMIR-J1 Questionnaire Summary(*1)#Usersexcluding
Question/Answer#Users WG Members
Q2:Are the relevance levels(A,B and C) useful?

yes
yes but
level C is unnecessary
single relevance levelis sucient
there should be
as many levels as possible
others
TOTAL

21

15

5

3

2

1

1
2
31

0
1
20

there should be more
60 is sucient
there are too manyothers
TOTAL

16
10
1
1
28

8
7
1
1
17

Q3:Is the number of searchrequests appropriate?

Q8-1:What kinds of text should we add
to our document collection?

patents
technical papers/
abstracts
magazine articles
encyclopedia
manuals
web pages
system diagnosis/ logs
law reports
texts in foreign languages/
multilingual texts
TOTAL(*1)standard set + additional set
6

Conclusions

14

11

15
3
3
3
3
2
2

5
2
2
3
1
2
2

2
47

2
30

16

9

4
2
2
2

3
2
2
1

Research in English-based information retrieval has a history of some half a century. Compared to this, Japanese
information retrieval is still in its infancy. Already, about
40 sites have completed the user registration of BMIRJ2, and we are beginning to see published research work
based on BMIR-J2 [1] [6] [7] [12] [13] [15] [16] [17] [18] .
We hope that BMIR-J2 will be widely used and that it
will foster the development of advanced IR systems forJapanese. We also hope that our experiences will help
in the development of other new test collections. In fact,
inspired by our activities, the rst TREC-like workshops
for retrieval of Japanese text, namely IREX [3] and NTCIR [4] will start in 1999. The test collections associatedwith these workshops will be much larger than BMIR-J2
with around two to three hundred thousand documents.
Some of the BMIR working group members are involvedin planning these new projects.
The BMIR working group itself stopped its activities
just after the release of BMIR-J2. Currently, the Third
Subcommittee (Hypertext and Information Retrieval)under IPSJ SIGDBS is in charge of matters concerning
BMIR-J2. The contact address is bmir@rd.nacsis.ac.
jp.

2

1

Acknowledgements

1

1

5
1
35

2
0
21

Q8-2:Which domains should we add
to our document collection?

engineering/
computer science
politics/
international
legal matterssociety/culture
hobbies/home a airs
advertisement/press releases
editorials/
survey articles
miscellaneous
domains
restricted domains
TOTAL

Table 5: Number of Requests in Each Complexity Group
Group BMIR-J1 BMIR-J2(*1)
A
10
14+0=14
B
5
3+1=4
C
6
10+1=11D
12
9+2=11
10
4+3=7
E
F
17
10+3=13
total
60
50+10=60(*1)For Q8-1 and Q8-2, the users were allowed to choose
more than one answer.

In the construction of BMIRJ1, Yoshifumi Masunaga(ULIS), Tomohiro Tanaka(NTT),
Tadanobu Miyauchi(Fuji Xerox) and Seiji Miike(Toshiba)
also took part as members of the working group. The
authors would like to thank Professor Katsumi Tanaka,
chair of IPSJ SIGDBS, for his continuous support of the
Benchmark Database Working Group.

References[1] Chun, Y. et al.: A Utility-based Information Retrieval System for User Information Usage - UBIR System, Proc. of The 3rd International Workshop on Information Retrieval with Asian Languages , pp. 22{27(1998).
[2] Harman, D.(Moderator): Panel:Building and Using
Test Collections, Proc. of ACM SIGIR '96 , pp. 335{
337 (1996).
[3] http://cs.nyu.edu/cs/projects/proteus/irex/
index-e.html[4] http://www.rd.nacsis.ac.jp/~ntcadm/index-en.
html[5] http://www.rwcp.or.jp/home-E.html[6] Jones, G. et al.: Experiments in Japanese Text Retrieval and Routing using the NEAT system, Proc. of
ACM SIGIR'98 , pp. 197{205 (1998).
[7] Jones, G. et al.: A Comparison of Query Translation
Methods for English-Japanese Cross-Language Information Retrieval, Proc. of ACM SIGIR '99 , to appear(1999).
[8] Kimoto, H. et al.: Construction of a test collection for
the evaluation of Japanese information retrieval systems(in Japanese), IPSJ Transactions , Vol. 40, No. 9,
to appear (1999).
[9] Kitani, T. et al.: Information Retrieval Using a
Full-text and Extracted Keywords(in Japanese), IPSJSIGFI Notes , 96{FI{43(96{NL{115), pp. 129{134(1996).
[10] Kitani, T. et al.: Lessons from BMIR-J2: A Test
Collection for Japanese IR Systems, Proc. of ACM
SIGIR '98 , pp. 345{346 (1998).
[11] Matsui, K. et al.: Test Collection for Information
Retrieval Systems from the Viewpoint of Evaluating
System Functions, Proc. of International Workshop
on Information Retrieval with Oriental Languages , pp.
42{47 (1996).
[12] Mochizuki, H. et al.: Passage-Level Document Retrieval Using Lexical Chains, IPSJ SIGFI Notes , 98{
FI{51, pp. 39{46 (1998).
[13] Ogawa, Y. et al.: Overlapping Statistical Segmentation for E ective Indexing of Japanese Text, Information Processing & Management, to appear.
[14] Sakai, T. et al.: Generation and Evaluation of Search
Queries Using Boolean Expressions and Document
Structure for Information Filtering(in Japanese), IPSJ
Transactions , Vol. 39, No. 11, pp. 3076{3083 (1998).
[15] Sakai, T. et al.: Application of Query Expansion
Techniques in Probabilistic Japanese News Filtering,

Proc. of The 3rd International Workshop on Information Retrieval with Asian Languages , pp. 46{55 (1998).

[16] Sakai, T. et al. : A Study on English-to-Japanese /Japanese-to-English Cross-Language Information Retrieval using Machine Translation (in Japanese), IPSJ
Transactions , submitted (1999).
[17] Sugai, T. et al.: The Hierarchical Information Filtering Method and its Evaluation (in Japanese), JSAI
12th National Conference Proceedings , pp. 390{393(1998).
[18] Yamada, H. et al.: A Proposal and Evaluation of a
Text Retrieval Method Based on Numerical Value Extraction from Text(in Japanese), IPSJ SIGFI Notes ,99{FI{53, pp. 17{22 (1999).