Cross-Lingual Information Access and Its Evaluation
Noriko Kando
Research and Development Department
National Center for Science Information Systems (NACSIS), Japan
URL: http://www.rd.nacsis.ac.jp/~{ntcadm,kando}/
Abstract
This paper introduces the outline of the first
NTCIR Workshop1 [1], August 30 - September 1,
1999, which is the first evaluation workshop
designed to enhance research in Japanese text
retrieval and cross-lingual information retrieval, thensuggests some thoughts on the future directions ofcross-lingual information access in the research anddevelopment of digital libraries.
Keywords: Cross-Lingual Information Retrieval,
NTCIR, Large-Scale Lexicon, Evaluation

1. Introduction
1.1 Cross-Lingual Information Retrieval
Cross-lingual information retrieval (CLIR) is asearch that a user submit a query in a language andthe system retrieves documents which may
represented in other languages as well as the
documents in the query language. As shown in Fig.1,
the term "multilingual information retrieval" mayrepresent CLIR in this sense, but it may represent apaired single language search. To avoid such
ambiguity, the term "Cross-Lingual" (or, CrossLanguage, or Cross-Linguistic) is used in the IR
community.

Cross-Lingual IR
EnglishQuery

Eng docs
Jap docs
FrenchChinesedocs

Multilingual IR: it may be cross-lingual, but
it may be a paired monolingual retrieval.
EnglishQuery

Eng docs

JapaneseQuery

Jap docs

Fig.1 Cross-Lingual vs Multilingual1This project is supported by “Research for the Future”
Program JSPS-RFTF96P00602 of the Japan Society for
the Promotion of Science

Importance of CLIR in digital libraries is
extremely increasing in the Internet environment. It
is more practical than translating every documents
into every other languages. A user can issue a queryin the most fluent language and avoid multiple
queries in each language. It would be useful for
people who are not fluent enough to create a queryin a foreign language, but who can read the language
to understand document contents and judge theirrelevance. CLIR also helps to reduce the cost of
manual translation by discarding irrelevant
documents before translation. Translating retrieved
documents may be unnecessary since (1) users may
be able to read other languages; (2) relevant
information objects may be non-textual ones, suchas photos, paintings, etc.; (3) to know the existenceof the documents may be sufficient for the users,
such as bibliometrics, visualization, etc.; and (4)translations may be provided, and so on.
Scholarly or scientific documents are one of the
focuses of research and implementation of the digital
libraries in these days. CLIR technique is critical in
information access and delivery of Japanesescholarly information since scholarly information
produced in Japan is substantially multilingual inboth document and lexical levels.
For example, a journal may contain both English
articles and Japanese articles. Japanese papers often
contain English abstracts and author-givenkeywords. In Japanese documents, a concept may
represented in four different forms; Japanese term,
English term in original spelling using romanalphabet, English acronyms, and transliterated formof English term using Japanese phonetic charactersof Katakana. Especially technical terms or new
concepts in Japanese papers are quite often
expressed in English or other foreign languages withoriginal spelling, and they are important as searchkeys. Word mismatch caused by the variation may
decrease the search effectiveness and to overcome
the problems, CLIR technique is needed[2].
1.2 Evaluation Workshop
An "evaluation workshop" usually provides a setof data usable for experiments and unified procedureof evaluation of experiment results. Eachparticipating research group conducts research andexperiments using the data provided in various
approaches. The first and one of the most successful
examples of the evaluation workshops in
information retrieval is the workshop series called
Text Retrieval Conference (TREC) organized by
National Institute of Standards and Technology(NIST) in the United States [4] since 1992.
Importance of large-scale standard test collections in
IR research has been widely recognized. Providing alarge scale test collection, which is equivalent to thesize of documents data searched in the operationalsetting, and unified evaluation procedures, various
new techniques have been developed through TREC,
and both exchanging research ideas among researchgroups and technology transfer from researchlaboratories to commercial products have been
facilitated so much.
Regarding evaluation of CLIR, TREC has hosted
Cross Language Retrieval Subtrack since 1996.
CLIR between English and European languages
such as Spanish, French, German, Italian has been
investigated and English-Mandarin Chinese isplaned for the next year, 2000. It has leaded theresearch and development of techniques of CLIR.
Topic Detection and Tracking (TDT) [5] uses
newswire documents on both English and Chinese.
Fundamental IR procedures like stopping, stemming,and query analysis contains language dependent
procedures. Therefore to test IR systems which canhandle Japanese documents, a large-scale testcollection containing Japanese documents is needed.
In the followings, the next section envisages theoutline of the first NTCIR Workshop, Section 3
introduces the research on CLIR at NACSIS, andSection 4 discusses the research trends towards
cross-lingual information access.

2. NTCIR Workshop
2.1 Overview of the NTCIR Workshop 1
The First NTCIR Workshop was held on August30-31, 1999, in Tokyo. The participation to theWorkshop was limited to the active participants, i.e.
the members of the research groups that submitted
the results of the tasks, advisors and members of theorganizing group. It had fifty-two participants andended in great enthusiasm. The NTCIR/IREX Joint
Workshop and IREX Workshop, the anotherevaluation workshop of IR and information
extraction (named entity) using Japanese newspaper
articles were held consecutively.
The purposes of the NTCIR Workshop are as
follows;(1) to encourage research in information retrieval,
CLIR and related areas by providing a large-

scale Japanese test collection and a common
evaluation setting that allows cross-systemcomparisons(2) to provide a forum for research groups interested
in comparing results and exchanging researchideas or opinions in an informal atmosphere(3) to investigate methods for constructing largescale test collections and IR laboratory-type
testing, and create large scale test collections
based on the investigation.
The test collection used in this Workshop is
called “NACSIS Test Collection 1” or “NTCIR-1”
and consists of more than 330,000 documents, withmore than half presented as English-Japanese pairs.
Although there is a Japanese test collection called
BMIR-J2 consisting of 5,080 newspaper articles [6]and its contribution to IR research in Japan istremendous, enhancement of the Japanese testcollection in both variety of text types and scale wasneeded. We place emphasis on CLIR since it iscritical in the Internet environment and for Japanesescientific information retrieval as discussed in
Section 1.
Thirty-one groups, including participants fromsix countries, have enrolled in the first NTCIR
Workshop. Among these, 28 groups have enrolled in
IR tasks (23 in the Ad Hoc Task and 16 in the CrossLingual Task) and nine in the Term Recognition
Task.
Regarding IR tasks, twenty-three groups
submitted the search results. The search results of 48
runs were submitted for the Ad Hoc Retrieval Task
from 17 groups and 69 runs for the Cross-Lingual
Retrieval Task from 10 groups. Nine groups arefrom Japanese companies, 11 are from Japanese
universities or national research institutes, and fourare non-Japanese groups. Two groups are from theUnited States, one group is from Australia, one
group is from Taiwan and 19 groups are from Japan;
some of this latter group have non-Japanese
members or have collaborated with research groups
outside Japan. Two groups worked without any
Japanese language expertise.
2.2 The Tasks
Each participant has conducted one or more of
the following tasks.
The Ad Hoc Information Retrieval Task: toinvestigate the retrieval performance of systems thatsearch a static set of documents using new search
topicsThe Cross-Lingual Information Retrieval Task:
an ad hoc task in which the documents are in
English and the topics are in Japanese
The Automatic Term Recognition and Role
Analysis Task: (1) to extract terms from titles andabstracts of documents, and (2) to identify the terms
representing the “object”, “method” and “mainoperation” of the main topic of each document
The Ad Hoc Task is also substantially crosslingual since in the Task, as shown in Fig. 2, JE
collection, mixture of Japanese documents andEnglish documents, is used since in Japanese
operational IR environments, especially for retrievalof scholarly, scientific or technical documents andWorld Wide Web documents, retrieving both
Japanese and English documents at the same time isquite natural. The E Collection, English document
collection, is used in the Cross-Lingual Task. The J
Collection, Japanese document collection, is used inmonolingual retrieval, which serves as a baseline forcomparing the search effectiveness with the resultsin the Cross-Lingual Task.

Monolingual IR
J
Collection

JE
Collection

E
Collection

Training
Topics0001--0030
Japanese
Cross-Lingual
IR
Test Topics
Ad Hoc 0031--0083
Japanese

Fig 2. Relation of the Document Collections and
Tasks

The Procedures
From November 1, 1998, delivery to each IR
tasks-participant of the document data, 30 search
topics (0001–0030) and their relevance judgments
began, so the participants could train their systems.
Among them, 21 topics were used as cross-lingual
topics. The topics were written in Japanese. Fiftythree new test topics (0031–0083) were distributed
to participants on February 8, 1999, and the search
results for these new topics were submitted fromeach participant by March 4 as official test runs. The
test topics are common for both IR tasks.
A participant could submit the results of morethan one run. Both automatic and manual query
constructions were allowed. In the case of automaticconstruction, the participants were required tosubmit at least one set of results for searches using
the <DESCRIPTION> fields of the topics only as
the mandatory runs as baseline of cross-systemcomparison. For optional automatic runs and manual
runs, any fields of the topics could be used. In

addition, each participant was required to complete
and submit a form describing the detailed features of
their system.
Top ranked documents of each topic werecollected from each run and formed document poolsof candidates of relevant documents. Human
analysts assessed the relevance of documents in thepools for each topic. Based on the assessments,
interpolated recall and precision at 11 points,
average precision (non-interpolated) over all
relevant documents, and precision at 5, 10, 15, 20,
30, 100 documents were calculated using TREC’s
evaluation program, which is available from the ftp
site of Cornell University.
2.3. The Test Collection
The test collection used in the Workshop, “Test
Collection 1” or “NTCIR-1”, consists of documents,
topics, and relevance assessments for each search
topic.
Documents
The documents are author abstracts ofconference papers that were presented at academic
meetings hosted by 65 Japanese academic societies[7]. Subject domain, length and format of the
documents are diversified according to each society.
A wide range of subject domains from pure sciences,
technology and engineering, social sciences andhumanities is included, although about half of the
documents are from electronic engineering andcomputer sciences.
The Collection contains three document
collections: the JE Collection; the J Collection; andthe E Collection. The JE Collection contains
339,483 documents, of which more than half are
English-Japanese paired, the most of the rest are
Japanese, and rather small number of documents are
English documents. The author(s) of the document
prepared both Japanese and English parts. The J and
E Collections are constructed by extracting theJapanese or English parts, respectively, of the
documents in the JE Collection.
Documents are plain text with SGML-like tags.
A record may contain document ID, title, a list of
author(s), name and date of the conference, abstract,
keyword(s) that were assigned by the author(s) of
the document, and the name of the host society.
Topics
A topic is a formatted description of a user’s
information need. We defined the topics as
statements of “user need” rather than “queries”,
which are the strings actually submitted to thesystem, since we would like to allow both manualand automatic query construction from the topics.

<REC><ACCN>gakkai-0000011144</ACCN><TITL TYPE="kanji">電子原稿・電子 出版・電子図書館「SGML 実験誌」の作成実験を通して</TITL><TITE TYPE="alpha">Electronic manuscripts, electronic
publishing and electronic library </TITE><AUPK TYPE="kanji">根岸 正光</AUPK><AUPE TYPE="alpha">Negishi,Masamitsu</AUPE><CONF TYPE="kanji">研究発表会(情報学基礎)</CONF><CNFE TYPE="alpha">The Special Interest Group Notes of
IPSJ</CNFE><CNFD>1991. 11. 19</CNFD><ABST TYPE="kanji"><ABST.P>電子出版というキーワードを中心に、文献の執筆、編集、印刷、流通の過程の電子化について、その現状を整理して今後の動向を検討する。とくに、電子出版に関する国際規格である SGML(Standard
Generalized Markup Language)に対するわが国での動きに注目し、学術情報センターにおける「SGML 実験誌」およびその全文 CD-ROM 版の作成実験を通じて得られた知見を報告する。また電子図書館について、その諸形態を展望する。
出版文化に依拠するこの種の社会システムの場合、技術的な問題というのは、その技術の社会的な受容・浸透の問題
であり、この観点から標準化の重要性を論じる。
</ABST.P></ABST><ABSE
TYPE="alpha"><ABSE.P>Current
situationonelectronic processing in preparation, editing, printing anddistribution of documents is summarized and its future trend isdiscussed, with focus on the concept: "Electronic publishing.
"Movements in the country concerning an international standard
on electronic publishing, SGML (Standard Generalized Markup
Language), are assumed to be important, and the results from anexperiment at NACSIS to publish "SGML Experimental Journal"
and to make its full-text CD-ROM version are reported. Various
forms of "Electronic library" are also investigated. The author
puts emphasis on standardization, as technological problems for
those social systems based on cultural settings of publication of
the country, are the problems of acceptance and penetration of thetechnology in the society.</ABSE.P></ABSE><KYWD TYPE="kanji">電子出版 // 電子図書館 // 電子原稿 //SGML // 学術情報センター // 全文データベース</KYWD><KYWE TYPE="alpha">Electronic publishing // Electronic
library // Electronic manuscripts // SGML // NACSIS // Full text
databases</KYWE><SOCN TYPE="kanji">情報処理学会</SOCN><SOCE TYPE="alpha">Information Processing Society ofJapan</SOCE></REC>

Fig. 3. A Sample of the Document Record
The query format is similar to that used in
TREC-1 and 2 and contains SGML-like tags. A
query consists of a title of the topic, a description, adetailed narrative and a list of concepts and field(s).
The title is a very short description of the topic andcan be used as a very short query that resembles
those often submitted by end-users of Internet searchengines. Each narrative may contain a detailed
explanation of the topic, term definitions,background knowledge, the purpose of the search,
criteria for judgment of relevance, and so on.
Topics were collected from users. Their subject
domains are diversified such as, computer sciences,
pharmacology, biochemistry, social sciences such as
education, linguistics, and so on.
The Collection contains 30 training topics and 53
test topics. Among them, 21 training topics and 39

test topics are usable for cross-lingual retrieval. All
the topics are written in Japanese. English andKorean versions will be available.
<検索課題q=0005><タイトル >
特徴 次 元リダクション</タイトル ><検索要求>
クラスタリングにおける特徴次 元リダクション</検索 要求><検索要求 説 明>
オブジェクトのクラスタリングを行なうとき、オブジェクトを特徴ベク
トルで表現することが望まれる。アプリケーションによっては、オブジェクトの次元は数千、数万となることがある。このような場合、
事 前に次 元を落とすことが必要になる。正解文書は、特徴次元リダクションの方法について、理論 面から、または実験によって、
提 案、比較などを行なっているもの。画像 処 理などの実験の操
作の一 部として特徴 次 元リダクションを用いているだけでは要求を満たさない。
</検索 要求 説 明><概念>
特徴 選 択, 主成分分析, 情報の粒度, 幾 何クラスタリング
</概念>
<分野>
1.電子・情報・制御</分野></検索 課題>

Fig. 4 A sample Topic
The criteria of topic preparation are as follows.
(1) Statements of “user need” rather than “queries”(2) <Description> containing every concept needed
to describe the topic(3) Not too easy:(3-1) Simple word matching of query terms
cannot retrieve every relevant document. (Since a
concept may be represented by various terms.)(3-2) A document containing query terms can be
non-relevant. (Since a word may have different
meanings.)(4) Five or more relevant documents in the top 100
documents retrieved by the NACSIS retrieval
systemSentences were modified when they were too
restricted or ambiguous. The function category ofeach topic was analyzed and assigned based on thefunction category proposed by BMIR [6]. The
category indicates the required level of techniques
and knowledge to conduct a search of the topic. We
tried to balance the topic length, number of relevant
documents, and “difficulty”; however, we found that
estimating “difficulty” is difficult.
Relevance Judgments (Right Answers)
The relevance judgments were undertaken bypooling methods. A certain number of top-ranked
documents were collected from each submitted run
and created a pool of possibly relevant documents.
Human analysts assessed the relevance of eachdocument in the pool against the topic. The
relevance assessment was undertaken using threegrades: relevant, partially relevant, non-relevant.
Two analysts assessed the relevance to a topic
separately, and then the primary analyst of the topic,
who created the topic, made the final judgment
based on the negociation between two analysts.

3. In terms of exhaustiveness, pooling the top 100
documents from each run worked well for topics
with less than 50 relevant documents. For the topics
with more than 100 relevant documents, although
the top 100 pooling covered only 51.9% of the total
relevant documents, the coverage reached 90% if
combined with interactive searches.

Relevance judgment files contain not only therelevance of each document in the pool but also
contain extracted phrases or passages showing thereason why the analyst assessed the document as“relevant”. Since a narrative of topics may containsome description related to the user’s situation or the
purpose of the search, situational-oriented relevancejudgments were conducted as well as topic-oriented
relevance judgments, which are more common in
ordinary IR systems laboratory testing. However,
only topic-oriented judgments are used in the formal
evaluation of this Workshop. The relevancejudgments are often called "right answers" in theJapanese IR community.

Regarding test topics, based on the analyses
above we decided to use the top 100 pooling andinteractive search for the topic with more than 50
relevant documents.

Linguistic Analysis
Part of the J Collection contains detailed handtagged part-of-speech (POS) tags [8]. Because of theabsence of explicit boundaries between words inJapanese sentences, we set three levels of lexicalboundaries (i.e., word boundaries, strong and weakmorpheme boundaries), and assigned detailed POS
tags based on the boundaries and types of origin, sothat the Collection could be used to examine suitableterm segmentations of Japanese texts for retrieval
purpose. The Tagged Corpus was not used for
official evaluation of the IR tasks but was distributed
to the IR tasks participants on request.

Regardless of the inconsistency of the relevanceassessments, we found strong correlation between
the system rankings produced using the relevant
judgments by the primary analysts, the secondary
analysts, and the final judges [9-10,13]. Regarding
pooling methods, we compared the relevancejudgments based on the document pool created bycollecting top X documents from one run per systemand top X documents from every run, regardless ofthe number of runs submitted by a system.

Robustness of the System Evaluation using theTest Collection 1
The Test Collection 1 itself has been tested fromthe following aspects so that it is usable as a reliable
tool for IR system testing:(A) exhaustivity of the document pool(B) inter-analysts consistency and its effect forsystem evaluation(C) topic-by-topic evaluation.
The results of these studies have been reported
and published on various occasions [9-13]. A briefsummary of the studies is shown here.
1. The initial pooling, which was held in NACSIS
to create relevance judgments for the training topics,
worked well and covered 97% of the total relevant
documents.
2. Interactive searches were effective for someparticular topics and found 17.5% of unique relevant
documents.

4. The top 100 pooling method has the effect of
reducing the size of the document pool to 21.9% of
its possible size for training topics and 20.9% fortest topics.
5.
Very high similarity was found between thesystem rankings produced using different sets ofrelevance assessments, regardless of the different
coverage and pooling methods and regardless ofinconsistencies between relevance assessments ofhuman analysts for both training and test topics.

As a result, regardless of the different coverageof pooled methods, we found strong correlation
between the system rankings produced based on the
different pooling methods for both training topics
and test topics [10,13]. We concluded that the testcollection is reliable as a tool for system evaluation
based on these analyses.
2.4. Retrieval Results
This section reports an overview of the retrieval
results from the aspects of system effectiveness andanalyzes some of the similarities and differences of
the approaches taken by each participating group.
Since one of the main purposes of the NTCIR
Workshop is to enhance research in Japanese text
retrieval and cross-lingual retrieval, examination ofvarious approaches using the NTCIR Collection has
been encouraged. For the official test runs, 53 new
topics (topics 0031–0083) were delivered. Fortyeight ad hoc runs and 69 cross-lingual runs were
submitted. Relevance judgments were undertaken byhuman analysts and delivered to the active
participating groups by the beginning of June 1999.
For further details of each approach, please consult

each system paper in the Workshop Proceedings.
The Proceeding will be available online;
http://www.rd.nacsis.ac.jp/~ntcadm/
Cross-Lingual Runs
The Cross-Lingual Retrieval Task is an ad hoc
retrieval task in which the documents are in English
and the topics are in Japanese. Sixty-nine retrieval
result sets were submitted from 10 participatinggroups and the NTCIR IR organizer. The organizer
did this to assess the effectiveness of the systemused for the initial pooling for this collection without
any extra tuning.

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0

Cross-Lingual All Runs

0

1.0
0.9

0.1

0.2

0.3

0.4

0.5 0.6

0.7

0.8

0.9

1

BKJEBDDS,auto

TSB4,interactive

1KE3,auto

NTE153,auto

CRL1,auto

BKJJBIDS,monolingual short

0.8

Fig.6 Cross-Lingual ShortQueries w/o Concepts(Relevant) Top Runs (Best run for each group)

0.7
0.6

(1) Approaches
Every runs took the approach of “Query
Translation” as overall approach for CLIR. Most ofruns were dictionary-based or using machinetranslation. One group used corpus-based approach.

0.5
0.4
0.3
0.2
0.1
0.0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

BKJEBKFU,auto,t,d,n,jc,ec,ac,f

FLAB11,inter,t,d,n,jc

tstar7,auto,t,jc,ec,ac

TSB4,inter,d1KE3,auto,dNTE154,auto,t,d1

BKJJBIFU,Monolingual,longFig. 5 Cross-Lingual All Runs (Relevant) Top
Runs （ Best run for each group））
In the Workshop, various approaches towards
tasks were reported. Results are as follows;
1. A search using a longer topic tended to obtainbetter results, however, some runs gave
opposite results and could not utilize the<NARRATIVE> field of the search topics to
improve search effectiveness.
2. Interactive systems obtained better results thanautomatic system
3. A search using <CONCEPT> in the topic
obtained better results than a search without<CONCEPT>, but
thesearchusing<CONCEPT> only worked poor
4. Indexing methods used were various; bi-gram,morphological analysis, and extended n-gram
such as, adaptive segmentation, etc. were used.
5. Retrieval model: Vector space, probabilistic
model, extension of probabilistic model, etc.

For the dictionaries, most groups preparedbilingual dictionaries internally from various
resources. The size of them were from 20,636
entries to 579,115 entries. EDR and EDICT are also
used by a few groups.
(2) Technical Terms
One of the difficult components of the NTCIR-1
is that of translating technical terms. Terms are
sometimes too specific or technical, or too new tolook up in ordinary dictionaries.
On the other hand, quasi-paired documents usinga native language and English, such as ntc1-je0
documents, are rather common in non-English
speaking countries. These documents can be foundon the Web and in scholarly documents, commercial
documents describing a company’s products, andgovernment documents. Using these documents to
prepare bilingual or multilingual lexical resourcesthat are usable for cross-lingual information accesswill be one of the practical approaches to the
problem.
(3) Query Expansion and Disambiguation
Query expansion and word sense disambiguation
were conducted by several groups. For example, one
group looked at post-translation expansion anddisambiguation using various kinds of English

monolingual corpora. On the other hand, the othergroup looked at pre-translation expansion anddisambiguation using J Collection or another parallel
corpus. Another group incorporated automatic local
feedback. Several groups employed more naturaland naïve query expansion, i.e., translating terms
into more than one target language, translation andexpansion at the same time.
(4) Transliteration
A group used transliteration of Katakana words
and worked well. Especially it seemed to work wellon technical terms and is expected to effective to
reduce the problems caused by the word mismatch
because of the various ways of expression of aconcept in Japanese document, which werediscussed in Section 1.1. More investigation is
expected on this matter.
2.5 Next NTCIR Workshop
NACSIS will change its name into National
institute of Informatics in the next spring and plans
to run a second evaluation after the change. It will
include at least Japanese and English with training
data available in May 2000, test data available inapproximately in September, and the workshop itself
scheduled for February or March, 2001. Some partof the schedule may be changed through the effort tocordinating other evaluation project like TREC,
TDT, or TIDES.
Meanwhile, we are planing details of the tasks,
subtasks, evaluation scheme, collection, andresources. The needs of training courses andtutorials on evaluation of information retrieval
systems including interactive systems for Japanese
new comers in Japanese language and ones on
Japanese text processing and available resources for
non Japanese researchers in English are suggested
from the advisory group. Any comments, advises,
and leads are welcome.

3. Large-Scale Multilingual Lexicon
One of the problems of the CLIR is drawbacks
associated with the availability of resources [16-17].
For dictionary-based methods, the coverage ofdictionaries or thesauri are often not sufficiently
broad and deep, thus domain specific terms or new
concepts, which are critical for retrieval of technicaland scientific documents used here, tend not to belisted. Lack of resources is also problematic in
corpus-based methods; parallel corpora are notalways readily available. Regarding machinetranslation techniques, we can not ignore the cost of
linguistic
analysis.
Preparinglarge-scale
multilingual
lexiconthroughinternational

collaboration and encourage the research on themethods to generate such lexicon automatically are
one of the promising directions to facilitate CLIR invarious types of information.
In the NTCIR Workshop, many groups
constructed bilingual lexical resource internally.
Some of them were from the document collection in
NTCIR-1. We are very grateful that a group kindly
offered to share the resources with participants of thenext workshop. We ourselves also conducted
research on CLIR using automatically generated
bilingual keyword clusters based on graph-theory[18]. The clusters were generated using Japaneseand English keywords, which are assigned toscientific papers by the authors.
In the contrast to the problems of resource shown
above, our approach for the CLIR using keyword
clusters has several advantages: regarding theresources, the keyword data we used has advantages,i.e., subject-specific bilingual keyword corpora arereadily available in machine-readable form for agreat many subject domains. They are also readily
segmented into terms, and well aliened (albeit withsome noise). Therefore, they are rather easy tohandle for our purpose. In addition to this, the graphtheoretic approach as advantages: (1) by utilizing
topological features of the graph, low-frequency
keywords can be treated properly and are usable inIR; (2) the clusters contain not only J-E pairs, butalso J-J and E-E pairs so that they are usable forquery expansion in monolingual retrieval as well,and (3) this was achieved with reasonable
computational cost.
We used keyword clusters based on only lessthan 10% of database as a corpus and achieved theeffectiveness of 52.4% of monolingual at average
precision for all relevant documents at “relevant”
level and 65.7% for "relevant & partial relevant”
level with automatic phrase identification andsynonym processing, without any manual interaction,
and, in monolingual IR, 13.4% over the baseline at
the average precision for relevant level, and 14.2%
for "relevant & partial relevant”. This is comparable
to existing researches and fairy well as a start point.
However this still leaves a lot of room for
improvement both in the aspect of IR and clustering
as further discussion is found in [18].
The majority of scientific papers published inJapan have such bilingual keyword lists and such
lists are widely found in non-English speaking
countries. These are one of the good candidates toenhance the lexical resources usable for CLIR
through international collaboration.

4. Towards Cross-Lingual Information
Access.
The occasion of acting as one of the panel
speakers at ACM-DL'99 and SIGIR'99 Joint
Workshop on Multilingual Information Discovery
and Access (MIDAS) brought me to think about"information discovery and access". From my pointof view, one of the focuses of research anddevelopment in digital libraries is to enhance theuniversal information discovery and access. It
includes (1) to know the appropriate place to searchnecessary information, (2) to search and acquire it,and (3) to access it.
To know the right place to search, or "where togo", is the first step of the information discovery.
Any of Internet search engines can not be perfect in
the aspect of coverage of the documents indexed.
Traditional libraries or information centers haveprovided such service called "referral service". Thiskind of function should be incorporated to digital
libraries. To search and acquire is covered byinformation retrieval.
To access the information has two meanings, i.e.,
to access the information physically andintellectually. The latter includes that users do
information work such as decision making, problem
solving, paper writing using the information
retrieved. To support and facilitate the users to do
information work using retrieved information is oneof the new areas of investigation in digital libraries.
For example, translation of the retrieved documents,
skimming, automatic summarization, extracting
answer passages, clustering the retrieved docs,
visualization, text mining, comparison of thecontents in the documents etc
Cross-lingual information discovery and accessis one of the key aspects to achieve the universal
information discovery and access. CLIR is a part of
the whole process of the cross-lingual information
discovery and access. Expansion of the scope of
research from CLIR to cross-lingual information
discovery and access is one of the interesting
directions of future research.

ACKNOWLEDGMENTS
We thank all the participants for their contributions
and the assessors. Special thanks are due to Donna
Harman, Ellen Voorhees, Ross Wilkinson, Sung H.
Myaeng, and Mun Kew Leong for their substantial
advice and continuous support. Hard work of the
NTCIR project members should be acknowledged;
Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi,
Souichiro Hidaka, Hideyuki Kato, and Yuko
Miyamoto.

REFERENCES[1] NTCIR Project. http://www.rd.nacsis.ac.jp/~ntcadm/[2] Kando, N. et al. NACSIS Test Collection Workshop(NTCIR-1) [poster]. Proceeding of ACM-SIGIR '99.
Aug.,1999, Berkeley, CA[3] Kando, N. Cross-Linguistic Scholarly Information
Transfer and Database Services in Japan. Annual
Meeting of the ASIS. (Nov. 1997) Washington DC.
[4] TREC URL: http://www.nist.gov/trec[5] TDT URL: http://www.nist.gov/speech and selecting"benchmark tests" and then "TDT-3"[6] Sakai, T. et al. BMIR-J2: Test Collection for
Evaluation of Japanese Information Retrieval
Systems. SIGIR Forum (to appear).
[7] The list of 65 academic societies is available at URL,
http://www.rd.nacsis.ac.jp/~ntcadm/acknowledge/tha
nks1-en.html.
[8] Kageura, K. Overview of Term Recognition Task, In
Proceedings of the First NTCIR Workshop, August30-September 1, 1999, Tokyo (to appear).
[9] Kando, N. et al. NTCIR-1: Its Policy and Practice.
IPSJ SIG Notes, Vol.99, No.20, pp. 33-40, 1999 [inJapanese].
[10] Kuriyama, K. et al. Pooling for a Large Scale Test
Collection: Analysis of the Search Results for the
Pre-test of the NTCIR-1 Workshop. IPSJ SIG Notes,
Vol. 99, (May, 1999) [in Japanese].
[11] Kuriyama, K. et al. Construction of a Large Scale
Test Collection: Analysis of the Training Topics of
the NTCIR-1. IPSJ SIG Notes, Vol. 99 (July, 1999)[in Japanese].
[12] Kando, N. et al. Construction of a Large Scale Test
Collection: Analysis of the Test Topics of the
NTCIR-1. In Proceedings of IPSJ Annual Meeting (toappear) [in Japanese].
[13] Kuriyama, K. et al. Pooling for a Large Scale Test
Collection: Analysis of the Search Results for the
first NTCIR Workshop. In Proceedings of IPSJAnnual Meeting (to appear) [in Japanese].
[14] Rorvig, M. et al. The N-Gram Hypothesis Applied toMatched Sets of Visualized Japanese-English
Technical Documents. In Proceeding of the ASIS
Annual Meeting (to appear).
[15] Voorhees, E.M. Variations in Relevance Judgments
and the Measurement of Retrieval Effectiveness. In
Proceedings of the 21st Annual International ACM
SIGIR Conference on Research and Development inInformation Retrieval, Melbourne, Australia, August,
1998, pp. 315-323.
[16] Ballesteros, L. et al. Resolving ambiguity for crosslanguage retrieval. Proceedings of the 21st Annual
International ACM SIGIR Conference of Researchand Development in Information Retrieval, Aug.,
1998, Melbourne, Australia, p.64-71[17] Shauble, P. et al."Cross-language Information
Retrieval (CLIR) Track overview" In Proceedings of
TREC6, p.25-30, 1998[18] Kando, N. et al., Cross-Lingual Information Retrieval
using
Automatically
Generated
Multilingual
Keyword Clusters. In
Proceedings of the 3rdInternational Workshop of IR with Asian Language,Oct., 1998, Singapore