Preface
The Third NTCIR Workshop is the third venture in a series of evaluation workshops designed toenhance research in information access technologies including text retrieval, cross language information
retrieval, automatic text summarization, information extraction, and question answering on Japanese andAsian language text. It is co-sponsored by the Research Center of the Information Resources at theNational Institute of Informatics (RCIR/NII, Director: Jun Adachi) and Japanese MEXT Grant-in-Aid for
Scientific Research on Informatics (#13224087).
The goals of the NTCIR Workshops are as follows:
1. to encourage research in information access technologies by providing large-scale test collections
reusable for experiments and common evaluation infrastructures,
2. to provide a forum of research groups interested in cross-system comparison and exchanging
research ideas, and
3. to investigate a) methodologies and metrics for evaluation of information access technologies andb) methods for constructing large-scale reusable test collections.
With the rapid growth of the Internet, the importance of research in information access technologies isincreasing greatly. A term "information access" includes a whole process to make information in thedocuments usable to users and we have looked at the technologies to support users to utilize theinformation in the documents in a vast collection of various types of documents for various tasks.
Research and development of information access technologies requires solid evidence based on
experiments to show the superiority of the proposed system and/or strategy over previous ones. A testcollection is a data set used for such testing. The importance of the large-scale test collections in thisresearch and the forum for researchers to learn each other is widely recognized. We hope the NTCIR
Workshop serves as one such opportunity, and that wide-ranging insight has been produced as a result ofthe Workshop.
The Third NTCIR Workshop started with the document data distribution in October 2001 and its
Meeting was held on 8-10 October 2002, NII, Tokyo. We call the whole process from document
distribution to the meeting as workshop since we have placed emphasis on the interaction andcollaboration among participants and organizers through this activity. It hosted five tasks;(1) Cross-Language Information Retrieval Task (CLIR),(2) Patent Retrieval Task (PATENT),(3) Question Answering Challenge (QAC),(4) Text Summarization Challenge, (TSC2), and(5) Web Retrieval Task (WEB).
This was the first NTCIR workshop for PATENT, QAC, and WEB. CLIR and TSC included newsubtasks. A new challenging task was termed a "challenge" in the NTCIR. However, we found that all
five tasks contained new and unique challenging issues regarding research and development of thetechnologies as well as their evaluation and test collection construction, and therefore each task was a"challenge" for both the participants and the task organizers.
Each task was coordinated and organized by the task organizers, a group of researchers of the topic.
CLIR is organized in cooperation with National Taiwan University, Korea Institute of Science andTechnology Information (KISTI), and NII. PATENT is organized in cooperation with Japan Intellectual
Property Association (JIPA) as a joint research with the NII. In addition to email discussion, organizers
had periodical meetings to discuss the issues in detail. Each task had several so-called "Round Table
Meeting", and participants, task organizers, advisory committee members and sometimes non-participant

researchers who were interested in the evaluation of the technologies discussed detailed task description,evaluation methodologies, results analysis, and future direction.
Sixty-five research groups from nine different countries and areas conducted the tasks set by theorganizers and submitted the results. Of these groups, 14 are from companies, 44 from universities andseven from national or independent research institutes. These groups were listed in Table 1 in theoverview of the third NTCIR Workshop in this volume. We greatly appreciate the efforts and energy ofeach group in their research for this workshop.
For the Workshop Meeting held on 8-10 October 2002, 207 people attended. The meeting included
plenary sessions for overview and system papers, two invited talks, digital poster sessions in which
demonstration or poster presentations were provided by every participating group, and pre-meeting
lectures and some round table discussion meetings.
The following test collections were created for each task of the Third NTCIR Workshop:
・ NTCIR-3CLIR: 690,832 (944MB) news articles published in Taiwan ROC, Korea, and Japan. The
1998-99 sub-collection consists of articles in the languages of Traditional Chinese, Japanese,Taiwan English, and Japan English articles published in 1998-1999, and 50 topics. The 1994
sub-collection consists of articles in Korean language published in 1994 and 30 topics. All thetopics are available in all four languages, Traditional Chinese, Korean, Japanese and English.
4-grade relevance judgments.
・ NTCIR-3PATENT: 697,262 full text Japanese patents published in 1998-1999 (18GB) and1,706,154 exactly translated English and Japanese abstracts in 1995-1999 (4GB). Thirty-one
topics are available in Traditional and Simplified Chinese, Korean, Japanese, and English.
Relevance judgments were done in three grades by real patent search professionals.
・ NTCIR-3QAC: Documents are the same Japanese newspaper collection used in CLIR (220,078
documents). This includes 60 questions for dry run, 240 questions for formal evaluation, about
900 additional questions, and the list of the exact answers to those questions.
・ NTCIR-3Summ: Documents are the same as QAC. As human-created summaries which including
key-data for evaluation, seven types of summaries were prepared by each of three different
summarization experts (professional experienced captionists) for each of 60 articles of various
genres, and two types of summaries by each of three different experts for each of 50 sets ofarticles on particular topics.
・ NW100GB, NW10GB: 100 GB and 10 GB web documents crawled from the '.jp' domains andlinked pages to them. Forty-seven topics were used in the formal evaluation. Relevancejudgments are in 4 grades and relative judgments like best, second best, etc., based on "oneclick-distance model" of the documents, in which the assessors allowed to see the documents
hyperlinked from the document to be judged.
A system for relevance assessments for CLIR, PATENT, and WEB constructed to handle hyperlinked
documents, and various scoring tools and scripts were created for each task.
We are extremely grateful to the following organizations for their understanding of the project and forproviding permission to use document data for the Test Collections : Central Daily News, China Daily
News, China Times, China Times Commercial, China Times Express, Korea Economic Daily, Mainichi
Newspaper, PATOLIS (Japan), Taiwan News, and United Daily News. We also thank deeply National
Taiwan University, KISTI, and JIPA for their cooperation and collaboration.
I am grateful to the advisors of the Third NTCIR Workshop. Special thanks go to Drs Ellen Voorhees
and Donna Harman, National Institute of Standard and Technology (NIST), USA for their support andsubstantial advices based on their experience from the beginning of the project, for permitting the use ofTREC's evaluation program. We thank CLEF, the Cross-Language Evaluation Forum for support andcollaboration in various occasions.

Analysts who created topics or questions, undertook the relevance judgments and evaluation of thesubmitted results, and prepared "right answers" have all worked very hard. The hard work of taskorganizers, Hsin-Hsi Chen, Kuang-hua Chen, Koji Eguchi, Atsushi Fujii, Jun'ichi Fukumoto, TakahiroFukusima, Emi Ishida, Makoto Iwayama, Tsuneaki Kato, Kazuaki Kishida, Kazuko Kuriyama, Sukhoon
Lee, Fumito Masui, Sung Hyon Myaeng, Hidetsugu Nanba, Manabu Okumura, Keizo Oyama, andAkihiko Takano and their groups, the secretariat in the NII: Ai Kuba, Urara Numata, and Naoko Tsuchiya,
and those elsewhere must be acknowledged. We have been able to list some of these people on the"Workshop Organization" page.
January 2003

Noriko Kando
Program Chair of the Third NTCIR Workshop
National Institute of Informatics