Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Overview of the Seventh NTCIR Workshop
Noriko Kando
National Institute of Informatics
Tokyo 101-8430, Japan
Noriko.Kando@nii.ac.jp

Abstract
This paper outlines the seventh NTCIR
Workshop, which is the latest in a series. It briefly
describes the background, tasks, participants, andtest collections of the workshop. The purpose ofthis paper is to serve as an introduction to theresearch described in detail in the rest of theproceedings of the seventh NTCIR Workshop.
Keywords: evaluation, information access,information
retrieval,complexquestionanswering,
test
collections,cross-lingual
information retrieval, cross-lingual question
answering, multilingual opinion analysis, patenttranslation, patent mining, trend analysis,
visualizationsuch as indexing includes language-dependent
procedures. The NTCIR project therefore started
in late 1997 with emphasis on, but not limited to,Japanese or other East Asian languages, and its
workshop series has attracted international
participation.
In NTCIR, a unit of activity is one and a halfyears, i.e. eighteen months. Because we respect
the interaction between participants, we consider
the whole process from call-for-task-participation
to the final meeting to be a “workshop”. Eachworkshop selects several research areas called“tasks”. Each task has been organized by the
researchers of the domain and a task may consistof more than one subtask.
In addition to the tasks, NTCIR-7 introduced“cluster structure” of tasks, so that we can conducta module-based evaluation as well as investigate
the best combination of modules from different
parties to search toward a “dream system”.

1. Introduction
The NTCIR Workshop [1] is a series of
evaluation workshops designed to enhanceresearch in information access (IA) technologies
including information retrieval (IR), cross-lingual
information retrieval (CLIR), question answering(QA), automatic text summarization, text mining
and so on by providing large-scale test collections
and a forum for researchers. It has been done as acollaborative community-based effort to build
research infrastructure for testing and evaluation.
The aims of the NTCIR project are:
1. Encourage research in information accesstechnologies by providing research
infrastructures for evaluation and testing;
2. Provide a forum for researchersinterested in exchanging research ideas
in an informal atmosphere; and3. Investigate methodologies to evaluate
information access technologies.

1.1 Information Access
The term “information access” (IA) refers thewhole process from when a user realizes his/her
information needs, through the activity ofsearching for and finding relevant documents, andthen utilizing information in them. We havelooked at IA technologies to help users utilize theinformation in large-scale document collections.
IR, summarization and QA are part of a “family”
of the technologies aiming at the same target,
although each of them has been investigated byrather different communities

By providing infrastructure for large-scale
experiment and evaluations, NTCIR has
encouraged the researches on the selected tasks in
IA and then speed the research and technology
transfer. The importance of such infrastructure in
IA research has been widely recognized.
Fundamental text processing procedures for IA,―1―

Lab-type IR Test
Asian Languages/cross-language
Variety of Genre
Parallel/comparable Corpus

New Challenges
Intersection of IR + NLP
To make information in the
documents more usable forusers!
Realistic eval/user task

Forum for Researchers.

Idea Exchange
Discussion/Investigation on
Evaluation methods/metrics

Figure 1. Focus of NTCIR Workshops

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

From the beginning of the project, we havelooked at both traditional laboratory-type IR
system testing and the evaluation of challenging
technologies, as shown in Figure 1. For theformer, we have placed emphasis on text retrievaland CLIR with Japanese or other Asian languages
and testing on various document genres.
For the challenging issues, the target is to shift
from document retrieval to technologies thatutilize “information” in documents, andinvestigation of methodologies and metrics formore realistic and reliable evaluation. For theformer, technologies and social environments are
kept improving and enhancing and then an
appropriate research infrastructure for these newtechnologies must be continuously built andmaintained by the research community. NTCIR
thus have to continue the activity as a communitybased collaborative effort. These two directions
have been supported by a forum of researchers
who are interested in cross-system comparisonand by their discussions.

2. Tasks at NTCIR-7
2.1 Overview
For the Seventh NTCIR Workshop (NTCIR-7)[2-3], the process started from the call-for-taskparticipation in September 2007 and the meetingis held on 16-19 December 2008 [4], at National
Institute of Informatics (NII) in Tokyo. It issponsored by NII.
As shown in Figure 2, NTCIR-7 selected six"tasks" grouped into four clusters:

Cluster 2. User-Generated :- Multilingual Opinion Analysis
Cluster 3. Focused Domain : Patent- Patent Translation ; English -> Japanese,- Patent Mining paper -> IPC
Cluster 4. MuST :- Multi-modal Summarization of Trends

MuST; Visualization Challenge

Cluster 1. Advanced CLIA
- Complex CLQA (Chinese, Japanese, English)- IR for QA (Chinese, Japanese, English)

1.2 Focus of NTCIR

Figure 2. NTCIR-7 Task Clusters
MOAT is the continuation of NTCIR-6’sOpinion Analysis (OPINION) Pilot Task andadded a new language, Simplified Chinese. MuSTis also the continuation of NTCIR-5 and -6’s Pilot
Workshops of MuST, which was originally held
as domestic events, and added evaluation subtask
called “T2N” in addition to the Free subtask.
Cluster 1 and 3 at NTCIR-7 are the extensionof the tasks well investigated in past NTCIRs withlargely changing the focuses of the investigation.
Cluster 1, or ACLIA, at NTCIR-7 is a modulebased evaluation combining "Complex CLQA"
and "IR for QA" and it is an extension of CLIR,
Question answering Challenge (QAC) and CLQA
which have been held in NTCIR-3 to -6. And
Cluster 3 proposed new research tasks upon the
prolonged experiences of “Patent Retrieval” and“Patent Classification targeting text mining forautomatic patent map creation” at NTCIR-3 to -6.
(Figure 3)

Japanese IR
T Cross-lingual IRa Patent Retrievalsmap/classif
k Web RetrievalsNavigational
Geo
Result Classification
Term Extraction
QuestionAnswering
Info Access Dialog
Summ metrics
Cross-Lingual
Text Summarization

Cluster
1:
Advanced
Cross-Lingual
Information Access (ACLIA)+ Complex Cross-Lingual Question Answering(CCLQA) task
+ Information Retrieval for Question
Answering (IR4QA) task
Cluster 2: User Generated Contents+ Multilingual Opinion Analysis Task (MOAT)
Cluster 3: Focused Domain : Patent+ Patent Mining (PAT MN) task
+ Patent Translation (PAT MT) task
Cluster 4: Multimodal Summarization ofTrend (MuST)

1st
sci

2nd

3rd 4th

5th 6thnews

Trend Information
Opinion Analysis

Figure 3. Tasks at Past NTCIRs

2.2 Advanced Cross-Lingual Information
Access (ACLIA) Cluster [5][6]It is a module-based evaluation combining"Complex CLQA" and "IR for QA" as shown in
Figure 4.

―2―

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

ACLIA
Complex Cross-lingual Question Answering(CCLQA) Task

Small teams that
do not possess anentire QA system
can contribute

Different teams
can exchange
and create a“dream-team”
QA system
IR and QA communities can collaborate

Figure 4. Structure of ACLIAThe basic ideas to propose a module-based
evaluation are:- Advanced IA systems for any types of
Questions and returning answers in
relevant formats.
- Glass-box evaluation- Build a “dream system” by merging QA
and IR communities- Extended research infrastructure not only
traditional “test collections” but also API
or modules, and- Provide opportunity that a small researchgroup can make significant contribution
This target to complex QA including any typesof questions in QA-viewpoints, and it is a kind of
focused retrieval for particular question types in
IR viewpoints.
As shown in Figure 3, QAC started from
NTCIR-3, and investigated factoid for two cycles,
Evaluation of Series of Questions was started
from NTCIR-3 but the design was drasticallymodified into the evaluation of Information
Access Dialog (IAD) in NTCIR-4 and NTCIR-5.
Cross-language started in NTCIR-5. In NTCIR-6,
it started a new QA experimental design to cover
every kinds of questions including complex
questions like definition, how and why as well asfactoid, and evaluate using a metric called BE,
Basic Element, which was originally proposed aswhich will be used in n automatic evaluation ofsummarization. This was done only on Japanese
documents. Then we extended it to cross-lingual
in NTCIR-7
Detailed module-based analysis of the resultsof Cross-lingual Question Answering (CLQA)
tasks at NTCIR-5 and -6 revealed that the one of
the major problems in CLQA between Asian
languages and English was “retrieval” module,
and the acute performance drops were occurred in
retrieval modules. CLQA systems generally
worked well on “question type analysis” and“answer extraction and formatting” once thecandidate documents were properly retrieved.
However, in many cases, CLQA system failed toretrieve the appropriate document subsets

containing potential answers from the whole
document collection especially for the queries
including proper names or any named entities
which are often not included in ordinarytranslation dictionaries or machine translation
systems.
On the other hand, CLIR however has been
investigated for long from NTCIR-1 with shifting
focuses of the research until NTCIR-6. One of the
major problems that all the participants havetackled is the solution of Out-of-Vocabulary(OOV) problems. In CLIR between the languages
using completely different alphabets, cognatematching cannot be used and searching proper
names is one of the most challenging tasks
although proper names and named entities areoften used in queries issued by the users inordinary life setting. Various methodologies for
OOV have been proposed and tested through past
NTCIRs.
It seems promising if we can combine thetechnologies investigated in QA community andthe better retrieval functionalities including OOVsolutions tested in the past CLIR tasks, we could
see a “dream system”. Although CLQA, QAC and
CLIR have been organized in NTCIR series forlong, the research communities for QA and IR are
rather separate each other, and a few groups
participated in both QA- and IR-oriented tasks.
The organizers then proposed a module-based
evaluation aiming combine the best QA and thebest IR systems. We also expected that suchenvironment encourage the participants to try
other part of the whole IA mechanisms.
CCLQA is highly complicated tasks consistingof many modules. When we conduct an end-toend evaluation, it is not easy for the researchers to
identify which parts are problematic in theirhighly complicated systems. If we can provide a
module-based evaluation environment, it is rather
easier to identify the problems and to learn whichpart of their systems need the improvement andrevisions. This module-based environment also
encourages the participation from small researchgroups or new comers who can work on only partof the whole process of CLQA, but can work
better on a focused module.
2.2.1. EPAN
ACLIA provide an evaluation environment
called EPAN, Evaluation Platform for ACLIA andNTCIR. It enable
1) Module-based evaluation by uploading anddownloading interim results from each
module of any participating systems, and
2) Community-based test collection creation
To sharing actual modules from each systemneeds extra burden of significant engineering
effort of integration and documentation of thefunctionality and usage. ACLIA then choose to―3―

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

achieve it by exchanging the interim results fromeach of the modules in CCLQA by a fixed XML
format. Using EPAN, anybody can contribute thedata creation. In NTCIR-7 CCLQA at ACLIA,
dry run datasets were constructed by theparticipating research groups and some of theparticipants volunteered to contribute thejudgments in formal runs as well. Evaluation is
critical for the researchers themselves. This is akind of healty cycle that the researchers
community work collaboratively to build a
research infrastructure of evaluation with somecoorination. I do appreciate all the contribution,and expect to see such efforts shall make NTCIRor any other evaluation campaign sustainable.

Genre
News
Review
Blog

Subjectivity Holder
NTCIR-6
NTCIR-6
NTCIR-7
NTCIR-7
NTCIR-8
NTCIR-8

StakeholderTemporal
NTCIR-7
NTCIR-8

NTCIR-8

Polarity
NTCIR-6
NTCIR-7
NTCIR-8

Strength
NTCIR-7
NTCIR-8

LanguageGranuality Application
C,J,E
single-sent Summarization
Cs,Ct,J,E clause
QA
multi-sent Opinion tracking
Cs,Ct,J,E document Consistency checking
Trend

Figure 6. Roadmpap for Opinion Analysis
at NTCIR
At NTCIR-7, it was originally porposed to use
multilingual blog corpus, but with various
reasons, we gave up to deliver the new corpus
from NII and change the corpus to news. But wecould added a new language, Simplified Chinese.
“Continuing for two cycles” is one of our basic
policies for task selection and it is expected toeffective to initiate a new task for the first, andenhance the researches in the second. We
therefore tried to keep the same task design asmuch as possible with minor changes of the
problematic parts. At the previous OPINION task
at NTCIR-6, the evaluation metrics were slightly
changed across the language subtasks, but thenthe unified metrics were applied at NTCIR-7. A
new opinion-related element, Opinion Target, wasadded at NTCIR-7.
NTCIR-6 OPINION test collection was built
top upon the past NTCIR’s CLIR topics whichwere completely translated into four languages
and their relevant documents in each language.
MOAT at NTCIR-7 utilized some of the topics
from NTCIR-7 ACLIA and added some topics
and did own relevance judgments to select the
documents to be annotated at MOAT.

2.4 Patent Machine Translation [8]After four cycles of Patent Retrieval evaluation
both in monolingual and cross-lingual and PatentClassification targeting to automatic patent map
creation, NTCIR decided to go into a newtechnological domains related to patent
information processing:Statistical machinetranslation of patents and patents mining.
yp• NTCIR-3 (2001-2002)– Technology survey

Figure 5. Sample Displays of EPAN

• Applied conventional IR
problems to patent data

2.3 Multilingual Opinion Analysis Task
(MOAT) [7]• NTCIR-4 (2003-2004)

The roadmap for the task is shown in Figure 6.
Though it was originally proposed to use Chinesemonolingual corpus, but extended into
multilingual ones at the first pilot task at NTCIR6. It means that we started from very complicatedtask from the beginning.

• NTCIR-5 (2004-2005)―4―– Invalid patent searches• Addressed patent-specificIR problems– Enlarged invalid searches– Passage searches• NTCIR-6 (2006-2007)– Added English patents

2 years of patentapplications at JPO* JPO = Japanese Patent Office

5 years of patentapplications at JPO
10 years of patentapplications at JPO
10 years of USPTO
patents granted* USPTO = US Patent & Trademark Office

Figure 7. History of Patent Retrieval at
NTCIRProceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Patent Machine Translation (MT) is realistic asparallel corpora can potentially be produced from
10 years Japanese and USPTO’s patent fulltext
from NTCIR-5 and -6 collections using patentfamily relationship. Decoders for statistical MT(SMT) are also now available. At the PATMT
task at NTCIR-7, both statistical machinetranslation (SMT) and rule-based MT canparticipate and be tested.

aligned corpus, and it is the largest one between
Japanese and English to our knowledge.
Evaluation was done by both intrinsic using
BLEU and readability by human assessor, andextrinsic evaluation in CLIR task.
NTCIR-5Extrinsic evaluation
Patent claim

Human

Search topic
in Japanese

Evaluation by BLEU

JPO applications
1993-2002
Comparable(3.5-M docs)(not parallel)

J

E

J

E

Sentence-alignment method[Utiyama and Isahara, 2007]
Targeting“background” and“description”

USPTO grants
1993-2002(1.3-M docs)

J

Training data
1.8-M sentencepairs• System training• Parameter tuningE

JPO applications
1993-2002

MT system

Invalidate patent
Translation in
JapaneseIR system
Ranked doc.
list

Evaluation by Mean Average
Precision (MAP)

Patent family
Patent set forsame invention

Performed byorganizers

Search topic
in English

Figure 9. Extrinsic Evaluation at PAT MT
using CLIR Task.

Sentence pairs
Parallel (alignment accuracy= 90%)

The results demonstrated:- SMT is much better for CLIR
- Rule-based MT is good for humanevaluations- Human evaluations and creation ofreference translations must be carefully
done (in the real world, professional patenttranslators do use MT).

Figure 8. Producing Parallel Corpora

2.5 Patent Mining [9]Patent classification and mining has been
investigated from NTCIR-4. The goal is
automatic generation of patent map like Figure
10. The scenario is -- A topic is given, and a set of
relevant documents are retrieved and provided tothe mining system. It shall automatically analyze
the documents to find a set of “problems to besolved” and the “solutions”, and make a table andallocate relevant patents to each column
automatically.
gpp

Example: Blue light-emitting diodes

Given

Problems to be solved
Crystalline

Reliability

Structure of active
layer

SolutionsIntellectual property (IP) activities are global.
IP experts have to investigate to the patents
published in other countries in various languages
including the ones cannot understand. CLIR and
MT are inevitable to support such practice. CLIR
must be strong support for pre-selection of thecandidates to be translated for further
investigation. MT has been already used in thereal world and usable for some extent. But thequality is not sufficient for human comprehension
in details. And there are still various positive
reasons to provide a research infrastructure forlarge-scale evaluation for patent MT both inscientifically and for the value in real-world
applications:- MT is necessary for relevance judgmentsof the CLIR results by the users when theusers are not familiar with the target
languages.
- Resources for SMT is usable for CLIR
- SMT is especially effective for CLIR
- Bi-directorial approach (combination ofquery-translation
anddocument
translation) for CLIR has been known
effective.
Especially SMT has promises in applicability
to any language pairs as far as parallel corpora
will be available without labour-intensive efforts
to maintain dictionaries and rules up-to-date. And
for Patents, there is high possibility to obtainlarge-scale parallel corpora using patent family
relationship in various language pairs.
The scale and quality of the parallel corpora are
the key element for SMT. NTCIR-7 PAT MT
provided 1.8 M sentences exact sentence-levelLong
operatinglife

Emissionstability

Emission
intensity

1998-145000
1998-233554

Electrodecomposition

1998-107318

1998-190063
1998-209498

1998-209495

Electrodearrangement

1998-215034
1998-223930

1998-173230
1998-209499
1998-256602

1998-242515
1998-270757

Structure of lightemitting element

1998-135516
1998-242586
1998-247761

1998-242518

1998-135514
1998-256668

1998-012923
1998-247745
1998-256597

Systems automatically identify rows and columns

Figure 10. Automatic Patent Map Creation
The history of the patent classification andmining tasks at NTCIR is as following:
NTCIR-4
subtask:―5―(2003-2004):

Patent-map-creation

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan- Direct approach to creation of patent maps
- Hard tasks and insufficient evaluation
NTCIR-5 (2004-2005): Classification subtask:- Categorize
patents
topre-defined
categories called F-terms (multi-facetedand structured)- Relatively small number of test documents- Evaluate only strict matches in F-term
hierarchy
NTCIR-6 (2006-2007): Classification subtask:- Increased the number of documents andtopics (108 topics)- Evaluate partial matches in F-term
hierarchy
A facet in the F-term is selected at NTCIR-5 as atarget since the granularity is relevant for theclasses for the patent map. As an extension of thepast classification and mining task, NTCIR-7
PAT MN test the effectiveness of the mining
patents and scientific papers into a sameclassification scheme, so that the patent map canbe created using both scientific papers andpatents, and mapping patents and scientific papers
are realistic task in IP practice.
The task structure of PAT MN at NTCIR-7 isshown in Figure 11. It can be conducted as aclassification of scientific papers into IPC PatentClassification scheme, or it can be conducted as a
Cross-lingual document retrieval of patents, andfind related scientific papers using citation
relationship between them. Twelve groups
completed the experiments and submitted theresults.
n m n ng

N()

Searches and/or classifying patents and scientific papers into IPC
Research paper written in
Japanese (Japanese / J2E
subtasks)
A Participant System

Machine-translation
module (E2J / J2E)

Text classification module

Research paper written in
English (English / E2J
subtasks)

participated and completed the experiments in
MuST at NTCIR-7. This is an interesting mixtureof different communities, IR, NLP, Web
intelligence, Fuzzy, etc. A sample result was
shown in Figure 12.

Trend information (MuST)Example: Visualising the Japanesecabinet support rate

Gold standard

System output

Change over time

Very interesting results will be presented at NTCIR-7!

Figure 12. A Sample Results of MuST
The major achievements are, that we had
established an infrastructure for the investigationof the technologies for trend analysis. It includes,
a community of researcher consisting of a mixtureof related technological domains, a set of tags todescribe the trend-related information includingtime and geographical relationship, a set of tagged
corpus usable for the experiments, a platformsystem for visualization. The platform softwarefor visualization has been prepared as a software
can be used by researchers and is expected toserve as one of the common basis for further
evaluation of the trend analysis including
usability or user-involved evaluation for
exploratory. A sample display was shown in
Figure 13.

Japanese, English, and Crosslingual (J-to-E, E-to-J) subtasks
Patent data
- written in Japanese(Japanese / J2E)
- written in English(English / E2J)

List of IPC codes

Figure 11. Structure of PAT MN
The results showed the promise in n-KNN andother machine learning techniques.

2.6 Multimodal Summarization for Trend
Information [10]Multimodal
Summarization
for
Trend
Information (MuST) was organized as a pilot
workshop of the NTCIR-5 and -6, and became an
ordinary task at NTCIR-7. It investigates the task
to extract numeric expressions from a set of
documents, summarize, and visualize so that theusers easily understand the tendencies among theset of documents. The examples of the topics are
the stock market price, amount of import/exportof a particular products, etc. Thirteen groups

Figure 13. Visualization Platform for MuST

3. Test Collections
Test collections and data set built in the past
NTCIRs are listed in Table 1. Most of them areavailable for research purpose for nonparticipants. All the test collections constructed
through NTCIR-7 will be available for research
purpose for outside the NTCIR’s task participants
after the NTCIR-7 Meeting.

―6―Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

Table 1. Test collections constructed by NTCIRClass Collection

Tas
k

IR
Sci.
CLIR
NTCIR-1
abstract
on
TE*5
Scie
mtificSci.
NTCIR-2 IRabstract
CIRB010NTCIR-3
CLIR

NTCIR-4
CLIR

IR

IR

IR

News
News

News

News

CLIR
on
New
s

Filename
ntc1-je ++ntc1-j ++ntc1-e ++ntc1-tmrc ++ntc2-j ++ntc2-e ++CIRB010
KEIB010
CIRB011
CIRB020 ++Mainichi +&EIRB010Mainichi Daily ++CIRB011
CIRB020 ++Hankookilbo ++Chosenilbo ++Mainichi +&Yomiuri +&EIRB010Mainichi Daily ++Korea Times ++Hong Kong Standard
Xinhua +&CIRB040r ++Hankookilbo ++NTCIR-5
CLIR

NTCIR-6
CLIR

NTCIR-5
CLQA
CLQ
A

NTCIR-6
CLQA

IR

Chosenilbo ++Mainichi +&Yomiuri +&Mainichi Daily ++Korea Times ++Daily Yomiuri +&Xinhua +&CIRB040r ++Hankookilbo ++Chosenilbo ++Mainichi +&Yomiuri +&CIRB040r ++Yomiuri +&Daily Yomiuri +&

News

IR

News

QA

News

QA

News

CIRB020 ++Mainichi +&EIRB010Mainichi Daily ++Korea Times ++Hong Kong Standard
CIRB020 ++

OPI
IE/a
NTCIR-6
NIO OPINION naly
sis
N

NTCIR-3
PATENTIRNTCIR-4
PATENTIR

Patent
NTCIR-5
PATENT

NTCIR-6
PATENT
NTCIR-3
QA

QA

News

PatentAbstract
Abstract

CIRB040r ++Mainichi +&Yomiuri +&EIRB010Mainichi Daily ++Korea Times ++Hong Kong Standard
Xinhua +&kkh *3 ++jsh *3 ++paj *3 ++

Lang.
JE
J
E
J
J
E
Ct
K

Year
1988-1997
19861999**
1998-1999
1994

Ct
J

1998-1999

E
Ct
K
J

1998-1999

E

Ct

2000-2001

E
Ct
K

2000-2001

J
C
J
E

2000-2001

Ct
J
1998-1999
E

3 grades-49

4 grades50
30

4 grades4 grades

870MB

CtKJ
E

50

4 gradesca.3GB

CtKJ
E

60

4 grades

CtKJ
E

50

4 grades

135,124
199,681
658,719
12,15530,53017,741
198,624
901,446
85,250135,124
199,681
658,719
901,446
658,719
17,74150(selecte
d from 4 grades
NTCIR3,4)
smpl:30
3 grades0,
CJE*7
test:200

CtKJ
E

*6
J-E/JJ/E-J:
200, C- 3 grades
CJE*7
E/C-C/EC/E-E:
150

249,203
220,078
10,204
12,723
19,59996,68332, 30,28 doc901,446
32
sets for419,759 C, J, E,(selecte
1,032,277 respectiv CtKJ d from 2 types,
3
10,204 ely, were E
NTCIRmetrics
242,878 selected
3,-4,-5
50,129CLIR)
from
96,683 those409,971 describe
697,262 18GB
CtCs
31
3 grades
1,706,154 1,883MB
KJE
1,701,339 2,711MB

J
1998-2001
E

1998-1999
1995-1999
1995-1999

Publication ofunexamined patentJ

1993-1997

ca. 1,700,000 ca.27GB

Abstract

Patent Abstracts ofJapan (PAJ) ++E

1993-1997

ca. 1,700,000 ca.5GB

patent full

Publication ofunexamined patentapplication++J

1993-2002

3,496,252 ca.45GB

Abstract

Patent Abstracts ofJapan (PAJ) ++E

1993-2002

3,496,252 ca.10GB

Patent grant data
published from USPTO++

E

1993-2000

981,948

News

Mainichi +&J

1998-1999

220,078J

1998-1999

220,078 ca.776M
373,558
BJ*

IR patent full

83
60
-

249,203

Ct

J
J
E

Topic/ Question Relevanc
e judge
#

Size
Lang
577MB
312MB
J
218MB
600MB
JE
200MB
132MB CtE
74MB CtKJ

85,250K
J# of docs
339,483
332,918
187,080
2,000
400,248
134,978
132,173
66,146132,173249,508
220,078
10,204
12,723
132,173249,203
149,921104,517220,078
373,558
10,204
12,723
19,59996,683
208,167
901,446

patent fullIR

QA

Task data

Documents
Genre

260MB

E

Main:
34, Add: 3 grades
69

JE

34+1189
in NTCIR5, added
3 grades349+1681
in NTCIR6

E3221

3 grades

J*

1200

exact
answer

NTCIR-4
QA

QA

News

Mainichi +&Yomiuri +&NTCIR-5
QA

QA

News

Mainichi +&J

2000-2001

199,681 260MBJ*NTCIR-6
QA

QA

News

Mainichi +&J

1998-2001

419,759 535MB

J

197
exact
199
answer251
50
gradedseries100 Q graded(any kind (3 types,of Q) 4 levels)

11,038,720 100GB
4 grades
NW100G-01++
NTCIR-3Web
multiple* crawled inJ*
47IR+ relative
1,445,466 10GB
NW10G-01++
WEB(html/text
4
2001
multiple* crawled in
NTCIR-4
WebWEB
11,038,720 100GB
NW100G-01++J*
3 gradesIR(html/text)
4
2001
WEB
multiple* crawled in
NTCIR-5
Web
98,870,352 1.36TB
NW1000G-04++
J* 269 + 847 3 gradesIR(html/text)
4
2004
WEB
Apr.2004 to
QA siteYahoo! Q&A corpus
available
OTH
J(Chiebukuro) ++Oct.2005
on Webfor future
ERs
News
Singpore Press ++Cs
1998-2001
task
J:Japanese, E:English, C:Chinese (Ct:Traditional Chinese, Cs: Simplified Chinese), K:Korean;
All the topics/questions and relevance judgements/answers are available for research purpose for free"++" indicates the document collections available from NII for research purpose"+&" indicates the document collections available for task participants for free, and available
for research purpose use from research purpose other than NTCIR participation from other party with fee* English translation is available** gakkai subfiles: 1997-1999, kaken subfiles: 1986-1997*3: kkh : Publication of unexamined patent application, jsh: Japanese abstract, paj: English translation of jsh*4: almost Japanese or English (some in other languages)*5: Term extraction/ role analysis*6: 300+200 questions for C documents, and 300+200 questions for JE documents*7: Right, Unsupported, Wrong―7―

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

4. Participants
Table 2 is a list of the active participating
research groups in the NTCIR-7. A hundred andEitgty-seven groups registered, and eighty-two
from fifteen different countries and areas wereremained as active participants to the final
meeting.

NTCIR-782

15
12NTCIR-536

8
20# of groups

65# of countries

28

6
0

A brief overview of the Seventh NTCIR
Workshop is reported here. The details of theachievements from each task and those of eachparticipant are reported by the papers in thisvolume [3].
The test collections used in the tasks of the
NTCIR-7 and the archives of the system produced
submission raw data will be available for research
purpose. We expect that many of the researchgroups involved in the larger NTCIR community

74

9

NTCIR-1

5. Summary

77

10

NTCIR-3

submitted

85

15

40

60

80

As shown in Figures 14 the numbers ofparticipants who completed the tasks have been
almost stable in these NTCIRs. Different tasks
attracted different research groups. Manyinternational participants enrolled in ACLIA,
MOST, and PATMN. One piece of good news for
NTICIR-7 was large number of new comers joing
NTCIR. We plan to investigate the reasons why
such a large number of participants could notcomplete the experiments in time, and learn toimprove the future NTCIR organization.
Categorization.

100 120 140 160 180 200

Figure 14. Number of Participating Groups

Table 2. Active Participating Groups of the Seventh NTCIR Workshop[CCLQA]•Academia Sinica•Beijing Univ of Posts &Telecoms,
China•Carnegie Mellon Univ
•NICT
•NTT Corporation•Shenyang Institute of AeronauticalEngineering•Wuhan Univ
•Yokohama National Univ

•Hiroshima City Univ
•Information and Communications Univ
•Chinese Academy of Sciences(ISCAS)•Keio Univ
•City Univ of Hong Kong
•National Taiwan Univ
•NEC
•Northeastern Univ
•Peking Univ
•Pohang Univ of Science and Technology•Swedish Institute of Computer Science•Technical Univ of Darmstadt•Graduate Univ for Advanced•Tornado Technologies Co., Ltd.,•Toyohashi Univ of Technology•Univ of Neuchatel•Univ of Sussex[IR4QA]•Carnegie Mellon Univ
•Chaoyang Univ of Technology•Chinese Academy of Sciences(ICT)•Harbin Institute of Technology +Heilongjiang Institute of Technology•National Taiwan Univ
[Must]•Open Text Corporation•Hiroshima City Univ
•Shenyang Institute of Aeronautical•Keio Univ
Engineering•Mie Univ•Toyohashi Unive of Technology•NICT•Univ of California, Berkeley•NEC•Univ of Montreal•Ochanomizu Univ (2 Groups)•Wuhan Univ
•Wuhan Univ of Science and Technology •Okayama Univ
•Osaka Prefecture Univ
•Otaru Univ of Commerce[MOAT]•Tokyo Metropolitan Univ
•Beijing univ
•Chinese Academy of Sciences(NLPR- •Tokyo Denki Univ
•Univ of Sheffield
IACAS)•Chinese Univ of Hong Kong + Hong Kong •Yokohama National Univ
Polythechnic Univ+ Tsinghua Univ
•DAEDALUS, S.A.

―8―

[PAT MIN]•Hiroshima City Univ
•Hitachi, Ltd.,•Huafan Univ
•Nagaoka Univ of Technology•Northeastern Univ
•NTT Corporation•Peking Univ
•Shenyang Institute of AeronauticalEngineering•Toyohashi Univ of Technology•Univ of California, Berkeley•Univ of Montreal•Xerox[PAT MT]•Fudan Univ
•Harbin Institute of Technology +Heilongjiang Institute of Technology•Hitachi,Ltd.,•Japan Patent Information Organization•Kyoto Univ
•Massachusetts Institue of Technology•Nara Institute of Science andTechnology + NTT•NICT•National Taiwan Normal Univ
•NTT Corporation•Pohang Univ of Science and Technology•TOSHIBA•Tottori Univ
•Toyohashi Univ of Technology + Hosei
University•Univ of Tsukuba

Proceedings of NTCIR-7 Workshop Meeting, December 16–19, 2008, Tokyo, Japan

will work collaboratively to investigate the systemmechanisms and to analyze the further results, andthen learn each other from each other’s
experience.
WEB was not conducted at NTCIR-7. It was
not because of no interest. In opposite, we areinterested in evaluating information access on
WEB in various aspects. WEB has been
incredibly changed in many ways both in quality
and quantity of the information access in the realworld, and technologies required to support them.
Such tendencies include, for example, credibility,
quality, balance or skewedness of the retrieval
results, interaction and exploratory, social
networking and connectivities, and users
satisfaction with various tasks, etc. We then had abreak for better planning for the future tasks
investigating various aspects on the WEB as wellas those with other traditional document genresfor various usage and information seeking tasks
behind.
There are some large-scale research funding
programs on the research on WEB under multiple
government agencies in Japan, and each of themproposes to provide to the granted research groups
with large-scale shared infrastructures of researchand experiments, such as computing facilities likelarge-scale PC clusters or large-scale document
collections. But none of them provides Evaluation
infrastructures. Collaboration with these programs
shall be expected to be mutually beneficial.
Evaluation must adapt to technologicalevolution and the change in social needs. We areworking towards this goal together with researchcommunity as a collaborative community-based
effort. Any leads and suggestions are always
welcome.

References[1] NTCIR Project: http://research.nii.ac.jp/ntcir/[2] NTCIR
Workshop
7(2007-2007)
http://research.nii.ac.jp/ntcir/ntcir-ws7/meeting/:[3] Noriko Kando, Miho Sugimoto, (eds): NTCIR
Workshop 7: Proceedings of the Seventh NTCIR
Workshop Meeting on Evaluation of Information
Access Technologies: Information Retrieval,

Question
AnsweringandCross-Lingual
Information Access, Tokyo Japan, Dec. 16-19,
2008, NII, Tokyo 2008. (ISBN: 978-4-86049-0447) Available online: http://research.nii.ac.jp/ntcir/
workshop/OnlineProceedings7/[4] NTCIR Workshop 7 Meeting (Dec. 16-19, 2008)
http://research.nii.ac.jp/ntcir/ntcirws7/meeting/index.html[5] Teruko Mitamura, Eric Nyberg, Hideki Shima,
Tsuneaki Kato, Tatsunori Mori, Chin-Yew Lin,
Ruihua Song, Chuan-Jie Lin, Tetsuya Sakai,Donghong Ji and Noriko Kando. Overview of the
NTCIR-7 ACLIA Tasks: Advanced Cross-Lingual
Information Access. In Proceedings of the Seventh
NTCIR Workshop Meeting, Tokyo Japan, Dec 1619, 2008, NII, Tokyo 2008
[6] Tetsuya Sakai, Noriko Kando, Chuan-Jie Lin,
Teruko Mitamura, Hideki Shima, Donghong Ji,
Kuang-Hua Chen and Eric Nyberg. Overview of
the NTCIR-7 ACLIA IR4QA Task. In
Proceedings of the Seventh NTCIR Workshop
Meeting, Tokyo Japan, Dec 16-19, 2008, NII,
Tokyo 2008[7] Yohei Seki, David Kirk Evans, Lun-Wei Ku, Le
Sun, Hsin-Hsi Chen and Noriko Kando. Overview
of Multilingual Opinion Analysis Task at NTCIR7. In Proceedings of the Seventh NTCIR Workshop
Meeting, Tokyo Japan, Dec 16-19, 2008, NII,
Tokyo 2008
[8] Atsushi Fujii, Masao Utiyama, Mikio Yamamoto
and Takehito Utsuro. Overview of the Patent
Translation Task at the NTCIR-7 Workshop. In
Proceedings of the Seventh NTCIR Workshop
Meeting, Tokyo Japan, Dec 16-19, 2008, NII,
Tokyo 2008[9] Hidetsugu Nanba, Atsushi Fujii, Makoto Iwayama
and Taiichi Hashimoto. Overview of the Patent
Mining Task at the NTCIR-7 Workshop. In
Proceedings of the Seventh NTCIR Workshop
Meeting, Tokyo Japan, Dec 16-19, 2008, NII,
Tokyo 2008[10] Tsuneaki Kato and Mitsunori Matsushita.
Overview of MuST at the NTCIR-7 workshop -Challenges to Multi-modal Summarization for
Trend Information. In Proceedings of the Seventh
NTCIR Workshop Meeting, Tokyo Japan, Dec 1619, 2008, NII, Tokyo 2008―9―