Overview of NTCIRCharlie Clarke, Noriko Kando
NTCIR-14 General Chairs

Noriko kando @ ntcir-14 Kickoff

1NII Testbeds and Community for Information access Research
Research Infrastructure for Evaluating Information Access■A series of evaluation workshops designed to enhance research ininformation-access technologies by providing an infrastructure forevaluations. ■Data sets, evaluation methodologies, forum
Project started in late 1997
Data sets (Test collections or TCs)

4146 Users!

15780

112
108

16

NTCIR-10

17NTCIR-965

172
109

1582

12186

85

1577

109
IR: cross-lingual, patents, web, geo, time, NTCIR-3
8math, Spoken, recepe, microblogs, lifelogs
NTCIR-2
QA：monolingual, cross-lingual, interactive, NTCIR-1 6IR for QA, complex, exams
0summarization, trend info., patent maps,Inference, 2-layer summarization for mobile
opinion analysis, text mining, Intent,link
Noriko kando @ ntcir-14 Kickoff
discovery, Visual, info extraction,

13599

17NTCIR-5
NTCIR-4

105
91

20NTCIR-11

Scientific, news, patents, web, CQA,NTCIR-8
Wikipedia, exams, spoken, microblogs, lifelogs NTCIR-7
NTCIR-6Chinese, Korean, Japanese, and English

65

20NTCIR-1218 months Cycle

Tasks (Research Areas)

20

NTCIR-13

104

Task Participants

74
65# of countries

36# of active participating teams# of registered teams

28
50

100150200

2

NTCIR th 12omain Year '99 '01

3456
7
8
9'02 '04 '05 '07 '08 '10 '11

SocialMedia・UGC

Tasks inNTCIRs■

ModuleBased■

■
■

■

■

1011
12'13 '14 '16■

SpecialDomain■
□■■

■

■

□■
□■■■
□■■■■■■■■

QuesitonAnswering■■■■■
■■

■■

■■■

■

IE & IR
Semantic□■■■

Summarization

■

■

■
■

■
■

■
■

■
■

■
■

■

■Interactive■■MobileWeb

Noriko kando @ntcir-14 Kickoff

CrossLingual
TetxtRetrieval

■■■

■
■

■
■

■
■

■
■

■
■

■
■

■
■
■

■
■
■

■

■

■

■

■

■

■

■

■
■

■
■

■

■

■
■Tasks■ Microblog
Community QA
Opinion Analysis
IR + Inference
QA + IR■ Lifelog images
Recepe■ MedNLP■ Math Retrieval■ Exam (Entailment, QA)■ Spoken Q& Doc Retireval■ Temporalia
GeoTime Retrieval
Patent Retrieval■ Exam (multi-choice, essay
Any types of Qs
Dialogue
Cross-Lingual IR
Factoid, List
Link Discovery
Inference■ 2-layer (mobile)
Text Mining/Trend Info Visualization
Text Summarization■ Micoblog■ Spoken QueryVisualization■ MobileClick■ Intent, Diversification
SERPT Quality
Web Search
Machine Translation
Cross-Lingual IR
Non-EnglishIR
Ad Hoc, IR for QA

3

Tasks in
NTCIRs

MedWebQA LabNoriko kando @ntcir-14 Kickoff

Lifelog
STC
AKG
ECA
NAILS
WWW

4

IR Evaluations• TREC: Text Retrieval Conference• CLEF: Cross-Language Evaluation ForumConferece and Labs of the Evaluation Forum• NTCIR: NII Testbeds and Community forInformation access Research
Noriko kando @ ntcir-14 Kickoff

5Structure of NTCIR
NTCIRTasks
Program Co-chairs
Program Committee
Task Organizers

General Co-chairs
Steering Committee

EVIA
EVIA Co-chairs
Program CommitteeAuthers

Participants
Noriko kando @ ntcir-14 Kickoff

6

NTCIR changed name 3 times• Nacsis Test Collection for IR research– NTCIR-1• NII Test Collection for IR research– NTCIR-2～-8• NII Testbeds and Community for Information
access Research– NTCIR-9• NTCIR is a Community-Lead Activities forCreating Future Values
so, Let’s work Together!
Noriko kando @ ntcir-14 Kickoff

7

Procedures in NTCIRs• Call for Task Proposals– Selection of Task Proposals by Program Committee– Discussion about Task Design in Each Task

• TaskKickoff– Registration to Task(s)• Deliver Training Data (Documents, Topics, Answers)

Pls join
Discussion!!

– Experiments and Tuning by Each Participants• Deliver Test Data (Documents and Topics)– Experiments by Each Participants– Submission of Experimental Results– Pooling the Answer Candidates from the Submissions, and Conduct ManualJudgments– Return Answers (Relevance Judgments) and Evaluation Results– Analysis of the results, and improving the systems, etc.
– Paper writing for the Conference proceedings• Conference– Presenting the achievements of this Round– Discussion for the Next Round– Networking• Test Collection Release for non-participants
Noriko kando @ ntcir-14 Kickoff

8

Past NTCIR Conference& EVIA

9

Noriko kando @ ntcir-14 Kickoff

Past NTCIR Conference & EVIA

EVIA has attracted top experts on IR Evaluation

Noriko kando @ ntcir-14 Kickoff

10

Past NTCIR Conference

Satelite Lifelog Workshop in Glasgow
at NTCIR-12 Conference, June 2012

Noriko kando @ ntcir-14 Kickoff
11

Travel Supports are provided forOversee Task/EVIA Young Authors

Past NTCIR Conference & EVIA
Noriko kando @ ntcir-14 Kickoff

12

Uses of the Test Collections by Community# of users (who have effective licenses)■Research Purpose ■Task Participants
246

24165412 518
134 165 317

23

4

5

6

7811
81376 1366

910371

246

2060
1623 1877

1112746

4,164

519
371

97208 107
101721716

2228 2452

13142737 2980

1516173283 3418

1819International Users vs Domestic Users
92■International ■Domestic
108

1501141861251640
292327447390

38221656791

93

82
9942

18896892

68

101112107
3913138 111

4987
14211 53
116 116 132
15161782
18

Noriko kando @ ntcir-14 Kickoff

1913

Courses of a Task
Special Sessions at
Conference
Idea
Intensive Discussion- SWIRL, Dagstuhl, NII
Shonan- Post Conference WSs, etc

Journal Special
Issues
So far, IP&M, ACM TALIP,IRJ, etc

Paper Submissions
toOther CoferencesandJournals

Paper at NTCIRConference

Task Proposal

Task at NTCIR

Noriko kando @ ntcir-14 Kickoff

LNCS

Paper at EVIA

14

NII Testbeds and Community for
Information access ResearchWhy?
-Data sets / Users’ Information Seeking Tasks-Evaluation Methodology-Reusable vs Reproducibility-User-Centered Evaluation-Experimental Platforms-Open Advancement-Advanced NLP  Knowledge- or Semantic-based-Diversified IA Applications in the Real World-Best Practice for a technology
 Best Practice for Evaluation Methodology-Big Data (Documents + Behaviour data)
Noriko kando @ ntcir-14 Kickoff

15Mid-term Vision

Create Future
ValueConversation

Weaving out
KnowledgeMobileUsers Task andBig Data
Context
Intension

Integrating
NLP・IR

Reproducibility

Documents -> Answer,
Information

Datasets

Interaction &Visualization

Module-based Module-based Platform

Large-Scale Reusable TCs

Cross-lingual
Users’ Information Seeking
Tasks + Genres
NTCIR-1

NTCIR-3

Interaction &Live Evaluation(Open Advancement)

Opinion/Attitudes
Noriko kando @ ntcir-14 Kickoff
NTCIR-6

NTCIR-9

NTCIR-12

NTCIR-14
16