NII Testbeds and Community for Information access Research

Overview of NTCIRCharles L. A. Clarke
University of Waterloo

Noriko Kando
National Institute of Informatics

Yiqun Liu
Tsinghua University

Makoto P. Kato
University of Tsukuba

NII Testbeds and Community for Information access Research
Research Infrastructure for Evaluating Information AccessA series of evaluation designed to enhance research in information accesstechnologies by providing an evaluation infrastructure: datasets, evaluation
methodologies, and forum
Project started in late 1997
18 months cycle

Test collections

5,078 user teams

Scientific, news, patents, web, CQA, Wikipedia, exams,
spoken, microblogs, lifelogs
Chinese, Korean, Japanese, and English

Tasks (Research areas)
IR: cross-lingual, patents, web, geo, time, math, Spoken,
recipe, microblogs, lifelogs, open data
QA: monolingual, cross-lingual, interactive, IR for QA,
complex, exams
Summarization, trend info., patent maps, inference, 2-layersummarization for mobile, dialogue generation andevaluation
Opinion analysis, text mining, intent mining, link discovery,
information extraction, action detection

20NTCIR-16
NTCIR-1515NTCIR-14155382558047

NTCIR-1320

NTCIR-1220

NTCIR-1120# of participating teams

766510591

NTCIR-10

16NTCIR-917NTCIR-817NTCIR-71582

NTCIR-61285

NTCIR-515

NTCIR-4

10

NTCIR-3

9

NTCIR-2

8

NTCIR-1

60157

80

112
108

1359965172109186104

77# of countries /regions74# active teams

65# registered teams

36
2850100

150200Statistics of NTCIR Test Collections

3# of NTCIR test collection users(= # of NTCIR participants + # of applicants for using the collections)
研究目的利用
ワークショップ参加者■Applicants■NTCIR participants
721716

746

785

785

4,723

519

208

107

97

246

1877
101
1376 1366 1623
811

246

371

2060 2228

371

2452

3283
2737 2980

3418 3644

3938

518
134 165 317 412
65241999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

# of applicants for using the collection
from Japanese and oversea institutes
92

海外
国内
■Overseas■Japan108

150
114
73

90

206

91

93

82
99

107

138

45

111

94

49

211
116

116

132

53

181

112
87
82
68
68
25
42
39
38
27
18
22
18
16
16
64
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
402992

History of NTCIRYear

4
1999

2001

2002

2004

2005

2007

2008

2010

2011

2013

2014

2016

2017

2019

20202022

Task/NTCIR round

123456789101112131415

16

Automatic Term Recognition and Role Analysis (TMREC) (1)9Ad hoc/Crosslingual IR (1) -> Chinese/English/Japanese IR (2) -> CLIR (3-6)

283020262522989Web Retrieval (WEB) (3-5)

711

Question Answering Challenge (QAC) (3-6)

1618

78

Patent Retrieval [and Classification] (PATENT) (3-6)

10101312

Multimodal Summarization for Trend Information (MUST) (5-7)

131513Crosslingual Question Answering (CLQA) (5, 6) ->141219141221

1612111314

Text Summarization Challenge (TSC) (2-4)

7Advanced Crosslingual Information Access (ACLIA) (7, 8)
Opinion (6) -> Multilingual Opinion Analysis (MOAT) (7, 8)
Patent Mining (PAT-MN) (7, 8)
Community Question Answering (CQA) (8)4Geotemporal IR (GeoTime) (8, 9)

13128

21

21

Crosslingual Link Discovery (Crosslink) (9, 10)1110

INTENT(9, 10) -> Search Intent and Task Mining (IMine) (11, 12)

1611129One Click Access (1CLICK)(9, 10) -> Mobile Information Access (MobileClick) (11, 12)

48

4

11

Recognizing Inference in Text (RITE)(9,10)24

28

231012

11

7

Interactive Visual Exploration (Vis-Ex) (9)4Patent Translation (PAT-MT)(7, 8) ->15

Patent Machine Translation (PatentMT)(9, 10)-> Recognizing Inference in Text and Validation (RITE-VAL)(11)
IR for Spoken Documents (SpokenDoc) (9, 10)-> Spoken Query and Spoken Document Retrieval (SpokenQuery&Doc) (11, 12)
Mathematical Information Access (Math) (10, 11) -> MathIR (12)68

6Medical Natural Language Processing (MedNLP) (10, 11) -> MedNLPDoc (12)

1212

81112

Temporal Information Access (Temporalia) (11, 12)814

Cooking Recipe Search (RecipeSearch) (11)4-> MedWeb (13) -> Real-MedNLP(16)109

QA Lab for Entrance Exam (QALab) (11, 12, 13)1112-> QA Lab for Political Information (QALab-PoliInfo) (14, 15, 16)

Personal Lifelog Organisation & Retrieval (Lifelog) (12, 13, 14, 16)84

6

Short Text Conversation (STC) (12, 13, 14)22

27

13Open Live Test for Question Retrieval (OpenLiveQ) (13, 14)74

Actionable Knowledge Graph (AKG) (13)3Emotion Cause Analysis (ECA) (13)3Neurally Augmented Image Labelling Strategies (NAILS) (13)2We Want Web (WWW) (13, 14) -> We Want Web with CENTER (WWW) (15, 16)

5348

3Fine-Grained Numeral Understanding in Financial Tweet (FinNum) (14, 15, 16)

6

7

7

CLEF/NTCIR/TREC REproducibility (CENTRE) (14)1

Dialogue Evaluation (DialEval) (15, 16)

74

SHINRA 2020 Multi-lingual (SHINRA 2020-ML) (15)7

Data Search (Data Search) (15, 16)5

Micro Activity Retrieval Task (MART) (15)

56Session Search (SS) (16)3

Reading Comprehension for Information Retrieval (RCIR) (16)3

Unbiased Learning to Ranking Evaluation Task (ULTRE) (16)23739617479818066

102108939771

475253

Tasks in NTCIR (1999 ~ 2008) NII Test Collection for Information Retrieval systems
Year
Task/NTCIR round
Automatic Term Recognition and Role Analysis(TMREC) (1)Ad hoc/Crosslingual IR (1) ->Chinese/English/Japanese IR (2) -> CLIR (3-6)

Text Summarization Challenge (TSC) (2-4)
Web Retrieval (WEB) (3-5)
Question Answering Challenge (QAC) (3-6)

Patent Retrieval [and Classification] (PATENT) (36)Multimodal Summarization for Trend Information(MUST) (5-7)
Crosslingual Question Answering (CLQA) (5, 6) ->Advanced Crosslingual Information
Access (ACLIA) (7, 8)
Opinion (6) -> Multilingual Opinion Analysis(MOAT) (7, 8)
Patent Mining (PAT-MN) (7, 8)

51999

2001

2002

2004

2005

2007

2008

1

2

3

4

5

6

730
2026 IR25• Crosslingual

22

928

98

916

18

78

10

10

13

1213

15

1314

1219

1221

• Text summarization
7117• Patent retrieval/mining• QA• Web search• Opinion mining12

Tasks in NTCIR (2010 ~ 2019) NII Testbeds and Community for Information access ResearchYear

Task/NTCIR round
Crosslingual Link Discovery (Crosslink) (9, 10)
INTENT(9, 10) -> Search Intent and Task Mining (IMine) (11, 12)
One Click Access (1CLICK)(9, 10) -> Mobile Information Access (MobileClick) (11,12)
Recognizing Inference in Text (RITE)(9,10)-> Recognizing Inference in Text and Validation (RITE-VAL)(11)

IR for Spoken Documents (SpokenDoc) (9, 10)-> Spoken Query and Spoken Document Retrieval (SpokenQuery&Doc) (11, 12)
Mathematical Information Access (Math) (10, 11) -> MathIR (12)Medical Natural Language Processing (MedNLP) (10, 11) -> MedNLPDoc (12)-> MedWeb (13)QA Lab for Entrance Exam (QALab) (11, 12, 13)-> QA Lab for Political Information (QALab-PoliInfo) (14)Temporal Information Access (Temporalia) (11, 12)
Cooking Recipe Search (RecipeSearch) (11)Personal Lifelog Organisation & Retrieval (Lifelog) (12, 13, 14)
Short Text Conversation (STC) (12, 13, 14)Open Live Test for Question Retrieval (OpenLiveQ) (13, 14)

Actionable Knowledge Graph (AKG) (13)
Emotion Cause Analysis (ECA) (13)Neurally Augmented Image Labelling Strategies (NAILS) (13)
We Want Web (WWW) (13, 14)
Fine-Grained Numeral Understanding in Financial Tweet (FinNum) (14)CLEF/NTCIR/TREC REproducibility (CENTRE) (14)

62010

2011

2013

2014

2016

2017

2019

8

91011

12

13

14

•
•
•
•
••
•
•
•
•
•••
•
•••
•

66

Link discovery
111016 intent 11Searchmining 12
8
4Mobile4 IR
24
28
23
Text understanding
Spoken
10 IR
1211
IR for math8
Medical NLP 612
12
Q&A for entrance exam
11
Temporal IR
IR for recipes
8IR for lifelogs
4Dialogue generation
Online evaluation
Knowledge graph
Neural signal processing
Web search (again)
Numerical understanding
Reproducibility
10210893911768912111384622271374143325461977147

Recent Tasks in NTCIR (2020 ~ 2022)
Year

72014

Task/NTCIR round11

Medical Natural Language Processing (MedNLP) (10, 11) -> MedNLPDoc (12)

•••

11

-> QA Lab for Political Information (QALab-PoliInfo) (14, 15, 16)•••••

We Want Web (WWW) (13, 14) -> We Want Web with CENTER (WWW) (15, 16)

Fine-Grained Numeral Understanding in Financial Tweet (FinNum) (14, 15, 16)

2020

1213
11Lifelog
Activity recognition
Eye-gaze information for IR••

SHINRA 2020 Multi-lingual (SHINRA 2020-ML) (15)

Q&A for politicalinformation
4
5
Dialogue evaluation
6•

Micro Activity Retrieval Task (MART) (15)Multilingual document classification•••

Reading Comprehension for Information Retrieval (RCIR) (16)
Unbiased Learning to Ranking Evaluation Task (ULTRE) (16)

93

37771

4653Web search
Session searchUnbiased learning to rank

97

1285

• Advanced IR

Session Search (SS) (16)

16

3

7
Numeral understanding for financial texts
Data search
7• Collaboration

Data Search (Data Search) (15, 16)146• Numbers

Dialogue Evaluation (DialEval) (15, 16)

2022

10984
• Dialogues

Personal Lifelog Organisation & Retrieval (Lifelog) (12, 13, 14, 16)

2019

1213
14
15
Medical NLP
Q&A for political information
Numeral understanding for financial texts• Sensors-> MedWeb (13) -> Real-MedNLP (16)QA Lab for Entrance Exam (QALab) (11, 12, 13)

2016
2017
• Specialists

47

3
25253

NII Testbeds and Community for Information access ResearchWhy?

1. Find successes and failures–

8For participants

Conference papers do not always report failures

2. Boost up your research in an early stage–Well-defined tasks, large datasets, opportunities for discussion

3. Share the goal of research–

Especially when you are tackling a big problem

4. Develop large test collections–

For organizers

IR test collections need participants' contributions

5. Build international teams for multilingual tasks–

Many multilingual tasks have been proposed in NTCIRPast NTCIR Conferences (Offline)

99

Past NTCIR Conferences (Offline)

10

Past NTCIR Conferences (Offline)

11

NTCIR-16 Conference•••••••

12Report out from 10 different tasks
Tutorial by Tetsuya Sakai: Evaluating Evaluation Measures, Evaluating Information Access
Systems, Designing and Constructing Test Collections, and Evaluating Again
Keynotes:
Chengxiang Zhai: IR Evaluation as Search Simulation
Ellen Voorhees: Cranfield is Dead; Long Live Cranfield
Falk Scholer: The Impact of Query Variability and Relevance Measurement
Scales on Information Retrieval Evaluation
Invited talks on CLEF (Nicola Ferro), NeuCLIR (Douglas Oard),
Marth Larson &MediaEval (Marth Larson & Gareth Jones)
Gareth Jones

Tetsuya Sakai

Chengxian Zhai Ellen Voorhees Falk Scholer

Nicoal Ferro Douglas OardNTCIR-16 Conference - Statistics

13• 10 Evaluation Tasks• 59 Task Organizers (allows duplicates)• 69 = Papers in the NTCIR-15 proceedings• 226 = Authors• 298 Participants(as of June 16, 2022)

References:
NTCIR 10, June 2013: 208
NTCIR 11, Dec 2014: 199
NTCIR 12, June 2016: 245

NTCIR 13, Dec 2017: 203
NTCIR 14, June 2019: 160
NTCIR 15, Dec 2020: 270

NTCIR Book

17

• Sakai, T., Oard, D., and Kando, N. (eds.):
Evaluating Information Retrieval andAccess Tasks: NTCIR’s Legacy of
Research Impact, Springer, 2020.
– https://www.springer.com/gp/book/97898
11555534
Book Review;https://irsg.bcs.org/informer/2020/10/bookreview-evaluating-information-retrieval-andaccess-tasks-ntcirs-legacy-of-researchimpact/

OPEN ACCESS