NII Testbeds and Community for Information access Research

Overview of NTCIRCharles L. A. Clarke
University of Waterloo

Noriko Kando
National Institute of Informatics

Yiqun Liu
Tsinghua University

Makoto P. Kato
University of Tsukuba

NII Testbeds and Community for Information access Research
Research Infrastructure for Evaluating Information AccessA series of evaluation designed to enhance research in information accesstechnologies by providing an evaluation infrastructure: datasets, evaluation
methodologies, and forum
Project started in late 1997
18 months cycle

Test collections
Scientific, news, patents, web, CQA, Wikipedia, exams,spoken, microblogs, lifelogs
Chinese, Korean, Japanese, and English

Tasks (Research areas)
IR: cross-lingual, patents, web, geo, time, math, Spoken,recipe, microblogs, lifelogs, open data
QA: monolingual, cross-lingual, interactive, IR for QA,complex, exams
Summarization, trend info., patent maps, inference, 2-layersummarization for mobile, dialogue generation andevaluation
Opinion analysis, text mining, intent mining, link discovery,
information extraction, action detection

NTCIR-1555

NTCIR-144720

NTCIR-13766510520

NTCIR-129120NTCIR-11# of Participants

801578011216NTCIR-1010817NTCIR-99917NTCIR-86582

12NTCIR-674

9NTCIR-36NTCIR-10# of countries /regions

65

8NTCIR-2

10477

10

NTCIR-4

186

85

15

NTCIR-5172
10915

NTCIR-7135# of active participating teams

36# of registered teams

2850100150200

Statistics of NTCIR Test Collections

3# of NTCIR test collection users(= # of NTCIR participants + # of applicants for using the collections)
研究⽬的利⽤
ワークショップ参加者■Applicants■NTCIR participants

20810797246

18771011376 1366 1623811246371

2060 2228

371

2452

519

721

716

3283
2737 2980

746

785

3418 3644

785

4,723

3938

518
134 165 317 412
65241999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

# of applicants for using the collection
206

from Japanese and oversea institutes
海外
国内■Overseas■Japan92
108

150
114
73

90

91

93

82
99

107

138

111

116

132

49

45
94
211

116

53

181

112
87
82
68
68
25
42
39
38
27
18
2218
16
16
64
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019
402992

History of NTCIR
Year

4
1999

200120022004200520072008

2010

2011

2013

2014

2016

2017

2019

Task/NTCIR round

12

34

56

78910111213

14

Automatic Term Recognition and Role Analysis (TMREC) (1)9Ad hoc/Crosslingual IR (1) -> Chinese/English/Japanese IR (2) -> CLIR (3-6)

283020262522

98

9

Web Retrieval (WEB) (3-5)

711

Question Answering Challenge (QAC) (3-6)

16

18

7

8

Patent Retrieval [and Classification] (PATENT) (3-6)

10

10

13

12

Multimodal Summarization for Trend Information (MUST) (5-7)

1315

13

Crosslingual Question Answering (CLQA) (5, 6) ->14121914122116121112911

Text Summarization Challenge (TSC) (2-4)

7Advanced Crosslingual Information Access (ACLIA) (7, 8)
Opinion (6) -> Multilingual Opinion Analysis (MOAT) (7, 8)
Patent Mining (PAT-MN) (7, 8)
Community Question Answering (CQA) (8)4Geotemporal IR (GeoTime) (8, 9)

13

128

2121

Crosslingual Link Discovery (Crosslink) (9, 10)

11

10

INTENT(9, 10) -> Search Intent and Task Mining (IMine) (11, 12)1611Interactive Visual Exploration (Vis-Ex) (9)4Patent Translation (PAT-MT)(7, 8) ->15Patent Machine Translation (PatentMT)(9, 10)

One Click Access (1CLICK)(9, 10) -> Mobile Information Access (MobileClick) (11,12)484

Recognizing Inference in Text (RITE)(9,10)

2428

23

1012

11

7

-> Recognizing Inference in Text and Validation (RITE-VAL)(11)
IR for Spoken Documents (SpokenDoc) (9, 10)-> Spoken Query and Spoken Document Retrieval (SpokenQuery&Doc) (11, 12)
Mathematical Information Access (Math) (10, 11) -> MathIR (12)686Medical Natural Language Processing (MedNLP) (10, 11) -> MedNLPDoc (12)

1212

81112

Temporal Information Access (Temporalia) (11, 12)814

Cooking Recipe Search (RecipeSearch) (11)4-> MedWeb (13)9QA Lab for Entrance Exam (QALab) (11, 12, 13)11

13-> QA Lab for Political Information (QALab-PoliInfo) (14)Personal Lifelog Organisation & Retrieval (Lifelog) (12, 13, 14)84

6

Short Text Conversation (STC) (12, 13, 14)

22

27

13

Open Live Test for Question Retrieval (OpenLiveQ) (13, 14)7

4

Actionable Knowledge Graph (AKG) (13)3Emotion Cause Analysis (ECA) (13)3Neurally Augmented Image Labelling Strategies (NAILS) (13)2We Want Web (WWW) (13, 14)5Fine-Grained Numeral Understanding in Financial Tweet (FinNum) (14)46

CLEF/NTCIR/TREC REproducibility (CENTRE) (14)137396174

7981806610210893977147

Tasks in NTCIR (1999 ~ 2008) NII Test Collection for Information Retrieval systems
Year
Task/NTCIR round
Automatic Term Recognition and Role Analysis(TMREC) (1)Ad hoc/Crosslingual IR (1) ->Chinese/English/Japanese IR (2) -> CLIR (3-6)
Text Summarization Challenge (TSC) (2-4)
Web Retrieval (WEB) (3-5)
Question Answering Challenge (QAC) (3-6)
Patent Retrieval [and Classification] (PATENT) (36)Multimodal Summarization for Trend Information(MUST) (5-7)
Crosslingual Question Answering (CLQA) (5, 6) ->Advanced Crosslingual Information
Access (ACLIA) (7, 8)
Opinion (6) -> Multilingual Opinion Analysis(MOAT) (7, 8)
Patent Mining (PAT-MN) (7, 8)

51999

2001

2002

2004

2005

2007

2008

1

2

3

4

5

6

730
2026 IR25• Crosslingual

22

928

98

916

18

78

101013121315

131412191221• Text summarization
7117• Patent retrieval/mining• QA• Web search• Opinion mining12

Tasks in NTCIR (2010 ~ 2019) NII Testbeds and Community for Information access ResearchYear
Task/NTCIR round
Crosslingual Link Discovery (Crosslink) (9, 10)
INTENT(9, 10) -> Search Intent and Task Mining (IMine) (11, 12)
One Click Access (1CLICK)(9, 10) -> Mobile Information Access (MobileClick) (11,12)
Recognizing Inference in Text (RITE)(9,10)-> Recognizing Inference in Text and Validation (RITE-VAL)(11)
IR for Spoken Documents (SpokenDoc) (9, 10)-> Spoken Query and Spoken Document Retrieval (SpokenQuery&Doc) (11, 12)Mathematical Information Access (Math) (10, 11) -> MathIR (12)Medical Natural Language Processing (MedNLP) (10, 11) -> MedNLPDoc (12)-> MedWeb (13)QA Lab for Entrance Exam (QALab) (11, 12, 13)-> QA Lab for Political Information (QALab-PoliInfo) (14)Temporal Information Access (Temporalia) (11, 12)
Cooking Recipe Search (RecipeSearch) (11)Personal Lifelog Organisation & Retrieval (Lifelog) (12, 13, 14)
Short Text Conversation (STC) (12, 13, 14)Open Live Test for Question Retrieval (OpenLiveQ) (13, 14)
Actionable Knowledge Graph (AKG) (13)
Emotion Cause Analysis (ECA) (13)Neurally Augmented Image Labelling Strategies (NAILS) (13)
We Want Web (WWW) (13, 14)
Fine-Grained Numeral Understanding in Financial Tweet (FinNum) (14)CLEF/NTCIR/TREC REproducibility (CENTRE) (14)

62010

2011

2013

2014

2016

2017

2019

89101112

13

14

•
•
•
•
•
•
•
•••
•
•••
•
•
•
•

66

Link discovery
111016 intent 11Searchmining 128
4Mobile4 IR
24
28
23
Text understanding
Spoken10 IR12
11
IR for math8
Medical NLP 61212
Q&A for entrance exam
11
Temporal IRIR for recipes
8IR for lifelogs
4Dialogue generation
Online evaluation
Knowledge graph
Neural signal processing
Web search (again)
Numerical understanding
Reproducibility
10210893911768912111384622271374143325461977147

How a Task Has Been Evolved ‒ An Example of Patent Retrieval
Networking

Doc
Donation

IR experts
IP+IR+ISexperts
Brain
Storming

SIGIR
2000 WS

Collaboration
with IPExperts

7

Breakthrough!
NTCIR-7 to -12: SMT beats
Patent MT
RMT!
(thanks for NICT)

Datasets usable
for NMTCLEF-IPMultiLingualNTCIR-3 to-6: Patent IR
TaskTASKs

Professional
SearchTREC
Chemical IR

Hands-on Tutorials
Industries,Workshops, JSpecial issues, etc. POs

IRF
Collabrative
IR, Ontology,

NII Testbeds and Community for Information access ResearchWhy?
•-Datasets / Usersʼ Information Seeking Tasks
Evaluation Methodology
Reusable vs Reproducibility
User-Centered Evaluation
Experimental Platforms
Open Advancement
Advanced NLP à Knowledge- or Semantic-based
Diversified IA Applications in the Real WorldBest Practice of Technologies
à Best Practice of Evaluation Methodology
Big Data (Documents + Behavior data)

8Past NTCIR Conferences (Offline)

99

Past NTCIR Conferences (Offline)

10

Past NTCIR Conferences (Offline)

11

NTCIR-15 Conference - Virtual Conference!

12• 7 Evaluation Tasks• 43 Task Organizers• 57 = Papers in the NTCIR-15 proceedings• 193 = Authors• 270 Participants(as of Dec 11)

References:
NTCIR 10, June 2013: 208
NTCIR 11, Dec 2014: 199
NTCIR 12, June 2016: 245
NTCIR 13, Dec 2017: 203
NTCIR 14, June 2019: 160

NTCIR-15 Tasks• CORE TASKS
1. DialEval-1: Dialogue Evaluation
2. FinNum-2: Numeral Attachment in Financial Tweets
3. QA Lab-PoliInfo-2: Question Answering Lab for
Political Information
4. SHINRA2020-ML: SHINRA 2020 Multi-lingual
5. WWW-3: We Want Web with CENTRE• PILOT TASKS
6. Data Search: Data Search
7. MART: Micro Activity Retrieval Task

13

NTCIR-15 Conference Organizing Committee
General Co-Chairs

Charles L. A. Clarke
Noriko Kando
Madoka Ishioroshi
Sumika Masui
Tomoko Ohsuga
Yuko Tanaka

University of Waterloo, Canada
National Institute of Informatics, Japan
National Institute of Informatics, Japan
National Institute of Informatics, Japan
National Institute of Informatics, Japan
National Institute of Informatics, Japan

Makoto P. Kato
Hiroaki Ohshima
Organizing Committee
Emi Ishita
Yoshiyuki Shoji
Masao Takaku
Takehiro Yamamoto
Masaharu Yoshioka
Social Platform Chair
Tomoko Ohsuga
Hideyuki Shibuki
Publication Co-Chairs
Makoto P. Kato
Yiqun Liu
Publicity Chair
Hideyuki Shibuki
Program Committee Co-Chairs Noriko Kando
Makoto P. Kato
Yiqun Liu
Secretariat
Sumika Masui
Yuko Tanaka

University of Tsukuba, Japan
University of Hyogo, Japan
Kyushu University, Japan
Aoyama Gakuin University, Japan
University of Tsukuba, Japan
University of Hyogo, Japan
Hokkaido University, Japan
National Institute of Informatics, Japan
National Institute of Informatics, Japan
University of Tsukuba, JapanTsinghua University, China
National Institute of Informatics, Japan
National Institute of Informatics, Japan
University of Tsukuba, Japan
Tsinghua University, China
National Institute of Informatics, Japan
National Institute of Informatics, Japan

Local Arrangement Chair
Local Arrangements

Organizing Committee CoChairs

NTCIR Book

17

• Sakai, T., Oard, D., and Kando, N. (eds.):
Evaluating Information Retrieval andAccess Tasks: NTCIRʼs Legacy of
Research Impact, Springer, 2020.
‒ https://www.springer.com/gp/book/97898
11555534
Book Review;https://irsg.bcs.org/informer/2020/10/bookreview-evaluating-information-retrieval-andaccess-tasks-ntcirs-legacy-of-researchimpact/

OPEN ACCESS