Welcome!

Twitter: #ntcir11
Ust: ntcir11-kickoff

NTCIR-11 Kick-Off Event
Sep 02, 2013
English Session: 13:30Japanese Session: 15:151

Program• About NTCIR• About NTCIR-11• Accepted Tasks• Why participate?
• How to participate• Important Dates• Q&A2

About NTCIR
Introduction

3

NII Testbeds and Community for Information access Research
Research Infrastructure for Evaluating Information Access■A series of evaluation workshops designed to enhance research ininformation-access technologies by providing an infrastructure forlarge-scale evaluations. ■Data sets, evaluation methodologies, forum
Project started in late 1997
18 months Cycle

NTCIR-917
17

NTCIR-817

NTCIR-71582

NTCIR-61285

NTCIR-515

NTCIR-10

2,347 Users!

Data sets (Test collections or TCs)

10
NTCIR-4
Scientific, news, patents, web, CQA, Wiki,
9
NTCIR-3
Exams# Participants
NTCIR-2 to 8
6
Chinese, Korean, Japanese, and English
NTCIR-1
Task(s)Tasks (Research Areas)
IR: Cross-lingual tasks, patents, web, Geo,
Spoken
QA：Monolingual tasks, cross-lingual tasks
Summarization, trend info., patent maps,
Inference,
Opinion analysis, text mining, Intent, Link
2013-09-02Visual
ntcir-11 kick-off
Discovery,

0

125
90
65

110
103
# of countries

77
74# of Active
participant groups

65
36# of registered(types)

28
50

100

150

Task Proposal Discussion

NTCIR-10 (2012-2013)
125 Teams Registered to Task(s)963 Teams Registered so far

4Tasks in the NTCIR
NTCIR th 1 2 3 4 5 6 7 8 9 10 Tasks
DomainYear '99 '01 '02 '04 '05 '07 '08 '10 '11 '13■
Community QASocial Media・UGC■ ■ ■Opinion Analysis■ IR + Inference
Module-Based■ ■QA + IR■ MedNLP■ Math Retrieval■ ■ Entrance ExamSpecial Domain■ ■ Spoken Doc Retireval■ ■GeoTime Retrieval■ ■ ■ ■ □ □Patent Retrieval■ ■ ■ □
Any types of Qs
Quesiton■ ■Dialogue
Answering■ ■ ■ ■
Cross-Lingual IR■ ■ ■ ■
Factoid, List
IE & IR■ ■ Link Discovery
Semantic■ ■ Inference■ ■ ■ ■ ■Text Mining/
Summarization■ ■ ■Trend Info Visualization■ ■ ■Text Summarization
Interactive■
Interactive IR and Visualization■ ■ Intent mining, Diversified Searc
Web■ ■ SERPT Quality
Web Search■ ■ ■■ ■ ■ ■ Machine Translation
Cross-Lingual■ ■ ■ ■ ■ ■ ■ ■Cross-Lingual IR■ ■ ■ ■ ■ ■ ■ ■Non-EnglishIR
Tetxt Retrieval ■ ■ ■ ■ ■ ■ ■ ■Ad Hoc, IR for QA
Year that the conference was held, The Tasks started 18 Months before

2013-09-02 ntcir-11 kick-off

Tasks in NTCIR

5

Information retrieval (IR)• Retrieve RELEVANT information from vast collection to meetusers’ information needs• Using computers since the 1950s• First CS uses human assessments as success criteria– Judgments vary
– Comparative evaluations on the same infrastructure

Information access (IA)Whole process to make information usable by users.
ex.: IR, text summarization, QA, text mining, andclustering
2013-09-02 ntcir-11 kick-off

6

Procedures in NTCIR Workshops• Call for Task Proposals• Selection of Task Proposals by Committee• Discussion about Experimental Designs and Evaluation Methods(can be continued to Formal Runs)• Registration to Task(s)– Deliver Training Data (Documents, Topics, Answers)• Experiments and Tuning by Each Participants– Deliver Test Data (Documents and Topics)• Experiments by Each Participants• Submission of Experimental Results• Pooling the Answer Candidates from the Submissions, andConduct Manual Judgments• Return Answers (Relevance Judgments) and Evaluation Results• Conference & EVIA
Discussion for the Next Round2013-09-02 ntcir-11 kick-off

7

NTCIR-10 Conference and EVIA 2013 June 18-21, 2013
Keynotes

EVIA

Breakout Session
Poster andDemo

2013-09-02 ntcir-11 kick-off

Task Proposal
Discussion88

High Quality Access to WEB

NTCIR-10 (2011-2013) Organizers
Challenge to ML
CrossLink

INTENTMin Zhang
Tsinghua Univ. China

In-Su Kang

Shlomo Geva Kyungsung Univ. KR
QUT, AU

Ruihua SongMSR China

Haitao Mi
ICT, China

General Co-chairs

Andrew Trotman
Univ. Otago, NZ

RitsumeikanU

Patent MT
Ka Po Chow
Hong Kong Inst. Edu.

Yue Xu
QUT, AU

Tsuneaki Kato
U of Tokyo

Takehiro Yamamoto
Kyoto U

Makoto P. Kato
Kyoto U

Shahzad Rajput
Northeastern Univ. USA

Matthew Ekstrand-Abueg
Virgil Pavlu
Northeastern Univ. USA
Northeastern Univ. USA

Young-In Song
MSR, China

Noriko Kanod
NII

神門 典子Douglas W. OardNII Univ. Maryland, USA

Program Co-chairs

Isao Gogo

Tetsuya Sakai
MSR, China

Yiqun Liu

QUT, AU

Fuminori
KimuraMayu Iwata
Osaka U

Eric Tang

1CLICK

Zhicheng Dou
MRA, China

Mark Sanderson
RMIT, AUEVIA Co-Chairs
Akiko Aizawa
NII

Bin LuCity Univ. Hong Kong
Benjamin K. Tsou
Eiichiro Sumita
Hong Kong Inst. Edu.
NICT

SpokenDoc

Hideo Joho
Tsukuba U

Tetsuya Sakai
MSR, China

Ian Soboroff, NIST Ruihua Song
William Webber
MSR, China Univ. Maryland, USA

Gareth Jones,
Dublin City U
Kal Javelin,
Iadh Ounis
Charlie Clark Tempare Univ Univ. Glasgow, UK
U. Waterloo

Michael Kohlhase
Jacobs Univ. Bremen
Fredric C. Gey
U.C.Berkeley, USA

Program Committee

RITE

Akiba Tomoyosi
Seiichi Nakagawa
Kiiyoaki Aikawa
TUT
南條 浩輝
Tokyo Tech U
Cheng-Wei Lee
龍⾕⼤学
Academia Sinica, Taiwan
NICT

Tatsuya Kaahara
Yoishi Yamashita
Kyoto U
Ritsumeikan U

Takamitsu Nisizaki
Yamanashi U

Hideki Shima
CMU, USA
Teruko Mitamura

Challenge to Multi Media

MATH

Iadh Ounis
Univ. Glasgow, UK

Noriko Kando
NII

Chuan-Jie Lin
Yotaro Watanabe
National Taiwan Ocean Univ.
Tohoku U

Yusuke Miyao
NII

Teruko Mitamura
Noriko Kando
CMU
NII

Tomohide Shimaga
Kyoto U
Junta Mizuno太
NICT

Mizuki MoritaEiji Aramaki
U of Tokyo Kyoto U
Yoshinobu Kano
Presto/NII

MedNLP

Towards Knowledgebase Construction
Hiroshi Kanayama
IBM株)
Koichi Takeda
IBMfrom Texts in the Specific Domains

NTCIR-11 Organization<NTCIR-11 Program Co-Chairs>- Hideo Joho（University of Tsukuba, Japan）- Kazuaki Kishida（Keio University, Japan)<EVIA2014 Co-Chairs>-Ruihua Song (Microsoft Research Asia, China)-TBA<NTCIR-10 General Co-Chairs>- Noriko Kando (NII, Japan)- Tsuneaki Kato (The University of Tokyo, Japan)- Douglas W. Oard (University of Maryland, USA)
- Tetsuya Sakai (Waseda University)- Mark Sanderson (RMIT University, Australia)-Many Task Organizers from all over the world

2013-09-02 ntcir-11 kick-off

10

NTCIR Test Collections

Test Collections = Docs + Topics/Questions + Answers

2013-09-02 ntcir-11 kick-off

Available to Non-participants for Research Purpose

11

Evaluation Forums• Research teams gather up to solve sharedproblems; submit system output before
deadline• Systems evaluatedacross teams
Task andand
inputcompareddata
Participant
System output
Organisers
Results andvalue-added data

ParticipantParticipantParticipant

Share data and insights, discuss, improve2013-09-02 ntcir-11 kick-off

12Why Evaluation Forums?
• Compete and collaborate, accelerate research• Build large-scale test collections throughcollective efforts e.g. pooling• Foster interdisciplinary research towards
grand challenges, build new researchcommunities
NII Testbeds and Community for Information accessResearch
2013-09-02 ntcir-11 kick-off

13

Information Retrieval Evaluation
Forums• TREC (Text Retrieval Conference) 1992• NTCIR 1998/9- [sesquiannual]• CLEF (Cross-Language Evaluation Forum) 2000• INEX (Initiative for the evaluation of XML
retrieval ) 2002• TRECVID 2003• FIRE (Forum for IR Evaluation ) 2008• MediaEval (Benchmarking Initiative forMultimedia Evaluation) 2010[not exhaustive]
2013-09-02 ntcir-11 kick-off

14

Difficulty of retrieval varies with topics
1.0000J-J Le ve l1 D auto
EffectivenessacrossSYSTEMS

Effectiveness across TOPICS on
system 検索システム別の11pt再現率精度
10.8000

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

A

Average over 50
topicsB
CD

0.8

0.6000precision

E

precision

F

0.4000

G

0.6

H
I
J

0.4

K
L

0.2000

M
N

0.2

O

0.0000
0

2

4

6

8

0

0.

0.

0.

0.

0.

1.

P

recall
2013-09-02 ntcir-11 kick-off

0
0

0.1

0.2

0.3

0.4

0.5
recall

0.6

0.7

0.8

0.9

1

15

Difficulty of retrieval varies with topics
J-J Le ve l1 D auto

1.0000Effectiveness across
SYSTEMS

検索システム別の11pt再現率精度
Effectivenessacross TOPICS on
system10.8000“Difficult
topics”
vary with systems
Averageover 50
A
B

topicsJ-J Leve l1 D auto
0.8

0.60000.80000.4000

D

B

E

C
D

G

0.6

E

H

0.60000.4

0.40000.2000

I

F

J

G

K

H

L

I

M

J

N

0.2

0.2000K

O

0.0000LP

7
1 10 .2011
3
0
1 1 .4611
9
1 20 .
26
12
5
1 20
8.8
13
1
1 31
4.0
13
7
14
0
14
3
14
6
14
9

0

0

10

1

0
1 .

04 0

0.000010

precision

A

F

precision

Mean av. precision precision

1.0000C

recallT opic#Requests#101-150

2013-09-02 ntcir-11 kick-off

0.1

0.2

0.3

For reliable
and stableMevaluation,using substantial #
N 0.6
0.4
0.5
0.7
0.8
0.9
1
of recall
topicsisnecessary.
O
P

16

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

IR Systems Evaluation• Engineering Level: Efficiency• Input Level: ex. Exhaustivity, quality, novelty of DB• Process Level: Effectiveness ex. recall, precision• Output Level: Display of output• User Level: ex. Effort that users need
• Social Level: ex. Importance2013-09-02 ntcir-11 kick-off(Cleverdon & Keen 1966)

17

What are TCs usable for evaluating?

Pharmeceutical R & D
Phase I:
In vitro
experiments

2013-09-02 ntcir-11 kick-off

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

18

What are TCs usable for evaluating?
NTCIR

Users’ information-seeking
tasks

Test collections
Phase I:

Phase II:

Phase III:

Phase IV:

Laboratory-type
testing

Sharing modules,
Prototype
testing

Controlledinteractive testing
using humansubjects

Uncontrolled preoperational testing

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

Pharmeceutical R & D
Phase I:
In vitro
experiments

2.Input level、Levels ofevaluation
1. Engineering level:
3.Processlevel:kick-off
Effectiveness2013-09-02 ntcir-11

6.Social level

4.User level、5.Output level

Efficiency
19• Testing & Bench
marking• Information
Seeking Task

– To learn how andwhy the system
works better (worse)
than others– To learn how it canbe improved– Scientific
Understanding of
the effectiveness– document types +user community– user’s situation,
purpose of search,
realistic

Experiments are
Abstraction of theReal World Tasks.
Trade-off between“reality” and“controlable”
2013-09-02 ntcir-11 kick-off

20

Improvement of Effectiveness by Evaluation Workshops
1.5 – 2 times in 3
years

Mean Average Precision

0.5

Cornell University TREC Systems

0.4
0.3

TREC-1
TREC-2
TREC-3

0.2

TREC-4

0.1

TREC-60

TREC-5
TREC-7

'92
'93
'94
'95
'96
'97
'98
System System System System System System System
2013-09-02 ntcir-11 kick-off

21

Research Trends
Number of Papers Presented at ACM-SIGIR
450

400

WEB
350

User
Evaluation# of papers

300

Non-Text
QA & Summarization

250

NLP
Cross-Lingual

200

ML
Clustering

150

Efficiency
Filtering

100

Query Processign
IR Models

50

General
0'77-79'80-84'85-89'90-94'95-99'00-04'05-09PpublicationYears
2013-09-02 ntcir-11 kick-off

22

Research Trends
Number of Papers Presented at ACM-SIGIR
450

400

Many newtechnologies
were started andboosted thro
Evaluation Forum350# of papers

300

WEB
User
Evaluation
Non-Text
QA & Summarization

250

NLP
Cross-Lingual

200

ML
Clustering

150

Efficiency
Filtering

100

Query Processign
IR Models

50

General
0'77-79'80-84'85-89'90-94'95-99'00-04'05-09PpublicationYears
2013-09-02 ntcir-11 kick-off

23

Some Thoughts on Future• Propose and lead a new community!
• How to Evaluate Individual Applications and new technologies– Think and work in the community!

• Interactive and Exploratory Information Access– Users’ Intention, Diversity– Collaborative Search– Task Expertise, Knowledge, Literacy, etc.

• Ex. Key Topics discussed at SWIRL 2012
– Conversasional Retrieval– MobileIR– Zero Query (or Less)– SmarteIR– Query by walking around– Structure, Entity, Relation• Time-aware• Whole Session Evaluation2013-09-02 ntcir-11 kick-off

24About NTCIR-11
What’s new?

25

Recent NTCIRs Objectives• Solid foundation– Even stronger
structure than before• Task diversity– Covers a wide context
in Information Access– Studies rich media
types• Community-led taskorganisation– Sustainability of
research• Promotion of researchresources– Show case in the
NTCIR Conferences
26

NTCIR-11: Structure• General Co-Chairs– Noriko Kando (NII)– Tsuneaki Kato (The Univ. ofTokyo)– Douglas W. Oard (Univ. of
Maryland)• Program Co-Chairs– Hideo Joho (Univ. of Tsukuba)– Kazuaki Kishida (Keio Univ.)• Task Organisers– Tetsuya Sakai (Waseda Univ.)– Leading researchers
worldwide– Mark Sanderson (RMIT Univ.)– Participants (You!)TO• EVIA 2014 Co-Chairs– Ruihua Song (MSRA)
– TBA
27

NTCIR-11: Development so far
Jan - Feb
2013

Development for NTCIR-11 organisation started

March May 2013
Jun 2013
Jun 2013

Call for task proposal announced and
11 proposals were submitted
NTCIR-10 Conference
8 evaluation tasks were accepted by the programcommittee and co-chairs
Calls for task participation prepared
NTCIR-11 Kick-Off Event

Aug 2013
Sep 2013

28

NTCIR-11 Program Committee•

Hsin-Hsi Chen (National Taiwan University, Taiwan)•

Charles Clarke (University of Waterloo, Canada)•

Kalervo Järvelin (University of Tampere, Finland)•

Hideo Joho (Co-chair, University of Tsukuba, Japan)•

Gareth Jones (Dublin City University, Ireland)•

Noriko Kando (NII, Japan)•

Tsuneaki Kato (The University of Tokyo, Japan)•

Kazuaki Kishida (Co-chair, Keio University , Japan)•

Gary Geunbae Lee (POSTECH, South Korea)•

Douglas W. Oard (University of Maryland, USA)•

Maarten de Rijke (University of Amsterdam, The Netherlands)•

Stephen Robertson (Microsoft Research Cambridge, UK)•

Tetsuya Sakai (Microsoft Research Asia, PRC)•

Mark Sanderson (RMIT University, Australia)•

Ian Soboroff (NIST, USA)
29

NTCIR-11 Evaluation Tasks
Calls for task participation

30

Tasks accepted for NTCIR-11
CORE TASKS• [IMine] Search Task and Intent Mining• [Math-2] Mathematical Information Access• [MedNLP-2] Medical Natural Language Processing• [MobileClick] Mobile Information Access• [RITE-VAL] Recognizing Inference in Texts and Validation• [SpokenQuery&Doc] Spoken Query and Spoken Document
Retrieval
PILOT TASKS• [QALab] QA Lab for Entrance Exam
• [Temporalia] Temporal Information Access31

IMine32

The	
  NTCIR-­‐11	
  IMine	  Task	  Kickoﬀ	
  Mee4ng
Yiqun	
  Liu,	
  Ruihua	
  Song,	
  Min	  Zhang,	
  Zhicheng	
  Dou,	
  	
  Takehiro	
  Yamamoto,	
  Makoto	
  P.	
  Kato,	  Hiroaki	
  Ohshima,	
  	
  
Ke	
  Zhou	  http://www.thuir.org/IMine/
September,	  2013@NII,	
  Tokyo	
  Diversiﬁed	
  search

SERP	
  (Search	
  Engine	
  Result	
  Page)	
  • Given	
  an	
  ambiguous/underspeciﬁed	
  query,	
  produce	
  a	
  single	  Search	
  Engine	
  Result	
  Page	  that	  sa4sﬁes	
  diﬀerent	
  user	
  intents!	
  • Challenge:	
  balancing	
  relevance	
  and	  diversity
Highly	  relevant	  near	
  the	
  top	  Cover	  many	  intents	  Give	
  more	  space	  to	
  popular	
  intents?	
  Give	
  more	  space	  to	
  informa4onal	
  intents?	
  IMineUnderstanding	
  user	  intents	
  in	
  Web	
  search	

Intent	  Mining	曖昧	
  （ambiguous） 	

The	
  IMine	
  taskThree	
  subtasks	
• Subtopic	
  Mining	  (SM):	
  Chinese,	
  English,	  Japanese	  INPUT	
  	
  	
  	
  :	
  query	
  (e.g.	
  “harry	
  poYer”)	
  OUTPUT:	
  ranked	
  list	  of	
  subtopic	
  string	
  (e.g.	
  “harry	
  poYer	
  book,	  harry	
  poYer	
  ﬁlm,	
  harry	
  poYer	  the	  character…”)	
  • Document	
  Ranking	
  (DR):	
  Chinese,	
  English	  INPUT:	
  query	
  (e.g.	
  “harry	
  poYer”)	
  OUTPUT:	
  diversiﬁed	
  ranked	
  list	  of	
  web	
  pages	  • Search	
  Task	  Mining	
  (TASKMINE):	
  Japanese	  INPUT	  :	
  query	
  (e.g.	
  “pollen	
  allergy	
  treatment	  ”)	
  OUTPUT:	
  ranked	
  list	  of	  task	  string	
  which	  sa4sﬁes	
  the	
  given	
  query	
  （e.g:	
  “Laser	
  surgery”，”an4allergic	
  drug”，”allergy	  mask”）	
  	
  More	
  User	  Behavior	
  Data• THUIR@INTENT/INTENT2:	
  More	  user	  behavior	
  data	  led	  to	
  beYer	
  performance.	
  • More	  raw	
  click-­‐through	  data:	
  SogouQ	
  has	
  doubled	
  its	
  
size	
  to	
  include	
  click-­‐through	  data	  collected	
  from	  Sogou.com	
  in	
  2011	
  • 1.85GB	
  =>	
  3.85GB,	
  over	  40M	
  user	  clicks	
  • More	  subtopic	
  candidates	
  generated	
  from	
  	
  
more	
  recent	
  user	  behavior	
  data.	
  • Search	
  engine	
  data	  provided	
  by	
  two	  major	  	
  Chinese	
  search	
  engines	
  will	  be	
  adopted	
  	
  • Find	  out	
  whether	
  more	  logs	  help	
  improve	  	
  SM/DR	  performance

Intent	
  Annota4on	
  Using	
  Logs• SM	  results	
  in	
  pools	
  will	  be	
  clustered	
  with	  click-­‐
through/pseudo	
  RF	
  data	  at	
  ﬁrst	  to	
  generate	
  preliminary	
  candidate	
  intent	
  groups.	
  	
  • Query	
  frequency	
  informa4on	
  will	  be	
  taken	
  	
  
into	
  considera4on	
  during	
  subtopic	  	
  importance	
  vo4ng	
  process.	
  • Data	  source:	
  recently	
  collected	
  data	
  	
  
from	
  Sogou	
  for	
  Chinese	
  SM,	
  Bing	  for	  	
  English/Japanese	  SM	
  • More	  credible	
  SM	
  qrels	
  with	  less	  annota4on	
  eﬀorts	  • Perhaps	
  the	
  reusability	
  of	
  results	
  could	
  be	
  increased

Crowd-­‐Sourcing	
  based	
  Evalua4on• A	
  search-­‐engine-­‐like	
  annota4on	
  interface	
  to	
  collect	  feedback	
  informa4on	
  from	  a	
  rela4vely	
  large	
  number	  of	
  unprofessional	
  users	
  (e.g.	
  50+	
  undergraduate	  students)	
  • Data	  collected	
  from	  the	
  interface:	
  query,	
  click,	
  examina4on	
  • High	  correla4on	
  with	  examina4on	
  • Preliminary	  results	
  on	
  10	
  users.	
  	
  • KAPPA:	
  0.65,	
  Accuracy:	
  0.83.	
  • Find	  out	
  whether	
  D#	
  measures	  accords	
  	
  
with	
  user	  sa4sfac4on

INTENT	
  with/for	
  Knowledge	
  Graph• Fuji	  in	
  Wikipedia:	
  • Fuji	  the	
  actress	  • Fuji	  the	
  Mountain	  • Fuji	  the	
  apple	  • Ambiguous	
  queries:	
  NTCIR	  INTENT	
  #0205:	
  功夫(kung	
  fu)，#0206:	
  ⽣生⽇日快樂樂樂樂(happy	
  birthday)	
  • Mining	
  and	  evalua4ng	
  hierarchical	
  user	  intents	
  • SM	
  task:	
  return	
  a	
  two-­‐level	  hierarchical	
  list	  of	
  subtopics	  • First	  level:	
  at	
  most	  10	
  major	
  categories	
  of	
  user	  intent	  (“Microsoq	
  windows”	
  for	
  “windows”)	
  • Second	  level:	
  other	  minor	
  subtopics	
  under	  the	
  categories	
  (“windows	
  update”,	
  “windows	
  8.1	
  installa4on”)	
  Task	
  Mining	
  (TASKMINE)	
  subtask
Find	
  tasks	
  that	  saDsﬁes	
  the	
  given	
  query	
OUTPUT	

query	

move	
  to	
  a	
  
new	
  house	Stop	
  gas	Stop	
  electricity	
Submit	  document	
  of	
  
move	
  out	

Submit	  document	
  of	
  
move	
  in	

	
  	
  Organisers:	  	
  Takehiro	
  Yamamoto,	
  Makoto	  Kato,	  Hiroaki	
  Ohshima
http://www.dl.kuis.kyoto-u.ac.jp/ntcir-11/taskmine/Summary	
  of	
  IMineThank you
http://www.thuir.org/IMine/
http://www.dl.kuis.kyotou.ac.jp/ntcir-11/taskmine/Math-2

33

NTCIR-11
Math-2
マスタ タイトルの書式設定
What is Math Search?
Goal of NTCIR-Math
Task outline
Technical challenges
Contact information

September 2, 2013

NTCIR-11 Math-2

1

What
is
Math
Search?
マスタ タイトルの書式設定

NTCIR Math-1
Topic FT-6

September 2, 2013

NTCIR-11 Math-2

2

What
is
"math"?
マスタ タイトルの書式設定

Structured data withcomplex XML representation
called MathML

September 2, 2013

NTCIR-11 Math-2

3

The
Goal
of
NTCIR
Math
マスタ タイトルの書式設定
Exploring methods for mathematical content
accessTwo major goals of Math-2
Achieving reusable test collection
Establishing and supporting the Math IR
community (mathematicians and IR&NLP
researchers)Task
Outline:Dataset
マスタ タイトルの書式設定
Given a document collection, retrieve
relevant mathematical formulae for a given
query.
Dataset: Reuse and adapt NTCIR-10 dataset
100,000 papers from ArXiv.org
35,000,000 formulae
Converted into XML+MathML by arXMLiv project
Retrieval unit: minimal subsections of ArXivdocuments
September 2, 2013

NTCIR-11 Math-2

5

Task
Outline:Assessment
マスタ タイトルの書式設定
Pooling and assessment
50 Topics
Multiple assessment (two) for inter agreement
check
Pooling size: 100
Include “Relevant, Partially relevant, Nonrelevant, Can not be assessed”

September 2, 2013

NTCIR-11 Math-2

6

Task
Outline:Topics
マスタ タイトルの書式設定
Topic development
Topic structure• Topic ID• Query (formula + key words)• Description (short description of what a user is lookingfor)• Narrative (precise description of the user situationand information need and relevancy criteria)

All topics should include multiple relevant
documents
September 2, 2013

NTCIR-11 Math-2

7

Task
Outline:Runs
マスタ タイトルの書式設定
Runs
Submit compulsory automatic runs using query
only field
Encourage participants to submit manual runs
(with manually generated queries)
Results will include supporting evidence(formulaID, sentenceID,etc.), optional

September 2, 2013

NTCIR-11 Math-2

8

Technical
Challenges
マスタ タイトルの書式設定
Mathematical knowledge management
Digital math library
Large scale XML tree search (with variables)
Semantic search based on deep language
analysis
Domain specific information retrieval

September 2, 2013

NTCIR-11 Math-2

9

Contact
Information
マスタ タイトルの書式設定
E-mailorganizers ML: ntcadm-math@nii.ac.jp
community ML: ntcir-math@nii.ac.jp

Task Web page
http://ntcir-math.nii.ac.jp/

Community Site
http://ntcir.mathweb.org/

Task Organizers
Akiko Aizawa (National Institute of Informatics, Japan)
Michael Kohlhase (Jacobs University Bremen)
Iadh Ounis (University of Glasgow)

Tentative
Schedule
マスタ タイトルの書式設定
September, 2, 2013

NTCIR-11 kick-off meeting

November, 2013

Call for participation

February, 2014

Initial dataset and example topics release

May, 2014

Topic release

June, 2014

Result submission deadline

December, 2014

NTCIR-11

September 2, 2013

NTCIR-11 Math-2

11

MedNLP-2

34

MedNLP Task
at NTCIR-11 Kick-off
Eiji Aramaki
Yoshinobu Kano
Tomoko Ohkuma
Mizuki Morita

Kyoto University / JST PRESTO
JST PRESTO
Fuji Xerox
The University of Tokyo

Current Medical NLP Situation• Researchers– have methods– but no data

• Companies– have solution & money– but no data

• Only Hospitals have data
– want to use it

MedNLP Task could create a win-win situation

What kind of NLP is required?
Texts
De-identification
NER
Normalization
Mining

京大病院にてアクトス投与もむくみ出現
Named Entity (NE) removing for privacy protecting■■病院にてアクトス投与もむくみ出現
Patient information extraction■■病院にてアクトス投与もむくみ出現

MedNLP-1■■病院にてビオグリタゾン投与も浮腫出現What was MedNLP-1?
I think it was very successful• 12 teams joined to MedNLP-1• We will have an invited talk at
CLEFeHealth 2013

What’s Next?
Texts
De-identification
NER
Normalization
Mining

京大病院にてアクトス投与もむくみ出現■■病院にてアクトス投与もむくみ出現■■病院にてアクトス投与もむくみ出現■■病院にてビオグリタゾン投与も浮腫出現

MedNLP-2

Marchesani Syndrome

マルケサーニ症候群
マルケサニー症候群
マルケサニ症候群
マルケザーニ症候群
マルケザニー症候群
マルケザニ症候群

ICD 10 =
Q871

悪心増悪
胃のむかつき悪心 (sickness)
食後悪心
嘔気 (vomiting)
吐き気
悪阻 (nausea)

ICD 10 = R16
悪心及び嘔吐(sickness & vomiting)

MedDRA/JCorpus Example

Thank you!

Mizuki Morita
Yoshinobu Kano
Tomoko Ohkuma
Mai Miyabe
Eiji Aramaki

The University of Tokyo
JST PRESTO
Fuji Xerox
Kyoto University
Kyoto University / JST PRESTO

Annotated Example<Task1> & <Task2>

MobileClick

35

NTCIR-11 MobileClick Task
Makoto P. Kato
Matthew Ekstrand-Abueg
Virgil Pavlu
Tetsuya Sakai
Takehiro Yamamoto
Mayu IwataSuppose that …
 You are finding answers for a question“what’s the difference between Organic EL and
LCD?” in an electronics storeIR Systems in Ten-Blue-Link Paradigm
Enter query

Click SEARCH button
Scan ranked list of URLs

Click URL
Read URL contents

Get all desired information

Long way to get all desired information

MobileClick System
Enter query

The system outputs X-string

Click SEARCH button
OLED LCD difference
LCD is better in terms of the weight, sizeand energy saving.
OLED shows a better black color, a faster
response speed, and a wider view angle.

Advantage of OLEDAdvantage of LCD

Get all desired information

Phone: 046-2233636. Fax: 046223-3630. Address:
118-1 Nurumizu,
Atsugi, 243-8551.
Email:
soumu@shonanatsugi.jp. Visiting
hours: general ward
Mon-Fri 15-20;
Sat&Holidays 13-20/ Intensive Care Unit
(ICU) 11-11:30,
15:30, 19-19:30.

Phone: 046-2233636. Fax: 046223-3630. Address:
118-1 Nurumizu,
Atsugi, 243-8551.
Email:
soumu@shonanatsugi.jp. Visiting
hours: general ward
Mon-Fri 15-20;
Sat&Holidays 13-20/ Intensive Care Unit
(ICU) 11-11:30,
15:30, 19-19:30.

Task: Given a search query,
return a structured textual output

Go beyond the "ten-blue-link" paradigm, and tackle
information retrieval rather than document retrieval

MobileClick=

Immediate & Direct
Mobile Information AccessWhat’s New in MobileClick?
 A New Challenge: Two-layered Summarization

 Two Subtasks for Easy Participations
 iUnit Retrieval Subtask (Information Extraction)
 iUnit Summarization Subtask (Summarization)

iUnit Retrieval Subtask
 Generate a list of iUnits ranked according totheir importance for a given query iUnits: information pieces relevant to a given query

Input: Query
OLED LCD difference

Output: List of iUnits
Rank

iUnit

1

LCD is lighter than OLED

2

OLED shows a better black color

3

OLED has a wider view angle…

…

iUnit Summarization Subtask
 For a given query (and a list of iUnits),

generate a two-layered textual output
Input: Query
OLED LCD difference

Input: List of iUnits
Rank

iUnit

1

LCD is lighter than OLED

2

OLED shows a better black color

3

OLED has a wider view angle…

…

Output:Two-layered textual output
LCD is better in terms of the weight, size and energy
saving, while OLED shows a better black color, afaster response speed, and a wider view angle.

Advantage of OLEDAdvantage of LCD
OLED shows a better blackcolor, a faster response speed,
and a wider view angle.

LCD is better in terms of theweight, size and energy saving.

Who Should Participate in MobileClick?
 Researchers who are interested in: Information ExtractionQuery-driven information extraction and ranking

 Passage Retrieval


Finer-grained information pieces retrieval

 Question AnsweringIntent estimation and answer finding Summarization


Generating structured summaries

 Teams that participated in: Previous NTCIR 1CLICK tasks


MobileClick is an extension of the 1CLICK task

 Previous NTCIR INTENT tasks

Two-layered output may be suitable for handling ambiguous and faceted
queriesTentative Schedule

Aug 31, 13First CFP, and Web page launch

Oct 31, 13

Sample queries and iUnits released

Mar 31, 14

Test queries released

Apr 30, 14

Run submissions due

Aug 15, 14

Evaluation results released

Dec 9-12, 2014 NTCIR-11

Summary MobileClick Task (inherits 1CLICK Task)
 iUnit Retrieval Subtask
 iUnit Summarization Subtask

 New challenges
 Ranking information pieces
 Two-layered summarization

 MobileClick Homepage

http://www.dl.kuis.kyoto-u.ac.jp/ntcir-11/mobileclick/

RITE-VAL

36

Introduction to NTCIR-11 RITE-VAL Task
(Recognizing Inference in Text and Validation)

SuguruMatsuyoshi1

YotaroWatanabe2

YusukeMiyao3

Tomohide
Shibata4

1University

2Tohoku

3National Institute4Kyotoof Yamanashi

Universityof Informatics

University

TerukoMitamura5

ChuanJie Lin6

ChengWei Shih7

5Carnegie Mellon

6National Taiwan

7Academia

University

Ocean University

SinicaNTCIR-11 Kick-Off Event September 2nd, 2013

Overview of RITE-VAL• RITE is a benchmark task for automatically detecting
the following semantic relations between two
sentences:– entailment, paraphrase and contradiction.

• Given a text t1, can a computer infer that a hypothesis
t2 is most likely true (i.e., t1 entails t2) ?
– t1: Yasunari Kawabata won the Nobel Prize in Literaturefor his novel “Snow Country.”– t2: Yasunari Kawabata is the writer of “Snow Country.”• Target languages:– Japanese, Simplified Chinese, Traditional Chinese, andEnglish.

RITE-VAL Subtasks
Other
NTCIR tasks
QA Lab

Applications
QA
IR

Fact
Validation
Multiplesentencelevel
inference
Pyramid ofentailment
recognition
technology

Sentence-level
inference

sentence
sentenceentailment ?

BCparaphrase?
entailment ?
contradiction?

MCnegation…

modification

coordination

quantificationcase alternation

Unit
Test

entailment ?
contradiction?

documents

phrase rel.

Linguistic
phenomenalevel inferenceWorldknowledge

Search≡≡≡ ≡≡lexical rel.

SystemValidation

sentence

Application oriented

sent.

sent. sent. sent. sent. sent.

sent.

sent.

sent.

sent. sent. sent. sent. sent.

sent.

sent.

RITE1 RITE2 RITE-VAL

Foundation oriented

Main two tasks of RITE-VAL
Fact
Validation

t2: The Kamakura Shogunate began

in Japan in the 12th century.

sentence

Search≡≡≡ ≡≡documents

SystemValidation

Search for evidence or
counter-evidence for t2.
Docs contradict t2.

t1: In the Meiji Constitution, legal clear distinction

Category:
modification

sent. sent. sent. sent. sent.

sent.

sent. sent. sent. sent. sent.

t2: In the Meiji Constitution, distinction between the

…

sent.

between the Imperial Family and Japan had been
allowed.
Imperial Family and Japan had been allowed.

negation

modification

coordination

quantificationcase alternationphrase rel.

Unit
Test

lexical rel.

Worldknowledge

entailment ?
contradiction?

Docs entail t2.

t1: In the Meiji Constitution, distinction between theImperial Family and Japan had been allowed.
sent.

sent.

Category:
sent. sent.
meronymy

t2: In the Meiji Constitution, distinction between theEmperor and Japan had been allowed.

Why you should participate• Products from RITE-VAL can benefit various research fields
in NLP (not limited to recognizing textual entailment!).
– Core technologies: Semantic processing, Lexical knowledge
acquisition, Machine learning, etc.
– Applications: Information retrieval, Question answering,Document summarization, etc.

• We welcome a wide variety of participants from all over theworld.
– From undergraduate students to industry researchers.
– Resource pool, which we have built since NTCIR-10 RITE-2, will
help you make a prototype system quickly.
– Adding a new resource or tool to the pool is welcomed.

• Website: https://sites.google.com/site/ntcir11riteval/

SpokenQuery&Doc

37

NTCIR-­‐11	
  Core	  Task:	  Spoken	  Query	
  and	
  Spoken	
  Document	  Retrieval	  (SpokenQuery&Doc)	
Tomoyosi	  Akiba	  (Toyohashi	
  University	
  of	
  Technology)	
  Hiromitsu	
  Nishizaki	
  (Yamanashi	
  University)	
  Hiroaki	  Nanjo	
  (Ryukoku	
  University)	
  Gareth	  Jones	  (Dublin	
  City	  University)	

Focus	  of	
  SpokenQuery&Doc	
• InformaNon	
  Retrieval	
  by	
  using	  “spontaneously	
  spoken	  query”	
  targeNng	
  “spoken	
  documents”.	
  Mt.	
  Fuji	Search	  Engine	Mt.	
  Fuji	
  is	
  
the	
  highest	
  mountain	  in	
  Japan.	
  ..	

Overcome	
  it	
  by	
  using	  Speech!	
I'm	
  going	
  to	
  go	
  climbing	
  Mt.	
  Fuji	  in	
  next	  summer	  vacaNon.	
  Well,	
  I	
  
hear	
  the	
  accident	
  um...	
  I	
  hear	  the	  news	  about	  the	
  accident	
  in	
  the	
  climbing.	
  Well,	
  someone	  has	
  missed	
  or	
  someone	  lost	  one's	
  way	
  in	
  
the	
  mountain.	
  So	
  I	
  would	
  like	  to	
  know	
  what	  should	
  I	
  prepare	
  for	
  climbing,	
  what	  equipment,	
  equipment	
  is	
  necessary.	
  I	
  would	
  like	
  to	
  
know	
  what	  should	
  I	
  prepare	
  when	  I	
  go	
  climbing.	

Example	  of	
  Spontaneously	
  Spoken	
  Query	
今年の夏休み(summer	
  vacaNon)に始めて山(mountain)に登山に行くことになった
んですけれども。あのー登山は結構事故(accident-­‐1)があのーやはり夏(summer)になるとよくニュース(new)で聞きますし，ま誰々が行方不明(go	
  missing)になったとか
遭難(accident-­‐2)したとかそういう話が色々あると思うんですけれども。あのー山(mountain)に登る(climb)時にはどういった心構え(keep	
  in	
  mind)と言うか，あのー
装備(equipment)こういう装備が必要だとかこういうものがあるといいよ，とかそういった
あのー山登り(mountain	
  climbing)に関しての，山(mountain)に登る(climb)時についての心構え(keep	
  in	
  mind)について知りたいです		• PROS:	
  provides	
  rich	  clues	
  for	
  retrieval.	
  • CONS:	
  unclear,	
  ambiguous,	
  redundant,	
  error-­‐prone	  	
  Challenges	
  in	
  SpokenQuery&Doc	
• How	  to	
  get	
  beneﬁt	
  from	  spontaneously	
  spoken	
  queries?	
  – Make	  use	  of	
  long	  queries.	
  – DisambiguaNon.	
  – Improve	
  ASR	
  for	
  spontaneous	
  speech.	
  – Development	
  of	
  speciﬁc	
  IR	
  techniques.	
  • Spoken	
  document	
  retrieval	
  (conNnued)	
  – Evaluate	
  spoken	
  term	  detecNon	
  (STD)	
  and	
  spoken	
  content	
  retrieval	
  (SCR)	
  at	
  the	
  same	  Nme	  by	
  using	  common	
  (spoken)	
  query	
  topics.	
  – Text	  queries	
  are	
  also	  provided	
  (as	  in	
  SpokenDoc-­‐1	
  and	
  2.)	  

Spoken	
  query	
  (English)	

Spoken	  query	
  (Japanese)	

Task	
  Design	

SQ-­‐SCR	

Spoken	  Content	  Retrieval	Relevant	
  Segments	  (presentaNon	
  slides)	
  Spoken	  Documents	  (presentaNons	
  at	
  academic	
  meeNng)	

Spoken	  Content	  Retrieval	

Spoken	  Term	  DetecNon	

DetecNon	  Results	

Spoken	  Terms	

SQ-­‐STD	

STD-­‐SCR	Why	  parNcipate	
  in	
  SpokenQuery&Doc	
• ParNcipants	
  will	  be	
  provided	  many	  resources	
  for	  IR	  and	  spoken	
  language	
  processing	  research.	
  – Real	  recordings	
  of	
  spontaneously	
  spoken	
  queries.	
  – Real	  recordings	
  of	
  academic	
  meeNng	
  lectures.	
  – Their	
  rich	  transcripNons.	
  • manual.	
  • automaNc	
  (with	  various	
  condiNons.)	
  • alignment	
  with	  presentaNon	
  slides.	
  

• IR	
  and	
  NLP	  researchers	
  are	
  also	  welcomed.	
  – Easy	  to	
  apply	
  your	  exisNng	
  techniques	
  to	
  provided	
  text	  transcripNons	
  of	
  speech.	
  • Web	
  site	  will	  be	
  open	
  soon.	
  – hap://www.nlp.cs.tut.ac.jp/ntcir11/	

QA Lab38

Proposal for
NTCIR-11 Pilot task

QA Lab
Madoka Ishioroshi (NII)
Kelly Itakura （U Waterloo)
Noriko Kando (NII)
Yoshinobu Kano (PRESTO/NII)
Teruko Mitamura (CMU/NII)
Eric Nyberg (CMU)
Hideyuki Shibuki (Yokohama National University)
Advisor: Tatsunori Mori (Yokohama National University)
NTCIR-11 Kickoff Event
Sept. 2, 2013
Thanks Madoka for slides

Main Idea
• Challenge for Real World Problem of QA– University Entrance Exams - New Q-types not covered byexisting QA systems. Q is not a single sentence. Need
context (ntcir-11), inference (ntcir-12).

• Challenge for a new structure of evaluation – Modulebased, building workable system as a joint effort of all
the participants– QA is very complicated– Hard for a research group to be good on every
components– Your module used and rated by other people• Continuous Plan-Run-Analysis-Improve Cycle

Background-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」[factoid] People, GeoName,
Organization, Numeric value, etc.
Document Retrieval[non-factoid] Definition, Reasons,Methods, etc.

Extracting Answer
Candidates- Extracting focus from
Question- Analyzing what is asked(QuestionType)
Inf used forscoring:・Verify the Q with the sentence
that
therelated
answerdocuments
candidate was
Search(orextracted,
Textualpassage) using
theEntailment
extracted
・#of KWsaroud theanswerfocus
of questions
andQ typecandidate・Verify by the Q type and theExtracting
Answer Candidates
answercandidatefromthe retrieved・Context
where thedocuments
answerpassagescandidateorwas
extracted from

Answers「634 Meters」

Answer Generation

Ranking the answer candidates
based on the scores of “answer
likeliness”

Evaluation Method-1

Systems• QA-Platform– UIMA-based Module-base QA system– Follows: ACLIA modules• Q analysis• IR• Answer Extraction• Answer Generation– 2 baseline systems for Japanese• Javelin (CMU)• MinerVA (YNU)– 1 baseline systems for English (CMU)– Systems are usable either• local (by Kachaco) or on NII Server “bunbun”

UIMA Native UI for QA Platform to run a pipeline

Upload the Q fileSet various parameters to run

Specifies the index file for Indri (RS)Output directory

Run

Document directoryUI for the QA Platform to examine the results of each
moduleResults of each
moduleShow the representative results
for the first. To see moredetailed results by click on thebottun of “Expanding”

Click tojump tothemoduleAnswerCandidates

Evaluation Method-2• Questions & Answers

Corpus– Japanese National Center for University Admissions Tests (multiple
choices)•  use as a Yes-No / TF Questions• XML, Japanese and English translation– Second Exam for the U of Tokyo and some others•  factoid & Complex Questions• Will be XML tagged• Knowledge sources provided– High school Textbooks, Wikipedia– Ontology of World History– Ontology-annotated Textbook• Tools– Wide-range of Language annotation tools– RITE resource and tools

Evaluation Method-3

Tasks
Multiple Choice (Center Exam)Japanese

EnglishNTCIR-11

World Hitory

NTCIR-12

World Hitory

English

2nd Exam

JapaneseJapanese

Factoid

Biology, Politics& Economics

Factoid

2nd Exam

English?

English?

Factoid

Complex

Factoid

ComplexEvaluate QA end-to-end with Every Possiblecombination of the modules submitted so far
and base systems

Tentative Schedule:

Multiple Opportunities to Run
Jan-14Feb-14Mar-13Apr-14May-14Jun-13

Jul-14Aug-14

Sep-14runs
1st Roundreturnresults
R-tableruns

2nd Roundreturnresults
R-tableruns

3rd Roundreturnresults
R-tableSubmitted all rounds are not mandatory.
* R-table = Round-table meetingParticipant’s merits• Can participate in one phase or multiple
phases (flexible participation)• Can evaluate with Baseline or with othersystems and to improve your system for the
next phase• Can see the effectiveness of your modules invarious pipelines created by other teams or
combination of other teams.

Join Us!

Temporalia

39Temporal Information Access(Temporalia)
NTCIR-11 Pilot Task
https://sites.google.com/site/ntcirtemporalia

Hideo Joho

Adam Jatowt

Roi Blanco(Univ. of Tsukuba)(Kyoto University)(Yahoo! Research)

Motivation

Proposed Subtasks in NTCIR-11• Temporal Query Intent Classification (TQIC)– For a given query, participants classify it todifferent temporal classes such as atemporal,
recency, past, future, etc.

• Temporal Information Retrieval (TIR)– For a given topic, participants retrieve relevant
documents for different kinds of temporal queries
Extension to temporal summarization, visualization, and other applications will beconsidered in NTCIR-12.

Subtasks
Temporal Query Intent
Classification (TQIC)
Query

Temporal

Past

Recency

Non-Temporal

FutureTemporal Information
Retrieval (TIR)
Query

1.
2.
3.
4.
5.

DocA
DocB
DocC
…
..

Non-Recency
N. DocNthematic +temporalrelevance
rankingExample Queries
Non-Temporal

Recency

how to tie a necktie

dollar yen rate

pitgoras theorem

san francisco weather

facebookcalgary flood news

Past
dollar yen rate

Temporal

top comedies 2000s
calgary flood news

Non-Recency

watergate story

dollar yen ratewatergate story

olympics 2016

top comedies 2000s
why hitler did not attack switzerland

Futuregoogle glass release date
olympics 2016

Queries will possibly be collectedfrom real search query logs

opening of new train lineSubTasks Data
• Temporal Query Intent Classification (TQIC)– query strings (100 for training/dry run, 100 forformal run)– query issuing times• Temporal Information Retrieval (TIR)– 50 topic descriptions (title, description, narrative,
query issuing time)– qrels (relevant document list)

Document Collection• LivingKnowledge Corpus (150GB compressed)– Crawl time: April 2011 and March 2013
– Source: Annotated News Feeds– Size: 20G uncompressed (>5G zipped)• 3.8M documents from 1500 different blogs and news
sources• Provided annotations: Time Annotations, Named
Entities and Sentence Splitting
ClueWeb Corpora: ClueWeb09 (5TB, Jan-Feb 2009), ClueWeb12 (5.5 TB, Feb- May2012) will be considered in NTCIR-12.

Example Document

More details at the Task’s website: https://sites.google.com/site/ntcirtemporalia

Performance Metrics• Thematic relevance– Graded relevance (highly relevant, relevant, partiallyrelevant and not relevant)• Temporal relevance• Some function to integrate them, similarly as in:– GeoCLEF (Spatial relevance + Thematic relevance)– INEX (Structural relevance + Thematic relevance)Draft ScheduleThank you!
tc4fia@googlegroups.com

References• H. Joho, A. Jatowt, and R. Blanco. A Survey of Temporal Web SearchExperience. TempWeb 2013 Workshop @ WWW 2013
• A. Jatowt, C. M. Au Yeung and K. Tanaka. Estimating Document Focus Time.
CIKM 2013• R. Blanco, H. Halpin, D. M. Herzig, P. Mika, J. Pound, H. S. Thompson, D. T.
Tran: Repeatable and reliable search system evaluation using crowdsourcing.
SIGIR 2011
• M. Matthews, P. Tolchinsky, R. Blanco, J. Atserias, P. Mika, and H. Zaragoza.
Searching Through Time in the New York Times, HCIR 2010

NTCIR-11 Task Map
Summary

40

Context of Information AccessHistory
Temporaldata

EconomyTechnology
Society

Infrastructure

Roles,
work task
Organisation
Contextualtask

Interaction

Hyperlink
Reference
Multi-docs

Document
structure

Betweendocument

Time

User andInformation
User andsystemCharacter
Word
Syntax
Structure

Adapted from Ingwersen & Järvelin (2005)
41

NTCIR-11 Tasks
Covers a wide
range of context
and rich media
sources

Infrastructure

Work task,
roles

IMine

Document
structureTime

QA Lab

RITE-VAL
Math-2Math

MedNLP-2
Medical

Web

Exam
News

SpokenQ&R

Betweendocument

MobileClick

Temporalia

Interaction

Speech42

NTCIR’s Long-Term View
Covers a wide
range of context
and rich media
sources

Infrastructure

Work task,
roles

IMine

Document
structure

Time

NTCIR’sfor
QA Lab
RITE-VAL Infrastructure
IA Evaluation
Math-2Math

MedNLP-2
Medical

Web

Exam
News

SpokenQ&R

Betweendocument

MobileClick

Temporalia

Interaction

Speech43

Why participate?
Case for students and industry

44Why participate? (Students)• Perfect schedule– Task: Jan-Aug 2014
– Writing: Sep-Nov 2014
– Presentation: Dec 2014

• Easy start-up– Much of experimental
setup is provided– Performance measuresare (often) defined• Publications– Comparison with otherparticipants can produce
stronger arguments– Inspired by theinternational
community for future
work• Diverse tasks– Range of Information
access tasks to tackle

45

Why participate? (Industry)• Establish your brand– To your end-users andcompetitors– Recruit smart people• Fair benchmarking– Comparison with yourown products can beless reproducible– Critical self-assessments• Faster development– Brush up your product oreliminating bugs in alimited period of time

• Early access to resulted
resources– Secondary resources
developed by the task
are yours, too

46

How to participate
Simple six steps

47

How to participate
1. Read the task
description and CFP
carefully
2. Contact a TO if you
have questions
3. Decide a task toparticipate
4. Register as aparticipant at NTCIR
website5. Fill in User Agreement
Forms
6. Keep an eye on atask’s ML, website,
etc. to follow theactivity
Don’t hesitate
to send a
feedback to TO

48

Reminder for participants

49Participant’s agreement• Submit a paper to NTCIR-11 Conference– Demonstrate your systems and methods– Failure analysis is strongly encouraged• Attend to the conference and give a
presentation– Every group will give a poster presentation– Some will give an oral presentation, too

50Group ID policy• Group ID is independent of participating tasks– Please use the same Group ID if the groupmembers are identical, even if your group
participates in multiple tasks– Please use a different Group ID if the groupmembers are different across participating tasks• Group ID is first come first serve manner– NTCIR office might ask you to revise the name51

Important Dates
For your diary52

Important Dates
Sep 02, 2013

Kick-off event in Tokyo

Dec 20, 2013

Task registration due

Jan 05, 2014

Document set release

Jan – May, 2014

Dry runMar – Jul, 2014

Formal run

Aug 01, 2014

Evaluation results due

Aug 01, 2014

Task overview partial releaseSep 01, 2014

Participant paper submission due

Nov 01, 2014

All camera-ready copy for the Proceedings due

Dec 9-12, 2014

NTCIR-11 Conference, NII, Tokyo, Japan

Contact TO forthe exact
schedule

53

Wrap-up• The eleventh cycle of NTCIR has started– Will conclude in Dec 2014

• Eight exciting tasks are running– Organised by leading researchers worldwide• Lots of opportunities for innovative work
– Exchange great ideas with the community• What’s missing is your participation!
54http://research.nii.ac.jp/ntcir/ntcir-11/

Thank you for your attention!
For further enquiries, contact the NTCIR officentc-secretariat nii.ac.jp

Q&A

55