NTCIR-12 MobileClick-2
Makoto P. Kato (Kyoto University),
Virgil Pavlu (Northeastern University),
Tetsuya Sakai (Waseda University), andTakehiro Yamamoto (Kyoto University)

Let's seethe current
mobile search
2

"NTCIR"
What's NTCIR?
Your Search Stats

Clicks: 2
Time: 00:31

3

"NTCIR"
When isthe deadlineof NTCIR?
Your Search Stats

Clicks: 2
Time: 00:29

4

5

6

"NTCIR"
What's NTCIR?

Home | NTCIRThe NTCIR Workshop is a series of evaluation
workshops designed to enhance research ininformation access technologies including
information retrieval, summarization, extraction,question answering, etc.

Your Search Stats

Clicks: 0
Time: 00:03
NTCIR-12

Held on June 9(Tue)-12(Fri), 2016
at National Center of Sciences, Tokyo, JapanNTCIR-12 NewsNTCIR-12 Schedule
NTCIR-12 Tasks

7

"NTCIR"NTCIR-12 Schedule

When isthe deadlineof NTCIR?

20/Jan./2016:

Task Registration Due

06/Jan./2016:

Document Set Release

Home | NTCIRDry Run

Jan.-May/2016:

Mar.-July/2016:
The NTCIR Workshop is aFormal
series Runof evaluation
workshops designed to enhanceresearchin Due
01/Aug./2016:
Evaluation
Results
information access technologies
including
01/Aug./2016:Task overview
releaseinformation retrieval, summarization, extraction,
15/Sep./2016:
Paper submission Duequestion answering, etc.
01/Nov./2016:
All paper Due
NTCIR-1209-12/Dec./2016:
NTCIR-11 Conference
Held on June 9(Tue)-12(Fri), 2016
at National Center of Sciences, Tokyo, Japan

Your Search Stats

Clicks: 1
Time: 00:15

NTCIR-12 NewsNTCIR-12 Schedule
NTCIR-12 Tasks

8"NTCIR"NTCIR-11 Schedule

When isthe deadlineof NTCIR?

http://research.nii.ac.jp/ntcir/schedule

20/Jan./2014:

Task Registration Due

06/Jan./2014:

Document Set Release

Home | NTCIR

Jan.-May/2014: Dry Run

Your MobileClick
Search Score
Goal ofMar.-July/2014:
Formal Run
http://research.nii.ac.jp/ntcir/
The NTCIR Workshop
is aResults
series ofevaluation
01/Aug./2014:
Evaluation
Dueworkshops designed to enhance research in
01/Aug./2014: Early draft Task overview release
information access technologies including
15/Sep./2014:
Draft summarization,
paper submission
Due
information retrieval,extraction,question answering,etc.
01/Nov./2014:
All camera-readypaper Due

Clicks:1
Provide Direct and Immediate
Time: 00:15
09-12/Dec./2014: NTCIR-11 Conference
NTCIR-11Mobile Information Access
http://research.nii.ac.jp/ntcir/

Held on December 9(Tue)-12(Fri), 2014
at National Center of Sciences, Tokyo, JapanNTCIR-11 News
http://research.nii.ac.jp/ntcir/news
NTCIR-11 Schedule
http://research.nii.ac.jp/ntcir/schedule
NTCIR-11 Tasks
http://research.nii.ac.jp/ntcir/tasks

9"NTCIR"NTCIR-11 Schedule

What’s new?

When isthe deadlineof NTCIR?

http://research.nii.ac.jp/ntcir/schedule

20/Jan./2014:

Task Registration Due

06/Jan./2014:

Document Set Release

Home | NTCIR

Jan.-May/2014: Dry Run

Mar.-July/2014:
Formal Run
http://research.nii.ac.jp/ntcir/
The NTCIR Workshop
is aResults
series ofevaluation
01/Aug./2014:
Evaluation
Due
workshops designed to enhance research in
01/Aug./2014: Early draft Task overview release
information access technologies including
15/Sep./2014:
Draft summarization,
paper submission
Due
information retrieval,extraction,question answering,etc.
01/Nov./2014:
All camera-readypaper DueYour Search Score• Two subtasks
for En/Ja
Clicks:1• iUnits provided,
not extracted
Time: 00:15
09-12/Dec./2014: NTCIR-11 Conference
NTCIR-11
http://research.nii.ac.jp/ntcir/• Instant feedback• Leader board

Held on December 9(Tue)-12(Fri), 2014
at National Center of Sciences, Tokyo, JapanNTCIR-11 News
http://research.nii.ac.jp/ntcir/news
NTCIR-11 Schedule
http://research.nii.ac.jp/ntcir/schedule
NTCIR-11 Tasks
http://research.nii.ac.jp/ntcir/tasks

10

iUnit Ranking Subtask• Given a query and a set of iUnits,
rank them based on their estimated importance
Note: iUnits are information pieces relevant to a given query
Input: Query
Output: iUnit list

NTCIR
Input: iUnit setiUnit
1

A series of evaluation workshopsiUnit

2

Task Registration Due 20/Jun./2016

A series of evaluation workshops

3

Designed to enhance IA research

Designed to enhance IA research
Task Registration Due 20/Jun./2016

nDCG

0.5

… …

Challenge

Predict the importance of strings rather than documents
11

iUnit Summarization Subtask• Given a query, a set of iUnits, and a set of subtopics,
generate a two-layered summary
Input: Query
NTCIR

Input: iUnit setiUnit
A series of evaluation workshops
Designed to enhance IA research…

Input: Subtopics
Subtopics
News

Output: Two-layered summaryThe NTCIR Workshop is aseries of evaluation
workshops designed toenhance research ininformation accesstechnologies including
information retrieval,summarization, extraction,question answering, etc.

Evaluation metric
designed for mobile
information accessM-measure
2nd layer

News
Schedule
Tasks Challenge
20/Jan./2016:

Task Registration Due

06/Jan./2016:

Document Set Release

Jan.-May/2016:

Dry RunMar.-July/2016:

Formal Run

01/Aug./2016:

Evaluation Results Due

01/Aug./2016:

Task overview release

15/Sep./2016:

Paper submission Due

01/Nov./2016:

All paper Due

09-12/Dec./2016:

NTCIR-11 Conference

0.5

Lay out iUnits so that any types of users can be
immediately satisfied (similar to the IMine task)

Schedule…

12

Leader Board System

Q. When can we get our evaluation result?
A. Right after you submit your run!

Got a result

Submission

Leader board
Impressiveresult!

Feedback• Evaluation for training queries from Apr 2015
• Evaluation for test queries from Aug 201513

Who Should Participate?
• Those who are interested in– Mobile information access– Intent detection– Diversification– Summarization– Q&A– Nugget-based evaluation• Those who participated in– 1CLICK tasks– INTENT, IMine tasks– RITE tasks
14

MobileClick-2 Schedule
Month

EventApr, 2015

First CFP
Evaluation system for training queriesJun, 2015

Registration dueAug, 2015

Test query release
Evaluation system for training queries

Dec, 2015

Run submission due

Mar, 2016

Participant paper submission dueJun, 2016

NTCIR-12 Conference
15

"NTCIR"

MobileClick-2
NTCIR-11 Schedule

When isthe deadlineof NTCIR?

http://research.nii.ac.jp/ntcir/schedule

20/Jan./2014:

Task Registration Due

06/Jan./2014:

Document Set Release

Provide Direct and Immediate Mobile
Your Search Score
Information AccessHome | NTCIR

Jan.-May/2014: Dry Run

Mar.-July/2014:
Formal Run
http://research.nii.ac.jp/ntcir/
The NTCIR Workshop
is aResults
series ofevaluation
01/Aug./2014:
Evaluation
Due
workshops designed to enhance research in
01/Aug./2014: Early draft Task overview release
information access technologies including
15/Sep./2014:
Draft summarization,
paper submission
Due
information retrieval,extraction,question answering,etc.
01/Nov./2014:
All camera-readypaper Due

Clicks: 1Apr 2015
Time:00:15
Evaluation system for training queries open at
09-12/Dec./2014: NTCIR-11 Conference
NTCIR-11
http://research.nii.ac.jp/ntcir/

Held on December 9(Tue)-12(Fri), 2014
at National Center of Sciences, Tokyo, Japan

http://www.mobileclick.org/NTCIR-11 News
http://research.nii.ac.jp/ntcir/news

NTCIR-11 Schedule
http://research.nii.ac.jp/ntcir/schedule

Aug 2015 - Dec 2015
Real-time evaluation
NTCIR-11 Tasks
http://research.nii.ac.jp/ntcir/tasks

16