Welcome!

Twitter: #ntcir9
Ust: ntcir-9-kick

NTCIR-9 Kick-Off
ff Event
2010.10.05
日本語セッション: 13:30English
li h Session:Si
15:30301

Program• About
Ab t NTCIR• About NTCIR-9• Accepted Tasks• Why participate?
• How to participate• Important Dates• Q&A2

About NTCIR

3NTCIR: NII Testbeds and Community for Information access Research
Research Infrastructure for Evaluating IA

A series of evaluation workshops designed to enhanceresearch
h iin information-accessi ftit h l i b
technologies
by providing
idi aninfrastructure for large-scale evaluations.
■Data sets,, evaluation methodologies,
g , and forum

Project started in late 1997
Once every 18 months

Data sets (Test collections or TCs)
Scientific, news, patents, and web
ChiK
d English
E li hChinese,Korean, JJapanese, andTasks (Research Areas)
IR: Cross-lingual tasks, patents, web, Geo
QA：Monolingual tasks, cross-lingual tasks
Summarization, trend info., patent maps
Opinion analysis, text mining

Community-basedCi b d ResearchRh Activities
A i ii4

Information retrieval (IR)• Retrieve RELEVANT information from vast collection to meetusers’’ iinformationfti needsd• Using computers since the 1950s• First CS uses human assessments as success criteria– Judgments vary
– Comparative evaluations on the same infrastructure

Informationo at o access ((IA))
Whole process to make information usable by users.
ex.: IR, text summarization, QA,
Q text mining, anddclustering
5Tasks at Past NTCIRs
NTCIR

1

2

3

4

5

6

7

8

'99 '01 '02 '04 '05 '07 '08 '09User Generated■ Community QA■ ■ ■ Opinion Analysis

Contents
Module-Based
IR for Focused
Domain
QuestionA
Answeringi

Summarization /Consolidation
Web

■ ■
■
■ ■ ■ ■ □■ ■ ■Cross-Lingual QA + IR
Geo Temporal
PatentComplex/ Any Types

Dialog■ ■CLil■ ■ ■ ■ Cross-Lingual
Factoid, List
■ ■ ■ ■
■ ■ ■ ■ ■ Text Mining / Classification

■ ■ ■
■ ■ ■
■ ■ ■

Trend Info Visualization
Text SummarizationWeb■ ■ Statistical MT

Crosslingual
Retrieval■ ■ ■ ■ ■ ■ ■ ■ Cross-Lingual IR
Non English Search■ ■ ■ ■ ■ ■ ■ ■ Non-English

Text Retrieval■ ■ ■ ■ ■ ■ ■ ■ Ad Hoc IR, IR for QA

The Years the meetings were held. The tasks started 18 months before

6Procedures in NTCIR Workshopsp• Call for Task Proposals• Selection of Task Proposals by Committee• Discussion about Experimental Designs and Evaluation Methods(can beb continued
d to Formall Runs)• Registration to Task(s)– Deliver Training Data (Documents,(Documents Topics,
Topics Answers)• Experiments and Tuning by Each Participants– Deliver Test Data (Documents and Topics)• Experiments by Each Participants• Submission of Experimental Results• Pooling the Answer Candidates from the Submissions
Submissions, andConduct Manual Judgments• Return Answers (Relevance Judgments) and Evaluation Results• Workshop Meeting
Discussion for the Next Round
7

NTCIR Workshop Meeting
NTCIR:

http://research nii ac jp/ntcir/
http://research.nii.ac.jp/ntcir/
8

8

NTCIR-7 & -8 Program Committee

Mark Sanderson, Doug Oard, Atsushi Fujii, Tatsunori Mori,Fred Gey, Noriko Kando(and Ellen Voorhees,
Voorhees Sung Hyun Myaeng,
Myaeng Hsin-Hsi
Hsin Hsi Chen
Chen,
Tetsuya Sakai)

9

NTCIR Test Collections

Test
est CoCollections
ect o s = Docs
ocs + Topics/Questions
op cs/Quest o s + Answers
s es

Available to Non-participants for Research Purpose

10Focus of NTCIR
Lab-type
yp IR Test

New ChallengesgAsian Languages/cross-language
Variety of Genre
Parallel/comparable Corpus

Intersection of IR + NLP
To make information in the documents
more usable for users!
Realistic eval/user task
Interactive/Exploratory search
QA ttypes att ttopic
i creaForum for Researchers andOther Experts/users
Idea Exchange
Discussion/Investigation on Evaluation
methods/metrics
11

IR Systems Evaluation• Engineering Level: Efficiency• Input Level: ex. Exhaustivity, quality, novelty of DB• Process Level: Effectiveness ex. recall, precision• Output Level: Display of output• UserU Level:L l ex. Effort
Eff t thatth t users needd• Social Level: ex. Importance(Cleverdon & Keen 1966)

12

Difficulty of retrieval varies with topics
1.0000J-J Le ve l1 D auto
EffectivenessacrossSYSTEMS

Effectiveness across TOPICS on
system 検索システム別の11pt再現率精度
10.8000

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

A

Average over 50
topicsB
CD

0.8

0.6000precision

E

precision

F

0.4000

G

0.6

H
I
J

0.4

K
L

0.2000

M
N

0.2

O

0.0000
0

2

4

6

8

0

0.

0.

0.

0.

0.

1.

P

recall

0
0

0.1

0.2

0.3

0.4

0.5
recall

0.6

0.7

0.8

0.9

1

13

Difficulty of retrieval varies with topics
J JL
J-J
Le ve l1 D autot

1.0000Effectiveness across
SYSTEMS

検索システム別の11pt再現率精度
Effectivenessacross TOPICS on
system10.8000“Difficult
topics”
vary with systems
Averageover 50
A
B

topicsJ-JJJ Le ve l1 D auto
0.8

0.60000 80000.80000.4000

D

B

E

C
D

G

06
0.6

E

H

0.60000.4

0.40000.2000

I

F

J

G

K

H

L

I

M

J

N

0.2

0.2000K

O

0.0000LP

7110 .2
0
11
3
0
11 .4
6
11
9
120 .
26
12
5
120
8.8
13
1
131
4.0
13
7
14
0
14
3
14
6
14
9

0

0

10

1

0
1 .

04 0

0.000010

preecisionA

F

precisioon

Mean av. precision preecision

1.0000C

recallT opic#Requests#101-150#101 150

0.1

0.2

0.3

For reliable
and stableMevaluation,using substantial #
N
0.4
0.5
0.6
0.7
0.8
0.9
1
of recall
topicsisnecessary.
O
P

14

101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

What are TCs usable for evaluating?
g

Pharmeceutical R & D
Phase I:
In vitro
experiments

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

15

What are TCs usable for evaluating?
g
NTCIR

Users’ information-seeking
tasks

Test collections
Phase I:

Phase II:

Phase III:

Phase IV:

Laboratory-type
testing

Sharing modules,
P t tPrototype
testing

Controlledinteractive testing
using humansubjects

Uncontrolled preoperational testing

Phase II:

Phase III:

Phase IV:

Animal experiments

Tests with healthy
human subjects

Clinical tests

Pharmeceutical R & D
Phase I:
In vitro
experiments

2.Input level、Levels ofevaluation
1. Engineering level:
3.Process level: Effectiveness6.Social level

4.User level、5.Output level

Efficiency
16• Information Seeking
g Task– document types + user community– useruser’ss situation
situation, purpose of searchsearch, realistic
Experiments are Abstraction of the Real World Tasks.
Tasks
Trade-off between “reality” and “contorable”• Testing & Bench marking
To learn how and why the system works better (worse)
than others
To learn how it can be improved
Scientific Understanding
g of the effectiveness17

Improvementpof Effectiveness byy Evaluation Workshopsp
1.5 – 2 times in 3
years

Mean Avverage P
M
PrecisionnCornell University TREC Systems

0.5
0.4

TREC-1

0.3

TREC-2
TREC-3

0.2

TREC 4
TREC-4

0.1

TREC-5
TREC-6

0
'92

'93

'94

'95

'96

'97

'98

SystemSystemSystemSystemSystemSystemSystemTREC-718

Research Trends
Number of Papers Presented at ACM-SIGIR
450

400

WEB
350

User
Evaluation# of pap
pers

300

Non-Text
QA & Summarization

250

NLP
Cross-Lingual

200

ML
Clusteringg

150

Efficiency
Filtering

100

Query Processign
IR Models

50

General
0
77 79'77-79'80-8480 84'85-8985 89'90-9490 94'95-9995 99'00-0400 04'05-0905 09

PpublicationYears
19

Some Thoughts
g on Future• Requirements for Evaluating Individual Applications– Ex.
Ex Enterprise search,search Federated Search,
Search etc
etc.

• Interactive and Exploratory Information Access– Users
Users’ Intention,
Intention Diversity– Collaborative Search– Expert
p Search,, Search for Expertisep
and Knowledge,g ,Inference, etc.

• Answer “No”
• Using Ontology, Metadata• Multilingual, Cross-Lingual
etc.

20

About NTCIR-9
What’ss new?
What

21

NTCIR 9: Objectives
NTCIR-9:• Solid
S lid foundationfd i– New structure• Task diversity• Community-led
Ci l d taskk
organisation– Sustainability of
research– Covers a wide
context inInformation Access• Promotion ofresearch resources– Studies
St di richi h media
ditypes– Show case in the
NTCIR-9 Meeting
22

NTCIR 9: Structure
NTCIR-9:• General Co-Chairs– Noriko Kando (NII)– Tsuneaki Kato (Tokyo
University)– Eiichiro Sumita (NICT)• Evaluation Co-Chairs– Hideo Joho (Tsukuba
University)– Tetsuya Sakai (MSRA)• Task Organisers– 31 researchers
worldwide– Participantsp(You!)()• EVIA Co-Chairs– Markk Sandersond(RMIT)– William Webber(Melbourne University)

23

NTCIR 9: Development so far
NTCIR-9:
Jun 2010
July 2010
Aug 2010
Sep 2010
Oct 2010

New structure formedffor
f NTCIR-9
Call for task proposal announced and
10 proposals
l were submitted
b i d
7 proposals were accepted by the task selection
committee
itt anddE
Evaluationl ti co-chairs
h iCalls for task participation prepared
NTCIR-9 Kick-Off
k ff Event24

NTCIR-9 Evaluation Tasks
Calls for task participation

25

Tasks accepted for NTCIR-9
NTCIR 9CORE TASKS• [Intent] Intent (with One-Click Access)• [RITE] R
Recognizing
i i IInferencefi Textin
T t• [GeoTime] Geotemporal information retrieval• [SpokenDoc] IR for Spoken Documents
PILOT TASKS• [CrossLink] Cross-lingual Link Discovery• [Vis-Ex][Vis Ex] Interactive Visual Exploration• [PatentMT] Patent Machine Translation
26

Intent + One-Click Access
Calls for task participation

27

NTCIR-9 Intent Task

28

Subtasks• Subtopic Mining [Chinese and Japanese]– Given a real web query,
query participating systems minepossible intents from web collections and query logs
• A one-month click-through log is available for Chinese
QUERY: Harry Potter
INTENTS: Books? Movies? Character?...

– Submitted intent lists will be evaluated in terms ofcoverage and novelty; each intent will be weighted by
votes from many assessors• Ranking [Chinese and Japanese]– Participatingpg systemsyselectivelyy diversifyy search
results– Search results will be evaluated by diversity metrics
using key intents obtained from Subtopic Mining
29

Subtasks (cont.)• One Click Access (“1CLICK”) [Japanese]– Current search engines
User (1) enters query (2) clicks on Search button(3) scans ranked list (4) clicks on URL that looks relevant(5) reads the page (6) finds the answer– One Click Access (for desktop and mobile)mobile):User (1) enters query( ) clicks(2)
li k on SearchSh buttonb tt(3) finds the answer– ZeroZ
Cli
Clickk A
AccessUser (1) finds the answer
without
ith t clicking
li ki on Search!
Sh!
30

Introductiondto NTCIR-9 RITE Taskk((Recognizinggg Inference in TExt))

Hideki
Shima1

TerukoMitamura1

1Carnegie Mellon University

Hiroshi
Kanayama2

KoichiTakeda22IBM Research - Tokyo

NTCIR-9 Kick-Off Event
October 5th, 2010

Chuan-Jie
Lin3
3National Taiwan

Ocean University

Cheng-Weig
Lee4
4Academia Sinica

Overview of RITE
RITE isi abbenchmarkhk ttaskk ((nott a competition!)
titi !)
for automatically detecting entailment,
paraphrase,handd contradiction
t di ti iin ttext.
t
Given t1, can a computer infer thatt2 is most likelyy true?

– t1: Yasunari Kawabata won the Nobel Prizein Literature for his novel “Snow Country”– t2: YasunariYiK
Kawabata
b t isi th
the writer
it off “Snow“SCCountry”
t ”(Target languages: Japanese, Simplified Chinese, Traditional Chinese)Three Subtasks in RITE
Binary-class subtask•

Given a ttextt pairGii <tt1 ,tt2>, your system
t
will
ill d
detectt t whether
h th t1 entails
t il ahypothesis t2 or not

Multi-class ((5-way)y) subtask•

Given a text pair <t1 ,t2>, your system detects whether t1 and t2– has entailment relation: t1 t2 / t1  t2 / t1  t2– does not have entailment relation: contradiction / independence

RITE4QA subtask••

Same input and output as the binary-class subtask
Evaluated in an extrinsic way– Evaluation method: design the dataset/metric as if a system is ananswer-filtering module in a Question Answering system.
– Data: t2 is a question converted to affirmative statement with a wh-word
replaced with an answer candidate. t1 is a sentence/paragraphcontaining the answer candidate.

Why you should participate
In addition to researchers in entailment and paraphrase
paraphrase,
various research fields can benefit from RITE:• Core technologies: Semantic processing, Lexical acquisition,
Machine learning,g, …
• Applications: Information retrieval, Question answering,
Summarization, …
We tryWt hardh d tto welcome
lwide
id variety
i t off participations
ti i ti– fromf
undergraduate students to industry researchers, from all over theworld.
• Resource pool will be available to help you build a prototype systemquickly or participate in collaboration by sharing useful resource withothers.
• Mailing list is available for receiving important announcements andjoining the task design discussion
Website: http://artigas.lti.cs.cmu.edu/rite

GeoTime
Calls for task participation

35

GeoTime2 (E,(E JJ, K?)
Organisers: Fred Gey, Ray Larson (UCB), Noriko Kando (NII),• Second round of ad hoc IR for WHEN and WHERE
p How old was Max Schmelingg when
• GeoTime1 topic:
he died, and where did he die?
At GeoTime1
GeoTime1, docs that contain both WHEN and
WHERE were treated as relevant;Those that contain only WHEN or only WHERE info
were treated as partially relevant.
C we dCando bbetter??
Can we create more realistic
GeoTime topics?
36

GeoTime2: Some ideas to make the task morechallenging (under discussion):
Search New Information
all topics including timestamps to indicate thequery period. Search new information on a topic since somestart date (up to the query time).

Time Reasoning• Relative expression such as “yesterday”, “last week”, “after”,
etc.

Geo Reference• Geo Tagging:
gg g Annotating
g geographical
g g p
names
• Geo Coding: Coding geographical points by codes –geographical reasoning such as “near”, “part of” feasible
37

Test Collection• News Papers– English:g
New York Times (2002-2005)(
5) from LDC– Japanese: Mainichi News Papers (2002-2005)– Newspapers from Korea (2002-2005)(2002 2005) (?)• Topics 25 +?
• Topic creation and Relevance Judgments will
be done by the participants

38

NTCIR 9 Core Task
NTCIR-9

IR for Spoken Documents (SpokenDoc)
Tomoyosi Akiba (Toyohashi University of Technology)
Hiromitsu Nishizaki (Yamanashi University)
Kiyoaki Aikawa (Tokyo University of Technology)
Tatsuya Kawahara (Kyoto University)
Tomoko Matsui (The Institute of Statistical Mathematics)
Spoken Document Processing WG, SIG-SLP,IPSJ

39

Background• CurrentCt IR methods
th d assume cleanltargettt
documents, i.e. text without errors.
• However,
However the real-world
real world documents are noisy!
– UGC with typos and specific usage of terms.
– Text data obtained by automatic processing like OCR
or MT.
– Speechpdata (spoken( pdocuments),), e.g.
g p
podcasts,,
broadcast news clips, spoken lectures, etc.
– ...

• Require methodologies to deal with those noisy
documents for IR.

Our Focus
40

Task Overview• TargetTtD
Documentst– 2702 lectures in Corpus of Spontaneous Japanese (CSJ),
628hrs.

• Subtasks– Task1: Spoken Term Detection• Find the occurrences of the given queried term.

– Task2: Spoken Document Retrieval• Find the passages including the relevant information related to agiven query topic.

• We are going to provide the reference speech
recognition results (N(N-bestbest or lattice representation)
so that those who are interested in SDR but not in ASR
participatep
our tasks.
can p
41

Merits of the Participation• Organi
Organizing
ing grogroupsps (IPSJ SIG-SLP
SIG SLP SDP working
orking group)
gro p) have
ha e alreadyalreadreleased 2 test collections.
1.

CSJ STD test collection [Itoh et al., 2010]•2.

250 query termst
iin ttotal.
t l

CSJ SDR test collection [Akiba et al., 2009]• 39 query topics asking for passages in lectures.
• Relevance Judgment has been conducted manually (maybe imperfect).
imperfect)• Task Participants can get the extended and refined version of thosetest collections.
– MoreMquery termstandd query topics
t i– Pooling based relevance judgment– New evaluation metrics (including time and space efficiency,ddocument-levelt l l andd passage-levell l relevancy)l)• Visit our Web site for more information– http://www.nlp.cs.tut.ac.jp/~sdpwg/index.php?ntcir942

Cross-Lingual Link Discovery
Calls for task participation

43

Whyy Cross-Lingualg Link Discovery?
yexamplecross-lingual link discovery

but not linked
yet in the text
for the crabs
Actually, there is a
page about “花蟹”
in English

mono-lingual link
discovery

link these two pages
with each other

Language link here,
but there is no
language link
to apage inlanguages other thanChinese1.Trying
T i tto searchh whath t iis “花蟹” on Wikipedia,
Wiki diand maybe the “花蟹” articles in other languages.
2. Wikipedia brings you the “香港十元紙幣(HongKong ten-dollar note)” not the “flower crab”.
3. What about the actual crab? Where can we findthe English page about花蟹?
But there is no language link
here to the equivalent page in
4. What link discovery can do?
Chinese44

Cross-lingual Link Discovery Task in
NTCIR-9• Cross-lingual link discovery (CLLD) is a way of
gp
potential links between
automaticallyy finding
isolated documents in different languages.
• Task goal: to create a reusable resource for
evaluating automated cross language linkddiscovery
approaches.
h The
h results
l off thishg and refininggresearch will be used in building
systems for automated link discovery.
45

We Need Cross-Lingual
Cross Lingual Link Discovery• Not
N t just
j t forf Wikipedia
Wiki di andd nott just
j t forf wiki.
iki• Cross language link discovery can allow us todiscover documents on web or on a digital
library in languages which we either are morefamiliar with,
with or which have a richer set of
documents than in our language of choice.

logos of various Wiki software

FACT:•There are at least 83 popular Wikis in the world serving
people from different language background for different
needs.
needs•There are at least 44 wiki-style documentation
management software and their forks to help numerous
projects or corporations to manage knowledge.
(source from Wikipedia)
46

VisExInteractive Visual Exploration Task
Organized by

Tsuneaki KatoThe University of Tokyo

Mitsunori Matsushita
Kansai University47

Wh t isWhat
i VisEx
Vi E•

A pilot task for establishing an evaluation framework of explorative
information access environments•

ParticipantsPti it submitb it ththeir
i proposingiInformation
I ftiA
Access Environment
E it
Systems (IAESs)– which should be able to be embedded in a common framework– which shoud be able to hundle given experimental tasks•

Submitted IAESs are evaluated in laboratory experiments withhuman subjects for gathering subjective and objective data•

Reports are requested, that explain the experimental results interms of the process primives and process model of subimitted IAEs

An efficient and effective evaluation frameworkp
information access
A model of explorative

Final
Objective!

48

T kO
Task
Outlinetli
ProvideFramework
Organizer Provide
Baseline

Browser (Log Collection)
IAES Core

Mainly by the Organizer… etc. Func.

Laboratory
Experiments

Log Collection
Documents

Participants

EditorLog Collection
Display

SubmitExperimental
Tasks

Info. Retrieval
EngineIt is important to discuss the followings through the WS• I/F/ between
bet ee ana IAES
S coreco e anda d thet e frameworka e o• Taxonomy of process primitives• Detailed design of the laboratory experiments

Human
Subjects
49

E
Experimental
i
t lT
Tasks
k
•

Event Collection Task
– Uses the event-list questions in the NTCIR-7 ACLIA Task
• Please tell me about incidents where NATO has recognized cases offfriendly fire.
• Please tell me about airplane crashes that have happened in Asia.
– Requests subejcts to gather nuggets (event characteristics such as its timeand place) as many as possible in a given time period•

Trend Summarization Task
– Is on summarization of the trend (not only changes but also thosebackground and influence) of time-series statisitcal information such as the
subjects of NTCIR-5,6,7 MuST
• Pl
Please ttellll me a summary off th
the statest t off th
the cabinet
bi t approvall ratingti ffrom 1998
8 tto
1999.

– Requests subjects to gather nuggets as many as possible in a given time
period, which are primitivepp
information that constitute a requestedqsummaryy
50

Info.
Info• Schedule– End of Oct. 2010
Participation Registration (First) Due– End of Dec. 2010
IAE I/F description Release– Latter part of Mar. 2011
IAE Framework and Baseline IAE Core Release– Latter part of Jul. 2011 Laboratory Experiments– Latter part of Aug. 2011 Experiment Results Release• Contacts
C– Tsuneaki Kato kato@boz.c.u-tokyo.ac.jp– Mitsunori Matsushita mat@res.kutc.kansai
mat@res.kutc.kansai-u.ac.jp
u.ac.jp• Home Page
– http://must.c.u-tokyo.ac.jp/visex
51

Patent Machine Translation task
(P
(PatentMT)MT)
Call for task participation

52

Background• Patent information is important information insociety worldwide• There is a large need for translation to accesspatent information written in foreign languages
and to apply for patents in foreign countries• We
W havehorganized
i d patentt t machine
hi translationtl ti
tasks to address this significant practical need53Outline• Participants
P ti it machine
hi ttranslatel t patentt t sentencest
N
New

Subtasks
Chinese to English
Japanese to English
English to Japanese

Parallel data
1 million sentence pairs
3 million sentence pairs
Test data: 2
2,000
000 sentences
Data type: patent description• Participants select subtasks in which they want toparticipate• Human evaluations for MT quality will be carriedoutPrimary evaluation
54

Why is it so exciting to participate in?
• Patents are one of the challenging domains forMT– Patent sentences could be quite long and containcomplex structures• Participants
pwill receive reliable evaluation for
their MT quality– Human evaluations will be carried out• Participants can use large-scale patent paralleland monolingual corpora• Participants can choose subtasks from threelanguage directions, including the popularllanguagedidirectionti off Chinese
Chi
to
t E
Englishli h55

Highlights• Chinese to English subtask has been added• Human evaluations will be carried out• 1 million Chinese-English and 3 million
J
Japanese-English
E li h patent parallel
ll l corpora
will be provided

56

NTCIR-9 Task Map
Summary57

Context of Information AccessHistoryHit
Temporaldata

EconomyTechnology
Society

Infrastructure

Roles
Roles,
work task
Organisation
Contextualtask

Interaction

Hyperlink
Reference
Multi-docs

Document
structure

Betweendocument

Time

User andInformation
User andsystemCharacter
WorddSyntax
Structure

Adapted from Ingwersen & Järvelin (2005)
58

NTCIR 9 Tasks
NTCIR-9
Infrastructure

Work task,
task
roles

IntentInteraction
Betweendocument

Document
structure

Covers wide
contextcote t + richcmedia types

Vis-ExTime

CrossLink

RITE
GeoTimeNews

PatentMT
Web

Legal
g

SpokenDoc
Speechp
59

NTCIR’ss Grand Challenge
NTCIR
Infrastructure

Work task,
task
roles

IntentInteraction
Betweendocument

Impact to real
cchallengesa e ges inour society

Vis-ExNTCIR’s

Time

CrossLink for
RITE Infrastructure

IA Evaluation
Document
structure

GeoTimeNews

PatentMT
Web

Legal
g

SpokenDoc
Speechp
60Why participate?
Case for students and industry

61

Why participate? (Students)• Perfect
P f scheduleh d l– Task: Jan - Aug 2011
– Writing: Sep – Nov 2011
– Presentation: Dec 2011

• Easy start-up– Much of experimental
setup is provided– Performance measuresare (often) defined• Publications
P bli i
– Comparison with otherparticipants can produce
stronger arguments– Inspiredpbyy theinternational
community for future
work• Diverse tasks– Range of Information
access tasks to tackle

62Why participate? (Industry)• Establish your brand– To yyour end-users andcompetitors– Recruit smart ppeoplep• Fair benchmarking– Comparison withh yourown products can bebiased– Critical self-assessments• Faster development– Brush up
p youryp
product oreliminating bugs in ashort period of time

• Early access to resulted
resources– Secondary resources
developed
p byy the task
are yours, too

63

How to participate
Simple six steps

64How to participate
1. R
Read
d the
h taskkdescription and CFP
carefully
2. Contact a TO if you
have questions
3. Decide a task toparticipate
4. Register as aparticipant at NTCIR
website5. Fill iin U
User A
Agreement
Forms
6 Keep6.
K
an eye on atask’s ML, website,
etc to follow theetc.
activity
Don’t hesitate
to send a
feedback to TO

65

Important Dates
For your diary66

Important Dates
05/10/2010/ /Ki k ff event in
Kick-offi TokyoT k

20/12/2010

Task registration due

05/01/2011

Document set release01 - 05/2011

Dry run03 - 07/2011

Formal run

22/08/2011

Evaluation results due

22/08/2011

Task overview partial release

20/09/2011

Participant paper submission due

04/11/2011

All camera-ready copy for the Proceedings due

06-09/12/2011

NTCIR-9 Meeting
Meeting, NII,NII Tokyo,
Tokyo Japan

Contact TO forthe exact
schedule

67

Wrap up
Wrap-up• The ninth cycle of NTCIR has started– Under the new structure• Seven exciting tasks are running– Organised by 31 researchers worldwide• Lots
L t off opportunities
t iti forf innovativeiti workk– Exchange
g greatg
ideas with the communityy• What’s missing is your participation!
68http://research.nii.ac.jp/ntcir/ntcir-9/

Thank you for your attention!
For further enquiries, contact the NTCIR officentc-secretariat
ntc
secretariat nii.ac.jp

Q&A

69