Call for NTCIR-10 Task Proposals

NTCIR is a series of evaluation workshops designed to enhance research in Information
Access (IA) technologies such as Information Retrieval, Question Answering, Text
Summarisation, Opinion Analysis, etc. NTCIR-9, the ninth cycle of the workshops, saw
several changes, including community-led task coordination and a new workshop
management structure where Task Organisers, Program Chairs and General Chairs
share different responsibilities.

We were excited to see six brand new tasks that were accepted as NTCIR-9 tasks andrun successfully. Their outcomes will be presented at the NTCIR-9 Workshop Meeting
to be held in December 6-9, 2011 at NII (Tokyo).

It is time to call for NTCIR-10 task proposals. We are looking for proposals of evaluation
tasks that can help advance our understanding of Information Access processes,technologies, and systems. In NTCIR-10, we are looking for two types of evaluation
tasks: Core challenge tasks and Pilot tasks.

lCore Challenge Task: this is for benchmarking and assessment of solutions to aparticular Information Access problem within one or two NTCIR cycles. It also aims
to establish the best practices to address each of the problems. NTCIR-9 had four
core tasks including INTENT (Diversity search), RITE (Text Entailment for QA),
GeoTime (Spatial/Temporal IR), and SpokenDoc (IR for spoken documents).

l

Pilot Task: this type is suitable for the first cycle of a new task where evaluation
methodology needs to be developed. Or, a task can only looks at a component (i.e.,MT) and apply it to an IA application in the next cycle (i.e., MT for IR). NTCIR-9 had
three pilot tasks such as VisEx (Visualisation for Explorative Search), CrossLink(Cross-lingual Link Discovery), and PatentMT (Machine translation for patentdocuments).

NTCIR-10 Program Committee will carefully assess the proposals based on selection
criteria given below. Please make sure to include sufficient information for the1Committee to assess the proposals using the criteria. A seed funding will be available
for the accepted evaluation tasks to cover some of the expenses (data collection
purchase, relevance assessments, etc.). However, task organizers are encouraged toapply for external funding whenever appropriate, to make the resultant data publiclyavailable, and to involve the participants or community in the data creation process.

Coordinating an evaluation task is a great way to create a new research area andestablish an international community. We are looking for innovative and exciting ideas
for the evaluation tasks in Information Access! Please send your proposal in PDF format
to ntc-admin

by Midnight of 25 November 2011 (JST) with the subject:

NTCIR-10 Task Proposal.

Best regards,

Tetsuya Sakai and Hideo Joho
NTCIR Program Chairs

Noriko Kando, Tsuneaki Kato, and Eiichiro Sumita
NTCIR General Chairs

NTCIR-10 Programme Committee (Confirmed as of Oct 17, 2011)
Charles Clarke (University of Waterloo, Canada)
Kalervo Jarvelin (University of Tampere, Finland)
Hideo Joho (Co-chair, University of Tsukuba, Japan)
Gareth Jones (Dublin City University, Ireland)
Noriko Kando (NII, Japan)
Tsuneaki Kato (The University of Tokyo, Japan)
Tetsuya Sakai (Co-chair, Microsoft Research Asia, PRC)
Mark Sanderson (RMIT, Austraria)
Ian Soboroff (NIST, US)2ll

Important DatesØ25 November 2011: Deadline for proposalØ16 December 2011: Notification of the resultsØ

January 2012: Kick-off Event (Call for task participation)

Proposal FormatØØl

Main Part (A single A4 page)²

Task Name²

Task Type (Core or Pilot)²

Outline²

Purpose²

Methodology²

Expected results

Appendix (Free format)²Name and contact information of the organizer(s)²Expected participants²Resources (document sets and others) to be used or constructed²

Budget planning²

Schedule²

Other notes

Selection Criteria
Ø

Essential²

Responsibility: how do the organizers plan to advocate the significanceof the proposed task and commit to it?

²

Sustainability: how do the organizers plan to sustain the task activity
financially after the seed funding runs out?

²

Repeatability: Will the participants’ work be reproducible? Scientificmerits: what is the significance of potential research being conducted in
the task? What are the potential benefits to other work and related
technologies?

3

ØDesirable²

Social impact: what social problems/issues can the expected results ofthe task address?

²

Synergy: How will the task fit in with other prospective tasks? Can thegrand challenge task leverage core challenge tasks, or can the core
challenge task contribute to a grand challenge task?

²

Generalisability: how would the results of the task generalize to othersituations and data?

²

Accountability: how does the task plan to ensure that participants
perform and report a component-level evaluation as well as end-to-end
evaluation? Are there any potential components?

Last update: 2011.10.17

4