Call for NTCIR-11 Task Proposals

NTCIR (NII Testbeds and Community for Information access Research)is a
sesquiannual series of evaluation conferences that mainly focuses on
Asian language information access. The first NTCIR conference(NTCIR-1) took place in August/September 1999, and the tenth(NTCIR-10) will take place in June 2013. Research teams from all over
the world participate in one or more of the NTCIR tasks to advance thestate of the art and to learn from one another's experiences.
http://research.nii.ac.jp/ntcir/ntcir-10/index.html

As the tasks being run at NTCIR-10 are reaching the final stages, itis time to call for tasks for the next NTCIR (NTCIR-11) which will beconcluded in December 2014. Task proposals will be reviewed by theNTCIR Programme Commitee, following the schedule below:
IMPORTANT DATES:

May 10, 2013 Deadline for task proposals (new and existing tasks)

June 7, 2013 Notification of accepted tasks

June 18-21, 2013 Task planning sessions at the NTCIR-10 conferenceJanuary 31, 2014 Deadline for additional pilot task proposals

February 28, 2014 Notification of accepted additional pilot tasks

Organisers of exising NTCIR-10 tasks (CrossLink, INTENT, 1CLICK,PatentMT, RITE, SpokenDoc, MATH and MedNLP) are required to submit a
new proposal if they wish to continue them for NTCIR-11. New task
proposals are also very welcome. To organise an evaluation task is toidentify important research problems, tackle them strategically bycollaborating with other researchers (participants), build thenecessary evaluation framework to advance the state of the art, andmake an impact to the research community and to the future.

We will accept two types of tasks:

Core challenge task: this is for fostering research on a particular information accessproblem by providing researchers with a common ground for evaluation. New testcollections and evaluation methods may be developed through the collaboration
between task organisers (proposers) and the task participants. At NTCIR-10, thecore challenge tasks were: CrossLink, INTENT, 1CLICK, PatentMT, RITE andSpokenDoc.

Pilot task: this is recommended if the information access problem to be tackled isnew and there are uncertainties as to how to evaluate it. It may focus on a
subproblem of an information access problem. It may attract a smaller group ofparticipating teams than core challenge task, but may grow into a core challengetask in the next round of NTCIR. At NTCIR-10, the pilot tasks were: MATH andMedNLP.

NTCIR will provide a "seed funding" to each accepted task, which can be used for some
limited purposes such as hiring relevance assessors.
However, organisers of each task are expected to make the task as self-sustaining aspossible. The amount allocated to each task will vary depending on requirements andthe total number of accepted tasks, but typical cases would be: around
1000000JPY/year for a core challenge task and around 500000JPY/year for a pilot task.

Please submit your task proposal as a pdf to the following easychair siteby May 10, 2013 (Japan Time: UTC+9).
https://www.easychair.org/conferences/?conf=ntcir11tp

As indicated in the important dates, there will be another call for additional pilot tasks
after the NTCIR-10 conference, to accommodate "late-breaking information accessproblems."

TASK PROPOSAL FORMAT:Main part (a single A4 page)

Task name and short name

Task type (core challenge or pilot)

Abstract

Motivation

Methodology

Expected resultsAppendix
Names and contact information of the organisers

Prospective participants

Data to be used and/or constructedBudget planning

Schedule

Other notes



REVIEW CRITERIA:



Importance of the task to the information access community and to the societyTimeliness of the taskOrganisers' commitment in ensuring a successful task

Financial sustainability (self-sustainable tasks are encouraged)

Soundness of the evaluation methodology

Language scope



NTCIR-11 PROGRAMME COMMITTEE:
Hsin-Hsi Chen (National Taiwan University, Taiwan)
Charles Clarke (University of Waterloo, Canada)
Kalervo Jarvelin (University of Tampere, Finland)
Gareth Jones (Dublin City University, Ireland)Gary Geunbae Lee (POSTECH, South Korea)
Maarten de Rijke (University of Amsterdam, The Netherlands)
Stephen Robertson (Microsoft Research Cambridge, UK)
Ian Soboroff (NIST, US)
Hideo Joho (Co-chair, University of Tsukuba, Japan)
Tetsuya Sakai (Co-chair, Microsoft Research Asia, PRC)NTCIR General Chairs:
Noriko Kando (NII, Japan)
Tsuneaki Kato (The University of Tokyo, Japan)
Douglas W. Oard (University of Maryland, USA)
Mark Sanderson (RMIT, Australia)