=====================================================CALL FOR NTCIR-13 TASK PROPOSALS(Task Proposal Due: April 28, 2016)=====================================================NTCIR (NII Testbeds and Community for Information access Research) is asesquiannual series of evaluation conferences that mainly focuses on Asian language
information access.
The first NTCIR conference (NTCIR-1) took place in August/September 1999, and thelatest NTCIR-12 conference will take place in June 2016. Research teams from all over
the world participate in one or more of the NTCIR tasks to advance the state of the art
and to learn from one another's experiences.
As the tasks being run at NTCIR-12 (http://research.nii.ac.jp/ntcir/ntcir-12/index.html)
are reaching the final stages, it is time to call for task proposals for the next NTCIR(NTCIR-13) which will start in June 2016 and conclude in December 2017. Task
proposals will be reviewed by the NTCIR Program Committee, following the schedule
below:
IMPORTANT DATES:
Apr 28, 2016

Task Proposals due

May 27, 2016

Acceptance Notification

June 7-10, 2016 Task Planning Sessions and Presentation at the NTCIR-12
conference
NTCIR-13 Tentative Schedule:Dec 2016:

Task Registration Due

Jan 2017:

Dataset Release

Jan-May 2017:

Dry RunMar-July 2017: Formal Run
Aug 2017:

Evaluation Result Release

Dec 2017:

NTCIR-13 Conference & EVIA 2017 in NII, Tokyo, Japan

Organizers

ofexisting

NTCIR-12tasks(IMine,MedNLPDoc,MobileClick,SpokenQuery&Doc, Temporalia, MathIR, Lifelog, QALab, and STC) are required tosubmit a new proposal if they wish to continue them in NTCIR-13. We also welcomenew task proposals within the broad research area of information access. New taskorganizers will be given many opportunities to consult and learn from senior PC
members for task design and organization (see MENTORING). To organize anevaluation task is to identify important research problems, tackle them strategically bycollaborating with other researchers (participants), build the necessary evaluation
framework to advance the state of the art, and make an impact to the researchcommunity and to the future.
PROPOSAL TYPES:
We will accept two types of task proposals:- Proposal of a Core task:
This is for fostering researches on a particular information access problem by providing
researchers with a common ground for evaluation. New test collections and evaluation
methods may be developed through the collaboration between task organizers(proposers) and task participants. At NTCIR-12, the core tasks were: IMine,MedNLPDoc, MobileClick, SpokenQuery&Doc, Temporalia, and MathIR.
- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel information accessproblem and there are uncertainties in task designing and organization. It may focus ona sub-problem of an information access problem and may attract a smaller group ofparticipating teams than core tasks. However, it may grow into a core challenging task
in the next round of NTCIR. At NTCIR-12, the pilot tasks were: Lifelog, QALab, andSTC.
Organizers are expected to run their task mainly with their own fundings and to makethe task as self-sustaining as possible. A part of the fund can be supported by NTCIR,
which we call "seed funding". It is usually used for some limited purposes such as hiring
relevance assessors. The amount of the seed funding allocated to each task varies
depending on requirements and the total number of accepted tasks, but typical cases
would be: around 1M JPY for a core task and around 0.5M JPY for a pilot task (notethat the amount may be changed in NTCIR-13).
Please submit your task proposal as a pdf file to “NTCIR-13 Task Proposal” track in thefollowing EasyChair site by April 28, 2016 (UTC-12).
https://easychair.org/conferences/?conf=ntcir12

MENTORING:
We offer a mentoring program especially for those who propose a task for the first time.
If you would like help in your task organization, you may ask for a mentor: a person whowill help you organize a task usually via e-mail. A mentor can also familiarize you withthe standards and schedules of NTCIR. He or she will also help you understand how todesign tasks, test collections, and evaluation methodology. Mentors are volunteers who
are familiar with successful proposals. To request a mentor, please include a “Requestfor mentoring” section in your proposal, which must include problems for which youneed support (e.g. scheduling, design of task, evaluation methodology, etc.). Since thisprogram just begins from NTCIR-13 and is still at a trial stage, we may not fully meetyour requests.

DIFFERENCE FROM THE PAST NTCIR TASK PROPOSAL:- We have a plan to send acceptance notification before the NTCIR-12 conference, unlike
the previous rounds where it was sent after the conferences.
- Organizers at NTCIR-12 may keep the task name (e.g. TASKNAME-2) as long as thetask includes improvement based on the previous rounds.
- Please describe the data with as much detail as possible so that we can help your data
release after the proposal is accepted. In the past NTCIR, it took much time to create
memorandums, which sometimes slowed down the task organization.
- We offer a mentoring program.
- The proposal should not exceed four pages in A4 single-column format.
- We encourage applicants to- Emphasize real-world applications of the proposed task by using real-world data,real tasks, and addressing real-world problems.
- Address a challenge in information access technology evaluation such as a large
amount of assessments required for evaluation, privacy-preserving use of proprietarydata, and live test with real users.

TASK PROPOSAL FORMAT:The proposal should not exceed four pages in A4 single-column format.
Main part- Task name and short name- Task type (core or pilot)

- Abstract- Motivation- Methodology- Expected results
Appendix- Names and contact information of the organizers- Prospective participants- Data to be used and/or constructed- Budget planning- Schedule- Request for mentoring (Include this section if you would like to apply for thementoring program; otherwise, exclude this section)- Other notes

REVIEW CRITERIA:- Importance of the task to the information access community and to the society- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)- Soundness of the evaluation methodology- Language scope

NTCIR-13 PROGRAM COMMITTEE*The following list is tentative. The members may be changed.
Hsin-Hsi Chen (National Taiwan University, Taiwan)
Nicola Ferro (University of Padua, Italy)
Kalervo Jarvelin (University of Tampere, Finland)
Gareth Jones (Dublin City University, Ireland)
Gary Geunbae Lee (POSTECH, South Korea)
Mandar Mitra (Indian Statistical Institute, India)
Maarten de Rijke (University of Amsterdam, The Netherlands)
Ian Soboroff (NIST, United States)
Yiqun Liu (Co-chair, Tsinghua University, China)
Makoto P. Kato (Co-chair, Kyoto University, Japan)
NTCIR GENERAL CHAIRS:Charles Clarke (University of Waterloo, Canada)

Noriko Kando (NII, Japan)
Tetsuya Sakai (Waseda University, Japan)===