=======================================CALL FOR NTCIR-15 TASK PROPOSALS(Task Proposals Due: June 6, 2019)=======================================NTCIR (NII Testbeds and Community for Information access Research) is a series ofevaluation conferences that mainly focuses on information access with East Asian languages
and English. The first NTCIR conference (NTCIR-1) took place in August/September 1999,
and the latest NTCIR-14 conference will take place in June 2019. Research teams from allover the world participate in one or more of the NTCIR tasks to advance the state of the art
and to learn from one another's experiences.
As the tasks being run at NTCIR-14 (http://research.nii.ac.jp/ntcir/ntcir-14/index.html) arereaching the final stages, it is time to call for task proposals for the next NTCIR (NTCIR-15)
which will start in June 2019 and conclude in December 2020. Task proposals will be reviewed
by the NTCIR Program Committee, following the schedule below:** IMPORTANT DATES: **June 6, 2019

Task Proposals due*(*Extended*)*June 10-13, 2019 *Task Planning Discussions at the NTCIR-14 conference 9:00-12:30,Monday, June 10, 2019, and/or 13:00-17:00, Thursday, June 13, 2019
Based on your request, rooms will be prepared for your task during the above timeslots. The
first-come-first-served basis.
June 30, 2016

Acceptance Notification** NTCIR-15 Tentative Schedule:*
Dec 2019: Task Registration Due
Jan 2020: Dataset Release
Jan-May 2020: Dry Run
Mar-July 2020: Formal Run
Aug 2020: Evaluation Result Release
Dec 2020: NTCIR-15 Conference & EVIA 2020 in NII, Tokyo, Japan
March 2021: Publication of Post-conference Proceedings** WHO SHOULD SUBMIT NTCIR-15 TASK PROPOSALS?*Organizers of existing NTCIR-14 tasks are required to submit a new proposal if they wish tocontinue them in NTCIR-15. We also welcome new task proposals within the broad researcharea of information access.
To organize an evaluation task is to identify important research problems, tackle themstrategically by collaborating with other researchers (participants), build the necessary
evaluation framework to advance the state of the art, and make an impact to the researchcommunity and to the future.
We encourage applicants to emphasize real-world applications of the proposed task by using
real-world data, real tasks, and addressing real-world problems, and address a challenge in
information access technology evaluation such as a large amount of assessments required forevaluation, privacy-preserving use of proprietary data, and live test with real users.
** PUBLICATION*
We will use a peer-review process for participants’ papers and publish participants’ papers
and overview papers both in the form of conference notes and also post-conference LNCS
proceedings. Task organizers are expected to review participants’ paper and to make decisions
on their acceptance at the post-conference proceedings.
** PROPOSAL TYPES:*
We will accept two types of task proposals:- Proposal of a Core task:
This is for fostering researches on a particular information access problem by providing
researchers with a common ground for evaluation. New test collections and evaluation
methods may be developed through the collaboration between task organizers (proposers)
and task participants. At NTCIR-14, the core tasks were: Lifelog-3, OpenLiveQ-2, QALabPoliInfo, STC-3 and WWW-2.
- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel information accessproblem and there are uncertainties in task designing and organization. It may focus on a subproblem of an information access problem and may attract a smaller group of participatingteams than core tasks. However, it may grow into a core challenging task in the next round ofNTCIR. At NTCIR-14, the pilot tasks were: CENTRE, and FinNum.
Organizers are expected to run their task mainly with their own fundings and to make the task
as self-sustaining as possible. A part of the fund can be supported by NTCIR, which we call"seed funding". It is usually used for some limited purposes such as hiring relevance assessors.

The amount of the seed funding allocated to each task varies depending on requirements andthe total number of accepted tasks, but typical cases would be: around 1M JPY for a core task
and around .5M JPY for a pilot task (note that the amount may be changed in NTCIR-15).
Please submit your task proposal as a pdf file via EasyChair by June 6, 2019 (Anywhere onEarth).
https://easychair.org/conferences/?conf=ntcir15taskproposal** TASK PROPOSAL FORMAT: *The proposal should not exceed six pages in A4 single-column format. The first five pages
should contain the main part and appendix, and the last page should contain only description
about the data used in the task. Please describe the data with as much detail as possible sothat we can help your data release after the proposal is accepted. In the past NTCIR, it tookmuch time to create memorandums for data release, which sometimes slowed down the taskorganization.
Main part- Task name and short name- Task type (core or pilot)- Abstract- Motivation- Methodology- Expected results
Appendix- Names and contact information of the organizers- Prospective participants- Data to be used and/or constructed- Budget planning- Schedule- Request for mentoring (Include this section if you would like to apply for the mentoring
program; otherwise, exclude this section)- Other notes
Data (used in your task)- Details(Please describe the details of the data, which should include the source of the data, methods

to collect the data, range of the data, etc.)- License(Please make sure that you have a license to distribute the data, and details of the license. If
you have not had permission to release the data yet, please describe your plan to get thepermission.)- Distribution(Please describe how you will distribute the data to participants. There are mainly threechoices: distributed by the data provider, distributed by organizers, and distributed by NII.)- Legal / Ethical issues(If the data can cause legal or ethical problems, please describe how you address them. e.g.
some medical data may need approval from an ethical committee. e.g. some Web data mayneed filtering for excluding discriminative messages.)** REVIEW CRITERIA: *- Importance of the task to the information access community and to the society- Timelinessof the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)- Soundness of the evaluation methodology- Detailed description about data to be used
- Language scope** NTCIR-15 PROGRAM COMMITTEE*
Leif Azzopardi, University of Strathclyde
Ben Carterette, University of Delaware
Hsin-Hsi Chen, National Taiwan University
Tat-Seng Chua, National University of Singapore
Maarten de Rijke, University of Amsterdam
Nicola Ferro, University of Padova
Makoto P. Kato, University of Tsukuba
Gareth Jones, Dublin City University
Diane Kelly, University of Tennessee
Douglas Oard, University of Maryland
Tetsuya Sakai, Waseda University
Mark Sanderson, RMIT University
Ian Soboroff, NISTNoriko Kando (Co-chair, NII, Japan)
Yiqun Liu (Co-chair, Tsinghua University, China)** NTCIR-15 GENERAL CHAIRS: *Charles Clarke (University of Waterloo, Canada)
Noriko Kando (Co-chair, NII, Japan)------------------------------