======================================================NTCIR-14 CALL FOR TASK PROPOSALS(Task Proposal Due: September 15, 2017)======================================================NTCIR (NII Testbeds and Community for Information access Research) isa sesquiannual series of evaluation conferences that mainly focuses on
Asian language information access.
The first NTCIR conference (NTCIR-1) took place in August/September
1999, and the latest NTCIR-13 conference launched in June 2016.
Research teams from all over the world participate in NTCIR tasks toadvance the state of the art and to learn from one another's experiences.
As the tasks being run at NTCIR-13(http://research.nii.ac.jp/ntcir/ntcir-13/index.html) are reaching thefinal stages, it is time to call for task proposals for the next NTCIR(NTCIR-14) which will start in December 2017 and conclude in June 2019.
Task proposals will be reviewed by the NTCIR program committee,
following the schedule below:
IMPORTANT DATES:
Sep 15, 2017 NTCIR-14 task proposals due
Nov 10, 2017 Acceptance Notification
Dec 5-8, 2017 NTCIR-13 conference,
where the accepted tasks can be introduced by the task organizers
NTCIR-14 Tentative Schedule:Mar 2018 NTCIR-14 kickoff
May 15, 2018 Task registration due*Jun 2018 Dataset release*
Jun-Jul 2018 Dry run*
Aug-Oct 2018 Formal run*Dec 20, 2018 Evaluation result release
Dec 20, 2018 Task overview paper release (draft)
Feb 1, 2019 Submission due of participant papers
Mar 1, 2019 Acceptance notification

Mar 20, 2019 Final submission due of all camera-ready papers(task overview papers / participant papers)
Jun 2019 NTCIR-14 Conference & EVIA 2019 in NII, Tokyo(Those with * can be changed by each task)
WHO SHOULD SUBMIT NTCIR-14 TASK PROPOSALS?
Organizers of existing NTCIR-13 tasks are required to submit a new proposal
if they wish to continue them in NTCIR-14.
We also welcome new task proposals within the broad research area ofinformation access.
To organize an evaluation task is to identify important research problems,
tackle them strategically by collaborating with other researchers(participants),
build the necessary evaluation framework to advance the state of the art,and make an impact to the research community and to the future.
We encourage applicants to emphasize real-world applications of theproposed task
by using real-world data, real tasks, and addressing real-world problems,and address a challenge in information access technology evaluationsuch as a large amount of assessments required for evaluation,
privacy-preserving use of proprietary data, and live test with real users.

PUBLICATION
We are planning to introduce a peer-review process for participants’ papers,
and publish participants’ papers and overview papers from LNCS at this round.
This trial aims at increasing the quality and visibility of NTCIR papers,and encouraging more participants to present their work at NTCIR.
Task organizers are expected to review participants’ paper and to makedecisions on their acceptance at NTCIR-14.
Moreover, some deadlines became earlier at this round(e.g. evaluation result release and task overview paper release).
We hope that this change brings more benefits to organizers andparticipants than burdens.

PROPOSAL TYPES:
We will accept two types of task proposals:- Proposal of a Core task:
This is for fostering researches on a particular information accessproblem by providing researchers with a common ground for evaluation.
New test collections and evaluation methods may be developed through thecollaboration between task organizers (proposers) and task participants.
At NTCIR-13, the core tasks were: Lifelog-2, MedWeb, OpenLiveQ,
QALab-3, and STC-2.
- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel
information access problem and there are uncertainties in task designingand organization. It may focus on a sub-problem of an information accessproblem and may attract a smaller group of participating teams than coretasks. However, it may grow into a core challenging task in the next
round of NTCIR. At NTCIR-13, the pilot tasks were: AKG, ECA, NAILS, and WWW.
Organizers are expected to run their task mainly with their own fundings
and to make the task as self-sustaining as possible. A part of the fundcan be supported by NTCIR, which we call "seed funding". It is usuallyused for some limited purposes such as hiring relevance assessors. The
amount of the seed funding allocated to each task varies depending on
requirements and the total number of accepted tasks, but typical cases
would be: around 1M JPY for a core task and around 0.5M JPY for a pilottask (note that the amount may be changed in NTCIR-14).
Please submit your task proposal as a pdf file via EasyChair bySeptember 15, 2017 (Anywhere on Earth).
https://easychair.org/conferences/?conf=ntcir14taskproposal

TASK PROPOSAL FORMAT:The proposal should not exceed five pages in A4 single-column format.
The first four pages should contain the main part and appendix, andthe last page should contain only description about the data used inthe task. Please describe the data with as much detail as possible sothat we can help your data release after the proposal is accepted. In
the past NTCIR, it took much time to create memorandums for data
release, which sometimes slowed down the task organization.
*Main part*- Task name and short name- Task type (core or pilot)- Abstract- Motivation- Methodology- Expected results*Appendix*- Names and contact information of the organizers- Prospective participants- Budget planning- Schedule- Other notes* Data (used in your task) *- Details(Please describe the details of the data, which should include thesource of the data, methods to collect the data, range of the data, etc.)- License(Please make sure that you have a license to distribute the data, anddetails of the license. If you have not had permission to release thedata yet, please
describe your plan to get the permission.)- Distribution(Please describe how you will distribute the data to participants.
There are mainly three choices:
distributed by the data provider, distributed by organizers, anddistributed by NII.)- Legal / Ethical issues(If the data can cause legal or ethical problems, please describe how
you address them.
e.g. some medical data may need approval from an ethical committee.
e.g. some Web data may need filtering for excluding discriminative messages. )REVIEW CRITERIA:- Importance of the task to the information access community and to the society- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)- Soundness of the evaluation methodology- Detailed description about data to be used
- Language scope
NTCIR GENERAL CHAIRS:Charles L. A. Clarke (University of Waterloo, Canada)
Noriko Kando (NII, Japan)
Best regards,
NTCIR-14 Program Committee Co-chairs,
Makoto P. Kato and Yiqun Liu