CALL FOR NTCIR-14 ADDITIONAL PILOT TASK PROPOSALS(Task Proposals Due: May 20, 2018)
NTCIR (NII Testbeds and Community for Information access Research) isa sesquiannual series of evaluation conferences that mainly focuses on
Asian language information access.
The first NTCIR conference (NTCIR-1) took place in August/September
1999, and the latest NTCIR-13 conference was held in December 2017.
Research teams from all over the world participate in NTCIR tasks toadvance the state of the art and to learn from one another's experiences.

For NTCIR-14, six tasks have been already accepted and their activity started. The sixtasks include five core tasks (Lifelog-3, OpenLiveQ-2, QALab-PoliInfo, STC-3, andWWW-2) and a pilot task (CENTRE). Since there are still many emerging problems in
information access research, we now call for additional pilot tasks in NTCIR-14. The
additional pilot task proposals will be reviewed by the NTCIR-14 program committee,
following the schedule below:
IMPORTANT DATES:May 20, 2018

Task Proposals due

June 30, 2018

Acceptance Notification

NTCIR-14 General Schedule:Jun 2018

Dataset Release*

Jun-Jul 2018

Dry Run*Aug-Oct 2018

Formal Run*Feb 1, 2019

Evaluation Result ReleaseFeb 1, 2019

Task overview paper release (draft)

Mar 15, 2019

Submission due of participant papers

May 1, 2019

Camera-ready participant paper due

Jun 10-13, 2019 NTCIR-14 Conference & EVIA 2019 in NII, Tokyo(Those with * can be changed by each task)
Although the task registration due in the official schedule is June 2018, task organizers canset their own due as long as evaluation results will be released by Feb 1, 2019.

Please note that financial support (i.e., seed funding) will not be provided to the accepted
additional pilot task. Otherwise, the task will be treated in the same way as the otherpilot tasks in NTCIR-14, and in its proceedings. Please feel free to contact the program
co-chairs for any details of the task organization at ntc14-pcc@nii.ac.jp .
Please submit your task proposal as a pdf file via EasyChair by
May 20, 2018 (Anywhere on Earth).
https://easychair.org/conferences/?conf=ntcir14taskproposal

WHO SHOULD SUBMIT NTCIR-14 TASK PROPOSALS?
To organize an evaluation task is to identify important research problems,
tackle them strategically by collaborating with other researchers (participants),
build the necessary evaluation framework to advance the state of the art,and make an impact to the research community and to the future.
We encourage applicants to emphasize real-world applications of the
proposed task by using real-world data and addressing real-world tasks,
and address a challenge in information access technology evaluationsuch as a large amount of assessments required for evaluation,
privacy-preserving use of proprietary data, and live test with real users.

PUBLICATION
We introduce a peer-review process for participants’ papers,
and publish some of the participants’ papers and overview papers from LNCS at this round.
This trial aims at increasing the quality and visibility of NTCIR papers,
and encouraging more participants to present their work at NTCIR.
Task organizers are expected to review participants’ paper and to makedecisions on their acceptance at NTCIR-14.

TASK PROPOSAL FORMAT:The proposal should not exceed five pages in A4 single-column format.
The first four pages should contain the main part and appendix, andthe last page should contain only description about the data used inthe task. Please describe the data with as much detail as possible sothat we can help your data release after the proposal is accepted. In
the past NTCIR, it took much time to create memorandums for data
release, which sometimes slowed down the task organization.

*Main part*- Task name and short name- Abstract- Motivation- Methodology- Expected results*Appendix*- Names and contact information of the organizers- Prospective participants- Budget planning- Schedule- Other notes* Data (used in your task) *- Details(Please describe the details of the data, which should include thesource of the data, methods to collect the data, range of the data, etc.)- License(Please make sure that you have a license to distribute the data, anddetails of the license. If you have not had permission to release thedata yet, please
describe your plan to get the permission.)- Distribution(Please describe how you will distribute the data to participants.
There are mainly three choices:
distributed by the data provider, distributed by organizers, anddistributed by NII.)- Legal / Ethical issues(If the data can cause legal or ethical problems, please describe how
you address them.
e.g. some medical data may need approval from an ethical committee.
e.g. some Web data may need filtering for excluding discriminative messages. )REVIEW CRITERIA:- Importance of the task to the information access community and to the society- Timeliness of the task

- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)- Soundness of the evaluation methodology- Detailed description about data to be used
- Language scope
NTCIR GENERAL CHAIRS:Charles L. A. Clarke (University of Waterloo, Canada)
Noriko Kando (NII, Japan)
Best regards,
NTCIR-14 Program Committee Co-chairs,
Makoto P. Kato and Yiqun Liu