CALL FOR NTCIR-17 TASK PROPOSALS(Task Proposals Due: July 8, 2022)
NTCIR (NII Testbeds and Community for Information access Research) is a series ofevaluation conferences that mainly focus on information access with East Asian languages andEnglish. The first NTCIR conference (NTCIR-1) took place in August/September 1999, andthe latest NTCIR-16 conference will take place in June 2022. Research teams from all over
the world participate in one or more NTCIR tasks to advance the state of the art and to learn
from one another's experiences.
As the tasks being run at NTCIR-16 (http://research.nii.ac.jp/ntcir/ntcir-16/index.html) arereaching the final stages, it is time to call for task proposals for the next NTCIR (NTCIR-17)
which will start in September 2022 and conclude in December 2023. Task proposals will be
reviewed by the NTCIR Program Committee, following the schedule below:* IMPORTANT DATES:
July 8, 2022

Task Proposal Submission Due

August 26, 2022 Acceptance Notification of Task Proposals* SUBMISSION LINK:
https://easychair.org/my/conference?conf=ntcir17taskproposal* NTCIR-17 TENTATIVE SCHEDULE:January 2023:
January-June 2023:

Dataset Release*
Dry Run*March 2023-July 2023:

Formal Run*

August 1, 2023:

Evaluation Result Release

August 1, 2023:

Draft Task Overview Paper Release

September 1, 2023:

Draft Participant Paper Submission Due

November 1, 2023:

All Camera-ready Paper Submission Due

December 2023:

NTCIR-17 Conference in NII, Tokyo, Japan(* indicates that the schedule can be different for different tasks)* WHO SHOULD SUBMIT NTCIR-17 TASK PROPOSALS?
Organizers of existing NTCIR-16 tasks are required to submit a new proposal if they wish tocontinue them in NTCIR-17. We also welcome new task proposals within the broad researcharea of information access.

To organize an evaluation task is to identify important research problems, tackle themstrategically by collaborating with other researchers (both co-organizers and participants),
build the necessary evaluation framework to advance the state of the art, and make an impact
on the research community and to the future.
We encourage applicants to emphasize real-world applications of the proposed task by using
real-world data, real tasks, and addressing real-world problems, and address a challenge in
information access technology evaluation such as a large number of assessments required forevaluation, privacy-preserving use of proprietary data, and live test with real users.

* PROPOSAL TYPES:
We will accept two types of task proposals:- Proposal of a Core task:
This is for fostering research on a particular information access problem by providing
researchers with a common ground for evaluation. New test collections and evaluation
methods may be developed through the collaboration between task organizers (proposers)
and task participants. At NTCIR-16, the core tasks are Data Search 2, DialEval-2, FinNum3,QA

Lab-PoliInfo-3,WWW-4,

andLifelog-4(Detailscanbefoundat

http://research.nii.ac.jp/ntcir/ntcir-16/tasks.html).
- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel information accessproblem and there are uncertainties either in task designing or organization. It may focus ona sub-problem of an information access problem and may attract a smaller group ofparticipating teams than core tasks. However, it may grow into a core challenging task in thenext round of NTCIR. At NTCIR-16, the pilot tasks are RCIR, Real-MedNLP, SS, andULTRE (Details can be found at http://research.nii.ac.jp/ntcir/ntcir-16/tasks.html).
Organizers are expected to run their tasks mainly with their own funding and to make the task
as self-sustaining as possible. A part of the fund can be supported by NTCIR, which is called"seed funding". It is usually used for some limited purposes such as hiring relevance assessors.
The amount of seed funding allocated to each task varies depending on requirements and thetotal number of accepted tasks. Typical cases would be around 1M JPY for a core task andaround 0.5M JPY for a pilot task (note that the amount is subject to change).
Please submit your task proposal as a pdf file via EasyChair by July 8, 2022 (Anywhere onEarth).
https://easychair.org/my/conference?conf=ntcir17taskproposal* TASK PROPOSAL FORMAT:The proposal should not exceed six pages in A4 single-column format. The first five pages
should contain the main part and appendix, and the last page should contain only a descriptionof the data to be used in the task. Please describe the data in as much detail as possible so that
we can help your data release process after the proposal is accepted. In the past NTCIRs, it
took much time to create memorandums for data release, which sometimes slowed down thetask organization.
Main part- Task name and short name- Task type (core or pilot)- Abstract- Motivation- Methodology- Expected results
Appendix- Names and contact information of the organizers- Prospective participants- Data to be used and/or constructed- Budget planning- Schedule- Other notes
Data (to be used in your task)- Details(Please describe the details of the data, which should include the source of the data, methods
to collect the data, range of the data, etc.)- License(Please make sure that you have a license to distribute the data, and details of the licenseshould be provided. If you do not have permission to release the data yet, please describe yourplan to get the permission.)- Distribution(Please describe how you plan to distribute the data to participants. There are mainly threechoices: distributed by the data provider, distributed by organizers, and distributed by NII.)- Legal / Ethical issues(If the data can cause legal or ethical problems, please describe how you propose to address
them. e.g. some medical data may need approval from an ethical committee. e.g. some Webdata may need filtering for excluding discriminative messages.)* REVIEW CRITERIA:- Importance of the task to the information access community and to the society- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)- Soundness of the evaluation methodology- Detailed description about the data to be used
- Language scope* NTCIR-17 PROGRAM Co-Chairs
Takehiro Yamamoto (Co-chair, University of Hyogo, Japan)
Zhicheng Dou (Co-chair, Renmin University of China)* NTCIR-17 GENERAL CHAIRS:Charles Clarke (University of Waterloo, Canada)
Noriko Kando (National Institute of Informatics, Japan)
Makoto P. Kato (University of Tsukuba, Japan)
Yiqun Liu (Tsinghua University, China)---------------------------------------------------