Dear NTCIR community,

As was announced during the NTCIR-8 Meeting, there will be some changes in thecentral coordination of the next NTCIR-9 based on careful reflection of the past NTCIRactivities. First of all, NII (National Institute of Informatics) and NICT (National Institute ofInformation and Communications Technology) will co-host NTCIR-9. The NTCIR-9
General Co-chairs will be Noriko Kando (NII, Japan), Tsuneaki Kato (University of Tokyo,Japan), and Eiichiro Sumita (NICT).

Second, NTCIR-9 will be more community-led than previous NTCIRs. Submitted
proposals will be assessed by the NTCIR-9 Task Selection Committee. The task
selection process will be chaired by the NTCIR-9 Evaluation Co-chairs, Tetsuya Sakai(Microsoft Research Asia, China) and Hideo Joho (University of Tsukuba, Japan). Toemphasize the new community-led structure, we are considering to change the name of
NTCIR to "NII/NICT Testbeds and Community for Information access Research"(without changing the acronym).

These changes, however, mean nothing unless the research and development inInformation Access are facilitated by the community. Therefore, we are looking for
proposals of evaluation tasks that can help advance our understanding of Information
Access processes, technologies, and systems. In NTCIR-9, we are looking for three
types of evaluation tasks: Core challenge task, Grand challenge task, and Pilot task.

Core

Challenge

Task: thisis

similar toprevious

NTCIRtasks. It isfor benchmarking and assessment of solutions to a particular Information Accessproblem within one or two NTCIR cycles. It also aims to establish the bestpractices to address each of the problems. NTCIR-8 had six tasks including IR forQA, Complex Cross-Language QA, Patent Machine Translation, Patent Mining,GeoTime Search, and Opinion analysis.

Grand Challenge Task: this task aims at more long-term goals than core challengetasks. Proposals should clearly demonstrate their potential impact on society. Also,
the technologies required to accomplish this task can be more complex than those1

for the core challenge tasks. In particular, this task should encouragecross-community collaboration and integration of multiple core technologies (e.g.,IR, QA, NLP, Visualization, ML, etc.).

Pilot Task: this type can be used when the evaluation task is potentially appealing
to a large community, but there are some uncertainties in the methodology, andthus, premature as a core or grand challenge task. In such a situation, you canapply for a pilot task to test the feasibility and obtain feedback from the community.
NTCIR-8 had a community QA pilot task.

The proposal will be carefully assessed by the NTCIR-9 Task Selection Committee
based on selection criteria given below. Please make sure to include sufficient
information for the Task Selection Committee to assess the proposals using thecriteria. A seed funding will be available for the accepted evaluation tasks to cover someof the expenses (data collection purchase, relevance assessments, etc.). Task
organizers are encouraged to apply for external funding whenever appropriate, to makethe resultant data publicly available, and to involve the participants or community in thedata creation process.

Coordinating an evaluation task is a great way to create a new research area andestablish an international community. We are looking for innovative and exciting ideas
for the evaluation tasks in Information Access! Please send your proposal in PDF format
to ntc-admin

by Midnight of 16 July 2010 (JST) with the subject: NTCIR-9

Task Proposal.

Best regards,Tetsuya Sakai and Hideo Joho
NTCIR-9 Evaluation Co-Chairs

Noriko Kando, Tsuneaki Kato, and Eiichiro Sumita
NTCIR-9 General Co-Chairs2

Important Dates


16 July 2010: Deadline for proposal

8 August 2010: Notification of the results

1 September 2010: Kick-off Event (Call for task participation)

Proposal Format


Main Part (A single A4 page)Task NameTask Type (Core or Grand challenge or Pilot)

Outline

Purpose

Methodology

Expected results

Appendix (Free format)
Name and contact information of the organizer(s)Expected participantsResources (document sets and others) to be used or constructedBudget planning

Schedule

Other notes

Selection Criteria


EssentialResponsibility: how do the organizers plan to advocate the significanceof the proposed task and commit to it?

Sustainability: how do the organizers plan to sustain the task activity
financially after the seed funding runs out?

Repeatability: Will the participants’ work be reproducible? Scientificmerits: what is the significance of potential research being conducted inthe task? What are the potential benefits to other work and relatedtechnologies?

Desirable
Social impact: what social problems/issues can the expected results of3the task address?
Synergy: How will the task fit in with other prospective tasks? Can thegrand challenge task leverage core challenge tasks, or can the core
challenge task contribute to a grand challenge task?

Generalisability: how would the results of the task generalize to othersituations and data?

Accountability: how does the task plan to ensure that participants
perform and report a component-level evaluation as well as end-to-endevaluation? Are there any potential components?

NTCIR-9 Task Selection CommitteeCharles Clarke (University of Waterloo, Canada)Fred Gey (UC Berkeley, USA)

Gareth Jones (Dublin City University, Ireland)

Noriko Kando (NII, Japan)Sung-Hyon Myaeng (KAIST, South Korea)

Stephen Robertson (Microsoft Research Cambridge, UK)Eiichiro Sumita (NICT, Japan)

4