CALL FOR PARTICIPATIONNTCIR-8 Community QA Pilot Task
http://research.nii.ac.jp/ntcir/ntcir-ws8/yahoo/index-en.html
We are very pleased to announce that NTCIR-8 plans a pilot task using a Community QA
dataset from Yahoo! Japan.
The importance of the Community QA like Yahoo! Answers has been increased in ourordinary life. Wide variety of questions are asked and answered. It also provides a good
opportunity to investigate the real users' (a kind of) relevance judgments for the "bestanswers" in large-scale.
We are fortunately provided a large-scale Community QA dataset from Yahoo! Japan forresearch purpose. It consists of about 100,000,000 items (about 100 GB). Then we plan apilot task using the dataset as following;
1. Task:A. Main Task: Best Answer Estimation
B. Sub Task: Question Type Classification
2. Data set:
100,000,000 items from Yahoo! Japan's "Yahoo! Chiebukuro"(http://chiebukuro.yahoo.co.jp/), a Community-based QA site which is equivalent toYahoo! Answers. It consists of about 26,000,000 questions and 76,000,000 answers tothe questions, about 100 GB in total. All items are written in Japanese.
Main task:- Training data will be more than 99% of the above including the best answer
specification.
- Test data will be several thousand questions and their answers discarded the bestanswer specification.
Subtask:- In addition to the above, set of questions/answers which manually annotatedquestion types for training and for evaluating the test results.

3. Task Sketch
Main Task:
All the participants are expected to submit the result to Main Task. In the test, we will
use the "Best answers" selected by the askers.
Subtask (optional):
As Community QA has the wide variety of question types from simple factoid/navigate
questions which often input to the ordinary web search engines to those asking theadvises or opinions from the community.
To improve the effectiveness of the best answer estimation and investigating users'intentions, question types is one of the clues. Then we are interested in to test theautomatic classification of the question types, and see the effectiveness of the question
type classification in the best answer estimation. We will propose a question typeclassification scheme, and provide a training set and the answers for the test set whichcontaining manually annotated question types. The participants can propose their own
classification scheme other than one provided by NTCIR, do the question type
annotation, and test the effectiveness of those classification in the best answer typeestimation over the runs without question types and/or those with the other set ofquestion types including the one which NTCIR will provide.
4. Task Organizers and Contact Information:
Daisuke Ishikawa and Noriko Kando (NII)

Email: ntcadm-yahoo (at) nii.ac.jp

The task design is still rough, and you are welcome to join the discussion on the task
design and evaluation methodology. Please consider to participate in this new
exciting task.
5. How to Participate:
To join the discussion mailing list, please email to ntcadm-yahoo (at) nii.ac.jp.
To register as a participant, please read the "How to participate" at;http://research.nii.ac.jp/ntcir/ntcir-ws8/howto-en.html,
then register at; http://research.nii.ac.jp/ntcir/ntcir-ws8/ntcir8regist-e.html by selecting"pilot task" at the bottom of the form and describe "Community QA Task" in the box forthe system description.
You are most welcome to join us!