NTCIR-12 QA-Lab Task
second Pilot
Hideyuki Shibuki*1, Kotaro Sakamoto*1,*2,
Yoshionobu Kano*3,*4, †, Teruko Mitamura*5,
Madoka Ishioroshi*2, Tatsunori Mori*1, Noriko
Kando*2,*6*1: Yokohama National University, *2: National Institute ofInformatics, *3: Shizuoka University, *4: PRESTO, *5: Carnegie
Mellon University, *6: The Graduate University for Advanced
Studies (SOKENDAI)

2015-02-27

qalab rtm1

Challenges in QA Lab• Real-world Question Answering using university entrance
exam on World History• Questions are often multiple sentences• Context• Various Question-Format (multiple choices, Yes/No, Slot
Filling, Chronological Ordering, Essay, Combo)• Distribution of the Question Types is unknown and vary
year by year
• Some questions require inference• Explore the role of Semantic Processing• A module-based platform for advanced question answeringsystems• Multiple Phases (you can try more than one time)• Japanese and English
2015-02-27

qalab rtm

2

Question XML Format

2015-02-27

qalab rtm

3

Question XML Format
Questions (Multi-Sentence, SuggestContext)

2015-02-27

qalab rtm

4

Question XML Format 1(Center Exam)
Questions (Multi-Sentence, SuggestContext)

Context
Sub-Questions

Multiple Choices
2015-02-27

qalab rtm5

What’s New in NTCIR-12• Increasing Training Data

• 1 year (NTCIR-11)•  3 years (NTCIR-12) + some additions for specificquestion type•  Joint Effort to Construct Large Scale Quasi-Examples• By Extracting sentences from Textbook, Collecting
data from Exercise Books or related test, creating by
Human experts, etc.

• Training and 1 test by Question Types (New)• 1 Test use a whole questions in an examination(same as NTCIR-11)• Vertical Combinations of Different modules or QA
Systems (and explore more complex ways forbetter integration)
2015-02-27

qalab rtm

6

Question Types Proposed• 【A1:大論述（Complex Essay, CE）】
• 【A2:小論述(Simple Essay, SE)】• 【B1:Factoid（F）】• 【B2:穴埋め（Slot-Filing, SF）】
• 【C:真偽判定型(True-or-False, TF)】
• 【D:特殊（Unique, U）】
2015-02-27

qalab rtm7

Data and Resources• Questions and Answers• Training data – Three Years of Examinations (NationalCenter Exam, Secondary Exam for 5 Univ)• Test data – 2 Sets

• Knowledge Sources• High School Textbook (4 types from 2 publishers,Japanese Only)• Wikipedia• Participants can use any resources (and need to report)

2015-02-27

qalab rtm

8

Tools Provided• Two Baseline Systems (Japanese and English)• UIMA module-based end-to-end QA systems• One Passage Retrieval Module• to enhance the module-based collaboration.

2015-02-27

qalab rtm

9

Evaluation• Non-Essay• Scores set by National Center or Each University• Accuracy• Essay• Rouge• BE-based automatic quasi-pyramid• Pyramid using human extracted nuggets and nugget
voting (30 Complex and 50 Simple Essays)• Welcome! More discussion about the evaluation
methodologies and metrics
2015-02-27

qalab rtm

10

Proposed Schedule -1• April – June 2015
Discussion on task description, Question types
ontology, Evlauation Methodologies and metrics• End June 2015 Fixed task detailed task description

2015-02-27

qalab rtm

11

Proposed Schedule -2<Phase 1>• July 2015 1st runs
• Aug 2015 Results of the 1st runs return to the participants• Sept2015 Round-table discussion on the above results<Phase 2>• Dec 2015 2nd runs
• Jan 2016 Results of the 2nd runs return to the participants• Feb 2016 Round-table discussion on the above results<Towards Conference>• May 1: 2016 Final Paper submission• June 2016 NTCIR-12 Conference
2015-02-27

qalab rtm

12

Module Structure(Pipeline planned in NTCIR-11)

2015-02-27

qalab rtm

13

Question

Common Modules
Basic Element Analysis

Question Format Analysis…

Essay-Specifying-Words
Type Answering
Essay-No-Specifying
Type Answering

True-or-False TypeAnswering

Blank Type Answering

Factoid Type Answering

Dedicated Modules

Term Recognition

Answer gathering
2015-02-27

AnswerChronological Analysis…
Information Retrieval
call

Another pipeline example
qalab rtm

14

Join Us !!
Email: qalab-admin@nii.ac.jp

2015-02-27

qalab rtm

15