Proposal for
NTCIR-11 Pilot task

QA Lab
Madoka Ishioroshi (NII)
Kelly Itakura
Noriko Kando (NII)
Yoshinobu Kano (PRESTO/NII)
Teruko Mitamura (CMU/NII)
Hideyuki Shibuki (Yokohama National University)
Advisor: Tatsunori Mori (Yokohama National University)
NTCIR-11 Kickoff Event
Sept. 2, 2013
Thanks Madoka for slides

Main Idea
• Challenge for Real World Problem of QA– University Entrance Exams - New Q-types not covered byexisting QA systems. Q is not a single sentence. Needcontext (ntcir-11), inference (ntcir-12).

• Challenge for a new structure of evaluation – Modulebased, building workable system as a joint effort of all
the participants– QA is very complicated– Hard for a research group to be good on everycomponents– Your module used and rated by other people• Continuous Plan-Run-Analysis-Improve Cycle

Background-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」
Document Retrieval

Extracting Answer
Candidates
Answers「634 Meters」

Answer Generation

Background-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」[factoid] People, GeoName,Organization, Numeric value, etc.
Document Retrieval[non-factoid] Definition, Reasons,Methods, etc.

Extracting Answer
Candidates
Answers「634 Meters」

Answer Generation- Extracting focus from
Question- Analyzing what is asked(Question Type)Background-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」
Document Retrieval

Extracting Answer
Candidates
Answers「634 Meters」

Answer Generation- Extracting focus from
Question- Analyzing what is asked(Question Type)
Search related documents (orpassage) using the extracted
focus of questions and Q typeBackground-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」
Document Retrieval

Search related documents (orpassage) using the extracted
focus of questions and Q type

Extracting Answer
Candidates

Extracting Answer Candidates
from the retrieved documentsor passages

Answers「634 Meters」- Extracting focus from
Question- Analyzing what is asked(Question Type)

Answer Generation

Background-1

Modules in a QA System
Q

Question Analysis「How tall the Sky-Tree in Tokyo?」- Extracting focus from
Question- Analyzing what is asked(Question Type)

Document Retrieval

Search related documents (orpassage) using the extracted
focus of questions and Q type

Extracting Answer
Candidates

Extracting Answer Candidates
from the retrieved documentsor passages

Answer Generation

Ranking the answer candidates
based on the scores of “answerlikeliness”

Answers「634 Meters」

Background-1

Modules in a QA SystemQ

Question Analysis「How tall the Sky-Tree in Tokyo?」
Document Retrieval

Extracting Answer
Candidates- Extracting focus fromQuestion- Analyzing what is asked(QuestionType)Inf used forscoring:・Verify the Q with the sentence
that
therelated
answerdocuments
candidate was
Search(orextracted,Textualpassage) using
theEntailment
extracted
・#of KWs
aroud theanswer
focusof questions
andQ type
candidate・Verify by the Q type and theExtracting
Answer Candidates
answercandidatefromthe retrieved・Context
where thedocuments
answerpassagescandidateorwas
extracted from

Answers「634 Meters」

Answer Generation

Ranking the answer candidates
based on the scores of “answerlikeliness”

Evaluation Method-1

Systems• QA-Platform– UIMA-based Module-base QA system– Follows: ACLIA modules• Q analysis• IR• Answer Extraction• Answer Generation– 2 baseline systems for Japanese• Javelin (CMU)• MinerVA (YNU)– 1 baseline systems for English (CMU)– Systems are usable either• local (by Kachaco) or on NII Server “bunbun”

UIMA Native UI for QA Platform to run a pipeline

Upload the Q fileSet various parameters to run

Specifies the index file for Indri (RS)Output directory

RunDocument directoryUI for the QA Platform to examine the results of eachmodule

Results of eachmodule

Show the representative results
for the first. To see moredetailed results by click on thebottun of “Expanding”

Click tojump tothemoduleAnswer
Candidates

Evaluation Method-2• Questions & Answers

Corpus– Japanese National Center for University Admissions Tests (multiplechoices)•  use as a Yes-No / TF Questions• XML, Japanese and English translation– Second Exam for the U of Tokyo and some others•  factoid & Complex Questions• Will be XML tagged• Knowledge sources provided– High school Textbooks, Wikipedia– Ontology of World History– Ontology-annotated Textbook• Tools– Wide-range of Language annotation tools– RITE resource and tools

Evaluation Method-3

Tasks
Multiple Choice (Center Exam)Japanese

EnglishNTCIR-11

World Hitory

NTCIR-12

World Hitory

English

2nd Exam

JapaneseJapanese

Factoid

Biology, Politics& Economics

Factoid

2nd Exam

English?

English?

Factoid

Complex

Factoid

ComplexEvaluate QA end-to-end with Every Possiblecombination of the modules submitted so far
and base systems

Tentative Schedule:

Multiple Opportunities to Run
Jan-14Feb-14Mar-13Apr-14May-14Jun-13

Jul-14Aug-14

Sep-14runs
1st Roundreturnresults
R-tableruns

2nd Roundreturnresults
R-tableruns

3rd Roundreturnresults
R-tableSubmitted all rounds are not mandatory.
* R-table = Round-table meetingParticipant’s merits• Can participate in one phase or multiple
phases (flexible participation)• Can evaluate with Baseline or with othersystems and to improve your system for thenext phase• Can see the effectiveness of your modules invarious pipelines created by other teams or
combination of other teams.

Join Us!