QA Lab-3:QA Lab for Entrance ExamK O TA R O S A K A M O T O * 1 , * 2 , * 3 , H I D E Y U K I S H I B U K I * 1 , M A D O K A I S H I O R O S H I * 2 , A K I R A F U J I TA * 2 , T E R U K O
M I TA M U R A * 3 , N O R I K O K A N D O * 2 , * 4 , TAT S U N O R I M O R I * 1 , Y O S H I N O B U K A N O * 5* 1 : Y O K O H A M A N AT I O N A L U N I V E R S I T Y, * 2 : N AT I O N A L I N S T I T U T E O F I N F O R M AT I C S , * 3 : C A R N E G I E M E L L O NU N I V E R S I T Y, * 4 : T H E G R A D U AT E U N I V E R S I T Y F O R A D V A N C E D S T U D I E S ( S O K E N D A I ) , * 5 : S H I Z U O K A U N I V E R S I T YGoalUsing Japanese university entrance exams and their English
translations on the subject of “World history”
Investigation of the real-world complex QA technologies◦ Question Analysis◦ Information Retrieval◦ Information Extraction◦ Automatic Summarization

Overview of QA Lab-2Tasks◦ National Center Test (multiple-choice type, EN & JA)◦ Second-stage examinations of 5 universities (free-description type, EN & JA)◦ Benesse mock exam (multiple-choice type, JA)◦ Yozemi mock exam (multiple-choice type, JA)◦ Sundai mock exam (free-description type, JA)

Multiple-choice type questions◦ 12 teams participated◦ Good results

Free-description type questions◦ 3 teams participated

Named-Entity questions
 Similar to factoid QA
 Relatively good results

Free to participate
either of exams

Focus on essayquestions at QA Lab-3

Essay questions
 Complex and laborious task
 Similar to query-biased multi-document summarization
 Not good results

QA Lab-3 tasks
Multiple-Choice Question Task
Named-Entity Question Task
separateｘ English / Japanese

Essay Question Task
◦ End-to-End subtask◦ Extraction subtask◦ IN: Question OUT: Passages including texts of Gold Standard Data◦ Summarization subtask◦ IN: Question + Passages OUT: Essay◦ Evaluation Method subtask◦ IN: Question + Essays + Gold Standard Data OUT: Ranking of Essays (with score)

Research run
How much did the QA technologies improved from QA Lab-1?
◦ Using the same training/test sets as the past QA Lab runs, comparison withthe past results◦ Using the systems participating in the past QA Lab runs, comparison with thepresent systems

What’s new in NTCIR-13
Restructuring tasks based on question types◦ Multiple-Choice Question Task
◦ Named-Entity Question Task
◦ Essay Question Task

Breaking down Essay Question Task into manageable subtasks◦ Extraction subtask and Summarization subtask in order to be highly
participative◦ Evaluation method subtask for automated evaluation of essays

Research run for the progress so far

Challenges in QA Lab-3
1) essay questions that require logical summaries along a historical theme2) competition with more than 3,500 students, examinees, from all over Japan (JA only)
3) questions with context4) answer by text as high-compress-ratio query-biased summarization
5) advanced entity-focused passage retrieval6) enhance knowledge resources
7) semantic representation and sophisticated learning8) appropriate evaluation measure for essay9) research run using the past QA Lab data/systems

Question and Answer Data
Training Data
◦ 8 years of National Center Test (JA and EN)◦ 5 years of Second-stage Examinations for 5 univs (JA and EN)◦ 11 years of Essay questions of 2 univs’ entrance exams (JA)◦ Nugget data for 11 years of Essay questions of 2 univs’ entrance exams (JA)Test Data◦ 2 sets (JA and EN)

From
QA Lab-1and -2
training andtest data

Knowledge Sources
Four Japanese high school textbooks◦ Available in Japanese

A snapshot of Wikipedia◦ Available in English and JapaneseWorld History Ontology◦ Available in JapaneseParticipants are free to use any resourcesToolsTwo Baseline Systems (Japanese and English)◦ UIMA module-based end-to-end QA systems

One Passage Retrieval Module◦ to enhance the module-based collaboration.

Scorer for National Center Test
Format Checker for National Center Test and Second-stage Examination

Proposed Schedule (1/2)
Aug 2016

First version of task description released

Sep 2016

Corpus, Training set, Baseline systems will be released to the participants

Aug – Sep 2016 Discussion on task description
End Sep 2016 Fixed task detailed task description
Oct 2016only)

1st Competitive runs with more than 3,500 high-school students, examinees, from all over Japan (JA<Phase 1>
Feb 2017

1st runs (End-to-End, Extraction, Summarization, Evaluation method)

Mar 2017

Results of the 1st runs return to the participantsApr 2017

Round-table discussion on the above results

Proposed Schedule (2/2)<Phase 2>
Jun 2017

2nd runs (End-to-End, Extraction, Summarization, Evaluation method)

Jul 2017

Results of the 2nd runs return to the participants<Research Run>Jul 2017

Research ｒuns

Aug 2017

Round-table discussion on the above results<Towards Conference>Oct 2017

Draft paper submission

Oct 2017only)

2nd Competitive runs with more than 3,500 high-school students, examinees, from all over Japan (JANov 1: 2017

Camera-ready Paper submission

Dec 2017

NTCIR-13 ConferenceMailing List for Discussion
Englishqalab@nii.ac.jp
Japaneseqalab-ja@nii.ac.jp

Join Us !!!
Email: qalab-admin@nii.ac.jp