NTCIR16 QA Lab-PoliInfo-3
Organizers:
1.

Yasutomo Kimura, Otaru University of Commerce, Japan

2.

Hideyuki Shibuki, National Institute of Informatics, Japan

3.

Hokuto Ototake, Fukuoka University, Japan

4.

Yuzu Uchida, Hokkai-Gakuen University, Japan

5.

Keiichi Takamaru, Utsunomiya Kyowa University, Japan

6.

Madoka Ishioroshi, National Institute of Informatics, Japan

7.

Kazuma Kadowaki, The Japan Research Institute, Limited

8.

Masaharu Yoshioka, Hokkaido University, Japan

9.

Tomoyoshi Akiba, Toyohashi University of Technology, Japan

10.

Yasuhiro Ogawa, Nagoya University, Japan

11.

Minoru Sasaki, Ibaraki University, Japan

12.

Kenichi Yokote, Hitachi, JapankimurashibukiototakeuchidaKimuraShibukiOtotakeUchidatakamaru

ishioroshi

kadowaki

yoshioka

Takamaru

Ishioroshi

Kadowaki

Yoshioka

akiba

ogawa

sasakiYokote-san

Akiba

Ogawa

SasakiYokotearaki

sekinemorimitamura

Araki

SekineMori

Mitamura

Advisers:
1.

Tatsunori Mori, Yokohama National University, Japan

2.

Kenji Araki, Hokkaido University, Japan

3.

Satoshi Sekine, RIKEN, Japan

4.

Teruko Mitamura, Carnegie Mellon University, USA

NTCIR16 QA Lab PoliInfo-3
March 2021

NTCIR16 QA Lab-PoliInfo3● Background● Related work
● What is QA Lab PoliInfo-3 ?
● Sub tasks
1.

Question Answering

2.

QA Alignment

3.

Fact Verification

4.

Budget Argument Mining● Summaryhttps://poliinfo3.net/June 2022

Background●Fake news articles are easily propagated through various online
media platforms● Fake news detection and fact-checking have recently become anemerging research that is attracting tremendous attention● Fact-Checking, Fake News, Propaganda and Media Bias
○ [Preslav Nakov+ EMNLP2020 Tutorial]● Fact-checking is important to check the primary information source○ [Aizawa+ 2020]

Related work
CLEF-2020
Profiling Fake News
Spreaders on Twitter

CLEF-2020
CheckThat! Lab

NTCIR 15QA Lab-PoliInfo-2

DatasetTwitterTwitter
Political debate

Assembly minutes,
newsletter and Wikipedia

Task

Profiling Fake News
Spreaders on Twitter

Task1: Check-Worthiness on
tweetsTask2: Verified claim retrieval
Task3: Evidence retrieval
Task4: Claim verification
Task5: Check-Worthiness on
debates

Task1: Stance Classification
Task2: Dialog Summarization
Task3: Entity Linking
Task4: Topic Detection

Language

English and Spanish

English and ArabicJapaneseWhat is QA Lab-PoliInfo-3 ?
● We aim to resolve issues related to fake news detection and fact checking on
political information (minutes and newsletters)● In particular, we are conscious of accurately dealing with primary information

1. Question Answering
2. QA Alignment
3. Fact Verification
4. Budget Argument Mining

NTCIR14
QA Lab PoliInfo

NTCIR15QA Lab PoliInfo-2

NTCIR16
QA Lab PoliInfo-3

Summarization

DialogSummarization

Question Answering

Segmentation

Topic Detection
QA Alignment

Classification
Entity Linking

StanceClassification

Fact Verification

Budget Argument
MiningQuestion Answering

What is QA Lab-PoliInfo3 ?

Fact Verification

Diet or local assembly

SummaryNewsletter
Person asking a question

Person answering the question

Assembly minutes
質問

QA Alignment

Budget Argument Mining
答弁

Open dataTranscript(Assembly minutes)

Budget list

Question Answering
Purpose:▶

Question Answering aims to generate a summary of the corresponding answers given a summary ofthe questions in the Minutes.

▶

Questions of Q&A sessions in the newsletters (JSON format)▶

The minutes of the Tokyo Metropolitan Assembly (JSON format)

Input:

Output:▶

Short answer of the question in a few lines (JSON format)

Evaluation:▶

ROUGE (for automatic evaluation)▶

Quality questions (for human evaluation)

QA Alignment
Purpose:▶

QA Alignment aims to associate each question with an answer when given a question and answer inthe form of a batch question and answer.

▶

Assembly minutes, which includes a batch question and answer

Input:

Output:▶

Question and Answer pair

Evaluation:▶

Fact Verification
Purpose:▶

Fact Verification aims to determine when summaries and minutes are given, whether the contents ofthe summaries really exist in the minutes.

▶

Summary of remarks

Input:

Output:▶

Binary classification of true and false▶

Checking evidence

Evaluation:▶

Budget Argument Mining
Purpose:▶

Budget Argument Mining aims to extract the part of the meeting that is being discussed when a budget
item or amount is given, and determines the argument label (claim, premise, etc.) of the content.

▶
▶

Budget list
Assembly minutes

Input:

Output:▶
▶

Classification of argument labels
Linking documents

Evaluation:▶
▶

Accuracy
F-measureImportant dates
Mar. 24th 2021NTCIR-16 QALab-PoliInfo-3 Kickoff meeting

Mar. 29th 2021

NTCIR-16 Kickoff event

Apr. — 2021

First round table meeting

May — 2021

Second round table meeting

Jun. 15th 2021

Dataset Release*Aug. 10th – Nov. 12th 2021
Nov. 19th 2021

Dry Run*Deadline for resignation*

Nov. 22th -Nov. 30th 2021
Dec 6th -24 th 2021

Formal Run*

Human Evaluation*

Feb. 1st 2022

Evaluation Result ReleaseFeb. 1st 2022

Task over view paper release (draft)

Mar. 15th 2022

Submission deadline of participant papers (draft)

May 1st 2022

Camera-ready participant paper submission deadline

Jun. 10th – 13th 2022

NTCIR-16 Conference ＆ EVIA 2022 (in NII, Tokyo)

Question Answering

What is QA Lab-PoliInfo3 ?

Fact Verification

Diet or local assembly
Input : Question
Output : Answer
We consider QA summary in
the newsletter to be thecorrect answers.

Person asking a question

Person answering the question

Input : Summary of speech
Output :1. Binary classification of trueand false
2. Checking evidence

Assembly minutes
質問

QA Alignment

Budget Argument Mining
答弁

Input : Assembly minutes
Output : QA pair
We consider the ne to be thecorrect answers.

Transcript(Assembly minutes)

Input : Budget list andminutes
Output :1. classification of argumentlabel
2. Linking documents