NTCIR-17 QA Lab-PoliInfo-4
Organizers:

1. Yasuhiro Ogawa, Nagoya University, Japan
2. Yasutomo Kimura, Otaru University of Commerce, Japan
3. Hideyuki Shibuki, Besna Institute Inc.

kimurashibukiototakeuchida

4. Hokuto Ototake, Fukuoka University, Japan

Ogawa

KimuraShibukiOtotake

7. Madoka Ishioroshi, National Institute of Informatics, Japan

takamaru

ishioroshi

kadowaki

8. Kazuma Kadowaki, The Japan Research Institute, Limited

Kobayashi

UchidaTakamaru

Ishioroshi

Kadowaki

Akiba

Sasaki

Kobayashi

Yoshiokaaraki

sekineMoriAraki

5. Yuzu Uchida, Hokkai-Gakuen University, Japan
6. Keiichi Takamaru, Utsunomiya Kyowa University, Japan

9. Tomoyoshi Akiba, Toyohashi University of Technology, Japan
10. Minoru Sasaki, Ibaraki University, Japan
11. Akio Kobayashi, National Agriculture and Food Research
Organization, JapanAdvisers:

akiba

1. Masaharu Yoshioka, Hokkaido University, Japan
2. Tatsunori Mori, Yokohama National University, Japan
3. Kenji Araki, Hokkaido University, Japan
4. Satoshi Sekine, RIKEN, Japan
5. Teruko Mitamura, Carnegie Mellon University, USA

mitamura
SekineMitamuraNTCIR-17 QA Lab-PoliInfo-4● Background● Related work
● What is QA Lab PoliInfo-4 ?
● Sub tasks
1.

Question Answering-22.

Answer Verification

3.

Stance Classification-2

4.

Budget Argument Mining-2 Minutes-to-Budget Linking● Summary

Background● Fake news articles are easily propagated through various online
media platforms● In political news, the one reason is the complexity of politicalresources● To deal with such fake news, we have addressed these tasks fromNTCIR-14 to NTCIR-16.

Related Work
CLEF-2020
Profiling Fake News
Spreaders on Twitter

CLEF-2022
CheckThat! Lab

NTCIR 16
QA Lab-PoliInfo-3

DatasetTwitterTwitterNews articles

Assembly minutes,
newsletter

Task

Profiling Fake News
Spreaders on Twitter

Task 1 - Check-Worthiness
Estimation
Task 2 - Verified Claim Retrieval
Task 3 - Fake News Detection

Task 1: Question Answering
Task 2: QA Alignment
Task 3: Fact Verification
Task 4: Budget Argument Mining

Language

English and Spanish

Arabic, Bulgarian, Dutch, English,
German, Spanish, and Turkish

JapaneseWhat is QA Lab-PoliInfo-4 ?
● We aim to resolve issues related to fake news detection and fact checking on
political information (minutes and newsletters)● In particular, we are conscious of accurately dealing with primary information

1. Question Answering-2 (QA2)
2. Answer Verification (AV)
3. Stance Classification-2 (SC2)
4. Budget Argument Mining-2 (BAM2) Minutes-to-Budget Linking (MBLink)

NTCIR14
QA Lab PoliInfo

NTCIR15QA Lab PoliInfo-2

NTCIR16
QA Lab PoliInfo-3

NTCIR17QA Lab PoliInfo-4

Summarization

DialogSummarization

Question Answering

Question
Answering-2

QA Alignment

Answer Verification

Fact Verification

StanceClassification-2

Budget Argument
MiningMinutes-to-Budget
Linking

Segmentation

Topic Detection

Classification
Entity Linking
StanceClassification

Question Answering-2

Newsletter

What is QA Lab-PoliInfo-4 ?
Diet or local assembly

Person asking a question

Person answering the question

Stance Classification-2

Topic &Opinion
sentences

Assembly minutes
質問

check
Answer Verification
check

Fake answer

Assembly minutes(Transcript)

Minutes-to-Budget Linking

答弁

Budget table

Question Answering-2 (QA2)
Purpose:▶ The second question answering (QA2) subtask is the same as the question answering (QA)
subtask at NTCIR-16 QA Lab-PoliInfo-3.

▶ Collaborate with the answer verification (AV) subtask at NTCIR-17 QA Lab-PoliInfo-4.
Input:▶ Questions of Q&A sessions in the newsletters▶ The minutes of the Tokyo Metropolitan Assembly

Output:▶ Short answer of the question in a few lines

Evaluation:▶ ROUGE (for automatic evaluation)▶ Quality questions (for human evaluation)

Answer Verification (AV)Purpose:▶ Answer Verification aims to check whether the results of QA is correct.

Two Stages:
1.

Expanding training data including fake answers
 Participants tackle to make fake answers or seemingly wrong answers thatbaseline classifiers will misjudge.
2.

Building binary classifiers that can judge the test data set correctly These classifiers are used as new baseline classifiers in the 1st stage

AV 1st Stage: Fake-answer Generation
Purpose:▶ The 1st stage of AV aims to generate answers difficult for classifiers to judge▶ This idea is from GAN (Generative Adversarial Networks)

Input:▶ Sets of a question and its answer in Q&A sessions in Newsletters of the Tokyo Metropolitan
Assembly▶ The minutes of the Tokyo Metropolitan Assembly

Output:▶ Fake answers of the question in a few lines▶ Correct but seemingly wrong answer the question in a few lines

Evaluation:▶ Decrease rate of the accuracy of estimated results by fact checking baseline system

AV 2nd Stage: Fact-checking Classification
Purpose:▶ The 2nd stage of AV aims to judge whether the answer is fake

Input:▶ Questions of Q&A sessions in the newsletters▶ List of correct or fake answers of the question in a few lines▶ The minutes of the Tokyo Metropolitan Assembly

Output:▶ List of “True” or “False” (binary) (“True” indicates that the answer is not fake.)

Evaluation:▶ Accuracy

Stance Classification-2 (SC2)
Purpose:▶ The stance classification-2 aims to classify the stance of a political party▶ It expands target local governments (Tokyo, Sapporo… ) and is more difficult than

NTCIR15 stance classification-1.

Input:▶ A topic

Stance Classification▶ An assembly member's opinion sentences on the given topic
without explicit statement of neither approval nor disapproval.
Output:▶ Relevance (Binary) : Either approval or disapproval on a given topic
Evaluation:▶ Accuracy (for automatic evaluation)

Budget Argument Mining - 2
Minutes-to-Budget Linking (MBLink)
Purpose:▶ Minutes-to-Budget Linking aims to link minutes to budget tables when a sentence of assembly
minutes is given, and extracts the evidence of the discuss.
Input:▶ Budget tables (HTML); each cell has an ID assigned by the organizer▶ Assembly minutes; each sentence has an ID assigned by the organizer
Output:▶ For each sentence in the minutes,✔ “linkedCellIDs”: a list of cell IDs that are related to the sentence.
✔ “containsReason”: whether the sentence contains a reason for the claim.
Evaluation:▶ “linkedCellIDs” and “containsReason” are evaluated using F-measure.

NTCIR-17 PoliInfo-4 Important Dates
IMPORTANT DATES●July 8, 2022
Task Proposal Submission Due●
August 26, 2022
Acceptance Notification of Task Proposals●
September 28, 2022
NTCIR-17 Kickoff Event●
November 8, 2022
First table meeting for PoliInfo-4●
January 2023:
Dataset Release*●
March 2023:
Second round table meeting●
January-June 2023:
Dry Run*●June 2023:
Release of Formal Run dataset●
June-July 2023:
Formal Run*●
August 1, 2023:
Evaluation Result Release●August 1, 2023:
Draft Task Overview Paper Release●September 1, 2023:
Draft Participant Paper Submission Due●November 1, 2023:
All Camera-ready Paper Submission Due●December 2023:
NTCIR-17 Conference in NII, Tokyo, Japan(* indicates that the schedule can be different for different tasks)

Question Answering-2

What is QA Lab-PoliInfo-4 ?
Diet or local assembly

Input: Topic and Opinion
sentences
Output : Approval ordisapproval

Input : Question
Output : Answer
We consider Q&A summary inthe newsletter to be the correctanswers.

Stance Classification-2

Person asking a question

Person answering the question

Assembly minutes

Answer Verification
1st stage:
Input : Q&A in newslettersand assembly minutes
Output : Fake or seemingly
wrong answers
nd2 stage:
Input : Assembly minutes andquestions andquestionable answers
Output : “True” or “False”

質問

答弁

Minutes-to-Budget Linking

Transcript(Assembly minutes)

Input: Budget tables andassembly minutes
Output :1. Linked cell ID
2. Presence or absence ofreasons