NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Overview of the NTCIR-15 QA Lab-PoliInfo-2 Task
Yasutomo Kimura

Hideyuki Shibuki

Hokuto Ototake

Otaru University of Commerce, JapanRIKEN, Japankimura@res.otaru-uc.ac.jp

National Institute of Informatics,
Japan
shib@nii.ac.jp

Fukuoka University, Japanototake@fukuoka-u.ac.jp

Yuzu Uchida

Keiichi Takamaru

Madoka Ishioroshi

Hokkai-Gakuen University, Japanyuzu@eli.hokkai-s-u.ac.jp

Utsunomiya Kyowa University, Japan
takamaru@kyowa-u.ac.jp

National Institute of Informatics,Japanishioroshi@nii.ac.jp

Teruko Mitamura

Masaharu Yoshioka

Tomoyoshi Akiba

Carnegie Mellon University, U.S.A
teruko@andrew.cmu.edu

Hokkaido University, Japan
yoshioka@ist.hokudai.ac.jp

Toyohashi University of Technology,Japanakiba@cs.tut.ac.jp

Yasuhiro Ogawa

Minoru Sasaki

Kenichi Yokote

Nagoya University, Japanyasuhiro@is.nagoya-u.ac.jp

Ibaraki University, Japan
minoru.sasaki.01@vc.ibaraki.ac.jp

HITACHI, Japan
kenichi.yokote.fb@hitachi.com

Tatsunori Mori

Kenji Araki

Satoshi Sekine

Yokohama National University, Japan
mori@forest.eis.ynu.ac.jp

Hokkaido University, Japanaraki@ist.hokudai.ac.jp

RIKEN, Japansatoshi.sekine@riken.jp

Noriko Kando
National Institute of Informatics,
Japan
SOKENDAI, Japankando@nii.ac.jp

ABSTRACT

Fact-Checking Network of the Poynter Institute established that
April 2 would be considered as International Fact-Checking Day
from 2017. In addition, fact-checking is difficult for general Web
search engines to deal with because of the filter bubble developed by Eli Pariser[7], which keeps users away from informationthat disagrees with their viewpoints. For fact-checking, we should
confirm primary sources such as assembly minutes. The description of the Japanese assembly minutes is a transcript of a speech,
which is very long; therefore, understanding the contents, including the opinions of the members at a glance is difficult. New information access technologies to support user understanding areexpected, which would protect us from fake news.
We provide the Japanese Regional Assembly Minutes Corpus as
the training and test data, and investigate appropriate evaluation
metrics and methodologies for the structured data as a joint effortof the participants.
The QA using Japanese regional assembly minutes has the following challenges to consider:

The NTCIR-15 QA Lab-PoliInfo-2 aims at real-world complex Question Answering (QA) technologies using Japanese political information such as local assembly minutes and newsletters. QA LabPoliInfo-2 has four sub tasks, namely Stance classification, Dialog
summarization, Entity linking and Topic detection. We describe
the used data, formal run results, and comparison between humanmarks and automatic evaluation scores.

TEAM NAME
Task Organizers

SUBTASKS
Overview

1

INTRODUCTION

The QA Lab-PoliInfo-2 (Question Answering Lab for Political Information 2) task at NTCIR-15 aims at complex real-world question answering (QA) technologies, to show summaries of the opinions of assembly members and the reasons and conditions for such
opinions, from Japanese regional assembly minutes.
We reaffirm the importance of fact-checking because of the negative impact of fake news in the recent years. The International

1: comprehensible summary of a topic;
2: beliefs and attitudes of assembly members;
3: mental spaces for other assembly members;
4: contexts, including reasons;
5: several topics in a speech; and101

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Figure 1: Comparison with related shared tasks
6: colloquial Japanese including dialect and slang.

conducted the Check-worthiness and Factuality tasks in both English and Arabic, based on debates from the 2016 US Presidential
Campaign[1].
Figure 1 shows a comparison with the related shared tasks suchas Profiling Fake News Spreaders, CheckThat! and QA Lab PoliInfo2. The organizers of Profiling Fake News Spreaders addressed the
problem of fake news detection from the author profiling perspective[8].
CheckThat! addressed the development of technology capable ofspotting check-worthy claims in English political debates in addition to providing evidence-supported verification of Arabic claims[3][2].

In addition to the QA technologies, this task will contribute to thedevelopment of a semantic representation, context understanding,information credibility, automated summarization, and dialog systems.
In the NTCIR-15 QA Lab PoliInfo-2 (hereinafter called "PoliInfo2"), stance classification, dialog summarization, entity linking
and topic detection sub tasks were held. The stance classification
task is an expansion of the classification task in the NTCIR-14 QA
Lab-PoliInfo[4]. Although the classification task aimed to infer individual political policies of assembly members from their speech,
the stance classification task aims to infer political party stancesof bills from speeches of assembly members in the party. The dialog summarization task is a combinational expansion of the segmentation and the summarization tasks in the NTCIR-14 QA LabPoliInfo. The segmentation task aimed to find a description related to a given short text from speeches as source document, andthe summarization task aimed to summarize a description from aspeech without changing the meaning. The dialog summarization
aims to find and summarize descriptions related to a given topic
word from question and answer speeches without changing themeaning. In the NTCIR-14 QA Lab-PoliInfo, We observed several
inconsistent spellings with the same meaning. To deal with this,we held the entity linking task that is to extract and map descriptions to law names. The topic detection task is an additional task
to study a role of political information in order to cope with theoutbreak of COVID-19.

2

3 JAPANESE REGIONAL ASSEMBLY MINUTESCORPUS
Kimura et al.[5] constructed the Japanese Regional Assembly Minutes Corpus that collects minutes of plenary assemblies in 47 prefectures of Japan from April 2011 to March 2015. Figure 2 shows
an example of the minutes of the Tokyo Metropolitan Assembly.
Japanese minutes resemble a transcript. In the question-and-answer
session, a member of assembly asks several questions at a time, anda prefectural governor or a superintendent answers the questions
under his/her charge at a time. A speech is too long to understandthe contents at a glance; therefore, information access technologies such as QA and automated summarization, will aid in understanding. For the QA Lab-PoliInfo task, we distributed a subset ofthe corpus, which is narrowed down to the Tokyo Metropolitan
Assembly.

RELATED WORK
4 TASK DESCRIPTION

Fake news detection and Fact-checking have recently received significant research attention. Fake News Challenge1 and CLEF-2018
Fact Checking Lab2 are shared tasks dealing with political information. Fake News Challenge conducted the Stance Detection task
estimating the relative perspective (or stance) of two pieces of text
relative to a topic, claim or issue. CLEF-2018 Fact Checking Lab

We designed the stance classification, the dialog summarization,
the entity linking and the topic detection tasks. We put the tasks
at the elemental technologies of information credibility or factchecking for political information systems. Figure 3 shows a relation of the tasks.
Human evaluation has advantage in terms of detailing and deep
understanding, while automatic evaluation has advantage in termsof labor and time savings. We used automatic evaluation so thatparticipants could confirm their results immediately during the dry

1 http://www.fakenewschallenge.org/
2 http://alt.qcri.org/clef2018-factcheck/

102

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Figure 2: Example of the minutes of the Tokyo Metropolitan Assembly
Table 1: Data fields used in the assembly minutes of the
stance classification task
Field name
Date
Prefecture
ProceedingTitle
Proceeding
URL

Explanation
Date
Prefecture nameTitle of proceeding
List with Speaker and Utterance as elements
Tokyo Metropolitan website

4.1.2 Data. We distributed the assembly minutes and an answer
sheet. Table 1 and 2 show the data fields of the minutes and theanswer sheet, respectively. The data sizes of them are shown in
Table 3 and 4, respectively. Examples of the minutes and the answer sheet are shown as below.
ソースコード 1: Minutes for Stance Classification
1
2
3

[
{
"Date": "2001/8/8","Prefecture": "東京都","ProceedingTitle": "平成十三年第一回臨時会会議
録","URL": "https://www.gikai.metro.tokyo.jp/record/
extraordinary/2001-1.html","Proceeding": [{"Speaker": " 議会局長（細渕清君） ","Utterance": " 議会局長の細渕でございま
す。"
}
]

4
5

and formal runs. After the formal run, human evaluation was used
for detailed analysis.
For automatic evaluation, we introduced leader boards of thetasks, which were published on the QA Lab PoliInfo-2 website3 .
Participants could post their system results once a day.

6
7
8
9
10

4.1

Stance classification

4.1.1 Purpose. Stance classification task aims at estimating politician’s position from politician’s utterances. In PoliInfo2, system
participating in the task estimates the stances of political parties
from the utterances of the members of the Tokyo Metropolitan Assembly. Given the Tokyo Metropolitan Assembly, topics (agenda),
member’s list and political denomination list, and the systems classify their stance into two categories (agreement or disagreement)
for each agenda.
3 https://poliinfo2.net/

11
12
13
14

}
]

ソースコード 2: Answer sheet for Stance Classification
1
2
3
4

103

[
{
"ID":"PoliInfo2-StanceClassification-JA-DryTraining-02543","Prefecture":"東京都",

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Figure 3: Relation of the three tasks
Table 2: Data fields used in the answer sheet of the stance classification task
Field nameID
Prefecture
Meeting
MeetingStartDate
MeetingEndDate
Proponent
BillClass
BillSubClass
Bill
BillNumber
SpeakerList
ProsConsPartyListBinary
ProsConsPartyListTernary

Explanation
Identification code
Prefecture name
Meeting nameStart date of the meeting (Date type)End date of the meeting (Date type)
Proponent is either a governor or an assembly member .
Class
Sub classBill nameBill identification
Assembly member and political party
Answer section agreement or disagreement (binary)Answer section agreement, disagreement or NS (ternary)

Table 3: Tokyo Metropolitan assembly minutes for the
stance classification task
Minutesregular and extra meetingscommittee meetings

5

Number of files
230

Table 4: Data size of the answer sheet in the stance classification task

File size
97MB
462MB

"Meeting":"平成３１年第１回定例会、第１回臨時会",

Answer sheet
Training
Test

6
7

104

Number of questions
2,622
479

"MeetingStartDate":"2019/2/20","MeetingEndDate":"2019/3/28" ,File size
8.1MB
1.4MB

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

Table 5: Data fields used in the assembly minutes of the dialog summarization task

"Proponent" : "知事提出議案","BillClass" : "予算","BillSubClass": "３１年度予算","Bill" : "一般会計","BillNumber" : "第一号議案","SpeakerList" :{"増子ひろき" : "都ファースト","吉原修" : "自民党",},"ProsConsPartyListBinary" :{"都ファースト" : "賛成","公明党" : "賛成","自民党" : "反対","日本共産党" : "反対",},"ProsConsPartyListTernary" :{"都ファースト":null,"公明党":null,"自民党":null,"日本共産党":null,}

8

Field nameID
Line
Prefecture
VolumeNumber
Year
Month
Day
Title
Speaker
Utterance

Table 6: Data fields used in the answer sheet in the dialog
summarization task}

Field nameID
Date
Prefecture
Meeting
MainTopic
QuestionSpeaker
SubTopic
QuestionSummary
QuestionLength
QuestionStartingLine
QuestionEndingLine
AnswerSpeaker
AnswerSummary
AnswerLength
AnswerStartingLine
AnswerEndingLine

]

Input

OutputEvaluation

Tokyo Metropolitan assembly minutes(Proceedings, Committees)
Answer sheet
Binary : agreement or disagreement
Ternary : agreement, disagreement or NS
The average of accuracy

4.1.3 Evaluation. For automatic evaluation, pros and cons of bills
published by assembly secretariat were used as gold standard data.
1 ∑ NCA(i)N B i=1 N PP(i)NB

Score =

(1)

where N B is Number of Bill, NCA(i) is Number of Correct Answers
for Billi and N PP(i) is Number of Political Party,

4.2

Explanation
Identification code
Line number
Prefecture nameVolumeDay of the meeting
Year
Month
Day
Title
Speaker
Utterance

Explanation
Identification code
Date
Prefecture name
Meeting nameMain topic
Question speaker
Sub topic
Summary of question
Limit length of summary
Starting line of question
Ending line of question
Answer speaker
Summary of answer
Limit length of summary
Starting line of question
Ending line of question

Table 7: Tokyo Metropolitan assembly minutes for the dialog summarization task

Dialog summarization

4.2.1 Purpose. Dialog summarization task aims at summarizing
the transcript of local assembly, taking the structure of dialogue
into account. In PoliInfo2, systems participating in this task summarize the transcript based on the dialogue structure, which consists of Members’ questions and Governor’s answer . Given
the transcript and summary conditions (speaker name and number of summary characters etc), they generate the structured document.

Minutes
Proceedings

Number of files
1

File size
42MB

Table 8: Data size of answer sheet in the dialog summarization task
Answer sheet
Training with segment
Training without segment
Test

4.2.2 Data. For the dialog summarization task, the minutes of theTokyo Metropolitan Assembly from April 2011 to March 2015 and asummary of a speech of a member of assembly described in Togikaidayori4 , a public relations paper of the Tokyo Metropolitan Assembly are provided. Table 5 and 6 show the data fields of the minutes and the answer sheet, respectively. The data sizes of them are

Number of questions
438
325
254

File size
414KB
292KB
161KB

shown in Table 7 and 8, respectively. An example of the answer
sheet is shown as below.

4 https://www.gikai.metro.tokyo.jp/newsletter/ (in Japanese)

105

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

1
2
3
4
5
6
7
8
9
10
11
12

ソースコード 3: Minutes for the dialog summarization{"ID":"130001_230617_2","Line":2,"Prefecture":"東京都","Volume":"平成 23年_第２回","Number":"1","Year":23,"Month":6,"Day":17,"Title":"平成 23年_第２回定例会(第７号)","Speaker":"和田宗春","Utterance":"ただいまから平成二十三年第二回東京都議会定例会を開会いたします。"}

where S is a set of summaries the participants assessed, and a is aconstant representing whether acceptable summaries that are different from the gold standard summary are regarded as correct ornot. If such summaries are regraded as correct, a is 2; otherwise, a
is 0.
Input
OutputEvaluation

4.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

1. Tokyo Metropolitan assembly minutes
2. Answer sheet in Json format
A summary that takes into account the structureof the dialogue between question and answer.
ROUGE-1 and human marks

Entity Linking

4.3.1 Purpose. Entity linking task aims at identifying political terms
included in politicians’ statements, and is to resolve mention recognition, disambiguation and linking the mention with the knowledge base. In PoliInfo2, Entity linking is the task of assigning aunique identity of "law name" which is one of the political terms.
Given local assembly member’s utterances, and systems extract amention of law name and link the mention with the list of law
names or Wikipedia.

ソースコード 4: Answer sheet for the dialog summarization[{"AnswerEndingLine": [ 532 ],"AnswerLength": [ 50 ],"AnswerSpeaker": [ "知事" ],"AnswerStartingLine": [ 528 ],"AnswerSummary": ["全国の先頭に立ち刻苦する被災地を支援するのは当然。
今後も強力に後押しする。"],"Date": "2011-06-23","ID": "PoliInfo2-DialogSummarization-JA-Dry-TrainingSegmented-00001","MainTopic": "東京の総合防災力を更に高めよ<br>環境に配慮した都市づくりを","Meeting": "平成 23年第 2回定例会","Prefecture": "東京都","QuestionEndingLine": 276,
"QuestionLength": 50,"QuestionSpeaker": "山下太郎（民主党）","QuestionStartingLine": 266,
"QuestionSummary": "被災地が真に必要とする支援に継続して取り組むべき。知事の見解は。","SubTopic": "東日本大震災"}]

4.3.2 Data. Table9 shows data fields used in the answer sheet ofof the entity linking task. The answer sheet is TSV format, which
is similar to AIDA CoNLL-YAGO Dataset.The data size is shown in
Table 10. Figure 4 shows an example of the answer sheet.

4.2.3 Evaluation. For the Leader board for automatic evaluation,
we used ROUGE-1 Recall[6] to calculate the score as . We used asummary of newsletter as the gold standard data.
For human evaluation, we used the following quality questions
by the participants. The quality questions were assessed by a threegrade evaluation (A, B and C) for content, well-formed, non-twisted,
sentence goodness and dialog goodness, respectively. However, forthe content evaluation, we prepared an extra grade X because asummary that does not include contents of gold standard data may
be acceptable. The quality question score QQ(v) from viewpoint v
was calculated using the following expressions:∑s ∈S д(s, v)
QQ(v) =(2)|S |
 2 (дradeA) 1 (дradeB)д(s, v) =(3)0 (дradeC) a (дradeX )

Figure 4: Example of the entity linking file
4.3.3 Evaluation. Because the gold standard data was made by human workers and checked by participants, human evaluation did
not conducted.
2 × Precision × Recall
Score =
(4)
Precision + Recall
NCM
Precision =
(5)
NOM
NCM
Recall =(6)
NGSM
where NCM is the number of correct mentions, NOM is the number of outputted mentions, and NGSM is the number of gold standard mentions. The correct mention means that the expression is106

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 9: Data fields used in the answer sheet of the entity
linking task (TSV format)
Field name
Column1
Column2
Column3
Column4
Column5

Explanation
Word segmented tokens by morphological analysis
A tag is either B (beginning of a mention)
or I (continuation of a mention)
A full mention used to search for candidate entities
Wikipedia title
Wikipedia URL

11
12
13

Input
OutputEvaluation

Number of morphemes
260,366
209,862

File size
2.7MB
1.9MB

September 30, 2019: QA Lab-PoliInfo Kickoff Meeting
October 18, 2019: First round table meeting in NII
December 15, 2019: Second round table meeting in NII
February 15, 2020: Dataset release

1. Answer sheet in TSV format
2. Wikipedia titles (2019-12-01)
Column 2 and 3 : An entity mention extraction
Column 4 : Wikipedia title for the mention
End-to-end evaluation from column2 to column4

Dry RunApril 23, 2020: First online round table meeting
May 7, 2020: Dry Run
May 27, 2020: Second online round table meeting using zoom
June 27, 2020: Third online round table meeting using zoom
June 30, 2018: Submission Deadline for Dry Run
Formal Run
July 1- 12, 2020: Update of dataset
July 31, 2020: Task Registration Due for Formal Run (This is not
required for Dry Run participants)
July 13 - 31, 2020: Formal Run (Stance classification, Dialog summarization and Entity linking)

Topic detection

NTCIR-15 CONFERENCE
August 1 - 7, 2020: Evaluation by participants
August 8 - 14, 2020: Evaluation by organizers
August 15, 2020: Evaluation Result Release
August 26, 2020: Fourth online round table meeting using zoom
September 1, 2020: Task overview paper release (draft)
September 20, 2020: Submission due of participant papers
November 1, 2020 Camera-ready participant paper due
December 8-11, 2020: NTCIR-15 Conference & EVIA 2020

Input Newsflashes
Output Lists of dialog topic words/phrases per speaker
Data. An example of the answer sheet is shown as below.

5 PARTICIPATION

ソースコード 5: Minutes for the topic detection
1
2
3
4
5
6
7
8
9

Schedule

The NTCIR-15 QA Lab-PoliInfo task has been run according to thefollowing timeline:

4.4.1 Purpose. In order to cope with the outbreak of COVID-19, it
is important to speedily provide the latest information of COVID19 for citizens. Considering the possibility of information accesstechnologies, we held another task using the local assembly minutes, namely, topic detection task.
Newsletters issued by local government can consistently provide arguments in local assembly to citizens, while it takes a longtime to make them. Although local government also provides newsflashes for providing arguments promptly, there is room for improvement of comprehensibility at a glance. Therefore, the topic
detection task aims to make a list of argument topics from newsflashes of local assembly minutes.

4.4.2

]

4.5

exactly matched the gold standard and that the mapped entity iscorrect.

4.4}}

4.4.3 Evaluation. We did not conduct quantitative evaluation. Task
organizers and participants discussed appropriate topic words andthe application.

Table 10: Data size the entity linking task
Answer sheet
Training
Test"Utterance": "ただいまから令和二年第一回
東京都議会定例会を開会いたします。\
n これより本日の会議を開きます。
\n
"

10

Eighteen teams were registered, but only 15 teams participated actively, namely, submitted any results. Table 11 shows the active
participating teams.

[{"Date": "2020/2/19","Prefecture": "東京都","ProceedingTitle": "令和二年東京都議会会議録第一号","URL": "https://www.gikai.metro.tokyo.jp/record/
proceedings/2020-1/01.html","Proceeding": [{"Speaker": "議長（石川良一君）",

6 SUBMISSIONS
Table 12 shows the number of submissions. The number in brackets means the number of late submissions. In the dry run, there
were 19 submissions from 5 teams for the stance classification, 2
submissions from 2 teams for the dialog summarization and a submission from a team for the entity linking. In the formal run, there

107

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 11: Active participating teams
akbl∗
knlab
wer99
Ibrk∗
LIAT∗
HUHKA∗
JRIRD
selt
nukl∗
rnyk∗∗
Forst∗SKRA
TKLB
wfrnt∗
TO∗

Toyohashi University of Technology
Shizuoka University
Tokyo Institute of technology
Ibaraki University
RIKEN Center for Advanced Intelligence Project (AIP)
Hokkaido University
The Japan Research Institute, Limited
Waseda University
Nagoya University
individuals
Yokohama National University
Hokkaido University
Osaka Electric-Communication University
HITACHItask organizers∗ Task organizer(s) are in the team∗∗ only the dry run
Table 12: Number of submitted runs

Team ID

akbl
knlab
wer99
Ibrk
LIAT
HUHKA
JRIRD
selt
nukl
rnyk
Forst
SKRA
TKLB
wfrnt
TO
Sum

Stanceclassification
7
1
4
2
5
19

Dry runDialogsummarization
1
1
2

Entitylinking
1
1

Stanceclassification
5
6
9
4
2(3)
26(3)

Formal run
Dialogsummarization
-(1)
1(1)
3
4
5(5)
1
2
3
19(7)

Entitylinking
8(4)
4
1(1)
4
17(5)

Topic
detection
2
1
1
1
6

7 RESULT
7.1 Dry run

were 29 submissions (including 3 late submissions) from 5 teams
for the stance classification, 26 submissions (including 7 late submissions) from 8 teams for the dialog summarization and 22 submissions (including 5 late submissions) from 4 teams for the entity
linking. Fpr the topic detection, there were 6 submissions from 5
teams. In total, there were 90 submissions from 15 teams.

We conducted only automatic evaluation in the dry run. Table 13,
14 and 15 show results of the stance classification, the dialog summarization and the entity linking in the dry run, respectively. Because the test data of the stance classification was corrected on
May 22 and June 5, we separated the results according to the period.

108

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 13: Accuracy in the stance classification task at the dryrunTable 16: Accuracy in the stance classification task at the formal run

ID teamAccuracy
from May 7 to May 21
116 rnyk.9457
72 akbl.9437
90 wer99.9375
114 rnyk.9325
115 wer99.9284
from May 22 to June 4
120 rnyk.9493
125 akbl.9467
124 wer99.9416
119 rnyk.0001
from June 5 to July 4
144 Ibrk.9569
143 knlab.9523
129 rnyk.9499
139 akbl.9494
126 akbl.9472
132 akbl.9466
131 akbl.9422
130 wer99.9382
136 akbl.8927
140 Ibrk
.8839

ID
175
177
202
191
196
186
182
205
180
149
167
203
214
199
158
160
156
204
218
198
153
154
193
169
171
164

Table 14: ROUGE-1-R scores in the dialog summarization
task at the dry run
ID
141
137

team
JRIRD
TO

team
wer99
wer99
wer99
wer99
wer99
wer99
wer99
Ibrk
Ibrk
Ibrk
Ibrk
knlab
knlab
knlab
knlab
knlab
akbl
akbl
akbl
akbl
wer99
wer99
knlab
akbl
Forst
Forst

Accuracy
.9976
.9976
.9976
.9970
.9952
.9923
.9910
.9650
.9644
.9600
.9598
.9531
.9531
.9529
.9520
.9520
.9498
.9498
.9496
.9492
.9481
.9461
.9452
.9399
.9388
.9382

ROUGE
.2865
.2436

8 OUTLINE OF THE SYSTEMS
We briefly describe the characteristic aspects of the participatinggroups systems and their contribution below.
The akbl team tackled the Stance Classification, the Dialog Summarization, and the Topic Detection tasks. For the Stance Classification task, they used a rule-based analyzer on the opinion statements at first, then, for those left undetermined, they applied a
BERT-based stance classifier on the debate statements. For the Dialog Summarization task, they firstly searched for the relevant segment, then extracted the final sentence to form the output summary. For the Topic Detection task, they employed a clustering
algorithm on the BERT embeddings of initial topic candidates extracted by using regular expressions, then their final topics wereselected based on the centroid of each cluster.
The knlab team tackled the Stance classification task. For the
Stance Classification task, they designed features obtained from asentiment dictionary and BERT, then trained LightGBM to classify
the stances.
The wer99 team tackled the stance classification task. They designed a set of rules to recognize an explicit mention to a stancefor a bill. When a party does not mention a stance explicitly, theyuse clues in the bill name to predict a stance.

Table 15: F-measures in the entity linking task at the dry run
ID
108

7.2

team
HUHKA

ROUGE
.4049

Formal run

Automatic evaluation and human evaluation were conducted in
the formal run. Table 16, 17 and 18 show automatic evaluation results of the stance classification, the dialog summarization and theentity linking in the formal run, respectively.
Table 19 shows a human evaluation result of the stance classification.
Table 20, 21 and 22 show human evaluation results of the dialog
summarization. Table 23 shows the Cohen’s kappa scores for thehuman evaluation of the dialog summarzation.
Although the deadline was July 31, we accepted submissions until August 31. They were treated as late submissions. Table 24, 25
and 26 show results of the late submissions of the stance classification, the dialog summarization and the entity linking, respectively.

109

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 17: ROUGE-1-R scores in the dialog summarization
task at the formal run
ID
189
185
195
216
148
215
187
161
172
200
194
157
208
151
181
176
184
211
206

team
JRIRD
JRIRD
JRIRD
nukl
TO
Forst
nukl
nukl
nukl
Forst
Forst
TO
wfrnt
TO
wfrnt
Forst
Forst
SKRA
LIAT

Table 19: Accuracy of stance classification task at the formalrun (human evaluation results)

ROUGE
.3208
.2980
.2980
.2581
.2436
.2410
.2387
.2274
.2198
.2145
.2093
.1331
.1171
.1164
.1058
.0782
.0729
.0696
.0555

ID
149
153
154
156
158
160
164
167
169
171
175
177
180
182
186
191
193
196
198
199
202
203
204
205
214
218

Table 18: F-measures in the entity linking task at the formal
run
ID
212
201
155
174
197
150
192
217
183
147
166
146
173
178
179
213
190

team
HUHKA
HUHKA
HUHKA
HUHKA
HUHKA
HUHKA
HUHKA
Forst
Forst
Forst
HUHKA
Forst
selt
selt
selt
selt
nukl

F-measure
.6035
.4887
.4747
.4468
.4468
.4049
.3980
.3910
.3656
.3389
.3247
.3089
.2980
.2978
.2978
.2930
.2375

team
Ibrk
wer99
wer99
akbl
knlab
knlab
Forst
Ibrk
akbl
Forst
wer99
wer99
Ibrk
wer99
wer99
wer99
knlab
wer99
akbl
knlab
wer99
knlab
akbl
Ibrk
knlab
akbl

Accuracy
—–
—–
—–
0.668
0.805
0.834
0.144
—–
0.838
0.852
0.978
0.978
—–
0.978
0.978
0.978
0.834
0.978
0.675
0.805
0.982
0.805
0.892
—–
0.805
0.888

decomposed the task into border detection, topic matching, and extractive summarization and used an attention mechanism to solve
each subtask.
The HUHKA team tackled the Entity Linking task. For Entity
Linking task, they extracted mentions of “law name” with BERT,
and filter the extracted mentions. For the extracted mentions, they
performed disambiguation using exact match, Wikipedia2Vec, mentionentity prior, and e-Gov.
The JRIRD team tackled the Dialog Summarization subtask. For
the Dialog Summarization subtask, they developed a BERT-based
module that extracts candidate sentences, and a UniLM-based module that generates a summary from the extracted sentences.
The selt team tackled the Entity Linking tasks. For the Entity
Linking task, they detected the mentions using fine-tuned BERT
and disambiguate the entities of them with wikipedia2vec. To improve their system performance, they used some rules for mention
and entity decision.
The nukl team tackled the Dialog Summarization, Entity Linking, and Topic Detection tasks. For the Dialog Summarization task,
they applied Progressive Ensemble Random Forest (PERF) developed at the NTCIR-14 QA Lab-PoliInfo to sentence extraction andsentence reduction. For the Entity Linking task, they applied simple matching. For the Topic Detection tasks, they used a rule-based
approach.

The ibrk team tackled the Stance Classification task. They develop rule-based system for the Stance Classification task by detecting the word "agree" or "disagree" with each bill in speaker’s
utterances. If its word is not obtained in the utterances about abill, they categorize his/her opinion into "agree" or "disagree" according to some heuristics.
The LIAT team tackled the Summarization task. For the Summarization task, they took an approach of sentence extraction. They110

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 20: Quality question scores of the dialog summarization in the formal run (Content, Well-formed and Sentence goodness)

ID
148
157
176
181
185
187
189
206
208
211
215
216

team
TO
TO
Forst
wfrnt
JRIRD
nukl
JRIRD
LIAT
wfrnt
SKRA
Forst
nukl

num ofsummaries
533
533
533
533
533
533
533
533
533
533
533
533

Content(X=2)(X=0)
398.5 0.748 357.5 0.671
258.0 0.484 210.0 0.394
188.5 0.354 146.5 0.275
157.0 0.295 138.0 0.259
540.5 1.014 479.5 0.900
422.5 0.793 375.5 0.705
576.5 1.082 519.5 0.975
176.0 0.330 136.0 0.255
175.0 0.328 158.0 0.296
184.5 0.346 135.5 0.254
414.5 0.778 355.5 0.667
442.0 0.829 398.0 0.747

Table 21: Quality question scores of the dialog summarization in the formal run (Non-twisted)

ID
148
157
176
181
185
187
189
206
208
211
215
216

team
TO
TO
Forst
wfrnt
JRIRD
nukl
JRIRD
LIAT
wfrnt
SKRA
Forst
nukl

num ofevaluable
259
172
90
102
360
270
373
104
109
118
274
292

all
539.0
377.0
279.0
224.0
650.0
557.5
701.5
247.0
248.5
283.5
556.5
598.5

team
TO
TO
Forst
wfrnt
JRIRD
nukl
JRIRD
LIAT
wfrnt
SKRA
Forst
nukl

num oftopics
254
254
254
254
254
254
254
254
254
254
254
254

Sentence
goodness389.0 0.730
228.5 0.429
138.0 0.259
146.0 0.274
555.5 1.042
423.5 0.795
601.5 1.129
122.5 0.230
160.5 0.301
151.5 0.284
415.5 0.780
445.5 0.836

The Forst team tackled the Stance Classification, the Dialog Summarization, and the Entity Linking tasks. For the Stance Classification task, they used a rule-based approach taking into account thedate of assembly, speaker name and bill name. For the Dialog Summarization task, they extracted sentences using word embedding
similarity between a sentence and a passage including it. For the
Entity Linking task, they extracted mentions using BiLSTM-CRF
model and disambiguated the entities using RNN model.
SKRA team tackled Dialog Summarization task. They extracted
key sentences using an unsupervised extraction method based on
EmbedRank++. The team TKLB tackled the Topic Detection task.
For the task, they proposed to find differences of opinions and positions among the participants based on the co-occurrence graph.
To reflect the broader contexts that all of the given discussions provide, they applied the Latent Dirichlet Allocation (LDA) to weight
each word.
The wfrnt team tackled the Dialog-Summarization task. For the
Dialog-Summarization task, they investigated whether heuristics
of conclusion extraction in Japanese is useful to develop a baseline
system for summarization. They quantitatively verified the validity of examination of language use such as ”English begins withconclusion, Japanese begins with background.”Non-twisted
evaluable
1.011 429.5 1.658
0.707 255.5 1.485
0.523 113.5 1.261
0.420 159.0 1.559
1.220 569.0 1.581
1.046 456.0 1.689
1.316 638.5 1.712
0.463 147.0 1.413
0.466 172.5 1.583
0.532 164.5 1.394
1.044 435.5 1.589
1.123 496.5 1.700

Table 22: Quality question scores of the dialog summarization in the formal run (Dialog goodness)

ID
148
157
176
181
185
187
189
206
208
211
215
216

Well-formed
843.0 1.582
803.0 1.507
748.0 1.403
688.5 1.292
975.5 1.830
867.5 1.628
990.5 1.858
867.0 1.627
730.5 1.371
888.5 1.667
906.5 1.701
896.0 1.681

Dialoggoodness
124.0 0.488
69.5 0.274
33.5 0.132
22.0 0.087
215.5 0.848
138.5 0.545
238.0 0.937
28.0 0.110
27.0 0.106
43.0 0.169
153.5 0.604
156.5 0.616

9 CONCLUSIONS
We described the overview of the NTCIR-15 QA Lab-PoliInfo-2
task. The goal is realizing complex real-world question answering(QA) technologies, to show summaries of the opinions of assembly
members and the reasons and conditions for such opinions, fromJapanese regional assembly minutes. We conducted in a dry run
and a formal run, which are including the stance classification, dialog summarization, entity linking and topic detection sub tasks.
There were 105 submissions from 15 teams in total. We described
the task description, the collection, the participation and the results.

REFERENCES[1] Pepa Atanasova, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Wajdi
Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018.
Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and111

NTCIR 15 Conference: Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies, December 8-11, 2020 Tokyo Japan

Table 23: Cohen’s kappa scores for human evaluation of dialog summarization task in the formal run

Team
JRIRD
nukl
Forst
wfrnt
SKRA
LIAT
TO

Content
0.602
0.378
0.287
0.317
0.328
0.449
0.586

Well-Formed
0.454
0.303
0.109
0.156
0.311
0.417
0.417

Non-Twisted
0.387
0.337
0.354
0.392
0.369
0.360
0.325

Table 24: Accuracy of the late submissions of the stance classification task in formal run
ID
234
232
226

team
Forst
Forst
Forst

team
Forst
Forst
Forst
Forst
Forst
LIAT
akbl

Accuracy
.9408
.9391
.8642

ROUGE
.1384
.1219
.1155
.1133
.1040
.0946
.0621

Table 26: F-measures of the late submission of the entity
linking task in the formal run
ID
238
233
236
229
225

team
HUHKA
HUHKA
HUHKA
HUHKA
nukl

Dialoggoodness
0.310
0.314
0.358
0.354
0.325
0.365
0.369

[4] Yasutomo Kimura, Hideyuki Shibuki, Hokuto Ototake, Yuzu Uchida, Keiichi
Takamaru, Kotaro Sakamoto, Madoka Ishioroshi, Teruko Mitamura, Noriko
Kando, Tatsunori Mori, Harumichi Yuasa, Satoshi Sekine, and Kentaro Inui. 2019.
Overview of the NTCIR-14 QA Lab-PoliInfo Task. Proceedings of The 14th NTCIR
Conference (6 2019).
[5] Yasutomo Kimura, Keiichi Takamaru, Takuma Tanaka, Akio Kobayashi, Hiroki Sakaji, Yuzu Uchida, Hokuto Ototake, and Shigeru Masuyama. 2016.
Creating Japanese Political Corpus from Local Assembly Minutes of 47 prefectures. In Proceedings of the 12th Workshop on Asian Language Resources(ALR12). The COLING 2016 Organizing Committee, Osaka, Japan, 78–85.
https://www.aclweb.org/anthology/W16-5410[6] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries.
In Text Summarization Branches Out. Association for Computational Linguistics,
Barcelona, Spain, 74–81. https://www.aclweb.org/anthology/W04-1013[7] Eli Pariser. 2011. The Filter Bubble: What the Internet Is Hiding from You. Penguin
Books.
[8] Francisco Rangel, Anastasia Giachanou, Bilal Ghanem, and Paolo Rosso. 2020.
Overview of the 8th Author Profiling Task at PAN 2020: Profiling Fake News
Spreaders on Twitter. In CLEF 2020 Labs and Workshops, Notebook Papers, Linda
Cappellato, Carsten Eickhoff, Nicola Ferro, and Aurélie Névéol (Eds.). CEURWS.org.

Table 25: ROUGE-1-R scores of the late submissions of the
dialog summarization task in the formal run
ID
235
231
242
239
240
224
237

Sentence
goodness
0.447
0.392
0.400
0.400
0.349
0.414
0.477

F-measures
.5863
.5518
.5000
.3980
.3813

Verification of Political Claims. Task 1: Check-Worthiness. CoRR abs/1808.05542(2018). arXiv:1808.05542 http://arxiv.org/abs/1808.05542[2] Alberto Barrón-Cedeño, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov,
Bayan Hamdan, Alex Nikolov, Shaden Shaar, and Zien Sheikh Ali. 2020.
Overview of CheckThat! 2020 — Automatic Identification and Verification ofClaims in Social Media. In Proceedings of the 11th International Conference of
the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, andInteraction (CLEF ’2020). Thessaloniki, Greece.
[3] Tamer Elsayed, Preslav Nakov, Alberto Barrón-Cedeño, Maram Hasanain, Reem
Suwaileh, Giovanni Da San Martino, and Pepa Atanasova. 2019. Overview of
the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims.
In Experimental IR Meets Multilinguality, Multimodality, and Interaction (LNCS).
Lugano, Switzerland.

112