Pseudo Ambiguous and Clarifying Questions
Based on Sentence Structures
Toward Clarifying Question Answering System

Yuya Nakano, Seiya Kawano, Koichiro Yoshino,
Katsuhito Sudoh and Satoshi Nakamura
Nara Institute of Science and Technology (NAIST)
Guardian Robot Project, RIKEN
奈良先端大
知能コミュニケーション研究室
Augmented Human
Communication Laboratory,
NAIST

2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc1

Question Answering (QA)
◆QA systems generate an answer to a given question⚫Retrieval approach is widely used to realize QA system⚫The problem is defined as a two-stage task⚫Document retrieval to find documents that may containanswers to questions⚫Span prediction to predict
Q. Who is the inventor of Volleyball?
QA modelA. William G. Morgan

Question & Document
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

answer
ACL workshop DialDoc2

Problem of existing QA systems◆User questions are often ambiguous⚫It is hard to predict one answer
What is the masterpiecedrawn by Leonardo da Vinci?
Hmm… There areseveral candidates…

2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc3

Solution: clarifying QA◆The system can ask the user clarification questions⚫Additional information is extracted to find the answer
Ambiguous Q

What is the masterpiecedrawn by Leonardo da Vinci?
Clarifying Q1

Which museum
displays themasterpiece?
the Louvre

Clarifying Q2What is themotif?
A woman

The answer is “La Joconde”
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc

4Paper focus
Ambiguous Q
What is the masterpiece drawn byLeonardo da Vinci?
Clarifying Q1
Which museumdisplays themasterpiece?
the Louvre

Clarifying Q2

Problem 1
Existing QA datasets
do not include such
ambiguous questions

Solution 1
Pseudo ambiguous
question generation
using existing datasets

Problem 2
How does the systemgenerate appropriate
clarifying questions?

Solution 2
Clarifying question
generation using
semantic structures

What is themotif?
A woman

The answer is “La Joconde”◆This paper focus on pseudo ambiguous question
generation and clarifying question generation
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc5

Related works◆Rephrasing ambiguous question [Otsuka19]◆Clarification using target document [Rosenbaum20]

Pseudo ambiguous question to extend existing dataset◆Question generation based on sentence structure[Furukawa18]
Predicting clarifying part based on sentence structure2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc6

System overview
Question

QA dataset

What was the first comic book written by thewriter who had a series developed into a 2010
film with Bruce Willis and Morgan Freeman?

Pseudo Ambiguous QAnswerTransmetropolitan

conversion

What was the first comic book written by thewriter who had a series developed into a 2010
film with Bruce Willis and Morgan Freeman?

Case frame

Generated Clarifying Q
When did the writer have a series?

2010

The answer is“Transmetropolitan”
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKENClarifying Q generation

Concatenation
QA modelACL workshop DialDoc7

Pseudo ambiguous question
What was the first comic book written by thewriter who had a series developed into a 2010
film with Bruce Willis and Morgan Freeman?

QA
model“Transmetropolitan(ROOT(SBARQ(WHNP (WP What)) (SQ (VBD was)
(NP (DT the) (JJ first) (JJ comic) (NN book))(VP (VBN written) (PP (IN by)(NP (NP (DT the) (NN writer)) (SBAR (WHNP (WP who)) (S(VP (VBD had)
(NP (NP (DT a) (NN series)) (SBAR (S(VP (VBD developed)(PP (IN into) (NP (DT a) (CD 2010) (NN film)))(PP (IN with)(NP (NP (NNP Bruce) (NNP Willis)) (CC and)(NP (NNP Morgan) (NNP Freeman)))))))))))))))(. ?)))

What was the first comic book written by thewriter who had a series developed into a 2010
film with Bruce Willis and Morgan Freeman?

2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

CompareQA
model

ACL workshop DialDoc

red8

Clarifying question generation
1. Predicate identification
What was the first comicbook written by the writer
who had a series?

Parser

3. Target case decision
4. Interrogative word decision
Missing cases

P-A•
••
•
•

2. Missing case extraction

was-what (cop)written-book (acl)written-writer (obl:by)
had-writer (nsubj)
had-series (obj)Case frame…
• write-* (advmod)• have-* (advmod)…

HotpotQA
train

2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

Casedecision…
• write-* (advmod)• have-* (advmod)…

Casefrequency

Question
generation

When did thewriter have aseries?

Parser
ACL workshop DialDoc9

Evaluation
Question

HotpotQA

AnswerWhat was the first comic book written by
the writer who had a series developed
into a 2010 film with Bruce Willis andMorgan Freeman?

Pseudo Ambiguous Q
Crowd Worker

Transmetropolitan

conversionQA model

What was the first comic book writtenby the writer who had a series
developed into a 2010 film with Bruce
Willis and Morgan Freeman?

Transmetropolitan

Generated Clarifying Q
When did the writer
have a series?

2010
The answer is“Transmetropolitan”

Clarifying Q gen.

Answer to theoriginal Q

Concatenation
QA modelThe answer is “Spider-Man”
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc10

Experimental setting◆We prepared the eval data in the following procedure⚫Prepare ambiguous questions by the pseudo ambiguousquestion generation. We selected samples by:⚫the eliminated words are less than 50% of the original Q⚫the eliminated words do not contain interrogative⚫QA model results are changed from correct to incorrect◆Original dataset:⚫HotpotQA◆QA model used in the evaluation:⚫BERT-based model◆Answers to the generated clarifying Q:⚫collected by crowdsourcing (AMT)
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc

11

Results
CategoryEM

F1

#q

#eval

w/o Clarify

0.0

0.0

850

4,250

Yes+No49.52

57.28850

4,250

Yes

50.21

57.82

486

2,430

◆We also have asked crowd-workers to evaluate theclarifying questions itself⚫Yes: appropriate, No: not appropriate◆Half of samples we prepared were recovered⚫Clarifying questions can get additional information to improve
the QA accuracy⚫Crowd-workers often provide useful information even if theclarifying question is not appropriate
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc12

Examples
Original Question

Original Question

What is the name of the executive
producer of the film that has a scorecomposed by Jerry Goldsmith?

Lt Col. Stewart Francis Newcombe was aBritish army officer and associate of amilitary officerthat was given what title?

Pseudo Ambiguous Q (A)WorkerWhat is the name of the executive
producer of the film that has a scorecomposed?

Pseudo Ambiguous Q (A)WorkerLt Col. Stewart Francis Newcombe andassociate of a military officerthat wasgiven what title?

Clarifying Q (C)which composed?

Resp.

Jerry Goldsmith

Answer to A+R

Clarifying Q (C)Resp.

which was the Newcombe
and associate given?

a military officer

Answer to A+R

Ronald Shusett

Lawrence of Arabia

Answer to A

Answer to A

Jerry Goldsmith2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

British archaeologist, military
officer, diplomat, and writer
ACL workshop DialDoc13

Summary◆We built pseudo ambiguous question generation andclarifying question generation system toward clarifyingquestion answering system⚫Important to consider sentence structure in generation⚫Predicate (event)-oriented NLP is important to generateclarification questions⚫Even if the clarification question generation failed, theclarification will be a help to input additional information byusers◆We have to build better algorithms to use the pseudodata◆We also have to evaluate the system with real
ambiguous questions by users
2022/04/27 ⒸKoichiro Yoshino,
Guardian Robot Project, RIKEN

ACL workshop DialDoc14