Extracting Structure From Text as
Questions, Entities, and Answers

William W. Cohenwith Wenhu Chen, Pat Verga, Haitian Sun, Nitish Gupta, Michiel De Jong, John Wieting, and others

Motivation and BackgroundQuestion-Answering with a KB: A Timeline of KBQA

Chat-80
Warren &Pereira

1980

KBQA: A Timeline

Chat-80
Warren &Pereira

1980

CHILLZelle &Mooney

1996

Engineering is replaced with learning from pairs (x, y)
where●
x = a natural language query●y = a formal query against a structured DB

KBQA: A Timeline

KBQA for FreeBase,a large open-domain
KB that integrated
many other smaller
KBs

WebQuestions
Berant et al
KBQA with complex logical questions

Chat-80
Warren &Pereira

1980

Questions that exploit
the modularity ofKBs: assertions in a
KB can be combinedComplex
WebQuestions
Talmor &BerantCHILL
about virtually any information stored inZelle &the world’s largest KB!
MooneyModularity implies that the number ofanswerable questions is exponentiallymore than the number of stored facts

1996

2013

2018

KBQA: A Timeline

NaturalQuestions
Kwaitkowski etal

WebQuestions
Berant et al

Chat-80
Warren &Pereira

1980

ComplexWebQuestions
Talmor &Berant

CHILL
Zelle &Mooney

1996

2013

2018 19

“Natural” = relatively complex Google queries

KBQA: A Timeline

Many of the questions people
actually ask are notanswerable using existing
KBs.

WebQuestions
Berant et al

Chat-80
Warren &Pereira

1980

NaturalQuestions
Kwaitkowski etal

Complex
WebQuestions
Talmor &Berant

CHILL
Zelle &Mooney

1996

2013

2018 19

KBQA only works for the “right” questions
Four datasets, three methods:●

Natural Questions●

Trivia QA: mostly entity-related
trivia questions.

●WebQuestionsSP: must be
FreeBase answerable andhave a semantic parse●WebQuestionsSP/WD - also
have answers in a 1M-entity
WikiData subset

Methods: FILM uses KB; RAG
retrieves documents and uses NLP toread them; T5 11B answers using
model parametersText-based “Open Book” QA is more robust
Four datasets, three methods:●Natural Questions●

Trivia QA: mostly entity-related
trivia questions.

●WebQuestionsSP: must be
FreeBase answerable andhave a semantic parse●WebQuestionsSP/WD - also
have answers in a 1M-entity
WikiData subset

Methods: FILM uses KB; RAG
retrieves documents and uses NLP toread them; T5 11B answers using
model parametersIs open book QA >> KBQA?
No: Open-book QA still has trouble with compositionality

Complex WebQuestions - with KB

Complex WebQuestions - with Text

Best of both worlds?
Can we build KR/QA systems that●●

exploit the modularity of KBs to storeand query information compositionallyandhave good coverage on the questions
that people naturally ask?

Our proposal: a KB of question-answer
pairs generated automatically from text.

Best of both worlds?

Our proposal: a KB of question-answerpairs generated automatically from text.
●●KB extraction and QA system (Chen etal AAAI 2023, Chen et al EACL 2023)
Ongoing work on KB extraction (Sun etal, 2023)

Questions, Answers, and Entities as a KB

Generating questions and answers from a KB
QA datasets (like SQuaD and NQ) containmany triples (question, passage, answer), so
you can train a model Pr(answer | question,passage)
Alternatively you can train a modelPr(question, answer | passage) and generatemany Q-A pairs from a passage.

Generating questions and answers from a KB
references rq, rp
Additional structure has been added to theseannotated corpora to support explainable
question answering [Lamm, et al, “QED”2021].
●●Question and answer contain annotated“references” (often entities)
Equality between references isannotated.
○ Entity links from the passage canbe propagated to the questions.

answer a

The annotations (rp, q*, a ) are like a KB triple:(Howls_Moving_CastleQ42713, film.writer, Hyao_MiyazakiQ743313)

Abstract version of question q*: who wrote $1?

QEDB: Key Ideas(1) Annotated questions ~= KB relation

Peter Dinklage1…. is an American actor and producer. He
received acclaim for portraying Tyrion
Lannister on the television series Game ofThrones (2011–2019)...

Who is the actor who plays
tyrion in game of thrones

Peter Dinklage1●1

A question-answer pair (with annotations) describe arelationship between NPs in the answer passage(doc 1)○ Peter Dinklage1, Tyrion Lannister1, and Game ofThrones1 are related

QEDB: Key Ideas(1) Annotated questions ~= KB relation

Peter Dinklage1…. is an American actor and producer. He
received acclaim for portraying Tyrion
Lannister on the television series Game ofThrones (2011–2019)...

Who is the actor who plays
tyrion in game of thrones
Tyrion Lannister1 Game of Thrones1
who is the actor
who plays $1 in $2

Peter Dinklage1●

A question-answer pair (with annotations) describe arelationship between NPs in the answer passage(doc 1)○ Peter Dinklage1, Tyrion Lannister1, and Game ofThrones1 are related●Question explanations are modular, atomic objects- a lot like KG relations! But each QA explanation isaligned with user questions○ Information is written about and also askedabout1

QEDB: Key Ideas (2) Large-scale QG + crosslinking
creates a “KB”
Peter Dinklage1…. is an American actor and producer. He
received acclaim for portraying Tyrion
Lannister on the television series Game ofThrones (2011–2019)...

Game of Thrones
Game of Thrones is an American fantasy
drama television series created by David
Benioff and D. B. Weiss for HBO.

Currently using
1.

2

1

2.
Who is the actor who plays
tyrion in game of thronesTyrion Lannister1 Game of Thrones1

what channel is game of thrones onGame of Thrones2

who is the actor
who plays $1 in $2

Peter Dinklage1

where can I watch game of thrones

HBO

What channelis $1 on

3.

Entity linking is on
passages in Wikipedia○ Lots of context!
About ½ of questionphrases are entities andmost contain entities
60M generated, filtered
Wikipedia questions fromthe Facebook PAQ project**4.

T5-based question
annotators trained on
QED/NQ data

We also can soft-matchquestions.

QEDB: A KB of grounded, atomic relationships based on what is asked
Peter Dinklage1…. is an American actor and producer. He
received acclaim for portraying Tyrion
Lannister on the television series Game ofThrones (2011–2019)... his breakthroughcame as Finbar Mcbride in the
comedy-drama The Station Agent (2003)
Who is the actor who plays
tyrion in game of thrones

Game of Thrones
Game of Thrones is an American fantasy
drama television series created by David
Benioff and D. B. Weiss for HBO. It is anadaptation of a series of fantasy novels
by George R. R. Martin.
2Rebels, Rogues & Sworn Brothers…The name of the album comes fromthe appendix of Storm of Swords ,the third book in George R.R. Martin's
Song of Ice and Fire series.
3

1

what channel is game of thrones on

Tyrion Lannister1 Game of Thrones1
who is the actor
who plays $1 in $2

Peter Dinklage1
who played $1 in $2

Finbar McBride The comedy-drama The
Station Agent (2003)

Game of Thrones2
what is game ofthrones based onwhat is $1 based ona series of fantasy novels
by George R. R. MartinHBO

Storm ofSwords3

what is the thirdbook in george r r
martin’s a song ofice and fire

What channel
is $1 onWhat is the third bookin $1 ‘s $2

George R. R. Martin3

A Song of Ice and Fire 3

who is the author of $1

who is the author of a song of ice and fire

who wrote a song of ice and fire

Claim 1: Components in the QEDB are accurate

Claim 2: Components in the QEDB are modular
QEDB

HotpotQA

Claim 2: Components in the QEDB are modular…and we can answer multi-hop
questions with these components!
QAMAT and QAMAT+ [Chen, et al EACL
2023 https://arxiv.org/abs/2204.04581]**Prior work [Lewis et al, 2021, PAQ dataset and RePAQ system] explored answering questions with a
DB of question-answer pairs, but not the possibility of linking QA pairs into a modular KB.

Claim 3: KBQA using a QEDB is aligned with users’ questions
Recap: KBQA only works for the “right” questions●

Trivia QA: mostly entity-related
trivia questions.

●

WebQuestionsSP: must be
FreeBase answerable andhave a semantic parse●WebQuestionsSP/WD - also
have answers in a 1M-entity
WikiData subset

Claim 3: KBQA using a QEDB is aligned with users’ questions
Recap: KBQA only works for the “right” questions●NQ: natural questions●Trivia QA: mostly entity-related
trivia questions.

●WebQuestionsSP: must be
FreeBase answerable andhave a semantic parse●WebQuestionsSP/WD - also
have answers in a 1M-entity
WikiData subset

Claim 3: The QEDB is aligned with users’ questions
Augmenting Pre-trained Language Models
with QA-Memory Chen, et al EACL 2023
https://arxiv.org/abs/2204.04581
Performs well vs comparably
sized memory-augmentedmodels, or larger models using
closed-book approaches
Outperforms prior work
using QA memories(RePAQ)

hardneutralgood

Aside: this problem is not solved by recent LLMs
NQ leaderboard

Claim 3: The QEDB is aligned with users’ questions
An Efficient Memory-Augmented Transformer
for Knowledge-Intensive NLP Tasks:
Wu, Zhao, Hu, Minervini, Stenetorp, Riedel,
2022
Long-form dialog

Still trails thebest openbook
methods: QG
has limited
recallFast key slow value
Slow key slow value

High-recall question generation

The PAQ question generation process
1.
2.
3.

Generate answer strings in each document d: a ~ Pr(a | d)Generate questions conditioned on the answer and document: q ~ Pr(q|a, d)Filter the questions to make sure they make sense as “open book” questions
76% of questions are discarded as having multiple “interpretations”
d = “The Michigan Wolverines men’s basketball team have played their home games at
Crisler Center since 1967. Crisler also became the home stadium of the women’sbasketball team since 1974... The home stadium of the Wolverines football team is theMichigan Stadium.”a = “Crisler Center”
a = “1967”a=…

qq==“wherethetheMichigan
men’s basketballteamplay“wheredodoesmen’s basketballteam play
their
their
homegames”
homegames”

The PAQ question generation process
1.
2.
3.

Generate answer strings in each document d: a ~ Pr(a | d)Generate questions conditioned on the answer and document: q ~ Pr(q|a, d)○ Use a question generation model trained on NQ
Filter the questions to make sure they make sense as “open book” questions

AmbigQA dataset (Min et al, 2021) is an annotated subset of NQ questions●●identified as ambiguous in an open-book context
and edited by crowdworkers to be unambiguousq = “where does the men’s basketballteam play their home games”

q = “where does the University ofMichigan men’s basketball teamplay their home games since1967”

The SIXPAQ* question generation process
1.
2.

3.

Generate answer strings in each document d: a ~ Pr(a | d)Generate questions conditioned on the answer and document: q ~ Pr(q|a, d)○ Use a question generation model trained on NQ AmbigQA
crowdworker-edited questions, with a single “interpretation”
Filter the questions to make sure they make sense as “open book” questions

Other differences:●●●

Haitian Sun, paper
to appear soon!

Generate from all passages○ PAQ generates from a subset
Generate one question from each answer○ PAQ generates two, often redundant
Filter questions generated from a, d if a is
not the best answer to q in d
○ PAQ does this also

Result:●●●21m passages
137M questions
127M answers

vs PAQ’s 60M questions after filtering*Single Interpretation eXtensions to Probably Asked Questions

SIXPAQ coverage
For 89.9% of NQ questions q some PAQ
question q’ has the same answer as q.
For 70.3% of NQ questions q some PAQ
question q’ has the same answer as q
and is semantically the same as q

*Single Interpretation eXtensions to Probably Asked Questions

SIXPAQ coverage???

*Single Interpretation eXtensions to Probably Asked Questions

SIXPAQ applications
Question-based retrieval

recall@10

Retrieve passages indirectly by matchingasked questions q to generated questions
q’ (and then taking associated passage)*Single Interpretation eXtensions to Probably Asked Questions

SIXPAQ applications
Long-form QA for ambiguous questions
Dataset: ASQA [Stelmakh et al. 2022]
A: “The original artist of the song sound of silence
released in 1966 is Paul Simon and Art Garfunkel. In
2016, Australian recording artist Dami Im recorded adiﬀerent song by the same name. The album …”

DISAMBIG-F1

Q: who wrote the sound of silence

Input: an AmbigQA question with-Top-5 retrieved Wikipedia passage, plus
Top-k most similar SIXPAQ questions

Metric: DISAMBIG-F1*Single Interpretation eXtensions to Probably Asked Questions

Recap:Questions, Answers, and Entities as a KB

Recap and conclusions●●●●40 years of work has mostly solved KBQA
○ LLM-based semantic parsers and a KB inference engine work well…but broad-coverage KBs can be misaligned with information needs○ Building KBs data-first means that some things users ask about will get missed
Text-based open book QA methods still are challenged by○ Multihop reasoning○ Providing multiple answers○ …
Closed book QA methods with LLMs still lag in performance and in justifying answers

Recap and conclusions●QA based KBs are a plausible bridge from structured to unstructured knowledge○ can be easily linked to structured KBs (via entity linking on original text)○ are made of modular, easily composed units (like a standard KB)○ are useful as external memory for LLMs
■ for both short answers (NQ, TQA, …) and long-answer tasks○ can be efficient as external memory for LLMs
○ have good coverage for user questions (latest: mid-80s for NQ)

Thanks for your attention!