Knowledge Augmentation for
Reasoning in Language
P rt of the ACL 2023 Tutori l“Complex Re soning in N tur l L ngu ge”(Bill) Yuchen Linaaaaaaa

https://yuchenlin.xyz

Knowledge Augmentation for Reasoning
A question.

Retrieve

Blablabla……

Knowledge Bases

Relevant
Knowledge

Now I can reason better!

Intro|Structured Knowledge|

Un/Semi-structured Knowledge|Parametric Knowledge|Conclusion2

Knowledge Sources🕸 Structured Knowledge

ASCENT++ [2]

ATOMIC [3][1]

📚 Un/Semi-structured Knowledge

GenericsKB [4]🤖 Parametric Knowledge

COMET[7]Intro|

Structured Knowledge|

OpenAI GPTs

[8]

Un/Semi-structured Knowledge|

Parametric Knowledge|

Conclusion

3🕸 Structured Knowledge

Task: Multiple-Choice QAQuestion: xxxxx?

Options: A) a_1, B) a_2, C) a_3

Am I too old?

Symbolic Structures of Knowledge

Neural Language Models (e.g., BERT [9])How can we incorporate structured knowledge into neural language models?
Intro|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge|

Conclusion4Knowledge GraphFeatures
LM encoder to encode the text of question + candidate

Multiple-Choice QAQuestion: xxxxx?

Graph encoder to encode knowledge subgraphs

Candidates: A) a_1, B) a_2, C) a_3

Late fusion of Text Embeddings + Graph EmbeddingsInterpretable ⬇

A related knowledge subgraphextracted from ConceptNet

KagNet [10]Intro|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge|Conclusion

5

GreaseLM [13]MHGRN [11]Multi-hop graph encoder ➡ Scalable

QA-GNN [12]Early fusion of graphs and text encoders
Text embeddings as virtual nodes in graph encoding
Intro|Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge|Conclusion

6📚 Un/Semi-structured Knowledge
KGs can be limited:
1. Incompleteness2. Only for binary relations

Tree
Trees are perennial plants that have long woody trunks.
Most trees add one new ring for each year of growth.
Trees grow using photosynthesis, absorbing carbon dioxide
and releasing oxygen.
…

GenericsKB

Task: Open-Ended QA
Question: xxxxx?
(A target knowledge corpus.)Complex questions need multiple reasoning steps.

Who voices the dog in the TV show Family Guy ?
What can help alleviate global warming?
What will separate iron filings from sand?
Intro|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

7

Dense Passage Retrieval (DPR) [14]

Intro

|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

8

Multi-hop Dense Retrieval (MDR) [15]

Intro

|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

9

Di erentiable Fact-Following Operations (DrFact) [17]GenericsKBffIntro|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

10

Di erentiable Fact-Following Operations (DrFact) [17]ffIntro|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

11

Di erentiable Fact-Following Operations (DrFact) [17]ffIntro|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

12A quick comparison
Methods

BM25

DPR [14] / MDR [15]DrKIT [16]DrFact [17]Concept-Fact
Hypergraph

Knowledge Structure

A set of documents

A set of documents

Mention-Entity
Bipartite Graph

Multi-hop Reasoning
Formulation-- / Multiple-Round

Entity-Following

Fact-Following

Index for Dense Retrieval-Passage Embeddings

Mention Embedding

Fact Embeddings

Sparse Retrieval Method

TF-IDF based Index+
BM25 Ranking Func.

-Entity-Mention
Cooccurrence

Fact-to-Fact Matrix
A single modelw/ Self-Following
Distant Supervision

Multi-Hop Questions-- / Single model

Aggregating
Multiple Models

Intermediate Supervision--N/AIntro|

Structured Knowledge|

Un/Semi-structured Knowledge |Parametric Knowledge|

Conclusion

13📖 Textual Knowledge
🕸 Structured Knowledge📚 Un/Semi-structured Knowledge

We are very accurate, easy to modify,trustworthy, and veri iable; However,
we are incomplete & hard to query!

🤖 Parametric Knowledge
We are super generalizable and easy toquery! But, we often hallucinate andmake mistakes.
GPT-x|

Structured Knowledge
fIntro|

Un/Semi-structured Knowledge|

Parametric Knowledge| Conclusion

14• We can be very expensive.
• You have no idea about our data/weights.
GPT-x

Symbolic Knowledge Distillation [19]Intro|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge| Conclusion

15

Generated Knowledge Prompting (GKP) [20][22]

Input: The word children means <mask> or more kids.
Knowledge: The word child means one kid. The wordchildren is the plural form of the word child.
Prediction: two

Intro

|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge| Conclusion

16

Reinforced Knowledge Introspector (Rainier) [21]PPOGenerated Knowledge Prompting(GKP)• A tripod is a kind of easel.
• A human is a kind of easel.

Intro|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge| Conclusion

17

Conclusion

Skipped. See website ⬇

GKP [20]

QA-GNN [12]MHGRN [11]GreaseLM [13]DPR [14]

MDP [15]DrKIT [16]DrFact [17]WOW!

SKD [19]

ReCross [18]KagNet [10]Rainier [21]

📎 Reference is on the website: https://yuchenlin.xyz/
Intro|

Structured Knowledge|

Un/Semi-structured Knowledge|

Parametric Knowledge|Conclusion

18

Future Directions
1⃣ How can we merge all these di erent sources of knowledge
into a uni ied knowledge model?

2⃣ How can we deliver a more faithful and interpretablereasoning models with low cost?

3⃣ How do we collect and ground knowledge ofreal world and social interactions between humans?

Structured Knowledge|

Un/Semi-structured Knowledge
ff|
fIntro|Parametric Knowledge|

Conclusion

19