Semantic Parsing in the Era ofLarge Language Models
Rui Zhang, Assistant Professor, Penn State University
UPenn CLunch on Oct 31, 2022 …...Happy Halloween!

1Semantic Parsing for Many Things

Text-to-SQL for Natural Language Interfaces to Databases

Instruction Following for Robotics

Question Answering over Knowledge Graphs

Language-to-Code Generation2

Semantic Parsing History: From Leibniz to Symantec
Leibniz (1685) developed aformal conceptual language,
the characteristica universalis, foruse by an automated reasoner,
the calculus ratiocinator.

Richard Montague (1970) developeda formal method for mapping natural
language to FOPC using Church’s
lambda calculus.

Bertrand Russell and Alfred North
Whitehead (Principia Mathematica, 1913)
finalized the development of modern
first-order predicate logic (FOPC).

Figure Adapted from Ray Mooney

Dave Waltz (1975) developed the next
NL database interface (PLANES) toquery a database of aircraft
maintenance for the US Air Force.

Bill Woods (1973) developed the first NL
database interface (LUNAR) to answer
scientists’ questions about moon rocks 12 usinga manually developed Augmented Transition
Network (ATN) grammar.

Gary Hendrix founded Symantec (“semantictechnologies”) in 1982 to commercialize NL
database 14 interfaces based on manuallydeveloped semantic grammars, but they switchedto other markets when this was not profitable.
3

Learning-based Semantic Parsing Research
Zelle and Mooney (1996)
Dong and Lapata (2016)

Zettlemoyer and Collins (2005)
Jia and Liang (2016)

Hwang et al. (2019)

Wang et al. (2019)4

Semantic Parsing Research: Paradigm Shifts
Zelle and Mooney (1996)
Dong and Lapata (2016)

Zettlemoyer and Collins (2005)
Jia and Liang (2016)

Non-Neural Network

E2E Neural Networks

Hwang et al. (2019)Wang et al. (2019)

Contextualized Embeddings andPretrained Language Models

5Delicate Data Curation

https://yale-lily.github.io/spider

6Overspecialized Model Design

Hand-Crafted Rules and Features (Zettlemoyer and Collins, 2005)

Customized Decoders to Follow Grammar (Yin et al., 2017)

Single Utterances vs Conversations (Suhr et al., 2018)

Handling Different Forms of Data (Yi et al., 2018)7

Pretrained Language Models are Getting Bigger

Figure Adapted from Myle Ott8

Large Language Models Open a New Era for NLP Researchhttps://beta.openai.com/examples

9Focus on this Talk: How Large Language Models Help Semantic Parsing?
Zelle and Mooney (1996)
Dong and Lapata (2016)

Zettlemoyer and Collins (2005)
Jia and Liang (2016)

Non-Neural Network

E2E Neural Networks

Hwang et al. (2019)

Wang et al. (2019)

Contextualized Embeddings andPretrained Language Models

LLMs

10

Large Language Models Help Semantic Parsing in Three Ways
Unified Semantic Parsing Framework
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text
Language Models, EMNLP 2022

Semantic Parsing
Multilingual Semantic Parsing
XSemPLR: Benchmarking Cross-Lingual Semantic Parsing in Multiple Languages andMeaning Representations, work in progress 2022
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual
Text-to-SQL Semantic Parsing, Findings of EMNLP 2022

Large Language Models
Semantic Parsing for Numerical and Logical Reasoning
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data, ACL
2022
FOLIO: Natural Language Reasoning with First-Order Logic, arXiv 2022
11Unified Semantic Parsing Framework
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text
Language Models, EMNLP 2022

Semantic Parsing
Multilingual Semantic Parsing
XSemPLR: Benchmarking Cross-Lingual Semantic Parsing in Multiple Languages andMeaning Representations, work in progress 2022
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual
Text-to-SQL Semantic Parsing, Findings of EMNLP 2022

Large Language Models
Semantic Parsing for Numerical and Logical Reasoning
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data, ACL 2022
FOLIO: Natural Language Reasoning with First-Order Logic, arXiv 2022

12UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge
Grounding with Text-to-Text Language Models
Xie et al., EMNLP 2022
https://github.com/hkunlp/unifiedskg

13

Many NLP Tasks need Structured Knowledge

14

Unified Structured Knowledge Grounding

15

Unify 21 Tasks Across 6 Task Families

16Unified Architecture using Text-to-Text Pretrained Language Models

17Unified Architecture using Text-to-Text Pretrained Language Models

18One Architecture, Multiple SOTA

19

Unified Semantic Parsing Framework
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text
Language Models, EMNLP 2022

Semantic Parsing
Multilingual Semantic Parsing
XSemPLR: Benchmarking Cross-Lingual Semantic Parsing in Multiple Languages andMeaning Representations, work in progress 2022
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual
Text-to-SQL Semantic Parsing, Findings of EMNLP 2022

Large Language Models
Semantic Parsing for Numerical and Logical Reasoning
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data, ACL 2022
FOLIO: Natural Language Reasoning with First-Order Logic, arXiv 202220XSemPLR: Benchmarking Cross-Lingual Semantic Parsing inMultiple Languages and Meaning Representations
Zhang et al., Work in Progress

21

Multilingual Information Access(Data from Technology report: Usage statisticsof content languages for websites)

User language preferences online (EU)22

Multilingual Semantic Parsing

Multilingual Semantic Parsing

23

Multilingual Semantic Parsing
Databases

Knowledge Graphs

Virtual AssistantsMultilingual Semantic Parsing
Smart Home Devices

SQL, SPARQL, GraphQL, Lambda-Calculus
Functional Query Language, Prolog
Intent and Slot, ThingTalk Query Language

Human-Robot Interaction

Code Generation
24

Multiple Input Languages, Multiple Output Meaning Representations25

mT5 is the Best26

Multilingual Multitask Learning

27

Multilingual Multitask Learning Helps28

Transfer Learning

29

Need Better Zero/Few-shot Learning for Other Languages

30

XRICL: Cross-lingual Retrieval-Augmented In-Context Learning
for Cross-lingual Text-to-SQL Semantic Parsing
Peng Shi, Rui Zhang, He Bai, Jimmy Lin

Findings of EMNLP 2022

31

In-Context Learning:
Learning from Examples without Parameter Updates

Figure from http://ai.stanford.edu/blog/in-context-learning/32

Our Task: Cross-lingual Text-to-SQL
Databases

IdTourney

1

AustralianOpen…
Year Winner id…
…
2018
3…

RankinPoints Player id Toursg

19,985

3

11

……
……

Multilingual User Questions

AnswerEnglish: Which European
countries have players whowon the Australian Open atleast 3 times?

Switzerland, Serbia, German

SQL Execution

Spanish: Qué países
europeos tienen jugadoresque ganaron el Abierto deAustralia al menos 3 veces?

Text-to-SQL
Id

Name

Nation Continent … …

1

DjokovicSerbia

Europe……

23OsakaJapan
Asia
Federer Switzerland Europe……
……

Chinese: 哪些欧洲国家有至少赢得过 3 次澳网冠军的球员?

SELECT T1.nation
FROM players AS T1 JOIN matches AS T2
ON T1.id = T2.winner_id
WHERE T2.Tourney = “Australian Open”
AND T1.continent = “Europe”
GROUP BY T2.winner_id
HAVING COUNT(*) >= 3Arabic: ‫ﻣﺎ ھﻲ اﻟدول اﻷوروﺑﯾﺔ اﻟﺗﻲ‬‫ﻟدﯾﮭﺎ ﻻﻋﺑون ﻓﺎزوا ﺑﺑطوﻟﺔ أﺳﺗراﻟﯾﺎ‬‫ ﻣرات ﻋﻠﻰ اﻷﻗل؟‬3 ‫اﻟﻣﻔﺗوﺣﺔ‬33

Our Goal
Our Goal
1. Use In-Context Learning with LLMs for Cross-lingual Semantic Parsing
2. Have only English Annotations of Text-to-SQL pairs.

34

Our Goal, Challenges, and Solutions
Our Goal
1. Use In-Context Learning with LLMs for Cross-lingual Semantic Parsing
2. Have only English Annotations of Text-to-SQL pairs.
Challenges
1. Find most relevant English examples.
2. Facilitate translation.

Solutions→→Cross-lingual Retrieval
Translation-based Prompt

35

XRICL Framework

36

Prompt Examples in XRICL

37

Two New Benchmarks
XSpider:English, Chinese, Vietnamese, Farsi, Hindi

XKaggle-DBQA: English, Chinese, Farsi, Hindi

38

Cross-lingual Exemplar Retriever39Translation-Augmented Prompts

40Unified Semantic Parsing Framework
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text
Language Models, EMNLP 2022

Semantic Parsing
Multilingual Semantic Parsing
XSemPLR: Benchmarking Cross-Lingual Semantic Parsing in Multiple Languages andMeaning Representations, work in progress 2022
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual
Text-to-SQL Semantic Parsing, Findings of EMNLP 2022

Large Language Models
Semantic Parsing for Numerical and Logical Reasoning
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data, ACL 2022
FOLIO: Natural Language Reasoning with First-Order Logic, arXiv 2022

41

MultiHiertt: Numerical Reasoning overMulti Hierarchical Tabular and Textual Data
Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang

https://github.com/psunlpgroup/MultiHiertt
ACL 2022

42

Numerical Reasoning over Text

(Roy and Roth, EMNLP 2015)43

Numerical Reasoning over Tables(Roy and Roth, EMNLP 2015)(Cheng, ACL 2022)

44

Our Research Question: Numerical Reasoning from Both?

Numerical Reasoning over Tabular and Textual Data
45

Both Tables and Text

46

Complex Table Hierarchy

47

Fact Retrieval Across Tables and Text

48

Fact Retrieval Across Different Tables

49

Multi-hop Reasoning byGenerating Arithmetic Equations

50

MT2Net Model51

More Challenging with More Tables and Reasoning Steps52

FOLIO: Natural Language Reasoning withFirst-Order Logic
Han et al., arXiv 2022

https://github.com/Yale-LILY/FOLIO

53

Logical Reasoning: Deductive vs Inductive

54

Syllogism
Major premise: All men are mortal.
Minor premise: Socrates is a man.
Conclusion:

Therefore, Socrates is mortal.

55

Syllogism by Venn Diagram
Major premise: All men are mortal.
Minor premise: Socrates is a man.
Conclusion:

Therefore, Socrates is mortal.

M: men

S: Socrates

P: mortal

56

Syllogism by First-Order Logic
Major premise: All men are mortal.

∀x Men(x) ⇒ Mortal(x)Minor premise: Socrates is a man.

Men(socrates)

Conclusion:

Mortal(socrates)

Therefore, Socrates is mortal.

57

FOLIO: A New Dataset for Natural Language Reasoning with First-Order Logic

58

FOLIO is Different59Large Language Models as Soft Logic Reasoners[8 NL-Label Examples] Premises: Allmen are mortal. Socrates is a man.
Conclusion: Therefore, Socrates ismortal. True, False, or Unknown?

GPT-3

Output60

Large Language Models as FOL Semantic Parsers[8 NL-Label Examples] Premises: Allmen are mortal. Socrates is a man.
Conclusion: Therefore, Socrates ismortal. True, False, or Unknown?

GPT-3[8 NL-FOL Examples] Premises: All men
are mortal. Socrates is a man.
Conclusion: Therefore, Socrates ismortal. FOL isCodex

Output∀x Men(x) ⇒Mortal(x)Men(socrates)
Mortal(socrates)

FOL Inference EngineOutput

61

FOLIO is Challenging for Large Language Models62

Conclusions
Unified Semantic Parsing Framework
Semantic Parsing

Multilingual Semantic Parsing
Semantic Parsing for Numerical and Logical Reasoning

Large Language Models63

Future Directions
Unified Semantic Parsing Framework
Semantic Parsing

Multilingual Semantic Parsing
Semantic Parsing for Numerical and Logical Reasoning
Interactive Semantic Parsing for New Domains / New
Meaning Representations (e.g., Iyer et al., 2017)

Large Language Models

Multimodal Semantic Parsing for jointly incorporatingtable/text/images (e.g., Pix2Struct, Lee et al., 2022)
Pretraining for Semantic Parsing for Reasoning andStructured Knowledge (e.g., TAPEX, Liu et al., 2021)
64Thanks! Any Questions?

65