SQL and Large Language Models:
A Marriage Made in Heaven?
Paolo PapottiInternational Workshop on Databases and Machine Learning – 13th May 2024

SQL and LLMs?
Input :Query, Schema

NL Prompt/Question“Engine” :Relational DBMS

Neural Net

Output :Table

NL Text

2

SQL and LLMs Vows
“I will help your users write SQL queries”[Veltri et al, ICDE 2023]“I will help your users benchmark data tasks”“We will answer queries jointly”
3[Papicchio et al, NeurIPS 2023][Saeed et al, EDBT 2024]

User Input:
NL Question

Relations

Table QA

Storage:

Documents

Question answering(QA)

SQL Query

Query Execution

Semantic Parsing
Table Retrieval
Fact Checking

4[Badaro et al, 2023]

Semantic Parsing
Please translate in SQL query:“Give me all the employees withsalary above 2k”“Select nameFrom Emp
Where salary>2000”

for the schema

Emp(name, age, salary)•Text to SQL: example of NL text to code•

LLMs do very well… according to results on public benchmarks
5

Spider: Semantic Parsing and Text-to-SQL Challenge•Manually annotated corpus [EMNLP 2018]
5.7k (NL Question, SQL query) on 200 databases

https://yale-lily.github.io/spider
6

Can we adopt these models?
•Solutions are validated on public benchmark•

Risks:••

Overfit – systems optimized for queries in this dataset•

Contamination - examples are on the Web

What if I need to pick a model for my proprietary data?
Will it work? How well?
7

Custom benchmark on user data•

Given proprietary table D•

Automatically rank existing LLMs on D for SM

User data

Problem for any tabular data task with (NL text, tabular data)
8Table Question Answering
Please give me all the employeeswith salary above 2k sorted by
name“Laure, Mike”

for dataset:
Emp(name, age, salary)(Mike, 33, 2900)(Laure, 45, 3200)(John, 21, 1900)•

LLMs can do it… according to some papers•

No established benchmark
9Custom benchmark on user data•

Given proprietary table D•

Automatically rank existing LLMs on T for data-task

User data

Table QA
Semantic Parsing

10

QATCH: Query-Aided TRL Checklist
Given proprietary data D and task T••

Create a set of tests QT on D (NL question, result GT data)•Measure the quality of LLMs on QT and DNLquestion

GenerateTests

Predict
Model Output

ComputeMetrics

Input data
Ground Truth
QATCHLLM
11

How to get ‘good’tests?

SQL to the rescue•

Query generation to create (NL question, result GT data) pairs•Focus on query complexity: 1 to n attributes/conditions, …

•

Simple text: no ambiguity, no failure, plain English

Input
data D12

NL question

Ground Truth =SQL (input data D)

QATCH: Query-Aided TRL Checklist•Given proprietary data D and task T•

Create a set of tests QT on D (NL question, result GT data)•Measure the quality of LLMs on QT and D
NL question

Predict

GenerateTests

Model Output

ComputeMetrics

Input data
Ground Truth

X

Sim(SQL script 1, SQL script 2)
QATCH

LLM
13

Focus on data

TQA - ChatGPT
Results for TQA

Proprietarydatasets

Failure!

ECOMMERCE

14

TQA - all tests, models
Results for TQA
Tapas, Tapex,
OmniTab: Fine-tuned
Tabular LMs (TRL)

Synthetic examples
effective for test onproprietary datause them for domainspecific fine tuning[ongoing]

15

Fine tuning would fix it?
•fine-tune GPT-3.5 and ChatGPT using 18 table-tasks- 3.2M tables, 1k training examples per task

TQA

SP

Table-GPT [Li et al, 2023]

16

SP - all tests, models
Results for SP

Promising results!
With simple text

17

Data-Ambiguous Questions“Is Curry the best shooter in NBA?”
It depends

TRUE

shooterPlayer

Team

FG%

3FG%

Apps

t1

CurryGSW

48.0

44.7

826

t2

CurryNets

47.7

43.9

377

t3

Jordan76ers

67.3

8.3

780

18[Veltri et al, ICDE 2023]

SP - all tests, models
Results for SP

NL text with attribute ambiguity,avg over 13 datasets

Simple NL text without data ambiguity
Evaluating Ambiguous Questions in Semantic Parsing [Papicchio et al, 2024]

19

SQL and LLMs Vows
“I will help your users write SQL queries”[Veltri et al, ICDE 2023]“I will help your users benchmark data tasks”“We will answer queries jointly”
20[Papicchio et al, NeurIPS 2023][Saeed et al, EDBT 2024]

SQL•

Powers- scalable and cheap  big data- declarative  expressive and precise- logic and relational model exact relations•Data Applications- rich analytics- data warehouse- data collection/preparation for ML on
structured data (prediction)

21

LLM
What are the top 5 Indian cities
with population higher than allother cities with bigger size?

•Powers- self supervised  pre-training- transfer learning  easy to customize- memorization  factual knowledge•Text Applications- text analysis/creation/processing (QuestionAnswering)- chatbots/virtual assistant/code-…

Pre-training
Text doc

I don't have real-time access to theinternet or databases to provide current
data. However, you could obtain such datafrom a reliable source such as the World
Factbook by the CIA, World Bank
datasets, or WHO databases•Data Applications
22

23

User Input:

Storage:

NL Question
Documents

SQL Query

Question answering(QA)

Query Execution

Relations

Table QA
Semantic Parsing

24Applications
RAG, e.g., Llama3 with DB ofembeddings from PDFs

• Hybrid querying

Enterprise own resources

DBLLM

SQL

SELECT c.researchTopic, AVG(e.salary)
FROM LLM.Employees c, DB.Employees e
WHERE c.eid = e.eid
GROUP BY c.researchTopic
25

Galois: SQL querying LLMs•

Input: SQL,arbitrary schema with key•

Storage: LLM•

Output: Relation

26

Challenges•

LLMs store factual data, but•

Input: Not trained to execute SQL faithfully•

Engine: Struggle with complex tasks•

Output: Not trained to (precisely) return relations

LLM

SQL
27

ErrorsQuery processing in 1 slide
Relational Algebra

SQL Querytree of thought

SELECT S.name
FROM Reserves R, Sailors S
WHERE R.sid = S.sid
AND R.bid = 100
AND S.rating > 5

Query Parser

bid=100⋀rating>5(⋈R.sid=S.sid Sailors))(Physical) Query Plan:

𝜋𝜋S.name

𝜎𝜎R.bid=100 ⋀ S.rating > 5
R.sid=S.sid

Reserves

S.name(Reserves(Logical) Query Plan:

⋈

𝜋𝜋 𝜎𝜎

Sailorstables by construction

will produce…

𝜋𝜋S.name

𝜎𝜎S.rating>5⋈

R.sid=S.sid

Operator Code
B+-TreeIndexed Scan
Iterator

𝜎𝜎R.bid=100
Reserves

Sailors

On-the-fly

Project Iterator
On-the-fly

Select Iterator
Indexed Nested
Loop Join Iterator
Heap ScanIterator

Query processing in 1 slide
Relational Algebra

SQL Querytree of thought

SELECT S.name
FROM Reserves R, Sailors S
WHERE R.sid = S.sid
AND R.bid = 100
AND S.rating > 5

Query Parser

bid=100⋀rating>5(⋈R.sid=S.sid Sailors))(Physical) Query Plan:

𝜋𝜋S.name

𝜎𝜎R.bid=100 ⋀ S.rating > 5

will produce…

Sailorstables by construction

𝜋𝜋S.name

𝜎𝜎S.rating>5⋈R.sid=S.sid

R.sid=S.sid

Reserves

S.name(Reserves(Logical) Query Plan:⋈𝜋𝜋 𝜎𝜎Access LLM

𝜎𝜎R.bid=100
Reserves

Sailors

DBMS

Access LLM
DBMSAccess
LLM

NL prompts

Physical Query Plan30

Physical Query Plan31

Factuality•

Decoder returns next token based on trainingdata•

Such token may be based on either reliableacquired knowledge, or it may be a guess hallucinations+ Models keep increasing the factuality of theiranswers*+ Encouraging results from Galois*“GPT-4 scores 40% higher than GPT-3.5 on our factuality evaluations”
32

https://github.com/vectara/hallucination-leaderboard

Experiments - data•

•Corpus of 46 SQL “reasonable” queries/questions from Spider (200datasets)•No: “How many heads of the departments are older than 56?”

•Yes: “What are the names of the countries that became independent
after 1950?”

Tested 4 LLMs: GPT-3 and ChatGPT better than Flan based33

Experiments – QA as “upper bound”
QE

LLM
DBSQL
q1

A
a1
a2
a3

B
b1b2b3

C
c1c2
c3

SQL
q1

D
d1d2d3Ground truth

RM
A
a1
a2a3

B
b1b2

A
a1a3

B
b4

C
c1
c2
c3

Dd5d2

C D E
c1 d1 e1c3 d3 e3

QA

LLM

QA(q1)

TMtext34

Results ChatGPT•

Similarity in output results between ground truth and•our method RM (SQL queries)•

manually parsed traditional TM (NL questions)

QE

35

QA

Error analysis•

Different formats:join country code “IT" with “ITA” for entity Italy•

Entity linking: “Brussels” vs “Bruxelles”•

Verbose output: “The city of Paris”•ChatGPT trained to output NL text adhering to human preferences36

Next Steps
Query optimization•

Physical: reduce hallucinations
 prompts using data examples
 Reconfidencing [Chen et al, 2024]•Logical: Reduce LLM calls  pushdown selections (“get names of cities
with > 1M population”)•Optimize cost, quality.. Without
metadata/catalog

37

Open Questions•• LLM first:

DB first:
use LLM in operators – Galois

Consuming structured data in pretraining, extensions, fine tuning…. But
fine tuned ChatGPT obtains only 0.53
accuracy for TQA[Jo and Trummer, 2023], [Urban et al, 2023](Physical) Query Plan:DBMS

𝜋𝜋S.name

𝜎𝜎S.rating>5⋈R.sid=S.sid

AccessLLM

𝜎𝜎R.bid=100

Sailors[Badaro et al, 2023] [Li et al, 2023]

Access LLM• LLMs + Agents?
DBMS

SP better results than TQA
 Use LM for NLU, SQL/code fordata operations

Access
LLM[Arora et al, 2023]

Reserves

38

SQL and LLMs?

https://github.com/enzoveltri/pythia

http://www.eurecom.fr/~papotti/@paolopapotti

https://github.com/spapicchio/QATCH

https://gitlab.eurecom.fr/saeedm1/galoisInternational Workshop on Databases and Machine Learning – 13th May 2024

Solution• LLMs store factual data, but• Input: Not trained to execute SQL faithfully use simple NL prompts to get data

• Engine: Struggle with complex tasks

 chain of thought* with simple tasks• Output: Not trained to return relations

 tables by construction as in DBMS

* breaking a problem down into intermediate reasoning steps increases LLM abilities40

If only LLMs had SQL powers…
DB/DW“Paolo works at Eurecom andhis colleague Pietro leads theDS dept…”

Excel …
ML

QueryUnsup.
Train

LLM

Query

Data applications: we could immediately query text documents!
41

Evaluate on output data
1.

Benchmark multiple tasks: QA output is data

2.

Data comparison enables accurate metrics for SP: execute correct SQL andgenerated SQL on D, compare data outputs

X

Sim(SQL script 1, SQL script 2)

42How do LLMs work? Big Picture1- Develop LM through pre-training using large unlabeled text corpora

Transformers

2’- Describe a task (with examples) inthe prompt of the LLM

2- Fine-tune LM using (small) labeledtraining data for target application

NeutralTransformer
Based LM

FineTunedLM

3- Given a new paragraph, predict sentiment
Fine-TunedLM

Language Model

NeutralPlease label thesentiment in thefollowing text

LLM

NeutralReconfidencing [Chen et al, 2024]

Background
Query LLM with SQL queries. Different from•

SP: translate NL questions to SQL
SP

•

TQA
QA on tabular data: querying a relation with NL questions•Neural DBs: textual facts encoded with a transformer and NLquestions [Thorne et al., 2020]
QA45

References•

Papicchio et al, QATCH: Benchmarking SQL-centric tasks with Table Representation Learning Models on YourData. NeurIPS 2023•

Saeed et al, Querying Large Language Models with SQL. EDBT 2024•

Saeed and Papotti, You Are My Type! Type Embeddings for Pre-trained Language Models. EMNLP 2022

•Thorne et al, From Natural Language Processing to Neural Databases. Proc. VLDB Endow. 2021

•Veltri et al, Data Ambiguity Profiling for the Generation of Training Examples. ICDE 2023

•Yu et al, Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing andText-to-SQL Task. EMNLP 2018•

Badaro et al. Transformers for Tabular Data Representation: A Survey of Models and Applications TACL 2023
48

SQL and LLMs Vows
“I will help your users write SQL queries”“I will help your users benchmark data tasks”[Papicchio et al, NeurIPS 2023]“We will answer queries jointly”[Saeed et al, EDBT 2024]
56