Evaluating Ambiguous Questions in Semantic
Parsing
Simone Papicchio, Paolo Papotti, Luca Cagliero

International Workshop on Databases and Machine Learning – 13th May 2024

Semantic Parsing
Please translate in SQL query:“Give me all the employees withsalary above 2k”“Select nameFrom Emp
Where salary>2000”

for the schema

Emp(name, age, salary)•Text to SQL: example of NL text to code•

LLMs do very well… according to results on public benchmarks
2

Spider: Semantic Parsing and Text-to-SQL Challenge•

Manually annotated corpus [EMNLP 2018]
5.7k (NL Question, SQL query) on 200 databases

https://yale-lily.github.io/spider3

Can we adopt these models?
•Solutions are validated on public benchmark•

Risks:•

Overfit – systems optimized for queries in this dataset•

Contamination - examples are on the Web•

Assumptions – clear and complete questions
4

Today
13:30

Assumptions in Benchmarks vs Reality

What is the size of theAbalone fish with Id 1?
“customers use compact and informal language to interact with our systems”[Microsoft - Floratou et al, CIDR 2024]
5

Related Work•

Analysis: 45% questions attribute ambiguity [Wang et al, EMLNLP 2023]•Ambiguity detection in Semantic Parsing:- fine tuning encoder [Veltri et al, ICDE 2023],- add documentation, data examples with GPT4 [Huang et al, TRL 2023]- GPT4 high agreement with humans [Floratou et al, CIDR 2024]
 Revise the workflow of NL2SQL?

•Top k solutions [Bhaskar et al, EMNLP 2023]
6

What is a good answer?
L, D, H data-ambiguous wrt label “size”

Ranking:
SELECT Length FROM ….
SELECT Diameter FROM ….

What is the size of theAbalone fish with Id 1?

1SELECT Length, Diameter, Height FROM ….
SELECT AbaloneId FROM ….
70

Benchmarking models on tables withattribute ambiguity•

Given a table D with attributes A1, .., An data-ambiguous wrt label L•

Rank existing LLMs on D for Semantic Parsing

Table withambiguity

DAMBER

8

DAMBER: Data-AMBiguity testER•DAMBER built on top of QATCH (Query-Aided TRL Checklist)

DAMBER: Data-AMBiguity testER•QATCH metrics computed ondata outputs•Measure model predictionagainst the best matchingtarget query

Experiments setting•

corpus: 13 tables (UCI repository and WebTables),10 annotators identify ambiguous attributes+label for each pairE.g., “weight” and “height” as ambiguous  label “measure”
1321 attribute pairs, 252 ambiguous pairs•

three TRL models: RESDSQL, GAP, UNIFIEDSKG
two LLMs: CHATGPT 3.5 Turbo and Code-LLAMA

11

Results for Semantic Parsing

avg over 13 tables and all tests• ChatGPT avg results range from 0.98 in WDC_631 to 0.60 in Abalone"length", "diameter", and "height" vs label "distance“• ChatGPT returns all relevant attributes when faced with uncertainty: higher recall
than precision, struggle with aggregate queries
12Evaluating Ambiguous Questions in Semantic Parsing• Semantic Parsing is a maturetechnology… under assumptions
common in benchmarks• Attribute Ambiguity affects the qualityof the results• Keep exploring the impact of othertypes of ambiguity
https://github.com/spapicchio/QATCH

http://www.eurecom.fr/~papotti/@paolopapottiInternational Workshop on Databases and Machine Learning – 13th May 202414

Evaluate on output data
1.

Benchmark multiple tasks: QA output is data

2.

Data comparison enables accurate metrics for SP: execute correct SQL andgenerated SQL on D, compare data outputs

X

Sim(SQL script 1, SQL script 2)

15

References•

Papicchio et al, QATCH: Benchmarking SQL-centric tasks with Table Representation Learning Models on Your
Data. NeurIPS 2023•

Saeed et al, Querying Large Language Models with SQL. EDBT 2024•

Saeed and Papotti, You Are My Type! Type Embeddings for Pre-trained Language Models. EMNLP 2022

•Thorne et al, From Natural Language Processing to Neural Databases. Proc. VLDB Endow. 2021

•Veltri et al, Data Ambiguity Profiling for the Generation of Training Examples. ICDE 2023

•Yu et al, Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing andText-to-SQL Task. EMNLP 2018

•Badaro et al. Transformers for Tabular Data Representation: A Survey of Models and Applications TACL 2023
18

User Input:
NL Question

Relations

Table QA

Storage:

Documents

Question answering(QA)

SQL Query

Query Execution

Semantic Parsing
Table Retrieval
Fact Checking

20[Badaro et al, 2023]