BLENDSQL: A Scalable Dialect for Unifying Hybrid Question Answering in Relational Algebra (2024)

Abstract: Many existing end-to-end systems for hybrid question answering tasks can be boiled down to a “prompt-and-pray” paradigm, where the user has limited control and insight into the intermediate reasoning steps used to achieve the final result. Additionally, due to the context size limitation of many transformer-based LLMs, it is often not reasonable to expect that the full structured and unstructured context will fit into a given prompt in a zero-shot setting, let alone a few-shot setting. We introduce BLENDSQL, a superset of SQLite to act as a unified dialect for orchestrating reasoning across both unstructured and structured data. For hybrid question answering tasks involving multi-hop reasoning, we encode the full decomposed reasoning roadmap into a single interpretable BLENDSQL query. Notably, we show that BLENDSQL can scale to massive datasets and improve the performance of end-to-end systems while using 35% fewer prompt tokens. Our code is available and installable as a package at github.com/parkervg/blendsql.

heading: 1 Introduction
text: Problem Decomposition In settings involving both fine-tuning and in-context learning, it has been demonstrated that generating explicit intermediate steps for complex problems can enhance accuracy. In their aptly titled “Scratchpad” framework, Nye et al. (2021) revealed that predicting the complete execution trace led to better results when fine-tuning language models to generate execution output. More recent work has shown that prompting LLMs to decompose complicated problems into smaller reasoning steps before generating the final answer can improve results (Wei et al., 2022; Yao et al., 2023; Wang et al., 2022). However, at their core, these methods use natural language as their intermediate reasoning representation. With BLENDSQL, we argue that natural language alone is inherently a lossy and ambiguous intermediate
reasoning representation. Instead, by decomposing a problem into a SQL-like syntax, we allow for more deterministic reasoning capabilities and better interpretability of intermediate steps by leveraging the compositional nature of relational algebra (Codd, 1970).
Agents Previous works have explored agents: an experts-based framework where the main task is broken down into parts that each agent or expert1 carries out (Parisi et al., 2022; Liang et al., 2022; Schick et al., 2023; Shen et al., 2023; Liang et al., 2023; Lu et al., 2023). Furthermore, these works are based on the presence of a ‘routers’ or ‘planners’ that understand the given task and break it down into smaller tasks. Although rule-based routers have been widely used, the ability of LLMs to generalize to complex texts has enabled their use as
1Here an agent or an expert can be anything from a Python script, an API call, a tool or a language model.
453
planners. LLMs, as planners, process the input and as output generate text containing the list of subtasks (Shen et al., 2023), the agent(s) to use from a given set, what are the inputs to the agent(s) (Schick et al., 2023; Qin et al., 2023; Cheng et al., 2023), what is expected as response from the agent(s) (Parisi et al., 2022). With the scope of the tasks carried out by the agents being very narrow, recent works have primarily focused on the planner component of the framework. For the same, harnessing the in-context learning capabilities of the LLM (Shen et al., 2023; Qin et al., 2023) or finetuning the planner LLM (Parisi et al., 2022; Schick et al., 2023; Patil et al., 2023; Tang et al., 2023; Wang et al., 2023; Lu et al., 2023) have been the two main methods. Additionally, works have used LLMs’ in-context learning capability for dataset creation (Patil et al., 2023; Tang et al., 2023). The parser in BLENDSQL acts as a planner and generates the APIs or Ingredients to use in answering the input question. However, it differs from the previous works as it goes beyond generating a single string representation of complicated problem decomposition and simplifies the agent framework using robust and complicated abstract syntax tree (AST) structures to be encoded into a single string. It is also important to note that BLENDSQL does not just rewrite the input statement using formal constructs (Lu et al., 2021; Wu et al., 2022), it also adds information using the reference database, input prompt, and internal knowledge of the language models.
Text-to-SQL On the widely used text-to-SQL dataset Spider (Yu et al., 2018), many works demonstrate impressive performance in the fewshot or zero-shot setting (Gao et al., 2023; Pourreza and Rafiei, 2023; Dong et al., 2023; Liu et al., 2023). Even in settings where text-to-SQL is not the focus, existing work details the effectiveness of a common and well-understood intermediate representation like SQL for other reasoning tasks. Hu et al. (2022) frames dialogue state tracking (DST) as a text-to-SQL task by encoding domains and slots from MultiWOZ (Budzianowski et al., 2018) into a serialized database format and using SQL as an intermediate representation, showing significant improvement over the “traditional” key-value style prediction format.
Hybrid Question Answering Unlike text-toSQL, hybrid question answering involves an im-
plicit decision to access tabular data, unstructured data, or both (Chen et al., 2020; Zhu et al., 2021; Chen et al., 2021). Li et al. (2021) make this more explicit by routing between end-to-end models and a parser that generates an intermediate SQL representation. Their implementation frames the two as isolated, specialized systems which are unable to pass information to one another.
For the scope of this work, we define Hybrid Question Answering as answering questions using a corpus of tabular and unstructured multi-modal content. While this multi-modal content can encompass text, images, audio, and video, the primary focus of this paper is on unstructured text in conjunction with tabular data. The potential to adapt this process to other formats, such as images, is also discussed in Appendix C.
One prior work, UniK-QA (Oguz et al., 2022), proposes converting structured tabular data into unstructured text data. Our work does the opposite, and instead converts all unstructured text data into tabular data. Empirically, UniK-QA improves on several knowledge-base (KB) QA benchmarks using a KB-to-text preprocessing model. For example, for the KB-to-text model mentioned in the UniK-QA paper, the relationship between different entities is assumed to have a simple triplet representation: <subj, pred, obj> where pred (predicate) defines the relationship from the table name. However, relationships between <subj> and <obj> are often complicatedly embedded in heterogeneous information sources (e.g., audio, documents, or column names of the table), which would then demand more efforts in developing dedicated KB-to-text models.
Neuro-Symbolic Frameworks Most similar to our work is the BINDER method presented in Cheng et al. (2023), which integrates LLM reasoning into symbolic languages like SQL and Python. While we take inspiration from their neurosymbolic framework for reasoning, BLENDSQL differs in the following ways. (1) We create an API specialized for SQL, enabling reasoning over multi-table databases containing unstructured content, and are capable of executing and optimizing queries containing JOIN statements, aliases, conditional table expressions, and most other SQLite operations (2) We enable constrained decoding according to the database schema (3) We design our API such that users are able to easily create any number of custom functions themselves to use
within a BLENDSQL script.

heading: 1.1 Main Contributions
text: In summary, we make the following contributions.
• To the best of our knowledge, we are the first to propose framing the context of hybrid question answering as a relational database.
• We introduce a new open-source query language, BLENDSQL, to orchestrate and optimize hybrid functions across SQL logic and LLM reasoning.
• We demonstrate that with only a small number of in-context exemplars, BLENDSQL can outperform end-to-end methods using 35% fewer tokens and without direct access to unstructured context.

heading: 2 BLENDSQL: Overview
text: We implement BLENDSQL as a superset of the SQLite relational database management system (RDBMS)2. An executed BLENDSQL script returns a smoothie object as output, containing the final result and intermediate reasoning steps taken. To make a smoothie, we need some ingredients and a blender. We describe these components below.

heading: 2.1 Blender vs. Parser
text: We use the term “blender” to describe the LLM which receives the prompts used to perform each ingredient function within a BLENDSQL script. Prior to execution, the “parser” receives a set of few-shot examples and generates a BLENDSQL query given a question and database context. We use GPT-4-0613 (Achiam et al., 2023) as both the blender and the parser for our core experiments.

heading: 2.2 Ingredients
text: Ingredients are at the core of a BLENDSQL script. They are subprograms used to pass certain logical operations through LLM-based functions, and are denoted by double-curly braces (“{{” and “}}”). BLENDSQL syntax is represented as a parsing expression grammar, implemented via PyParsing3 (McGuire, 2007). Like vanilla SQL, ingredient calls are fully recursive, and the context passed to
2https://www.sqlite.org/ 3https://github.com/pyparsing/
pyparsing
one can include operations invoking other BLENDSQL ingredients.
LLMMap The LLMMap ingredient is a unary scalar function, much like LENGTH or ABS in standard SQLite. The output of this operation is then set as a new column in a temporary table, for later use within the wider query. Specifically, it accepts table and column arguments in the string format “table::column”. It then creates a new column c′ with mapped values v′, which are the output of the scalar function. With nc as the number of values in the target column c, we get the following.
V = {vi,∀i∈{1,...,nc}} fmap(V) → {v′1, ..., v′nc}
For example, Figure 2 depicts the LLMMap ingredient in action. For each value, an LLM is prompted to answer the question: “Is this a team event?” Given the diversity in how team events are denoted in this column (“team event”, “4x100 medley relay”, etc.), it becomes difficult to write SQLite logic for a generalized solution across the entire table via pure string parsing. LLMMap, instead, implements a generalized solution to transform values based on their semantic denotations.
LLMQA The LLMQA ingredient is an aggregate function that transforms a table subset T into a single scalar value v′.
fqa(T ) → v′
As shown in Figure 1, this ingredient may be restricted with the options argument, which ensures that the output of the function will be an existing value in a specific column c, v′ ∈ {vi,∀i∈{1,...,nc}}4. In Figure 2, the LLMQA function receives a piece of unstructured document context, and the question: “Which NBA season was suspended due to COVID-19?” Additionally, it receives a set of column values to use in constraining the language model’s generation.
For fact verification tasks, we implement a modified version of this ingredient, LLMValidate, which always returns either a true or false boolean.
LLMJoin The LLMJoin ingredient creates a custom mapping between a pair of value sets V 1 and
4This constrained decoding is implemented using guidance, which manipulates the logit_bias argument in the OpenAI API.
V 2. This mapping is then used to create an auxiliary table to carry out a SQL INNER JOIN operation.
fjoin(V1,V2) → {(v1′1, v2′1), ...(v1′nv1, v2′nv2)} This ingredient is particularly useful in situations when proper foreign keys do not exist, but some semantic alignment is still possible. In Figure 2, “joshua fields” and “josh fields (pitcher)” share the same referent. We use the LLMJoin ingredient to align the two distinct references, effectively performing a form of entity linking.

heading: 2.3 Query Optimizations
text: When possible, we execute all SQL predicates in a subquery first and assign their outputs to a temporary session table. Since native SQLite operations are relatively inexpensive, we ensure that the expensive LLM-based ingredient functions receive no more (and no less) the subset of data required to generate a faithful execution output. As demonstrated below, we only pass the subset of rows from w where w.school = ’university of georgia’ to the LLM-based ingredients in 3. We validate these optimizations with an extensive test suite5.

heading: 3 Datasets
text: Due to the joint usage of SQL-like syntax and LLM-based ingredients, we expect our BLEND-
5https://github.com/parkervg/blendsql/ tree/main/tests
SQL to perform well on hybrid question answering datasets. To validate our hypothesis, we experimented on several challenging benchmark datasets requiring reasoning over both structured and unstructured knowledge.

heading: 3.1 HybridQA
text: Compared to previous QA tasks utilizing only passages (unstructured) or tabular information (structured), HybridQA contains challenging questions whose answers demand heterogeneous forms of information collected from Wikipedia tables and passages (Chen et al., 2020). In an ablation study, the original authors show that a table-only model achieves an accuracy of 8.4 on the dev set, and a passage-only model achieves an accuracy of 19.5. Combining both sources, their baseline model improves drastically to 44.0, highlighting the importance of a model adept at combining both hybrid data sources.

heading: 3.2 OTT-QA
text: The Open Table-and-Text Question Answering dataset (OTT-QA) is a “decontextualized” variant of HybridQA, which requires both text and table retrieval over a large corpus (Chen et al., 2021). On the OTT-QA dataset, we demonstrate the potential of BLENDSQL to act as both the retriever and reader within a unified query language. In the multi-hop reasoning OTT-QA requires, the relevant passages can only be found after completing some operation on the structured tables, and vice versa. We enable this communication between the reader and retriever components via BM25 full-text search enabled with the built-in FTS5 extension7 in SQLite. For example, in Figure 1, we first retrieve unstructured text from the documents table with the highest BM25 relevancy to the query “nba OR covid”. Then, given the structure of the table “Lebron James Career Statistics”, we constrain the possible generations given our unstructured context to only a value occurring in the “Year” column.

heading: 3.3 FEVEROUS
text: The FEVEROUS (Fact Extraction and VERification Over Unstructured and Structured information) dataset contains claims accompanied by context sentences and tables from Wikipedia (Aly et al., 2021). Each claim is classified as either “supports”, “refutes”, or “not enough info”. By using BLEND-
7https://www.sqlite.org/fts5.html
SQL as an intermediate representation, we are able to frame the notion of “truth” as a function over facts found within a database. Additionally, by generating an intermediate representation of the FEVEROUS claim, we not only produce an interpretable decomposition of the implicit truth claims, but we also have a reusable blueprint for future fact verification even if the values in our underlying database are updated. This differs from the traditional end-to-end approach, where the full hybrid context must be passed each time a prediction is made.
BLENDSQL as Predicate Logic BLENDSQL transforms the propositional claims of FEVEROUS into predicate logic, providing a new language to evaluate the truth value of a statement given world knowledge in a relational database. For example, we can take the following (abbreviated) example from the FEVEROUS dev set.
Pesamino Victor (an association footballer) and his team lost in all their in-
ternational matches.
Given the constant p for Pesamino Victor and two-place predicates PLAYSON, PLAYEDBY, and WON, this becomes the following.
∃t(ASSOCIATIONTEAM′(t) ∧ PLAYSON′(p, t)) ∧¬∃m(INTERNATIONALMATCH′(m) ∧PLAYEDBY′(t,m) ∧ WON′(t,m))
Below, we display the BLENDSQL program corresponding to the same statement, given the underlying structured (the Pesamino Victor table) and unstructured (documents) context.

heading: 4 Experiments

heading: 4.1 Dataset Pre-processing
text: In all the datasets we evaluate, we place both structured and unstructured contexts into a SQLite database as shown in Table 1. We use the FTS5 extension8 to create a full-text index with BM25 ranking over the title and content columns in our documents table.
8https://www.sqlite.org/fts5.html
Additionally, we use the fuzzy string matching described in Lin et al. (2020) to align a given question to relevant values in the underlying database, and provide these “bridge” matches as hints in our prompt. We serialize our databases as code, as proven effective for text-to-SQL in the study by Nan et al. (2023). Specifically, we include the CREATE clause for each table in the database, along with n example rows for every table except documents. In many datasets, the unstructured text includes lengthy passages that are often irrelevant to the given question. By withholding the unstructured content from the parser, we are able to scale to massive datasets such as OTT-QA.
End-to-End Prompting on HybridQA On the HybridQA dataset, many data points contain context data that is too large to fit into a single GPT-4 prompt. To solve for this, we include all tabular data, and truncate the values of the unstructured content to a max of 400 characters.

heading: 4.2 Teaching BLENDSQL via In-Context Learning
text: We implement our few-shot prompting via the guidance toolkit9, which supports handlebars-style syntax to control the prompting workflow10. For each dataset, we randomly sample ∼10 examples from the train set and annotate their BLENDSQL representation. For HybridQA and FEVEROUS, we use 12 examples; for OTT-QA, we use 9.

heading: 4.3 Metrics
text: In an attempt to measure the true performance of our approach without overfitting to spurious answer formats present in the free-text question-answering datasets, we employ the denotation accuracy metric used by Cheng et al. (2023), from the script here. This metrics measures semantic “denotation accuracy”, normalizing different output formats (such as “seven” and “7”). On the HybridQA dev set, for example, 101 gold answers contain natural language numbers (five million, three campuses), and 697 gold answers contain numeric representations of numbers (4 million, 6). The official HybridQA evaluator would judge these different answer formats as incorrect, and therefore, we opt for measuring semantic denotation instead.
9https://github.com/guidance-ai/ guidance/tree/0.0.64
10For the full prompt, see Appendix B

heading: 5 Results
text: We direct the curious reader to Table 6 in the Appendix for example predictions and more analysis.

heading: 5.1 HybridQA
text: Unlike in intermediate representations based on natural language, a BLENDSQL script may result in an execution error and fail to produce a response. Additionally, the returned LLMQA ingredient may deem the question as unanswerable given the provided context and respond with something in the spirit of “This table does not provide the necessary info.” In these cases, we experiment with falling back to the end-to-end prompting style as a last resort, as denoted by “BlendSQL + End-to-End” in Table 2.
With 12 few shot exemplars, the parsed BLENDSQL query was unable to generate a prediction on 33% (1,173 examples out of 3,466) of the dev set. Even with this shortcoming, it outperformed the truncated context, end-to-end method by 3.76%. By using BLENDSQL first, and falling back to endto-end prompting, we boost performance by 8.63% to 57.76%, outperforming the oracle document retriever approach of Sui et al. (2023). Notably, on the subset of 2,293 questions that generated a valid BLENDSQL script, we reached an accuracy of 64.43%, suggesting that improved few-shot examples and perhaps more refined ingredients may have the potential to further boost performance.

heading: 5.2 OTT-QA
text: Depicted in Table 3, BLENDSQL allows for competitive performance among existing finetuned benchmarks, with only 9 few-shot examples. Even more promising is BLENDSQL’s use of a naive BM25 document retriever, and ability to encode the unified reasoning roadmap into a single, debuggable query. While we use the predictions from the table retriever described in Ma et al. (2022), it is up to our parser to write a satisfactory FTS5 query for retrieving documents from the BM25 index store. For these reasons (in-context learning, with a naive BM25 retriever), it is difficult to compare our results to existing literature directly. Future work involving a more robust retrieval algorithm (e.g., retrieval with vector embeddings) has the potential to further improve results.
10‡ We follow the previous work in Sui et al. (2023) and only evaluate on claims which include table-based evidence. However, we discard data points with the label “not enough

heading: 5.3 FEVEROUS
text: As shown in Table 4, BLENDSQL does not outperform the end-to-end prompting style on the FEVEROUS dataset. As described in 3.3, this is likely due to the complexity and ambiguity of outlining predicate logic to evaluate the truth value of a given claim, compared to the other hybrid QA tasks. Not only have we prompted our parser to compose a script that evaluates to the correct judgment (“supports” or “refutes”), but we have implicitly asked it to identify the various atomic truth claims made within a claim, and set boundaries for each with respect to database context. Additionally, the table structures in FEVEROUS tend to deviate far from the traditional relational model, with many subtables and empty values.

heading: 6 Error Analysis
text: To better understand the advantages and limitations of BLENDSQL, we annotate randomly sampled 50 BLENDSQL question-answer pairs from HybridQA mistakes. Our chosen denotation accuracy metric judges each datapoint as either a 1 or 0. Out
enough”, making the results not directly comparable. ∗ To normalize different output formats (such as “seven” and “7”), we employ the denotation accuracy metric used by Pasupat and Liang (2015), from the script here. It is unclear if the previous methods use the same evaluation script.
of the 3,466 datapoints in the HybridQA dev set, we see 1,464 with an accuracy score of 0. Although the number 1,464 seems intimidating, it is important to remember that we achieved a literature-comparable score of 57.76 and to realize that there are many false negatives from those errors. To distinguish those false negative errors, we describe Annotation Categories as the parent categories containing false negative errors and the BlendSQL Error Categories as the child categories containing only true negative errors.

heading: 6.1 Annotation Categories
text: Figure 5 shows an overview of the annotation categories, among the 50 mistakes, there are considerable amount of mistakes are false negatives because those BLENDSQL answers are semantically equivalent to the HybridQA’s ground-truth answer. The annotation categories, along with their definitions, are described below:
Lengthy Our LLMQA ingredient tends to provide a more elaborate answer instead of naively an address, name, or date. These elaborate but semantically equivalent answers are annotated as LENGTHY.
Normalization Even with the normalization script of Cheng et al. (2023), some unwanted formatting variations remain. This second source of false negative is due to text normalization such as "$20 millions" versus "20,000,000", "Belize City" versus "belize" and "METREX" versus "Metrex Network". We annotate this second source of false negative as NORMALIZATION.
HybridQA-Centric There are also some groundtruth answers from HybridQA that we believe are partially wrong, ambiguous with multiple answers, or require additional background information not present in the document or the table. We annotate those observations into HYBRIDQA-WRONG, HYBRIDQA-MULTIPLE, and HYBRIDQA-MISSING.
BlendSQL The mistake caused by BLENDSQL only occupies 17 out of the 50 sampled mistakes, amounting to 34% of the true negative rate.

heading: 6.2 BlendSQL Error Categories
text: Focusing on those 17 true negative BLENDSQL mistakes, we further break down the error into 6 major categories with Figure 5.
Parser The PARSER category indicates the incorrect answer is caused by a mistake in BLENDSQL syntax. For example, the “greater than” syntax (WHERE "date" > {{LLMQA(...)}}) is incorrectly generated as “equals” (WHERE "date" = {{LLMQA(...)}}).
Blender-Centric Besides the most common PARSER error, we see five additional categories indicating the direction of the multi-hop reasoning and the topic of the mistakes. Take the following question as an example.
What is the difference in time between Jose Reliegos of Spain and the person born 5 September 1892 who competed at the 1928 Olympics?
The answer requires a full-text search of the athlete name within the document via documents MATCH ’born 5 september 1892’, then a SQL query to calculate the time difference based upon the searched athlete name. In this given example, the error is annotated as BLENDER-DOC>TABLE(DATE).

heading: 6.3 Error Causes
text: A majority of the BLENSQL mistakes came from the parser generation. We find that sometimes the generated BLENDSQL syntax uses LLMQA to answer questions that are better suited for the LLMJoin
operation. Those inappropriate parses hence incur incorrect answers related to date and numerical operation.
Among those blender mistakes, we find the multi-hop reasoning from DOC->TABLE is usually more challenging than TABLE->DOC. The reason is that searching for a potential filter over an unstructured document is more difficult and open-ended than a column filter over a structured table.

heading: 7 Open Model Evaluations
text: Closed models like GPT-4-0613, while powerful, lack a level of transparency and interpretability that is valuable to the research community. To this end, we also evaluate two open-source language models built for coding tasks: DeepSeek-Coder-6.7bInstruct11 (Guo et al., 2024) and StarCoder2-15b12 (Lozhkov et al., 2024). Both models are autoregressive decoders with a context window of 16k tokens. We run experiments on the HybridQA dataset, and use each respective model as both the parser and blender to best reproduce the core experiments with GPT-4-0613.

heading: 7.1 Open Model Error Analysis
text: As shown in Table 5, the two open models fail to match the performance of GPT-4-0613. This can be partly attributed to the ability of the parser to generate valid BLENDSQL syntax: while GPT-4-0613 only generated bad syntax on 3% of the HybridQA dev set, DeepSeek-Coder and StarCoder2 saw rates of 13% and 21%, respectively. We explore specific execution errors yielded by different parser models in Figure 6. In this Figure, we consider everything to the right of “No Results” to be a syntax error. Referring back to section 6.2, these errors are subsets of the PARSER category.
11https://huggingface.co/deepseek-ai/ deepseek-coder-6.7b-instruct
12https://huggingface.co/bigcode/ starcoder2-15b

heading: 8 Future Work
text: We hope to study the ability of LLMs not only to use functions from a previously unseen SQL dialect but also to create functions that generalize certain logical patterns, as described in Cai et al. (2023). Additionally, while instruction fine-tuned LLMs show impressive abilities in executing BLENDSQL scripts, we hope to experiment with more specialized and inexpensive models for modular tasks such as fact verification and DPR retrieval.
Finally, as many errors came from BLENDSQL scripts that were unable to execute properly, we hope to explore hybrid QA tasks as a form of interactive semantic parsing (Elgohary et al., 2020; Glenn et al., 2023).

heading: 9 Conclusion
text: We introduce BLENDSQL, a scalable dialect for problem decomposition and hybrid questionanswering. Results show competitive performance on popular benchmarks while using only ∼10 fewshot examples. Additionally, on the HybridQA dataset, we improve the performance of a naive end-to-end system by 8.63%, while using 45% fewer prompt tokens. We open-source all code and present an installable Python package for future researchers to further explore BLENDSQL as an intermediate representation for hybrid questionanswering.

heading: 10 Limitations
text: As our core experiments were conducted using GPT-4-0613, the API costs associated with reproducing the experiments may be a limiting factor. In an attempt to remedy this, we make our work open source and share all evaluation outputs.
Additionally, while we show that our approach can minimize prompt tokens and improve performance, we require an extra step of pre-processing all context into a SQLite database. We aim to streamline this pre-processing by adding support for more database management systems, and creating helper scripts for transforming a hybrid question-answering context into the appropriate database format.

heading: A Appendix
text: A.1 Inferring Outputs from SQL Syntax Ingredients in BLENDSQL are inherently stateless; they do not receive any external information other
than that which is explicitly passed in as arguments. This approach leads to a view of problem decomposition where each ingredient call is only responsible for the specialized task it’s been setup to solve for.
However, oftentimes, an ingredient’s placement within the larger SQL syntax can provide useful signals for the downstream LLMs. For example, consider the BLENDSQL query below. SELECT * FROM w WHERE {{
LLMMap( 'What state is this city in?', 'w::city'
}} = 'CA'
We have a single LLMMap ingredient call, which will receive all distinct values from the city column. Examining the query, we also understand that we expect a returned string that looks like ’CA’, as opposed to ’California’, or any other equivalent denotation. By traversing the AST and extracting the arguments of predicates like =, >, and <, we are able to inject this prior knowledge into the LLMMap call with a simple flag “Here is an example output: CA”. Additionally, by utilizing open models with publicly accesible logits, we are able to directly enforce string patterns at the decoding level (Willard and Louf, 2023). For example, take the below query. SELECT * FROM w WHERE {{
LLMMap( 'Is this a team event?', 'w::event'
}} = TRUE
By using the same AST traversal logic described above, we can infer that we expect the LLMMap to return a boolean datatype. Using constrained generation, we are able to only produce a response from a language model that matches the regular expression ((true|false);)+13.

heading: B Few-Shot Prompt
text: Below we show the few-shot prompt we used for our core experiments, along with an example database serialized in the “code” format. {{#system~}} Generate BlendSQL given the question to answer
the question correctly. BlendSQL is a superset of SQLite, which adds
external function calls for information not found within native SQLite.
These external functions should be wrapped in double curly brackets.
If question-relevant column(s) contents are not suitable for SQL comparisons or calculations, map it to a new column using the new function: `LLMMap('question', '{table}::{column}')`
13We use a semicolon as a separator here
If mapping to a new column still cannot answer the question with valid SQL, turn to an end-to-end solution using the new function: `LLMQA('{question}', ({blendsql}))`
If we need to do a `join` operation where there is imperfect alignment between table values, use the new function: `LLMJoin(({blendsql}), options='{table}::{column}')`
ONLY use these BlendSQL functions if necessary. Answer parts of the question in vanilla SQL, if
possible. {{~/system}}
{{#user~}} {{few_shot_examples}}
{{serialized_db}} Question: {{question}} BlendSQL: {{~/user}}
{{#assistant~}} {{gen "result" temperature=0.0}} {{~/assistant}}
CREATE TABLE "w" ( "index" INTEGER, "no" INTEGER, "rider" TEXT, "team" TEXT, "motorcycle" TEXT ) /* 3 example rows: SELECT * FROM w LIMIT 3 index no rider team
motorcycle 0 1 carl fogarty ducati performance ducati 996 1 4 akira yanagawa kawasaki racing team kawasaki zx-7rr 2 5 colin edwards castrol honda honda rc45
*/
CREATE VIRTUAL TABLE "documents" USING fts5(title, content, tokenize = 'trigram')

heading: C Hybrid QA over Images
text: The main paper discusses the Hybrid Question Answering task with text as the unstructured media in addition to tabular data. The methodology used for
text can easily be extended for images for Visual Question Answering (VQA) task14, as shown in Bae et al. (2024). This section presents an example of how adding of a VQA Ingredient to BlendSQL allows us to do QA over a different unstructured media (images). In this example, a table and the corresponding image from a Wikipedia page15 are used for answering a question. Figure 7 shows the table from the Wikipedia page containing information about different fountains along with the corresponding image of the same. A tiny VQA model16 is used for answering the question once the correct image has been retrieved. The byte representation of the images is stored in the SQLite database and used for the question answering task.
Below, we show an example natural language to BLENDSQL pairing for this database context. The below query corresponds to the question “How many animals are in the fountain designed by Georgia Gerber?”. When executed, it invokes the VQA model on the retrieved database content to return “There are three animals in the fountain”.
14https://github.com/parkervg/blendsql/blob/main/examples/vqaingredient.ipynb
15https://en.wikipedia.org/wiki/Fountains_in_Portland,_Oregon 16https://huggingface.co/bczhou/tiny-llava-v1-hf

heading: Future Work
text: We open-source all code and present an installable Python package for future researchers to further explore BLENDSQL as an intermediate representation for hybrid questionanswering