Title: UQE: A Query Engine for Unstructured Databases
Venue: NeurIPS 2024
Accepted: yes
Keywords: unstructured data

sampling and optimization

database

language models
Abstract: Analytics on structured data is a mature field with many successful methods.
However, most real world data exists in unstructured form, such as images and conversations.
We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics.
In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators.
The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution.
In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls.
We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation.

ˇ

Abstract

Analytics on structured data is a mature field with many successful methods. However, most real world data exists in unstructured form, such as images and conversations. We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics. In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections. This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators. The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution. In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls. We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation.

1 Introduction

Data analysis [13] is essential for making well founded decisions and enabling businesses and society to function more effectively. Relational databases [12, 32] and the Structured Query Language (SQL) [7] have delivered huge successes in structured data management and analysis. Typically, such data is collected and organized in a pre-defined schema [14], where the data properties and relationships have been pre-specified, and downstream analysis is restricted to this schema.

In most real-world applications, however, data exists in unstructured formats, such as images, documents and audio recordings. Without preprocessing such data into structured forms, traditional SQL engines can only support limited queries. Preprocessing, including document entity retreival [45] and form understanding [42], also require training on downstream tasks given a predefined taxonomy. This naturally motivates the question we consider in this paper:

How can one perform unstructured data analysis in a flexible and efficient way?

In the literature, full-text search engines [20] support scalable regexp-matching search on unstructured data, but this becomes infeasible for more complex semantic reasoning queries. Retrieval-Augmented Generation (RAG) [39, 24, 18] allows question answering on a subset of related data, but is not directly applicable to generic analytical tasks with aggregation and semantic queries that spans over an entire large database. Recent advances in Large Language Models (LLMs) [4, 2] unlock the ability to perform flexible question answering, especially with recent long-context models [33]. However, setting aside the cost per query, data analytics can still be challenging for LLMs without fine-tuning [25] or few-shot demonstrations [9], even given structured tables.

movie

rating

review_text

Interstellar

7

Very entertaining...

Glitter

5

No purposeful plot, ...

Source Code

8

Bold, brash, exciting...

reason

sentiment

Fun to watch...

positive

Bad plot...

negative

Refreshing

positive

structured

columns

unstructured

columns

virtual columns

concrete columns

Recently, a promising line of work considered marrying LLMs with programming frameworks [38], where logical or arithmetic operations are offloaded to program interpreters [17]. A most relevant example for analytics and table understanding tasks is Cheng et al. [11], which augments classical SQL semantics with LLMs as user-defined functions (UDF). While promising, the execution of such SQL programs that embed LLM calls still requires sweeping over the entire database, which is too costly for large collections of unstructured content. To overcome this barrier, we leverage the synergy between LLMs and programmatic execution to define an Unstructured Query Language (UQL) that augments SQL for flexible semantic queries, with a focus on improving scalability and efficiency.

A key observation is that the efficiency of classical SQL engines relies on (1) indexing structures that avoid the need to scan the entire database, and (2) a compilation system that determines the best execution order for operations. Based on these ideas, we propose the Unstructured Query Engine (UQE), which refines and extends this design principle to unstructured data analytics. To achieve similar effect to indexing, UQE casts the problem as learning to search or sample, seeking to avoid a full database scan with statistically sound methods. Additionally, a compilation system is developed that determines the best execution order and operator combination for different clauses in a UQL query, with the goal of minimizing LLM calls while preserving query semantics.

As part of this project, we have created four new benchmark datasets, with both text and image modalities, along with three common analytic tasks. Compared to baseline methods, such as longcontext LLMs and embedding based retrieval, UQE achieves significant improvements in terms of the accuracy and cost reduction on these benchmarks.

2 Problem

Before defining the problem we are solving, we first establish the terminology and notation we will use throughout the paper. A concrete illustration of the following terms is given in Figure 1.

Table / database : We define a table T = {T i } N i =1 as an unordered set of |T | = N rows, where each row T i = [ T i, 1 , T i, 2 , . . . , T i,M ] is an array of M elements such that M is the total number of columns in the table. Each row can consist of elements T i, · of heterogeneous types ( e.g. , datetime, float, enum) with different modalities ( e.g. , text, image), while elements in each column T · ,j must be of the same format and modality.

Structured data : A column T · ,j is structured w.r.t. a query if it can be accessed quantitatively, such as by algebraic operations over numeric data, comparison over string labels with predefined vocabulary ( e.g. , categorical labels), datetime functions, etc. .

Unstructured data : A column T · ,j is unstructured if a query cannot access it using standard quantitative access. Typically, such a column does not belong to a predefined taxonomy. Examples include text ( e.g. , dialogs), images, videos, and other forms of data that usually require semantic understanding and preprocessing before performing any algebraic operations.

Concrete column : A column is concrete if it already exists in the table.

Virtual column : A column is virtual if it does not already exist in the table, but a query is able to operate on it. Conceptually, one needs to derive (partial rows of) these columns by processing the data from concrete columns. In our work, we bypass this step by creating such columns lazily and selectively, which is the key to achieving efficiency and performance gains.

SQL engines can perform analytic queries on databases by manipulating structured data in concrete columns. The focus of this paper is to propose a new query engine that can perform analytics on databases with both structured and unstructured data, with queries that operate over both concrete and virtual columns. Standard analytics tasks that we seek to enable over unstructured data are:

Conditional aggregation : perform aggregation operations on a sub-table filtered by a condition.

Semantic retrieval : collect relevant rows specified by semantic filters.

Abstraction and aggregation : group the rows based on abstractions and then performs aggregation.

Since optimizing queries on structured data within concrete columns is well-studied, we focus instead on techniques for handling queries on unstructured data over virtual columns. However, the UQE implementation also supports operations over structured data within concrete columns. In the following, unless stated otherwise, the term unstructured databases refer to databases containing both structured and unstructured data.

3 Unstructured query language

First, we need to formally define the query language, UQL, that talks to the unstructured databases. The idea of defining a natural query language for unstructured data is not completely new ( e.g. , Cheng et al. [11], even though UQL has richer semantics), nor is the specific syntax or design of UQL the main focus of this paper. However, we need to define the scope of queries that the engine can handle, and breakdown the semantic meaning of each clause.

3.1 UQL semantics

We assume a basic familiarity of SQL, upon which UQL is based. UQL can be considered to be a dialect of SQL that has augmented functionalities for handling unstructured and virtual column queries. The SQL clauses that we support in UQL, along with necessary modifications to support unstructured semantic queries, are described as follows.

SELECT is a mapping function that maps the operand (usually a row or collection of rows in a grouped query) to a new row of elements. In traditional SQL, this mapping is usually a subset selection over concrete columns, or algebraic operators over those columns. UQL provides additional semantic mapping capability as:

SELECT "the attribute specified by natural language" AS attribute_name

For example, one can write SELECT "the sentiment of the movie review" over an unstructured movie review column, and retrieve "positive" or "negative" as a structured output.

FROM specifies the source of the table. In SQL one can additionally specify table joins, but we limit our attention to sourcing from a single table in this paper.

WHERE intrinsically specifies a binary classifier over rows, which is used to retrieve a subset of the database. In addition to comparator operators on structured columns, we also allow semantic specifications in the form of:

WHERE "the row satisfies some natural language specifications"

The predicates in WHERE are organized in disjunctive normal form (DNF) with AND and OR syntax, so a user can arbitrarily express predicates over concrete and virtual columns.

GROUP BY partitions the table into groups, where rows within each group share the same attributes over the keys being grouped by. UQL allows partitioning over virtual columns via natural language:

GROUP BY "the abstraction criteria specified in natural language"

Similar to WHERE , one can GROUP BY over both concrete and virtual columns by concatenating multiple criteria, with the resulting partition corresponding to grouping by a tuple of these keys.

We also reuse other clauses from SQL including: ORDER BY , which simply inherits the SQL semantics to rank the resulting rows according to a specific concrete column. In most analytics tasks, sorting is applied over structured columns with well defined ordering comparators. LIMIT is applied during processing in the form of LIMIT num_rows , which limits the number of output rows.

Assumptions: We rely on the ability of an LLM to perform intra-row semantic understanding and analysis tasks. For example, we assume that LLMs are able to correctly judge the specification in WHERE for a single row. Similarly, LLMs should be able to extract the information specified by SELECT or GROUP BY for a single row. We build programmatic functionality on top of this fundamental ability of LLMs to handle analytics for large databases.

3.2 UQL queries

A UQL query is a composition of clauses that can be categorized as an aggregation or a nonaggregation, as illustrated in Figure 2. Aggregation queries perform a summary on groups of aggregated rows, such as COUNT the number of rows, or summarize a common attribute in a group of rows (as defined in GROUP BY above). Non-aggregation queries perform operations on individual rows, which usually means the rows can be processed in parallel. A UQL query will only belong to one of the above two types and we do not consider nested queries for now. The query type determines how UQE will optimize and execute the query.

4 Unstructured Query Engine

One straightforward way to run UQL is to use an interpreter that executes queries imperatively. Cheng et al. [11] implements an engine in this form, which is able to handle tables of relatively small size. By analogy, this is similar to executing an SQL program using a linear database scan. While it is valid, the latency and cost are prohibitive and generally prevent scaling to real world scenarios.

There are (at least) two key techniques in SQL databases for making query execution efficient:

Indexing , which organizes concrete columns via data structures for fast search with sublinear cost.

Compilation , which considers alternative query plans and executes the most efficient one.

UQL queries over virtual columns pose challenges in both indexing and compilation. In this section, we present effective approaches to indexing (Section 4.1) and compilation (Section 4.2) for unstructured databases, along with the implementation details of low-level primitives (Section 4.2.2).

4.1 Indexing

Executing traditional SQL queries over indexed columns can be made efficient by avoiding an entire database scan to find the relevant rows to process. However, for UQL queries over virtual columns, it is hard to predict or predefine an index that can enable efficient searching, since these columns are not concrete and defined via arbitrary natural language specifications.

Our first key contribution is to introduce a proxy for "indexing" that allows one to leverage the intrinsic semantic content of a virtual column to efficiently execute queries without scanning the entire database. The main idea is to use statistically sound sampling techniques to approximately retrieve relevant rows for processing. Based on the two types of queries defined in Section 3.2, we develop corresponding "indexing" counterparts.

4.1.1 Unbiased estimation for aggregation queries

We use a simple query to illustrate the idea of unbiased estimation to obtain a query result without scanning over an entire virtual column.

Given a row T i (a natural language movie review), it is relatively easy for an LLM to tell whether it satisfies the WHERE condition. If we use f : ( T i , cond ) ↦→{ 0 , 1 } to represent the LLM's classification of whether row T i satisfies the conditions specified in cond , then the goal is to estimate the quantity

There are many approaches that can be used to estimate the finite sum in the above equation, with different tradeoffs between bias and variance. One unbiased but potentially high variance estimator is to simply use Monte Carlo samples from a uniform distribution over 1 . . . N . A typical technique for reducing variance is to use importance sampling with a proposal p , according to

A theoretically optimal proposal p is given as follows:

Algorithm 1 Stratified sampling for unbiased aggregation

Embed each row T i as ⃗ e i , using a multi-modal embedding over the unstructured columns of T i .

Cluster the embeddings { ⃗ e i } N i =1 into K disjoint groups { C k } K k =1 , C k ⊆ { 1 . . . N } , where k can be a predefined constant or automatically selected [48]. Each group has size | C k | and ∑ K k =1 | C k | = N . We use c : { 1 . . . N } ↦→ { 1 . . . K } to denote the cluster index of each row.

Perform stratified sampling over these groups and obtain samples S ⊆ { 1 . . . N } .

Then we can obtain the following estimator for the expectation:

Algorithm 2 Online active learning for non-aggregation retrieval

Embed each row T i as ⃗ e i , using multi-modal embedding over the unstructured columns of T i .

Maintain ˆ g ( i ) that approximates f ( T i , cond ) . Initialize ˆ g ( i ) ∝ U (0 , 1) uniformly.

Maintain the collection of sampled rows S = ∅ at step t = 0 .

At step t , obtain a batch S t of samples where S t = arg max S t ⊆{ 1 ...N }\ S ∑ i ∈ S t ˆ g ( i ) + ϵ t,i ; observe f ( T i , cond ) for each sample i ∈ S t ; Update S ← S ∪ S t .

Fit ˆ g with samples and corresponding observations from S . Go to step 4 if | S | < B or return the positive samples found in S .

Proposition 1 The optimal proposal distribution p that minimizes the variance of estimation in Eq 2 is p i ∝ f ( T i , cond ) , which achieves zero variance.

Prop 1 indicates that an ideal proposal should sample rows that have positive f with equal probability, while sampling negative rows with zero probability. However, given that f is the response of an LLM it is expensive to execute over all rows, forcing us to consider efficient approximations.

Stratified sampling leverages the ability to partition a population into homogeneous subpopulations. As shown in Prop 1, a good proposal p i should predict whether a row T i satisfies the target property cond . To trade-off between cost and variance reduction, we propose Algorithm 1. In Eq 3 we normalize the importance weights w i to further reduce the estimation variance.

Extension The above estimator can be used for other aggregation operations such as SUM and AVERAGE , including GROUP BY , and allowing concrete columns as operands as well. However, some aggregations such as MAX does not admit such an estimator. UQE in this case can only provide estimates with greater effort. We discuss limitations in Section 7.

4.1.2 Online learning for non-aggregation queries

A non-aggregation query can be viewed as a search problem where we want to find a relevant subset of rows to process. As before, we begin with a concrete example:

SELECT dialog_ID FROM dialogs WHERE "the customer is unhappy with the agent's manner" In practice, we aim to identify as many rows as possible that satisfy the given condition while adhering to budget constraints (e.g., the total number of tokens allowed to expense). This can be formulated as an online learning problem: given the token budget (or approximately, the number of LLM calls over individual rows, denoted as B ), we seek to balance exploration (to better understand the semantic landscape of all rows) with exploitation (to maximize recall). Drawing inspiration from Bayesian optimization, which employs a surrogate model learned on-the-fly to inform sequential decision-making [36, 19, 22], we use a cheap proxy ˆ g as a surrogate for f ( T i , cond ) . At each step t , we re-train ˆ g with the observed data and select its maximizer to query in the next step. See Algorithm 2. Here we use random noise ϵ t,i to allow some degree of exploration that decays with t . Compared to the typical max inner-product search method prevalent in RAG systems, we rely on the online learning to adjust the beliefs, instead of solely relying on predefined embedding similarities.

4.2 Compilation

Classically, the goal of a compiler is to translate a high level program to low-level machine code, maintaining exact execution results with improved execution speed. Such a lowering process is usually accompanied by optimizations e.g. , fusing, selecting the optimal instructions, and kernel optimizations. Our goal is similar: we would like to compile high-level UQL into low-level machine code, with the distinction that the "machine" is an LLM, and the "low-level code" is the orchestration of prompt calls to the LLM. Given that the primary bottleneck is the LLM API calls, we attempt to maintain the execution semantics while minimizing the cost of LLM calls, as in Figure 3.

Low-level IR

(optimized)

High-level IR

(unoptimized)

Input: Program

Input: Query

High-level IR

(optimized)

Low-level IR

(optimized)

Assembly

(sequence of instructions)

high-level optimization

(e.g. algebraic simplification,

CSE, DCE)

lowering, low-level optimization

(e.g. instruction selection,

scheduling, register allocation)

code generation

optimized ops

High-level IR (unoptimized)

High-level IR

(optimized)

Prompt + Orchestration

(sequence of LLM calls)

WHERE

LIMIT

SELECT

AGG

AGG_WHERE

high-level optimization

obj: min token_count

cost

estimation

fusion

optimization

AGG_WHERE

Active Learning

Stratified

Random

Round Robin

sample efficiency

optimization

lowering, low-level optimization

obj: tradeoff

- bias v.s. variance

- exploration v.s.

exploitation

prompt

optimization

code generation

accuracy | FM

Foundation

Model

sequence of

executable ops

4.2.1 Planning

Lowering a query into sequences of concrete execution units is a planning problem: The action space includes the order of clause execution, as well as ways to fuse clauses to execute together. The objective is to minimize the (estimated) LLM cost. Figuring out the best decomposition and combination is usually an NP-hard problem. Fortunately, the number of clauses is very limited for a single query, so we can enumerate possible combinations of ordering and fusions with little overhead.

The outcome of planning is a specification of a sequence of kernel executions. The input and output of each kernel can be one of the following:

Concrete table : a standard table with only concrete columns.

Stochastic table : the outcome of unbiased sampling of a table. Importance weights will be attached to each row of the table, and the operation ( e.g. , SUM , AVG ) on this table takes weights into account.

In the following 3 sections we will explain the building blocks of the compiler, including the kernel implementation, the cost estimation and final instantiation in detail.

4.2.2 Kernel implementation

Each kernel is an standalone execution unit that reads and produces a (stochastic) table.

SELECT on structured columns is straightforward. When operating on unstructured columns, we prompt the LLM to extract semantic attributes from the input data. If several extractions share the same source column, we can also group these together into a single prompt to reduce cost.

WHERE takes a logical formula in disjunctive normal form, such that each conjunction can contain predicates over both unstructured and structured columns. One optimization we make in this case is to perform evaluations over the structured columns first, then simplify ( e.g. , remove a conjunction if any of the structured column evaluates to false) the logical formula. Any remaining predicates over unstructured columns are then executed on the table filtered by predicates over structured columns.

GROUP BY first gathers a representative subset of rows from the table, then calls an LLM to extract a taxonomy ( i.e. , the description of each cluster) for a cluster abstraction. Then the taxonomy is used to classify rows sampled according to the methods defined in Section 4.1. Finally, each row is classified into one of the clusters with the corresponding cluster description in the taxonomy.

Other standard kernels like ORDER BY are implemented as-is since they are efficient to execute.

Kernel fusion: Certain clauses can be fused together to achieve significant efficiency gains.

WHERE + LIMIT can be terminated earlier for non-aggregation queries, once the number of rows specified by LIMIT are retrieved. This is particularly useful for rare event finding.

GROUPBY + WHERE can share the same sampling proposal for the aggregation queries.

SELECT + GROUPBY when executed together, the semantic attribute extraction of SELECT and taxonomy classification in GROUPBY can be done in the same LLM call to save cost.

When and how to fuse clauses relies on the planning technique introduced in Section 4.2.1.

4.2.3 Cost estimation for each kernel

We only consider the cost of calling the LLM, as this dominates the overall cost per query. Assuming the length of each row in the unstructured data is more or less uniform across rows, then the cost is proportional to the number of rows that fed to the LLM, which we use as the surrogate for estimation.

SELECT maps each row, hence the cost is |T | for the table T fed to SELECT .

GROUP BY consists of two steps, where taxonomy construction consumes a subset of the input table T , and classification runs |T | LLM calls in parallel.

WHERE depends on the proposal p . In practice we set a budget B and try to minimize the variance of unbiased estimator or maximize the recall in online learning, as explained in Section 4.1.

Whenever clauses are fused together, each implementation is responsible for providing a reasonable cost estimate. For example when SELECT and GROUP BY are fused, the estimated cost is the same as GROUP BY alone, as the classification stage of GROUP BY shares the input tokens with SELECT .

4.2.4 Instantiation of kernels

The last step of compilation is to generate the machine specific code ( e.g. , x86 assembly code) from the intermediate representations (IR). For UQE, this is the process of generating the LLM-specific prompts. For example, when GPT is deployed as the "machine", a system prompt like "You are a helpful assistant" will be added to the queries. This step also sets the correct context ( e.g. , the correct structured/unstructured column to associate to, the description of the databases) for the LLM. When such information is not available, one can also leverage the LLM to provide a good suggestion.

5 Related work

While the unstructured data analytics engine is relatively new, there are several related works in the context of unstructured data query and analysis. Approaches like pattern or regexp matching [20] is scalable but not feasible for complex semantic reasoning. RAG [24, 18] based approaches rely on the retrieval quality and is not directly suitable for aggregation queries over entire database. LLMs [4, 2, 33] depict the ability of table analytics [15] to some extent [9, 25], but are still not reliable for large unstructured database analytics yet.

Our work is closely related to neural symboilic approaches for unstructured data analytics. Early attempts in this line aim to design specialized neural architectures with inductive biases (e.g., attention) to capture a particular form of operation (e.g., filtering a list of objects based on a natural language predicate by their attention scores) [43, 29, 3]. Those differentiable neural 'operators' can then be chained together to model more compositional queries, and trained end-to-end using gradient descent. Another direction, in line with our work, is to augment symbolic programs with learnable operators parameterized by neural networks [10]. Those programs are often modeled as discrete latent variables, which can be hard to optimize. In contrast, UQE leverages predictions from LLMs as supervision to train an efficient proxy query model in an online fashion. Similar to UQE, some recent work [11, 38] also adopts LLMs as fuzzy query operators. However, the generated programs treat LLMs as an UDF in a SQL program, which can be very expensive to execute on large databases. Our UQE implements similar but augmented semantics with the focus on the cost efficiency and scalability. Liu et al. [26] optimizes a similar query engine from the system perspective like cache optimization and deduplication, while our work mainly considers algorithmic improvements and is considered as an approximate query engine [28]. These system and algorithm optimizations are actually orthogonal and can be beneficial to jointly consider both for future works.

In a distantly related topic, text2SQL [47, 46, 21, 35] also leverages models talking to databases, but is mainly for semantic parsing purpose. While it also leverages the advances in LLMs [44, 16, 31, 37], the execution is still on pure SQL and thus is not suitable for unstructured databases. There are also works on leveraging formal query languages to better query LLMs [34, 6], with the focus on controllability of the LLM itself rather than performing analytics on external unstructured data.

6 Experiments

We benchmark the accuracy and incurred cost of UQE on multimodal unstructured data analytics tasks, with the goal to show and understand when and why UQE can improve accuracy while keeping the cost low. Since the unstructured database analytics is a relatively new task, we construct and compare against several baseline approaches, on a set of tasks created from existing datasets.

Baselines: We design the following baselines for comparison

lc-LLM denotes the long-context LLMs that can directly take a subset of database and a natural language question as input, and produce the desired analysis. We mainly evaluate against several model families, including GPT-4 [2] and Claude-3 [4]. Of course, when evaluating the lc-LLM based approaches, we use the natural language instead of UQL as the prompt.

RAG-based can be applied to some non-aggregation queries, such as semantic retrieval. For the retrieval part we use max inner-product search (MIPS) on top of the same embeddings that are used by UQE, for a controlled experiment.

UDF simply treats the LLM calls as User-defined function of an SQL engine, with the same budget as UQE by default. This approach will not have the advanced sampling / search algorithm as used in UQE, which also serves as an ablation for the effectiveness of our UQE.

Datasets: We evaluate different approaches on common analytical tasks in three widely used application domains. We use the datasets that were previously created for discriminative tasks, as these datasets contain both the unstructured columns and the structured ones (the labels in the corresponding dataset). We then hide these structured label columns and perform analytical tasks on the unstructured columns, where these hidden structured columns will be used to compute the ground-truth. The text based tasks include IMDB [27] movie reviews, customer service dialogs including Action-Based Conversations Dataset (ABCD [8]) and AirDialog [41], and image based Clevr [23] dataset. Please refer to Appendix C.1 for more information.

Setup: We use voyage-2 [1] to embed the text-based unstructured columns, and Vertex [40] for multimodal embeddings. For budget constraint queries, we allow different approachces to access at most 128 rows in the database by default.

Benchmarks

Metrics

Methods

lc-gpt-4-turbo

lc-claude-3-opus

UQE-claude-3-haiku

AirDialog

EMD

↓

0.143

±

0.034

0.121

±

0.014

0.111

±

0.019

cost

$0.21

$0.37

$0.04

ABCD

account_access-EMD

↓

0.154

±

0.031

0.113

±

0.010

0.110

±

0.016

single_item_query-EMD

↓

0.031

±

0.034

0.011

±

0.006

0.005

±

0.002

cost

$0.34

$ 0.56

$ 0.07

Uniform

Stratified

0.45

0.50

0.55

0.60

0.65

Query "book" on airdialog

Uniform

Stratified

0.125

0.150

0.175

0.200

0.225

0.250

0.275

0.300

0.325

Query "no_flight" on airdialog

Uniform

Stratified

0.125

0.150

0.175

0.200

0.225

0.250

0.275

0.300

0.325

Query "no_reservation" on airdialog

Uniform

Stratified

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

Query "account_access" on abcd

Uniform

Stratified

0.06

0.08

0.10

0.12

0.14

0.16

Query "single_item_query" on abcd

Uniform

Stratified

0.40

0.45

0.50

0.55

0.60

Query "positive" on imdb

6.1 Main results

We run queries on different datasets by instantiating the template shown in each of the sections below. The exact natural language and UQL queries can be found in Appendix D and more information including statistics of conditions we used for query and hyperparameters (for UQE we simply use the default hyperparameters for sampling and online learning) can be found in Appendix C.

6.1.1 Conditional aggregation

This task provides aggregated statistics over databases with specified conditions, with the template as:

SELECT COUNT(*) FROM {table} WHERE {"satisfies natural language specified condition"} predict true_count true_count

We report the relative estimation error ( i.e. , | -| / ) and its standard deviation in Table 1. For lc-LLM baselines we estimate the count based on groups of unbiased data samples that fed into the prompt.

For text based aggregation we use claude-3-haiku as the backbone model, where UQE deploys 10 × reduction in relative errors while reducing the cost by a factor of 20 × or more. For the image dataset, since only limited set of LLMs are capable right now, we use gpt-4o as the backbone, and compare with lc-LLM baselines. Thanks to the improved sampling method in UQE, the same gpt-4o consistently achieves improved performance out-of-the-box. To verify this, we feed one image at a time to gpt-4o and manually aggregate the count, the estimation error would be 17.10% ± 13.95 and 19.35% ± 13.81% for the two queries of Clevr, which is twice higher than UQE in the worst case.

6.1.2 Semantic retrieval

This task filters rows in databases that satisfy specified conditions, with the template as:

SELECT * FROM {table} WHERE {"satisfies natural language specified condition"} LIMIT B While we limit the output size to be B = 256 to keep the total cost within a reasonable budget. The challenging scenarios are when the number of rows that satisfy the predicate is few ( i.e. , "rare event finding"). Table 2 shows similar sets of comparison, but the metric is F1 score which evaluates the quality of SELECT -ed rows. Overall UQE (with claude-3-haiku as backbone LLM) consistently achieves comparable or better performance than the baseline methods. MIPS which uses the same embedding of unstructured data as UQE, has high variance across different types of queries. The queries such as "dialogs with account access issues" would be very suitable for MIPS as the embedding similarity is able to capture that well. For queries involving reasoning ( e.g. , find the images with less than 4 objects), it is pretty hard for pretrained embeddings to express this.

6.1.3 Abstraction and aggregation

This task abstracts the intrinsics of each row, and then performs semantics-based GROUP BY , grouping the common intrinsics across all rows. Finally, it provides aggregated statistics over each group: SELECT derived_attribute, COUNT(*) FROM {table} GROUP BY {"extract an abstract intrinsic attribute specified in natural language"} AS derived_attribute LIMIT 10

The challenging problems in this task are (i) building a taxonomy with good coverage, and (2) bias and variance reduction for groups with small population. The result of this query is a list of

128

256

Number of iterations

0.00

0.25

0.50

0.75

1.00

Recall

(Moving average)

128

256

Number of iterations

0.00

0.25

0.50

0.75

1.00

Recall

(Moving average)

128

256

Number of iterations

0.00

0.25

0.50

0.75

1.00

Recall

(Moving average)

128

256

Number of iterations

0.00

0.25

0.50

0.75

1.00

Recall

(Moving average)

tuples of derived attributes and their number of occurrences in the dataset. We use the earth mover's distance (EMD [30]) as the evaluation metric to compare the extracted tuples and ground-truth tuples. The distance between a pair of attributes is defined by one minus the cosine similarity of their text embeddings. We can see from Table 3 that UQE consistently outperforms baselines while achieving much lower cost. We also show in Appendix C with more qualitative results comparisons.

6.2 Ablation studies

We study the effectiveness of UQE for aggregation queries in Section 6.2.1 and non-aggregation queries in Section 6.2.2, and the quality/cost trade-off of UQE in Section 6.2.3. In appendix we provide more results on other modalities C.3, consistency C.4 and latency C.5.

6.2.1 Variance of different sampling approaches for aggregation queries

To decouple the variance introduced by the algorithm and the bias introduced by the LLM based predictors, here we use the ground-truth label as the predictive result and focus on the effectiveness of variance reduction. Figure 4 shows the box plot of different sampling methods. We can see using stratified sampling over the embeddings of unstructured content achieves significant lower variance compared to the uniform random sampling. Also both of these achieve similar expected values, which also justifies the correctness or unbiasedness.

6.2.2 Efficiency of online learning for non-aggregation queries

We show the effectiveness of the online learning in terms of the recall as a function of the iteration steps in Figure 5. Compared to the dashed line in the figure which indicates the results of uniform random sampling, the online learning can achieve significant boost in terms of the recall. While for some queries the variance at early iterations can be high, these all converge well in the end.

6.2.3 Trade-off between cost/latency and accuracy

Generally the larger compute budget B the better quality UQE will get, and we verify this in Table 4 5. UQE can achieve pretty good quality even with very low budget, and notably compared to the baseline, it achieves similar quality with 16x reduction of the compute needed. We show more results in Section C.5 regarding the compute efficiency.

7 Conclusion

This paper proposed an unstructured query engine that leverages 1) the flexibility of LLMs for data understanding; 2) the advances in sampling and online learning for efficient data scanning; 3) and the compiler that bridges these algorithmic workflows with LLMs. We demonstrated its efficiency and accuracy over three analytic tasks on four datasets with two different modalities. However the current work is still very limited in terms of 1) the semantics it lacks, including table join and other types of aggregations; 2) an automated selection of LLMs and sampling configurations; 3) and scaling to even larger databases. We hope to investigate these further in future works.

Acknowledgments and Disclosure of Funding

We thank Carsten Binnig, Howie Xu, Ras Bodik, Xinyi Chen, and the anonymous reviewers for providing helpful discussion and suggestions.

References

Voyage 2. Voyage 2. https://github.com/voyage-ai/voyageai-python .

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 39-48, 2015. URL https://api.semanticscholar.org/CorpusID:5276660 .

AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card , 2024.

Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojciech Samek. Audiomnist: Exploring explainable artificial intelligence for audio analysis on a simple benchmark. Journal of the Franklin Institute , 2023. ISSN 0016-0032. doi: https://doi.org/10.1016/j.jfranklin.2023.11.038. URL https://www.sciencedirect.com/science/article/pii/S0016003223007536 .

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages , 7 (PLDI):1946-1969, 2023.

Don Chamberlin. A complete guide to DB2 universal database . Morgan Kaufmann, 1998.

Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. arXiv preprint arXiv:2104.00783 , 2021.

Wenhu Chen. Large language models are few (1)-shot table reasoners. arXiv preprint arXiv:2210.06710 , 2022.

Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Xiaodong Song, and Quoc V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations , 2020. URL https://api.semanticscholar.org/CorpusID:212814759 .

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding language models in symbolic languages. arXiv preprint arXiv:2210.02875 , 2022.

Edgar F Codd. The relational model for database management: version 2 . Addison-Wesley Longman Publishing Co., Inc., 1990.

Nada Elgendy and Ahmed Elragal. Big data analytics: a literature review paper. In Advances in Data Mining. Applications and Theoretical Aspects: 14th Industrial Conference, ICDM 2014, St. Petersburg, Russia, July 16-20, 2014. Proceedings 14 , pages 214-227. Springer, 2014.

Ramez Elmasri and Shamkant B Navathe. Database systems: models, languages, design, and application programming . Pearson, 2013.

Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, and Christos Faloutsos. Large language models on tabular data-a survey. arXiv preprint arXiv:2402.17944 , 2024.

Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. ArXiv , abs/2308.15363, 2023. URL https://api.semanticscholar.org/CorpusID:261276437 .

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning , pages 10764-10799. PMLR, 2023.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 , 2023.

Roman Garnett. Bayesian optimization . Cambridge University Press, 2023.

Otis Gospodnetic, Erik Hatcher, and Michael McCandless. Lucene in action . Simon and Schuster, 2010.

Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and D. Zhang. Towards complex text-to-sql in cross-domain database with intermediate representation. ArXiv , abs/1905.08205, 2019. URL https://api.semanticscholar.org/CorpusID:159041042 .

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5 , pages 507-523. Springer, 2011.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.

Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263 , 2023.

Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E Gonzalez, Ion Stoica, and Matei Zaharia. Optimizing llm queries in relational workloads. arXiv preprint arXiv:2403.05821 , 2024.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .

Barzan Mozafari and Ning Niu. A handbook for building an approximate query engine. IEEE Data Eng. Bull. , 38(3):3-29, 2015.

Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. CoRR , abs/1511.04834, 2015. URL https://api.semanticscholar.org/CorpusID:6715185 .

Ofir Pele and Michael Werman. Fast and robust earth mover's distances. In 2009 IEEE 12th international conference on computer vision , pages 460-467. IEEE, 2009.

Mohammad Reza Pourreza and Davood Rafiei. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. ArXiv , abs/2304.11015, 2023. URL https://api.semanticscholar.org/CorpusID:258291425 .

Raghu Ramakrishnan and Johannes Gehrke. Database management systems . McGraw-Hill, Inc., 2002.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jeanbaptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024.

Mohammed Saeed, Nicola De Cao, and Paolo Papotti. Querying large language models with sql. arXiv preprint arXiv:2304.00472 , 2023.

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. ArXiv , abs/2109.05093, 2021. URL https://api.semanticscholar.org/CorpusID:237491759 .

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104 (1):148-175, 2015.

Ruoxi Sun, Sercan O Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. Sql-palm: Improved large language modeladaptation for text-to-sql. arXiv preprint arXiv:2306.00739 , 2023.

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11888-11898, 2023.

James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. Neural databases. arXiv preprint arXiv:2010.06973 , 2020.

Vertex. Vertex API. https://cloud.google.com/vertex-ai/generativeai/docs/embeddings/get-multimodal-embeddings .

Wei Wei, Quoc Le, Andrew Dai, and Jia Li. Airdialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 3844-3854, 2018.

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pretraining of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1192-1200, 2020.

Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. Neural enquirer: Learning to query tables. ArXiv , abs/1512.00965, 2015. URL https://api.semanticscholar.org/CorpusID:6715526 .

Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. ArXiv , abs/2005.08314, 2020. URL https://api.semanticscholar.org/CorpusID:218674345 .

Lijun Yu, Jin Miao, Xiaoyu Sun, Jiayi Chen, Alexander G Hauptmann, Hanjun Dai, and Wei Wei. Documentnet: Bridging the data gap in document pre-training. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages 707-722, 2023.

Tao Yu, Michihiro Yasunaga, Kai-Chou Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir R. Radev. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. ArXiv , abs/1810.05237, 2018. URL https://api.semanticscholar.org/CorpusID:52979524 .

Tao Yu, Rui Zhang, Kai-Chou Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Z Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and crossdomain semantic parsing and text-to-sql task. ArXiv , abs/1809.08887, 2018. URL https://api.semanticscholar.org/CorpusID:52815560 .

Chunhui Yuan and Haitao Yang. Research on k-value selection method of k-means clustering algorithm. J , 2(2):226-235, 2019.

A Proof

Proposition : The optimal proposal distribution p that minimizes the variance of estimation in Eq 2 is p i ∝ f ( T i , cond ) . The variance gets 0 with this proposal.

Let's simplify the notation a bit and use f ( x ) for f ( T i , cond ) and prove the variance reduction in general cases for binary function f . For the simplicity let's omit the constant |T | and focus on the estimation of the expectation term. If we come up with a new proposal distribution p : T ↦→ [0 , 1] where ∑ x ∈T p ( x ) = 1 , then we get a new estimator in the following form:

We hope this new estimator would have lower variance. Let's define u ( x ) = q ( x ) p ( x ) f ( x ) for the ease of notation, and look at its variance first:

Since our estimator is unbiased, E 2 p [ u ( x )] = E 2 q [ f ( x )] and thus has nothing to do with p , let's focus on minimizing the first term. More specifically we have the optimization as:

Let:

and we can find the saddle point of min p max λ x λ 2 ( L ( p, { λ x } , λ 2 )) using K.K.T condition.

and we can get the optimal solution of

and put it back into Eq(1) we can see the optimal variance would be

which means one sample would be good enough! In our context, q ( x ) is usually just a constant (e.g., q ( x ) = 1 T for the [Case Count]), and f ( x ) ∈ { 0 , 1 } . In this case, a simplified optimal proposal would be:

or in another word, ideally we should have zero-chance to sample from regions where f ( x ) = 0 , and have equal chances to sample where f ( x ) = 1 .

B UQL specifications

B.1 Tokenizer

We use the following pattern matching to tokenize the UQL programs/queries.

B.2 Grammar

Below we show the context free grammar of the UQL. Note that this represents a subset of the "natural query" analogy to SQL, and we leave other clauses like table join into the future work.

| VAR_NAME COMPARE_OPERATOR FLOAT

l i m i t _ c l a u s e :

LIMIT INTEGER

t o _ c l a u s e :

TO VAR_NAME

C Experiments

C.1 Datasets

We evaluate different approaches on common analytical tasks in three widely used application domains. We use the datasets that were previously created for discriminative tasks, as these datasets contain both the unstructured columns and the structured ones (the labels in the corresponding dataset). We then hiden these structured label columns and perform analytical tasks on the unstructured ones, where these hidden structured columns will be used to compute the groundtruth.

User review mining . We use IMDB [27] for semantic analysis of user sentiment. The entire dataset contains 50K highly polar movie reviews with positive and negative sentiment labels.

Goal-oriented (customer service) dialogue systems . We use two datasets for this category, including 1) Action-Based Conversations Dataset (ABCD [8]) for intent identification, which has 10,042 dialogs with 10 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success; and 2) AirDialog [41] for conversation outcome understanding. It contains 402,037 goal-oriented conversation on flight booking with 5 possible ground-truth states.

Image understanding . We use Clevr [23] dataset for multimodal data understanding and retrieving. Specifically we use the val split of the dataset, which contains 15,000 images of objects containing different number of cylinders, cubes and spheres with different sizes/colors. We down-sample the image to size no more than 128 × 128 , and only feed the images to the LLMs while holding out scene metadata for evaluation only.

Also see Table 6 for the detailed statistics of different query conditions. Rare events like cancel in AirDialog is typically challenging to find and aggregate on.

C.2 Parameter and experiment setup

For the embeddings, we use the voyage-2 for text and for images, we use Google Vertex API with dimensionality of 512. We preprocess the embeddings for all the datasets and keep them static during the queries.

For the aggregation queries, we use faiss 2 to cluster the embeddings into 10 groups, and perform stratified sampling on top.

For the online learning setting for non-aggregation queries, in our experiment we simply set ϵ t,i to be 0. We start training the function g when we collect at least one possible and one negative example labeled by the LLM. Then after every minibatch of samples collected, we train g via linear logistic regression and simply leverage sklearn for that.

Other parameters that might matter include: the sampling budget B for aggregation queries is 128 and for non-aggregation queries it is 256. For group-by queries the UQE needs a step in building the taxonomy, where the budget we use for that is 16.

We set these parameters based on educated guess and keep them as default across all the queries over all the datasets.

C.3 Audio modality

We use the audio MNIST [5] data for this experiment. This dataset contains 30k wav files from 60 speakers pronouncing digits 0-9. We perform the audio semantic retrieval experiments. The query is first converted to audio space using TTS and the corresponding audio embedding is used for MIPS search. We use Gemini Pro 1.5 as the backend model, as it supports the audio inputs nicely. As is shown in Table 7, the proposed UQE consistently does better than alternatives, and it also allows complex queries that require reasoning, while embedding based MIPS is limited to certain types of queries.

C.4 Consistency of the execution results

While UQE is able to leverage the foundation models to analyze the unstructured databases directly, the execution result is not deterministic due to the nature of the stochasticity of the algorithm and the LLMs themselves. In our paper we aim at reducing the variance so as to improve the consistency. However when the backend LLM gets updated, it might also cause the potential inconsistency. Below we analyze the effect of different LLM backends.

Table 8 shows the effect on the IMDB dataset, where we see little variation. Actually the difference caused by the model switching is much lower than using a worse query engine. Of course, this behavior shift would be task related, but with the advances of LLMs we believe this variation would converge and be more stable to different prompts in the future.

C.5 Latency

We report the runtime of UQE with claude-3-haiku as backbone, and lc-gpt-4-turbo as the baseline method in Table 9. We can see UQE achieves low latency in aggregation operations, but higher latency in retrieval. This is due to the online update and re-evaluation of the g function described in Section 4.1.2. The experiments were run on MacBook Pro CPU, so we expect this bottleneck would be alleviated with better engineered system, which we will focus in our future works.

C.6 Group By qualitative results

For single_item_query in ABCD, the items mentioned in the dialogs found by UQE is:

For account_access in ABCD, the issues mentioned in the dialogs found by UQE is:

Which is very close to the ground truth (recover_username, reset_2fa, recover_password).

For airdialog, the outcomes found by UQE is

Where the ground truth has one more additional outcome (cancel). But since the percentage of cancellation is very small, it is expected that this might be missing from the group by abstraction when number of occurs are very limited.

D Prompts

D.1 Prompts for lc-LLMs

D.1.1 Task: Conditional aggregation

IMDB dataset

System prompt

Read the following movie reviews, and categorize them into either positive or negative class, depending on the sentiment of the review.If the movie has a mixed sentiment, try your best to classify into positive or negative class based on the overall sentiment.In the end, please just output a single number, which is [the total number of positive reviews]

User prompt

ABCD dataset

System prompt

The following dialogs between a customer service agent and a customer. Dialogs start by headers such as **Dialog 1**, **Dialog 2**, and so on. Your task is to classify whether the dialog content is about the theme "account access issue". Then count how many dialogs are talking about this theme. In the end, output the count as a single number.

Theme "<THEME>" dialogs content is about THEME_EXPLANATION. Please perform thorough analysis for each of the dialog. In the end, please **only** output a single number, which is [the total number of dialogs] that talks about the theme "account access issue".

Here is more detailed explanation about Theme "<THEME>". Be sure to use this information when you classify.

<THEME_EXPLANATION> = the customer could not access the account and was locked out, such as couldn´ t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account

<THEME_EXPLANATION> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to

System prompt

Please read the following images, and count how many of them show that <CONDITION>.

<CONDITION> = there are more than 3 spheres in the image User prompt image\_0: <base64\_encoded\_image> image\_1: <base64\_encoded\_image> ... Please output a single number, which is the total number of images that ↪ → satisfy the condition.

D.1.2 Task: Semantic retrieval

IMDB dataset

System prompt

User prompt

ABCD dataset

System prompt

The following are dialogs between a customer service agent and a customer. Dialogs start by headers such as **dialog_1**, **dialog_2**, and so on. Your task is to find out the dialogs where <CONDITION>. Please only output a list of indices in the format of [dialog_3, dialog_7, ...]

<CONDITION> = the customer could not access the account and was locked out, such as couldn´ t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account <CONDITION> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to cart or purchase an item, or inquiries about subscription

User prompt

AirDialog dataset

System prompt

Read the following customer support dialog between an agent and a customer, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise.

<WHERE_CLAUSE> = the customer could not access the account and was locked out, such as couldn´ t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account <WHERE_CLAUSE> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to cart or purchase an item, or inquiries about subscription

User prompt

...

AirDialog dataset

System prompt

Read the following airline ticketing dialog between the customer and the agent, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise.

<WHERE_CLAUSE> = the agent has booked a flight for the customer (not including the flight change)

<WHERE_CLAUSE> = the customer aims to book a flight from departure to destination but finds no flights between departure and destination <WHERE_CLAUSE> = the customer wants to change or cancel the flight but there is no valid reservation under this customer

<WHERE_CLAUSE> = the agent canceled the existing valid reservation for the customer

User prompt

Clevr dataset

System prompt

Read the following image, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise.

User prompt

D.2.2 Task: Abstraction and aggregation

ABCD dataset

Building taxonomy

System prompt

Group-wise conditional aggregation

System prompt

User prompt

AirDialog dataset

1. Building taxonomy

System prompt

The following are dialogs between a airline ticketing agent and a customer. Dialogs start with headers such as **dialog_1**, **dialog_2**, and so on.

Your task is to analyze all the dialogs, and summarize "<ABSTRACT_ATTRIBUTE>" into groups. Please output the table of your analysis, in the format of pairs of ("<ABSTRACT_ATTRIBUTE>", number_of_dialogs belong to that). Specifically in the format as: group 1,number_of_dialogs

2.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provided thorough experiments and explanations of the algorithms to jusify the claim.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed that in the conclusion section and we pointed out three major limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have the proof for the optimal variance reduction proposal in the appendix. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use public LLMs on public benchmarks, and the prompts are all included in the appendix. For our work we have provided enough details for reimplementation, and we will work on open sourcing as well.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The data and prompts are all public and we have provided the information. We are working on open sourcing the code after going through the internal approval process.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the information in the appendix and main paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provided the variance of the results in almost all tables when applicable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mainly use public API.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is a pure algorithmic and enginnering study for new forms of databases.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No risk as far as we can see.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited the models (GPT4, Claude3) we used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: no crowdsourcing nor research

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a Universal Query Engine to directly draw insights from unstructured data. For aggregation queries, this paper designed an unbiased sampling algorithm to solve the problem that it’s hard to apply index on virtual columns. Inspired by the workflow of the C++ compiler, this paper also proposed a workflow of optimizing and generating prompt to achieve more efficient query.\n\nSTRENGTHS:\n1. This paper suited an interesting and important problem, by using LLM, more complex, semantics -related SQL queries can be achieved.\n2. This paper took efficient into consideration and proposed corresponding algorithm and workflow to improve the query speed.\n3. Provided theory proof of the unbiased sampling algorithm.\n4. The idea of introducing compiler’s workflow into such problem to accelerating the LLM’s inference is interesting.\n\nWEAKNESSES:\n1. UQE seems to lack scalability on the Semantic Retrieval task. The latency reported in Table 5 is as high as 38.08 seconds. Also, the paper didn’t state that whether these numbers are the total latency or average latency per query. If they are latency per query, using UQE on the Semantic Retrieval task still has a far distance from practice.\n2. The cost of embedding operation of Algorithm 1 and 2, and the cost of Algorithm 2 are not discussed in the paper. Although the authors wrote in the appendix that the g function in Algorithm 2 caused the high latency problem, no quantitative analysis was provided.\n3. Lack of comparison with important baselines. I noticed that the references [9] and [33] mentioned in related work are not included in the baselines. Using LLM as UDF may significantly slow down the queries, however, they may have better accuracy on aggregation queries. Thus, I think it’s necessary to include them in the baselines to further evaluate the performance of the trade-off made by Algorithm 1 between accuracy and latency. \n4. Lack of evaluation of UQE’s time cost in the main text of the paper. I noticed those results are reported in the appendix. However, in the database area, the latency and cost are significant metrics that should be reported in the main text of the paper.\n\nQUESTIONS:\nI noticed that lc-gpt-4-turbo also has quite a high latency on Semantic Retrieval tasks, is it possible to do more optimization on the model itself to improve the performance to make such a method practice?\n\nLIMITATIONS:\nYes, the authors have mentioned the limitations of this work, but I suggest to provide more comprehensive evaluations, detailed discussions about UQE’s scalability and cost. After all, these are critical problems about the practicality.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.\nFurther, a Universal Query Language (UQL) is proposed as a dialect of SQL that provides full natural language flexibility in specifying conditions and operators.\nUQE leverages the ability of LLMs to conduct analysis of unstructured data and achieve efficient and accurate query execution.\n\nSTRENGTHS:\n1. The idea of designing new query language and query engine for unstructured databases leveraging the power of LLMs is interesting.\n2. The paper is also very clear with thorough experiments and analysis.\n3. Well-organized presentation of the proposed method and its components.\n\nWEAKNESSES:\n1. Many details about the design and implementation of UQE are missing. For example, the description of the generation of machine-specific code (assembly instructions) during the compilation stage (L251) is not clear. Please clarify the details of this strategy. Additionally, the authors claim that calling the LLM dominates the overall cost per query (L239). Providing detailed information about the LLM-calling stage would be beneficial.\n2. The indexing technique introduced by UQE is related to the sampling strategy based on specific queries. However, database indexing is traditionally meant to facilitate quick searches. If each query requires sampling, will this impact efficiency? Additionally, doesn't this compromise accuracy?\n3. This paper primarily addresses unstructured datasets, including video and audio. However, the experiments only involve text and images, lacking validation for broader generalizability. \n4. Furthermore, there are plenty of multi-modal vector databases, yet the authors only compare the RAG-based query. Can UQE outperform recent multi-modal query databases, such as Milvus and MillenniumDB? If so, demonstrating this would make the evaluation more convincing.\n5. Does the query support join operations, such as joining across multiple different modal files?\n6. It's hard to validate the results without supplying code/models.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nNo discussion on limitations, I suggest the authors to add one.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes an ambitious new framework for analytics on unstructured databases, the Universal Query Engine (UQE). The authors first present a list of semantics and clauses for querying unstructured databases and propose methods for implementing these functionalities, including indexing and compiling. Experiments on multimodal unstructured data analytics tasks show that UQE outperforms other methods, demonstrating the promise of utilizing UQE in database analytics.

STRENGTHS:
- The motivation is quite interesting and necessary to broaden the functionalities of databases.
- The paper is clearly written and well-presented.
- The authors presented the exact scope of the task and proposed methods accordingly.
- The proposed approach is reasonable and performs well within the scope of the work.

WEAKNESSES:
- Could you elaborate more on the difference between BINDER and this approach (line 39)? Is it due to the lack of indexing and compatibility with unstructured information?
- Similar to the first point, the baselines seem a bit weak. Why have the authors not used BINDER as their baseline?
- I want to hear the authors' opinions on how advancements in LLMs and vision models might affect the guarantee that databases consistently return the same results. Traditional database engines, which are widely used, provide this consistency over time. Could these technological advancements affect the reliability of database analytics and, subsequently, compromise their reliability?

QUESTIONS:
I have listed my questions in the Weaknesses section.

LIMITATIONS:
The limitations of the work are properly stated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

comments: SUMMARY:
This paper proposes an unstructured query language based on a small segment of SQL. Its key feature is that it further supports unstructured texts and images because the authors assume that LLMs work on intra-row texts. During the query, the authors propose using online learning on LLM outputs over batches to improve the sample quality in the upcoming batches. Empirical results convey important information; that is, feeding longer context into larger LLMs does not work as well as using smaller LLMs in a more organized way.

STRENGTHS:
This paper discusses several possibilities for using LLMs in relational databases with text columns. It reveals a new possibility that one could do something between 1. using LLMs as a black box and letting them handle everything in the context. and 2. using LLMs to parse the question into SQL and let the database engine do the job with much smaller LLMs.

For queries requiring sampling, a simple online learning algorithm is used to improve the sample efficiency.

WEAKNESSES:
- This paper is more like a technical report or a guideline for building certain products. The overall product presented is simplified from the introduction's proposed question. Realizing this system under the strong assumption that LLMs work on intra-row semantic understanding is not too difficult: the virtual column, although it seems to be filled by LLMs, can be realized in several prompts, such as closed-domain classification for WHERE and open-domain classification for GROUPBY.

QUESTIONS:
- Given that the output in the virtual column will work with other parts of the SQL, what is the failure rate in output formatting in the experiments?

LIMITATIONS:
Yes,
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-18