Title: Lucy: Think and Reason to Solve Text-to-SQL
Venue: NeurIPS 2024
Accepted: no
Keywords: Large Language Models

Automated Reasoning

Text-to-SQL
Abstract: Large Language Models (LLMs) have made significant progress in assisting users to query databases in natural language. While LLM-based techniques provide state-of-the-art results on many standard benchmarks, their performance significantly drops when applied to large enterprise databases. The reason is that these databases have a large number of tables with complex relationships that are challenging for LLMs to reason about. We analyze challenges that LLMs face in these settings and propose a new solution that combines the power of LLMs in understanding questions with automated reasoning techniques to handle complex database constraints. Based on these ideas, we have developed a new framework that outperforms state-of-the-art techniques in zero-shot text-to-SQL on complex benchmarks

Abstract

Large Language Models ( LLM s) have made significant progress in assisting users to query databases in natural language. While LLM -based techniques provide state-of-the-art results on many standard benchmarks, their performance significantly drops when applied to large enterprise databases. The reason is that these databases have a large number of tables with complex relationships that are challenging for LLM s to reason about. We analyze challenges that LLM s face in these settings and propose a new solution that combines the power of LLM s in understanding questions with automated reasoning techniques to handle complex database constraints. Based on these ideas, we have developed a new framework that outperforms state-of-the-art techniques in zero-shot text-to-SQL on complex benchmarks.

1 Introduction

Large Language Models ( LLM s) have significantly enhanced AI agents' capacity to assist humans in a variety of important tasks, including co-pilot programming [Chen et al., 2021, GitHub, Inc., 2021], program verification [Wu et al., 2024, Chakraborty et al., 2023], and math problem solving [Zhou et al., 2024]. One of the fastest-growing areas in this space is the development of LLM -based assistants for querying SQL databases. In this task, a user poses a question to a database in natural language. The agent's goal is to generate an SQL query that, when executed against the database, answers the user's question. Such assistance enables users with different levels of expertise to effectively analyze their data.

Recently, LLM -based solutions have made significant progress in addressing the text-to-SQL problem [Gao et al., 2024, Li et al., 2024a]. While GPT-based methods have quickly reached near-human performance on academic benchmarks, like Spider [Yu et al., 2018], they struggle to provide high-quality user assistance on large industrial databases [Sequeda et al., 2023, Li et al., 2023]. One of the core challenges is that industrial databases model many objects with complex relationships between them. To transform a natural language question into an SQL query, the LLM must effectively reason about these intricate relationships, which is highly non-trivial for LLM models. Interestingly, we found that gpt4 can even indicate in some cases that it needs help with logical reasoning on complex databases. Here is a common gpt4 output message on a question that requires multiple joins from ACME insurance database [Sequeda et al., 2023]: 'This join may need adjustment based on the actual logic of relating claims to policy coverage details.' . While we do provide the database schema as part of the input, it is still challenging for LLMs to formally reason about database logic.

In this work, we propose a new text-to-SQL framework, Lucy , designed for large databases with complex relationships between objects. Our main underlying idea is to combine the ability of LLM models to effectively relate user questions to database objects with the power of automated reasoning to analyze relationships between these objects. The Lucy workflow consists of three high-level steps. First, upon receiving a user's question, we identify the relevant objects and their attributes in the target database. In the second step, we employ an automated reasoner to build a view that joins the relevant tables based on relational constraints defined by the database schema. This view contains all the necessary information

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

Star

A

Star

B

Snowflake

Retention

Client

[

id

,

name

,

loc_id

]

Location

[

id

]

[

id

,

name

,

loc_id

]

Bonus

Payment

[

id

]

PayAmount

[

id

,

pay_id

,

amount

]

Tax

[

payamt_id

]

Supercharge

[

payamt_id

]

Income

[

payamt_id

]

Compute

,

[

id

,

dc_id

]

ResourcePool

[

id

]

[

id

,

rspool_id

]

Runtime

[

id

,

rspool_id

]

cMemory

[

config_id

,

overheadlimit

]

cCPU

[

config_id

,

overheadlimit

]

rMemory

[

,

overallusage

]

rCPU

[

,

overallusage

]

♠

:

♠

RsPool2Client

to answer the user's questions. In the third step, we construct a query targeting this view to produce an answer for the user. Our contributions are summarized as follows:

We propose a text-to-SQL framework Lucy capable of querying large industrial databases. To the best of our knowledge, Lucy is the first framework designed to support logical reasoning in the context of the text-to-SQL problem.

Lucy offers several advantages:

-alleviates the need for complex reasoning from a LLM , allowing it to focus on tasks where it currently excels,

-supports modeling and reasoning about complex, commonly used design patterns to model relationships, like many-to-many, Star , and Snowflake ,

-its modular workflow allows for effective debugging of failures,

-performs zero-shot generation and does not require fine-tuning of LLM s.

Our experimental results demonstrate significant performance improvements on several standard benchmarks as well as introduced large benchmarks. We also demonstrate the debugging capabilities of Lucy .

2 Motivation

To provide high-quality user assistance in text-to-SQL tasks, we face two types of challenges.

58 The first type of challenge comes from the formulation of the user's question. A question 59 can be poorly specified, ambiguous, or require additional knowledge that is not present in 60 the question. For example, the user might ask to list clients eligible for a loan; however, the 61 eligibility criteria are not present in the question [Li et al., 2023, 2024b]. The second class is 62 related to the complexity of the queried database that can have a large number of tables 63 with complex relations between them [Sequeda et al., 2023, Li et al., 2023]. In this work, we 64 focus on the second class. One approach to deal with complex relationships is to introduce 65 an intermediate layer, like a knowledge graph or ontology structure, that contains rich 66 information about the underlying database. Then, LLMs generate queries to this knowledge 67 graph using specialized languages, e.g., SPARQL, [Sequeda et al., 2023]. In turn, these 68 queries can be automatically translated to SQL. While this approach does show promise, it 69 does not alleviate the core issue: an LLM is still expected to reason about complex relations 70 between objects in this intermediate representation. Moreover, such a rich intermediate layer, 71 like an ontology, might not be easy to obtain for a database. Other standard techniques, 72 like additional training, multi-shot or fine-tuning, also rely on LLM s to perform constrained 73 reasoning steps [Gao et al., 2023, Pourreza and Rafiei, 2024, Gao et al., 2024]. To the best of 74 our knowledge, dealing with complex relationships in text-to-SQL remains an open problem. 75

In order to isolate the underlying challenges in this problem, we created an example database 76

that covers standard relationship patterns adopted in industry and academia. We identified a 77 set of simple and clearly formulated questions and demonstrated that even on this simplified 78 schema and clear questions, state-of-the-art LLM s struggle to assist the user. 79

80

2.1 Database description

We describe a minimal example database schema that contains basic relations, like ✶ : ✶ 81 and ✶ : ♠ , and more advanced relationship patterns, like ♠ : ♠ and Star , and analyze the 82 performance of LLM s on this schema (See Appendix A for relational database definitions). 83 Suppose a business sells cloud compute resources to customers and uses a database, ddo , 84 to manage its Day-to-Day Operations. Figure 1 shows objects' corresponding tables, their 85 relationships, and a subset of attributes. In particular, each table has a primary key, e.g., 86 Location . id , and might have foreign keys to refer to another table, e.g., Client refers 87 to Location using Client . loc_id . All attributes relevant to our examples are shown in 88 Figure 1 with self-explanatory names. ddo manages payments ( Payment ) and marketing 89 retention strategies ( Retention ) for clients ( Client ) and resources ( ResourcePool ) 90 in datacenters ( Datacenter ). This example is in part inspired by the VMware vSphere 91 data model (discussed in Section 5). The full data model contains hundreds of types of 92 resources that form deep tree-like structures [Managed Object, 2024]. Next, we consider 93 how relationships between objects are modeled in ddo . Figure 1 already defines basic 94 relationships, including : (dotted edges) and : (solid edges). 95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

✶ ✶ ✶ ♠ Many-to-many ( ♠ : ♠ ). Client and ResourcePool are related via a ♠ : ♠ relationship (the dashed edge) meaning that a client might use multiple resource pools and one resource pool can serve multiple clients. The table RsPool2Client models this relation.

Star. A Star pattern is a type of database schema composed of a single, central fact table surrounded by dimension tables. There are two groups of objects connected in a Star patterns in our example. Star A keeps track of retention marketing strategies for each client that can be either Gift or/and Bonus . Star B records clients' payments ( Payment ). Payments' amounts are stored in the PayAmount table. Each amount can be exactly one of three types: Tax , Supercharge , and Income .

Snowflake. A Snowflake schema consists of one fact table connected to many dimension tables, which can be connected to other dimension tables through a many-to-one relationship. In ddo , database resource pools are modeled using the snowflake pattern. Each resource pool has configurations ( Config ) and snapshots of the current usage ( Runtime ). Config and Runtime have two children nodes each to define CPU and memory properties.

Lookup. A lookup table is a table that contains descriptions and code values used by multiple tables, e.g., zip codes, country names. etc. In ddo , Location is a lookup table that stores geo-location related data for quick access.

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

Question

MatchTables

GenerateView

QueryView

dbModel

Relevant

tables (

R

T

)

Summary

view (

V

)

V

,

Q

2.2 User questions

We consider three simple questions to ddo that are well formulated: outputs are explicitly specified, so no additional information is needed to answer them. We use gpt4 ('gpt-4-0125preview'), and promptB [Sequeda et al., 2023] for these questions. For each question, we present a ground truth answer and a GPT answer. Table 1 presents both questions (Q3 is presented in Appendix C.1).

Question Q1 is 'List customers who use datacenters with names starting with 'dev'. Output clients and datacenters names' . The user asks for information that relates clients and datacenters. Consider GPT 's answer. GPT misses the core logic of the database: clients and datacenter resources are related via a ♠ : ♠ relation (modeled with RsPool2Client ). GPT outputs clients and datacenters that share the same location, which is incorrect.

Question Q2 is 'List resource pool names with CPU overhead limit greater than runtime overall usage by 100' . Here the user asks about resource pool properties. However, the GPT answer ignores the database's primary/foreign relations. It performs an inner join between ResourcePool , cCPU , and rCPU tables, using non-existent attributes ResourcePool . config_id and ResourcePool . runtime_id , which is clearly incorrect.

In summary, these examples demonstrated that LLM s struggle to handle complex relationships between objects.

3 Framework design

In this section, we present our framework Lucy . Figure 2 illustrates the workflow diagram, and Algorithm 1 shows the main steps of the workflow. There are two inputs to the framework. The first input is a user question Q . The second input is dbModel , which is a description of the database schema that we discuss in the next section (Section 3.1). The workflow consists of three sequential subtasks: MatchTables , GenerateView , and QueryView . MatchTables identifies the relevant tables and their attributes related to the user question (Section 3.2). GenerateView finds a combined view of relevant tables taking into account database constraints (Section 3.3). The third phase, QueryView , takes V and the user question Q and produces an SQL query Q for V (Section 3.4). To simplify notations, we assume that dbModel is a global variable in Algorithm 1.

3.1 Database model (dbModel)

We start with dbModel , or dbm for short. dbm is a data structure that contains aggregated information about the database, maintained as a JSON structure. dbm should be constructed once for a database as the structure of the database is relatively stable. dbm can always be extended if the database requires modifications. Here are the two main blocks of dbm :

Database schema. The schema is written using the SQL Data Definition Language (CREATE TABLE statements). It includes table names, names and types of columns in each table, and database constraints such as primary and foreign keys. It can also contain optional user comments associated with each table and column. We refer to tables and constraints as dbm .tables and dbm .constraints, respectively. We extract this information in the form of JSON. Appendix D.1.1-D.1.2 shows examples of these structures.

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

Patterns summary. The user can optionally list higher-level design patterns that are not captured by the schema explicitly. This information can help to improve the accuracy of the algorithm. We support ♠ : ♠ , Star , Snowflake , and lookup patterns, but the model is extendable to support other patterns. The user identifies these patterns manually, based on the logic of the target domain. In the future, we envision that the process can be partially automated. Appendix D.1.3 shows the JSON format used to specify pattern structures.

Formal notations. We introduce formal notations. dbm . tables contains a list of tables t i , i ∈ [1 , m ] where m is the number of tables. dbm . constraints contains a set of pairs ( t i , t j ) such that t i and t j are related via ✶ : ✶ , ✶ : ♠ or ♠ : ✶ relation. We denote dbm . ♠ : ♠ as a set of triplets ( t i , t j , t k ), where a join table t k models a ♠ : ♠ relation between tables t i and t j . Note that ( t i , t k ) and ( t j , t k ) must be in dbm . constraints. Additionally, we denote dbm . lookup as the set of lookup tables. For example, in the ddo database, dbm . ♠ : ♠ = { ( Client , ResourcePool , RsPool2Client ) } and dbm . lookup = { Location } . For a tree-like pattern, like Star or Snowflake , we distinguish between root table and inner tables using two predicates, e.g., star_root ( t ) returns True if t is the root table of a Star and star_inner ( t ) returns True if t is an inner table (not root) of a Star .

3.2 The MatchTables phase

The first phase, MatchTables , needs to find relevant tables and their attributes to the user question. One approach to achieve that can be to provide the schema and a question to an LLM and ask for this information. However, one of the distinguishing features of real-world databases is their large number of tables and attributes. Hence, feeding all of them along with their descriptions to the prompt might not be feasible for many LLM models. Therefore, we build an iterative procedure that takes advantage of database tree-like patterns. In general, this procedure can be customized to best support the structure of a database.

Algorithm 1 Lucy

Algorithm 1 shows MatchTables in lines 2-12. First, the algorithm focuses on tables that are not inner tables of any patterns. We refer to such tables as core tables (core_tables in line 3). For example, Figure 3 shows core tables for ddo . Next, we ask LLM to find relevant tables among these core tables using promptA in line 5. (Appendix D.2.1 shows a promptA with a few examples.) As a result, we obtain a set of relevant core tables. We explore them one by one in the loop in line 7. If it is a root table of a pattern, we perform a search inside the corresponding pattern to find more relevant tables using a breadth-first deepening procedure, IterativePrompting , in line 10 (Algorithm 2 shows IterativePrompting 's pseudocode in Appendix D.2). Otherwise, we use promptA to obtain relevant attributes in line 12.

Example 3.1. Consider questions Q 1 and Q 2 from Table 1. Figure 3 shows ddo 's core tables. For Q1, a LLM identifies relevant core tables: T = { Client , Datacenter }

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

Retention

Client

Location

Payment

RsPool2Client

Compute

ResourcePool

...

...

...

(line 5). Since none of these tables is a root of a Snowflake or a Star , we prompt for relevant attributes for each table in line 12 to get R T = { Client . name, Client . gender, Datacenter . name}. Now consider Q 2 . LLM identifies ResourcePool as a relevant table in line 5. As ResourcePool is the root table of Snowflake (see Figure 1), we begin to explore the pattern tree in a breadth-first order using IterativePrompting in line 10. ResourcePool has two child nodes, Config and Runtime , and several attributes. We query the LLM and find that both Config and Runtime are relevant as well as its attribute ResourcePool . name. Following the breadth-first search order, we consider Config with two descendants cCPU and cMemory and discover cCPU is relevant (Example D.4 in Appendix shows a full version).

3.3 The GenerateView phase

The MatchTables phase identifies a set of relevant tables and their attributes. Next, we construct a view table that combines relevant tables and attributes into a single table.

We build an abstract schema graph G which provides a graph view of dbm , and define a CSP over this graph. For each table t i in dbm . tables, we introduce a node in G . We use the names t i to refer to the corresponding nodes. For each pair of tables t i and t j , s.t. ( t i , t j ) ∈ dbm . constraints, we introduce an edge that connects them. We denote V the set of nodes in G and E its edges. Figure 3 illustrates a part of the graph (core tables) for ddo .

Algorithm 1 shows three main steps of this phase: build an abstract graph representation G of the schema (line 15); formulate and solve CSP to obtain a path P (line 17); and perform joins along this path to obtain the designed view V (line 19). Next, we describe these steps.

Problem Formulation. Let T = tables( R T ) be a set of relevant tables returned by MatchTables . We formulate the problem of finding a path P in G that visits a set of nodes T and satisfies a set of database constraints.

( C 1 ) P must be a valid path in G . This ensures that we follow primary/foreign keys relationships, i.e., : , : , and build a valid view.

♠ ♠ ( C 4 ) If t ∈ P and t ∈ dbm . lookup then t 's predecessor equals its successor in P . This ensures that a lookup table serves as a look-up function for each table individually.

✶ ✶ ✶ ♠ ( C 2 ) P visits all relevant tables T . This ensures combining all relevant tables to a view. ( C 3 ) Consider ( t i , t j , t k ) ∈ dbm . ♠ : ♠ . If t i ∈ P and t j ∈ P then t k must occur in P once between t i and t j . These constraints enforce : relationships.

( C 5 ) Cost function: we minimize the number of occurrences of tables outside of T in P . A shorter path that focuses on the tables in T allows us to build more succinct views.

( C 1 )-( C 5 ) are common constraints that we encounter in the benchmark sets. In general, the user can specify more constraints to capture the logical relationships of the modeled data. Constraint satisfaction problem (CSP). We define a CSP formulation S of constraints ( C 1 )-( C 5 ). We start with a basic formulation. Let n be the maximum length of the path P . For each node t i in G and step r , where r ∈ [1 , n ], we introduce a Boolean variable b r i . b r i is true iff t i is the r th node in P . We also introduce a sink-node Boolean variable b r d for each layer to model paths that are shorter than n . S contains the following logical constraints:

229

230

231

232

233

Consider the encoding S . Equations 2 specify integer variables, occ i , for i ∈ [1 , n ], that count the occurrences of each table in the path. Equations 8 encode that only one node belongs to a path at each step. Equations 9 encode that if the path visits the sink node, then it must stay there. Other equations encode constraints ( C 1 )-( C 5 ). By construction, Equations 1-9 generate a valid path in G that satisfies the constraints ( C 1 )-( C 5 ).

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

Example 3.2. For Q1, solving S gives the green path between Datacenter and Client in Figure 3. S rules out the red path as we enforce constraint ( C 4 ) and optimization ( C 5 ).

Improvements of CSP. Our basic model S can be improved to take advantage of Star and Snowflake patterns. Namely, we can leverage the decomposition of G and find a path P among core tables only. Then, for each core table in P that is a pattern root, and for each inner relevant table in this pattern, we build a path P ′ along the corresponding branch. For example, Figure 1 shows two paths from ResourcePool to cCPU (an orange path) and rCPU (a blue path). We use left join to combine tables along each such branch. Finally, we combine P and P 's into a single view.

Summary view. Given a path P in a graph, we join tables along the path using their primary and foreign key relations. We keep the same set of attributes that MatchTables identified. An example of the V for Q1 that corresponds to the green path in Figure 3 is shown in the listing in Table 7 in Appendix D.3.1.

3.4 The QueryView phase.

QueryView takes the summary view V along with the user question, and prompts an LLM to obtain the final SQL using promptC (line 21 in Algorithm 1). promptC is defined in Appendix D.4.1. The listing in Table 7 shows an SQL Q to answer Q1 (Appendix D.3.1).

4 Discussion on strengths and limitations

Strengths. Lucy is designed based on the principle of separation of responsibilities between generative tasks and automated reasoning tasks: each step focuses on either an NLP-related subproblem or a constraint reasoning subproblem. This separation allows us to support a number of unique capabilities. First, Lucy shifts the burden of complex reasoning from LLM s to constraint solvers. Second, we support reasoning on complex relationships, like ♠ : ♠ , lookup , Star or Snowflake . Third, our framework is flexible and extensible as it is easy to incorporate domain-specific constraints as soon as they can be expressed by constraint modeling language. This assumes that the user has a data analytics role and understands the logic of the database. Such formal reasoning capability is important, as it is hard to control LLM s via prompts when non-trivial reasoning is required. Fourth, we can evaluate each phase and diagnose Lucy failure modes. For example, if MatchTables misses relevant tables, this indicates that we need to provide more information about the schema to an LLM . Fifth, based on our evaluation, Lucy can support complex queries that include multiple filtering operators and aggregators, e.g. average or sum. This capability follows from the QueryView phase as the final call to an LLM is performed on a single view table.

Limitations. The first limitation is that we cannot guarantee that the SQL query answers the user's question. Given the current state of the art, providing such guarantees is beyond the reach of any copilot method that takes natural language descriptions and outputs structured text, like code or SQL. However, our solution does guarantee that V satisfies database constraints, which is a step forward in this direction. Second, we do not support questions

that require union operators in the GenerateView phase. In fact, there are no benchmarks 272 available that require the union operator to answer questions. Supporting union would 273 require an extension of MatchTables and GenerateView . Third, we observed experimentally 274 that Lucy struggles with certain types of queries that involve a particular interleaving 275 ordering of filtering and aggregate operators or question-specific table dependencies, like a 276 lookup table that has to be used multiple times to answer the user's question. We further 277 discuss such questions in our experiments. 278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

5 Experimental evaluation

In our experimental evaluation, we aim to answer the main questions:

Is Lucy competitive with existing LLM -based approaches?

Can we debug Lucy to gain insights about failure modes?

Can Lucy handle complex questions?

Setup. We compare with the following zero-shot baselines: gpt4 , nsql , and chat2query ( c2q for short). gpt4 and c2q methods are the best zero-shot techniques according to the BIRD leadership board that are accessible for evaluation [Li et al., 2024b]. nsql is the best open-source large foundation model designed specifically for the SQL generation task [Labs, 2023b]. chat2query is closed-source but the authors kindly extended their API that we can run experiments with gpt4 . We provide all benchmarks and frameworks' results in the supplementary materials. For gpt4 and Lucy , we use the 'gpt-4-0125-preview' API without fine-tuning. We use OR-Tools as a constraint solver [Perron and Didier, 2024] (Appendix E.1 provides full details of the experimental setup).

Evaluation metrics. We use the standard Execution Accuracy ( ex ) [Li et al., 2023]. In addition, we consider a relaxation of this metric. We noticed that frameworks often add additional attributes to the output as the exact format of the output is rarely specified. Hence, we extend ex to esx metrics that check if the output of a framework contains the ground truth outputs. To better understand performance characteristics and possible failure modes, we consider the coverage metric that captures whether a framework correctly identified a subset of relevant tables and attributes. Let sql G be the ground truth answer and sql F be a generated query. Then we assess the percentage of the ground truth content slq F captures:

where tables () and attributes () are functions that return a set of tables and attributes. 301

302

303

304

305

306

307

308

309

310

311

312

313

314

ACME insurance. We consider the ACME insurance dataset that was recently published [Sequeda et al., 2023]. The dataset represents an enterprise relational database schema in the insurance domain. The authors focused on a subset of 13 tables out of 200 tables and proposed a set of 45 challenging questions. We identified two Star patterns in this database. The authors showed that their method ( dw ) solved 24 out of 45 problems using intermediate representation of a knowledge graph, while gpt4 solved only 8 problems. However, results are not publicly available, so we cannot perform coverage analysis and compute esx .

We reran the experiment on gpt4 with the same promptB (Appendix C.1.1) and obtained similar results to those reported in [Sequeda et al., 2023]. In addition, we extended the schema with descriptions of table attributes from dbModel in the form of comments, which we called gpt4ex (See Appendix E.2 for examples). Table 2 shows our results. First, we observe that there is a strong correlation between coverage and accuracy metrics in the results. c2q and Lucy show good coverage, meaning that they can correctly identify most of

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

the required tables and attributes. They also demonstrate better performance compared to other methods. Our framework shows very high coverage and solves about 30 of benchmarks according to the ex metric, which outperforms dw that solves 24 and other methods.

Lucy still cannot solve 13 benchmarks, which is surprising given high coverage. We performed a study to locate where Lucy fails on these benchmarks (See Appendix E.2.1 for all questions where Lucy was unsuccessful). In summary, the majority of failures come from under-specified output attributes or nonstandard aggregators, like specialized formulas to compute an average. In four cases, MatchTables missed a table, and in one case, QueryView missed the attribute to output. The most interesting mode of failure is when we need to perform multiple lookups on the same table. The reason for that is the MatchTables phase identifies only relevant tables but ignores possible relationships between them. Extending MatchTables to retrieve relationships between tables is interesting future work.

BIRD datasets. Next, we consider the state-of-the-art dataset BIRD [Li et al., 2023]. From the development set, we chose two datasets with complex relationships between objects: financial (106 instances) and formula 1 (174 instances) 1 . The accuracy of chat2query on the BIRD development set is ∼ 58%; however, its accuracy on financial and formula 1 are much lower, ∼ 45%. We compare with results from gpt4'23 and c2q available from [AlibabaResearch, 2020] and [TiDBCloud, 2020a], respectively. However, we reran these benchmarks with gpt4 and gpt4ex as the gpt4'23 results are nearly one year old. Table 5 and Table 4 show results on financial and formula 1, respectively. Lucy and c2q have higher coverage and good accuracy. Lucy shows the best results in most cases. Again, Lucy has very good coverage on financial but was able to solve only 68 out of 106 queries based on the esx metric. We manually performed an questions study on the failed questions. There are two major groups there that are interesting. First, Lucy has difficulty if there are multiple orderings, especially nested or ordering in different directions. Second, sometimes, MatchTables adds an additional table that is not needed to find the answer. The rest are either ambiguous questions or small mistakes like outputting a wrong attribute, i.e., id instead of name . See Appendix E.3.3 for examples of questions where Lucy was unsuccessful.

Cloud resources. Next, we propose a new benchmark based on the vSphere API data model [VMware, Inc., 2024]. We experimented with this publicly available data model of an industrial product, as it is well-documented and easily accessible via a web interface. It describes the state of the system as well as its configuration parameters. States are stored in a database and queried by customers to keep track of performance, maintenance, and data analysis. We extracted the descriptions of main objects in Managed Object [2024], including data centers, resource pools, hosts, and virtual machines and their properties, and built a database that captures these relationships using 52 tables. Overall, we have two Star s, five Snowflake s and two ♠ : ♠ s patterns. For each table and an attribute, we get descriptions from [Managed Object, 2024]. As these can be a lengthy description, we use GPT to shorten it to 15 words (see promptD in Appendix E.3.2) . We generated data randomly using sqlfaker [Kohlegger, 2020]. We create 20 challenging questions for this benchmark.

Table 3 shows our results. nsql cannot process this benchmark due to a limited context window. We again see that Lucy outperforms other models in both coverage and accuracy. c2q failed on 6 questions with an error 'Unable to generate SQL for this database due to its extensive tables' and it often does not follow instructions on the output columns. In terms of failure mode, Lucy failed in the third phase as it hallucinated some attribute names when names are long, e.g., 'Resourcepoolruntimemory' instead of 'Resourcepoolruntimememory'.

References 362

363

364

365

AlibabaResearch. BIRD-SQL: A BIg Bench for Large-Scale Relational Database Grounded Text-to-SQLs. https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/bird , 2020. Accessed: May 2024.

Beaulieu. Learning SQL . O'Reilly Media, Inc., 2nd edition, 2009. ISBN 9780596520830. 366

Chakraborty, S. Lahiri, S. Fakhoury, M. Musuvathi, A. Lal, A. Rastogi, 367 N. Swamy, and R. Sharma. Ranking llm-generated loop invariants for pro368 gram verification. In 2023 Empirical Methods in Natural Language Processing , 369 December 2023. URL https://www.microsoft.com/en-us/research/publication/ 370 ranking-llm-generated-loop-invariants-for-program-verification/ . EMNLP371 Findings 2023. 372

Chang and E. Fosler-Lussier. How to prompt llms for text-to-sql: A study in zero-shot, 373 single-domain, and cross-domain settings. arXiv preprint arXiv:2305.11853 , 2023. 374

Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, 375 Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, 376 G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, 377 M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, 378 E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, 379 I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, 380 V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, 381 P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. 382 Evaluating large language models trained on code. CoRR , abs/2107.03374, 2021. URL 383 https://arxiv.org/abs/2107.03374 . 384

385

386

datadotworld, Inc. data instances. https://github.com/datadotworld/ cwd-benchmark-data/issues/3 , 2024. Accessed: May 2024.

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

Dong, C. Zhang, Y. Ge, Y. Mao, Y. Gao, lu Chen, J. Lin, and D. Lou. C3: Zero-shot text-to-sql with chatgpt, 2023.

Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou. Text-to-sql empowered by large language models: A benchmark evaluation, 2023.

Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou. Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow. , 17(5):1132-1145, 2024. URL https://www.vldb.org/pvldb/vol17/p1132-gao.pdf .

GitHub, Inc. GitHub Copilot. https://copilot.github.com/ , 2021. Accessed: May 2024.

Hui. Panel of BIRD Annotation Issues. https://github.com/AlibabaResearch/ DAMO-ConvAI/issues/39 , 2024. Accessed: May 2024.

IBM, Inc. Dimensional schemas. https://www.ibm.com/docs/en/ida/9.1.2?topic= design-dimensional-schemas , 2021. Accessed: May 2024.

Kohlegger. sqlfaker. https://pypi.org/project/sqlfaker/#description , 2020. Accessed: May 2024.

S. Labs. Nstext2sql: An open source text-to-sql dataset for foundation model training, July 2023a. URL https://github.com/NumbersStationAI/NSQL?tab=readme-ov-file .

S. Labs. Nstext2sql: An open source text-to-sql dataset for foundation model training, July 2023b. URL https://github.com/NumbersStationAI/NSQL .

Li, J. Zhang, H. Liu, J. Fan, X. Zhang, J. Zhu, R. Wei, H. Pan, C. Li, and H. Chen. Codes: Towards building open-source language models for text-to-sql, 2024a.

Li, B. Hui, G. Qu, B. Li, J. Yang, B. Li, B. Wang, B. Qin, R. Cao, R. Geng, N. Huo, 407 X. Zhou, C. Ma, G. Li, K. C. Chang, F. Huang, R. Cheng, and Y. Li. Can LLM 408 already serve as A database interface? A big bench for large-scale database grounded 409 text-to-sqls. CoRR , abs/2305.03111, 2023. doi: 10.48550/ARXIV.2305.03111. URL 410 https://doi.org/10.48550/arXiv.2305.03111 . 411 J. Li, B. Hui, G. Qu, B. Li, J. Yang, B. Li, B. Wang, B. Qin, R. Cao, R. Geng, N. Huo, X. Zhou, 412 C. Ma, G. Li, K. C. Chang, F. Huang, R. Cheng, and Y. Li. BIRD_SQL: Leaderboard 413 - Execution Accuracy (EX). https://bird-bench.github.io/ , 2024b. Accessed: May 414 2024. 415 A. Liu, X. Hu, L. Wen, and P. S. Yu. A comprehensive evaluation of chatgpt's zero-shot 416 text-to-sql capability, 2023. 417 Managed Object. Managed Object Types Overview, 418 VMware, Inc. https://vdc-download.vmware.com/ 419 vmwb-repository/dcr-public/c476b64b-c93c-4b21-9d76-be14da0148f9/ 420 04ca12ad-59b9-4e1c-8232-fd3d4276e52c/SDK/vsphere-ws/docs/ReferenceGuide/ 421 index.html , 2024. Accessed: May 2024. 422 L. Perron and F. Didier. Cp-sat, 2024. URL https://developers.google.com/ 423 optimization/cp/cp_solver/ . 424 M. Pourreza and D. Rafiei. Dts-sql: Decomposed text-to-sql with small large language 425 models, 2024. 426 F. Rossi, P. van Beek, and T. Walsh, editors. Handbook of Constraint Programming , volume 2 427 of Foundations of Artificial Intelligence . Elsevier, 2006. ISBN 978-0-444-52726-4. URL 428 https://www.sciencedirect.com/science/bookseries/15746526/2 . 429 J. Sequeda, D. Allemang, and B. Jacob. A benchmark to understand the role of knowledge 430 graphs on large language model's accuracy for question answering on enterprise sql 431 databases, 2023. 432 L. Silverston, W. H. Inmon, and K. Graziano. The data model resource book: a library of 433 logical data models and data warehouse designs . John Wiley & Sons, Inc., USA, 1997. 434 ISBN 0471153648. 435 TiDBCloud. chat2query bench. https://github.com/tidbcloud/chat2query_bench , 436 2020a. Accessed: May 2024. 437 TiDBCloud. Get Started with Chat2Query API. https://docs.pingcap.com/tidbcloud/ 438 use-chat2query-api , 2020b. Accessed: May 2024. 439 VMware, Inc. VMware vSphere API Reference Documentation. 440 https://vdc-download.vmware.com/vmwb-repository/dcr-public/ 441 c476b64b-c93c-4b21-9d76-be14da0148f9/04ca12ad-59b9-4e1c-8232-fd3d4276e52c/ 442 SDK/vsphere-ws/docs/ReferenceGuide/index.html , 2024. Accessed: May 2024. 443 N. Wretblad, F. G. Riseby, R. Biswas, A. Ahmadi, and O. Holmström. Finan444 cial annotations, March 2024a. URL https://github.com/niklaswretblad/ 445 the-effects-of-noise-in-text-to-SQL/blob/main/annotations/financial_ 446 annotations.xlsx . 447 N. Wretblad, F. G. Riseby, R. Biswas, A. Ahmadi, and O. Holmström. Understanding the 448 effects of noise in text-to-sql: An examination of the bird-bench benchmark, 2024b. 449 H. Wu, C. Barrett, and N. Narodytska. Lemur: Integrating large language models in 450 automated program verification. In The Twelfth International Conference on Learning 451 Representations , 2024. URL https://openreview.net/forum?id=Q3YaCghZNt . 452

Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, 453 Z. Zhang, and D. Radev. Spider: A large-scale human-labeled dataset for complex and cross454 domain semantic parsing and text-to-SQL task. In E. Riloff, D. Chiang, J. Hockenmaier, 455 and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural 456 Language Processing , pages 3911-3921, Brussels, Belgium, Oct.-Nov. 2018. Association for 457 Computational Linguistics. doi: 10.18653/v1/D18-1425. URL https://aclanthology. 458 org/D18-1425 . 459

Zhou, K. Wang, Z. Lu, W. Shi, S. Luo, Z. Qin, S. Lu, A. Jia, L. Song, M. Zhan, and H. Li. 460 Solving challenging math word problems using GPT-4 code interpreter with code-based 461 self-verification. In The Twelfth International Conference on Learning Representations , 462 2024. URL https://openreview.net/forum?id=c8McWs4Av0 . 463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

A Background

Relational databases. Let D 1 , . . . , D n be a set of domains. A relation or table, t , is defined over subset of domains: t ( X i 0 , . . . , X i k ) ⊆ D i 0 × . . . × D i k , X i j ⊆ D i j , j ∈ [0 , k ]. In addition, t defines a set of attributes (or columns) names X i 0 , . . . , X i k . Projection is a unary operation on a set of attribute names Y , Y ⊆ X . The result of such projection is the set of tuples that is obtained when all tuples in t are restricted to attributes Y . Inner join, or simply join , is a binary operator between two tables t 1 and t 2 over their common attributes Y that returns a set of all combinations of tuples in t 1 and t 2 that are equal on Y . Left join, left join , is similar to the join but returns all rows of t 1 filling unmatched rows of t 2 with null values. A database can support a large set of constraints over tables. The two main constraint types are related to primary and foreign keys. A primary key is the smallest subset of attributes guaranteed to uniquely differentiate each tuple in a table. A foreign key is a subset of attributes Y in a table t 1 that corresponds with (usually) a primary key of another table t 2 , with the property that the projection of t 1 on Y is a subset of the projection of t 2 on Y [Beaulieu, 2009].

Design patterns. A database typically represents entities and their interactions in realworld processes, e.g., the financial management of a company. To effectively model these complex entities, several design patterns have been developed [IBM, Inc., 2021, Silverston et al., 1997]. A many-to-one pattern ( ♠ : ✶ ) specifies a relationship when any number of attributes from one table is associated with unique attributes of the same or another table, typically enforced by foreign key and primary key relationships. A many-to-many relationship ( ♠ : ♠ ) occurs when any number of attributes from one table is associated with any number of attributes from the same or another table. It is typically modeled with an auxiliary join table that refers to the primary keys of the tables in the relationship. The lookup table is a table that contains descriptions and code values used by multiple tables, e.g., zip codes, country names. etc. A Star pattern is a type of relational database schema composed of a single, central fact table surrounded by dimension tables. A Snowflake schema consists of one fact table connected to many dimension tables, which can be connected to other dimension tables through a many-to-one relationship.

Constraint satisfaction. A constraint satisfaction problem (CSP) consists of a set of variables, each with a finite domain of values, and a set of constraints specifying allowed combinations of values for subsets of variables [Rossi et al., 2006]. A solution is an assignment of values to the variables satisfying the constraints. In the constraint optimization problem, we are looking for a solution that optimizes a given cost function. Constraint solvers typically explore partial assignments enforcing a local consistency property using either specialized or general-purpose propagation algorithms and employ conflict-driven learning to store information from failures as the search proceeds. We used OR-Tools CP-SAT solver [Perron and Didier, 2024] in our experiments.

B Related work

We focus on the zero-shot text-to-SQL problem, which has received significant attention in the last few years. Liu et al. [2023] performed a comprehensive evaluation of ChatGPT on the Spider dataset and demonstrated that it shows good performance. In [Dong et al., 2023], a new framework based on the GPT model was proposed, involving several techniques for promoting and post-processing the output to get more consistent results. Chang and Fosler-Lussier [2023] proposed several techniques to improve the performance of ChatGPT. [TiDBCloud, 2020b] represents the most recent zero-shot method. According to the API documentation [TiDBCloud, 2020b], the authors construct a data summary object that contains 'AI exploration information of the given database.' This method performs very well on the BIRD dataset. However, it relies on LLM s to reason about database relationships. Sequeda et al. [2023] performed an interesting investigation of the performance of LLM s on large industrial databases. They identified that GPT does not perform well when it needs to reason about complex relationships. The authors proposed a two-step approach to tackle this problem. As a knowledge graph is available for these benchmarks, the authors proposed

using the knowledge graph as an intermediate representation. Namely, the user's question is 517 answered using the KG structure with SPARQL, and this answer is automatically translated 518 to SQL using a given mapping from ontology to SQL (R2RML). However, while reasoning 519 on a knowledge graph can be easier for LLM s, it is still challenging to take all complex 520 relationships into account. 521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

C Motivation (additional materials)

C.1 User's questions

The third question Q3, 'What are the total tax payments, which is the sum of Tax and Supercharge?' , asks about the total amount of taxes paid from all payments (Table 6). There are a few issues with the GPT answer. First, it outputs all payment amounts that are both tax and supercharge. We reminded that that each payment amount can be of one type, so the result will be empty. Second, it hallucinates as there are no amount columns in the Tax or Supercharge tables.

C.1.1 Definition of promptB

D Framework design (additional materials)

D.1 Database model (dbModel)

D.1.1 Example of a table from dbm.tables

Here is a JSON structure for the Client table from the financial dataset [Li et al., 2023]. It contains the table name, primary keys, attributes, their types, and descriptions. This information is available in the dataset. The description of the table is generated by gpt4 using the prompt promptD .

6 545 7 "path_to_types": ""<path-to>/Client_types.json" 546 8 } 547 548 Here is the JSON structure for Client.json: 549 550 1 { 551 2 "NameField": "Client", 552 3 "DescriptionField": "Focuses on client information, 553 encompassing unique client identifiers, gender, birth 554 dates, and the location of the branch with which they 555 are associated.", 556 4 "client_id": "the unique number", 557 5 "gender": " Description: 'F: female; M: male '", 558 6 "birth_date": "birth date", 559 7 "district_id": "location of branch" 560 8 } 561 562 Here is the JSON structure for Client_types.json: 563 564 1 { 565 2 "NameField": { 566 3 "type": "varchar(100)", 567 4 "default": "DEFAULT NULL" 568 5 }, 569 6 "DescriptionField": { 570 7 "type": "varchar(5000)", 571 8 "default": "DEFAULT NULL" 572 9 }, 573 10 "client_id": { 574 11 "type": "bigint", 575 12 "default": "NOT NULL" 576 13 }, 577 14 "gender": { 578 15 "type": "varchar(46)", 579 16 "default": "NOT NULL" 580 17 }, 581 18 "birth_date": { 582 19 "type": "date", 583 20 "default": "NOT NULL" 584 21 }, 585 22 "district_id": { 586 23 "type": "bigint", 587 24 "default": "NOT NULL" 588 25 } 589 26 } 590 591 D.1.2 Example of a ♠ : ✶ relation from dbm.constraints 592 Here is the JSON structure for the Client and District relation from the financial dataset [Li 593 et al., 2023]. 594 595 1 "Client, District": { 596 2 "type": "Relationships", 597 3 "sqlrelation": "M:1", 598 4 "foreign_relation": { 599 5 "FOREIGN": [ 600 6 "district_id" 601 7 ], 602

D.1.3 Example of a ♠ : ♠ pattern from dbm.patterns 610 Here is the JSON structure for the Account and District ♠ : ♠ relation from the financial 611 dataset [Li et al., 2023]. 612

Here is the JSON structure for the Snowflake pattern rooted ta ResourcePool ( Cloud 637 Resources dataset). 638

D.2 MatchTables 652

D.2.1 promptA 653

promptA requires three inputs: a user question, a set of tables (can be empty), and a set of 654 attributes for a given table t (can be empty). 655

656

657

658

659

660

Inputs: Question, Tables, Attributes promptA: Here is a json schema. Please treat json schema objects as a description of tables in a database <JSON(Tables, Attributes)>. The user has a query to answer <Question>. What are all relevant json elements to a user query from the list [<list of json elements>]? Output is a list of elements, [element, element, element,...]. Do not explain. Returns: We post-process the output to extract a set of tables and their attributes

( R T ) and relevant tables T

In the prompt, we provide description of tables and attributes from dbm . We show a few examples of <JSON(Tables, Attributes)> and the corresponding <list of json elements>.

Example D.1. Here is an example of a JSON(Tables, {}) used in line 5 in Algorithm 1 for financial dataset. The goal is to determine relevant core tables.

Moreover, if a table is a root table of a pattern, we provide inner tables and their attribute 765 names so that an LLM can determine the relevance of Snowflake to the user question. 766

767

768

Example D.3. Here is an example of the Snowflake summary rooted at ResourcePool from the Cloud Resources benchmark.

D.2.2 Description of the IterativePrompting algorithm. 799

Algorithm 2 IterativePrompting

Example D.4 (Full version of Example 3.1 for the question Q2) . Consider Q 2 from Table 1. 800 Figure 3 shows ddo 's core tables. LLM identifies ResourcePool as a relevant table 801 in line 5, along with its attribute ResourcePool . name. Since ResourcePool is the 802 root table of a Snowflake pattern, we begin to explore the pattern tree in a breadth-first 803 order using IterativePrompting in line 10. See Figure 1 for the structure of the the 804 Snowflake pattern. ResourcePool has two child nodes, Config and Runtime , and 805 several attributes. We then query an LLM and find that both Config and Runtime are 806 relevant as well its attribute ResourcePool . name. Following the breadth-first search order, 807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

we next consider Config which has descendants cCPU and cMemory and a few attributes. We discover that only one of them, cCPU , is relevant. We then move to the next table in order, Runtime . It has two descendants rCPU and rMemory and a few attributes. We discover that only one of them, rCPU , is relevant. Next, we identify relevant attributes of cCPU in line 7 (Algorithm 2) and find that cCPU .overheadlimit is relevant to the user query. Finally, we identify relevant attributes of rCPU in line 7 in (Algorithm 2) and find that rCPU .overallusage is relevant to the user query.

D.3 The GenerateView phase

D.3.1 Summary view.

Consider again the question Q1 from Example 3.1. The view V that corresponds to the green path in Figure 3 is shown in the listing in Table 7. We keep the same set of attributes that MatchTables identified. In addition, we also perform renaming of all attributes, as we can control the length of the aliases (in case they are too long). For example, Client . name gets an alias Client _ name , Client . gender gets Client _ gender , so on.

D.4 The QueryView phase

D.4.1 promptC

Here is promptC that we use in the final phase QueryView (Algorithm 1, line 21). The function name() returns name of the view V .

Inputs: Question, V promptC I created a view table <name( V )> with all relevant information. Here is a view < V >. Please write MySQL query to name( V ) view to answer the following question: <Question>. Use only name( V ) columns in the query. Absolutely NO columns renaming. Absolutely NO HAVING operators. Absolutely NO COUNT(*). Output query that I can run via python interface. Output '''sql...'. Do not explain. Returns: SQL

We used a few assertive statements that we discuss next. 'Absolutely NO column renaming' means that we want to use aliases in the view table to form a valid SQL query. The statement 'Absolutely NO HAVING operators.' reflects our observation that gpt4 cannot generate valid SQL when using HAVING in combination with GROUP BY. It is a subject of future research to deal with MySQL constraints, so we encourage QueryView to avoid this operator.

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

Finally, we discourage the use of COUNT(*), 'Absolutely NO COUNT(*)', to ensure that gpt4 focuses on counting the entities specified in the user's question.

We noticed that better results are obtained if we provide a description of tables that are used to generate this view together with their relevant attributes. Here is an extended version of promptC where we provide relevant tables and their attributes that are used to obtain the V . We also provide an evidence if available.

Inputs: Question, V , DB_SCHEMA promptC' (with evidence and a part of the schema) Here is a SQL schema for in MySQL: <DB_SCHEMA> I created a view table <name( V )> with all relevant information. Here is a view < V >. Please write MySQL query to name( V ) view to answer the following question: <Question>. Additional knowledge to answer: <Evidence> Use only name( V ) columns in the query. Absolutely NO columns renaming. Absolutely NO HAVING operators. Absolutely NO COUNT(*). Output query that I can run via python interface. Output '''sql... ' . Do not explain. Returns: SQL

E Experimental evaluation (additional materials)

E.1 Setup

We run experiments on a laptop Intel(r) Core 2.40Hz and 32GB of memory. For nsql we use the largest model with 7B parameters (NumbersStation/nsql-llama-2-7B [Labs, 2023a]). For gpt4 and Lucy , we use the 'gpt-4-0125-preview' model as a LLM and set the temperature to 0.2 . We do not fine-tune a LLM . We require 20 answers from gpt4 for each question. If the number of correct answers is more than 5, then we count that benchmark as solved.

In the case of Lucy , we require 5 answers for each GPT call for the MatchTables phase. We sort tables based on the number of occurrences in these answers and take at most 8 candidates among relevant tables from each promptA output. Similarly to gpt4 , we require 20 answers from QueryView and decide on the success as described above. We use ORTools as a constraint solver [Perron and Didier, 2024].

We support MySQL as a relational database. However, BIRD uses SQLite. We automatically, converted queries from sqlite to MySQL.

We provide all benchmarks and their results in the supplementary materials.

E.2 ACME insurance

Note on the database. There are a few issues with broken relational constraints due to missing tables, as reported [datadotworld, Inc., 2024], which we fixed by adding the missing tables from the original database.

Extended schema examples. Example of tables extended with comments that describe each attribute for the ACME insurance benchmark.

911

912

913

914

915

916

E.2.1 Challenging questions

In this section, we present 13 questions that Lucy found challenging to answer and identify reasons for these failures.

Question1: What are the loss payment, loss reserve, expense payment, expense reserve amount by claim number and corresponding policy number, policy holder, premium amount paid, the catastrophe it had, and the agent who sold it? Reason: Multiple lookups. "policy holder" and "agent" require a look up to the same table Agreement_Party_Role.

Question2: What are the total loss, which is the sum of loss payment, loss reserve, expense payment, expense reserve amount by claim number and corresponding policy number, policy holder and premium amount paid?

Reason: Phase 1 issue. Phase 1 misses the relevant table Agreement_Party_Role.

Question3: What is the total amount of premiums that a policy holder has paid? Reason: Phase 3 issue. Phase 3 makes a mistake in the group by clause.

Question4: What are the total loss, which is the sum of loss payment, loss reserve, expense payment, expense reserve amount by catastrophe and policy number? Reason: Ambiguous question. By "by catastrophe", the user means to output Catastrophe's attribute Name. However, Phase 1 identifies Catastrophe's attribute Identifier as relevant instead of Name.

917

Question5: What is the average policy size which is the the total amount of premium divided by the number of policies?

Reason: Ambiguous question. The definition of average is not standard, as the same policy can have multiple amount values.

918

Question6: What are the loss payment, loss reserve, expense payment, expense reserve amount by claim number and corresponding policy number, policy holder, premium amount paid and the agent who sold it? Reason: Multiple lookups.

919

Question7: Return agents and the policy they have sold that have had a claim and the corresponding catastrophe it had.

Reason: Ambiguous question. The output includes Company_Claim_Number, although this information is not specified in the question.

920

Question8: What is the loss ratio of each policy and agent who sold it by policy number and agent id?

Reason: Ambiguous question. "the loss ratio" is a complex formula here, making it hard to guess without its proper specification.

921

Question9: What are all the premiums that have been paid by policy holders? Reason: Ambiguous question. Policy.Policy_Number and Party_Identifier should be included in the output. But they are not specified in the question.

922

Question10: What are the loss payment, loss reserve, expense payment, expense reserve amount by claim number and corresponding policy number, policy holder and premium amount paid?

Reason: Phase 1 issue. Phase 1 misses the relevant table Agreement_Party_Role.

923

Question11: What is the loss ratio, number of claims, total loss by policy number and premium where total loss is the sum of loss payment, loss reserve, expense payment, expense reserve amount and loss ratio is total loss divided by premium? Reason: Phase 1 issue. Phase 1 misses the relevant table Policy.

924

Question12: What are the total loss, which is the sum of loss payment, loss reserve, expense payment, expense reserve amount by claim number, catastrophe and corresponding policy number?

Reason: Phase 1 issue. Phase 1 misses the relevant table Catastrophe.

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

Question13: What is the total amount of premiums that a policy holder has paid by policy number?

Reason: Ambiguous question. Party_Identifier is included in the output. But it is not specified in the question.

E.3 BIRD datasets

E.3.1 Additional notes on the dataset.

Note on dbModel. We used attribute descriptions available in BIRD in dbModel . We also build table descriptions in the following way. We provided the description from BIRD to an LLM to generate a short summary description using promptD defined in Section E.3.2.

Note on datasets. It has been shown that there are a number of incorrect ground truth SQLs in BIRD datasets [Hui, 2024, Wretblad et al., 2024b]. For example, Wretblad et al. [2024b] found that 72 out of 106 benchmark questions in financial have errors of various types. Most of the issues have been reported to the authors from multiple sources, and we also reported additional problems via private communication. The authors acknowledge these issues and are working on them. To provide an example we reported from formula 1:

Question: 'Where can the introduction of the races held on Circuit de BarcelonaCatalunya be found?'

Ground truth SQL: select distinct circuits.url from circuits inner join races on races.circuitId = circuits.circuitId where circuits.name = 'Circuit de BarcelonaCatalunya'.

The issue is that select should be on race.url rather than circuits.url as the user requests information about the race, not the circuit.

On top of that, there are logical inconsistencies in ground truth answers for the financial dataset. Often, users ask for information about clients' accounts. Client and account tables have a ♠ : ♠ relationship modeled using an additional table disp. At the same time, they are both related to a lookup table district. Unfortunately, many ground truth SQL statements perform a join between clients and accounts via the district table, which is incorrect. Let's consider an example.

Question: 'Please provide the IDs of the 3 female clients with the largest loans.'

Ground truth SQL: select T1.client_id from client as T1 inner join account as T2 on T1.district_id = T2.district_id inner join loan as T3 on T2.account_id = T3.account_id where T1.gender = 'F' order by T3.amount DESC LIMIT 3

The issue is that the answer relates clients and accounts that have the same dis-trict_id. However, this does not mean that the client is related to the account.

As the authors are working on corrections, we analyzed the reported issues and manually corrected the ground truth. We only adjusted the SQL ground truth values to match the user questions; we did not alter the questions or evidences. We provide the corrected benchmarks in the supplementary material. Specifically, we corrected the financial and formula 1 benchmarks and used the correct answers to evaluate all methods. Interestingly, the performance of all frameworks improved by a few percentage points when we fixed these ground truth SQL statements.

E.3.2 promptD

Here is promptD that we use to generate tables summaries for financial and formula 1 datasets.

promptD Give me a very brief description of the <NAME> table.

968

969

Example for the Loan table (the financial database) Prompt: Give me a very brief description of the Loan table. original_column_name,column_name,column_description,data_format, value_description loan_id'the id number identifying the loan data,integer, account_id'the id number identifying the account,integer, date'the date when the loan is approved,date, amount'approved amount,integer,unit: US dollar duration'loan duration,integer,unit: month payments,monthly payments,monthly payments,real,unit: month status'repayment status,text,"'A' stands for contract finished, no problems; 'B' stands for contract finished, loan not paid; 'C' stands for running contract, OK so far; 'D' stands for running contract, client in debt" Return: The Loan table manages loan-related data, offering insights into each loan's unique identifier, associated account details, approval dates, amounts, durations, and monthly payments.

E.3.3 Challenging questions

We discuss three major groups of challenging questions with examples. 970

The first group contains ambiguous questions. Here are a few examples. 971

Question: List out the no. of districts that have female average salary is more than 6000 but less than 10000?

Reason: Ambiguous question. 'no. of districts' refers to the district number based on the ground truth. However, Lucy counts the number of districts.

Question: Wthat the client whose card was opened in 1996/10/21 made? Reason: Ambiguous question. Lucy filters on 'card issued date', while ground truth filters on 'account opened date' . However, the user is indeed asking about 'card open date' in this question. This issue was also independently observed in [Wretblad et al., 2024a].

972

973

The second group contains complex filtering, ordering, and/or formulas to compute. Here 974 are a few examples. 975

Question: List out the account numbers of clients who are youngest and have highest average salary?

Reason: Phase 3 issue. There are two filtering conditions that have to be applied in order. First, we find the youngest clients, then select the one with the highest average salary among them. Lucy treats these conditions as a conjunction, resulting in an empty output.

Question: List out the account numbers of female clients who are oldest and has lowest average salary, calculate the gap between this lowest average salary with the highest average salary?

Reason: Phase 3 issue. Two filtering conditions are required: first, in descending order, and then in ascending order. However, Lucy fails to perform them in this sequence.

976

977

978

979

980

981

982

983

984

985

Question: For the client who applied the biggest loan, what was his/her first amount of transaction after opened the account.

Reason: Phase 3 issue. Two filtering conditions are required: first, in ascending order, and then in descending order. However, Lucy fails to perform them in this sequence.

The third group contains questions where the MatchTables phase either adds an extra table, or occasionally misses a table or attributes. Here is an example.

Question: How many accounts have an owner disposition and request for a statement to be generated upon a transaction?

Reason: Phase 1 issue. Lucy identifies "Tran" (transaction) as a relevant table, but it is not needed to answer the query.

E.4 Cloud resources

Note on the cost of running. One note here is that GPT and c2q models are costly to run. For example, in the Cloud Resources experiment, the costs are as follows: c2q costs $15, gpt4 $2, and gpt4ex $5, while Lucy costs $0.5.

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, we do.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, see Section 4

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We model a part of the problem as an optimization problem and provide formal encoding. See Section 3.3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we describe all algorithms and an optimization model.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the data in supplementary materials and describe prompts. We will make code publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips. cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specified parameters of prompts. We do not train new models.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We provide details for Lucy and gpt4 . Existing methods either provide their results as a single answer [Li et al., 2024b] or are too costly to run multiple times.

Guidelines: 1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

1194

1195

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we describe the experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

1214

1215

1216

1217

1218

1219

1220

1221

1222

1223

1224

1225

1226

1227

1228

1229

1230

1231

1232

1233

1234

1235

1236

1237

1238

1239

1240

1241

1242

1243

1244

1245

1246

1247

Justification: We believe it has a positive impact as we enhance users with new capabilities.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

1248

1249

1250

1251

1252

1253

1254

1255

1256

1257

1258

1259

1260

1261

1262

1263

1264

1265

1266

1267

1268

1269

1270

1271

1272

1273

1274

1275

1276

1277

1278

1279

1280

1281

1282

1283

1284

1285

1286

1287

1288

1289

1290

1291

1292

1293

1294

1295

1296

1297

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide all benchmarks in supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

1298

Justification:

1299

1300

1301

1302

1303

1304

1305

1306

1307

1308

1309

1310

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of developing effective LLM-based assistants for querying SQL databases. In this context, users pose questions to a relational database in natural language, and the goal is to generate a SQL query that correctly answers the user's question when executed. The authors focus on overcoming a limitation of current text-to-SQL approaches: the difficulty LLMs face in handling databases with numerous tables and complex relationships, making it hard to determine the necessary table joins for the query.\n\nTo tackle this issue, the authors propose a workflow that begins with using the LLM to identify relevant tables and their attributes. In the second step, a constraint satisfaction solver (CSP) is employed to determine the necessary joins while adhering to database constraints. In the third step, a materialized view is created by joining the relevant tables. Finally, this view, combined with the user's question, is used to prompt the LLM to generate the final SQL query.\n\nSTRENGTHS:\nThis paper tackles a very relevant practical problem, which is attracting significant attention in both academia and industry. The proposed workflow may add practical value.\n\nWEAKNESSES:\nThe paper lacks depth, and the writing does not, in my opinion, meet the quality standards required for a venue like NeurIPS. Additionally, several potential limitations of the proposed workflow are not discussed.\n\t•\tAccuracy of the answers is not the only important requirement in generating SQL from text. Database users also expect query generation to be time-efficient. The proposed workflow includes several computationally intensive steps: first, solving an NP-complete problem (CSP), and second, creating a potentially enormous materialized view by joining many tables. I would have expected a discussion on the computational limitations of this approach.\n\t•\tThe workflow lacks sufficient precision and clarity. For example, it is unclear whether the final query is expressed with respect to the materialized view as the only table or with respect to the original schema. Additionally, how lookup tables and various schema design patterns are identified in the input database is not well-explained. The authors claim that their approach guarantees the generated query respects database constraints, but this guarantee is not clearly defined. Algorithm 1 is underspecified; at this high level of detail, the algorithm seems redundant and could be subsumed by the text description. The exact SQL fragment covered by this approach is also unclear. While the limitations section mentions that queries requiring the union operator are not supported, it is unclear if other standard SQL constructs are also unsupported.\n\t•\tWhile relevant related work is cited, the main body of the paper lacks a detailed discussion on the contribution in relation to recent approaches.\n\t•\tThe evaluation section is somewhat lacking. The tables are confusing, and it is unclear what each of the rows actually represents.\n\nQUESTIONS:\nPlease comment on the limitations concerning unsupported SQL constructs as well as on the computational limitations of the proposed approach.\n\nLIMITATIONS:\nThe limitations section mentions some but not all of the relevant limitations of this approach.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors introduce LUCY, a new LLM based framework for converting text-to-SQL to query databases. Primarily this framework focusses on addressing user queries to databases that contain a large number of tables with complex relations between them.\nThe core idea of this approach is to decompose the query generation process into distinct stages. LLMs (GPT-4) are utilized for generative tasks such as identifying relevant tables and attributes and generating the SQL query. Meanwhile a deterministic constraint solver (OR-Tools) is employed to map relationships between these elements. In essence LUCY processes a user query through 3 phases namely MatchTables, GenerativeView and QueryView phases. In the MatchTables phase the goal is to identify the relevant tables and attributes. This is accomplished by iteratively prompting a Large Language Model (such as GPT-4) to identify relevant tables and attributes based on the user query and the database model, which includes the schema and an optional list of high-level design patterns. The database model is presented in a hierarchical manner and explored using a breadth-first search approach. Once the relevant relations and attributes are identified a schema graph is constructed and solved using a constrainst solver (i.e to identify the optimal path to join the tables) to build a view in the GenerativeView phase. A LLM is then prompted to generate a SQL query given the summary view and the user query in the QueryView phase. The authors further conduct experiments that demonstrate that the proposed technique achieves a better execution accuracy as compared to the existing state-of-the-art techniques on standard datasets(ACME, BIRD). Furthermore, they also introduce a show large improvements on a new benchmark dataset (Cloud Resources).\n\nSTRENGTHS:\n1. The literature review is comprehensive and the paper does a good job at clearly defining the problem to solve. \n2. The novelty of the proposed approach lies in the decomposition of tasks involved in generating SQL queries. By employing LLMs to handle specific subtasks, it effectively circumvents the need for LLMs to perform complex reasoning. A core distinguishing factor from prior research is the use of constraint solvers to identify the relevant paths for joining the identified tables. \n3. The authors also demonstrate that the proposed approach achieves a better execution accuracy than the existing SOTA on several benchmarks.\n\nWEAKNESSES:\n1. The paper ends abruptly without a clear and comprehensive conclusion. The paper presentation needs improvement in this regard.\n2. The authors introduce a new benchmark for evaluation but do not offer sufficient details regarding it. A detailed overview of the queries and an analysis of why the existing SOTA techniques do not perform well on the same could be provided which could greatly inform future work.\n3. The practical utility of the proposed technique seems to be limited as each user query requires multiple calls to be made to LLMs thereby entailing both increased latency and cost.\n4. The error analysis is not very comprehensive and could be improved. For instance how does this technique fare when the names of entities in database schemas are not semantically meaningful or if there are conflicts in descriptions etc (as is often the case in real-world industrial databases).\n\nQUESTIONS:\n1. For the new benchmark introduced based on the Cloud Resources dataset it is shown that LUCY significantly outperforms the state-of-the-art. What are the queries used in this benchmark? What sets them apart from the standard benchmarks where the execution accuracy of SOTA is comparable/close to LUCY?\n2. Has any analysis been done to measure the performance gain vs cost tradeoff when compared to SOTA.\n3. What happens when 2 entities in a database have similar descriptions? How sensitive is this technique to having semantically meaningful table names?\n4. Is there any mechanism used to handle hallucinations from the MatchTable phase in the GenerativeView Phase?\n\nLIMITATIONS:\n1. As the technique leverages LLMs it seems to be heavily reliant on having semantically meaningful entity names /descriptions \n2. The proposed technique seems sensitive to hallucinations as it involves processing a query through multiple LLM phases. The errors in any of the earlier phases would result in it propagating to the next stage. For instance as the authors pointed out if the MatchTables phase produces an extra table this could in turn effect the end output.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a new method, Lucy, designed to handle large databases with complex relationships between objects. Lucy operates through three steps: MatchTables, GenerateView, and QueryView. It first identifies relevant tables and attributes using LLMs, constructs a combined view with an automated reasoner, and generates the final SQL query. Lucy shifts complex reasoning from LLMs to a CSP solver, supporting various database design patterns. Experiments on ACME insurance, Cloud Resources, and the two BIRD databases show that Lucy outperforms other zero-shot text-to-SQL models.\n\nSTRENGTHS:\n- The proposed method offers a fresh perspective on tackling text-to-SQL research with a logical workflow.\n- The paper is well-written and easy to follow.\n\nWEAKNESSES:\n- I am not convinced by the motivation of zero-shot text-to-SQL with the example of industrial databases having complex relationships. Text-to-SQL systems deployment in real-industry requires high performance. I doubt that people won't be using zero-shot models for real use applications. In KaggleDBQA, it also states \"we believe the zero-shot setting is overly-restrictive compared to how text-to-SQL systems are likely to be actually used in practice.\" I would like to hear the authors' thoughts on this.\n- The paper does not appear to be well-grounded in text-to-SQL research. For example, one way to handle complex relationships in text-to-SQL using LLMs is through schema linking. However, the paper does not mention this area of research and instead proposes MatchTables, seemingly ignoring the rich literature of text-to-SQL works. Other approaches include least-to-most prompting attempts in text-to-SQL for task decomposition and Natural SQL for intermediate representation (although it does not handle query nesting). Properly discussing these relevant methods of the proposed method will better situate the work.\n\nQUESTIONS:\nPlease address my above concerns. I am willing to raise the score if convinced.\n\nLIMITATIONS:\nThe limitations of the work are well-stated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces Lucy, a framework for solving Text2SQL by LLMs, particularly for complex enterprise databases. Lucy leverages LLMs' understanding and reasoning capabilities to handle intricate database relationships and constraints. The framework operates in three phases: identifying relevant tables and attributes (MatchTables), constructing a view through constraint reasoning (GenerateView), and generating the final SQL query (QueryView). The empirical studies show Lucy achieves performance improvements on several zero-shot Text2SQL tasks.

STRENGTHS:
Text2SQL is an essential problem in commercial scenarios.

WEAKNESSES:
The draft seems far from complete, so leave some high-level suggestions.
1. Make the title, abstract, and introduction more concrete. It is hard to tell the contribution or uniqueness of this work among other papers about Text2SQL by LLMs.
2. Survey related works and clearly state the contribution/novelty of the proposed method against others.
3. Define the terminologies or abbreviations before their first appearance.
4. Make the draft concise by removing unnecessary content. For example, the first challenge introduced in Motivation section is not relevant to this work.
5. The empirical studies could be more convincing by following others' evaluation protocols, such as BIRD.
6. Lack of comparison to other competitors.
7. The figures, tables, and their captions should be self-explanatory.

QUESTIONS:
1. What is the key difference between the proposed framework and competitors (e.g., MCS-SQL, MAC-SQL, Chat2Query)?
2. How does the proposed method perform if evaluated by the same protocol as the BIRD-SQL leaderboard?

LIMITATIONS:
There is a discussion about the limitation, though the first limitation seems too broad and unnecessary.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-15