Venue: NAACL 2025

# FLEX: Expert-level False-Less EXecution Metric

for Reliable Text-to-SQL Benchmark

Heegyu Kim1, Taeyang Jeon1, Seunghwan Choi1, Seungtaek Choi, Hyunsouk Cho1,2

###### Abstract

Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces **FLEX (False-Less EXecution)**, a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen's kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models' performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.


overcomes existing limitations through comprehensive context analysis, considering the natural language question, database schema, and external knowledge. It also exploits elaborately devised evaluation criteria with detailed guidelines for assessing query correctness. Our approach is validated through strong consistency between **FLEX** judgments and human expert evaluations, demonstrating significantly higher agreement (Cohen's kappa of 87.04) than the existing EX metric (62.00) and outperforming previous LLM-based methods.

To demonstrate the practical impact of **FLEX**, we re-evaluated 50 publicly available text-to-SQL models on the Spider and BIRD1 benchmarks, revealing several key insights. As shown in Table 1, significant shifts in model rankings, particularly for BIRD, provide a more accurate assessment of model capabilities. Additionally, **FLEX** categorized the error cases and identified noisy annotations as a primary factor in underestimating model performance in current benchmarks. As illustrated in Fig. 1, **FLEX** scores are generally higher than EX scores, suggesting that **FLEX** captures previously underestimated aspects of model capability. Furthermore, **FLEX** detected models are overestimated in BIRD's challenging questions, highlighting areas for future research focus.

Footnote 1: Our study utilized the 20240627 version of the BIRD-dev set, yielding different EX scores from prior studies due to modified ground truth.

Our key contributions are as follows: 1) we identify key limitations in current text-to-SQL evaluation metrics, especially the Execution Accuracy (EX) metric, and 2) we introduce **FLEX**, a novel LLM-based evaluation method that aligns with human expert reasoning and shows better agreement with expert assessments. 3) By re-evaluating 50 text-to-SQL models using **FLEX**, we uncover significant ranking changes and provide a clearer understanding of benchmarks and state-of-the-art models, emphasizing the need for advanced evaluation methods. **FLEX** framework is publicly available2.

Footnote 2: [https://github.com/HeegyKuKim/FLEX](https://github.com/HeegyKuKim/FLEX)

## 2 Related Work

**Exact Matching (EM) and Execution Accuracy (EX)** were proposed to evaluate text-to-SQL systems in the Spider Yu et al. (2018). EM assesses the syntax-level equivalence of two queries; however, it is prone to high false negative rates because the same logical intent can be expressed in various query formulations. Therefore, EX has been utilized as a primary metric for query accuracy in BIRD Li et al. (2023c), focusing on the execution results rather than the syntactic form. Despite this improvement, EX still suffers from false negatives due to annotation quality issues and may produce false positives by rewarding incorrect queries that coincidentally yield the same results as the ground truth. These limitations underscore the necessity for more robust semantic evaluation approaches.

**LLM-based evaluation** has gained popularity as an alternative to human evaluators for assessing human preference in natural language generation models. Frameworks like MT-Bench Zheng et al. (2023) and AlpacaEval Li et al. (2023e) utilize LLMs as judges to provide faster and more cost-effective evaluations than human assessments. However, these frameworks primarily rely on the proprietary GPT-4 OpenAI et al. (2024), which poses reproducibility risks since providers may alter or discontinue models without notice. In response, Kim et al. introduced the open-source LLM judge _Prometheus_, offering a reproducible and cost-free alternative at GPT-4's performance level. Nonetheless, _Prometheus_ is mainly focused on general human preference evaluations rather than specialized tasks like text-to-SQL evaluation. To specifically leverage LLMs for text-to-SQL evaluation, Zhao et al. proposed _LLM-SQL-Solver_. Without distinguishing between the false positives and false negatives, this approach showcases the capability of LLMs to assess execution equivalence between two queries using two prompting strategies: _Minature-and-Mull_, evaluating seman

Figure 1: Performance comparison of EX vs. FLEX metrics on Spider benchmark. The red identity line shows an equivalent score.

 

tic equivalence between two queries, and _Explain-and-Compare_, asking LLMs to compare their significant logical differences. Despite its advancements, _LLM-SQL-Solver_ remains insufficient to fully replace EX as an evaluation metric. We will investigate these detailed limitations in section 4.3.

## 3 Preliminaries

This section introduces the notation and definitions used throughout the paper. We begin by establishing the following conventions: Let \(D\) denote a database, \(S\) its schema, and \(X\) a set of natural language questions. For each question \(x\in X\), we define contextual information \(\mathbb{C}\) to assist in the evaluation of metrics, including \(Q_{gt}(x)\) as the ground truth SQL query for question \(x\), and \(Q_{gen}(x)\) as the SQL query generated by a text-to-SQL model for \(x\). We then execute these queries on the database \(D\) to obtain their execution result sets. Specifically, we define \(R_{gt}(x)=\text{Execute}(Q_{gt}(x),D)\) as the result of the ground truth query and \(R_{gen}(x)=\text{Execute}(Q_{gen}(x),D)\) as the result of the generated query. We define evaluation metrics: EM, EX, and LLM-based evaluation (LX) as:

EM \[=\frac{1}{|X|}\sum_{x\in X}\mathbb{I}\left(Q_{gen}(x)=Q_{gt}(x)\right)\] (1) EX \[=\frac{1}{|X|}\sum_{x\in X}\mathbb{I}\left(R_{gen}(x)=R_{gt}(x)\right)\] (2) LX \[=\frac{1}{|X|}\sum_{x\in X}\mathbb{I}\left(x\underset{\mathbb{C} }{\leftrightarrows}Q_{gen}(x)\right)\] (3)

where \(\mathbb{I}(\cdot)\) is the indicator function that returns 1 if the condition inside the parentheses is true, and 0 otherwise. The notation \(x\underset{\mathbb{C}}{\leftrightarrows}Q_{gen}(x)\) denotes that the generated query \(Q_{gen}(x)\) semantically matches the question \(x\) under the contextual information \(\mathbb{C}\) based on an LLM.

For EM, the condition checks if \(Q_{gen}(x)\) exactly matches the \(Q_{gt}(x)\) syntactically. For EX, the condition checks if the \(R_{gen}(x)\) exactly matches the \(R_{gt}(x)\). However, LX handles semantic match between \(Q_{gen}(x)\) and question \(x\) under contextual information. \(\mathbb{C}\) may vary depending on the evaluation method. These metrics produce a score between 0 and 1, where 1 indicates perfect performance (100% accuracy) and 0 indicates complete failure (0% accuracy).

## 4 Analyzing the Limitations of Current Text-to-SQL Evaluation Methods

In this section, we address key research questions to investigate the shortcomings of existing text-to-SQL evaluation metrics. We aim to understand the types of errors these metrics produce and explore whether LLMs can offer a better alternative.

### What Types of Errors Occur in Text-to-SQL Evaluation?

To identify the limitations of current evaluation methods, we first examine the errors that arise when using the Execution Accuracy (EX) metric: false positives (FP) and false negatives (FN) using Table 2.

**False Positives** (\(x\underset{\mathbb{C}}{\leftrightarrows}Q_{gen}(x),\text{but }R_{gen}( x)=R_{gt}(x)\)): The generated query \(Q_{gen}(x)\) with semantically different structures or logic may produce the same execution results due to the current database states, leading to overestimating system performance. _Example 1_ includes an unnecessary condition age < 19 but coincidentally produces the same results as the ground truth because the database has no students older than 18. Unlike humans, EX does not assess whether the generated SQL accurately represents the intent of the natural language question, potentially rewarding queries that yield correct results through incorrect means.

**False Negatives** (\(x\underset{\mathbb{C}}{\leftrightarrows}Q_{gen}(x),\text{but }R_{gen}(x)\neq R_{ gt}(x)\)): Semantically correct queries that accurately translate the natural language questions may produce different results from ground truth queries,

\begin{table}
\begin{tabular}{l} \hline _Question: Who has the highest score?_ \\ \hline Ground Truth Query \\ \hline SELECT finance, name FROM student \\ ORDEX BY score DESC LIMIT 1 \\ ¿ Emily, Carter \\ \hline Example 1: False Positive \\ \hline SELECT finance, name FROM student \\ WHERE age ¡ 19 \# Unnecessary condition. \\ ORDES BY score DESC LIMIT 1 \\ ¿ Emily, Carter \\ \hline Example 2: False Negative \\ \hline SELECT hame, frame FROM student \\ ORDEX BY score DESC LIMIT 1 \\ ¿ Carter, Emily \# The column order is different \\ \hline Example 3: False Negative \\ \hline SELECT fname, name FROM student \\ WHERE score — (SELECT MAX (score) FROM student) \\ ¿ Emily, Carter I Liam, Thompson \\ \# Two students have the highest score. \\ \hline  & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Illustrated examples of false positive / negative.

 

underestimating system capabilities. Prior studies (Pourreza and Rafiei, 2023; Wretblad et al., 2024) have raised concerns about annotation quality, potentially leading to false negatives during evaluation. They reported that the Spider and BIRD benchmarks have many annotation issues--at most 49% at specific databases--including ambiguous questions and incorrectly annotated ground truths.

Ambiguous questions can be translated into multiple queries due to limited instruction about the output structure. For instance, _Example 2_ is marked incorrect simply because lname precedes fname in the column order. This structural rigidity in evaluation can lead to false negatives, potentially overlooking semantically correct queries. Additionally, _Example 3_ shows noisy ground truth, which can occur when multiple students have the same highest score. The query handles this issue correctly but is evaluated as incorrect because its execution result differs from the ground truth.

These limitations reduce reliability and hinder the development of more robust text-to-SQL systems, potentially leading to deployed systems that fail in real-world scenarios. To address these issues, we must explore evaluation methods that align more closely with human judgment.

### How Closely Does EX Align with Human Expert Evaluations?

Building on our analysis of identified errors, we conducted a human evaluation study to assess the alignment between EX and human experts in evaluating SQL query correctness. We randomly sampled 200 query pairs from the BIRD dataset, generated by TA-ACL Qu et al. (2024) and SuperSQL Li et al. (2024a). The samples were evenly divided between queries that produced the same result as the ground truth (_Equivalent set_) and those with different results (_Not equivalent set_). Three SQL experts with over three years of experience independently evaluated these queries for semantic correctness. Discrepancies were resolved through consensus. Before reaching a consensus, the inter-annotator agreement measured by Fleiss' kappa was 79.32, indicating strong agreement.

We employed Cohen's kappa and accuracy score to measure the agreement between human experts and EX. The agreement between human consensus and EX yielded a Cohen's kappa of 62.0, indicating substantial agreement but leaving much room for improvement, where human annotators identified 21 false positives in the _Not equivalent set_ and 17 false negatives in the _Equivalent set_.

These findings raise concerns about the reliability of the EX metric and underscore the need for evaluation methods that better reflect human judgment by considering semantic correctness and providing a more expert-level assessment of query accuracy. Appendix C provides real examples illustrating these discrepancies.

### Can LLMs Replace EX in Evaluating Text-to-SQL Systems?

Given EX's limitations and misalignment with human judgment, we explore LLMs as a potential alternative for evaluating text-to-SQL systems. We thus employed two LLM-based evaluation methods: 1) _Prometheus-2_Kim et al. (2024), an open LM judge, and 2) _LLM-SQL-Solver_Zhao et al. (2024) with GPT-4o OpenAI et al. (2024), a prompting method utilizing a proprietary LLM.

**Human Experts vs. LLMs:** Table 3 reveals that existing LLM-based evaluation methods did not surpass EX performance. _Prometheus-2_ showed a marginally lower agreement than EX, with a Cohen's kappa of 61.14, while _LLM-SQL-Solver_ performed considerably worse, even when using a flagship proprietary LLM. We propose three main factors contributing to the underperformance of current LLM-based evaluation methods:

* **Limited Context and Noisy Ground Truth:**_LLM-SQL-Solver_ neglects contextual information \(\mathbb{C}\), including the natural language question, external knowledge 3, and execution results. It solely compares queries \(Q_{gen}(x)\) to \(Q_{gt}(x)\)under schema \(S\), disregarding whether \(Q_{gen}(x)\) accurately represents question \(x\), thus limiting its ability to detect false positives and negatives. This issue is particularly critical in the Spider and BIRD benchmarks due to annotation quality

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline
**Model** & **Kappa** & **Acc** & **EQ** & **NEQ** \\ \hline EX & **62.00** & **81.0** & **79** & **83** \\ ‘LLM-SQL-Solver & 52.29 & 76.5 & 70 & **83** \\ Prometheus-2-7B & 61.14 & 80.5 & 78 & **83** \\ Prometheus-2-8x7B & 60.66 & 80.0 & 78 & 82 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Agreement between human consensus and other evaluation methods. **Acc** denotes overall accuracy score, **EQ** denotes _Equivalent set_ accuracy score, **NEQ** denotes _Not equivalent set_ accuracy score.

 

concerns discussed in section 4.1.
* **Ambiguous Criteria:** The _LLM-SQL-Solver_ may underperform even with advanced LLMs due to unspecified evaluation criteria. Its _Explain-and-Compare_ strategy uses vague prompts like "Are there significant differences?" Similarly, _Prometheus-2_ employs a 1-5 rubric scoring system, but designing optimal rubrics becomes heuristic and ambiguous, especially for scores between 2 and 4 in binary decisions (correct and incorrect)4 We claim that LLM should evaluate SQL queries using deeper reasoning steps and several binary criteria; otherwise, the LLM shows low evaluation performance from the ambiguous criteria. Footnote 4: Therefore, we empirically selected rubrics with a threshold (\(\geq 4\)) yielding the highest agreement scores.

Our study empirically demonstrates that prior LLM-based evaluation methods are insufficient compared to EX for evaluating text-to-SQL systems. This highlights the need for a more advanced evaluation paradigm that leverages LLMs effectively while addressing their current limitations.

## 5 Our Proposed Metric: FLEX

This section introduces our proposed evaluation metric **FLEX (False-Less EXecution)**. **FLEX** addresses the limitations of existing metrics by leveraging LLMs to provide a more accurate and humanlike assessment of generated SQL queries.

### Evaluation Process

**FLEX** instructs the LLM with the question, \(Q_{gen}(x)\) and optimal contextual information \(\mathbb{C}_{FLEX}\), including the question \(x\), generated query \(Q_{gen}(x)\), ground truth query \(Q_{gt}(x)\), execution results \(R_{gt}(x)\) and \(R_{gen}(x)\), schema \(S\), external knowledge \(K\) and two criteria \(T_{EQ}\) and \(T_{NEQ}\lx@note{footnote}{The detailed prompts are provided in Appendix F.1.}\). In other words, **FLEX** prompts LLM to focus on detecting false positives or false negatives after obtaining EX results whether \(R_{gen}(x)=R_{gt}(x)\).

\[\begin{split}&\mathbb{C}_{FLEX}=\begin{cases}\mathbb{C}_{EQ}& \text{if $R_{gen}(x)=R_{gt}(x)$}\\ \mathbb{C}_{NEQ}&\text{if $R_{gen}(x)\neq R_{gt}(x)$}\end{cases}\\ &\mathbb{C}_{base}=\{x,S,K,Q_{gt}(x),Q_{gen}(x)\}\\ &\mathbb{C}_{EQ}=\mathbb{C}_{base}\cup\{T_{EQ}\}\\ &\mathbb{C}_{NEQ}=\mathbb{C}_{base}\cup\{T_{NEQ},R_{gt}(x),R_{ gen}(x)\}\\ \end{split} \tag{4}\]

### Optimal Context (\(\mathbb{C}_{Flex}\))

**1) Execution Results Match (\(\mathbb{C}_{EQ}\))**

If \(R_{gen}(x)=R_{gt}(x)\), there is a possibility of false positives--queries that produce correct results by coincidence or through incorrect logic. To evaluate false positives, 1) \(\mathbb{C}_{EQ}\) does not contain \(R_{gt}(x)\) and \(R_{gen}(x)\). LLM occasionally judges false positives as correct due to the equivalent execution results. 2) \(\mathbb{C}_{EQ}\) utilized the following criteria (\(T_{EQ}\)) to evaluate semantic correctness between the questions and \(Q_{gen}(x)\):

* **Schema Alignment**: Whether the tables and columns used in \(Q_{gen}(x)\) align with the intent of question \(q\) and are consistent with the schema \(S\).
* **Correct Filtering Conditions**: Whether the WHERE clauses in \(Q_{gen}(x)\) accurately reflect the conditions specified in the question \(x\).
* **Handling of Nullable Columns**: Whether \(Q_{gen}(x)\) properly handles cases where the nullable columns in aggregation functions (SUM, COUNT, AVG) of WHERE clauses, as improper handling can lead to unexpected results.
* **Accounting for Multiple Rows**: Whether \(Q_{gen}(x)\) correctly handles cases where multiple rows satisfy the query conditions such as one-to-many relation and multiple min/max tuples.
* **Abusing of Clauses**: Whether clauses like GROUP BY, HAVING, ORDER BY, or DISTINCT are used appropriately, avoiding unnecessary complexity that could alter the intended results.

The LLM analyzes differences between \(Q_{gen}(x)\) and \(Q_{gt}(x)\) to detect logical discrepan 

cies that might not affect the current execution result but could lead to incorrect outcomes if the database state changes.

**2) Execution Results Differ (\(\mathbb{C}_{NEQ}\))**

Although \(R_{gen}(x)\neq R_{gt}(x)\), \(Q_{gen}(x)\) may still be correct due to ambiguities in the question or annotation issues with the ground truth query. In this scenario, \(\mathbb{C}_{NEQ}\) contains \(R_{gt}(x)\), \(R_{gen}(x)\), and \(T_{NEQ}\) which is designed to evaluate semantic correctness between the question and \(Q_{gen}(x)\) by considering:

* **Acceptable Output Structure Variations**: Whether differences in the output structure (e.g., column order, extra or missing columns) are acceptable given the way \(x\) is phrased.
* **Representation of Values**: Whether differences in value formats (e.g., numerical precision, percentage representation, boolean values as YES/NO) are acceptable for readability and do not alter the meaning.
* **Multiple Answers Available**: Whether \(x\) is interpretable in multiple valid ways, each leading to different correct queries.
* **Incorrect Ground Truth**: Whether the ground truth query \(Q_{gt}(x)\) is incorrect or suboptimal, while \(Q_{gen}(x)\) correctly answers the question.

The LLM thus considers the flexibility required due to ambiguous or underspecified questions and assesses whether \(Q_{gen}(x)\) is a valid translation of the question. By focusing on semantic correctness and closely aligning with human expert judgments, **FLEX** provides a more reliable evaluation of text-to-SQL models than traditional execution-based metrics. It effectively mitigates issues with false positives and false negatives, offering a more accurate assessment of model performance.

## 6 Experiments

To validate the effectiveness of our **FLEX** approach, we conducted a comprehensive study comparing the agreement between LLM judgments and human evaluations. We tested various state-of-the-art language models to determine which aligns most closely with human judgment.

### Does FLEX Outperform Other Metrics?

As shown in Fig. 3, **FLEX** generally outperforms the traditional EX metric across various LLMs, with GPT-4o achieving the highest agreement with human judgments. Open-source models like Mistral-small-Instruct-2409 Jiang et al. (2023) show substantial improvement over the EX metric, indicating that expert-level evaluations are achievable without reliance on proprietary solutions. LLM performance has improved significantly over time, with more recent models generally demonstrating higher Cohen's Kappa scores. Although open-source models have consistently underperformed proprietary models such as GPT-4o, they are steadily advancing and closing the gap. For instance, DeepSeek-V2-Chat DeepSeek-AI et al. (2024) shows close performance to Claude-3.5-sonnet Anthropic (2024), a proprietary LLM in its contemporary time.

**FLEX** also offers substantial efficiency gains in both time and cost. GPT-4o evaluated the BIRD dev set (1,534 instances) in under 20 minutes at approximately $6, whereas volunteers required an average of 2 hours for 200 instances. These results demonstrate **FLEX**'s viability as an efficient (42x faster) and cost-effective alternative to both EX and human evaluation for text-to-SQL tasks. Using Batch API 6 can reduce the input processing cost 75% and more number of threads can reduce the processing time of the **FLEX**.

Footnote 6: [https://platform.openai.com/docs/guides/batch](https://platform.openai.com/docs/guides/batch)

### What Factors are Beneficial?

We conducted an ablation study to validate the

Figure 3: Agreements between human evaluation and **FLEX** across LLM models over time. The red line shows EX metric agreement. Dots represent other LLMs, illustrating lower agreement than previous SOTA. Details are illustrated in Fig. 8.

 

efficacy of the optimal context \({\mathbb{C}}_{FLEX}\) and each contextual information. As shown in Table 4, including execution results only in \({\mathbb{C}}_{NEQ}\) yields the best performance across all metrics. When execution results are included in both \({\mathbb{C}}_{EQ}\) and \({\mathbb{C}}_{NEQ}\), performance is strong in the _Equivalent set_ (true positives + false positives) but declines in the _Not equivalent set_ (true positives + false negatives). Excluding execution results from both contexts shows the opposite trend. The inclusion of execution results impacts the model's ability to detect errors. Equivalent execution results may hinder false positive detection even if the generated query has incorrect logic. Conversely, for non-equivalent queries, including execution results helps reduce false negatives, as the model can directly compare minor differences in execution results. Therefore, the optimal context selection approach enhances evaluation for \(Q_{gen}(x)\), aligning it more closely with human expert assessments.

Table 5 shows the impact of removing different components from the evaluation context. Removing the ground truth query and results is the most significant factor, causing a substantial performance drop across all metrics. This underscores the importance of a reliable reference point in LLM-based text-to-SQL evaluations. Removing the natural language question or external knowledge also decreases notable performance, emphasizing their role in providing a comprehensive evaluation context. The removal of evaluation criteria also impacts performance, highlighting the importance of well-defined guidelines. This ablation study demonstrates that **FLEX**'s performance stems from the synergy of all its components, each playing a vital role in achieving high-quality text-to-SQL evaluation.

## 7 Leaderboard Re-evaluation

In this section, we evaluated the published result in the Spider and BIRD benchmark using robust capabilities of the **FLEX**. Three key findings were observed to demonstrate crucial limitations of current text-to-SQL evaluation. We first introduce experimental settings and demonstrate three crucial limitations.

### Experiment Setup

For sections 7.2 and 7.4, we conducted experiments using all of the publicly available 50 models on both the Spider and BIRD benchmarks. For section 7.3, we examine error categories on the top ten models in BIRD and Spider benchmarks. Our study utilizes the dev set of the BIRD benchmark because it has not disclosed their test set. For a fair comparison, we employed a dev set of the Spider benchmark; most studies published their prediction results of the dev set. GPT-4o was employed for the query evaluation as an LLM judge, which shows an outperforming agreement in our human evaluation.

### Findings 1: Leaderboard Shifts

The change in the performance of text-to-SQL models in our re-evaluation leads to a significant shift in the leaderboard. We found that the EX metric mostly underestimates the models. As illustrated in Fig. 1 and 4, the average increases from EX to **FLEX** are \(+2.63\) and \(+2.6\) in Spider and BIRD benchmarks, respectively. This result causes an average rank change of about 1.7 and 2.2 in both benchmarks. Specifically, the maximum rank changes are as large as five positions in BIRD and eight in Spider. These findings demonstrate that the EX metric obscures models' true performance, leading to inaccurate leaderboard rankings.

The extent of these shifts varies across different model types: Proprietary, open pre-trained language models (open PLMs), and open language models with supervised fine-tuning (open SFT). As described in Fig. 5, proprietary models and

Figure 4: Performance comparison of EX vs. FLEX metrics on BIRD benchmark. The red identity line shows an equivalent score.

\begin{table}
\begin{tabular}{l|c c c c} \hline
**Ablation Settings** & **Kappa** & **Acc** & **EQ** & **NEQ** \\ \hline _w/o Question_ & 80.10 & 90.0 & 84 & 96 \\ _w/o Knowledge_ & 79.09 & 89.5 & 82 & 97 \\ _w/o Criteria_ & 74.08 & 87.0 & 81 & 93 \\ _w/o Ground Truth_ & 29.36 & 64.0 & 72 & 56 \\ \hline \end{tabular}
\end{table}
Table 5: Comparison of human agreement results using different contextual information.

 

open PLMs show significant improvements when re-evaluated using the **FLEX**, whereas open SFT models display only marginal changes. A closer examination of the error types explains this difference. Open SFT models have a lower false negative ratio than proprietary and open PLM models. In contrast, all model types show relatively similar false positive ratios.

We hypothesize that this phenomenon can be attributed to differences in the training approaches. The supervised fine-tuning process enables SFT models to predict SQL queries structurally similar to those in the training dataset Li et al. (2024). This similarity results in execution results closely matching those of the ground-truth queries, reducing false negatives. In contrast, proprietary and open PLM models are more likely to generate more diverse SQL queries. This diversity increases the likelihood of structural differences between the execution results of the predicted and ground-truth queries. Consequently, these models are more susceptible to false negatives under the EX metric, which **FLEX** can correct.

### Findings 2: Models are Underestimated due to Annotation Quality

We analyze the causes of false negatives, which lead to underestimation of text-to-SQL models' performance, by leveraging a categorized error summary from our framework7. As illustrated in Fig. 6, the primary causes of false negatives are different output structures (Struct) and incorrect ground truth (GT). The inherent ambiguity in questions allows for several possible answers, leading to many false negatives due to different output structures (58.13% in BIRD). Incorrect ground truth SQL queries also account for a significant portion of false negatives (54.54% in BIRD). These results support the observation of previous works Wang et al. (2023); Wretblad et al. (2024) and show that this limitation remains unaddressed even in state-of-the-art models.

Footnote 7: Detailed process is provided in Appendix B.3.

### Findings 3: Models are Overestimated in Difficult Questions

Our study reveals that text-to-SQL models demonstrate lower **FLEX** score than EX in BIRD's challenging questions while higher in easier ones. This discrepancy can be elucidated through an analysis of false positive and false negative ratios illustrated in Fig. 7. In Spider, considered less complex than BIRD, the false positive ratio is consistently lower than the false negative ratio for text-to-SQL models, indicating that text-to-SQL models show higher **FLEX** than EX. This trend persists for simple and moderate questions in the BIRD. However, in challenging questions of BIRD, the false positive ratio exceeds the false negativ

Figure 5: Average model performances and error ratios across different model types.

Figure 6: Categorized result of FN ratios in top 10 models. Struct denotes an acceptable output structure variation, Value denotes a different representation of value, GT denotes incorrect ground truth, and Multiple Ans denotes multiple answers available.

Figure 7: The models’ average FP and FN ratios across difficulty levels. The y-axis denotes the percentage (%). In the Spider, text-to-SQL models tend to be underestimated as the questions become more difficult. Conversely, in the BIRD, models are overestimated with more challenging questions.

 

decrease in the model's performance. These results underscore the significant impact of false positives and false negatives in text-to-SQL evaluation.

## 8 Discussion

**FLEX** demonstrates promising results in LLM-based text-to-SQL evaluation, yet it faces limitations primarily due to insufficient contextual information. To enhance the accuracy of LLM-based evaluations, the future text-to-SQL benchmarks should incorporate multiple ground-truth queries per question, more comprehensive schema details, and additional context regarding database relationships and constraints. GPT-4o exhibits potential biases in the human-AI agreement. To address this issue in the future, we suggest implementing advanced prompting techniques such as employing an ensemble of LLM judges. These strategies aim to mitigate individual model biases and enhance LLM-based evaluations' overall reliability and fairness. Continuous fine-tuning of the evaluation process based on human expert feedback is crucial for maintaining and improving the system's performance over time.

**FLEX** uncovered a substantial performance gap between proprietary models like GPT-4 and open alternatives such as CodeS Li et al. (2024) and RESDSQL Li et al. (2023). While proprietary models exhibit superior performance, their use in real-world text-to-SQL applications raises concerns, particularly regarding data contamination Ranaldi et al. (2024). **FLEX** could be employed for diverse and high-quality data augmentation, potentially enhancing open SFT models. This will narrow the performance disparity between open-source and proprietary models and advance the field of text-to-SQL translation.

## 9 Conclusion

In this paper, we found the limitations in text-to-SQL evaluation, the vulnerability of the EX metric to false positives and negatives, which can result in inaccurate model assessments. Although LLM-based evaluations have emerged as alternatives to human evaluators, they still reveal several limitations. To overcome this, we introduce a novel metric, **FLEX (False-Less EXecution)**. It utilizes the comprehensive context and elaborate criteria based on the advanced language understanding of LLMs to achieve more expert-level evaluations. Our comprehensive experiments show the effectiveness of our proposed metric and provide several empirical findings to promote future studies on text-to-SQL evaluation.

## 10 Limitations

While our **FLEX** evaluation approach offers significant improvements over traditional metrics like Execution Accuracy (EX), there are some limitations to note: **FLEX** has a strong agreement; Cohen's kappa of 87.04 is imperfect and could be further improved through advanced reasoning Wang et al. (2022); Madaan et al. (2024) or fine-tuning Wang et al. (2023); Zhu et al. (2023) of the LLM evaluator. Our evaluation is currently limited to the Spider and BIRD benchmarks, necessitating further testing on other text-to-SQL datasets and real-world enterprise databases to validate **FLEX**'s effectiveness fully. **FLEX**'s reliance on LLMs makes it computationally intensive and more time-consuming than simple execution-based metrics, which may limit scalability for million-scale evaluations. Moreover, proprietary LLMs pose reproducibility risks, as providers can change or deprecate models without notice. While **FLEX** provides a more comprehensive assessment, it does not eliminate the need for human review, especially in critical applications. Addressing these limitations presents opportunities for future work to refine further and expand LLM-based evaluation approaches for text-to-SQL systems.

## Acknowledgements

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2022-0-00680, Abductive inference framework using omni-data for understanding complex causal relations), the National R&D Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (RS-2024-00407282 and RS-2024-00444182), and the Artificial Intelligence Convergence Innovation Human Resources Development Program (IITP-2025-RS-2023-00255968) funded by the Korean government (MSIT) 



## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Anthropic (2024) Anthropic. 2024. Introducing claude 3.5 sonnet.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, and et al. Luo Ji. 2023. Qwen technical report. _arXiv preprint arXiv:2309.16609_.
* Arl (2024) Cohere For AI. 2024. c4ai-command-r-plus-08-2024.
* DeepSeek-AI et al. (2024) DeepSeek-AI, Axin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenqqi Dengr, Chong Ruan, Damasi Dai, Daya Guo, Dejian Yang, et al. 2024. Deepsek-v2: A strong, economical, and efficient mixture-of-experts language model. _Preprint_, arXiv:2405.04434.
* Dong et al. (2023) Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Jinshu Lin, Dongfang Lou, et al. 2023. C3: Zero-shot text-to-sql with chatgpt. _arXiv preprint arXiv:2307.07306_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jaulri, Abhinav Pandey, Abhinav Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhi Matur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Ooyal, Anthony Hartshorn, and Aobo Yang et al. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Gao et al. (2023) Dawei Gao, Haibin Wang, Yaliang Li, Xiuyun Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. _arXiv preprint arXiv:2308.15363_.
* Team et al. (2024) Gemini Team, Ying Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcocer, and Roy Frostig et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _Preprint_, arXiv:2403.05530.
* Team et al. (2024) Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivierz, Mihir Sanay Kale, Juliette Love, Pouya Tafti, and Leonard Hussenot et al. 2024. German: Open models based on gemini research and technology. _Preprint_, arXiv:2403.08295.
* the rise of code intelligence. _Preprint_, arXiv:2401.14196.
* Hong et al. (2024) Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. 2024. Next-generation database interfaces: A survey of llvm-based text-to-sql. _Preprint_, arXiv:2406.08426.
* Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego De Las Casas, Emma Bou Hanna, Florian Bressand, and et al. Lengyel. 2024. Mixxtral of experts.
* Jiang et al. (2023) AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b (2023). _arXiv preprint arXiv:2310.06825_.
* Kim et al. (2024) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Huaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing fine-grained evaluation capability in language models. _Preprint_, arXiv:2310.08491.
* Kim et al. (2023) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Huaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In _The Twelfth International Conference on Learning Representations_.
* Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pageattention. In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
* Li et al. (2024a) Boyan Li, Yuyu Luo, Chengliang Chai, Gioliang Li, and Nan Tang. 2024a. The dawn of natural language to sql: Are we fully ready? _Preprint_, arXiv:2406.01265.
* Li et al. (2023a) Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. _Preprint_, arXiv:2302.05965.
* Li et al. (2024b) Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024b. Codes: Towards building open-source language models for text-to-sql. _Proceedings of the ACM on Management of Data_, 2(3):1-28.
* Li et al. (2023b) Jinyang Li, Bingyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. 2023b. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 37, pages 13076-13084.
* Li et al. (2023c) Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023c. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-ssls. _arXiv:2305.03111_.



Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Koerckov, Chenghao Mou, Marcarone, Christopher Akiki, Jia Li, Jenny Chim, and Qian Liu et al. 2023d. Starcoder: may the source be with you! _Preprint_, arXiv:2305.06161.
* Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023e. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval).
* Maadaan et al. (2024) Aman Maadaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. _Advances in Neural Information Processing Systems_, 36.
* OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, and Igor Babuschkin et al. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Pourreza and Rafiei (2023a) Mohammadraeza Pourreza and Davood Rafiei. 2023a. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Pourreza and Rafiei (2023b) Mohammadraeza Pourreza and Davood Rafiei. 2023b. Evaluating cross-domain text-to-sql models and benchmarks. _arXiv preprint arXiv:2310.18538_.
* Pourreza and Rafiei (2024) Mohammadraza Pourreza and Davood Rafiei. 2024. Dts-sql: Decomposed text-to-sql with small large language models. _Preprint_, arXiv:2402.01117.
* Qu et al. (2024) Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. 2024. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-sql generation. _Preprint_, arXiv:2405.15307.
* Team (2024) Qwen Team. 2024. Qwen2.5: A party of foundation models.
* Ranaldi et al. (2024) Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Rainiero Romagnoli, and Fabio Massimo Zanzotto. 2024. Investigating the impact of data contamination of large language models in text-to-sql translation. _arXiv preprint arXiv:2402.08100_.
* Roziere et al. (2024) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and et al. Jeremy Rapin. 2024. Code llama: Open foundation models for code. _Preprint_, arXiv:2308.12950.
* Shi et al. (2024) Liang Shi, Zhengju Tang, and Zhi Yang. 2024. A survey on employing large language models for text-to-sql tasks. _arXiv preprint arXiv:2407.15186_.
* Talaei et al. (2024) Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. Chess: Contextual harnesssing for efficient sql synthesis. _arXiv preprint_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, and Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _Preprint_, arXiv:2307.09288.
* Wang et al. (2023a) Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023a. Know what i don't know: Handling ambiguous and unknown questions for text-to-sql. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5701-5714.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Wang et al. (2023b) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023b. Pandalm: An automatic evaluation benchmark for llvm instruction tuning optimization. _arXiv preprint arXiv:2306.05087_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funotowicz, and et al. Joe Davison. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wretblad et al. (2024) Niklas Wretblad, Fredrik Godh Riseby, Rahul Biswas, Amin Ahmadi, and Oskar Holmstrom. 2024. Understanding the effects of noise in text-to-sql: An examination of the bird-bench benchmark. _arXiv preprint arXiv:2402.12243_.
* Yang et al. (2024) An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, and et al. Haoran Wei. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. _arXiv:1809.08887_ 

* Zhao et al. (2024) Fuheng Zhao, Lawrence Lim, Ishtiyaque Ahmad, Divyakant Agrawal, and Amr El Abbadi. 2024. Llim-sql-solver: Can llms determine sql equivalence? _Preprint_, arXiv:2312.10321.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging Illm-as-a-judge with mt-bench and chatbot arena. _Preprint_, arXiv:2306.05685.
* Zhu et al. (2023) Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. _arXiv preprint arXiv:2310.17631_.

 

Experiment Setting

### Frameworks and Hardware

We utilized NVIDIA A6000x4 GPUs with Huggingface transformers (Wolf et al., 2020) and VLLM (Kwon et al., 2023) for generating outputs from open LLMs. To evaluate open models larger than 30B, we utilize the Huggingface Inference API, TogetherAI Inference, and AI/ML API.

### Generation Hyperparameter

We set the temperature to 0 in every experiment to ensure consistent results for each attempt and the maximum number of tokens to 2,048.

### Execution Results in the LLM Prompt

The execution results in the LLM prompt were converted to a markdown table. However, the SQL query can produce extensive rows, increasing the prompt length and API costs. To reduce the overhead, we truncated the middle of the table with a mark as '...' and left 50 heads and tails for more than 100 rows. The shape of the table is appended to the markdown table to help LLM's judgment. Some TEXT-type columns can also be longer. After the 50 characters, we truncated them and appended '... _k chars_', where k indicates an original number of characters.

### Source of Prediction Results

The prediction results were obtained from various sources, including NL2SQL360 (Li et al., 2024a) frameworks, published studies, and reproduced predictions. Table 7 provides an overview of these sources. In DAIL-SQL (Gao et al., 2023), we simplified model results in their repository8 for readability, as shown in Table 6.

Footnote 8: [https://github.com/BeachWang/DAIL-SQL/tree/main/results](https://github.com/BeachWang/DAIL-SQL/tree/main/results)

### Human Annotators

Three of the authors are annotators. They have taken database classes at the university and have more than three years of experience operating the database system.

## Appendix B Additional Experiment Result

### FLEX Human Agreement across Various LLMs

Fig. 8 shows the human-LLM agreement results. We employed the following cutting-edge LLMs to evaluate human-LLM agreements. For proprietary models, we employed OpenAI's GPT models (Achiam et al., 2023; OpenAI et al., 2024), Anthropic's Claude models (Anthropic, 2024), and Google's Gemini models (Gemini Team et al., 2024).

For open LMs, we employed Meta's Llama-3 and Llama-3.1 models (Dubey et al., 2024), MistralAI's models (Jiang et al., 2023, 2024), Google's Gemma-2 models (Gemma Team et al., 2024), Qwen-1.5 (Bai et al., 2023), Qwen-2 (Yang et al., 2024) and Qwen-2.5 models (Qwen Team, 2024) DeepSeek AI's DeepSeek-V2 (DeepSeek-AI et al., 2024) and DeepSeek-Coder (Guo et al., 2024), Coherence's Command-R (Cohere For AI, 2024).

### Full FLEX results

The full leaderboard of BIRD-dev and Spider-dev is described in Table 8 and 9, respectively.

### FP and TN Error Analysis

We generate a comprehensive evaluation report after obtaining judgments for all generated queries in a dataset. This report includes an overall accuracy

Figure 8: Detailed results of Agreement between Human Evaluation and FLEX across LLM models. The red line indicates the agreement of EX metric: “-I” denotes an instruction-tuned model





\begin{table}
\begin{tabular}{l l c} \hline
**Simplified Name** & **Original File Name** \\ \hline DAIL_SQL,7-SHOT_TH-0.85\_MP & BIRD_WITH_EVIDENCE-TEST_SQL\_7-SHOT_EUCDISMASKPREESKLSINTRH_QA-EXAMPLE\_CTX-150\_ANS-4096\_TH-0.85\_txt \\ \hline DAIL_SQL,7-SHOT_TH-0.8\_MP & BIRD_WITH_EVIDENCE-TEST_SQT\_7-SHOT_EUCDISMASKPREESKLSINTRH\_QA-EXAMPLE\_CTX-150\_ANS-4096\_TH-0.8\_txt \\ \hline DAIL_SQL,9-SHOT_MP & BIRD_WITH_EVIDENCE-TEST_SQL\_9-SHOT_EUCDISMASKPREESKLSINTRH\_QA-EXAMPLE\_CTX-150\_ANS-4096\_txt \\ \hline DAIL_SQL,9-SHOT_QM & BIRD_WITH_EVIDENCE-TEST_SQL\_9-SHOT_EUCDISQUESTIONMASK\_QA-EXAMPLE\_CTX-150\_ANS-4096\_txt \\ \hline DAIL_SQL & QUESTIONMASK\_GPT-4\_for\_bird.txt \\ \hline \end{tabular}
\end{table}
Table 6: Simplified names for DAIL-SQL model results

\begin{table}
\begin{tabular}{c p{236.1pt} r r r} \hline Rank & Model & FLEX & EX & \(\Delta\) \\ \hline
1 (-) & Super SQL & 91.20 & 87.04 & +4.16 \\
2 (7) & DINSQL & 91.10 & 82.79 & +8.32 \\
3 (13) & DAIL_SQL,SC & 90.14 & 83.56 & +6.58 \\
4 (-) & DAILSQL & 88.88 & 83.08 & +5.80 \\
5 (12) & TA-ACL & 88.78 & 85.01 & +3.77 \\
6 (4) & SFI_Codes\_TB & 87.91 & 85.40 & +2.51 \\
7 (13) & SFT_Codes\_1SB & 87.33 & 84.91 & +2.42 \\
8 (12) & C3_SQL & 87.04 & 82.01 & +5.03 \\
9 (5) & SFI\_Deepsek\_Coor\_TB & 84.72 & 80.75 & +3.97 \\
10 (-) & STFIT\_Code\_3B & 84.72 & 83.27 & +1.45 \\
11 (-) & RESDSQL\_NatSQL\_3B & 83.66 & 84.14 & \(-\)0.48 \\
12 (-) & REDSQL\_3B & 82.01 & 81.82 & +0.19 \\
13 (-) & Graphi\_PICARD\_3B & 81.72 & 80.95 & +0.77 \\
14 (-) & ressd\_test\_lexmatsl\_large & 81.43 & 82.01 & \(-\)0.58 \\
15 (+1) & ressd\_text\_lexmatsl\_large & 79.11 & 80.08 & \(-\)0.97 \\
16 (-) & ressd\_text\_lexmatsl\_base & 78.82 & 80.17 & \(-\)1.35 \\
17 (-) & SFT\_Code\_1B & 78.63 & 77.95 & +0.68 \\
18 (-) & pretrained\_deepesek\_coor\_Tb & 78.34 & 64.22 & +14.12 \\
19 (-) & SFT\_lama\_3B & 78.05 & 76.11 & +1.93 \\
20 (-) & USSR\_text\_lexmatsl\_base & 76.98 & 77.95 & \(-\)0.97 \\
21 (-) & SET\_codeLand\_TAB & 76.21 & 74.08 & +2.13 \\
22 (-) & Depeek-Code\^{*}/ TB & 75.15 & 73.50 & +1.64 \\
23 (-) & UFT\_Star\_Conder\_T\_B & 72.44 & 72.05 & +0.39 \\
24 (-) & preatternal\_lamas\_3B & 68.86 & 68.44 & +8.41 \\
25 (-) & SET\_I\_Lama\_2\_7B & 64.41 & 65.28 & \(-\)0.87 \\
26 (-) & CodeLang=7B & 60.35 & 60.93 & \(-\)0.58 \\
27 (-) & Preattenzated\_strocoder\_Tb & 59.96 & 55.51 & +4.45 \\
28 (-) & Llama=7B & 59.67 & 60.83 & \(-\)1.16 \\
29 (-) & pretrained\_codellama\_7b & 55.71 & 51.64 & +4.06 \\
30 (-) & pretrained\_tlama\_2\_7b & 24.08 & 20.99 & \(+\)3.09 \\ \hline \end{tabular}
\end{table}
Table 7: Overview of prediction sources

\begin{table}
\begin{tabular}{p{142.3pt} p{236.1pt} r r r} \hline Rank & Model & FLEX & EX & \(\Delta\) \\ \hline
1 (-) & SuperSQL & 64.08 & 57.37 & +6.71 \\
2 (-) & CHESS-OPT-4o-mini & 62.71 & 59.13 & +3.59 \\
3 (-) & TA-ACL & 59.97 & 55.67 & +4.30 \\
4 (-) & DAIL\_SQL\_9-SHOT\_MP & 59.26 & 53.52 & +5.74 \\
5 (-) & DAIL\_SQL\_9-SHOT\_QM & 58.47 & 53.06 & +5.41 \\
5 (-) & DTTS-SOLIBRID-OPT-06x-082 & 58.47 & 58.08 & +0.39 \\
7 (-) & SFF\_Codes\_1SB\_EK & 56.98 & 56.52 & +0.46 \\
8 (-) & SEFT\_Codes\_TB\_EK & 53.59 & 54.89 & \(-\)1.30 \\
9 (-) & SFT\_Codes\_3B\_EK & 53.26 & 53.46 & \(-\)0.20 \\
10 (-) & DAIL\_SQL\_ SQL & 51.83 & 45.89 & +5.93 \\
11 (-) & DAIL\_SQL\_7-SHOT\_QM & 51.50 & 45.89 & +5.61 \\
12 (-) & CS\_SQL & 51.30 & 48.44 & +2.87 \\
13 (-) & DAIL\_SQL\_7-SHOT\_TH-0.8\_MP & 49.54 & 44.52 & +5.02 \\
14 (-) & DAIL\_SQL\_7-SHOT\_TH-0.85\_MP & 48.89 & 44.39 & +4.50 \\
15 (-) & SFT\_Code\_1B\_EK & 47.59 & 48.70 & -1.11 \\
16 (-) & RESDQ\_3B\_EK & 41.98 & 42.37 & +0.39 \\
17 (-) & GPT+4 turbo\_kg\_predict\_dev & 40.87 & 35.92 & +4.55 \\
18 (-) & RESDQ\_Large\_EK & 35.53 & 36.90 & \(-\)1.37 \\
19 (-) & RESDQ\_Base\_EK & 29.14 & 31.16 & \(-\)2.02 \\
20 (-) & GPT+4 turbo\_pedict\_dev & 25.68 & 22.75 & +2.93 \\ \hline \end{tabular}
\end{table}
Table 8: Spider full result

score, a breakdown of error types, and detailed explanations for each incorrect query. The LLM is prompted to categorize the error case based on its judgment and other contextual information. Judgments are categorized according to the criteria described in the previous subsection, and complex errors can belong to more than one category. Researchers can utilize this evaluation report to identify systematic errors in their text-to-SQL models and guide future improvements. Fig. 12 describes the system prompt to categorize false positives, and Fig. 13 describes the system prompt to categorize false negatives. Fig. 9 demonstrates the error categorization results of the top 10 BIRD-dev and Spider-dev models.

## Appendix C EX Error Case Study

We sampled the representative examples from our human study. Table 10, 11, 12 and 13 demonstrate the false positive examples. Table 14, 15, 16, 17 demonstrate the false negative examples. Table 18 demonstrates the example, including different column ordering and incorrect ground truth.

## Appendix D Artifacts

We confirmed that all models, datasets, and frameworks are allowed for research use.

## Appendix E AI Assitant

We use Microsoft Copilot as a coding assistant and Grammarly and Writefull as a writing assistant.

## Appendix F LLM Prompt

### Flex

We distinguished the prompt into system prompt and user prompt. As described in Fig. 14, the user prompt includes natural language questions, generated queries, ground truth queries, and execution results. The system prompt includes task instruction and criteria as described in Fig. 10 and 11. In the categorization, the user prompt includes natural language questions, generated queries, and ground truth queries. All system prompts instruct the LLM to output using Chain-of-Thought Wei et al. (2022) prompting followed by JSON formatted result to extract judgment from LLM output.

### Prometheus-2

_Prometheus-2_Kim et al. (2024) requires instruction, response, reference answer, criteria, and score rubrics for evaluation. Fig. 15 and 16 show optimal criteria for equivalent and not equivalent sets, respectively. We deemed scores of four and five as correct to achieve the highest level of agreement.

 



\begin{table}
\begin{tabular}{l|p{142.3pt}} Question & Among the schools with the average score in Math over 560 in the SAT test, how many schools are directly charter-funded? \\ Ground Truth Query & SELECT COUNT(T2.School Code) FROM assitccos AS T1 INNER JOIN frym AS T2 ON T1.cds = T2.CDSCode \\  & WHERE T1.AvgSerMath 560 AND T2.Charter Funding Type = ”Directly funded’ \\ Generated Query & SELECT COUNT(*) FROM satscores INNER JOIN schools ON sstacores.cds = schools.CDSCode \\  & WHERE astecors.AvgSerMath 560 AND schools.FundingType = “Directly funded’; \\ Human Judgment & Incorrect Scheme Linking: The generated query uses incorrect columns schools.funding type \\  & The generated query is incorrect \\
* The Prediction Query uses the satscores and schools tables, joining them on statscores.cds = schools.CDSCode. \\  &
- The Ground Truth Query uses the saxcess and frpm tables, joining them on T1.cds = T2.CDSCode. \\  & 2 Conditions: \\  &
- Both queries filter for schools with an average Math score greater than 560. \\  &
- The Prediction Query checks if schools.FundingType is ’Directly funded’. \\  &
- The Ground Truth Query checks if rfpn.Charter Funding Type is ‘Directly funded’. \\  & 3. Scheman Considerations: \\  &
- The hint indicates that schools.FundingType and frpm.Charter Funding Type are not equivalent, which means they may not represent the same data or criteria. \\  &
- The Ground Truth Query uses frpm.Charter Funding Type, which is the correct column to check for charter funding type according to the question. \\  & 4. Logical Correctness: \\  &
- The Prediction Query uses a different column (schools.FundingType) to determine if a school is directly charter-funded, which is not equivalent to fpm.Charter Funding Type. \\ This could lead to incorrect results if the data in these columns differ. \\ Given these points, the Prediction Query does not logically answer the question as it uses an incorrect column to determine the charter funding type. Therefore, the Prediction Query is incorrect. \\  & [”json] \\  & ["corrct”: false] \\  & \\ \end{tabular}
\end{table}
Table 11: Equivalent set #28 (BIRD #68): An incorrect condition filtering case demonstrating a generated query that erroneously adds ’StatusType = ’Closed’’ and uses ’SOC = ’11’’ as a string instead of an integer. The human judgment correctly identifies the missing ’StatusType = ’Closed’’ condition, while FLEX w/ GPT-4o provides a more comprehensive analysis, highlighting both the SOC value type discrepancy and the missing StatusType condition. This example emphasizes the critical nature of precise condition filtering in SQL queries.

\begin{table}
\begin{tabular}{l|p{197.1pt}} Question & Which county reported the most number of school closure in the 1980s with school ownership code \\  & belonging to Youth Authority Facilities (CEA)? \\ Ground Truth Query & SELECT schools.County FROM schooling WHERE schools.SOC = ’11’ \\  & AND struffme(‘$%Y’, schools.ClosedDate) BETWEEN ’1980’ AND ’1989’ \\  & GROUP BY schools.County ORDER BYCOUNT(schools.County) DESC LIMIT 1; \\  & ¿S Sacramento \\ Generated Query & SELECT County FROM schools WHERE struffme(‘$%Y’, ClosedDate) BETWEEN ’1980’ AND ’1989’ \\  & AND StatusType = ’Closed’ AND SOC = 11 \\  & GROUP BY County ORDER BYCOUNT(School) DESC LIMIT 1 \\  & ¿S Sacramento \\ Human Judgment & Incorrect Condition Filtering: The generated query misses a filtering condition StatusType = ’Closed’ \\  & The generated query is incorrect \\ \hline
**FLEX w/ GPT-to** & ”jon \\ (**Correct**) & [“correct”: false] \\  & ... \\  & The Prediction Query is incorrect for the following reasons: \\  & 1. SOC Value Type. The Prediction Query uses SOC = ’11’ with ‘11’ as a string, while the Ground Truth Query uses SOC = ’11 as an integer. This could lead to different results if the SOC column is of integer type. \\  & 2. StatusTypeCondition: The Prediction Query does not include the condition StatusType = ’Closed’, which is present in the Ground Truth Query. This condition is crucial to ensure that only closed schools are considered, which directly affects the result. \\  & These differences mean that the Prediction Query could potentially include incorrect data, leading to different results if the data set includes schools that are not closed or if the SOC column is of integer type. \\ \end{tabular}
\end{table}
Table 10: Equivalent set #3 (BIRD #9): An incorrect schema linking case demonstrating a generated query that mistakenly uses ’schools.FundingType’ instead of ‘T2.Charter Funding Type’. The human judgment accurately identifies this schema linking error, further corroborated by the detailed FLEX w/ GPT-4o analysis.





\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline \hline Question & What is the average writing score of each of the schools managed by Ricci Ulrich? \\  & List the schools and the corresponding average writing scores. \\ \hline Ground Truth Query & SELECT T2\_School, T1.AvgScrWrite FROM satscores AS T1 \\  & INNER JOIN schools AS T2 ON T1.cds = T2.CDSCode \\  & WHERE T2\_AdmFNamen l = ‘Ricci’ AND T2\_AdmLName1 = ’Ulrich’ \\  & ¿! Buchanan High ! 507.0.1 \\ Generated Query & SELECT T1\_name, AVG(T1.avgscrwrite) FROM satscores AS T1 \\  & INNER JOIN schools AS T2 ON T1.cds = T2.cdscode \\  & WHERE T2\_admfname1 = ’Ricci’ AND T2\_admlname1 = ’Ulrich’ GROUP BY T1\_name \\  & ¿! Buchanan High ! 507.01 \\ \hline Human Judgment & The generated query misses a GROUP BY clause. It can yield different results \\  & If there are several schools with the same name. \\  & The generated query is incorrect. \\ \hline
* astscores table contains cdse and avgscrwrite. \\  &
- schools table contains dsccode, sname, admfname1, and admlname1. \\  & Ground Truth Query Analysis: \\  &
- It selects School and AvgScrWrite from the satscores and schools tables. \\  &
- It joins these tables on T1.cds = T2.CDSCode. \\  &
- It filters for schools managed by Ricci Ulrich using T2.AdmFNamel l = ’Ricci’ AND T2\_AdmLName1 = ’Ulrich’. \\  & Prediction Query Analysis: \\  &
- It selects T1\_name and the average of T1.avgscrwrite. \\  &
- It joins the satscores and schools tables on T1.cds = T2.cdscode. \\  &
- It filters for schools managed by Ricci Ulrich using T2.admfname1 = ’Ricci’ and T2.admlname1 = ’Ulrich’. \\  &
- It groups the results by T1\_name to calculate the average writing score for each school. \\  & Comparison: \\  &
- The Prediction Query correctly joins the satscores and schools tables and filters for the correct administrator. \\  &
- The use of AVG(T1.avgscrwrite) and GROUP BY T1\_name in the Prediction Query is appropriate \\  & to calculate the average writing score for each school. \\  &
- The Ground Truth Query does not perform an aggregation, which suggests it assumes precomputed average scores are available in T1\_AvgScrWrite. \\  & Result Consistency: \\  &
- Both queries produce the same results, indicating that the Prediction Query correctly calculates the average writing scores. Given the analysis, the Prediction Query logically answers the question and produces the correct results. \\  & Therefore, the Prediction Query is correct. \\  & ”json \\  & [”correct": true] \\  & ... \\ \hline \hline \end{tabular}
\end{table}
Table 12: Equivalent set #66 (BIRD #45): This case showcases a generated query that unnecessarily includes a GROUP BY clause. Human judgment correctly identifies this issue, noting that it can yield incorrect results if there are multiple schools with the same name. FLEX w/ GPT-4o incorrectly assesses both queries as equivalent, overlooking the potential problems caused by the superfluous GROUP BY clause. This example highlights the importance of careful query structure analysis and the challenges in detecting subtle logical errors in SQL query generation and evaluation.




\begin{table}
\begin{tabular}{l|p{142.3pt}} Question & In chartered schools with charter number 00D2, what are the names of all the administrators? \\  & include the name of the school and the city to which it belongs \\ Ground Truth Query & SELECT AInfName1, AdmlName1, School, City FROM skools \\  & WHERE Crater = 1 AND CharteNum = ’00D2’ \\ Generated Query & SELECT school, city, admfname1, admname2, admfname2, admname2, admfname3, admname3 FROM schools \\  & WHERE charter = 1 AND charteum = ’00D2’ \\ Human Judgment & Extra Column: admname2, admname2, admfname3, admname3, but the other logic is correct. The generated query is correct. \\
**FLEX w/ GPT-do (Correct)** & The task is to find the names of all administrators in charted schools with charter number ’00D2’, including the name of the school and the city to which it belongs. Let’s compare the Prediction Query and the Ground Truth Query: Filtering Condition: Both queries correctly filter the schools where Charter = 1 and CharteNum = ’00D2’. This part of the query is consistent with the requirement to focus on chartered schools with the specified charter number. Selected Columns: – The Ground Truth Query selects AdmlName1, AdmLName1, School, and City. ‘This directly answers the question by providing the first administrator x’s first last name, along with the school and city. The Prediction Query selects school, city, admfname1, admname2, admname2, admname3, admname3. This includes additional columns for the second and third administrators’ names, which are not required by the question. However, this does not affect the logical correctness of the query as it still provides the necessary information. Output Structure: – The Ground Truth Result has four columns, while the Prediction Result has eight columns due to the inclusion of additional administrator name fields. – The extra columns in the prediction Result do not impact the ability to answer the question, as the required information is still present. Result Comparison: – The results from both queries contain the same schools and cities, and the first administrator’s names match. – The additional columns in the Prediction Result are filled with None for the second and third administrators, which does not affect the correctness of the answer to the question. – In conclusion, the Prediction Query logically answers the question by providing the necessary information about the first administrator’s name, school, and city, even though it includes additional columns. “The extra columns do not introduce any errors or inconsistencies in the context of the question. ”jon [”correct”; true] \\ \end{tabular}
\end{table}
Table 14: Not equivalent set #90 (BIRD #63): A case showcases FLEX’s ability to evaluate SQL queries with different column selections correctly. The Generated Query includes extra columns for additional administrator names (admname2, admname3) that are not present in the Ground Truth Query. Despite this difference, FLEX accurately judges that both queries are logically equivalent in answering the question about administrators in chartered schools with charter number ’0002’. This demonstrates FLEX’s sophisticated understanding that the presence of additional, non-conflicting information does not alter the fundamental correctness of a query in meeting the specified requirements.

\begin{table}
\begin{tabular}{l|p{142.3pt}} Question & What is the total number of non-chartered schools in the county of Los Angeles \\  & with a percent (\%) of eligible free meals for grades 1 through 12 that is less than 0.18\%? \\ \hline Ground Truth Query & SELECT COUNT(chatternum) FROM schools WHHERE malicity = ’Hickman’ AND chartermum = ’00D4’ \\ Generated Query & SELECT COUNT(*) FROM schools WHHERE CharterNum = ’00D4’ AND MailCity = ’Hickman’ \\  & \textgreater{}3 \\ \hline Human Judgment & Missing handling of the nullable column: chartermum is the nullable column in the schema \\  & CONTUT(*) counts the null values while COUNT(‘characterrum) does not. The generated query is incorrect. \\ \hline
* The difference in column case sensitivity (charttermum vs CharterNum and malicity vs MailCity) is not an issue in SQLite, as it is case-insensitive with respect to column names. \\  &
- The use of COUNT(“dartemurum) in the Prediction Query versus COUNT(*) in the Ground Truth Query does not affect the result \\  & as long as chartermum is not nullable. If chartermum were nullable, it could potentially affect the count, but the question implies that it is not an issue here since both queries produce the same result. \\  & Given these considerations, the Prediction Query correctly answers the question. \\  & ’json \\  & ’correct’; true \\ \end{tabular}
\end{table}
Table 13: Equivalent set #73 (BIRD #64): A case highlighting the importance of handling nullable columns in SQL queries. The human correctly identifies that COUNT(*) and COUNT(charternum) treat null values differently, potentially affecting the result if chartermum is nullable. FLEX w/ GPT-do incorrectly deems the queries equivalent, overlooking this crucial distinction. This example reveals a limitation in the AI system’s query analysis, emphasizing the need for careful consideration of null value handling in SQL evaluation.




\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline Question & What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. \\ \hline Ground Truth Query & SELECT schools.Street, schools.City, schools.Zip, schools.State FROM schools \\  & INNER JOIN assucces ON schools.CDSCode = satscores.cols \\  & WHERE stancscrews.NumTfsfsTark I = 0 ORDRBY (stacscores.NumGE1500 + 1.0 / stancscrews.NumTsftTak) ASC LIMIT 1; \\ Generated Query & SELECT T2.Street, T2.City, T2.State, T2.Zip FROM sattscscors AST T1 \\  & INNER JOIN schools AS T2 ON T1.cls = T2.CDSCode \\  & ORDER BY CAST(T1.NumGE1500 AS REAL) / T1.NumTsftAK ASC LIMIT 1 \\  & ¿ 1111 Van Ness Avenue ! Fresno 93721-20021 CA1 \\ \hline Human Judgment & Only the column order is different, so the generated query is correct. \\ \hline
* \\  & Both methods are valid for ensuring floating-point division in SQLite. \\ Filtering: \\  &
- The Prediction Query includes a condition sastccres.NumTxtsTark I= 0 to avoid division by zero, which is a good practice. \\  &
- The Ground Truth Query does not explicitly include this condition, but it is implied that NumTr@flar should not be zero for a valid excellence rate calculation. \\ Ordering and Limiting: \\  &
- Both queries order the results by the calculated excellence rate in ascending order and limit the result to the first entry, which corresponds to the school with the lowest excellence rate. \\ Selected Columns: \\  &
- The Prediction Query selects Street, City, Zip, State, while the Ground Truth Query selects Street, City, State, Zip. \\  & The order of columns in the SELECT clause does not affect the correctness of the query in terms of answering the question. \\ Result Discrepancy: \\  &
- The Prediction Result and Ground Truth Result differ in the order of the State and Zip columns. \\ However, this does not affect the logical correctness of the query in terms of providing the complete address. \\  & In conclusion, the Prediction Query logically answers the question by providing the complete address of the school with the lowest excellence rate. \\  & – The difference in column order in the result does not impact the correctness of the query. \\  & ¿json \\  & [“correct”: true] \\ \hline \end{tabular}
\end{table}
Table 15: Not equivalent set #17 (BIRD #37): A case where FLEX correctly judges two SQL queries as logically equivalent despite a difference in column order. The Generated Query selects columns as T2.Street, T2.City, T2.State, T2.Zip, while the Ground Truth Query selects them as Street, City, State, Zip. Despite this variation in the SELECT clause, both queries aim to find the complete address of the school with the lowest excellence rate. FLEX accurately determines that the column order difference does not affect the logical correctness of the query in answering the question, demonstrating its ability to evaluate query semantics beyond superficial syntax.




\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline Question & What is the eligible free or reduced price meal rate for the top 5 schools in grades 1-12 with the highest free or reduced price meal count of the schools with the ownership code 66? \\ \hline Ground Truth Query & SELECT CAST(T1.FRPM Count (K-12) AS REAL) / T1.Enrollment (K-12) FROM fpm AS T1 INNER JOIN schools AS T2 ON T1.CDSCode = T2.CDSCode WHHERE T2.SOC = 66 ORDER BY T1.FRPM Count (K-12) DESC LIMIT 5 > 0.9993 1.9719 0.9142 0.8965 0.8958! \\ Generated Query & SELECT ROUND(if(fpnn.FPM Count (K-12) / fppn.Enrollment (K-12)) * 100, 2) AS eligible_rate FROM fppm INNER JOIN schools ON fpnp.CDSCode = schools.CDSCQUE WHEEK schools.SOE = ‘66’ ORDER BY frpmCount (K-12) DESC LIMIT 5 > 199.931 91.791 19.421 89.65 189.58! \\ \hline Human Judgment & Different output value representation: The generated query multiplies 100 to the result for readability. Other logic is correct; the generated query is correct \\ \hline
* The Ground Truth Query calculates the eligible rate as a floating-point number by dividing FRPM Count (K-12) by Enrollment (K-12). \\  &
- The Prediction Query also performs this division but multiplies the result by 100 and rounds it to two decimal places, effectively converting the rate into a percentage. \\ Result Representation: \\  & The Ground Truth Result presents the eligible rates as decimal values (e.g., 0.8964987714987716). \\  &
- The Prediction Result presents the eligible rates as percentages (e.g., 89.65). \\ Logical Correctness: \\  & Both queries logically answer the question by selecting the top 5 schools based on the FRPM Count (K-12) and calculating the eligible rate for these schools. \\  & The difference in representation (percentage vs. decimal) does not affect the logical correctness of the Prediction Query. \\  &
- The conversion to percentage is a valid transformation of the rate. \\ SQLite3 Compatibility: \\  & Both queries are compatible with SQLite3, and the operations performed are valid within this context. \\  &
- In conclusion, the Prediction Query is correct in terms of logic and functionality. \\  &
- The difference in result representation (percentage vs. decimal) is acceptable and does not impact the query’s ability to answer the question. \\ \hline \end{tabular}
\end{table}
Table 16: Not equivalent set #16 (BIRD #32): A different output value case demonstrates the accuracy of FLEX judgment in SQL query evaluation. While the Generated Query used the ROUND function to present results as percentages, the Ground Truth Query expressed them as decimal values. Despite this difference in output format, FLEX correctly determined that both queries produce logically equivalent results. This showcases FLEX’s ability to understand and evaluate query semantics and functionality beyond simple text comparison.




\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline Question & How many test takers are there at the school/s whose mailing city address is in Fresno? \\ \hline Ground Truth Query & SELECT T1.NumTsTark FROM asscores AS T1 \\  & INNER JOIN schools AS T2 ON T1.cls = T2.CDSCode WHERE T2.MailCity = ’Fresno’ \\  & > ! !



\begin{table}
\begin{tabular}{l|p{284.5pt}} \hline Question & What is the school type of the school with the highest latitude? \\  & Indicate the name of the school as well as the latitude coordinates. \\ \hline Ground Truth Query & SELECT T1.School Type, T1.School Name, T2.Latitude FROM frmn AS T1 INNER JOIN schools AT S2 ON T1.CDSCe = T2.CDSCode ORDER BY TzL.latitude DESIC LIMIT ! \\  & ¿I High Schools (Public) | Leggett Valley High 144.219305! \\ Generated Query & SELECT schools.school, schools.lattitude, frpm.school type FROM schools JOHN frmn ON schools schwocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@rns.cdsocabe r@ns.



* [1] The PredictionResultmatchestheReferenceResult. However, thisdoes not guaranteethatthePredictionQueryiscorrect. Carefullyanalyze thePredictionQueryandevaluateitscorrectnessconsideringthefollowingcriteria:
* [2] CorrectPredictionQuery

* [6] IncorrectPredictionQuery
* DoesnotlogicallyanswertheQuestionord containssignificanterrors.
* Producesdifferentresultsduetocincorrectfilteringormissingconditions,JOIN redundancy,orotherfatalissues.
* [7] Failstohandlenullvalues,multiplerowsoftorecriticalaspectsoftheQuestion.
* Doesnotconsidernumlablecolumnsinaggregationfunctions(SUM,COUNT,AVG) andNULLvaluecanleadunexpectedresults.
* [8] Forexample,COUNT(x)inthedrictionbutCOUNT(school)inthegroundtruthwillproducedifferentresultsiftthecolumnschooldosenthaveaNOTNULLconstraintintheschema.

* [9] SQLite3Compatibility
* BothqueriesareSQLite3compatible,meaningintegerdivisionistnot automaticallyconvertedtotfloat division,andlogicaloperators,column/tablenamesarecase-insensitive.

* [10] AnalysisGuidelines
* [11] ComparethePredictionQuerywiththeGroundTruthQuerywithinthecontextofttheprovidedschemaandquestion.
* [12] Predictthequery'slogicalcorrectnessbasedonthecriterimentionedabove.
* [13] Finally,scoretherPredictionQuerysfollows:
* [14]json
* [15]("correct":trueorfalse)
* [16]...

Figure 10: **FLEX system prompt to judge the generated query producing the same results as ground truth.**

Figure 9: Average error categorization result of top 10 Spider-dev and BIRD-dev models using **FLEX**. The gray line denotes the standard deviation. Schema denotes incorrect schema alignment, Filtering denotes incorrect filtering conditions, Null denotes incorrect handling of nullable columns, Multi Rows denotes incorrect accounting of multiple rows, Abused denotes abused clauses, and Error denotes SQL compile error.

 

* [1] The Prediction Result differs from the Ground Truth Result. However, this does not necessarily mean that the Prediction Query is incorrect. Analyze the differences between the Prediction Query and Reference Query, considering the following:
2
3Correct
4- The Prediction Query logically answers the Question, even if the output structurediffers from the Ground Truth Query.
5- Donot consider column naming, column/row ordering.
6- Some extra column or missing column in the output structure is acceptable if it doesn affect the query's ability to answer the Question.
7- Differences in the representation of values, such as formatting (percentile, YES /NO) or data types, are acceptable if they do not affect the query's logical correctness.
8- Ambiguous questions may have multiple correct answers, so the Prediction Query may differ from the Ground Truth Query.
9- Multiple rows are acceptable when the calculate themin/max
10
11Incorrect
12- The Prediction Query does not logically answer the Question or contains significant errors.
13- The Prediction Query produces different results due to incorrect filtering or missing conditions, JOIN redundancy, or other fatal issues.
14- The Prediction Query fails to handle null values, multiplerows, or other critical aspects of the Question.
15- The result of the Prediction Query is significantly different from the Ground Truth Query events instructive is similar.
16
17SQLite3 Compatibility
18- Both queries are SQLite3 compatible, meaning integer division is not automatically converted to float division, and logical operators are case-insensitive.
19- If the tables schema and description aredifferent, follow the schema provided in the prompt.
20
21Provide a detailed comparison of the Prediction Query and Ground Truth Query, focusing on the nature and significance of their differences. If the Prediction Query is incorrect, explain the specific errors and how they affect the query's ability to answer the Question.
22
23Finally, score the Prediction Query asks:
24~json
25{"correct": true/false}
26...
27Figure 11: **FLEX system prompt to judge the generated query producing different results as ground truth.

 

* [1] You are an expert SQL evaluator responsible for assessing the correctness and quality of SQL queries.
* [2] Analyze the judgment thoroughly and provide a categorized evaluation based on the following criteria:
* [3] Two queries have no compile-time errors, which means theyare both valid SQL queries, integer division is not automatically converted to float division, and logical operators arecase-insensitive.
* [4]### Criteria
* [6] I. Incorrect Schema Linking: Utilized tables and columns in the predicted query do not align with the question and the provided schema. Different columns are permissible if they are described as equivalent in the table description or if the question does not specify exact column names.
* [7] 2. Incorrect Filtering Conditions: The prediction query incorrectly filters data based on the given conditions, ensuring the WHERE clause is used appropriately to match expected results.
* [8] 3. Missing Handling of Nullable Column: Check if the query correctly handles nullable columns in aggregation functions (SUM, COUNT, AVG) or other operations, as improper handling can lead to unexpected results. Do not consider NULL values in arithmetic operations.
* [9] 4. Missing Handling of Multiple Rows: Determine if the query correctly accounts for scenarios where multiple rows might satisfy the condition (e.g min/max, multiple transactions in a day), potentially leading to incorrect answers. Consider primary and foreign keys, as well as unique constraints, which ensure uniqueness.
* [10] 5. Abused Clauses: Evaluate if SQL clauses like GROUP BY, HAVING, ORDER BY, and DISTINCT are used unnecessarily, which could produce incorrect results.
* [11] 6. Other Fatal Logical Issues: Identify any additional logical problems not covered by the above criteria.
* [12]
* [13]
* [14] *** Output Format NOTE: If nosise is found, the explanation can be an empty string, Output only the JSON object containing your evaluation results. Make sure the JSON is properly formatted and valid, with all boolean values in lowercase (true/false) and all strings properly enclosed in double quotes.
* [15] ...json
* [16] {
* [17] "incorrect_schema_linking": {
* [18] "issued": true/false,
* [19] "explanation": "Your explanation here"
* [20] },
* [21] "incorrect_filtering_condition": {
* [22] "issued": true/false,
* [23] "explanation": "Your explanation here"
* [24] "missing_handling_of_nullable_column": {
* [25] "issued": true/false,
* [26] "explanation": "Your explanation here"
* [27] "missing_handling_of_multiple_rows": {
* [28] "issued": true/false,
* [29] "Xor Explanation": "Your explanation here"
* [30] "missing_handling_of_multiple_rows": {
* [31] "issued": true/false,
* [32] "issued": true/false,
* [33] "abused_clauses": {
* [34] "isued": true/false,
* [35] "explanation": "Your explanation here"
* [36] "other_fatal_issues": {
* [37] "issued": true/false,
* [38] "explanation": "Your explanation here"
* [39] "explanation": "Your explanation here"
* [40] "explanation": "Your explanation here"
* [41] ...
* [42] ...

Figure 12: **FLEX system prompt to categorize the error case of false positives.

 

* [1] You arean expert SQL evaluator responsible for assessing the correctness and quality of SQL queries.
* [2] Analyze the judgment thoroughly and provide a categorized evaluation based on the following criteria:
* [3] Two queries have no compile-time errors, which means they are both valid SQL queries, integer division is not automatically converted to float division, and logical operators arecase-insensitive.
* [4] ### Criteria
* [6] 1. Different Output Structure: Columns selection, ordering is different, but should not consider output column naming.
* [7] 2. Different Output Value Representation: Differences in value representation, such as formatting or data types, should not be fatal unless they affect the logical correctness of the query.
* [8] 3. Incorrect Ground Truth Query: The ground truth query is incorrect and does not logically answer the question.
* [9] 4. Multiple Answers Available: The questions ambiguous and has multiple correct answers.
* [10] 5. Other minor issues: Otherminor issues that don't affect the logical correctness of the query.
* [11]
* [12] ## Output Format
* NOTE: If no issue is found, the explanation can be an empty string. Output only the JSON object containing your evaluation results. Make sure the JSON is properly formatted and valid, with all boolean values in lowercase (true/false) and all strings properly enclosed in double quotes.
* [13] ...json
* [14] {
* [15] "different_output_structure": {
* [16] "issued": true/false,
* "explanation": "Your Explanation here"
* [17] "different_output_value_representation": {
* [18] "isued": true/false,
* [19] "explanation": "Your explanation here"
* [20] "multiple_answers_available": {
* [21] "issued": true/false,
* [22] "evaluation": "Your explanation here"
* [23] "other_minor_issues": {
* [24] "issued": true/false,
* [25] "explanation": "Your explanation here"
* [26] }
* [27] ...

Figure 13: **FLEX system prompt to categorize the error case of false negatives.

 

* **Schema**
2{schema}
3
4**Question**
5{question}
6
7**PredictionQuery**
8{pred_query}
9
10**PredictionResult**
11{pred_result}
12
13**GroundTruthQuery**
14{gt_query}
15
16**GroundTruthResult**
17{gt_result}
18
19**Hint**
20{evidence} ```

Figure 14: **FLEX user prompt. Execution results are excluded for generated queries that do not produce the same results as ground truth. Hint indicates the external knowledge (evidence) that provided in BIRD.

Figure 15: Criteria and rubrics utilized in _Prometheus-2_ for the equivalent set.

 

* [15] M. C. Dickson, R. J. Thorne, and A. K. Szyminski, "Theoretical Population of the Universe: The Role of Life in Nature," _Journal of Modern Physics_**20**(4), no. 3, p. 79-86, 1996.
* [16] G. F. Hinton, "A Bayesian approach to estimating the age of a population of interest", _Annals of Statistics_**19**(1), pp. 105-118, 1970.
* [17] P. H. Anderson, "Bayesian inference for the age of an individual's population", _Nature_**406**(7258), 197-208, 1971.

[MISSING_PAGE_POST]