## 1 Introduction  
While interacting with SQL databases through natural language interfaces makes them significantly more accessible to non-experts, the task of automatically mapping natural language requests to SQL queries for relational databases, known as Text-to-SQL, remains challenging. Lately, the advent of the transformer [1] and large language models (LLMs) [2, 3] has led to momentous advancements in this field. Notably, LLMs have overcome several challenges in Text-to-SQL, evidenced by their dominance in leaderboards for popular benchmarks like the Spider dataset [4] and the more challenging BIRD dataset [5], underscoring their effectiveness in handling complex, multi-table SQL query generation that previous approaches had struggled with. Evaluating Text-to-SQL models is also challenging because SQL equivalence has been shown to be undecidable [6]. Text-to-SQL models are tested using two metrics: Execution Accuracy (\(\mathtt{EXE}\)) and Exact Set Matching Accuracy (\(\mathtt{ESM}\)). \(\mathtt{EXE}\) checks if the SQL execution result of the predicted query matches that of the gold standard query. However, \(\mathtt{EXE}\) can yield false positives, as semantically different queries may produce the same output (Figure 0(a)). On the other hand, \(\mathtt{ESM}\) assesses the predicted query by comparing sets of keywords and their arguments with those of the gold query. While more rigorous than \(\mathtt{EXE}\), \(\mathtt{ESM}\) is prone to false negatives because SQL queries may be semantically equivalent yetsyntactically diverse (Figure 1b). These issues raise the need for a more robust evaluation metric that accurately evaluates the performance of Text-to-SQL models. Models using pretrained LLMs without fine-tuning, such as GPT (henceforth **PLM**), perform particularly well on EXE, which is the main metric used on the Spider and BIRD leaderboards. Surprisingly, they do not show a similar level of performance on ESM. When using ESM as the primary metric, no PLM-based models rank highly, a stark contrast to the Spider and BIRD leaderboards. Therefore, it is critical to examine these metrics and determine the most suitable approach for an accurate evaluation of model performance, as the disparity between them disproportionately impacts PLM-based models compared with those using fine-tuned LLMs (henceforth, **FLM**). Our main contributions are as follows:  
* We identify and analyze critical mismatches between commonly used standard Text-to-SQL evaluation metrics (Execution Accuracy and Exact Set Matching). * We introduce and implement Enhanced Tree Matching (ETM), a novel metric that integrates syntactic normalization and rule-based semantic equivalence to better evaluate structural correctness in SQL generation. * We conduct comprehensive empirical studies across nine models on the Spider and BIRD benchmarks, showing that ETM reduces variance in evaluation and resolves failure cases missed by EXE and ESM. * We present detailed error analyses and rule-level ablations, providing insight into the kinds of reasoning and structure that ETM captures more effectively than existing metrics. This paper first examines potential issues in EXE and ESM and proposes a new enhanced metric, called ETM, which addresses many shortcomings present in the original metrics (Section 3). Nine state-of-the-art models are evaluated on the Spider and BIRD datasets, comparing their performance using EXE, ESM, and ETM (Section 4). Finally, a comprehensive error analysis is conducted on the evaluation results using these three metrics, revealing the superior robustness of ETM (Section 5). We posit that ETM will serve as a pivotal metric for assessing the real capabilities of LLM-based Text-to-SQL models, thereby enabling them to reach new heights of performance.