## 5 Discussion
### Error Analysis  
To assess whether our new metric gives a more accurate evaluation, we manually analyzed the false positives and false negatives generated by EXE, ESM, and ETM for each model on the Spider evaluation set and the BIRD development set. Since disabling either distinct or value checks leads to numerous false positives in both EXE and ESM, and most current state-of-the-art models predict values, we enabled these checks for our analysis.  
Table 6 presents the error analysis results. Despite enabling distinct and value checks, EXE and ESM still yield a high volume of false positives and false negatives, respectively. Notably, ESM's low false positive rate on BIRD partially stems from poor internal parsing, as it very rarely evaluates any two queries in BIRD as equivalent. For all models, the amount of false positives from EXE and false negatives from ESM decreases significantly in ETM. The decrease in false positives from EXE stems from the new constraints in ETM that correctly identify mismatches, while the false negative decrease from ESM is attributed to our equivalence rules in Table 2. The false positive rate in ESM results from the issues described in Section 3.2, which are fixed in ETM.  
On both EXE and ESM, false positives and false negatives disproportionately affect certain models. In particular, PLM-based models have higher false positive rates for EXE and higher false negative rates on ESM than FLM-based ones. In contrast, ETM exhibits a notably smaller discrepancy between the best and worst models, indicating it is less biased than either EXE or ESM. The few remaining false positives stem from inconsistencies in the database schema itself, where the actual database does not comply with the specified schema requirements. On Spider, the false negative rate is entirely eliminated, whereas on BIRD, a small number of false negatives persist. Although this is far fewer than the false positives for EXE, it suggests that additional equivalence rules could further reduce the false negative rate, which we will explore in future work.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **Model** & & \multicolumn{2}{c}{**EXE**} & \multicolumn{2}{c}{**ESM**} & \multicolumn{2}{c}{**ETM**} \\  & & **FP** & **FN** & **FP** & **FN** & **FP** & **FN** & **FP** & **FN** \\ \hline \multirow{8}{*}{Spider} & DAIL & P & 16.3 & 0.0 & 5.0 & 3.2 & 0.1 & 0.0 \\  & DIN & P & 19.5 & 0.0 & 6.1 & 9.8 & 0.0 & 0.0 \\  & C3 & P & 23.0 & 0.0 & 2.8 & 17.5 & 0.1 & 0.0 \\  & Super & P & 15.0 & 0.0 & 3.2 & 8.2 & 0.0 & 0.0 \\  & R+N & F & 10.0 & 0.0 & 4.6 & 3.8 & 0.1 & 0.0 \\  & R+P & F & 10.0 & 0.0 & 4.1 & 4.3 & 0.1 & 0.0 \\  & CodeS & F & 12.1 & 0.0 & 4.7 & 4.0 & 0.0 & 0.0 \\ \hline \multirow{8}{*}{BIRD} & DAIL & P & 17.4 & 0.0 & 1.6 & 26.3 & 0.3 & 1.1 \\  & C3 & P & 17.5 & 0.0 & 1.1 & 20.7 & 0.0 & 2.7 \\ \cline{1-1}  & Super & P & 17.9 & 0.0 & 1.2 & 27.1 & 0.2 & 1.2 \\ \cline{1-1}  & RESD & F & 11.1 & 0.0 & 1.7 & 20.7 & 0.1 & 1.0 \\ \cline{1-1}  & CodeS & F & 14.8 & 0.0 & 1.2 & 28.9 & 0.1 & 0.8 \\ \hline \hline \end{tabular}
\end{table}
Table 6: False positives and negative rates (%) for all models with respect to the three metrics on the Spider evaluation set and BIRD development set. ESM and EXE are evaluated with distinct and value checking enabled.