## 4 Results
### Results  
Table 4 shows the results of the seven Spider models (Section 4.1) with respect to EXE, ESM, and ETM. For the development set, Super performs the best on EXE, while R+N performs best on ESM. Super's EXE score is 2.6% higher than the next best model, CodeS, although its ESM score is 7.3% lower. This discrepancy is diminished to 1.5% with ETM; however, this still indicates it is not as strong as CodeS overall. DAIL, despite scoring better than every FLM model except CodeS on EXE, scores significantly worse on ETM, getting outperformed by both R+N and G+P. The trend is evident; FLM-based models exhibit 3-7% decreases in performance from ESM to ETM, whereas PLM-based models show 1-14% increases. This impact is especially dramatic for zero-shot methods. For example, C3 performs relatively well on EXE (only 3.7% lower than CodeS) but extremely poorly on ESM (32.5% lower than CodeS) and then substantially recovers on ETM (16.2% lower than CodeS). Likewise, DIN also underperforms on ESM and thus gets a significant 4.6% boost when evaluated using
On the evaluation set, the trend between ESM and ETM remains consistent. Overall, PLM-based models dominate FLM-based models on EKE: the best PLM-based model, Super, gives a 2.3% higher score than the best FLM-based model, CodeS. Unfortunately, we were unable to run the G+P model, so its results on the evaluation set are omitted from Table 4. This pattern continues with the other PLM models, which all outperform R+N and R+P on EKE. However, for ESM, CodeS outperforms Super by 8.2%, although the gap narrows to 2.8% on ETM. Note that the EKE scores of PLM-based models decrease from their originally reported values to our replicated results, while FLM-based models see similar or slightly higher scores. CodeS's scores on the evaluation set weren't reported, and CodeS was not submitted to the Spider leaderboard. We attribute these differences to the absence of distinct and value checking (Section 2.2), as well as the high variance in PLM-based approaches, discussed further in Section 5.2. Table 5 illustrates the results of the five models evaluated on BIRD's development set (Section 4.2). All models perform poorly on ESM, largely because the parser in the evaluation script fails to handle the complexity of the queries in the BIRD dataset. Consequently, runtime errors are common, and the script automatically classifies queries with such errors as syntactically incorrect, leading to notably low scores and indicating that ESM is not a suitable metric for a model's performance on this dataset. By contrast, ETM--unaffected by these shortcomings--still shows a marked difference from EKE. Similarly to the Spider results, models across the board perform worse on ETM than on EKE, though some are affected more severely than others. While Super achieves the highest EKE score, it is surpassed by CodeS-15 on ETM. C3 and RESD have a similar trend; despite outperforming RESD on EKE, C3 suffers a bigger drop off from EKE to ETM, causing RESD to score higher on ETM. A notable shift occurs in model rankings from EKE to ETM, as PLM-based models that dominate on EKE often underperform relative to FLM-based models on ETM. A likely explanation is that PLMs, which excel in broader language understanding, make implicit assumptions about the databases that prove problematic when the queries are evaluated on ETM. As discussed further in Section 5.1, these types of assumptions typically do not affect performance except in edge cases which may be absent in the dataset. In contrast, FLMs--though weaker on EKE--may avoid some of these pitfalls, resulting in higher ETM scores. Meanwhile, the performance decrease from ESM to ETM for the FLM-based models on Spider