EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing
Authors: Yizhang Zhu
Runzhi JIANG
Boyan Li
Nan Tang
Yuyu Luo
Venue: COLM 2025 (2025)
Keywords: Text-to-SQL
Routing
Cost-Efficiency
Large Language Model
TL;DR: We propose EllieSQL, a complexity-aware routing framework for Text-to-SQL, enhancing cost-efficiency while maintaining performance by directing queries to suitable SQL generation methods.

Abstract: Text-to-SQL automatically translates natural language queries to SQL, allowing non-technical users to retrieve data from databases without specialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQL approaches on leaderboards, their unsustainable computational costs—often overlooked—stand as the "elephant in the room" in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we exploratively propose EllieSQL, a complexity-aware routing framework that assigns queries to suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity of Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in our study, EllieSQL with the Qwen2.5-0.5B-DPO router reduces token use by over 40% without compromising performance on Bird development set, achieving more than a 2× boost in TEP over non-routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL. Our source code and model are available at https://elliesql.github.io/.

Published as a conference paper at COLM 2025
ElLieSQ:
Text-to-SQL with Complexity-Aware Routing
Yizhang Zhu" Runzhi Jiang"*, Boyan Li', Nan Tang Yuyu Luo
'Te Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China The Hong Kong University of Science and Technology, Hong Kong SAR, China
Abstract
Text-to-SQL automatically translates natural language queries to SQL, al- lowing non-technical users to retrieve data from databases without spe- cialized SQL knowledge. Despite the success of advanced LLM-based Text-to-SQI. approaches on leaderboards, their unsustainable computa- tional costs-ofter overlooked stand the "elephant in the room" in current leaderboard-driven research, limiting their economic practicability for real-world deployment and widespread adoption. To tackle this, we ex- ploratively propose EllieSQL, a complexity-aware routing framework that assigns queries suitable SQL generation pipelines based on estimated complexity. We investigate multiple routers to direct simple queries to efficient approaches while reserving computationally intensive methods for complex cases. Drawing from economics, we introduce the Token Elasticity Performance (TEP) metric, capturing cost-efficiency by quantifying the responsiveness of performance gains relative to token investment in SQL generation. Experiments show that compared to always using the most advanced methods in study, EllieSQL with the Crren..5-0.5--DD router reduces token use by over 40% without compromising performance Bird development set, achieving more than 2x boost in TEP over non- routing approaches. This not only advances the pursuit of cost-efficient Text-to-SQL but also invites the community to weigh resource efficiency alongside performance, contributing to progress in sustainable Text-to-SQL. Our source code and model are available at https://ellleest gitb io/
1 Introduction
Text-to-SQI. converts natural language into SQL queries, significantly lowering the barriers to database interaction and advancing the democratization of data science (Qin et al., 2020; Luo et al., 2018). With the advent of large language models (LLMs) and their powerful se- mantic understanding capabilities, LLM-based Text-to-SQL approaches have demonstrated remarkable performance on prominent benchmarks (Liu et al., 2024; et al., 2024b; Yu et al., 2018). These methods often require sophisticated and computation-intensi tion processes, including fine-grained task divisions (Talaei et al., 2024), online synthesized examples and divide-and-com Search-enhanced reasoning (Li et al., 2025).
SQL genera-
approaches (Pourreza et al., 2025), and Monte Carlo Tree
Despite the success of these advanced LLM-based Text-to-SQL solutions, there is a crucial challenge: excessive token consumption. While these approaches achieve high ranking on leaderboards, the performance improvements are marginal relative to their exponential com- putational demands. When handling massive volumes of queries real-world applications, the computational expenses associated with sophisticated pipelines can render such systems economically unsustainable. This issue represents a fundamental yet frequently overlooked limitation, posing major obstacle to deployment outside the laboratory. We argue this is an "elephant in the room" for the current lraderboard-drive Text-to-SOi research.
'Equal contribution.
'Yuyu Luo is the corresponding author. E-mail: yuyaludehkust-gr.eduuu

---

Published as a conference paper at COLM 2025
A notable observation is that existing approaches apply uniform processing pipeline to all queries, despite the fact that not all queries necessitate such sophisticated workflows to generate correct SQL. Even basic baseline models, without any reasoning enhancements, are capable of bolving a fair number of cases. For many simple queries, indiscriminately applying a complicated process not only results in large amount of unnecessary token overhead but may also lead to overthinking, increasing the risk of errors in straightforward cases (Cuadron et al., 2025).
To address it, we propose EllieSQL, a complexity-awaie routing framework for SQL gener- ation. EllieSQL employs three-tiered SQL generation module (Basic, Intermediate, and Advanced) that dynamically routes queries to the appropriate tier based on their estimated complexity. By efficiently handling simpler cases with lightweight SQL generation methods and reserving computationally intensive pipelines for genuinely complex ones, EllieSQL. significantly enhances cost-efficiency while maintaining high performance.
To quantitatively assess the cost-efficiency of SQL translation, we introduce Token Elasticity of Performance (TEP), a metric grounded in economic principles (Marshall, 2013). TEP treats tokens as a form of computational investment and quantifies the responsiveness of performance improvements relative to increased token consumption. This provides a standardized measure to evaluate the cost-efficiency trade-offs inherent in different SQL generation approaches, which is particularly crucial for real-world deployment scenarios where both performance and computational efficiency are essential considerations.
We aim to invite the whole community for further research into balancing performance with resource expenditure, hoping to contribute to more practical and sustainable Text-to- SQL. This paper serves as an exploration, a stepping stone rather than a definitive answer, intended to spark further research and discussion. In this paper, we try confront the elephant with contributions summarized as follows:
Elephant in the Room. We highlight a critical yet overlooked cost-efficiency limitation in current Text-to-SQL methods, hindering their practical deployment beyond laboratories.
TEP Metric. We introduce Token Elasticity of Performance (TEP), an economic-inspired metric for evaluating the responsiveness of performance gains relative to token invest- ments in SQL generation.
EllieSQL. We propose EllieSQL framework with various router implementations to direct queries to appropriate tiered SQL generation pipelines based on estimated complexity. Extensive Experiments. Experiments exhibit the potential and effectiveness of EllieSQL with various routers. Notably, with the Swen2.5-0.5B-DPF router, reduce token use by over 40% without sacrificing performance compared with consistently deploying the most advanced SQL generation pipeline, achieving more than 2x boost in TEP.
2 Related Work
Text-to-SQL. Text-to-SQL converts natural language queries into SQL statements. Since large language models (LLMs)) have demonstrated remarkable capabilities and versatility in data analysis (Zhu et al., 2024; Wu et al., 2025), researchers have increasingly explored their potential in Text-to-SQL tasks (Chen et al., 2024a; Liu et al., 2025). Earlier work, such as DAIL-SQL (Gao et al., 2024), has systematically benchmarked LLM-based Text-to-SQL. Li et al. (2024a) propose NL2SQL360, a testbed for comprehensively evaluating Text-to- SQL methods from multiple angles. DIN-SQL (Pourreza & Rafiei, 2023) performs internal classification, heuristically categorizing queries into manually-definel categories to select category-specific few-shot examples and handcrafted prompts to improve accuracy.
More recent efforts emphasize fine-grained task decomposition and advanced reasoning. CHESS (Talaci et al., 2024) meticulously designs a highly refined schema pruning technique. CHASE-SQL (Pourreza et al., 2025) enhances the SQL generation with three Chain-of- online synthesis, and query planning. Monte Carlo Tree Search has also been utilized to strengthen reasoning in Text-to-SQL. systems like Alpha-SQL (Li et al., 2025), which iteratively infers SQL construction actions based on partial SQL query states. YORO (Kobayashi et al., 2025) internalizes database schema into the parametric knowledge of LLMSs via fine-tuning to reduce input tokens in Text-to-SQI..
Thought strategies: divide-and-conquu
2

---

Published as a conference paper at COLM 2025
However, these approaches uniformly apply computationally intensive techniques to all queries, regardless of their difficulty. The inefficiency of overprocessing simple queries that lightweight methods could handle uncovers an opportunity for optimization via complexity- aware routing.
LLMs Routing. Routing trains model to select the most suitable LLM for each query without running inference on all candidates (Zhang et al., 2024; Xiang et al., 2025). RouteLLM (Ong et al., 2025) fine-tunes routers to make binary routing decisions via classification token, while ZOOTER (Lu et al., 2024) distills rewards from training queries to guide routing. RouterDC (Chen et al., 2024b) employs dual contrastive learning, Hybrid LLM (Ding et al., 2024) assigns queries based on predicted difficulty and desired quality level. Malekpour et al. (2024) conduct preliminary experiments on routing for a proper LLM out of three for SQL generation yet sacrificing performance. LLM cascading offers sequential alternative to assign tasks based on confidence scores post-generation, yet increasing costs for complex queries (Varshney & Baral, 2022). While existing LLM-level routing methods primarily focus on selecting models for each query, Text-to-SQL research re- veals that optimized reasoning pipelines play fundamental role in achieving performance improvements in SQL generation. Thus, the strategic selection of suitable SQL generation pipelines is fundamental, but complexity-aware routing for them remains unexplored.
3 Overview of EllieSQL Framework
In this section, we will first present the overview of EllieSQL. framework and its three key phases. Following this, we discuss the metrics involved to assess Text-to-SQI. and routers' performance, as well as the newly introduced Token-Elasticity Performance (TEP) for evaluation on cost-efficiency in SQL generation.
3.1 Overview
Given a database D and a user's natural language query Q, the goal of Text-to-SQL. is to produce a SQL. statement y based on D and Q. Formally, y = fDD, 2). To achieve this goal efficiently across varying query complexities without compromising performance, we propose EllieSQL, a complexity-awan
routing framework for cost-efficient Text-to-SQL.
As shown in Figure 1, our EllieSQL operates in three key phases: schema linking, routing, and tiered SQL generation pipelines. At the heart of EllieSQL lies the router, serving as the dicision-making core, which dynamically directs queries to the appropriate tier of SQL generation pipelines based on estimated query complexity.
Our motivation stems from the following observation: in practice, Text-to-SQL. tasks exhibit significant heterogeneity in complexity, where not all queries require the most sophisticated and resource-intensive methods for effective resolution. Existing leading approaches, how- ever, indiscriminately apply complex reasoning and computationally expensive techniques to all queries, resulting in inefficiency as simple queries could be adequately redses by lightweight methods. Moreover, Text-to-SQL research reveals that enhancing reasoning within SQL generation is fundamental for improving performance, particularly for complex queries, which has driven the development of diverse, meticulously designed SQL genera- tion pipelines. Such diversity underscores clear opportunity for complexity- at the pipeline level, which promises to optimize computational efficiency while providing enterprises with the flexibility to integrate tailored pipelines for their specific requirements. Next, we elaborate on the overview and formulation of each phase within EllieSQL.
routing
3.2 Phase I: Schema Linking
Overview. Given a database D and natural language query Q, the database schema Sp often contains numerous elements irrelevant to Q. Schema linking is the process of identifying the tables and columns necessary for SQL generation aligned with the user's 's intended query (Lei et al., 2020; Liu et al., 2024). By eliminating irrelevant schema elements, schema linking reduces input complexity and prevents LLMs from being overwhelmed with redundant information that could lead to performance degradation, making schema linking a critical component in Text-to-SQL, (Cao et al., 2024; Talaei et al., 2024; Li et al., 2025).
3

---

Published as a conference paper at COLM 2025
DB 1
NL Query
SQL Query
What is the total number superheroes without full name?
CB
SELECT COUNT(je) FROM superhero
WHERE /naee
Schema Linking
GM
DB
Query
SQL Query
What was the release date for the set which card "Evacuation" in
SELECT releasedate FROM 72.code T1,setCode WHERE 11 nome Evocuation
S
D
NL Query
Router
SQL Query
List down top three German drivers who has the shortest average pit stop duration and were 1980-1985.
SELECT foronamm, 12aumam: pitscops 11.dhveid "2 cecereer 60 EER retticicaltyy Cermon STATT TINETTY GROUP ORDER AVG 3.duradd)) UMT
S
Phase I I Phase I1-+
Phase III
Figure 1: Framework of EllieSQL, consisting of three phases: 1) schema linking; 2) routing to appropriate SQL generation method; and 3) three-tiered SQL generation pipelines (Gg: Basic, Gm: Intermediate, GA: Advanced, with performance and cost increasing in this order).
Formulation. Formally, when deploying an LLM M, the schema linking in Phase I can be denoted as S M(Sp, where S/ represents the filtered schema.
3.3 Phase II: Routing
Overview. Within the framework, the router is strategically positioned in Phase II, following the schema linking phase, schema linking constitutes standard preprocessing in Text-to- SQL. More importantly, the filtered schema Stp provides valuable structural information that strongly correlates with query complexity. For instance, heuristically, query with a linked schema involving only a single table and two columns (e.g., "List all customers' names from California") can be less complex than one requiring joins across multiple tables with numerous columns (egg." "Find the average order value of premium customers who made purchases both January and February"). Additionally, the natural language query itself contains important complexity cues, particularly regarding functions or calculations potentially required in the SQL generation (e.g,, COUNT or STRFTIME) (Li et al., 2025). Conse- quently, in Phase 11, the router is well-positioned to estimate the complexity of the query at hand by leveraging both the natural language query Q and its corresponding filtered schema Spp. We discuss various router implementations in Section 4.1.
Formulation. Let G denote the set of available SQL generation pipelines. The router R, is defined as a function targeted to map a natural language query Q, and its corresponding filtered schema StD, to an optimal SQL generation pipeline G. E G which is the most efficient method capable for the query. Specifically, the router R. utilizes Q and S to estimate query difficulty in its routing decision. Formally, the routing process is denoted as R Q x S G, such that R(2,5') = G..
3.4 Phase III: SQL Generation Pipelines
Overview. The three-tiered SQL generation pipelines in EllieSQL tackle the fundamental trade-off between performance and computational cost. We introduce hierarchical set of three-tiered SQL generation pipelines-Bi -Bsiic (GB), Intermediate (GM), and Advanced (Ga)-each offering progressively better performance at correspondingly higher token consumption. This tiered structure enables our router to assign queries to the most cost- efficient pipeline capable of handling a given query's s complexity.
Formally, each SQL generation pipeline G, E G {GB, Gm, GA takes as input a natural language query Q and its schema linking result Stp, producing a SQL query = (O)))) After the router assigns the generation method G. in Phase II, the SQL generation process in Phase III can be formulated as y = ( S)). We discuss the implementation of three-tiered SQL generation pipelines in Section 4.2.
Formulation.
4

---

Published as a conference paper at COLM 2025
3.5 Metrics
EX. Execution Accuracy (EX) (Yu et al., 2018) evaluates the performance of the Text-to-SQL. system by comparing whether the execution result sets of the gold SQL queries and the predicted SQL queries are identical:
EX 1 Ri)
(1)
N
where Vi and V; represent the execution result sets of the i-th predicted and gold SQL query, and N is the total number of examples. Let the EX of basic (GB), intermediate (GM) and advanced (Gx) SQL generation pipelines be EXB, EXM and EXA, respectively.
Performance Gap Recovered (PGR) to evaluate routers' performance. This metric captures how much of the performance difference between the weak and strong methods is recovered by the router R:
PGR. Following Ong et al. (2025),
EXR EXB
PGRR
(2)
EXA EXB
TEP. Inspired by the concept of elasticity in economics (Marshall, 2013), which measures how responsive one variable is to changes in another, we define Token Elasticity of Perfor- mance (TEP) as a metric to evaluate how sensitively the performance of SQL generation pipeline, denoted G, responds to changes in token investment. Token consumption for method is defined as T Tin + jTTout, where Tin represents consumed prompt tokens (input), Tout the completion tokens (output), and şi is a multiplier factor for completion tokens. The average token consumption then computed as T T/N, where N is the number of samples. TEP is formally expressed as:
% change in performance
AEX; EXB
TEP % change in average token consumption
(3)
ATc/TB
where ΛEX; EXg EXg represents the performance improvement of the tested SQL generation method G over baseline method GB, measured in terms of EX. Similarly, ATG = Tc Tв denotes the additional average token consumption of method G relative to the baseline. A higher TEP value indicates greater efficiency in translating token investment into performance gains, reflecting better cost-effectiveness.
4 Implementation Details
4.1 Router Implementation
We investigate various router implementations across three categories for experiments: classification-baeed routers, cascading routers, and preference learning-based routers.
Classification-based Routers directly predict the most suitable SQL generation pipeline from the set of available methods G by treating routing as multi-class classification problem. We explore two types of classification approaches: k-nearest neighbors (KNN) and supervised fine-tuning (SFT) lightweight language models as classifiers.
KNN. The KNN router represents simple heuristic approach: Queries involving more tables and columns are likely to be more complex, potentially requiring more sophis- ticated generation pipelines. Using training set of queries with known optimal SQL generation pipeline allocations, it extracts feature vectors v = ([T), [C]) representing the number of tables and columns in S.p For new query, it 1) Extracts v, Finds k nearest neighbors in the training set, and 3) Assigns the most common pipeline among neighbors.
SFT. We fine-tune two lightweight language models classifiers: 1) RoBERTa-bass encoder-only model; 2) Qwen2.5-0.51 with an added linear classification head that maps its hidden states to probabilities of SQL generation pipelines for multi-class classification.

---

Published as a conference paper at COLM 2025
Cascading Routers implement sequential decision-making process using series of binary classifiers. Each generation method has a corresponding classifier that determines if can handle the current query. Different from Varshney & Baral (2022); Chen et al. (2023), our cascading routers still predictively route queries without performing SQL generation. The cascading routing follows waterfall pattern:
classifier
G1
(2,
if
Recase (2,5p)
G
if (()))))0)))))))))) (B)(())))= = 1,ie n2 1-1, 1}, (4)
Gn
otherwise
where Bj is the binary classifier for method G, We implement these classifiers by fine- tuning RoBERTa-base and Qwen2.5-0.5B models. Compared with multi-class classification, the cascading approach offers better extensibility (adding methods requires only one new classifier without affecting existing ones) and eases data collection.
Preference Learning-baat based Routers approach the routing task learning relative prefer- ences between generation methods rather than making hard classifications. We construct preference pairs (G; Gj Sp)) and fine-tune Qwen2.5-0.5B with two preference learning techniques: Pairwise Ranking and Direct Preference Optimization (DPO).
Pairwise Ranking is used to train the model to distinguish between better and worse generation methods using margin-based ranking loss (Joachims, 2002):
Lrank = max(0, s(G)1 s())))
(5)
where (GI is the logit assigned to G, and À is the margin hyperparameter.
DPO directly optimizes the preference model by maximizing the probability of preferred generations over less preferred ones (Rafailov et al., 2023) using:
Ti G 2,5)) Te (G 2,
Blog No (T()))))))
LDPO -E(G,-G, ( log
(6)
n (nrr( GI
where 7i is the policy model being trained, and Thef is a reference model that remains unchanged during the training process. The B is temperature parameter controlling the divergence between Ty and Tef--
4.2 Three-tiered SQL Generation Pipelines Implementation
We introduce the implementation of our three-tiered SQL generation pipelines in Phase III as follows. All of the used prompts shown in Appendix Section A.3.
Basic SQL Generation Pipeline (GB). This tier adopts direct input-output approach, generating SQL queries solely based on the given natural language query and schema linking results, without any intermediate reasoning steps. Although it is less effective for complex queries, it can still handle straightforward cases. As the simplest and most lightweight method, it has the lowest token consumption.
Intermediate SQL Generation Pipeline (Ga). To improve the handling of complex queries, this tier follows CHASE-SQL (Pourreza et al., 2025) and deploys a Divide-and-Conquer Chain-of-Though strategy. Specifically, as illustrated by Phase Ill-GM in Figure 1, the original query is decomposed into sub-questions, which are solved independently and then combined into complete solution. A final refinement step is applied to correct any issues in the generated SQL, such as non-executable queries or queries yielding empty results. This method is reported to be more effective for handling complex queries but comes with increased token consumption per example.
Advanced SQL Generation Pipeline (G^). Building on the intermediate tier, this approach further integrates the online synthesis mechanism adopted from CHASE-SQL (Pourreza et al., 2025) to enhance SQL generation. In addition to the decomposition and refinement steps in GM, online synthesis is applied to each subtask. This mechanism dynamically

---

Published as a conference paper at COLM 2025
Training Set
GB
SQL
GB
Gv
4770
GB
776
Gy
3476
GA
Total
9012
GM
SQL
SQL
Pipeline
Label
Distribution of as- signed labels in training set
Table 1:
Figure 2: Training data construction for routers
synthesizes relevant examples based the given schema shown in Figure 1 Phase IIII- GA, providing specific few-shot demonstrations tailored to the test query. By constructing demonstrations using pertinent tables and columns, this approach understanding of the underlying data schema, making GA the most powerful- also the most computationally expensive basei method in our study.
gain a better
Experiments
5.1 Experimental Setup
Dataset. Bird (Li et al., 2024b) provides cross-domain Text-to-SQL tasks, which are diverse and proportionate in terms of query complexity. Therefore, we utilize the Bird development set with 1534 (NL, SQL) pairs on 11 databases for evaluation and the Bird training set with 9012 valid (NI., SQL) pairs on 69 databases for fine-tuning. Importantly, the training and development sets in Bird are constructed based entirely separate databases without any overlap, ensuring that the router is challenged to generalize across different databases during evaluation. Moreover, to evaluate the generalizability of EllieSQL, we assess its out-of-distribution (OOD) performance using the Spider-dev, Spider-Reer Realistic-dev (Deng et al., 2021), and Spider-test (Yu et al., 2018) datasets.
To train our routers, we construct the training data using a waterfall approach, as shown in Figure 2. Each query is first attempted with the basic SQL generation pipeline (Gg); successful, it receives a G3 label. Failed queries cascaded to the next-tier intermediate pipeline (GM) and then the advanced pipeline (GA) necessary. In this way, each query is assigned the label with the most efficient capable SQL generation pipeline. Table 1 shows the label distribution in the training set. For preference learning, we derive preference pairs from the label, such as GB > Gm and GB GA for query assigned with label Gg.
Training and Evaluation. As mentioned in Section 3.3, RoBERTa-base and Qwen2.5- 0.5B are used as base models to train different routers. For brevity, hereafter, we refer to them as RoBERTa and Qwen, respectively. We fine-tune our routers using the LoRA (Hu et al., 2022) on four RIX 4090 GPUs with a learning rate of le-4. For evaluation, to allow a fair and controlled assessment of the performance and cost-efficiency inherent in each SQL generation pipeline, we consistently deploy pot-4o-mini-2024-07- model across three tiers of SQL generation pipelines in Phase III, where μ = and all temperatures set to 0.
the backbone
The primary objective our experiments is to validate whether our complexity-aware routing framework for Text-to-SQI. can maintain performance parable to consistently using the most advanced pipeline while significantly reducing token consumption. Therefore, our analysis focuses on relative performance differences and comparisons between different methods, rather than absolute values.
Experiment Objective.
5.2
Experimental Results and Analysis
5.2.1 Performance
As illustrated in Table 2, our experimental results reveal clear performance progression across base methods on the Bird dev, with Advanced (Gx) outperforming Intermediate (GM) and Basic (Gg) SQL generation pipelines, particularly on hard cases. Among routing approaches, Qwen DPO and RoBERTa Cascading stand out as the top routers with no performance sacrifice- even slightly outperforming the standalone GA. The Qwen DPO router demonstrates best performance in challenging queries among routers though still slightly weaker then consistently using GA- Meanwhile, most router implementations
7

---

Published as a conference paper at COLM 2025
EX(%) Performance
Cost-efficiency
Category
Method
Totalf
Sim.
Mod.
Chal.
PGR
T
Time.
Basic (Gn)
51.83
59.78
42.03
32.41
695.55
114
Base
Intermediate (GM)
54.17
61.62
44.40
37.93 93
11792.16
0.283
Advanced (Gx)
55.02
62.07
43.53
42.76
13002.91
0.348
653
KNN
52.93
60.97
42.24
35.86
3615.67
0.345
0.506
RoBERTa SFT
54.30
61.95
44.83
35.86
6244.72
0.774
0.597
Qwen SFT
54.30
61.95
44.18
37.93
7220.68
0.774
0.508
402
Routing
RoBERTa Casc.
55.15
62.92
45.04
37.93
8211.00
1.040
0.593
Qwen Casc.
54.50
62.38
44.18
37.24
8481.66
0.836
0.460
644
Owen Pair. Rank
53.72
60.22
45.47
6476.39
0.592
0.439
Qwen DPO
55.41
62.38
45.91
41.38
7641.51
1.122
0.692
396
Table 2: Execution Accuracy (EX) with breakdowns across Simple (Sim.), Moderate (Mod.), and Challenging (Chal.) examples, Average Token Consumption per example (T), Per- formance Gap Recovered (PGR) by Router, Token Elasticity of Performance (TEP), and inference time duration in seconds on Bird dev. The bold and underlined results are the best and second best in each category.
Taken Elasticity of Perf. (x10-=)
Owen DPO
2.7
Qwen DPO
EX Performance Bird-dev
36.0
RoBERTa Cascading
Ro SFT
RollE Cascading
06.5
RodERTs SFT
Qwen Caorading
3.5
Owen SET
Intermediate
Cascading
2.-
Owan Pairwise Ranking
53.
Owen Pairwise Ranking
Pareto Front
KNN
Base Methods
32-
Routing (Classification)
Routing (Cascading)
52.
Routing (Classification)
(Cascading)
2.
Routing (Preference Learning)
52.0
Basic
Routing (Preference Learning.
0/
32000
Performance Gap Recovered by Router
Avg Token Consumption per Example
Figure 3: Performance Gap Recovered (PGR) Figure 4: EX performance vs. average token by Router vs. Token Elasticity of Performance consumption per example (T) on the Bird dev, (TEP) on Bird dev.
and the Pareto Frontier.
showcase competitive performance, with approaches like RoBERTa SFT, Qwen SFT, and Qwen Cascading achieving better overall execution accuracy than consistently using GB- Even when these approaches perform slightly below consistently using GA, the perfor- mance degradation remains acceptable. These results exhibit the feasibility and potential of complexity-awart routing in Text-to-SQL. confirming that with appropriate routing strate- gies, we can match the performance of consistently using the most advanced method, even without compromising performance.
Additionally, as outlined in Section 5.1, the training and development sets are partitioned with overlapping databases, which challenges routers to generalize effectively across distinct databases, thus their strong performance also further presents transferability and adaptability across diverse databases.
5.2.2 Cost-efficiency
The right side in Table 2 presents the average token consumption in SQL generation, Perfor- mance Gap Recovered (PGR), and Token Elasticity of Performance (TEP) across different methods. The PGR assesses the effectiveness of routers (Ong et al., 2025), while TEP measures the responsiveness of performance gains relative to token investments.
Base methods exhibit a clear trade-off between performance and cost, with more powerful SQL generation pipelines requiring significantly more tokens than Basic methods to achieve the performance improvements noted in the middle part of Table 2, resulting in lower TEP. In contrast, our routing approaches have proven effective by demonstrating greater efficiency in converting token investment into performance gains, with all router implementations showing improved TEP. With the perfforernnnng router, Qwen DPO, SQL generation consumes over 40% fewer tokens than GA without sacrificing performance. This doubles the
8

---

Published as a conference paper at COLM 2025
Spider-dev
Spidee--Ralisticdee
Spidertest
Category
Method
EX(%)
T
DX(%)
TEP
EXC%)
Tg
TEP "
Basic (Gy)
74.27
363.40
73.23
377.95
74.29
376.14
Base
Intermediate (Gy)
25.73
2432.13
0.345
74.80
2520.67
0.378
76.90
2422 84
0.645
Advanced (Ga)
76.60
3826.14
0.356
75.39
3584.01
0.537
KNN
25.05
1033.28
0.570
72.64
1171 93
0.383
75.08
974.26
0.669
RoBERTa SFT
75.92
1222.21
0.980
73.43
1128.54
0.138
75.78
1130.04
1.001
2014.60
75.39
2064.06
75.55
Routing
RaBERTa Casc.
76.21
1516.21
0.825
74.21
1593.84
0.416
76.90
1539 21
1.136
Owen Casc.
75.53
3410.96
022112
73.98
3526.75 75
0.451
77.04
3296 22
U.477
Qwen Pair, Rank
74.76
2334.71
0.122
74.61
2379,99
0.356
75.08
2245 64
0.214
Owen DPO
76.31
1906.79
0.647
75.79
1721.43
0.983
77.97
1850,8
1.263
Table 3: Experimental results on out-of-distribution (OOD) datasets. underlined results are the best and second best in each category.
The bold and
GA
GA Trend
(k)
PGR
TEP (x10-))
Distr
7 (k)
Distr
EX
Distr
PGR
Distr
TEP (x10-7)
56
121.0
0.8
0.8
0.0
55 0.0
0.8
0.7
0.8
0.6
0.6-
14
0.6
05
53
04
04
92-
0:2-
02 0.2
0.3
TT. Gx)
TEX,
T(PGR, 62)
T(EP Ga
RaBERts SFT
0.
KNN
ROBERT Carc
SFT
Owen Pairwise
Owen DPO
RaBERTs SFT SFT RoBERTa Case Owen Qwen Pairwise Owen DPD
0.0
KNN
RaBERS SFT
0.g
KNN
RoBERa Case Ownn Painwing
SFT
Owen DPD
ROBERTS SFT
0.00.0
KNN
ROBERA Case Owen Case Pairwise
SFT
Owen DPG
02
Figure 5: Comparison of GA allocation routing with other metrics.
TEP, making it twice responsive token consumption in terms of performance gains and achieving the best cost-efficiency. Although the RoBERTa SFT-based router experiences slight performance loss, it also demonstrates competitive cost-efficiency. Cascading routers are competitive in EX and routing performance but are slightly inferior in TEP compared to SFT-based routers due higher token costs.
Figure 3 illustrates the relationship between routers' PGR performance and TEP for SQL generation, revealing an overall positive correlation. This implies that superior routing performance tends to demonstrate greater token elasticity during SQL generation. Figure depicts the EX performance and token cost with Pareto Frontier (purple line), highlighting the cost-efficiency advantages of routers based on RoBERTa SFT and Qwen DPO. Neverthe- less, even though the RoBERTa Cascading router dominated by Qwen DPO on the Pareto Frontier, it remains a practical option in many scenarios due to its superior extensibility, as adding new methods requires only one new classifier without affecting existing ones, which can ease the data preparation and router training.
EllieSQL's 's efficiency extends to processing speed. Notably, under uniform settings, experi- ments on Bird dev also demonstrate that Qwen DPO router reduce total time duration by over 40% compared to the consistent deployment of GA- However, the cascading router design, while offering superior extensibility for future integrations, inherently demands more processing time. This presents a trade-off between immediate computational efficiency and the system's long-term architectural scalability.
5.2.3 Generalizability on Out-of-Distribuution (OOD) Datasets
To assess the generalizability of EllieSQL, we further evaluate several OOD datasets: Spider-dev,
and the Spider-test set. Note that routers are trained exclusively on the Bird training set and then directly deployed on these OOD datasets without any fine-tuning on the Spider training set.
As presented in Table 3, across all three OOD benchmarks, our proposed EllieSQL, par- ticularly with Qwen DPO and cascading routers, significantly reduces token costs while the Spider-test, EllieSQL deployed with
maintaining high performance. Specifically, Qwen DPO router not only surpasses GA in performance but does so with a 48% token reduction, yielding superior TEP. Likewise, on Spider-Realistieddee, EllieSQL with multi-

---

Published as a conference paper at COLM 2025
ple routers, including Qwen DPO, Qwen SFT and Qwen Cascading, achieves even better performance than standalone GA while reducing token consumption by 55% 46%, 8%, respectively, doubling even tripling the TEP metric. The strong OOD performance vali- dates that EllieSQL generalizes effectively to unseen datasets, confirming its robustness and efficiency for diverse Text-to-SQU. tasks.
5.2.4 Strategic Routing us. Advanced Method Selection.
Figure shows the proportion of GA assigned in routing across different router implemen- tations. We compare the usage frequency of the most powerful and costly SQL generation pipeline GA with other metrics, to analyze their relationship and investigate whether per- formance improvements across different routers simply come from increased selection of advanced methods or indeed stem from effective routing decisions.
As shown in Figure 5, obviously, higher allocation of GA is positively correlated with increased average token consumption. However, the EX performance exhibits a different pattern. Although the Qwen Cascading router assigns GA more frequently than any other router and incurs the highest token consumption, it does not rank at the top in terms of EX performance, resulting in a low TEP metric and indicating inefficiency. In contrast, RoBERTa Cascading and Qwen DPO routers select less GA than Qwen Cascading and Qwen SFT routers, yet they yield top performance and lead in TEP, proving their effectiveness, particularly the Qwen DPO router. This substantiates that performance gains are indeed driven by the effectiveness of routing strategies rather than merely deploying sophisticated methods more frequently.
6 Limitations and Future Directions
EllieSQL showcases better cost-efficiency via complexity-aware routing, yet limitations persist. First, labeled training data introduces an upfront cost for router implementation, although this expense becomes increasingly amortized with wider system adoption and EllieSQL presents great database transferability. Time and funding restricted us to three base methods and router categories, leaving room to explore additional SQL generation approaches (e.g,, MCTS-enhanced reasoning) and more advanced routing techniques.
Another limitation is the router's dependency on the upstream schema linking stage, where errors could propagate and may affect subsequent routing. We consider this a pragmatic design and validate the robustness of schema linking in our experiments, which presents 93% high recall while reducing the column by over 90% on average. More advanced schema linking methods could further strengthen downstream routing performance.
Also, introducing new SQL generation pipelines requires retraining new routers, which introduces incremental costs associated with data annotation and training. Future work could target data-efficient router training even training-free routing, while pursuing more scalable and generalizable routers for more sustainable and practical Text-to-SQL solutions.
Conclusion
In this paper, we propose EllieSQL to tackle the "elephant" in current leaderboard-driver Text-to-SQL research: the unsustainable computational costs. By dynamically assigning queries to different tiered SQL generation pipelines based estimated complexity, EllieSQL seeks to balance between performance and efficiency. We also introduce Token Elasticity of Performance (TEP), an cronomicinsspieed metric that evaluates the responsiveness of performance gains relative to token investments in SQL generation. Experiments demon- strate EllieSQL 's effectiveness: various router implementations significantly reduce token consumption while preserving competitive performance, even showing no performance compromise compared to consistently using the most advanced methods. Our findings establish the feasibility and potential of complexity-aware routing in Text-to-SQL, paving the way for further exploration of more advanced routing strategies. Ultimately, we hope EllieSQL can encourage the community to balance resource efficiency with performance on leaderboards, fostering efforts toward sustainable Text-to-SQL solutions that are both high-performing and cost-efficient for real-world adoption.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This paper is supported by NSF of China (62402409), Guangdong provincial project 2023CX10X008, Guangdong Basic and Applied Basic Research Founda- tion (202A11555105555, Guangzhou Basic and Applied Basic Research Foundation (2025A04]3935), HKUST(GZZ) Red Bird M.Phil. Program, and Cuangzhou-HKUUS((GZ)) Joint Funding Program (2025A03)3714).
References
Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, and Wei Chen. Rsl-sql: Robust schema linking text-to-sql generation. arXiv preprint ar (ix::2411.000733 2024.
Lingjaa Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arviv preprint arXit:2305.05176, 2023.
Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çagatay Demiralp, and Michael Stonebraker. Beaver: an enterprise benchmark for text-to-sql. ar Xiv preprint ar Xiix:2409.000 00000,,,, 2024a.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. Routerdc: Query- based router by dual contrastive learning for assembling large language models. Advances in Neural Information Processing Systems, 77766635-6666 2024b.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-acte dilemma agentic tasks. ar arviv preprint arXtv:2502.08233, 2025.
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. In The 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representutions, 2024. URL https: http::///pppreeeeeeeeeemme. net www.:nnet/orummmdo222ffffff
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-sql empowered by large language models: benchmark evaluation. Proc. VLDB Endow., 1755)11332 January 2024. ISSN 2150-8097. doi: /114777//64122443442222
URL / 14778/3641200 3641221.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: /wwp///eeeeee//e///////ee/e//eee net/ Forum?id=-ZeeKeeFFYy.
Thorsten Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, Pp- 133-142, 2002.
Hideo Kobayashi, Wuwei Lan, Peng Shi, Shuaichen Chang, Jiang Guo, Henghui Zhu, Zhiguo Wang, and Patrick Ng. You only read once (yoro): Learning to internalize database knowledge for text-to-sql. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1889-1901, 2025.
Wenqiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re-examining the role of schema linking in text-to-sqL. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6943-6954, 2020.
11

---

Published as a conference paper at COLM 2025
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to sql: Are we fully ready? Proceedings of the VLDB Endowment, 17(11):33183332 2024a.
Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. Alpha-sql: Zero-shot text-to-sql using monte carlo tree search. arv preprint arXio:2502.17228, 2025.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024b.
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. survey of NL2SQL with large language models: Where are we, and where are we going? Xiv preprint ar Xrto:2408.05109, 2024.
Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. NL2SQL-BUGs: A benchmark for detecting semantic errors in NL2SQL translation, 2025. URL 2503. 11984.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1964- 1974, 2024.
Yuyu Luo, Xuedi Qin, Nan Tang, and Guoliang Li. Deepeye: Towards automatic data Visualization. In ICDE, pp. 101-112. IEEE Computer Society, 2018.
Mohammadhossein Malekpour, Nour Shaheen, Foutse Khomh, and Amine Mhedhbi. To- wards optimizing sql generation via Ilm routing. ar preprint ar arXiv:2411.0431 2024.
Alfred Marshall. Principles of economics. Springer, 2013.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrew/e wttpevwwwwwweefummneeeeee mett/forumld--sSSNNtttet
Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL httpps://ppereeeeee http:://ppeeeeeeeeeeeeeeeernnnna mett/foruumPicrt533xxs1..
Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gau- rav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Ô Arik. CHASE- SQL: Multi-path reasoning and preference optimized candidate selection in text-to- SQL. In The Thirteenth International Conference Learning Representations, 2025. URL https://openreviee/ net
Xuedi Qin, Yuyu Luo, Nan Tang, and Guoliang Li. Making data visualization more efficient and effective: survey. VLDB J., 29(1):93- 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728 03741,, 2023.
Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin
Saberi. Chess: Contextual harnessing for efficient sql synthesis. arXiv preprint axXi::2405.16755, 2024.
Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of nlp systems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11007-11021, 2022.
12

---

Published as a conference paper at COLM 2025
Yifan Wu, Lutao Yan, Yizhang Zhu, Yinan Mei, Jiannan Wang, Nan Tang, and Yuyu Luo. Boosting text-to-chart retrieval through training with synthesized semantic insights. arviv preprint ar 1ati::2505.00444 2025.
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. Self-supervised prompt optimization. CoRR, abs/ foss2502.06855, 2025.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3911-3921, 2018.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation. CoRR, abs/2410.10722, 2024.
Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians? The Thirty-eighi Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024. URL https: http:///ppnneeeeeneeeee wwtp:e/eeee//oommm2311/4///a22 8M.
13

---

Published as a conference paper at COLM 2025
A Appendix
A.1 Model Transferability
As mentioned in Section 5.1, given that the training set and development set in the Bird dataset are constructed from distinct databases, routers are challenged to generalize across different databases. Their performance, as discussed in Section 5.2, highlights the transfer- ability of EllieSQL. across diverse databases. To further investigate the model transferability of routers within the EllieSQL, we experiment by replacing the backbone model in the SQL generation task with -ouudde--hiiuu-2222-2 respectively, without re-training any of the existing routers. Due to financial constraints, examine three representative routers in our experiments.
and much larger opt-40-2024-08-06
As reported in Table 4, the original routers demonstrate effectiveness even after the back- bone model substitution, showcasing transferability. These routers significantly enhance the responsiveness of performance gains relative to token investment, while maintaining competitive performance levels.
Specifically, when using claude- 3-iee---haikee hu- thaiku-2024033 20240307 the backbone LLM, the RoBERTa Cascading and Qwen SFT routers are effective in this transferability analysis. They achieve overall performance close to the standalone GA and reduce token costs by 39.5% and 45.8% respectively. This leads to an improvement in Token Efficiency Performance (TEP) by 29.8% and 34.4%. Furthermore, when gpt-40-2024-08 EllieSQL, all three examined routers also demonstrate high effectiveness. In particular, the RoBERTa Cascading router and the Qwen DPO router reduce token consumption by 41% and 44%, respectively, without any performance sacrifice, which significantly improves the TEP metric, increasing it by more than 2.5 times. These findings highlight the adaptability and efficiency of the proposed EllieSQL framework.
is deployed as the backbone in
claude-3-haiku 2-0------hhiuu2222007777
gpt-1o-2024-08-06
Category
Method
EX(%)
T
TEPf61009
EX(%)
T
TEP1(1103)
Basic (Gp) Intermediate (Gy)
51.17
736.91
56.13
768.03
Base
51.76
12153.93
0.074
57.82
9357.91
0.269
Advanced (Gx)
54.37
13444.51
0,363
58.34
10360.15
0.315
RoBERTa SFT
52.48
6094.96
0.352
58.15
4426.60
0.755
Routing
RoBERTa Casc.
53.59
8136.75
0.471
58.87
6073.55
0.706
Qwen DPO
53.26
7792.88
0.427
59.39
5832.57
0.881
Table 4: Performance of EllieSQL. with -laude-3-haiku-222444 and apt-44-2022-10
as backbone model on Bird dev dataset. The bold and underlined results are the best and second best in each category, respectively.
A.2 Comparison with Oracle Labels
We also deploy the process in Figure 2 to the Bird development set obtain Oracle labels in evaluation. Note that Oracle labels are the most efficient SQL generation pipeline capable for each query, which are guidance labels instead of absolute ground truth. Figure shows the distribution of Oracle labels on the Bird development set.
We measure the agreement between the router's predictions and the Oracle labels, and compare the agreement with EX performance, as illustrated in Figure 6-A. Our analysis reveals weak monotonic relationship between agreement with Oracle labels and EX performance (with only negligible Kendall's 7). Regarding disagreements with Oracle, directing a complex query to a simple method more detrimental than directing a simple query to complex method. While the latter may just incur some token overhead, the former can significantly increase the risk of errors.
To formalize this, and Oracle labels. Disagreements above the diagonal (simple queries routed to complex pipelines) are less severe, whereas those below the diagonal (complex queries routed to
disagreement matrix C to represent the router's predictions
14

---

Published as a conference paper at COLM 2025
Agreement with Oracle
Upper Triangular Ratio
TEP
Agreement
EX
UTR
EX
Agreement
TEP
UTR
TEP
0.8
08
35
55
0.4.5
53 9.4-
53
02
T(Arr, EX)
0.00
92
TITRR EX)
Aggr, TEP)
0.52
0.3-
TUTRR TEP)
KNN RoBERTS SFT Owen SFT ROBERTO case Owen
Owen Pair Pairwse Owen DPO
KNN RoBERTs SFT Owen SFT ROBERTO Case Owen Case Gwen Painerse Owen DPD
KNN ROBERT SFT
RoBERTa case Qken Cast
Owen Pairwise Owen DPO
ROBERTs SFT
KNN
Owen SFT ROBERS Casc Owen Case Owen Painwis Owen DPO
Owen
Figure 6: Comparison of agreement with Oracle and Upper Triangular Ratio (UTR) in routing with EX and TEP.
simple pipelines) more detrimental, discussed above. Thus, the Upper Triangular Ratio (UTR) can reflect the "reliability" of a router, since high UTR indicates a reduced risk of under-processing complex queries. Formally,
Lig Σij Cij
UTR
where Cjj is the number of queries cell (i,)). While higher UTR correlates with reduced misdirection of complex queries to simple methods (thereby improving reliability), it is insufficient standalone metric for effectiveness of routing. For instance, assigning all queries to the most complex pipeline would yield a full UTR of 1, yet such a strategy constitutes trivial and ineffective routing mechanism.
Figures 6-A/C demonstrate that agreement with Oracle exhibits a partial correlation with TEP but fails to reflect EX performance. In contrast, Figures 6-B/D reveal that while UTR achieves moderate correlation with EX (Kendall's T = 0.68), it suffers from significant lim- itations in assessing routing effectiveness. Specifically, RoBERTA SFT and Qwen Cascading routers present the highest agreement and UTR, respectively, but they do not demonstrate top performance either overall EX or TEP. Therefore, naive agreement-ratio-bsse com- parisons inadequate for evaluating routing performance cost-efficiency, motivating the introduction of the PGR metric (Ong et al., 2025) and TEP introduced in Section 3.5.
Figure 8 presents the disagreement matrix of the Qwen DPO router. While it is more effective at discerning tasks suitable for allocation to Gm and GA, the matrix suggests that distinguishing GM poses greater challenge for the router, likely due to the limited number of examples (Table 1 and Figure 7) and relatively narrow performance gap between Gm and GA- Although best-performing router, Qwen-DPO, achieves decent UTR of 0.79, there still remains notable room for further reducing below-diagona disagreements in future work to enhance reliability. This suggests considerable potential for enhancing Text-to-SQL routers in future studies, thereby improving both the overall performance and cost-efficiency of the Text-to-SQL. system.
Oracle Label
G
218
Oracle
38.3%
Gg: 51.6% (792)
Gy
67
51.6%
Gw: 10.0% (154)
Ga
193
67
328
10.0%
G 38.3% (588)
G.
GM
G^
Assigned Label
Figure 7: Distribution of Oracle labels in eval- uation on Bird dev
Figure 8:
Disagreement matrix between Ora- cle labels and Qwen DPO Router Predictions
15

---

Published as a conference paper at COLM 2025
A.3 Prompt Template
A.3.1 Prompt for Schema Linking
Schema Linking Prompt
You are smart and responsible SQLite SQL expert. Assist in identifying the database tables and columns involved natural language queries.
### Instruction:
Your task is to analyze the provided database schema, comprehend the posed ques- tion, and leverage the hint identify which tables are needed to generate SQL query for answering the question. The returned JSON format must strictly adhere to the following specifications:
"tables": [
"table": "table name", "columns": ["relevant column 1", "relevant column 2",
},
}
Each relevant column must belong to its respective table, and the output JSON object must be wrapped in a code block using json Please note that each table and column comes with detailed description information and example values for reference.
### Database schema: (schema_str)
### User question: {query)
### Hint: (evidence)
A.3.2
Prompt for Basic SQL Generation Pipeline
Basic SQL Generation Pipeline
You are intelligent and responsible SQLite expert.
### Instruction:
You need to read the database schema to generate SQL query for the user question. The outputted SQL must be surrounded by .. "sl"" code block.
### Database Schema: {schema}
### Hint: fevidence)
### User Question: {query)
The outputted SQL must be surrounded by -* "*"sq"" -.. code block.
16

---

Published as a conference paper at COLM 2025
A.3.3 Prompt for Divide-and-Conquee Chain-of-Thougih
Divide Prompt
You smart and responsible SQLite SQL expert. Given database schema and question, users want to know the corresponding SQL query. Your task is to understand the database schema and question, and decompose the question into sub-questions so user can better understand it. Each sub-question is enclosed in <<>> Here is an example for reference:
### Example:
## Given the database schema: (example.datee www.aaaae.ccheeeeeee
## Question: (
## Decompose the Question into sub-questions, each sub-question enclosed in <<>>
Sub-question 1: << (sub question }}>>
Sub-question 2: << question 2 >>>>
Sub-question 3: (sb question 3}>>
### Your task: decompose the question into sub-questions.
## Given the database schema: {schema}
## Question: {query)
## Hint: fevidence)
## Decompose the Question into sub-questions, each sub-question is enclosed in
<<>>;
17

---

Published as a conference paper at COLM 2025
Conquer Prompt
You are smart and responsible SQLite SQL expert. Given a database schema and question, your tasks are:
1. Parse user questions: Use natural language processing (NLP) techniques to parse user questions and extract query requirements and conditions.
Analyze database schema: Based on the database schema, understand the fields and relationships of the table, and build the basic framework of the SQL query.
3. Check sample data: Analyze the data characteristics based on the first three rows of the table values to help determine how to construct query conditions and filter results.
Generate SQL query: Based on user questions, query requirements and conditions, database schema, and sample data, build complete SQL query.
5. Verification and optimization: Check whether the generated SQL query is logical and optimize it if necessary.
### Database Schema:
[schema]
### Examples: (examples)
### Question: {query)
### Hint: (evidence}
Please generate the corresponding SQL query. SQL must be surrounded by sql code block.
18

---

Published as a conference paper at COLM 2025
Assemble Prompt
You are smart and responsible SQLite SQL expert. Given a database schema and question, users want to know the corresponding SQL query.
### Instructions:
We have decomposed the main question into sub-questions, now your task is based the SQL querys for corresponding sub-questions, assemble the final SQL for the main question:
Understand the database schema and the main question;
2. Read and analyze each sub-question and corresponding SQL query;
Analyze the relationship between sub-questions and the main question in order to assemble them properly; Generate the final SQL for the main question and optimize it if needed.
### Database Schema: {schema}
### Main question: {query)
### Hint: (evidence)
### Sub-questions and corresponding output, including SQL querys and explanation: {subs}
Based on the SQL querys for corresponding sub-questions, generate the final SQL for the main question in the end of your response, SQL must be surrounded by """sql"" code block.
19

---

Published as a conference paper at COLM 2025
A.3.4 Prompt for Online Syuthesis Mechanism
Online Synthesis Mechanism Prompt
### Instruction:
You are SQLite SQL expert. Your job is to create {k} examples, where each example consists of question and SQL query to fetch the data for it. want each example to look like this, question input and SQL output pairs:
### Example:
"Question": "What's 's the description of the series code SM. POP. TOTL for Aruba? (Hints: Aruba is the name of the country where ShortName 'Aruba')
"SQL": "SELECT T2. Descriptii FROM Country INNER JOIN CountryNotes AS ON T1. CountryCye = T2. Countrycode WHERE T1.ShortNamm 'Aruba' AND Seriecooe SM. POP. TOTL'"
### Task:
You should generate examples that examine and showcase different aspects and relationships of the following table schemas, described in "Table creation statements". " Understand the database tables and their relationships. Understand the columns and their types and meanings to construct interesting examples.
Generate a mixture of SQL examples that include:
some simple SQL query examples without JOIN
some SQL query examples with aggregates, like COUNT
some simple SQL query examples with JOIN
some complex SQL query examples with nested JOIN
## Database Schema: (TARGET DATABASE SCHEMA)
Generate total of (k) examples. Only output the examples ('question input' and 'SQL. output' pairs), and each example can be separated by a new line.
20


Reviews:
[{"review_id": "P4gmUNgbfU", "reviewer": "Reviewer_b7ZP", "summary": "The paper targets the problem of high computational costs in systems using advanced LLM-based text-to-sql approaches, which hinder their practicality for real-world deployment. It proposes EllieSQL, a complexity-aware routing framework that assigns queries to SQL generation pipelines based on their estimated difficulty, using lightweight pipelines for simple queries and computationally intensive pipelines for complex ones, along with a new efficiency metric called Token Elasticity of Performance (TEP). Experiments show that EllieSQL reduces token usage by over 40% without compromising accuracy and more than doubles cost-effciency (TEP) compared to always using the most complex pipelines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper targets an important problem, while prior work has explored routing different LLMs for prompts, routing within Text-to-SQL pipelines remains under-explored. \n\nThe idea of routing based on the results of schema linking makes a lot of sense, as it provides a natural signal of query complexity that can guide the selection of appropriate generation pipelines.\n\nThe results seem promising, showing significant reductions in token usage without compromising accuracy.", "reasons_to_reject": "The routing decision relies heavily on schema linking outcomes, yet schema linking itself is a challenging task. This dependency introduces potential fragility, if schema linking fails or produces noisy results, the entire routing strategy may misclassify query complexity.\n\nThe paper does not compare against existing LLM routing methods such as RouteLLM, which could also offer competitive performance in this setting. For example, the baseline can simply treat queries as standard prompts and use RouteLLM, or treat queries plus table schema or samples (potentially filter with retrieval) as the input to RouteLLM. It thereby opens the question of how much EllieSQL actually improves over simpler or existing alternatives.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7gRW8wF1yQ", "reviewer": "Reviewer_8cMx", "summary": "This paper introduces EllieSQL, a novel complexity-aware routing framework for Text-to-SQL tasks that explicitly addresses the under-discussed challenge of computational inefficiency in current LLM-based systems. Instead of treating all queries equally, EllieSQL estimates the complexity of each natural language query and dynamically assigns it to one of three tiered SQL generation pipelines: Basic, Intermediate, or Advanced. These pipelines trade off between performance and token usage.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an often-neglected yet pressing challenge in Text-to-SQL systems—computational cost and inefficiency—which is a major barrier for real-world deployment. The focus on cost-effectiveness adds significant practical value to the field.\n2. EllieSQL introduces a complexity-aware routing mechanism that dynamically directs queries to different SQL generation pipelines based on estimated difficulty. This pipeline-level routing is novel and distinct from existing model-level routing approaches.\n3. The proposed Token Elasticity of Performance provides an insightful and principled way to measure the cost-efficiency tradeoff, complementing traditional accuracy metrics and encouraging more sustainable model use.\n4. Experiments on the Bird benchmark show that EllieSQL—especially with the Qwen DPO router—achieves significant token reduction without sacrificing performance, confirming both its effectiveness and efficiency.", "reasons_to_reject": "The router in EllieSQL is tightly coupled with specific downstream SQL generation pipelines, making the approach difficult to generalize or directly apply to other Text-to-SQL systems. Adapting the framework to new tasks or models requires retraining the router, which introduces non-trivial annotation and engineering costs. While a similar automatic labeling strategy could be reused, many real-world Text-to-SQL scenarios may lack sufficient training data to support this. This limits the method’s modularity and practical applicability. A more generalizable or end-to-end trainable routing design would significantly improve usability.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CdfFY1mF3", "reviewer": "Reviewer_npNW", "summary": "This submission introduces a complexity-aware routing framework that assigns natural language queries to suitable text2sql generation models according to their complexity, with an aim to balance the system performance and inference costs. They also introduce a token elasticity of performance metric to measure the efficiency of generation, which measures how responsive one variable is to changes in another. The submission conducts experiments on the BIRD dataset with Roberta and Qwen2.5-0.5B as the baselines and foundation models for training routers, and gpt--4o-mini as the foundation model for SQL generation. Experiments show that the routing method can achieve a competing method compared with the direct SQL generation, but are more efficient in terms of generated tokens.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is well-motivated and the paper is generally well-written.\n2. The performance on cost-efficiency is greatly improved.", "reasons_to_reject": "1. It seems that the method can be applied to a larger scope of the application, including reasoning tasks. While I understand the choice of the task may be of the authors' interests, I still think the scope is somehow limited under the theme of COLM if the authors are focusing on the specific text2sql task, instead of the general cost-efficiency of LLMs on more tasks.\n2. Having said that, the experiments only focus on one dataset, and mainly one LLM (qwen2.5-0.5B) for the experiments. More datasets should be included. Also, since the cost-efficiency mainly focuses on the tokens that are generated instead of model sizes, larger LLMs should be included for comparison.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DyQpMIizkL", "reviewer": "Reviewer_367x", "summary": "This paper presents a cost-efficient Text2SQL system with complexity-aware routing, specifically, this system consists of three stages: 1) schema linking stage to collect sufficient information (e.g. NLQ, filtered schema) for query complexity estimation; 2) routing stage to decide the query difficulties: basic, intermediate or advanced; 3) SQL generation pipelines to translate NLQ into the final SQL query. To study the effectiveness of different routers, this paper implemented three different methods: classification-based router, cascading router and preference learning-based router. By conducting experiments on BIRD benchmark, the proposed system shows comparable Text2SQL performance but much better token efficiency.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed problem is very valuable and practically useful, not all NLQs share the same complexity, it is definitely needed to design a system to balance well between accuracy and inference cost.\n2. Three router implementations shows novelty, the performance on the BIRD benchmark also demonstrates the effectiveness.", "reasons_to_reject": "1. It's unclear about the router effectiveness on out-of-domain Text2SQL datasets, for example, we can evaluate the system on Spider related datasets (e.g. Spider, Spider-syn, Dr. Spider).\n2. Some Text2SQL baselines (e.g. DIN-SQL [1] ) also have internal routing mechanism, for example, DIN-SQL [1] dynamically classifies query into EASY, NON-NESTED, or NESTED, then select different prompting mechanisms for SQL generation. It will be great if this paper can do a comparison with these works.\n3. In addition to accuracy and token efficiency, in practical Text2SQL system, people also care about the latency, it's not clear about latency overhead for these router implementations.\n\n\n[1] DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper describes an efficiency-improving method for text-to-SQL processing in which a router decides whether a question can be answered by a lightweight model or requires a more expensive LLM call. Experiments, including additional results provided in response to reviewer concerns, demonstrate the effectiveness of the approach. Analysis addresses key questions about trade-offs in different variants of the approach.

Overall, this is a novel approach with positive results. The key reviewer questions have been addressed in the discussion period and adding the new results to the paper will strengthen it significantly.

Some particularly important points raised should be addressed in the final version of the paper:

- Add the new results presented in the discussion period and discuss them.
- Expand the discussion of other routing methods in section 2 to clarify where there is no direct comparison with RouteLLM (for example).
- Discuss the reliance on schema linking in the limitations section.
- Discuss other text-to-SQL methods that have some form of routing further, to distinguish this approach from the prior work.
Decision: Accept