AlioLeuchtmann/qwen-3.5-sql-27B-Distill-2B

Qwen3.5-2B · StructCoT SQL (27B-Distilled)

A 2B-parameter text-to-SQL model that matches 70B-class systems on the official BIRD benchmark.

Qwen3.5-2B fine-tuned for SQLite text-to-SQL through execution-verified structured chain-of-thought distillation from a Qwen 27B teacher. Evaluated officially by the BIRD team on the hidden test set.

• No reinforcement learning — no GRPO/PPO/DPO stage; the entire gain comes from execution-verified rejection-sampled distillation
• No test-time thinking — non-thinking decoding with a compact structured analysis (~500 output tokens), not long reasoning chains;

Official BIRD test-set results

Evaluated by the BIRD team (majority@7 self-consistency, single model, single A100):

metric	simple (949)	moderate (555)	challenging (285)	total (1789)
Execution Accuracy (EX)	77.13	62.70	50.88	68.47
Soft F1	77.46	63.87	53.41	69.41
R-VES	69.25	55.93	46.47	61.49

Size-for-score context (official BIRD test EX, from the public leaderboard)

model	size	test EX
Arctic-ExCoT-70B (Snowflake)	70B	68.53
this model	2B	68.47
Arctic-ExCoT-32B (Snowflake)	32B	68.19
Qwen3-Coder-480B-A35B	480B (MoE)	68.14
OmniSQL-7B	7B	67.97
AskData + GPT-4o (pipeline)	API	67.41
OmniSQL-32B	32B	67.05
Claude 4.5 Sonnet	frontier API	66.85
XiYanSQL-QwenCoder-32B	32B	64.65
GLM-4.7	frontier	62.94
DeepSeek-R1	671B (MoE)	60.93
Kimi-K2-Thinking	~1T (MoE)	59.87

At 2B parameters this model ties a 70B specialist, and outperforms frontier general-purpose LLMs — including Claude 4.5 Sonnet, DeepSeek-R1, and Kimi-K2-Thinking — as well as GPT-4o-based multi-stage pipelines and every open text-to-SQL model up to 32B listed below it. (Frontier-LLM rows are their official BIRD baseline entries; this model is a task-specialized fine-tune with self-consistency @7, declared as "Few" on the single-model track.)

Why this matters: distillation as a path to expert small models

This model is evidence for a broader thesis: in specialized domains, knowledge distillation can compress most of a large model's task competence into a model orders of magnitude smaller — here, ~93% of a 27B teacher inside 2B parameters, with no reinforcement learning and no test-time reasoning chains. The key enabler is that text-to-SQL is a verifiable domain: every candidate trace can be executed against the real database and checked againsining corpus can be filtered to contain only demonstrably correct reasoning. Wherever such a verifier exists — SQL execution, compckers, simulators — the same recipe applies: sample the teacher broadly, keep only what provably works, and fine-tune small.

The practical upside is significant. A 2B spsumer GPU (or CPU/edge hardware), answers in a few hundred tokens instead of thousand-token thinking chains, keeps data on-premises, ander API call — while, on its domain, outperforming general-purpose models hundreds of times its size. General frontier models remafor a fixed, well-defined task, this result suggests the efficient frontier is not a bigger generalist, it is a small model taugh

BIRD dev results (dev_20251106, execution accuracy, set-based comparison)

decoding	EX
greedy	56.89
majority@7 (1 greedy + 6 samples @ t 0.6)	61.40
pass@7 (oracle)	70.67

How it was trained

Every training target is an execution-verified chain-of-thought trace: a trace only enters the corpus if its final SQL, executed against the real database, reproduces the gold result set. No unverified text is ever trained on.

Stage 1 — Distillation SFT. 50,572 structured-CoT traces rejection-sampled from a Qwen 27B teacher over BIRD train, Spider, and synthetic schema corpora (multiple sampling rounds per item; longest correct trace kept). Each trace follows a 7-section template: question/hint mapping → schema selection → join path → filters → aggregation → edge checks → final checklist + SQL. 3 epochs, lr 2e-5, effective batch 32, NEFTune 5, bf16.

Stage 2 — Column-description anneal. BIRD databases ship per-column descriptions (database_description/*.csv); the schema prompt is augmented with them as comment blocks, and the model is annealed for 1 epoch at lr 3e-6 (cosine→0) on BIRD-train traces re-paired with these enriched prompts, plus 10% original-format replay. This stage alone adds ≈ +2.5 EX.

How to use it

The model expects the OmniSQL-style prompt (schema DDL + example rows + question + hint) and answers with a structured analysis ending in a fenced SQL block — always extract the last ```sql fence. Generate in non-thinking mode.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "AlioLeuchtmann/qwen-3.5-sql-27B-Distill-2B"
tok = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, dtype="bfloat16")

prompt = build_omnisql_prompt(schema_ddl, question, hint)  # see repo predict script
chat = tok.apply_chat_template([{"role": "user", "content": prompt}],
                               tokenize=False, add_generation_prompt=True,
                               enable_thinking=False)

# greedy answer
out = llm.generate([chat], SamplingParams(temperature=0.0, max_tokens=2048))

# full benchmark quality: majority@7 — add 6 samples at t=0.6 and vote by
# execution-result equality (see the BIRD submission's predict_maj7.py)

Tips for best results:

• Include 3 example rows per table in the schema block.
• If the database has database_description/*.csv files, inject the column descriptions as comments — the model is trained to exploit them.
• Majority@7 voting by executed result set gives ≈ +5 EX over greedy.

Intended use & limitations

Built for SQLite text-to-SQL (BIRD-style analytical questions with evidence hints). Not instruction-tuned for general chat. Performance on the challenging tier (50.9 EX) still trails large models — complex multi-step reasoning remains the 2B's bound.

Acknowledgements

Built on Qwen3.5-2B with a Qwen 27B teacher. Evaluated by the BIRD team (bird-bench.github.io). Developed by Alio Leuchtmann.