Cyrax321/QwerySmith-1.1

QwerySmith 1.1 (Qwen3-4B Text-to-SQL)

QwerySmith 1.1 is a specialized, production-ready Text-to-SQL model based on unsloth/Qwen3-4B, fine-tuned using QLoRA via Unsloth.

Benchmark Results

Results

set	system	valid SQL	exact match	execution acc (95% CI)	scored
in_dist	base_zeroshot	98.5%	6.0%	67.2% (54.7% to 77.7%)	61/200
in_dist	base_fewshot	98.0%	7.0%	67.2% (54.7% to 77.7%)	61/200
in_dist	finetuned	98.0%	84.5%	88.5% (78.2% to 94.3%)	61/200
gretel_test	base_zeroshot	92.7%	26.0%	52.3% (46.7% to 58.0%)	298/300
gretel_test	base_fewshot	88.7%	26.3%	47.3% (41.7% to 53.0%)	298/300
gretel_test	finetuned	90.7%	32.0%	55.7% (50.0% to 61.2%)	298/300
heldout_sqale	base_zeroshot	78.7%	12.8%	50.0% (35.8% to 64.2%)	44/47
heldout_sqale	base_fewshot	70.2%	12.8%	40.9% (27.7% to 55.6%)	44/47
heldout_sqale	finetuned	80.9%	10.6%	45.5% (31.7% to 59.9%)	44/47
heldout_large_schema	base_zeroshot	61.0%	1.9%	21.8% (15.8% to 29.3%)	142/154
heldout_large_schema	base_fewshot	53.9%	2.6%	19.7% (14.0% to 27.0%)	142/154
heldout_large_schema	finetuned	55.8%	1.9%	17.6% (12.2% to 24.7%)	142/154

• in_dist: fine-tuned vs 3-shot base, execution-correct only on one side: 13 wins, 0 losses
• gretel_test: fine-tuned vs 3-shot base, execution-correct only on one side: 43 wins, 18 losses
• heldout_sqale: fine-tuned vs 3-shot base, execution-correct only on one side: 4 wins, 2 losses
• heldout_large_schema: fine-tuned vs 3-shot base, execution-correct only on one side: 10 wins, 13 losses

How to read this

• valid SQL: the query runs in SQLite without error.
• exact match: normalized string equality with the gold query. Punishes correct queries written differently.
• execution acc: predicted and gold queries return the same rows. Only items whose gold query returns at least one row are scored (the "scored" column). in_dist has no real data, so tables are filled with random rows seeded from the gold query's literals; that can occasionally make two different queries look equal. gretel_test/heldout_* use each example's own INSERT rows where present (see below).
• If the confidence intervals overlap, do not claim one system beats the other.
• SQLite is not Postgres/MySQL: a few correct queries in other dialects will be marked wrong.
• Base-model output is parsed leniently (code fences and chatter stripped) so it is not punished for formatting.
• v1.1: once a source is part of --mix, its "test" split (e.g. gretel_test) is no longer a generalization check -- it is near-distribution. Only the heldout_ sets (sources named in --heldout, which must never also appear in --mix) answer "does this help on SQL styles the model never trained on". Each named source gets its own heldout_ set -- compare them individually rather than averaging, since they test different failure modes (SQaLe: real-world schema noise; large_schema: schema-linking pressure, though only its small-schema tail survives the size filter here).
• gretel_test: 79% of items ship with their own INSERT rows; the rest use random filler rows.
• heldout_sqale: 0% of items ship with their own INSERT rows; the rest use random filler rows.
• heldout_large_schema: 0% of items ship with their own INSERT rows; the rest use random filler rows.

How to Use with Unsloth

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained("Cyrax321/QwerySmith-1.1", max_seq_length=2048, load_in_4bit=True)
FastLanguageModel.for_inference(model)

prompt = '''<|im_start|>system
You are a text-to-SQL assistant. Given a database schema and a question, reply with exactly one SQL query and nothing else.<|im_end|>
<|im_start|>user
Schema: CREATE TABLE employees (id INT, name VARCHAR, salary INT, department VARCHAR);
Question: What is the highest salary in the engineering department?<|im_end|>
<|im_start|>assistant
<think>
</think>
'''

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, use_cache=True)
print(tokenizer.batch_decode(outputs)[0])