Cyrax321/QwerySmith-1.0

QwerySmith 1.0 (Qwen3-4B Text-to-SQL)

QwerySmith 1.0 is a specialized Text-to-SQL model based on unsloth/Qwen3-4B, fine-tuned using QLoRA with Unsloth on 10,000 text-to-SQL context pairs (b-mc2/sql-create-context).

📊 Benchmark Results

Results

set	system	valid SQL	exact match	execution acc (95% CI)	scored
in_dist	base_zeroshot	98.5%	6.0%	67.2% (54.7% to 77.7%)	61/200
in_dist	base_fewshot	99.5%	55.0%	60.7% (48.1% to 71.9%)	61/200
in_dist	finetuned	98.0%	87.5%	88.5% (78.2% to 94.3%)	61/200
external	base_zeroshot	92.7%	26.0%	52.3% (46.7% to 58.0%)	298/300
external	base_fewshot	89.7%	20.0%	47.3% (41.7% to 53.0%)	298/300
external	finetuned	93.0%	9.7%	34.6% (29.4% to 40.1%)	298/300

• in_dist: fine-tuned vs 3-shot base, execution-correct only on one side: 18 wins, 1 losses
• external: fine-tuned vs 3-shot base, execution-correct only on one side: 14 wins, 52 losses

How to read this

• valid SQL: the query runs in SQLite without error.
• exact match: normalized string equality with the gold query. Punishes correct queries written differently.
• execution acc: predicted and gold queries return the same rows. Only items whose gold query returns at least one row are scored (the "scored" column). in_dist has no real data, so tables are filled with random rows seeded from the gold query's literals; that can occasionally make two different queries look equal. external uses each example's own INSERT rows where present (see below).
• If the confidence intervals overlap, do not claim one system beats the other.
• SQLite is not Postgres/MySQL: a few correct queries in other dialects will be marked wrong.
• Base-model output is parsed leniently (code fences and chatter stripped) so it is not punished for formatting.
• external: 79% of items ship with their own INSERT rows; the rest use random filler rows.

🚀 How to Use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained("Cyrax321/QwerySmith-1.0", max_seq_length=2048, load_in_4bit=True)
FastLanguageModel.for_inference(model)

prompt = """<|im_start|>system
You are a text-to-SQL assistant. Given a database schema and a question, reply with exactly one SQL query and nothing else.<|im_end|>
<|im_start|>user
Schema: CREATE TABLE employees (id INT, name VARCHAR, salary INT, department VARCHAR);
Question: What is the highest salary in the marketing department?<|im_end|>
<|im_start|>assistant
<think>
</think>
"""

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256, use_cache=True)
print(tokenizer.batch_decode(outputs)[0])