qwen2.5-1.5b-sql-qlora-merged

Merged fp16 weights of Qwen/Qwen2.5-1.5B-Instruct, QLoRA-fine-tuned for text-to-SQL: given a CREATE TABLE schema and a natural-language question, emit a single SQLite query.

The LoRA adapter alone is at blaze-star/qwen2.5-1.5b-sql-qlora.

Results

Held-out test split, 1,000 examples never seen in training.

Model	Exact match	Token F1	Format compliance
Base, 0-shot (4-bit)	49.7%	0.925	26.7%
Base, 3-shot (4-bit)	52.3%	0.921	99.3%
QLoRA fine-tuned	74.8%	0.973	99.9%
Delta vs 0-shot	+25.1 pts	+0.049	+73.2 pts

Fine-tuning improved exact match by +25.1 points (49.7% -> 74.8%), a +51% relative gain.

Usage

The model expects the chat template with this system prompt — it was trained with it, and accuracy drops without it.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged", dtype=torch.float16, device_map="auto")
tok = AutoTokenizer.from_pretrained("blaze-star/qwen2.5-1.5b-sql-qlora-merged")

SYSTEM = ("You are a text-to-SQL engine. Given a SQLite schema and a question, reply with a "
          "single SQL query that answers the question. Output only the SQL query: no "
          "explanation, no comments, no markdown code fences.")

schema = "CREATE TABLE head (age INTEGER)"
question = "How many heads of the departments are older than 56?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: {question}\n\nSQL:"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=96, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
# SELECT COUNT(*) FROM head WHERE age > 56

Use greedy decoding (do_sample=False). Sampling hurts exact match on this task.

What the metric actually measures

Primary metric is normalized exact match against the dataset's reference SQL: lowercased, whitespace collapsed, " unified to ', backticks/brackets stripped, spacing normalized around operators and punctuation. It is strict — a semantically equivalent query that differs in alias naming (AS p vs AS T1) or literal quoting (= '15' vs = 15) counts as a miss.

That strictness is the point, but it must be read correctly: the base model already produces largely correct SQL content (token F1 0.925 before any training). Much of the headroom is conformance to this dataset's canonical SQL style, which is exactly what task-specific fine-tuning buys you. To keep that claim honest this project reports three separate baselines rather than one:

Baseline	Exact match	Format compliance	What it controls for
4-bit, 0-shot	49.7%	26.7%	matched conditions — same quantization the adapter trains on
4-bit, 3-shot	52.3%	99.3%	isolates output formatting from SQL convention
fp16, 0-shot	57.3%	99.4%	strongest untrained configuration

The 3-shot baseline is the important control. Three in-context examples raise format compliance to 99.3% — the model stops wrapping output in markdown fences almost entirely — yet exact match moves only to 52.3%. Formatting was therefore not the bottleneck, and gains above that line are genuine SQL-convention learning, not prompt-format cleanup.

Both models receive an identical prompt and identical output post-processing (fence stripping, leading-prose removal, first-statement extraction), so neither is advantaged by the harness. Secondary metrics: order-insensitive token F1 over SQL tokens (partial credit) and format compliance (fraction of raw generations that were already bare SQL).

Data and leakage control

b-mc2/sql-create-context — natural-language question + CREATE TABLE schema -> SQLite query.

The 78,577 raw rows are deduplicated on a SHA-1 of the normalized (question, schema) pair (4 exact duplicates dropped), shuffled with seed 42, and then test is carved off first, before val and train. Splits: 12,000 train / 750 val / 1,000 test.

Split disjointness is asserted at build time and recorded in data/split_report.json:

{"train_test_overlap": 0, "val_test_overlap": 0, "train_val_overlap": 0}

prepare_data.py raises if any of these is non-zero, so a leaking split cannot be trained on. The test split was used only by evaluate.py, never by train.py.

Training

Base model	Qwen/Qwen2.5-1.5B-Instruct
Method	QLoRA — frozen 4-bit NF4 base (double quant, bf16 compute) + LoRA adapters
LoRA	r=16, alpha=32, dropout=0.05, on q,k,v,o,gate,up,down_proj
Trainable params	18.5M of 1.56B (1.18%)
Optimizer	paged AdamW 8-bit, lr 0.0002, cosine schedule, 3% warmup, grad-clip 0.3
Effective batch	32 (16 x 2 accumulation)
Epochs	2
Max seq len	512 tokens (observed mean 125, max 254)
Loss	cross-entropy on the SQL completion only — prompt tokens masked to -100
Hardware	1x NVIDIA A100-SXM4-80GB
Wall time	11.7 min
Peak VRAM (training)	30.96 GB

Loss is computed only on the assistant turn, so the model is never rewarded for reproducing the schema or the question.

Quantization: latency, VRAM, and quality

Merged fp16 model re-quantized with bitsandbytes and benchmarked on the same A100. Latency is a single request generating exactly 64 tokens (20 runs after 3 warmups); quality is exact match on the first 300 test examples.

Precision	Weights VRAM	Peak VRAM	Latency (bs=1, 64 tok)	Decode tok/s	Batch-16 tok/s	Exact match
fp16	3.09 GB	3.3 GB	1862.4 ms	34.4	436.5	76.7% (n=300)
8bit	1.8 GB	2.1 GB	27963.2 ms	2.3	16.7	75.3% (n=300)
4bit	1.16 GB	1.54 GB	2320.7 ms	27.6	369.2	75.3% (n=300)

Limitations

• Single-table, synthetic-ish schemas. sql-create-context schemas are small CREATE TABLE statements derived from WikiSQL/Spider. Performance will not transfer directly to large multi-table production warehouses.
• Exact match is style-sensitive. A correct query written in a different but valid style scores zero. Token F1 is reported alongside for this reason.
• No execution-based evaluation. Queries are compared as strings, not run against a database, so semantic equivalence is undercounted.
• 4-bit inference costs accuracy. The base model loses ~8 points of exact match going from fp16 to 4-bit (57.3% -> 49.7%); see the quantization table for the fine-tuned model's own fp16/8-bit/4-bit spread.
• English only, and the model emits SQLite dialect.

Intended use

Converting natural-language questions into SQLite queries over small, explicitly-provided schemas — a component inside a larger system that supplies the schema and validates or sandboxes the generated query. Do not execute generated SQL against a production database without validation; the model can emit syntactically valid queries that are semantically wrong.

Training code

Full, reproducible pipeline: https://github.com/harshb20/qwen2.5-1.5b-sql-qlora

Citation

@misc{qwen25-1.5b-sql-qlora,
  title  = {QLoRA text-to-SQL fine-tune of Qwen2.5-1.5B-Instruct},
  author = {harshb20},
  year   = {2026},
  url    = {https://github.com/harshb20/qwen2.5-1.5b-sql-qlora}
}