cameronervin1/qwen2.5-0.5b-sql-lora

Qwen2.5-0.5B-Instruct text-to-SQL LoRA

A LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct. Given a schema (CREATE TABLE statements) and a question, it writes a SQLite query.

Trained on 10,000 rows of b-mc2/sql-create-context for 1 epoch. r=16, alpha=32, all attention and MLP projections. 8.8M trainable parameters (1.75%).

Results (500 held-out questions, greedy decoding)

Model	Exact match	Executes in SQLite
Base	41.4%	92.4%
Base + LoRA	69.0%	97.2%

"Executes" means the query runs without error against the schema. The dataset has no table rows, so this does not check that the answer is correct.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base)
model = PeftModel.from_pretrained(model, "cameronervin1/qwen2.5-0.5b-sql-lora")

messages = [
    {"role": "system", "content": "You write SQLite queries. Given a database schema and a question, return only the SQL query. No explanation."},
    {"role": "user", "content": "Schema:\nCREATE TABLE head (name VARCHAR, age INTEGER)\n\nQuestion: How many heads are older than 56?"},
]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Code: https://github.com/cameronervin/lora-sql