PhuocNg9604/qwen2.5-coder-3b-text2sql-sft

Qwen2.5-Coder-3B-Instruct — Text-to-SQL SFT Adapter

This repository contains the LoRA adapter fine-tuned on top of Qwen/Qwen2.5-Coder-3B-Instruct for the Text-to-SQL task, using the Spider benchmark with targeted failure mining.

Model Description

• Base Model: Qwen/Qwen2.5-Coder-3B-Instruct
• Architecture: Causal Language Model with LoRA PEFT adapter
• Task: Cross-domain Text-to-SQL generation
• Dataset: Yale Spider benchmark (curated training split with failure mining)
• License: Apache-2.0

Training Configuration

Hyperparameters extracted from configs/sft.yaml:

Hyperparameter	Value
Method	QLoRA (4-bit base quantization)
LoRA Rank ($r$)	16
LoRA Alpha ($\alpha$)	32
LoRA Dropout	0.0
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Max Sequence Length	2048 tokens
Training Epochs	3
Batch Size	2 per device
Gradient Accumulation	8 (effective batch size = 16)
Optimizer	adamw_8bit
Learning Rate	1e-4 (cosine schedule, 5% warmup)
Loss Computation	Completion-only loss (train_on_responses_only on assistant SQL output)

Prompt Format

The model was trained using the Qwen-2.5 chat template with database schema serialized as CREATE TABLE statements with foreign key comments:

System Prompt:

You are a SQL expert. Given a database schema and a natural language question, write a valid SQL query that answers the question. Output only the SQL query with no explanation.

User Prompt:

Given the following database schema:
CREATE TABLE employees (id INT, name TEXT, department_id INT);
CREATE TABLE departments (id INT, name TEXT);
-- FK: employees.department_id -> departments.id

Question: Find the names of employees in the Engineering department.

SQL:

How to Use

import re
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "Qwen/Qwen2.5-Coder-3B-Instruct"
adapter_id = "PhuocNg9604/qwen2.5-coder-3b-text2sql-sft"

# 1. Load tokenizer and base model in float16
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Attach and merge LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()
model.eval()

# 3. Prepare messages matching the pipeline format
messages = [
    {
        "role": "system",
        "content": (
            "You are a SQL expert. Given a database schema and a natural language question, "
            "write a valid SQL query that answers the question. "
            "Output only the SQL query with no explanation."
        ),
    },
    {
        "role": "user",
        "content": (
            "Given the following database schema:\n"
            "CREATE TABLE stadium (Stadium_ID INT, Location TEXT, Name TEXT, Capacity INT);\n"
            "CREATE TABLE concert (concert_ID INT, concert_Name TEXT, Stadium_ID TEXT, Year TEXT);\n"
            "-- FK: concert.Stadium_ID -> stadium.Stadium_ID\n\n"
            "Question: What is the capacity of the stadium hosting the concert in 2024?\n\nSQL:"
        ),
    },
]

# 4. Generate SQL
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )

input_len = inputs["input_ids"].shape[1]
raw_output = tokenizer.decode(outputs[0][input_len:], skip_special_tokens=True).strip()

# 5. Extract SQL
clean_sql = re.sub(r"<think>.*?</think>", "", raw_output, flags=re.DOTALL).strip()
match = re.search(r"```(?:sql)?\s*(.*?)```", clean_sql, re.DOTALL | re.IGNORECASE)
final_sql = match.group(1).strip() if match else clean_sql

print("Generated SQL:\n", final_sql)