qwen2.5-coder-1.5b-sql-lora-v2c

QLoRA adapter for Qwen/Qwen2.5-Coder-1.5B-Instruct fine-tuned on b-mc2/sql-create-context for text-to-SQL.

Config

• Base model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• LoRA: r=16, alpha=32, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training examples: 5000, held-out test: 100
• 1 epoch, batch 4 × grad-accum 4, LR 2e-4 cosine, warmup 5 steps
• 4-bit NF4 quantization with double-quant, bf16 compute
• Hardware: 1× NVIDIA T4 (Google Colab free tier)

Results (exact-match after normalization, 100 held-out examples)

Run	Baseline	Fine-tuned	Delta
V1 (Qwen2.5-1.5B-Instruct, r=16, α=32, 2500)	42%	61%	+19
V2C (Qwen/Qwen2.5-Coder-1.5B-Instruct, r=16, α=32, 5000)	35.0%	70.0%	+35.0

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         bnb_4bit_use_double_quant=True)
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-1.5B-Instruct",
                                            quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, "AmbikaSoni/qwen2.5-coder-1.5b-sql-lora-v2c")
tok = AutoTokenizer.from_pretrained("AmbikaSoni/qwen2.5-coder-1.5b-sql-lora-v2c")