Yathi28/llama-3.2-1b-text-to-sql

Llama 3.2 1B -- Text-to-SQL (QLoRA Fine-tuned)

"I fine-tuned a real LLM on my 6GB laptop GPU, for free, in ~12 minutes."

This model converts plain English questions into SQL queries based on a given database schema. Fine-tuned from unsloth/Llama-3.2-1B-Instruct using QLoRA (4-bit quantization + LoRA adapters).

---

What It Does

Input  (English): "Show top 5 highest paid employees"
Input  (Schema):  CREATE TABLE employees (id INT, name TEXT, salary REAL, dept TEXT)

Output (SQL):     SELECT name, salary FROM employees
                  ORDER BY salary DESC
                  LIMIT 5;

---

Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

MODEL_BASE = "unsloth/Llama-3.2-1B-Instruct"
ADAPTER    = "Yathi28/llama-3.2-1b-text-to-sql"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer  = AutoTokenizer.from_pretrained(MODEL_BASE)
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_BASE,
    quantization_config=bnb_config,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

PROMPT = """Below is a SQL question with its database schema.
Write the correct SQL query.

### Question:
{question}

### Schema:
{schema}

### SQL:
"""

def generate_sql(question, schema):
    prompt = PROMPT.format(question=question, schema=schema)
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    input_len = inputs["input_ids"].shape[1]
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=150,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
            repetition_penalty=1.1,
        )
    new_tokens = outputs[0][input_len:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

sql = generate_sql(
    question="Show the top 5 highest paid employees",
    schema="CREATE TABLE employees (id INT, name TEXT, salary REAL, department TEXT);"
)
print(sql)

---

Training Details

Parameter	Value
Base Model	unsloth/Llama-3.2-1B-Instruct
Technique	QLoRA (4-bit NF4 + LoRA rank 16)
Dataset	gretelai/synthetic_text_to_sql
Samples Used	5,000 out of 100,000+
Training Steps	200
Starting Loss	1.2786
Final Loss	0.6654
Training Time	~12.5 minutes
Hardware	NVIDIA GeForce RTX 3050 6GB Laptop GPU
Batch Size	1 x 8 gradient accumulation = effective 8
Learning Rate	2e-4 with linear decay
Max Seq Length	1024 tokens
Precision	bfloat16
Optimizer	adamw_8bit

LoRA Configuration

Parameter	Value
Rank (r)	16
Alpha	16
Trainable Params	11,272,192 (0.90% of 1.2B)
Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

---

Loss Curve

Step  10:  1.2786
Step  20:  0.7795
Step  50:  0.6632
Step 100:  0.6014
Step 150:  0.5782  <- Best
Step 200:  0.6031  <- Final

---

Limitations

• Trained for only 200 steps (0.32 epochs) -- more steps would improve quality
• Works best with CREATE TABLE style schemas
• English questions only
• Learning/demonstration project -- not tested for production use

---

Author

Yatheesh Pateel

• HuggingFace: Yathi28
• GitHub: yatheeshpateel

---

Framework Versions

• PEFT 0.13.2
• Transformers >= 4.40.0
• PyTorch 2.5.1+cu121
• bitsandbytes >= 0.43.0