Model Card for Llama3.2-SQL-1B

Model Details

This model is a fine-tuned version of Llama3.2-1B-Instruct, optimized for text-to-SQL generation tasks. It was trained on the gretelai/synthetic_text_to_sql dataset, which contains synthetic natural language questions and their corresponding SQL queries across a variety of domains.

The model learns to:

• Understand natural language instructions.
• Generate syntactically correct and context-aware SQL queries.
• Interpret structured schema information when included in the prompt.

Model Description

• Developed by: Rustam Shiriyev
• Model type: Instruction-tuned model on Text2SQL data
• Language(s) (NLP): English
• License: Llama3.2
• Finetuned from model: unsloth/Llama3.2-1B-Instruct

Uses

Direct Use

• Natural Language to SQL translation
• Educational or research applications
• Lightweight inference for SQL query generation on small-scale tasks or apps

Bias, Risks, and Limitations

• May not handle deeply nested or complex joins in SQL.

How to Get Started with the Model

from huggingface_hub import login
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

login(token="")  

tokenizer = AutoTokenizer.from_pretrained("unsloth/Llama3.2-1B-Instruct",)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama3.2-1B-Instruct",
    device_map={"": 0}, token=""
)

model = PeftModel.from_pretrained(base_model,"Rustamshry/Llama3.2-SQL-1B")


question = "What are the vehicle safety testing organizations that operate in the UK and France?"

context = 
"""
CREATE TABLE SafetyOrgs (name VARCHAR(20), country VARCHAR(10)); 
INSERT INTO SafetyOrgs (name, country) VALUES ('Euro NCAP', 'UK'); 
INSERT INTO SafetyOrgs (name, country) VALUES ('ADAC', 'Germany'); 
INSERT INTO SafetyOrgs (name, country) VALUES ('UTAC', 'France');
"""

instruction = (
    "You are a skilled SQL assistant."
    "Using the given database context, generate the correct SQL query to answer the question.\n\n"
    f"Context: {context.strip()}"
)

prompt = (
    f"### Instruction:\n{instruction}\n\n"
    f"### Question:\n{question}\n\n"
    f"### Response:\n"
)

input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **input_ids, 
    max_new_tokens=2048
)

print(tokenizer.decode(outputs[0]),skip_special_tokens=True)

Training Details

Training Data

• Dataset: gretelai/synthetic_text_to_sql which consists of 100,000 synthetic examples of natural language questions paired with corresponding SQL queries and explanations.

Training Procedure

The model was fine-tuned using the Unsloth and LoRA.

• LoRA rank: 8
• Aplha: 16

Training Hyperparameters

• batch size:8,
• gradient accumulation steps:4,
• optimizer:adamw_torch,
• learning rate:2e-5,
• warmup_steps:100,
• fp16:True,
• epochs:2,
• weight_decay:0.01,
• lr_scheduler_type:linear

Speeds, Sizes, Times [optional]

• Training time: 8 hour
• Speed: 0.22 steps/sec

Results

• Final Loss: 1.42 >> 0.48

Framework versions

• PEFT 0.14.0