README.md

---
base_model: unsloth/Llama-3.2-1B-Instruct
library_name: peft
license: llama3.2
datasets:
- gretelai/synthetic_text_to_sql
language:
- en
pipeline_tag: text2text-generation
tags:
- SQL
- Text-to-SQL
- SQL-generation
---

# Model Card for Llama3.2-SQL-1B


## Model Details

This model is a fine-tuned version of Llama3.2-1B-Instruct, optimized for text-to-SQL generation tasks.
It was trained on the **gretelai/synthetic_text_to_sql** dataset, which contains synthetic natural language questions and their corresponding SQL queries across a variety of domains.

The model learns to:
- Understand natural language instructions.
- Generate syntactically correct and context-aware SQL queries.
- Interpret structured schema information when included in the prompt.

### Model Description


- **Developed by:** Rustam Shiriyev
- **Model type:** Instruction-tuned model on Text2SQL data
- **Language(s) (NLP):** English
- **License:** Llama3.2
- **Finetuned from model:** unsloth/Llama3.2-1B-Instruct


## Uses


### Direct Use

- Natural Language to SQL translation
- Educational or research applications
- Lightweight inference for SQL query generation on small-scale tasks or apps

## Bias, Risks, and Limitations

- May not handle deeply nested or complex joins in SQL.

## How to Get Started with the Model

```python
from huggingface_hub import login
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

login(token="")  

tokenizer = AutoTokenizer.from_pretrained("unsloth/Llama3.2-1B-Instruct",)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Llama3.2-1B-Instruct",
    device_map={"": 0}, token=""
)

model = PeftModel.from_pretrained(base_model,"Rustamshry/Llama3.2-SQL-1B")


question = "What are the vehicle safety testing organizations that operate in the UK and France?"

context = 
"""
CREATE TABLE SafetyOrgs (name VARCHAR(20), country VARCHAR(10)); 
INSERT INTO SafetyOrgs (name, country) VALUES ('Euro NCAP', 'UK'); 
INSERT INTO SafetyOrgs (name, country) VALUES ('ADAC', 'Germany'); 
INSERT INTO SafetyOrgs (name, country) VALUES ('UTAC', 'France');
"""

instruction = (
    "You are a skilled SQL assistant."
    "Using the given database context, generate the correct SQL query to answer the question.\n\n"
    f"Context: {context.strip()}"
)

prompt = (
    f"### Instruction:\n{instruction}\n\n"
    f"### Question:\n{question}\n\n"
    f"### Response:\n"
)

input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **input_ids, 
    max_new_tokens=2048
)

print(tokenizer.decode(outputs[0]),skip_special_tokens=True)
```

## Training Details

### Training Data

- **Dataset**: gretelai/synthetic_text_to_sql which consists of 100,000 synthetic examples of natural language questions paired with corresponding SQL queries and explanations.

### Training Procedure

The model was fine-tuned using the Unsloth and LoRA. 

- LoRA rank: 8
- Aplha: 16

#### Training Hyperparameters

- batch size:8, 
- gradient accumulation steps:4,
- optimizer:adamw_torch,
- learning rate:2e-5,
- warmup_steps:100,
- fp16:True,
- epochs:2,
- weight_decay:0.01,
- lr_scheduler_type:linear

#### Speeds, Sizes, Times [optional]

- Training time: 8 hour
- Speed: 0.22 steps/sec

### Results

- Final Loss: 1.42 >> 0.48

### Framework versions

- PEFT 0.14.0