Yash-2330/llama3.2-spider-text2sql-lora-adapter

Developed as part of research and experimentation in cross-domain Text-to-SQL generation using parameter-efficient fine-tuning

Llama 3.2 Spider Text-to-SQL LoRA Adapter

Overview

This repository contains a LoRA adapter fine-tuned on the Spider Text-to-SQL benchmark dataset using Meta Llama 3.2 3B Instruct.

The objective of this adapter is to improve the model's ability to translate natural language questions into SQL queries across multiple database schemas and domains. The adapter serves as a SQL reasoning foundation that can be further adapted to domain-specific databases.

---

Base Model

• meta-llama/Llama-3.2-3B-Instruct

---

Training Dataset

Spider Text-to-SQL Dataset

Spider is a large-scale cross-domain Text-to-SQL benchmark containing complex SQL queries across multiple databases and schemas.

The dataset includes:

• Multi-table joins
• Aggregations
• GROUP BY and HAVING clauses
• Nested queries
• Complex filtering conditions
• Cross-domain schema generalization

Training examples consist of:

Natural Language Question → SQL Query

Example:

Question: How many heads of departments are older than 56?

SQL: SELECT count(*) FROM head WHERE age > 56;

---

Training Configuration

This adapter was trained using:

• Meta Llama 3.2 3B Instruct
• PEFT LoRA
• QLoRA (4-bit NF4 quantization)
• Hugging Face Transformers
• TRL SFTTrainer

LoRA Configuration:

• Rank (r): 16
• Alpha: 32
• Dropout: 0.05

Target Modules:

• q_proj
• k_proj
• v_proj
• o_proj
• gate_proj
• up_proj
• down_proj

Training Parameters:

• Training Examples: 7000
• Validation Examples: 1034
• Epochs: 3
• Gradient Accumulation Steps: 4

---

Training Results

Validation Loss:

Epoch	Validation Loss
1	1.0098
2	1.0938
3	1.1983

Best validation performance was observed after Epoch 1.

The adapter was retained for further domain adaptation experiments and downstream Text-to-SQL research.

---

Research Context

This adapter is part of a multi-stage Text-to-SQL fine-tuning pipeline.

Stage 1:

• Spider Dataset

Stage 2:

• Domain-specific Olist Business Analytics Dataset

The goal is to evaluate the impact of domain adaptation on SQL generation performance compared to:

1. Base Llama 3.2 3B Instruct
2. Spider Fine-Tuned Adapter
3. Spider + Domain-Specific Fine-Tuned Adapter

---

Intended Use

This adapter can be used for:

• Natural Language to SQL conversion
• SQL query generation
• Database question answering
• Text-to-SQL research
• Educational projects
• Further fine-tuning on domain-specific SQL datasets

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct"
)

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct"
)

model = PeftModel.from_pretrained(
    base_model,
    "Yash-2330/llama3.2-spider-text2sql-lora-adapter"
)

---

Limitations

• SQL correctness is not guaranteed.
• Generated queries should be reviewed before execution.
• Performance may vary across unseen schemas.
• The model does not execute SQL queries and only generates them.
• The adapter was trained on Spider and may require domain-specific adaptation for production databases.

---

Future Work

• Domain-specific Text-to-SQL adaptation
• Business analytics query generation
• Improved schema linking
• Multi-turn database question answering
• Execution-guided SQL generation

---

License

This adapter follows the licensing requirements of:

• Meta Llama 3.2
• Spider Dataset

Users must comply with the license terms of the underlying base model.