AbdiSQL-1.0: Domain-Specialized Small Language Model for Text-to-SQL

---

Executive Summary

AbdiSQL-1.0 is an instruction-tuned, domain-expert Small Language Model (SLM) engineered to translate complex natural language questions into precise, executable SQLite queries based on provided database schemas.

Trained as Model #1 of the ABDI Platform (a 15-model ecosystem of specialized lightweight models), AbdiSQL proves that targeted domain adaptation can achieve high-fidelity code generation on consumer-grade and constrained hardware without requiring massive parameter overhead or multi-GPU server infrastructure.

---

Model Specifications

Attribute	Specification
Model Name	AbdiSQL-1.0
Model Type	Causal Decoder-Only Transformer (Instruction-Tuned)
Base Model	Qwen/Qwen2.5-Coder-1.5B-Instruct
Target Dialect	SQLite
Total Parameters	1.54 Billion (1,543,714,816)
Trainable Parameters	1,089,536 (0.0705% of base parameters via LoRA)
Precision	16-bit Floating Point (float16 merged release)
Context Length	Up to 32,768 tokens (optimized for schemas with multi-table DDL)
Training Hardware	1x NVIDIA Tesla T4 GPU (16 GB VRAM) on Google Colab
License	Apache 2.0

---

Intended Use & Domain Scope

Primary Use Cases

• Natural Language to SQL Translation: Converting English questions into standard SQLite queries.
• Database Schema Understanding: Accurately mapping entity mentions and relational logic to foreign keys, primary keys, and table columns.
• Embedded & Edge Analytics: Low-latency on-device SQL generation for desktop apps, microservices, and localized business intelligence agents.

Scope Boundaries (v1.0)

• In-Scope: Single-turn natural language queries against single or multi-table SQLite schemas involving JOIN, GROUP BY, ORDER BY, aggregates (COUNT, SUM, AVG), and subqueries.
• Out-of-Scope (v1.0): Multi-dialect auto-translation (Postgres/MySQL specific functions), DDL schema mutations (ALTER TABLE, DROP), database performance index tuning, and multi-turn interactive disambiguation.

---

Dataset & Training Data

AbdiSQL was fine-tuned on cross-database examples derived from the prestigious Spider Text-to-SQL Benchmark (Yale University):

• Cross-Database Diversity: 1,350 instruction pairs spanning over 160 distinct relational schemas across varied domains (e.g., healthcare, finance, academic records, sports, logistics).
• Zero Schema Leakage: Training and evaluation splits are partitioned strictly by database schema (db_id), ensuring the model is evaluated on unseen database structures rather than memorized tables.
• Prompt Formulation: Every sample formatted using the Qwen-2.5 ChatML template:
  • System Prompt: Sets role and target dialect boundaries.
  • User Input: Database DDL / schema definition paired with the natural language query.
  • Assistant Target: Clean, syntactically valid SQLite code blocks.

---

Training Methodology

AbdiSQL-1.0 was trained using QLoRA (Quantized Low-Rank Adaptation) in 4-bit precision, maximizing parameter efficiency and stability on a single Tesla T4 GPU.

Hyperparameters

Hyperparameter	Value	Description / Rationale
Quantization	4-bit NormalFloat (NF4)	Double quantization enabled to compress base weights
Compute Precision	float16 / float32	Explicit FP32 casting for trainable LoRA layers on Turing architecture
LoRA Rank ($r$)	8	Balances representation capacity with small checkpoint footprint
LoRA Alpha ($\alpha$)	16	Standard $2 \times r$ scaling factor
LoRA Dropout	0.05	Prevents over-specialization on training schemas
Target Modules	Attention & MLP projections	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Epochs	2	Sufficient convergence across the 1,350 instruction samples
Learning Rate	1e-4	Tuned with Cosine annealing schedule
Warmup Ratio	0.03	Smooth gradient stabilization in initial steps
Optimizer	paged_adamw_8bit	Page-locked host memory prevents CUDA OOM spikes
Weight Merging	CPU RAM Offload	Full FP16 merge performed in system RAM to avoid VRAM allocation traps

---

Empirical Benchmark & Evaluation

Evaluation was performed using Execution Accuracy—the gold standard metric where generated SQL queries are executed against a live SQLite engine and their output record sets are strictly compared against ground-truth execution results (not superficial string matching).

Results on Held-Out Test Set (150 Unseen Cross-Database Queries)

Metric	Zero-Shot Base Model (Qwen2.5-Coder-1.5B)	AbdiSQL-1.0 (Fine-Tuned)	Delta
Execution Accuracy	51.33% (77 / 150)	54.67% (82 / 150)	+3.34%
Syntax Validity Rate	64.00%	69.33% (104 / 150)	+5.33%
Spider Dev Split Accuracy	43.00%	47.00% (47 / 100)	+4.00%

Key Observations

1. Schema Grounding: Fine-tuning significantly eliminated hallucinated column and table names that the base model occasionally assumed from common SQL naming conventions.
2. SQLite Dialect Alignment: Greatly reduced SQLite-incompatible function usage (e.g., using STRFTIME instead of Postgres-specific date truncation).
3. Execution Robustness: Higher percentage of error-free syntax execution on first-pass generations.

---

Quick Start / How to Use

1. Standard Inference with Hugging Face transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "abdulhayykhan/AbdiSQL-1.0"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Define schema and your question
schema = """
CREATE TABLE departments (
    dept_id INTEGER PRIMARY KEY,
    dept_name TEXT NOT NULL
);

CREATE TABLE employees (
    emp_id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    salary INTEGER,
    dept_id INTEGER,
    FOREIGN KEY (dept_id) REFERENCES departments(dept_id)
);
"""
question = "Find the name and salary of all employees in the 'Engineering' department who earn more than 75000."

# Format prompt using model chat template
messages = [
    {
        "role": "system",
        "content": "You are AbdiSQL, a domain-expert language model specialized in generating precise SQLite queries from database schemas and questions."
    },
    {
        "role": "user",
        "content": f"{schema.strip()}\n\nQuestion: {question}"
    }
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        temperature=0.0,
        pad_token_id=tokenizer.eos_token_id
    )

generated_sql = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
print("Generated SQLite Query:\n", generated_sql)

---

2. Low-Resource Deployment (4-bit via bitsandbytes in < 2GB VRAM)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "abdulhayykhan/AbdiSQL-1.0"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

---

3. Edge / Local CPU Deployment via GGUF & llama.cpp

AbdiSQL-1.0 can be easily converted to GGUF format for real-time offline CPU execution using llama.cpp, Ollama, or LM Studio:

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
pip install -r requirements.txt

# Convert downloaded HF weights to 4-bit GGUF
python convert_hf_to_gguf.py path/to/AbdiSQL-1.0 --outfile abdisql-1.0-q4_k_m.gguf --outtype q4_k_m

---

Limitations & Best Practices

• Security & Read-Only Execution: Like all code-generating LLMs, queries generated by AbdiSQL should never be executed directly on production databases with write permissions. Always execute generated queries in a sandboxed, read-only session (PRAGMA query_only = ON; in SQLite).
• Ultra-Complex Joins: Queries requiring 4+ table joins or recursive CTEs may occasionally require human verification or query refinement.
• Dialect Specificity: Optimized specifically for SQLite syntax. Dialect-specific keywords from SQL Server, Oracle, or PostgreSQL are not guaranteed to be compatible.

---

Citation & Acknowledgements

If you use AbdiSQL-1.0 in your research or application, please cite:

@misc{abdisql2026,
  author = {Abdul Hayy Khan},
  title = {AbdiSQL-1.0: A Domain-Specialized Small Language Model for Text-to-SQL},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/abdulhayykhan/AbdiSQL-1.0}}
}

Special acknowledgment to:

• The Yale LILY Lab for the Spider Benchmark.
• The Qwen Team (Alibaba Cloud) for the excellent Qwen2.5-Coder foundation series.
• The ABDI Platform initiative for pioneering accessible domain-specialized SLMs.