AkshAIML/tinyllama-sql-lora-adapter

TinyLlama SQL LoRA Adapter

A LoRA fine-tuned adapter for TinyLlama-1.1B-Chat-v1.0 that converts (table schema, question) pairs into clean, executable SQL queries.

The primary goal of this fine-tuning is to teach the model a consistent SQL-only response format rather than teaching SQL syntax from scratch. The base TinyLlama model already has general knowledge of SQL, but may respond conversationally by explaining the query, adding Markdown code fences, or providing additional commentary. This adapter is trained to produce a single SQL statement without unnecessary text.

Model Details

Model Description

• Developed by: AkshAIML
• Model type: LoRA adapter for a causal language model
• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning method: LoRA (Low-Rank Adaptation)
• Task: Text-to-SQL generation
• Primary input: Table schema + natural-language question
• Primary output: SQL query
• Languages: English
• License: See the base model license and repository information
• Adapter: AkshAIML/tinyllama-sql-lora-adapter

Model Sources

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Dataset: b-mc2/sql-create-context
• Project repository: Aksh-dev-code/Text-to-SQL

Uses

Direct Use

The adapter can be used for generating SQL queries from natural-language questions and database schemas.

Example use cases include:

• Natural-language database querying
• Text-to-SQL demonstrations
• SQL generation experiments
• Learning and experimenting with LoRA fine-tuning
• Prototyping AI-powered database interfaces

Downstream Use

The model can be integrated into applications where a user provides a database schema and natural-language question and the system converts the request into SQL.

For production systems, generated SQL should be validated before execution.

Out-of-Scope Use

This model is not intended to be used as a production-grade autonomous database agent without additional validation.

It should not be assumed that every generated SQL query is semantically correct, safe, or executable against an arbitrary database.

The model should not be given unrestricted database execution privileges without appropriate security controls.

Bias, Risks, and Limitations

This model is a demonstration of a LoRA fine-tuning workflow for Text-to-SQL rather than a production-ready SQL generation system.

Known limitations include:

• The model was fine-tuned on 3,000 examples rather than the full available dataset.
• Training was limited to 1 epoch.
• Exact-match evaluation is strict and may mark semantically equivalent SQL queries as incorrect.
• The model may generate syntactically or semantically incorrect SQL for unfamiliar schemas or questions.
• SQL generated by the model should be validated before execution.
• The model may not generalize equally well across different SQL dialects.
• Greedy decoding was used for reproducibility rather than production optimization.
• Execution-based evaluation was not performed in this demonstration.

Recommendations

Users should treat generated SQL as model output that requires validation.

For production applications, recommended safeguards include:

1. Validate generated SQL before execution.
2. Restrict database permissions using least-privilege access.
3. Prevent destructive operations unless explicitly authorized.
4. Consider SQL parsing or query validation before execution.
5. Evaluate using execution-based metrics in addition to exact-match accuracy.
6. Test the model on schemas and SQL dialects representative of the target application.

How to Get Started

The adapter can be loaded on top of the original TinyLlama model using transformers and peft.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
adapter_model = "AkshAIML/tinyllama-sql-lora-adapter"

tokenizer = AutoTokenizer.from_pretrained(adapter_model)

base = AutoModelForCausalLM.from_pretrained(
    base_model,
    dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base,
    adapter_model,
)

model.eval()

The model expects an input containing a database schema and a natural-language question formatted according to the training prompt/template.

Training Details

Training Data

The model was fine-tuned using 3,000 examples selected from:

b-mc2/sql-create-context

The dataset contains natural-language questions paired with database schemas and corresponding SQL queries.

A separate held-out evaluation split was retained for evaluation.

Training Procedure

The training pipeline consisted of:

1. Loading the TinyLlama-1.1B-Chat base model.
2. Formatting the dataset using TinyLlama's chat template.
3. Attaching LoRA adapters.
4. Fine-tuning using supervised fine-tuning (SFTTrainer).
5. Selecting the best checkpoint based on evaluation loss.
6. Evaluating the fine-tuned model on held-out examples.
7. Saving only the LoRA adapter rather than the complete base model.

LoRA Configuration

The adapter uses:

• LoRA rank (r): 16
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Training epochs: 1
• Trainable parameters: Under 1% of the total model parameters

LoRA was selected because the objective is primarily a response-format and task adaptation, making it unnecessary to update all parameters of the 1.1B-parameter base model.

Training Infrastructure

• Hardware: Google Colab NVIDIA T4 GPU
• VRAM: 16 GB available
• Peak VRAM usage: Approximately 4 GB
• Training time: Approximately 10 minutes for 3,000 examples and 1 epoch
• Environment: Google Colab

Evaluation

Testing Data

Evaluation was performed on a held-out evaluation split from the b-mc2/sql-create-context dataset.

The evaluation uses the raw, unformatted examples so that the generated SQL can be compared against the reference SQL.

Factors

The evaluation focuses primarily on whether the generated response matches the expected SQL query.

The model was also qualitatively compared before and after fine-tuning using three held-out questions.

Metrics

The primary metric used was:

Exact-Match Accuracy

The comparison is case-insensitive and whitespace-normalized.

This metric is intentionally strict. SQL queries that are semantically equivalent but differ in formatting, aliases, or query structure may still be counted as incorrect.

Results

The expected qualitative behavior after fine-tuning is:

Before fine-tuning:

The base model may produce verbose responses containing explanations, Markdown code fences, or additional commentary.

After fine-tuning:

The model is trained to produce a single SQL statement without unnecessary explanation.

Example target behavior:

SELECT name FROM employees
WHERE department = "engineering"
AND salary > 100000

The notebook also reports exact-match accuracy on the held-out evaluation set.

Note: The numerical exact-match score should be added here from the final notebook run rather than estimated or invented.

Environmental Impact

The model was trained using a Google Colab NVIDIA T4 GPU.

• Hardware: NVIDIA T4
• Approximate training time: 10 minutes
• Cloud provider: Google Colab
• Compute region: Not recorded
• Carbon emissions: Not measured

Carbon emissions should be calculated separately if accurate hardware utilization and regional energy information are available.

Technical Specifications

Model Architecture and Objective

The underlying model is TinyLlama-1.1B-Chat-v1.0, a causal language model.

LoRA adapters are added to the model's attention and MLP projection layers.

The objective is supervised fine-tuning for Text-to-SQL generation:

Input:

Table schema + natural-language question

Output:

SQL query

The adapter is designed primarily to improve task-specific behavior and output formatting while keeping the underlying base model frozen.

Compute Infrastructure

Training was performed on Google Colab using an NVIDIA T4 GPU.

The observed peak GPU memory usage was approximately 4 GB.

Software

The training workflow uses:

• Python
• PyTorch
• Hugging Face Transformers
• Hugging Face PEFT
• Hugging Face TRL
• Hugging Face Datasets
• Accelerate
• SentencePiece
• Protobuf

Adapter Size

The trained LoRA adapter is approximately 20 MB.

Only the adapter weights are stored and distributed; the original TinyLlama base model is loaded separately.

Repository

The complete training workflow is available in the project repository.

The repository contains the Google Colab notebook used to:

• Set up the environment
• Load the base model
• Prepare the dataset
• Configure LoRA
• Train the adapter
• Evaluate the model
• Save the adapter

Limitations and Future Improvements

Potential improvements include:

• Training on the complete ~78k-example dataset.
• Increasing training to 2–3 epochs.
• Evaluating larger language models such as Qwen2.5-3B.
• Using QLoRA for larger models on limited GPU memory.
• Adding execution-based SQL evaluation.
• Testing across multiple SQL dialects.
• Adding SQL syntax validation.
• Evaluating semantic equivalence rather than exact string matching.
• Testing on schemas outside the training distribution.

Citation

If you use this adapter in your work, please cite the project repository and the underlying TinyLlama model.

BibTeX

@misc{akshaiml_tinyllama_sql_lora,
  author = {AkshAIML},
  title = {TinyLlama SQL LoRA Adapter},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/AkshAIML/tinyllama-sql-lora-adapter}
}

Credits

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Dataset: b-mc2/sql-create-context
• Libraries: Hugging Face Transformers, PEFT, TRL, Datasets, Accelerate

Model Card Authors

AkshAIML

Model Card Contact

For questions, issues, or improvements, please use the project's repository or Hugging Face model page.