SQLQwen - Qwen2.5 Text-to-SQL Fine-Tuning with LoRA

LoRA fine-tuning of Qwen2.5-Coder-0.5B-Instruct for Text-to-SQL generation using the Gretel synthetic_text_to_sql dataset.

Text-to-SQL systems translate natural-language questions into SQL using a provided database schema or SQL context. SQLQwen is designed to specialize a lightweight code-focused language model for this task while keeping the fine-tuning process parameter-efficient and practical on day-to-day (normal) hardware.

Given:

1. a database schema or SQL context
2. a natural-language request

the model generates the relevant SQL query

Example

Database context:

CREATE TABLE customers (
    id INTEGER PRIMARY KEY,
    name TEXT,
    country TEXT,
    revenue DECIMAL(12, 2)
);

Natural-language request:

Find the five customers with the highest revenue.

Expected output:

SELECT id, name, country, revenue
FROM customers
ORDER BY revenue DESC
LIMIT 5;

Why LoRA

LoRA (Low-Rank Adaptation) provides a parameter-efficient alternative to full fine-tuning.

Instead of updating all parameters in the pretrained model, LoRA freezes the original model weights and introduces small trainable low-rank matrices into selected Transformer layers.

for this project, LoRA adapters are applied to the attention projection layers:

q_proj
k_proj
v_proj
o_proj

this reduces the number of trainable parameters, GPU memory requirements, and adapter storage size while preserving the capabilities of the original pretrained model.

Only 2,162,688 of 496,195,456 parameters, or approximately 0.4359%, were trainable during fine-tuning.

Model

• Base model: Qwen/Qwen2.5-Coder-0.5B-Instruct
• Fine-tuning: PEFT LoRA
• Training: TRL SFTTrainer
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj
• Training objective: Completion-only supervised fine-tuning

Qwen2.5-Coder was selected because SQL generation is fundamentally a structured code-generation task rather than conventional natural-language classification.

Dataset

Gretel Synthetic Text-to-SQL

gretelai/synthetic_text_to_sql

Gretel Synthetic Text-to-SQL dataset provides natural-language SQL requests, database context, target SQL queries, and supporting metadata.

3 fields are used directly during training:

Dataset field	Purpose
sql_context	Database schema or SQL context
sql_prompt	Natural-language request
sql	Ground-truth SQL completion

Training Run

Metric	Result
Training examples	20,000
Validation examples	1,000
Epochs	2
Optimizer steps	2,500
Final training loss	0.2381
Best evaluation loss	0.2198
Final evaluation loss	0.2200
Final evaluation token accuracy	93.54%
Best checkpoint	2,400
Training runtime	~1h 35m

Training was completed locally on an NVIDIA GeForce RTX 3050.

Validation loss decreased from 0.2839 at the first logged evaluation to a best value of 0.2198 at checkpoint 2,400. The final checkpoint shows an evaluation loss of 0.2200, indicating that training had largely converged by the end of the second epoch.

Held-Out Generation Evaluation

A held-out generation benchmark of 50 examples was used to evaluate SQL generation code quality.

Metric	Result
Evaluation examples	50
Exact-match accuracy	28.0%
SQL syntax validity	100.0%
Exact matches	14 / 50
Syntax-valid generations	50 / 50

All 50 generated SQL queries were successfully parsed by SQLGlot, resulting in a 100% syntax-validity rate.

28% exact-match score uses strict string-level comparison. Semantically or execution-equivalent SQL queries may differ from the reference query while still producing the correct result, so exact match should not be interpreted as the model's full semantic accuracy.

Model Limitations:

• model may hallucinate tables or columns when the supplied database context is incomplete.
• 0.5B parameter base model prioritizes lightweight training and inference over maximum reasoning capacity.
• Training uses synthetic Text-to-SQL examples, which may not represent every real-world database schema or production SQL workload.
• Exact-match evaluation does not account for all semantically equivalent SQL formulations.

Additional Note: SQL execution accuracy against live databases was not measured in the reported benchmark.

Results:

completed fine-tuning run shows:

• parameter-efficient adaptation, with approximately 0.4359% of model parameters trainable
• stable convergence, with validation loss reaching approximately 0.22
• 100% syntax-valid SQL generation across the 50-example held-out benchmark
• successful local fine-tuning of a code-focused language model using an NVIDIA GeForce RTX 3050

Imports (PEFT library):

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "AaronTekle/SQLQwen")

References:

• Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, 2021.](https://arxiv.org/abs/2106.09685)

• Hugging Face. PEFT LoRA Documentation. Parameter-Efficient Fine-Tuning documentation. (https://huggingface.co/docs/transformers/en/peft) (https://huggingface.co/docs/peft/en/package_reference/lora)

• Qwen Team. Qwen2.5-Coder-0.5B-Instruct Model Card

• Hui, B. et al. Qwen2.5-Coder Technical Report. arXiv:2409.12186, 2024

• Gretel.ai. synthetic_text_to_sql Dataset Card. Hugging Face Datasets