Model Card for deepseek-coder-7b-text2sql-magicoder-lora

LoRA adapter fine-tuned from deepseek-ai/deepseek-coder-7b-instruct-v1.5 for text-to-SQL generation, trained with TRL SFTTrainer.

Intended Use & Limitations

Intended use: generating a single SQL query from a natural-language question given a CREATE TABLE schema, in English, for single or lightly-joined relational databases similar in style to Spider / sql-create-context schemas.

Limitations:

• Not evaluated on production/adversarial inputs, non-English questions, or dialects outside SQLite-compatible syntax.
• Execution accuracy on Spider (60.8%) means roughly 2 in 5 generated queries on unseen schemas are still wrong — always validate generated SQL before running it against a real database, especially for destructive statements (this adapter was only trained/evaluated on read (SELECT) queries).
• JOIN over-generation: the SQL training data is almost entirely single-table, so on multi-table schemas the model tends to join every available table indiscriminately. This is the main source of exact-match errors on Spider.
• Code capability (HumanEval+) is retained better than the pure-SQL version but still ~4pp below the un-finetuned base model — for general-purpose coding tasks unrelated to SQL, the base model remains the stronger choice. See Results below.

What's on main vs pure-sql

• main (this version) — trained on a 50/50 mix of SQL and general code-instruction data. Better SQL generalization to unseen schemas and better retention of general code ability than the pure-SQL version.
• pure-sql branch — the original version trained on 100% SQL data. Comparable in-distribution SQL accuracy, but noticeably worse code capability retention and worse generalization to unseen database schemas.

# to load the pure-SQL version instead:
PeftModel.from_pretrained(model, adapter_id, revision="pure-sql")

Why mix in code data

The pure-SQL version showed catastrophic forgetting of general code generation ability:

Metric	Base	Pure-SQL SFT	Δ
HumanEval pass@1	52.0%	40.0%	-12.0pp
HumanEval+ (999 edge cases)	46.0%	34.0%	-12.0pp

Training data was rebalanced to 50% b-mc2/sql-create-context + 50% ise-uiuc/Magicoder-OSS-Instruct-75K (interleaved batch-wise via datasets.interleave_datasets, stopping_strategy="first_exhausted"), 1 epoch.

The two datasets are nearly the same size (~70.7k vs ~75.2k), so a 50/50 ratio consumes essentially all of both — 141,771 training examples, 8,861 optimizer steps.

Results

Code capability retention (n=50, HumanEval/HumanEval+)

Metric	Base	Pure-SQL SFT	This version (mixed)
HumanEval pass@1	52.0%	40.0% (-12.0pp)	44.0% (-8.0pp)
HumanEval+ (plus)	46.0%	34.0% (-12.0pp)	42.0% (-4.0pp)

SQL generalization on unseen schemas (Spider 1.0, n=1034, real databases + official eval)

Metric	Base	Pure-SQL SFT	This version (mixed)
Official Execution Accuracy	39.9%	50.4%	60.8%
Official Exact Match (structural)	32.1%	37.4%	47.3%

Compared with the pure-SQL version, this version generalizes substantially better to unseen schemas (+10.4pp execution accuracy) while also retaining more code capability. Mixing in code data appears to act as a regularizer against overfitting to the narrow single-domain SQL distribution.

LoRA configuration

r: 16
lora_alpha: 32
lora_dropout: 0.05
target_modules: [q_proj, k_proj, v_proj, o_proj]
quantization: 4-bit NF4 (QLoRA), bf16 compute

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "deepseek-ai/deepseek-coder-7b-instruct-v1.5"
adapter_id = "albertkingdom/deepseek-coder-7b-text2sql-magicoder-lora"

model = AutoModelForCausalLM.from_pretrained(
    base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)  # main = mixed training version

messages = [{
    "role": "user",
    "content": """Given the database schema below, write a SQL query that answers the user's question.
Only output the SQL query. Do not add any explanation.

### Schema
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(100))

### Question
Find all users with gmail addresses"""
}]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

The adapter was trained with exactly this instruction + ### Schema / ### Question layout wrapped in the DeepSeek chat template. Free-form prompts still work, but accuracy drops.

Training procedure

Trained with SFT (TRL SFTTrainer) on an interleaved SQL + code instruction dataset, 1 epoch over 141,771 examples (8,861 steps), learning rate 1e-4 with linear decay and 3% warmup, effective batch size 16 (per-device 4 × gradient accumulation 4), max sequence length 1024, paged_adamw_8bit, bf16, 4-bit QLoRA, on a rented RTX 5090 (~12h). Final train loss 0.3986, eval loss 0.3437.

Framework versions

• PEFT: 0.18.0
• TRL: 0.26.2
• Transformers: 4.57.3
• Datasets: 4.4.2
• PyTorch: 2.9.1+cu128

License

This adapter is a derivative of deepseek-ai/deepseek-coder-7b-instruct-v1.5, which is released under the DeepSeek Model License rather than a standard open-source license. Per that license, derivative models must carry forward at least the same use-based restrictions, so this adapter — and any model merged/derived from it — inherits them:

• No use for military purposes.
• No use that harms minors.
• No generation of false information intended to harm others.
• No creation of non-consensual personal identifiable information.
• No fully automated decision-making that adversely affects an individual's legal rights.
• No discrimination based on protected characteristics.
• See the full license text (Attachment A) for the complete list.

Commercial use is otherwise permitted, consistent with the base model's license.

Credits & Data Provenance

• Base model: deepseek-ai/deepseek-coder-7b-instruct-v1.5 (DeepSeek Model License)
• b-mc2/sql-create-context (CC-BY-4.0) — itself derived from WikiSQL and Spider; credit to both original sources per CC-BY-4.0 attribution terms.
• ise-uiuc/Magicoder-OSS-Instruct-75K (MIT) — generated via the OSS-Instruct method using gpt-3.5-turbo-1106. Outputs are subject to OpenAI's usage policies in addition to the dataset's own MIT license.

Citations

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}

@article{yu2018spider,
    title   = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task},
    author  = {Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir},
    journal = {arXiv preprint arXiv:1809.08887},
    year    = 2018
}

@article{wei2023magicoder,
    title   = {Magicoder: Empowering Code Generation with OSS-Instruct},
    author  = {Wei, Yuxiang and Wang, Zhe and Liu, Jiawei and Ding, Yifeng and Zhang, Lingming},
    journal = {arXiv preprint arXiv:2312.02120},
    year    = 2023
}