TinySQL-125M-WikiSQL

A 125M-parameter text-to-SQL model. Given a table schema and a question, it writes the SQL query. Every part of it was built from scratch: the tokenizer, the transformer, the optimizer, the training loop and the data pipeline. It was pretrained on 4.5B tokens on one GPU, then fine-tuned on WikiSQL.

• Paper: TinySQL: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens (Zenodo, 2026)
• Code: github.com/aditya0929/tinysql
• Base model: adiitya0929/TinySQL-125M-base

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "adiitya0929/TinySQL-125M-WikiSQL"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = (
    '<schema>CREATE TABLE t ("Player" TEXT, "No." TEXT, "Nationality" TEXT, "Position" TEXT, '
    '"Years in Toronto" TEXT, "School/Club Team" TEXT);</schema>\n'
    "<question>What position does the player who played for butler cc (ks) play?</question>\n"
    "<sql>"
)
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64)          # greedy; stops at </sql>
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
# SELECT "Position" FROM t WHERE "School/Club Team" = 'butler cc (ks)';

Prompt format. The model was trained on exactly this layout. The table is always called t. Column names are double-quoted, and each column is typed REAL (numeric) or TEXT. The question goes in <question>…</question>, and the prompt ends with <sql>. The model writes one SELECT statement and then </sql>.

Results (WikiSQL test, 15,878 questions)

Model	Pretraining tokens	Greedy	+ Schema + values constraints
TinySQL-125M (this model, mean of 3 seeds)	4.5B	77.95 ± 0.18	80.39 ± 0.20
GPT-2 124M	not published	78.93	82.36
Pythia-160M	300B	60.78	67.63
SmolLM2-135M	2T	82.57 ± 0.37	85.22 ± 0.21
SmolLM2-360M	4T	87.03 ± 0.19	88.01 ± 0.12

The table reports execution accuracy (%): the generated query is run against the table and its result compared with the gold query's. Every model was fine-tuned with the same recipe after its own learning-rate sweep. This checkpoint is seed 0, which scores 78.01 greedy and 80.44 constrained. The official WikiSQL evaluator gives 78.08 execution and 72.38 logical-form accuracy for it.

model.generate() above is plain greedy decoding. The constrained numbers use the grammar-constrained decoder in the GitHub repository (decode/). It only allows the table's real column names and values copied from the question, which gives 100% valid SQL.

Training

Architecture	Llama-style decoder: 30 layers, hidden size 576, 9 query / 3 key-value heads (GQA), SwiGLU 1536, RoPE, RMSNorm, tied embeddings, context 2048
Parameters	125,077,824
Tokenizer	own byte-level BPE, 32,768 tokens
Pretraining	4.51B tokens (FineWeb-Edu, The Stack SQL, Stack Exchange, Gretel synthetic text-to-SQL), 8,600 steps, about 46 h on one NVIDIA L4
Fine-tuning	WikiSQL train (56,355 examples), loss on the SQL only, 3 epochs, batch 32, peak LR 5e-5, best epoch by dev execution accuracy

The architecture matches Llama, so the weights are distributed in the standard LlamaForCausalLM format. The model was built and trained with the repository's own code, not with transformers. The export is checked to produce the same token ids, logits and generated SQL as the original implementation.

Limitations

• WikiSQL shape only: one table, one selected column, an optional aggregate, and up to four AND conditions. No joins, GROUP BY, ORDER BY or nesting. Do not expect it to handle Spider-style or production SQL.
• English only; small model; greedy decoding can produce column names that don't exist. Use the constrained decoder from the repository to rule that out.
• Label noise: WikiSQL's labels are noisy. A hand audit in the paper suggests that most remaining aggregation errors are label errors.
• Not for production databases without review: generated SQL should never be executed with write permissions.

Training data licenses

FineWeb-Edu (ODC-By 1.0), The Stack (dedup), SQL subset (permissively licensed source files, subject to The Stack's terms of use), Stack Exchange dba and datascience (CC BY-SA; 2.5, 3.0 or 4.0 depending on the post date), gretelai/synthetic_text_to_sql (Apache-2.0), WikiSQL (Salesforce, BSD-3-Clause). The pretraining data was decontaminated against WikiSQL dev and test questions.

Citation

@techreport{jha2026tinysql,
  title       = {{TinySQL}: A 125M-Parameter Text-to-SQL Model Built and Pretrained From Scratch on 4.5B Tokens},
  author      = {Jha, Aditya Narayan},
  year        = {2026},
  institution = {Zenodo},
  doi         = {10.5281/zenodo.23396770},
  url         = {https://doi.org/10.5281/zenodo.23396770}
}