Darwin-180B-NL2SQL

Darwin-180B-NL2SQL is VIDRAFT's text-to-SQL system built on Darwin-180B-RSI. It turns a natural-language question into a SQLite query, runs it against the database and revises the query from what the execution shows, then returns one final query per question.

This repository holds the system's predictions on BIRD Mini-Dev (SQLite) and the script used to score them.

Weights: the model is Darwin-180B-RSI, unchanged. Download the weights from that repository; no weights were changed or trained for this result.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "FINAL-Bench/Darwin-180B-RSI"  # weights for Darwin-180B-NL2SQL
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")

Result

BIRD Mini-Dev, SQLite, 500 questions: 83.80% execution accuracy (419 / 500).

Difficulty	Correct	Accuracy
Challenging	384 / 460	83.5%
Moderate	29 / 32	90.6%
Simple	6 / 8	75.0%
All	419 / 500	83.80%

• Data: the current canonical release on Hugging Face, birdsql/bird_mini_dev, release "Mini-Dev 1024", data/mini_dev_sqlite.jsonl (sha256 926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606), with the SQLite databases from minidev.zip.
• Metric: strict execution accuracy. A prediction is correct when the set of rows it returns equals the set of rows the reference SQL returns. Each predicted query may run for 60 seconds.
• Measured on 29 October 2026 (KST). The score was computed twice on an idle machine and was identical both times.

Evaluation conditions

Condition	Setting
Reference (gold) SQL used at any stage of inference	No
Oracle knowledge beyond the question and evidence given	No
Fine-tuning on BIRD	No
Final candidates per question	1
Database access during inference	Read-only execution of the system's own queries

The question and evidence in each item are the only task inputs. The reference SQL was used only to score the final predictions.

Files

File	Contents
predict_mini_dev_darwin-180b-nl2sql_SQLite.json	Predictions in the BIRD Mini-Dev submission format (SQL\t----- bird -----\tdb_id, keyed by question ID)
predictions.jsonl	The same predictions, one JSON object per line
score_ex.py	Strict execution-accuracy scorer used for the numbers above

To reproduce the score, place mini_dev_sqlite.jsonl next to score_ex.py and run:

python score_ex.py predictions.jsonl path/to/MINIDEV/dev_databases

About the system

The system pairs Darwin-180B-RSI with VIDRAFT's NL2SQL verification engine, which checks each generated query against the live database before an answer is returned. The engine is VIDRAFT's own implementation and is not released.

A public demo of VIDRAFT's NL2SQL engine (Korean and English, read-only, synthetic data) is available at final-bench-nl2sql-demo.hf.space.

Contact

VIDRAFT · vidraft.net · vidraft@mail.vidraft.net