QwerySmith-2.0-dev-seed3

A behaviour-tuned text-to-SQL adapter: it answers questions about a relational database from retrieved evidence, with machine-verifiable citations, and refuses when the evidence does not contain the answer. Facts come from retrieval - this adapter stores no data.

See the results table below.

What it is / is not

• Is: a QLoRA adapter (PEFT) for Qwen/Qwen3-8B teaching the output contract (SQL: / ANSWER: with [table:row_id] citations, or REFUSAL:), schema terminology, and calibrated refusal.
• Is not: a knowledge store. Unmodified Qwen/Qwen3-8B + the same retrieval achieves the fact-bearing part; per the v1.1 finding, fine-tuning taught style, not facts - this release is built on that constraint.

Training recipe (fully pinned)

Parameter	Value
Base model	Qwen/Qwen3-8B
Method	QLoRA, 4-bit NF4, double quantization
LoRA	r=16, alpha=32, dropout=0.05, targets=['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj']
Optimizer	lr=0.0001, cosine schedule, warmup 0.03, 3 epochs
Batch	1 per device x 16 grad accum, max_len 4096
Decoding (pinned)	non-thinking mode, T=0.7, top_p=0.8, top_k=20
Training data	RAFT-style triples from 30 questions (10 train_ok; held-out questions never trained)
Mix	~70% grounded / ~15% refusal / ~15% schema-only, hard-negative distractors
Seed	3

Training data construction, distractor sampling, and the refusal mix are produced by the harness (triples stage) and are reproducible from the repo.

Dataset

• online_retail_ii - https://archive.ics.uci.edu/dataset/502/online+retail+ii, CC BY 4.0 (UCI Machine Learning Repository)
• 30 questions with gold SQL + expected rows; difficulty-tagged; split mechanically: 10 train_ok / 20 held-out
• Held-out rule: questions whose gold result changes when the last 6 months of data are removed are tagged held-out (cutoff 2011-06-01) and are never used in training. Enforced by a clamped-shadow-database execution check.

Evaluation - measured artifacts only

All numbers below come from captured run artifacts (report.md, failures/), produced by the same harness on identical questions with byte-identical frozen evidence packs for every system.

(Evaluation pending - this card is generated from the training run only. The results table lands here after the eval matrix runs.)

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", device_map="auto")
model = PeftModel.from_pretrained(base, "QwerySmith-2.0-dev-seed3")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")

# The model expects the harness prompt contract:
#   SYSTEM contract + SCHEMA (CREATE TABLE text)
#   + QUESTION + RETRIEVED EVIDENCE rows as [table:row_id] lines
# and emits either:
#   SQL: <one SELECT>  /  ANSWER: <text with [table:row_id] citations>
#   or REFUSAL: <what is missing>

Execute generated SQL read-only, with a statement timeout and an AST SELECT-only guard (the harness ships one).

Figures

figures/loss_curve_seed3.png	training loss + LR schedule
figures/results_table.png	the evaluation table, as measured
figures/ + card diagram	pipeline & inference contract (below)

                    QWERYSMITH 2.0 - ARCHITECTURE & PIPELINE
================================================================================

  RAW CSVs          PREPARE (CPU, local)                TRAIN (T4, Colab)
============      =========================            ==================
 9 Olist tables -> ingest   -> Postgres/SQLite  RAFT   triples jsonl -> QLoRA
                  profile  -> cutoff frozen    TRIPLE       |
                  validate -> leak checks       BUILDER      v
                  retrieve -> evidence packs    ~70/15/15  Qwen3-8B 4-bit
                  triples  -> training data   grounded/   + LoRA r16 a32
                              (train_ok only)  refusal/    lr 1e-4 cosine
                                               schema      3 seeds
================================================================================

  EVAL MATRIX (L4, Colab) - identical questions + identical frozen packs
===============================================================================
   row 1  Qwen3-8B base        + packs --> scorer --> EX / flip / refusal
   row 2  Qwen3-8B + adapter   + packs --> scorer --> mean of 3 seeds
   row 3  Qwen3-30B-A3B AWQ   + packs --> scorer --> on-prem alternative
   row 4  frontier (API)      + packs --> scorer --> reference
                                        |
                     McNemar + GATE (pre-registered) <- report.md + failures/

================================================================================

  INFERENCE CONTRACT (what the fine-tune teaches - behaviour only)
===============================================================================
   prompt = SYSTEM contract + SCHEMA (CREATE TABLE text)
          + QUESTION + RETRIEVED EVIDENCE [table:row_id] rows
                 |
                 v
   SQL:  <one SELECT statement>
   ANSWER: <text with [table:row_id] citations>     or   REFUSAL: <why>
                 |
                 v
   sandboxed execution (read-only role, 30s timeout, sqlglot AST guard)
   facts ALWAYS come from retrieval - the adapter never stores data

Limitations (stated plainly)

• Evaluation is on online_retail_ii only; generalization to other databases is not claimed by this card.
• The adapter emits SQL for the schema it is shown; correctness depends on retrieval quality - the same dependency every system in the matrix has.
• Fine-tuned for the harness output contract; other prompt formats are out of distribution.
• Single-dataset behaviour tune; the 3-seed spread (see sibling repos) indicates run-to-run variance.

Provenance

• Trained by the QwerySmith v3 harness (one-command reproducible): see repository

• Hardware: Tesla T4, 14912 MB VRAM; packages: {"torch": "2.11.0+cu128", "transformers": "5.5.0", "trl": "0.24.0", "peft": "0.20.0", "unsloth": "2026.9.11", "datasets": "4.3.0", "bitsandbytes": "0.50.2"}

• v1.x lineage (1.0/1.1 fine-tuning pipeline and its evaluation): see the repository's legacy/ directory.

License

Apache-2.0 (base model license). Dataset retains CC BY 4.0 (UCI Machine Learning Repository).