# SCM-SQL — Dataset Statistics

Snapshot as of the 1.0 release (2026-07-30).

## Totals

- **Pairs**: 500
- **Turn-level trials**: 556 (L6 pairs are multi-turn)
- **Complexity levels**: 6 (L1 → L6)
- **Domains**: 4 (demand, finance, inventory, logistics)

## Per-level distribution

| Level | Style | Pairs | Turn-level trials |
|-------|-------|-------|-------------------|
| L1 | Single-table filter / aggregate | 100 | 100 |
| L2 | Two-table JOIN + GROUP BY | 100 | 100 |
| L3 | Nested subquery / semi-join / 3-table join | 130 | 130 |
| L4 | Window function | 60 | 60 |
| L5 | CTE + rollup | 60 | 60 |
| L6 | Multi-turn refinement (2-3 turns each) | 50 | 106 |
| **Total** | | **500** | **556** |

## Per-domain coverage

A single pair can carry multiple domain tags — cross-domain pairs
(especially L3 and L6) count in every domain they touch.

| Domain | Approx. pairs tagged |
|--------|----------------------|
| demand | ~ 145 |
| finance | ~ 140 |
| inventory | ~ 130 |
| logistics | ~ 130 |

## Provenance

| Bucket | Count | How it was authored |
|--------|-------|---------------------|
| Hand-crafted core | 100 | Author-written (NL + gold SQL) covering the queries a supply-chain analyst asks in practice |
| LLM-generated (execute-verified) | 400 | Generated by gpt-4o-mini from few-shots of the hand-crafted core; every candidate must (a) parse via sqlglot, (b) execute against the Odoo demo, and (c) return a non-empty non-null result set before acceptance |

## Reference-implementation performance on this dataset

Numbers below are for reference only — they are what the *reference*
multi-agent NL-to-SQL pipeline at
[github.com/AniruddhaPKawarase/scm-nl2sql](https://github.com/AniruddhaPKawarase/scm-nl2sql)
achieves on SCM-SQL. Your system will produce different numbers.

| Metric | Ours (gpt-4o-mini backbone) | MAC-SQL comparator (same backbone) |
|--------|-----------------------------|-------------------------------------|
| Execution Accuracy (EX) | 11.5 % | 10.1 % |
| Soft-EX | 11.9 % | 10.1 % |
| 95 % bootstrap CI (EX) | [8.8 %, 14.4 %] | [7.7 %, 12.8 %] |
| Wilcoxon signed-rank p | 0.325 | — |

**Interpretation**: the aggregate execution accuracy is intentionally
low. SCM-SQL is a hard benchmark by construction — 498-table Odoo
schema, JSONB translated-name fields, multi-turn dialogues,
multi-domain composition. This is a feature, not a bug: a benchmark you
never fail is not testing you.