Laplace-AI-Research/tarot-draw-1-7b

Tarot-Draw 1.7B

The small end of the Tarot-Draw ladder. It answers the same interface as the 8B and 4B, on a base model 4.7× smaller than the flagship.

input:  question + resolution criteria + resolution date + as_of
output: p ∈ [0,1]

Weights: Laplace-AI-Research/tarot-draw-1-7b on the Hugging Face Hub — LoRA adapter. Larger siblings: Tarot-Draw 4B and Tarot-Draw 8B — same 81,870 questions, same targets, same seed, same footing. Take the 4B unless it will not fit: it ties the 8B and beats this model by +0.0119 Brier [+0.0080, +0.0158] on the dev split. See Training variance below before reading that interval as tight. Evidence: Laplace-AI-Research/tarot-draw-1-7b on GitHub — the eval splits behind every claim, this model's raw forecasts (including the venue transfers where it failed), the metric code, and verify.py, which recomputes every number below without a model or a GPU.

No evidence, no retrieval, no market price. A question and a date, and a number.

Read this first: the trade this model offers

It is worse than the 4B, measurably, and that is the point of it existing.

	dev Brier (n=3,000)	vs Tarot-Draw 4B
Tarot-Draw 8B	0.1674	—
Tarot-Draw 4B	0.1684	—
Tarot-Draw 1.7B	0.1803	+0.0119 [+0.0080, +0.0158], significant

Paired, question-clustered bootstrap, 10,000 resamples, seed 20260822.

The offer is roughly +0.012 Brier for 57% fewer parameters. If you are running on a laptop, on CPU, or anywhere the 8B does not fit, that is the trade. If you are not constrained, use the 4B — it is free to run by comparison and it ties the 8B.

We are telling you to prefer a different model in most cases. That is the honest reading of our own numbers and it belongs at the top of the card, not in a footnote.

The headline number

Brier 0.2030 on the contamination-free published split, n=277 — questions that resolved after the training freeze, so no part of this model saw them.

	value
Brier	0.2030
Murphy calibration ↓	0.0042
Murphy resolution ↑	0.0518
base rate	0.4946
base-rate Brier	0.2500

uv run python verify.py recomputes every number above from the forecasts and eval split in this repository. It needs no GPU, no weights and no network.

Calibration without a binning choice

Expected calibration error depends on a binning scheme, and neither available scheme is defensible: equal-width leaves most bins too sparse to estimate a frequency on a skewed question set, and equal-mass has edges that move between checkpoints, so two models get scored against different boundaries. We have been publishing both numbers (0.0683 and 0.0647 here) because we could not choose.

CORP removes the choice. The reliability curve is the isotonic (PAV) fit, so there are no bins at all:

	value
MCB — miscalibration ↓	0.0106
DSC — discrimination ↑	0.0576
UNC — uncertainty (a property of the split)	0.2500

score = MCB − DSC + UNC holds exactly, with a residual of zero. The binned Murphy decomposition carries a residual that depends on the scheme; this does not.

Method: Dimitriadis, Gneiting & Jordan, Stable reliability diagrams for probabilistic classifiers, PNAS 2021 (arXiv:2008.03033). Recomputed from the shipped forecasts by verify.py; the ECE pair is retained above for continuity with earlier versions of this card.

---

Contamination

Zero of the 3,000 dev questions, zero of the 277 published questions, and zero of the 1,219 Kalshi transfer questions resolved before the base model was released. Outcome memorisation is closed by construction, not by argument.

• Freeze: 2026-08-15, committed in a dated, public git history before it passed.
• The split is gated on Qwen3-1.7B's public release date (2025-04-27), not on a stated pretraining cutoff — a release date is externally checkable, a cutoff is a vendor claim.

---

What the size ladder actually shows

This is the finding the three models exist to support, and it is more interesting than any one of them.

Calibration is size-invariant on this task. Discrimination is not.

dev, n=3,000	calibration ↓	resolution ↑
Tarot-Draw 8B	0.0010	0.0718
Tarot-Draw 4B	0.0007	0.0708
Tarot-Draw 1.7B	0.0004	0.0593

Across a 4.7× parameter range, calibration is flat — the differences are well inside the ±0.003 seed-noise floor we measured. Every point of the degradation is resolution.

The small model learns how confident to be as well as the large one. What it loses is which questions to be confident about.

On the published split (n=277) the calibration point estimates do differ — 0.0048 / 0.0034 / 0.0042 — but that split is far too small to resolve a difference of that size. That split is too small to resolve a calibration difference this size, so the claim above rests on the dev split, where there is power. Both are given.

Calibration: none applied

The shipped probability is the head's raw sigmoid(logit). No post-hoc calibration map is applied and none is distributed. The calibration figures above are what the model produces untuned.

What this model cannot do

1. It cannot beat the 4B. See the top of this card.
2. It cannot do mechanical threshold or counting questions. Questions needing a time series and precise arithmetic are outside a judgment prior.
3. Short horizons. Under 7 days it never beats a market crowd at any point in a question's life.
4. Calibration does not transfer to a new venue. A new venue needs its own calibration mapping, fit on that venue's own development split.

Transfer, measured on this model

Scored off-venue on 1,219 Kalshi judgment questions and 3,000 Polymarket questions. Measured on this model, not inherited from a sibling.

The Kalshi rows were rebuilt on 2026-08-30 and changed. The set they replaced had no builder anywhere — its selection rule was a metadata string no code implemented — and its prompts depended on a Kalshi API field present in no archive we kept. This set is built by scripts/kalshi_eval_set.py, kalshi_subtitles.py and kalshi_rebuild_prompts.py, which reproduce all 157 prompts shared with the old set exactly and refuse to write otherwise. verify.py recomputes every number below.

	n	Brier	resolution	BSS
Manifold dev (home venue)	3,000	0.1803	0.0593	+24.7%
Kalshi — Politics	170	0.0888	0.1136	+54.4%
Kalshi — Entertainment	500	0.2041	0.0820	+9.8%
Kalshi — Science & Technology	97	0.1969	0.0653	+18.7%
Kalshi — Economics	67	0.2556	0.0293	−5.1%
Kalshi — Elections	382	0.2192	0.0065	−5.8%
Kalshi — all	1,219	0.1949	0.0489	+12.7%
Polymarket (mechanical questions)	3,000	0.2196	0.0045	−17.2%

On Kalshi Politics this model discriminates better off-venue than at home — resolution 0.1136 against 0.0593. On Kalshi Elections it collapses to resolution 0.0065, and the aggregate is a composition artifact: Elections is 31% of this corpus and is a lookup rather than a forecast.

Polymarket is where it is worst. Resolution 0.0045 is close to none at all — those questions are mechanical (scores, thresholds, spreads) and outside what a judgment prior can do.

Elections is a subject-matter limit, not contamination

This card previously carried a section titled "the contamination-free Kalshi subset, where this model loses to a constant." That framing was wrong, and it was wrong in the direction that made us look careful.

That subset was 75% Elections. The model failed on it because it cannot forecast obscure local elections — which this card already says elsewhere about the aggregate — not because the freeze boundary revealed anything about contamination. Holding the category fixed shows it directly:

slice	n	Brier	BSS
Elections, resolved after the freeze	69	0.2087	−12.4%
Elections, resolved before the freeze	313	0.2215	−4.8%

Post-freeze is not better — it is slightly worse. If performance before the freeze were inflated by the base model having seen those questions, the clean subset would look worse than the exposed one. It does not. The limit is the subject matter, and it is present on both sides of the freeze.

On the rebuilt post-freeze subset as a whole (n=183) this model scores Brier 0.1577 against a constant's 0.2496 — −0.0920 [−0.1154, −0.0682], significantly better. That reverses the old card's claim, and it is a composition effect, not a contamination result: the old 117-row subset was 75% Elections, the new 183-row one is 46% Entertainment. Both numbers are artifacts of their category mix. Neither says anything about contamination, and neither should be quoted as though it does.

The honest summary is the one that survives every slicing: this model cannot forecast obscure local elections, and no amount of resampling changes that.

Training variance, and why the intervals above are narrower than the truth

This recipe was run twice, identically — same corpus, same seed, same learning rate, same footing — and produced two significantly different models.

	dev Brier
run 1	0.1752
run 2 (shipped here)	0.1803

run 2 − run 1 = +0.0051 [+0.0019, +0.0082], paired and question-clustered.

Individual forecasts between the two runs differ by up to 0.68.

Every interval in this card is a bootstrap over questions. It captures sampling variance and does not capture training variance at all. Two models from an identical recipe land outside each other's intervals, so a model-vs-model difference smaller than roughly 0.008 Brier should be treated as unresolved, whatever interval is printed beside it.

The +0.0119 gap to the 4B is above that bar. The calibration differences in the size-ladder table are not, and are not claimed.

The numbers in this card come from the run whose weights are published here, verified by recomputing them from the shipped forecasts. Run 1's figures are retracted, and are shown above only to document the spread.

Evaluation data

• published (headline, n=277) — Manifold questions resolving after 2026-08-15. Never trained on. The only source of an external number.
• dev (n=3,000) — resolving 2025-08-15 to 2026-08-15. Gates iteration, never published as a headline claim.

Eval splits carry question ids, dates and outcomes only — no question text. Each question_id is the venue's own stable identifier, so the text is retrievable from that venue under that venue's terms rather than ours.

All intervals are paired, question-clustered bootstraps (10,000 resamples). Seed-to-seed noise on this setup is ±0.003 Brier, so smaller differences are not findings. That floor is now known to be understated — see Training variance below.

---

Training

LoRA r=32, α=64 on Qwen/Qwen3-1.7B with a scalar regression head (Qwen3ForSequenceClassification, num_labels=1, modules_to_save=["score"]), Brier loss against the terminal 0/1 outcome. 81,870 Manifold questions resolving before the freeze. Seed 20260822, LR 1e-4 OneCycle, right padding, last-non-pad pooling, chat template on both train and score.

Identical corpus, seed, learning rate and footing to the 8B and 4B — the only variable across the three is parameter count. That is what makes the ladder above a measurement rather than three separate results.

A note on the base model

An earlier internal evaluation rejected Qwen3-1.7B as mode-collapsed: 226 of 500 answers exactly 0.5, 46.2% of probability mass within 0.05 of it, 3.02 bits of entropy. That was measured at Q4 quantisation on generated probability tokens.

This model is a bf16 scalar regression head, which does not generate. Its dev output distribution: 22.0% within 0.05 of 0.5, 6.19 bits of entropy, 1,257 distinct values across 3,000 questions, sd 0.2571. Not collapsed.

The earlier finding was about a substrate for prompt-based iteration, and it does not transfer to a regression head.

Licensing, in two parts

what	licence	commercial use
code — src/, scripts/, verify.py	Apache-2.0 (LICENSE-CODE)	permitted
adapter weights	CC BY-NC 4.0 (LICENSE)	not permitted
forecasts	CC BY-NC 4.0 (LICENSE)	not permitted
eval metadata — ids, dates, outcomes	CC BY-NC 4.0 (LICENSE)	not permitted
question text	not redistributed here	not ours to license
base model	Apache-2.0, by its authors	unaffected

The split is deliberate. The evaluation code contains no third-party rights and is permissively licensed, including for commercial use. The data-derived artifacts cannot be, because the corpus they come from restricts it.

Corrected 2026-08-29: an earlier version of this repository shipped the code with no licence grant at all, which under copyright means all rights reserved — published, but not usable by anyone. That was not intended and is fixed here.

---

Training data provenance and licensing

Adapter weights and evaluation metadata under CC BY-NC 4.0. Base model Qwen/Qwen3-1.7B is Apache 2.0 and unaffected.

No venue question text is redistributed here. Eval splits carry question ids, dates and outcomes only. Each question_id is the venue's own stable identifier, so the text is retrievable from that venue directly under that venue's terms.

Training data derives from the Manifold Markets public API and is not redistributed. Manifold's terms restrict bulk API data to personal and non-commercial use and prohibit training ML models for commercial purposes without a data licence; that licence is not ours to grant.

---

Reproducing

Every headline number is recomputed from the shipped forecasts by verify.py — no model, no GPU, no network:

uv run python verify.py

To regenerate those forecasts from the weights instead:

uv run python scripts/scalar_score.py \
  --adapter Laplace-AI-Research/tarot-draw-1-7b \
  --base-model Qwen/Qwen3-1.7B \
  --set eval/published_eval.json \
  --out published.jsonl \
  --chat-template

---

Intended use

Base rates and cold-start estimates where a larger model will not fit. Research on scale and calibration. Not for trading, and not as a substitute for the 4B where the 4B will run.

Provenance of claims

Every number here is recomputed from the shipped forecasts by verify.py, in this repository, with no model and no GPU. That is the check that matters and it is the one you can run.

Behind it sits an internal decisions log recording how each number was derived and which results failed — including, for this model, an earlier internal decision not to train this base model at all, which rested on a measurement taken at the wrong quantisation and on the wrong output head. That log is not public, so nothing in this card depends on it: every claim above is either reproducible from the files here or is stated as an unverifiable limitation.

---

The family

	parameters	dev Brier	use it when
Tarot-Draw 8B	8B	0.1674	you want the best number
Tarot-Draw 4B	4B	0.1684	almost always — it ties the 8B
Tarot-Draw 1.7B	1.7B	0.1803	the 4B will not fit

All three share one corpus, one seed, one learning rate and one footing. The only variable is parameter count.