GenbaLabs/torchcast-decision-12b-mlx

Torchcast Decision 12B — MLX, 8-bit

An 8-bit MLX copy of torchcast-ai/torchcast-decision-12b (revision 49107b5589bf2dc4ec3f5acd54316971dd5978b1, tag v1.0.0), made so the model runs on Apple silicon. Upstream ships a vLLM serving package and no MLX artifact; this is one.

It is not a chat model. It is a typed-decision model: one forward pass over a state plus a question plus lettered options, and the answer is the probability of each option letter at the answer slot — one output token, no reasoning, no prose. The authors' upstream card is the reference for what the checkpoint is and what it was trained on; this card covers only what changed here and how to run it.

	upstream	this copy
precision	bf16	8-bit affine, group 64 (8.5 bits/weight)
size	23.9 GB, one shard	12.7 GB, three shards
runtime	vLLM 0.30.0 + upstream shim	mlx_lm.server, letter readout on the client
modality	gemma4_unified (text/vision/audio)	text only — mlx-lm drops the towers it cannot run

Quickstart

python3 -m venv .venv && . .venv/bin/activate    # Python 3.10+
pip install mlx-lm                               # tested with mlx-lm 0.32.2, mlx 0.32.4
MODEL=$(pwd) sh runtime/run.sh                   # serves 127.0.0.1:8080, ~12 GB resident

Then ask one typed question — seven difficulty levels, read off the option letters:

python3 runtime/readout.py --state "fix the typo in the readme" \
  --instructions "Rate how hard this task is for an AI coding agent: how much focused work to finish well." \
  "trivial: a typo" "small: a quick edit" "routine: one focused change" "moderate: a feature, a few files" \
  "substantial: design across modules" "large: a refactor or migration" "huge: a multi-day build"

{"choice": "B",
 "probabilities": {"A": 0.266244, "B": 0.49741, "C": 0.207351, "D": 0.024765, "E": 0.002958, "F": 0.00096, "G": 0.000312},
 "prompt_tokens": 183, "ms": 374}

runtime/readout.py is stdlib-only and mirrors the upstream torchcast_shim.py prompt exactly. Any OpenAI-compatible client works the same way: max_tokens: 1, logprobs: true, top_logprobs: 11, chat_template_kwargs: {"enable_thinking": false}.

The readout protocol

The model answers with a letter. Recovery of a distribution is:

1. Render the state, the instructions and Options: as A. …, B. …, then Answer with the letter of exactly one option, and nothing else: and send it as the user message, with the upstream system message ("You are a calibration engine…"). The scaffolding is load-bearing: without it the model answers in prose and no distribution over options exists.
2. Read choices[0].logprobs.content[0].top_logprobs at the single output token.
3. Sum exp(logprob) per letter. A letter can surface as several token ids (A, A, A.), so summing per letter — not picking one token — is what upstream specifies. The letters come from a bf16 lm_head, so their logprobs sit on a coarse grid and ties happen: a three-way exact tie (C = D = F = -1.25) is reproducible on some inputs. Take the expectation over the levels rather than the argmax alone if a tied pick matters. A long-running server also shifts these low-order bits through its warm prompt cache: restart it before recording numbers you intend to quote.
4. Renormalize, and apply the per-type temperature from the bundled readout_config.json: score and choice 1.0, noul 0.2 (it fits calibration on public validation sets; do not re-tune it casually).

Types the checkpoint was trained for are noul (yes/no), choice (unordered options) and score (ordered options). Upstream's /v1/systemone wraps this protocol in HTTP; there is no equivalent here, see below.

The upstream /v1/systemone shim does not run on MLX

torchcast_shim.py and readout_config.json are included unchanged, but the shim cannot serve this copy. It needs two things mlx_lm.server does not provide: a structured_outputs letter mask on the upstream request (vLLM-specific, and load-bearing for the single-token answer), and an integer max_model_len in /v1/models. Pointed at MLX it exits at startup or returns 503 rather than answering. The MLX path is the client-side readout above, which is what the Pumpt! Claude Code mod does in production (runtime/readout.py is its Python equivalent, verified to reproduce the mod's distributions exactly).

Numbers, measured on an M5 Pro (48 GB, macOS)

• Load: ~17 s to /v1/models. First decision after start ~4.7 s (kernel warm-up).
• Warm decision: 100 ms fixed + ~1.1 ms per prompt token (900 tok/s prefill). 221-token prompt 402 ms; 402-token 593 ms; 931-token 1186 ms; a cached prefix 159 ms.
• Resident memory ~11 GB. mlx_lm.server keeps an LRU prompt cache, so a constant system+instructions prefix is nearly free.

What was checked here, and what was not: three decisions recorded from a live session of the mod were re-run against this copy — same argmax every time, max|Δp| between 0.0004 and 0.043 (the larger deltas come from context trimming, not the quantization). No bf16-vs-8-bit or MLX-vs-vLLM probability comparison exists. The upstream card's public-231 result (203/231, 87.88%) was measured on vLLM at bf16; it is not reproduced here, and 8-bit quantization can move calibrated probabilities — possibly toward over-confidence. If you depend on the calibration, measure the delta against the upstream bf16 artifact on the same inputs before trusting thresholds.

Limits

• Text only. Images and audio were in the upstream checkpoint; this copy cannot take them.
• top_logprobs is capped at 11 by mlx_lm.server, so this readout path reads up to 11 options (up to 26 with the upstream shim and SHIM_TOP_LOGPROBS=20).
• The protocol is one answer per question. Multi-question / multi-span cases are served upstream by serving/decision_server.py and are not covered here.
• English, decision-style inputs. The upstream card's intended-use and limitation notes apply unchanged.

Licence and attribution

Weights are CC BY-NC 4.0 — non-commercial use only, as upstream: they are a modified (8-bit quantized, text-only) copy of torchcast-ai/torchcast-decision-12b, which is a fine-tune of google/gemma-4-12B-it. Attribution to Torchcast AI (checkpoint), Google DeepMind (Gemma), and Cygnet/blockbrain-ai and NInfer (serving/readout foundation) is required; the changes made here are listed in NOTICE. LICENSE (the upstream CC BY-NC 4.0 text), LICENSE-gemma-Apache-2.0 and LICENSE-serving-MIT are carried over untouched — preserve them.

Use is also subject to Google's Gemma Prohibited Use Policy. The CC BY-NC label does not establish clearance of every training-source right; see the upstream card. This copy is not affiliated with Torchcast AI, Google, or the JevBench maintainers.