PhillyLabs/philadelphia-decider-4b

Philadelphia Decider 1.0 (4B)

A 4B decision model. It reads a state (text or JSON) and typed questions, and returns a probability distribution over the allowed answers:

• choice: pick one option;
• noul: yes or no;
• score: pick a level.

It reads the answer probabilities from a single forward pass and never generates text.

Philadelphia Decider is a fine-tune of Mapika/decider-4b v2.1, which is itself built on Qwen3.5-4B-Base. It uses the same architecture, prompt layout and inference library. The goals were:

1. Better-calibrated confidence on hard decisions.
2. Small, honest accuracy gains on hard, unfamiliar decision families, without trading away the base model's general skills.

Use

Same API as decider (pip install decider-ai):

from decider.infer import Decider
d = Decider("PhillyLabs/philadelphia-decider-4b")   # one CUDA GPU, bf16, about 8 GB
d.system_one(
    {"ticket": "I was charged twice for order A-104. Please refund the duplicate."},
    {"team": {"type": "choice", "instructions": "Which team should handle this?",
              "criteria": {"billing": "Charges, invoices, refunds", "shipping": "Delivery problems", "other": None}},
     "refund_requested": {"type": "noul", "instructions": "Does the customer ask for a refund?"}})

The bundled decider_config.json carries this release's per-type temperatures (choice 2.022, noul 2.457, score 1.663). decider-ai >= 1.4.0 applies them automatically.

Results

Everything below is compared against the base model (decider-4b v2.1). Deltas are paired: the same items are scored for both models.

Benchmark	Base (v2.1)	Philadelphia Decider 1.0	Notes
JevBench public, 231 items: correct	191	191 (merged release); 192 before merging	won 2 hard items, lost 2; one near-tie (top-p 0.54 vs 0.49) flipped in the BF16 merge
JevBench public, hard tier: ECE ↓	0.183 (as served, v2.1 temperatures)	0.081	JevBench's calibration axis is computed on the hard tier
Held-out synthetic practice set, 2,310 items in JevBench's family mix	70.95%	71.34% (+0.39 pp, z 1.5)	built from fresh generators and seeds; never used for training
Bespoke public suite, 12 human-labelled subsets	78.27	78.27 (+0.06 pp pooled)	no regression
Decision Index sample, 39 benchmarks, 4,621 questions	62.30%	62.14% (−0.16 pp, z −1.0)	within noise; see Limitations

The release blend strength (25%) was chosen with a rule fixed before the results were seen: the strongest blend whose Decision Index score stays statistically level with base, best on the held-out practice set. JevBench public was reported but never used for selection or for fitting temperatures. Stronger blends of the same adapter gained more on JevBench public and the practice set, but drifted on general knowledge and maths:

blend	JevBench public	practice set	Decision Index
35%	193	+0.74	−0.33
50%	194	+0.78	−0.51

How it was trained

Adapter. LoRA rank 16 (alpha 32) on attention and MLP, learning rate 1e-4, on top of decider-4b v2.1.

Training data. About 28k code-verified decision worlds. Every answer is computed by two independent solvers in code; no model wrote any answer. The families are:

• tradeoff and priority rules;
• response judging;
• temporal and numeric reasoning (yes/no and score items);
• information sufficiency and ambiguity;
• multi-hop lookups across sections.

Every family passed:

• a blind solve audit;
• shortcut probes (option position, mention order, note direction, numeric patterns, bag-of-words);
• an 8-gram overlap filter against JevBench's public items.

Anti-forgetting anchors. 40% of each step pulls the model toward the frozen base model's own predictions on 2,964 gold-free inputs from families the adapter does not train. Without this, a pilot lost 3.8 points on untrained families; with it, it lost none.

Selection. The checkpoint and the blend strength (WiSE-FT, 25% of the adapter) were chosen on held-out synthetic practice sets and the Decision Index rule above.

Temperatures. Fitted per question type by NLL on the held-out practice set.

No JevBench, Decision Index or Bespoke items were used for training, selection or temperature fitting.

Provenance note. The data generators (code that builds the worlds and computes their answers) were written with the help of AI coding assistants (OpenAI Codex and Anthropic Claude). The training examples themselves are produced by that code. A further 642 replay inputs from earlier synthetic sets were used only as targets toward the base model's own distribution.

Limitations

• Accuracy gains are small. On JevBench public the release ties base. The main improvement is calibration on hard items.
• The Decision Index sample is slightly below base (−0.16 pp, not significant). The losses concentrate in maths and science (GSM8K, GPQA), which this release does not train.
• Calibration trade-off. The per-type temperatures make very easy questions slightly under-confident. All-item ECE on JevBench public is 0.094 against 0.080 for base as served, in exchange for much better hard-item calibration.
• Synthetic evaluation sets. The practice sets are synthetic, built from our own generators, so they estimate rather than measure performance on unseen benchmarks.

License and attribution

Apache-2.0, as is the base model.

• Base: Mapika/decider-4b v2.1 (revision eb5fbdfc), which is built on Qwen3.5-4B-Base (Qwen team, Apache-2.0).
• The bundled decider/ inference package is Mapika's (Apache-2.0), included unchanged for convenience.