PhillyLabs/philly-decider-v2

Philly Decider v2 (4B)

Philly Decider v2 is a fine-tune of H2O-Lightning-4B v1.1 by H2O.ai. The base is h2oai/h2o-lightning-4b at revision 193ad740925b176a3b70a5a13a7cff2f2fadd01e, Apache-2.0, which is itself a post-trained Qwen/Qwen3.5-4B. H2O.ai deserves all credit for the base model, its prompt format and its serving shim. We changed the weights (a merged LoRA delta) and the readout settings, nothing else.

What this model is

• Base: H2O-Lightning-4B v1.1, revision 193ad740925b176a3b70a5a13a7cff2f2fadd01e. The tokenizer, chat template, prompt format and label tokens are unchanged.
• Our change: LoRA fine-tunes of the attention, Gated DeltaNet and MLP projections, trained on decision data with soft targets from the teachers below.
  • The fine-tunes are merged into the base in fp32 (W' = bf16(fp32 W + fp32 s B A), one rounding).
  • Decision agreement between the merged model and the unmerged adapter is 0.9907. The flips are near-ties, and the held-out score is unchanged.
• Lineage: dW = dW_r1 + 0.5 × dW_r2b on the H2O base, stored as one concatenated rank-256 LoRA and merged.

  • Round 1: distillation with KL replay toward the frozen H2O base.

  • Round 2b: a continuation on round-1 errors, blended in at 50%.

  • SHA-256 hashes:

    component	sha256
    base	c65443b1d28891ffc4b6f7d90028c542d6ccb24c2f9cd6a46a8294a44ab0ffae
    round-1 adapter	d1b0c627184d4600798470926366777e7a6109729b4b399593051b1f1f5cd8c4
    round-2b adapter	691de39e07456f2a77108ad0bf809523d4fd032596f43b149562480f0a382dd1
    blended adapter	cf13e392780223d7ecef0a39a4326357f7fb32d7e963c23cf4e4503f40967524

Teachers and labels

teacher	role	licence / terms
DeepSeek V4.1-Flash (API, thinking on)	verbalized per-option probabilities on hard-tier items (target teacher)	DeepSeek API terms
Qwen3.8-27B-FP8 (thinking, low effort)	labels on round-1 error items (round 2)	Apache-2.0
Chon-1.0-12B-Fast	logit teacher where no DeepSeek label exists	Apache-2.0
Torchcast-4B v1.2	yes/no second vote only, never a target	Apache-2.0

• Teacher probabilities are mixed with gold and are used only where the teacher agrees with reliable gold.
• Gold from programmatic or executed sources is authoritative.

Training data

• Sources: public decision-style datasets under permissive licences (CC0, CC-BY, MIT, Apache-2.0).
  • Round 1: legal, scientific, intent, policy and response-quality judgement sets.
  • Round 2 added tool and API selection (ToolBench, ToolACE, Seal-Tools, API-Bank), financial tables (TAT-QA, FinQA), executed SQL results (Gretel synthetic text-to-SQL), contracts (ContractNLI, CUAD, MAUD), state tracking (MultiWOZ 2.2, Taskmaster-3, TextWorld, StateBench) and a policy control (Aegis 2.0).
  • Counterfactual variants were re-executed or re-derived so that their labels are verified.
• No JevBench item text and no answers were used for training or for model selection. Training rows passed an 8-gram overlap filter against the public items. OpenDecisions was used for evaluation only.

Results (our held-out lockbox: 1,200 items never used for training or fitting)

model, as served	Intelligence (JevBench v1.6-style scorer)
H2O-Lightning-4B v1.1, stock readout	57.92
Philly Decider v2	61.96 (served through the real vLLM + shim path)
Philly Decider v2, also serving the 12 items over 6,144 tokens	62.94

These numbers come from our own evaluation set and scorer, not from the JevBench board.

Readout (served)

setting	value
temperature, choice	0.798298
choice position bias (logits added to options 1..8 before T)	[0.0, 0.15, 0.2, -0.15, -0.05, 0.0, 0.0, 0.0]
temperature, noul	0.933033
noul floor	0.801 (as H2O)
score readout	mirror: p = softmax(logits / 0.7), m = the median level of p; serve (p + p reflected about m) / 2, keeping on m the mass of levels whose mirror falls off the scale; the mean is m

These values were fitted on a held-out calibration set, never on JevBench items. H2O v1.1 serves T 0.8 for every type with the 0.801 floor.

Serving

bash serve.sh runs H2O's script: stock vLLM 0.30.0 serves this folder, with the shim on POST /v1/systemone, the same way H2O serves v1.1. One forward pass per decision; no generated tokens, no ensembles, no retries.

Weights

file	sha256
model.safetensors	70941015cc8d822bfba82134dd9b86e79e722df4c1be9bc39a8c99755322138e

Licence and attribution

Apache-2.0. LICENSE is H2O.ai's licence file, kept unchanged. Our modifications to the shim are marked [v37] and listed in NOTICE.