Gimbal-27B

An open decision model that reads text, then answers choice, yes/no and score questions with calibrated probabilities, one typed answer per question.

Beats hosted Jev on all three task types of the independent S1MB benchmark (137 benchmarks), is the most consistent model on JevBench (91.3, against 82.8 for hosted Jev), and is +6.1 points over hosted Jev on our sealed held-out set.

Named after the gimbal, which keeps a camera steady however it moves. Gimbal's answers stay put when a question is reworded or its options are reshuffled.

Gimbal-27B is Qwen/Qwen3.8-27B fine-tuned with LoRA as a 3-class cross-encoder. This repo holds the merged bf16 weights, ready to serve with vLLM, plus the original adapter. You describe the decision in plain words at request time, with your own labels, and get back a probability for every option. It needs no task-specific training, and there's no free-form text to parse.

{
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry":   {"type": "noul",   "instructions": "Is the customer angry?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}

The request and response follow the Jev-style systemone format, so clients written for that format work unchanged.

Quick start

python3.12 -m venv .venv && source .venv/bin/activate
pip install huggingface-hub==1.33.0
hf download Superwheat/Gimbal-27B --local-dir gimbal-27b
pip install -r gimbal-27b/helper/requirements.txt
PYTHONPATH=gimbal-27b/helper python -m gimbal_serve.server --model gimbal-27b --port 8000

Then POST the request above to http://127.0.0.1:8000/v1/systemone. It needs Linux, an NVIDIA GPU with about 51 GB free and a CUDA 13 driver, and the Python 3.12 headers. serve/SERVE.md has the full guide: flags, limits, rebuilding the merge from the adapter, and the equivalence tests.

Why Gimbal

• Consistent. Each option is judged on its own against the text, so reordering or rewording the options barely moves the answer. It scores 87.5 on JevBench option-order stability, against 76.4 for hosted Jev and 73.6 for OpenJev 27B.
• Calibrated. A 0.8 means about 80%. Expected calibration error on jev-dev choice questions is 1.9 with the shipped calibration (2.0 with the original one), against 8.5 for OpenJev 27B.
• Strong on held-out skills: math, logic, rule application and record reading, on templates and domains it never trained on.
• Open weights, Apache-2.0.

Results

S1MB English: 137 independent benchmarks

S1MB is a third-party benchmark for typed decision models: 137 English benchmarks, 26,269 decisions, scored by its own code.

S1MB english-v1 (baseline-adjusted, 0–100)	Gimbal-27B	Hosted Jev 1.13
Yes/no	72.7	64.6
Choice	72.4	67.2
Score	58.8	46.9

Complete coverage, zero failed decisions, nothing tuned on S1MB. These numbers come from a fresh full run on 2026-10-28 with the shipped calibration.json (calibration v2). It was fit before that run on our development data, with every case that matches or closely resembles an S1MB input removed. The first run, made with the original calibration, scored 72.72 / 72.38 / 58.84 against 72.65 / 72.40 / 58.83 for this run; it is preserved in the submission history. Training and calibration exposure, benchmark by benchmark, is in benchmarks/s1mb/training-exposure.md. The submission is pending review.

Our own evaluation

All models were measured with the same harness, on the same cases. Hosted Jev was measured through its public API for benchmarking only; none of its outputs were used for training. Intervals are 95% paired cluster-bootstrap intervals.

Typed decisions: the same 3,885 questions for every model

The questions are jev-dev and broad-dev: choice, yes/no and score questions drawn from hundreds of public datasets and generators, all held out from training.

Model	Accuracy
Gimbal-27B	79.0%
Gimbal 9B (r9b11)	76.1%
OpenJev 27B	75.7%
Hosted Jev 1.13	73.6%
OpenJev Flash 9B (FP8)	70.0%

On the sealed held-out halves, which were scored once on the final model only: Gimbal-27B 80.9% vs hosted Jev 1.13 74.8% on the same 3,836 questions, a difference of +6.1 [+4.8, +7.4].

Consistency: JevBench-mini

Model	Coherence	Gold accuracy (1,938)	Option-order stability
Gimbal-27B	91.3	90.9%	87.5
Gimbal 9B (r9b11)	86.8	90.2%	86.1
Hosted Jev 1.13	82.8	90.4%	76.4
OpenJev 27B	82.7	90.4%	73.6
Hosted Jev 1.14	81.9	90.3%	75.0
OpenJev Flash 9B (FP8)	75.9	89.1%	68.1

Pre-registered comparison against hosted Jev

Five primary metrics were fixed before the final scoring, and a result counts only when its 95% interval excludes zero.

• Claimed: typed accuracy on the sealed halves and JevBench consistency.
• Level, within noise: unseen task families (+1.4) and JevBench gold accuracy (+0.5).

Held-out skills

Test	Gimbal-27B	OpenJev 27B	OpenJev Flash 9B	Gimbal 9B
Math (gate-math)	80.9	75.9	59.2	75.2
Arithmetic composition, held out (choice)	96.9	80.7	59.4	74.5
Formula decisions, unseen families (choice)	50.6	42.5	41.2	39.6
Logic, held-out combinations (choice)	88.6	73.3	68.5	89.1
Rule menus, held-out shapes (choice)	97.4	66.7	45.2	82.2
Meta questions, held-out templates (choice)	81.5	50.0	48.1	66.7
Images (choice / yes-no)¹	95.9 / 80.6	96.7 / 84.9	96.3 / 82.0	95.7 / 80.1

¹ Our own image eval (1,500 cases). Gimbal-27B was not trained on images: for this row its text-trained adapter was run with the base model's vision tower, using our training code. The released checkpoint and server are text only, so this row can't be reproduced from this repository.

Bias and calibration

	Gimbal-27B	Gate
Civil statements wrongly flagged as offensive	0.2%	≤ 5%
Abusive-content recall	100%	≥ 90%
Topics with a viewpoint gap over 5 points (24 tested)	0	0

Expected calibration error (lower is better)	Gimbal-27B	OpenJev 27B	OpenJev Flash 9B
jev-dev choice / noul / score	2.0 / 2.8 / 2.8	8.5 / 3.3 / 7.4	5.7 / 7.3 / 4.7
broad-dev choice / noul / score	2.9 / 1.9 / 4.7	11.1 / 5.7 / 17.9	8.0 / 5.6 / 13.9

Gimbal-27B's values above use the original calibration. With the shipped calibration v2, recomputed on CPU from the same stored predictions, they are 1.9 / 2.8 / 2.8 on jev-dev and 3.2 / 1.9 / 4.7 on broad-dev (benchmarks/calibration-v2-ece.json).

Speed

One request at a time, B200, HTTP	Median	p95
Gimbal-27B, vLLM bf16 (this repo)	70 ms	88 ms
Gimbal-27B, plain PyTorch bf16	211 ms	213 ms
Hosted Jev 1.13 (client-side)	131 ms	180 ms
OpenJev 27B, vLLM FP8	59 ms	62 ms

vLLM serving isn't bit-identical to the PyTorch reference. No metric is worse, and on 5,321 dev questions it changes 27 decisions (0.5%), against 11 when the reference is simply rerun. All but one of those changes are near-ties. Details are in serve/SERVE.md.

Known weaknesses

• Multi-step formula application is 50–62%. Given the computed values, the model reaches about 99%, so the gap is in-context arithmetic over formulas.
• Two unseen slices still trail hosted Jev: fresh-family choice (77.1 vs 80.0) and retrospective-family score (35.7 vs 40.9).
• Hardware: the bf16 weights need about 51 GB of GPU memory. An FP8 build was tested and not published, because it changed 1.6% of decisions and wasn't faster.

Files

Path	Contents
model-*.safetensors, config.json, tokenizer files	merged bf16 checkpoint (Qwen3_5ForSequenceClassification, text only, float32 score head)
tokenizer-train.json	the exact tokenization used in training; the helper loads it, so any transformers version encodes identically
calibration.json	fitted per-kind temperatures (choice 1.267, noul 1.888, score 1.207), refit on 2026-10-28 with the 122 development cases that match or closely resemble S1MB inputs removed
calibration-2026-10-25.json	the original temperatures (choice 1.281, noul 1.895, score 1.211); the evaluation numbers on this card were measured with it, except the S1MB table and the v2 calibration errors
EXPORT_RECEIPT.json, gimbal_config.json	merge provenance, file hashes and serving settings
helper/	the gimbal_serve package: HTTP server, request schema, calibration, merge tool, pinned requirements
serve/SERVE.md	serving guide with speed and equivalence results
adapter/	the original LoRA plus head weights for Qwen/Qwen3.8-27B (revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0)
benchmarks/	full comparison table (md/json), pre-registered claims (dev and sealed), per-model aggregate reports, bias, latency, serving equivalence, provenance, calibration v2 errors, and the S1MB training and calibration exposure (benchmarks/s1mb/)
assets/	the charts above

The sealed test cases and raw hosted-API responses aren't included.

Training

• Phase 1: LoRA (rank 16, all layers) on Qwen3.8-27B, 1,000 updates.
• Phase 2: continued to one full epoch of the curriculum, 8,979 updates in total (1,138,104 examples).
  • Warmup-stable-decay schedule: hold at 5e-5, then linear decay to 1e-5 over the last 20%.
  • bf16 LoRA compute with fp32 master weights.
  • 4× B200, batch 128.
• Losses: hard labels, Brier 0.1, contrast 0.5.
• Curriculum: public decision datasets plus our own generators: record reading, policy menus, meta questions, arithmetic, logic, formula decisions, consistency groups and civility. Every eval and sealed set was screened out of training by id and content.
• Final model: chosen by a pre-registered dev-only rule. The sealed set was scored exactly once.

License

Apache-2.0, the same as the base model Qwen/Qwen3.8-27B. Use, modify and redistribute it, commercially or not, under that license's terms.

Training data terms. The curriculum combined public datasets with our own generated data. Some of those datasets don't allow commercial use, for example ANLI and BeaverTails (CC BY-NC 4.0), SciQ (CC BY-NC 3.0) and TabMWP (CC BY-NC-SA 4.0), and for others we found no stated license. We can't tell you whether their terms carry over to a model trained on them, so check this if it matters for your use.

Credits

• Data: the training data includes 23,937 rows from the train split of the release-v2-redistributable config of ZefanCai/Open-Jev at revision c67699e13d0ae25e35b77165a4b6b079bedc8aba, selected from 33,788 rows we filtered from it (customer-control rows excluded, exact duplicates and over-length rows dropped, game and painting rows downsampled); rows whose input states appear in our eval sets were left out. Open-Jev dedicates its original generated records under CC0-1.0; its third-party notices say the dedication doesn't relicense upstream wording, sources or other third-party material.
• Training objective: the hard labels with a Brier term of 0.1 came from studying ZefanCai/Open-Jev-9B (Apache-2.0). Its weights and predictions were not used.