OpenChoice-9B

CPU-first multimodal System-One-compatible decisions, no GPU required: about 5.8 GiB of serving weights, 6–7 GiB RAM, optional GPU offload, and an integrated LoRA → GGUF fine-tuning kit (tested on a 24 GB GPU).

This package contains the rank-16 OpenChoice decision adapter merged into the pinned Qwen/Qwen3.5-9B base and quantized as GGUF Q4_K_M, plus a Q8_0 vision projector. It directly returns probabilities for Choice, Score, and Noul; it does not generate answer text. The accompanying server implements the TypeSafe SDK 0.6-compatible POST /v1/systemone wire format. OpenChoice is independent of TypeSafe and is not a Jev implementation or parity claim.

The exact text file is 5.24 GiB / 5.63 decimal GB, plus a 595 MiB projector: about 5.8 GiB serving weights. The complete download is about 6.0 GiB including the tokenizer and retained training adapter. The base is a 9B-parameter model.

What OpenChoice is built for, and when to use something else

Official JevBench v1.5.1 results are pending, so this is not a quality ranking.

• OpenChoice is designed for long inputs (4,096 tokens per forward, 32,768 per decision), chart decisions on a CPU through the Python engine (the HTTP API is text-only), and your own labels and house rules: fine-tune with LoRA and export back to the same GGUF server. We'll publish benchmark results for these as they come in.
• Use a small encoder (Laya, one-click via Unsloth Desktop, or GLiNER2.5-Decide) for short-text routing on any laptop with the smallest footprint, and Laya multilingual for many languages.
• Use JevK5 v0.3 or Plumb-4B for the smallest GGUF with a published JevBench v1.5.1 score (OpenChoice's official score is pending).
• Use Jev, or Cygnet / Winnow-12B (v1.5.1 composites 72.1 / 73.7 / 73.2) for higher published accuracy; OpenChoice trails Jev on every aggregate benchmark compared. Use a generative model if you need explanations.

Quickstart: Docker (Linux containers; verified on Linux/amd64)

Use Docker Engine (Linux) or Docker Desktop with Linux containers (macOS/Windows). Recommend 16 GiB system RAM and at least 12 GiB assigned to Docker Desktop; this is not a tested minimum. Build from the public source, or replace the first three lines with docker pull ghcr.io/lutr0/openchoice:0.2.0-cpu and docker tag ghcr.io/lutr0/openchoice:0.2.0-cpu openchoice:cpu. The following single-line commands use a POSIX shell or PowerShell 7.3+:

git clone https://github.com/lutr0/openchoice.git
cd openchoice
docker build -t openchoice:cpu .
docker volume create openchoice-model
docker run --rm --mount type=volume,source=openchoice-model,target=/models openchoice:cpu download lutr0/OpenChoice-9B
docker run --rm --entrypoint python openchoice:cpu -c "import secrets; print('OPENCHOICE_API_KEY=' + secrets.token_hex(32))" > openchoice.env
docker run -d --name openchoice --env-file openchoice.env -p 127.0.0.1:8768:8768 --mount type=volume,source=openchoice-model,target=/models,readonly openchoice:cpu
docker logs -f openchoice

Keep openchoice.env private. Wait for “Application startup complete”; Ctrl-C exits log-following without stopping the service. Startup checks all package hashes and warms up. docker rm -f openchoice stops/removes the container but retains the model volume. For an existing local package, replace the volume mount with --mount "type=bind,source=/absolute/path/OpenChoice-9B-GGUF,target=/models,readonly"; see the source README for host permissions/SELinux notes.

The CPU-only image runs as non-root UID 10001, binds 0.0.0.0 internally, and therefore requires OPENCHOICE_API_KEY. Publish only the loopback host port shown above. It has no Torch, PEFT, bitsandbytes, CUDA wheels or build toolchain. No CUDA container variant or Apple Metal support is provided. The portable runtime builds all supported CPU variants with GGML_NATIVE=OFF and chooses features at runtime.

Verified scope: Linux/amd64 built and ran here; health, the request below and all 34 SDK smoke cases passed. Its image is 353 MiB excluding weights and startup to health was 7.39 s on a shared Ryzen desktop with warm filesystem caches. Linux/arm64 was built and smoke-run under QEMU emulation only, not verified on Apple Silicon/native arm64 hardware. Both default SVE/SME and Cortex-A76 emulated CPUs passed health and returned the curl request, but both full SDK runs exceeded the unchanged 180 s per-request timeout. macOS/Windows Docker Desktop host integration was not exercised. The amd64 container/native maximum absolute probability delta was 0.0012598 on this example: portable CPU variants/GCC 14.2 versus GGML_NATIVE=ON/GCC 16.2, not bit-identical execution. See container-verification.json for exact outputs, emulation results and limitations.

In a second POSIX terminal, load the key and send a decision:

. ./openchoice.env
curl --fail http://127.0.0.1:8768/health
curl --fail http://127.0.0.1:8768/v1/systemone -H "Authorization: Bearer $OPENCHOICE_API_KEY" -H 'Content-Type: application/json' -d '{"model":"openchoice-latest","state":"My card was charged twice. Please refund the duplicate payment.","questions":{"topic":{"type":"choice","instructions":"Classify the message.","criteria":{"billing":"Billing and refunds","weather":"Weather forecast"}},"severity":{"type":"score","instructions":"Rate the payment problem.","criteria":["No payment problem","A duplicate charge requiring a refund"]},"duplicate":{"type":"noul","instructions":"Was the customer charged twice?"}}}'

For PowerShell 7.3+, load the key with $env:OPENCHOICE_API_KEY = (Get-Content ./openchoice.env).Split('=', 2)[1], use curl.exe, and use $env:OPENCHOICE_API_KEY in the Authorization header. The source README includes the complete command. Expect normalized Choice/Score probabilities and a scalar Noul, not generated answer text. Do not expose the development server to untrusted networks.

Native Linux alternative

Linux x86-64, Git, Make and a C++17 compiler are required; uv supplies Python 3.12 and pinned CMake. Debian/Ubuntu prerequisites: sudo apt-get install git build-essential; Fedora: sudo dnf install git gcc-c++ make. After cloning the source as above:

bash scripts/build_llamacpp.sh
uv run hf download lutr0/OpenChoice-9B --local-dir ./OpenChoice-9B-GGUF
uv run python -m openchoice.server --model ./OpenChoice-9B-GGUF --port 8768

No separate sync step is needed. The script builds llama.cpp commit 4364bf7232e65c34eca8d9500c5464389662de6b locally; no full upstream BF16 download is needed. Native builds target the current CPU and are not portable between CPUs. Native loopback serving does not require an API key; non-loopback serving does. The native build script supports Linux only; native macOS/Windows, AMD GPU and Metal paths are not verified.

Optional native NVIDIA offload requires a CUDA toolkit and compatible driver: build with bash scripts/build_llamacpp.sh cuda, then add --gpu auto to the server command. The CPU instance stays resident; a second instance offloads four layers only after 60 game-free seconds and at least 4 GiB free VRAM. It releases the offload instance at request boundaries when Linux game detection or low free VRAM requires it. This is best-effort resource management, not an OOM guarantee. Four-layer measurements are about 1.9 GiB additional VRAM and about 12.35 GiB total peak RSS. Images always run on CPU.

Hardware and speed

Measured on a shared Linux desktop with a Ryzen 9 7900X3D, using 12 inference threads and CPU repacking disabled. These are observations, not hard latency guarantees or comparisons with hosted services.

Mode	Measured memory	Measured latency
CPU text, Q4_K_M	About 6.10 GiB peak RSS, including mapped weights	About 1.7–3.3 s for a 300–600-token Choice/Noul decision
CPU vision, Q4_K_M + Q8_0 projector	About 6.71 GiB peak RSS	Chart Choice mean 4.16 s; six-level Score mean 25.74 s without image-cache reuse
Optional four-layer CUDA instance	1,926 MiB peak VRAM in the recorded profile	Workload-dependent; not full-model GPU inference

Use --threads N to control CPU contention. Score evaluates each level independently; the bridge reuses aligned prefixes/image embeddings only within a request. No cross-client prompt/image cache. Recorded two-row integration verification measured a 13.22 s standalone six-level Score with image-embedding reuse; do not compare that directly with the uncached full-set mean.

Evidence is in the source repository's results/runtime/gguf-q4-norepack.json, vision-q4-q8_0.json, production-vision-verification.json, and production-gpu-verification.json, linked and qualified in RESULTS.md.

Evaluation: runtime-specific numbers

The released GGUF is not numerically interchangeable with the old NF4 implementation. On the frozen 200-row CPU feasibility sample it got 178/200, versus NF4 174/200, with 96% decision agreement. This reused sample is too small to establish improvement.

CPU vision, same Q4_K_M text + Q8_0 projector: 49/60 chart-development decisions (Choice 30/30, Score 19/30), versus NF4 52/60. Charts share a generator with image rows used in training. This does not establish broad photograph recognition, OCR, multiple-image, or video quality.

Full SciQ test on the released GGUF, CPU-only: 95.3% (953/1000), zero errors. This rerun uses the same frozen test inputs as the historical comparison. CLM-v0.1-8B NF4 previously measured 58.7% (587/1000): a 36.6-point cross-runtime accuracy gap, with no reused NF4 confidence interval or speed comparison. Public raw outputs: source results/gguf-sciq-predictions.jsonl; summary: results/gguf-sciq.json.

The 400-task panel scores 80.25% (321/400) on the released GGUF, CPU-only with zero errors. It covers 100 rows each of Aegis, BoolQ, HelpSteer2 and SNLI; those source families appeared in training. Summary/raw files in the source repository: results/gguf-panel400.json and results/gguf-panel400-predictions.jsonl.

Nimble public on the released GGUF: 75.41% micro (2926/3880); 74.72% macro (unweighted mean across 13 frozen subsets). One interrupted-runner boundary is conservatively counted wrong without retry; all remaining responses were recorded. Source summary/raw: results/gguf-nimble-public.json and results/gguf-nimble-public-predictions.jsonl. The exact frozen payload hashes and the execution interruption are disclosed there; this is native Score evaluation, not Nimble's enum/Choice rating path.

Historical NF4-only measurements, retained for context rather than attributed to GGUF:

Suite	Historical OpenChoice NF4	Comparator
SciQ official test, 1,000	95.6% (956/1000)	CLM-v0.1-8B NF4 58.7%, locally measured
400-task decision panel	79.5% (318/400)	CLM-v0.1-8B NF4 44.5%, locally measured
Nimble public, 3,880	75.0% (2909/3880)	Jev 77.3%, Nimble 75.9%, author-published
Nimble evidence-324	73.1% (237/324)	Jev 93.2%, Nimble 90.1%, author-published

Full release GGUF reruns, raw outputs, historical tables, and protocols are in source RESULTS.md. SciQ was held out from project tuning/selection, not proven absent from upstream pretraining. The 400-row panel's source families appeared in training and the panel was previously used for final evaluation. Nimble evidence-324 has synthetic/model-checked labels and is in-domain for Nimble. OpenChoice has no sealed JevBench score. Confidence is uncalibrated; published comparator results are not independent local reproductions.

Official JevBench v1.5.1: pending (sealed split requires maintainer submission). This is a 1,624-decision, four-axis composite, not raw task accuracy. ImageJevBench v0.1.3 is also not measured for OpenChoice. Legacy public-task figures appear only in the source RESULTS archive and must not be relabeled as current scores.

CPU, GGUF, probabilities-only decisions, vision, and LoRA tuning are not individually unique features: alternatives include Winnow, JevK5, Plumb, Imajev, and GLiNER2.5-Decide. OpenChoice offers a CPU-first multimodal deployment plus integrated customization/export path, not a smallest/fastest/best claim. See the source current artifact comparison for documented capabilities, sizes, exact-version caveats, and separately labeled published scores.

Vision and API limits

HTTP handles text and structured JSON, not image uploads. CPU image decisions use LlamaCppEngine.predict_rows with local image paths and SHA-256 hashes; see the source README's complete example. The native projector uses M-RoPE and the actual image encoder, not captions or image filenames as substitutes.

Choice supports 1–26 options, Score 1–10 levels; requests support up to 64 questions and 2 MiB JSON. No truncation: 4,096 tokens per forward and 32,768 tokens per decision. Over-limit/unsupported requests return 422. One worker serializes inference, excess queue entries return 503, and disconnected clients' work is cancelled between microbatches. Confidence is a concentration statistic, not a guarantee. English-centric training, option-order sensitivity, limited vision validation, and no coding/tool-trajectory evaluation remain limitations.

Fine-tune and export

The original adapter/, empty slot-model heads.safetensors, and model.json remain in this package for fine-tuning; they are not the serving backend. The slot model reads logits for option letters or Yes/No rather than a trained classifier head.

uv run --group train python -m openchoice.finetune --help

See FINETUNING.md for the validate, train, eval, and export commands. Training uses the full pinned base model plus PEFT; its GPU/RAM requirements are much higher than CPU serving. Export merges the adapter, writes Q4_K_M text weights and the projector, preserves processor files, and generates the exact portable manifest required by the server. Fine-tuned exports need their own evaluation.

Package contract and identity

manifest.json schema: openchoice-gguf-v1. It names model, projector, processor, checkpoint_config, quantization, base_model, base_revision, and llama_commit; files maps relative file paths to SHA-256. Serving rejects corrupted artifacts and paths escaping the package. Compiled local libraries are not shipped in this package. The manifest itself and card/license prose are not self-hashed.

• Text GGUF SHA-256: 6d8eef23223c2f53a772fa5a945842569e2cd646261bcb0a354e6c0f4e5ee238
• Q8_0 projector SHA-256: 44c7ddca015a856956bbf2ff4a88049a4bc1dcff47fabd93602f23554d307e0c
• Base revision: c202236235762e1c871ad0ccb60c8ee5ba337b9a
• Adapter: rank 16, alpha 32; original selected run used 100 updates and 3,200 rows / 3,234,666 tokens, seed 20260921.

License and data-rights caveat

The weights are declared CC-BY-SA-4.0 as a conservative choice because BoolQ uses CC-BY-SA-3.0 and SNLI uses CC-BY-SA-4.0. This is not a legal determination that these terms are sufficient or necessary for derived adapter/merged/quantized weights. The source code is separately Apache-2.0; the Qwen base is Apache-2.0. See LICENSE, NOTICE, and source PUBLISHING.md.

Training sources: NVIDIA Aegis AI Content Safety Dataset 2.0 human prompt labels (CC-BY-4.0), BoolQ (CC-BY-SA-3.0), NVIDIA HelpSteer2 human ratings of model-generated responses (CC-BY-4.0), Stanford SNLI (CC-BY-SA-4.0), and internally authored replay material. Benchmark inputs are not bundled as training data. Names of Jev, TypeSafe, and other models identify independent compatibility/comparisons, not affiliation or endorsement.