StandardThinking/StandardOne-8B-SH-GGUF

Standard One 8B SH — GGUF

Version: v4 (2026-10-13)

GGUF builds of the backbone of Standard One 8B SH v4 (mistral3 architecture, text model) for use with llama.cpp, together with the schema head and a small runner that serves the same POST /v1/systemone contract on CPU, Apple silicon or any other llama.cpp backend.

The schema head does not read text output. It reads two hidden states for every token — the final layer (after the last norm) and the output of decoder layer 25 — so stock llama-server cannot run it on its own. The runner here takes both states from llama.cpp through its evaluation callback (runner/sh_states.cpp) and runs the head in PyTorch.

Files

File	Quant	Size
StandardOne-8B-SH-BF16.gguf	BF16 (no quantization)	17.0 GB
StandardOne-8B-SH-Q8_0.gguf	Q8_0	9.0 GB
StandardOne-8B-SH-Q6_K.gguf	Q6_K	7.0 GB
StandardOne-8B-SH-Q5_K_M.gguf	Q5_K_M	6.1 GB
StandardOne-8B-SH-Q5_K_S.gguf	Q5_K_S	5.9 GB
StandardOne-8B-SH-Q4_K_M.gguf	Q4_K_M	5.2 GB
StandardOne-8B-SH-Q4_K_S.gguf	Q4_K_S	5.0 GB
StandardOne-8B-SH-Q4_0.gguf	Q4_0	4.9 GB
StandardOne-8B-SH-IQ4_XS.gguf	IQ4_XS (importance matrix)	4.7 GB
StandardOne-8B-SH-Q3_K_M.gguf	Q3_K_M (importance matrix)	4.2 GB
StandardOne-8B-SH-IQ3_M.gguf	IQ3_M (importance matrix)	4.0 GB
StandardOne-8B-SH-Q2_K.gguf	Q2_K (importance matrix)	3.4 GB
StandardOne-8B-SH-IQ2_M.gguf	IQ2_M (importance matrix)	3.1 GB
imatrix.gguf	importance matrix used for IQ4_XS, Q3_K_M, IQ3_M, Q2_K and IQ2_M	5.3 MB
head/	schema head, float32 (70.6M parameters) and its config	282 MB
tokenizer/	tokenizer and chat template used to render requests (empty default system message)	33 MB
runner/	sh_states.cpp (state extractor), serve_head_gguf.py (server) and the shared request/head code	—

BF16 down to Q4_0 are plain llama-quantize builds; IQ4_XS, Q3_K_M, IQ3_M, Q2_K and IQ2_M use an importance matrix (imatrix.gguf, llama-imatrix -c 2048 over our calibration corpus of decision requests, the same recipe as the StandardOne GGUF repos). SHA256 checksums: SHA256SUMS. Source and conversion details: release-manifest.json.

Quick start

# 1. llama.cpp (tested at commit 53ed051) and the state extractor
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp && git checkout 53ed051
cmake -B build && cmake --build build -j
hf download StandardThinking/StandardOne-8B-SH-GGUF StandardOne-8B-SH-Q8_0.gguf \
  --include "head/*" "tokenizer/*" "runner/*" --local-dir ../sh-gguf
g++ -O2 -std=c++17 -Iinclude -Iggml/include ../sh-gguf/runner/sh_states.cpp \
  -Lbuild/bin -lllama -lggml -lggml-base -Wl,-rpath,$PWD/build/bin -o ../sh-gguf/runner/sh-states
cd ../sh-gguf

# 2. the server (torch CPU build is enough)
pip install torch transformers safetensors numpy tokenizers
python runner/serve_head_gguf.py --gguf StandardOne-8B-SH-Q8_0.gguf --head head --tokenizer tokenizer \
  --sh-states runner/sh-states --port 30171 --threads 16

Requests and responses are the same as for StandardOne-8B-SH. The runner serves one request at a time (one llama.cpp context); --context sets the longest request in tokens (default 34,000).

Validation

Measured 2026-10-13 through the runner with a ROCm (HIP) build of llama.cpp 53ed051 on one AMD Instinct MI350X, against the official Decision Index 0.3 run of the BF16 release (StandardOne-8B-SH v4) on the same requests. Same sample as the FP8 check: 150 random requests from each of six benchmarks where the model is strong (Home appliance simulator, ContractNLI, HoVer, CLINC150, BANKING77, POP909-CL) and six where it is weak (MuSR, WinoGrande, GSM8K, CRUXEval, ChessBench, MMLU-Pro); ContractNLI has 123 in total.

BF16 reference (the official Decision Index run): 93.09 % on the strong benchmarks, 52.35 % on the weak ones.

Build	Accuracy, strong	Accuracy, weak	Same answer, strong	Same answer, weak	Same answer, all (5,019 questions)
Q8_0	93.11 %	52.47 %	99.93 %	98.15 %	99.64 %
Q6_K	92.94 %	52.47 %	99.67 %	95.56 %	99.00 %
Q5_K_M	92.92 %	52.96 %	99.67 %	92.96 %	98.59 %
Q5_K_S	92.99 %	53.21 %	99.64 %	92.59 %	98.51 %
Q4_K_M	93.01 %	53.09 %	99.19 %	87.90 %	97.37 %
Q4_K_S	93.09 %	52.72 %	99.26 %	86.54 %	97.21 %
Q4_0	93.01 %	53.33 %	98.93 %	86.30 %	96.89 %
IQ4_XS	92.99 %	53.21 %	99.14 %	88.27 %	97.39 %
Q3_K_M	92.66 %	52.22 %	98.50 %	83.95 %	96.15 %
IQ3_M	93.06 %	51.36 %	98.55 %	81.73 %	95.84 %
Q2_K	91.87 %	51.98 %	96.48 %	71.73 %	92.49 %
IQ2_M	92.02 %	49.88 %	96.84 %	73.58 %	93.09 %

Q8_0 is the recommended build. Down to Q4_0 (and IQ4_XS), accuracy moves by at most 1.0 points from BF16, in both directions; the answers that change are mostly ones the model was unsure about. Q3_K_M and IQ3_M move by at most 1.0 points. Q2_K and IQ2_M lose up to 2.5 points and change a quarter or more of the uncertain answers; use them only when memory is the limit.

Limitations

• Text input only (no vision projector in this repository).
• CPU prefill is slow for long requests: Q8_0 reads about 140 tokens per second on 16 server cores, so a 3,000-token request takes about 20 seconds.
• The runner is a reference implementation, not an inference engine: one request at a time, no batching.

License

Apache-2.0, the same terms as StandardOne-8B-SH and Standard One 8B, built on Ministral 3 8B (Apache-2.0).