Jeetu456/decider-2b-GGUF

Decider 2B GGUF

GGUF builds of Jeetu456/decider-2b for running locally with llama.cpp: on a Mac (Metal), a Windows or Linux PC, or a plain CPU. No PyTorch and no NVIDIA GPU needed.

File	Size	Same answer as the original (38 questions)	Largest probability difference	Confident flips
decider-2b-Q8_0.gguf (recommended)	2.0 GB	38/38	0.016	0
decider-2b-Q4_K_M.gguf (smallest)	1.3 GB	35/38	0.083	0

Reference: the original model in PyTorch float32 on CPU, 28 requests covering routing, moderation, tool choice, NLI, sentiment scores, knowledge with "I don't know", multi-question and empty-state requests (gguf_agreement.json). A BF16 GGUF matched 38/38, to 0.0024 (not uploaded).

How it works

llama.cpp runs the backbone with non-causal attention and returns every token's final hidden state. decider_gguf.py builds exactly the same prompt as the original model, applies the 4 MB decision head (decider_head.safetensors) in numpy, and returns a Jev/SystemOne response. This is the approach the community GGUF builds of Perplexity's pplx-decider use. The decision server built into llama.cpp does not know this model's head yet, so it is applied outside llama.cpp.

Run

1. Install llama.cpp (any recent build with Qwen3.5 support): brew install llama.cpp, or a release from https://github.com/ggml-org/llama.cpp/releases. These files were built with llama.cpp commit 71255fe (October 2026).
2. Download this repository: hf download Jeetu456/decider-2b-GGUF --local-dir decider-2b-gguf
3. Start the backbone. Non-causal attention is required; with the default causal attention the answers are wrong.

cd decider-2b-gguf
llama-server -m decider-2b-Q8_0.gguf --embeddings --pooling none \
  --override-kv qwen35.attention.causal=bool:false -c 12288 -b 12288 -ub 12288 --port 8081

4. Serve Jev/SystemOne on port 8000 (pip install numpy tokenizers safetensors):

python decider_gguf.py --model-dir . --serve --port 8000
curl localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "My card was charged twice and I want a refund.",
  "questions": {
    "team":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"billing": "Payments, charges and refunds", "technical": "App bugs and outages",
                            "account": "Login and profile"}},
    "urgent": {"type": "noul", "instructions": "Is this urgent?"}}}'

Or one request from the command line: python decider_gguf.py --model-dir . --request request.json. Routes: POST /v1/systemone (and /v1/decisions), GET /v1/models, /health.

Files

File	Purpose
decider-2b-Q8_0.gguf, decider-2b-Q4_K_M.gguf	Backbone for llama.cpp
decider_head.safetensors, decider_head_config.json	Decision head, temperature (1.2) and "I don't know" threshold (0.55)
decider_model.py, tokenizer.json	Request format and prompt layout (the runner uses the format part only, no PyTorch)
decider_gguf.py	Runner and /v1/systemone server
convert.sh	How these files were made from the original model
compare_gguf.py, compare_embed.c, gguf_agreement.json	Agreement check against the original model

Notes

• -ub (micro-batch) must be at least the prompt length, because non-causal attention needs the whole prompt in one pass. 12288 matches the model's pass length.
• Describe each choice option in a few words: accuracy is much better than with bare option keys.
• Set DECIDER_ABSTAIN_THRESHOLD=0 to turn off "I don't know" abstention.

License

Apache-2.0, as the original model and its base, Qwen/Qwen3.5-2B-Base.