MrScratchcat/Aurelis-2B-GGUF

Aurelis 2B GGUF

GGUF files of Aurelis 2B, a 2.5B decision model: it reads a state and a set of typed questions (multiple choice choice, or yes/no noul) and returns a probability for every option, one forward pass per question, over the /v1/systemone interface of Ollama and llama.cpp. Every file carries the decision type pplx-decider and the trained prompt as tokenizer.chat_template.systemone, so it works without a Modelfile template.

Q8_0 is the build that mrscratchcat22/aurelis:2b serves on ollama.com and that scored 51.29 on the Decision Index 0.3 public suite (run files). The other sizes were made from the F16 export of the same merged weights.

file	size	accuracy	choice (4,556)	yes/no (114)	score (58)	same top answer as Q8_0
Aurelis-2B-Q4_K_M.gguf	1.56 GB	80.4%	80.8	79.8	50.0	93.7%
Aurelis-2B-Q5_K_M.gguf	1.81 GB	80.7%	81.0	81.6	53.4	96.1%
Aurelis-2B-Q6_K.gguf	2.07 GB	80.8%	81.2	78.9	51.7	98.1%
Aurelis-2B-Q8_0.gguf	2.68 GB	80.6%	80.9	82.5	53.4	—
Aurelis-2B-F16.gguf	5.04 GB	80.9%	81.2	81.6	55.2	98.9%

Measured on the 4,728 held-out questions of the training mix (none of them from a benchmark's test set), one question per /v1/systemone request through Ollama; accuracy counts the top answer. Every size scores within half a point of Q8_0 overall (the yes/no and score columns have few questions, one question is 0.9 and 1.7 points); the smaller the file, the more individual answers change: Q4_K_M picks a different top answer than Q8_0 on about 6% of questions.

Use

Ollama (0.40.5 or newer), straight from this repo (the repo's params file sets the 16,384-token window):

ollama pull hf.co/MrScratchcat/Aurelis-2B-GGUF:Q4_K_M
curl -s localhost:11434/v1/systemone -d '{
  "model": "hf.co/MrScratchcat/Aurelis-2B-GGUF:Q4_K_M",
  "state": "Customer: my card was charged twice for one order and I need the money back before Friday.",
  "questions": {
    "route":  {"type": "choice", "instructions": "Which team handles this?",
               "criteria": {"billing": "Billing", "technical": "Technical support", "account": "Account"}},
    "urgent": {"type": "noul", "instructions": "Is this urgent?"}
  }
}'
# Q4_K_M: route -> billing 0.65 (technical 0.19, account 0.17), urgent 0.90

Any other size works the same way (:Q5_K_M, :Q6_K, :Q8_0, :F16). Or from a downloaded file, with this Modelfile next to it:

FROM ./Aurelis-2B-Q4_K_M.gguf
PARAMETER num_ctx 16384
CAPABILITY decision

then ollama create aurelis -f Modelfile and the same request with "model": "aurelis".

llama.cpp (b11669 or newer): llama-server -m Aurelis-2B-Q4_K_M.gguf -c 16384, then the same request to http://localhost:8080/v1/systemone.

At most 24 options per question; prompts are refused, never shortened, when they do not fit the window. Transformers weights, the prompt format and training details are in the main repo.

License

Apache-2.0, as the base model.