OpenAdminOS/openadmin-8b

OpenAdmin 8B

An open-weight model for Microsoft 365 administration — Intune, Entra and Defender — fine-tuned from Ministral 3 8B by the OpenAdminOS community.

4.9 GB at Q4_K_M. Runs on an 8 GB machine, CPU-only at roughly 14-16 tokens/second on a mini-PC. No GPU required.

What it is for

Grounded, read-only administrative work: answering questions about your tenant from documentation and Graph data, planning API calls with least-privilege scopes, and refusing to act on destructive requests. It is the local tier of a two-model setup; complex multi-step agentic work escalates to OpenAdmin 20B or a hosted provider.

What we optimised for, and why

Most of the effort went into three behaviours that matter more than raw scores when a model advises someone with production tenant access:

• It does not invent Microsoft features. Asked about a setting that does not exist, it says so instead of producing a plausible default value, licensing requirement and portal path. An earlier checkpoint scored our highest number and did exactly that; it was not released.
• It refuses to make changes, for the right reason. Destructive requests get a safety refusal naming the blast radius and pointing at the confirmation flow — not "I couldn't find that", which an earlier checkpoint produced and which is safe but incoherent.
• It answers ordinary questions directly. Definitions and comparisons are answered from knowledge; version numbers and limits it is unsure of are deferred to documentation.

Evaluation

Scored mechanically — schema validation, exact match, regex constraints, tool call verification. No LLM judges. Full harness, task set and results are public in the pipeline repository.

Behaviour suite (162 tasks, never used for training decisions)

category	tasks	score
Abstention — refuses to invent	24	23 / 24
Abstention — honours exact-reply contracts	12	12 / 12
Write-safety — with context	17	17 / 17
Write-safety — bare chat	17	17 / 17
Identity	30	27 / 30
Graph call planning	32	27 / 32
Answer quality — comparisons	20	19 / 20
Answer quality — data handling	10	8 / 10
Total	162	150 / 162

Every category has at least 30 tasks except where noted, and the naming pools are disjoint from the training data. This suite exists because our original one had categories of 4 and 5 tasks, where a single item was worth 25 percentage points and told us almost nothing.

The abstention miss is a correct refusal our regex did not match, not a fabrication: zero of the 24 invented-feature probes produced an invented answer. The identity and data-handling misses occur without a system prompt; the shipped prompt (below) states the base model and the local-execution facts, and with it all sixteen items of our release smoke script pass.

A higher-scoring checkpoint (154/162) was rejected for this release because three of its abstention answers invented default values and licence requirements. We would rather ship the model that scores four points lower and does not make things up.

Known limitations

• Multi-step agentic trajectories are the weak spot. On long tool-use chains it is measurably behind OpenAdmin 20B. Escalate that work.
• It is not a documentation database. Facts come from retrieval at query time. Without a documentation index in the prompt it answers from training memory, which is frozen at the base model's cutoff. Retrieval is worth roughly 30 tasks on our suite; use it.
• Graph planning covers the common Intune, Entra and Defender surface, not the whole API.

Usage

ollama run openadminos/openadmin-8b

Or with llama.cpp:

llama-server --model openadmin-8b-Q4_K_M.gguf --ctx-size 16384 --jinja

A default system prompt ships with the Ollama image and is included as system-prompt.txt. It matters: it tells the model to answer concept questions directly and to defer on uncertain specifics.

Training data

Synthetic and machine-validated before admission. Agent manifests are checked against the product's JSON schema, Graph plans against a curated endpoint table, and fleet-reasoning arithmetic is computed by the generator so it is correct by construction.

No tenant data. No scraped conversations. No distillation from proprietary model APIs. The dataset is published at OpenAdminOS/openadmin-sft.

Honest notes on how this was built

Sixteen training runs. Several scored well and were rejected: one fabricated documentation, one narrated its reasoning at the user, one answered a request to wipe 200 devices with "I can't find that number in the supplied data". Each was caught by reading the model's actual output, not by a benchmark.

The run log — including every failure and what it taught us — is published alongside the pipeline. We think that record is more useful than a headline number, and we have invalidated one of our own headline numbers already when a broader task set showed it was a tie.

Licence

Apache 2.0, matching the base model. Microsoft, Intune, Entra and Defender are trademarks of Microsoft Corporation; this project is not affiliated with or endorsed by Microsoft or Mistral AI.