damianborek/polaris-3

Polaris 3

Polaris 3 (polaris-3) is the autonoxis decision model: a LoRA adapter for bespokelabs/Bespoke-Nimble-9B (revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a), itself built on Qwen3.5-9B. It makes "conductor" decisions for autonomous coding agents. It does not generate text. Nimble scores the answer tokens and the adapter returns a probability for each allowed label.

It replaces Polaris 2. Polaris 3 performs about the same as Polaris 2 on real orchestrator packets (seed averages 77.1% vs 76.3% on three-model labels, within seed-to-seed variation), with a slightly more reliable confidence gate (79.8% vs 78.4%); it is the first Polaris trained on real-derived (rewritten, redacted) data. These are means over 8 Polaris 3 and 4 Polaris 2 training seeds on 94 real packets (v9, labels = majority of three frontier models). The released Polaris 3 seed scores 75/94, which may be optimistic (see Selection note); the released Polaris 2 seed scores 73/94 and Polaris 1 61/94. It answers every packet of v5–v8 correctly.

Two tracks, one question per request:

track	labels
decision: what should the conductor do next with this lane?	STOP, ASK, DISPATCH
manager: what should happen to the evidence returned for this lane?	ACCEPT, VERIFY, REJECT, REOPEN, ESCALATE

The question wording and the label criteria the adapter was trained on are in conductor-questions.json. They are the same questions as Polaris 2, so any client that works with Polaris 2 works with Polaris 3. The adapter's autonoxis.json carries its name (polaris-3), which autonoxis-server reports.

The 0.8 gate is not a guarantee. Three v9 mistakes are unsafe (a wrong ACCEPT or DISPATCH) at confidence ≥ 0.96, so the gate does not stop them. Treat ACCEPT as "verify first".

Links

• Polaris 2 model (previous): https://huggingface.co/damianborek/polaris-2
• Polaris 1 model (older): https://huggingface.co/damianborek/polaris-1
• Vega 1 model (earlier, generative): https://huggingface.co/damianborek/vega-1
• Pi plugin + server: https://github.com/damian87x/pi-autonoxis-model
• Claude Code plugin: https://github.com/damian87x/autonoxis-model

Use with the autonoxis plugins

The easiest way to use Polaris 3 is through the autonoxis server and one of its two clients: a Pi extension (pi-autonoxis-model 0.5.0 or later) and a Claude Code plugin (autonoxis-model 0.4.0 or later). Both clients send the trained call convention for you, act only at confidence ≥ 0.8 (never lower: Pi uses a fixed 0.8; the Claude CLI -t can only raise it), reject a label that does not belong to the requested track, and on any error (server down, bad response) return an error instead of a label, so nothing acts and the decision goes to a stronger model or a human.

The questions did not change from Polaris 2, so Polaris 3 also works with pi-autonoxis-model 0.4.x and autonoxis-model 0.3.x, and Polaris 2 still works with the new versions. Polaris 1 still works with the older plugin versions (pi-autonoxis-model 0.3.x, autonoxis-model 0.2.x).

1. Run the server. server.py and its full setup (NVIDIA GPU, about 20 GB in bf16; torch, transformers, peft, huggingface_hub) are in pi-autonoxis-model/server:

git clone https://github.com/damian87x/pi-autonoxis-model && cd pi-autonoxis-model
git clone https://github.com/bespokelabsai/nimble server/nimble
hf download bespokelabs/Bespoke-Nimble-9B --revision 594dfdcfb6f94e3d0c0db7535180d3c71689169a --local-dir base
hf download damianborek/polaris-3 --local-dir adapter
echo '{"model_path": "base", "max_input_tokens": 2048}' > nimble-model.json
python server/server.py --model-config nimble-model.json --adapter adapter --port 8765

It listens on http://127.0.0.1:8765 (loopback only, no authentication). Both clients read AUTONOXIS_URL and default to that address. GET /health returns {"status": "ok", "model": "polaris-3"}.

2. Pi.

pi install git:github.com/damian87x/pi-autonoxis-model

Tools: autonoxis_conductor ({track: "decision" | "manager", packet} → label, confidence, probabilities, act) and autonoxis_evaluate (raw Jev {state, questions}). Command: /autonoxis-model status | test.

3. Claude Code.

/plugin marketplace add damian87x/autonoxis-model
/plugin install autonoxis-model@autonoxis-model

(or claude plugin marketplace add damian87x/autonoxis-model and claude plugin install autonoxis-model@autonoxis-model from a shell). Then /autonoxis-model:autonoxis status | decision <packet> | manager <packet>, or the plugin's CLI directly:

python3 scripts/autonoxis.py conductor --track manager --file packet.txt
# exit 0 = ACT (conf >= 0.8), 3 = ESCALATE (below 0.8), 2 = error (never act)

4. Raw HTTP. One question with id label, state {"packet": ...}, and the instructions and criteria for the track copied verbatim from conductor-questions.json. With raw HTTP, apply the 0.8 gate yourself.

How to use without the plugins

You need the Nimble prompt builder and scorer from github.com/bespokelabsai/nimble: nimble.scoring.parallel_schema.prepare_prompts and nimble.training.schema_train.candidate_logits.

The call convention matches training. Use it exactly:

• ask one question per request, with the field id label;
• the state is the JSON {"packet": <packet text>};
• the field's description and choice descriptions come from conductor-questions.json (instructions and criteria for that track).

Load the base model and attach the adapter unmerged (merging in bf16 shifts the probabilities).

import json, torch
from transformers import AutoTokenizer, AutoModelForImageTextToText
from peft import PeftModel
from huggingface_hub import hf_hub_download
from nimble.scoring.parallel_schema import prepare_prompts
from nimble.training.schema_train import candidate_logits

BASE, REV = "bespokelabs/Bespoke-Nimble-9B", "594dfdcfb6f94e3d0c0db7535180d3c71689169a"
ADAPTER = "damianborek/polaris-3"

tok = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForImageTextToText.from_pretrained(BASE, revision=REV, dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, ADAPTER).eval()   # do not merge
questions = json.load(open(hf_hub_download(ADAPTER, "conductor-questions.json")))

def ask(packet: str, track: str) -> dict:
    q = questions[track]
    field = {"type": "enum", "description": q["instructions"],
             "choices": list(q["criteria"]), "choice_descriptions": q["criteria"]}
    p = prepare_prompts(tok, json.dumps({"packet": packet}, ensure_ascii=False), {"label": field}, 2048)
    ids, cands = p.full_ids[0], p.candidate_ids[0]
    batch = {"input_ids": torch.tensor([ids], device="cuda"),
             "attention_mask": torch.ones(1, len(ids), dtype=torch.long, device="cuda"),
             "candidate_ids": torch.tensor([cands], device="cuda"),
             "candidate_mask": torch.ones(1, len(cands), dtype=torch.bool, device="cuda")}
    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
        probs = candidate_logits(model, batch)[0].double().softmax(-1).tolist()
    return dict(zip(q["criteria"], probs))

Confidence gate. Compute conf = (n * max_prob - 1) / (n - 1), where n is the number of labels. Act on the top label only when conf >= 0.8. Below that, escalate to a stronger model or a human.

Results

"Unsafe" means the model predicted DISPATCH or ACCEPT where the reference label is different. The evaluation sets are not released. eval-summary.json holds the per-set metrics for this adapter (no packet text).

v9, 94 real orchestrator packets, labels = majority of three models:

	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt (approx.)
accuracy	77.1% (sd 1.9%)	76.3% (sd 1.4%)	75/94 (79.8%)	73/94 (77.7%)	61/94 (64.9%)	70/94 (≈74.5%)
decision / manager	34.6/48, 37.9/46	34.5/48, 37.25/46	36/48, 39/46	35/48, 38/46	28/48, 33/46	34/48, 36/46
unsafe	3.1	3.0	3	3	4	7
0.8 gate: kept / accuracy	85.9 / 79.8%	90.25 / 78.4%	91 / 81.3%	91 / 80.2%	89 / 66.3%	not measurable

All sets:

set	Polaris 3, mean of 8 seeds	Polaris 2, mean of 4 seeds	Polaris 3, released seed (may be optimistic, see selection note)	Polaris 2, released seed	Polaris 1	Jev contract prompt
v5 (40)	not reported	not reported	40/40	40/40	40/40	not measured
v6 (48)	not reported	not reported	48/48	48/48	48/48	not measured
v7 (48)	not reported	not reported	48/48	48/48	48/48	not measured
v8 (60)	not reported	not reported	60/60	60/60	58/60	60/60
v9, 3-model labels (94)	77.1%	76.3%	75/94 (80%)	73/94 (78%)	61/94 (65%)	≈70/94 (≈74%)
v9, original 2-model labels (96)	68/96 (70.8%)	67.25/96 (70.1%)	71/96 (74%)	69/96 (72%)	57/96 (59%)	70/96 (73%)

• v9 is 96 packets from real orchestrator runs (48 decision, 48 manager). Its reference labels are model labels, not human labels. Opus 5.5 (claude-opus-5-5), Grok 4.7 and Astra (gpt-6-astra) each labelled every packet blind; the reference is the label at least two of them chose. They were unanimous on 68, split 2–1 on 26, and all three differed on 2; those 2 packets have no reference and are left out (Polaris 3's answer on both matches one of the three votes).
• The original 2-model labels row is kept for continuity with the Polaris 2 card: Opus 5.5 and Grok 4.7 labelled blind and Opus 5.5 decided their 23 disagreements. With the third labeller, 6 of those labels changed and 2 packets were left with no majority.
• "Polaris 1" is the released Polaris 1 with its own questions. "Jev contract prompt" is untrained TypeSafe Jev given the same questions Polaris 2 and 3 were trained with. Only Jev's misses on v9 were kept, so its 3-model score assumes its other answers equal the original labels; it is approximate, and its gate cannot be recomputed.
• Polaris 3 v9 unsafe (3-model labels): two VERIFY packets answered ACCEPT (confidence 0.991 and 0.999) and one ASK packet answered DISPATCH (0.965). Polaris 2 makes the same three mistakes.
• Calibration holdout: 107 packets written from real agent-session moments (see What changed), never trained on: 102/107 (95.3%), 0 unsafe; with the 0.8 gate 106 kept at 95.3%.
• v5–v8 reference labels are Fable 5 (claude-fable-5). v8 was never used for training, but it is in-distribution. v5–v7 are not clean held-out sets (training examples were written from earlier mistakes on them). On v5–v8 Polaris 3 makes no unsafe predictions.
• Eight training seeds were run. On v9 (3-model labels) they score 69–75 of 94 (mean 77.1%).

Selection note. v9 was not used for training, and it is not in the selection key: best v5–v8 total, then most calibration-holdout packets correct, then lowest calibration-holdout log-loss. But seeds 1, 3 and 5 tied on the first two, and the log-loss tie-break was chosen after a per-seed table showing each seed's v9 score had already been viewed. It picked seed 1, which is also the best seed on v9. So the released seed's v9 figures (75/94, 79.8%) may be optimistically biased. The unbiased estimate for the Polaris 3 recipe is the 8-seed mean: 77.1% (sd 1.9%) on the 3-model labels, 3.1 unsafe, 0.8 gate 85.9 kept at 79.8%; 68/96 on the original labels. The fair comparison is with the Polaris 2 4-seed mean on the same labels: 76.3% (sd 1.4%), 3.0 unsafe, 0.8 gate 90.25 kept at 78.4%. (The released Polaris 2 seed was picked by v5–v8 total alone, a unique maximum, so no tie-break was needed there.)

What changed from Polaris 2

• Training data from real sessions. 293 new training rows written from moments in real agent sessions (worker reports, reviewer verdicts, owner rulings, "tests pass" claims). Every packet was rewritten into a new invented setting and redacted (names, paths, hashes, addresses, keys); rows that shared text with the source sessions or overlapped any evaluation set were rewritten or dropped. Material connected to the sources of the v9 packets was excluded.
• Soft labels. Opus 5.5 and Grok 4.7 labelled the new rows blind and agreed on 278 of 293. Where they disagreed (15 rows), the row is trained once with each label, so the model learns both are plausible.
• Calibration holdout. A further 107 rows from the same pipeline were held out, never trained on, and used to pick the seed.
• Same questions and training recipe as Polaris 2.

Limitations

• The gate is only slightly better than Polaris 2. On v9, averaged over seeds, packets kept by the 0.8 gate are correct 79.8% of the time (Polaris 2: 78.4%); for the released seeds, 91 kept at 81.3% vs 91 at 80.2%. The three unsafe answers are confident (≥ 0.96) and pass it. Treat ACCEPT as "verify first", and keep hard vetoes in code.
• Model labels, changed after training. v9 labels come from frontier models, not humans. The third labeller was added after Polaris 3 was trained and selected, so the 3-model numbers are on labels that did not exist when the model was built (see Selection note for how v9 relates to seed selection). On the original labels the gain over Polaris 2 is 2 packets (71 vs 69 of 96). Human labels could move the numbers either way.
• 2 ambiguous v9 packets have no majority label and are left out of the 94.
• No clear accuracy gain. Seed averages on v9 are 77.1% (Polaris 3, 8 seeds, sd 1.9%) vs 76.3% (Polaris 2, 4 seeds, sd 1.4%), a difference of under one packet in 94 and within seed-to-seed variation; Polaris 3 seeds range 69–75 of 94, Polaris 2 seeds 70–73. The released seeds differ by 2 of 94 (75 vs 73), but the Polaris 3 seed may be optimistically selected (see Selection note).
• The calibration holdout is in-distribution for the new training rows (same pipeline), so its 95% is much higher than v9 and should not be read as real-world accuracy.
• Privacy. The new training packets were derived from real agent sessions, rewritten and redacted as above. The training data, the calibration holdout and the evaluation sets are not released. This card and eval-summary.json contain no packet text.
• Prompts are limited to 2048 tokens. Longer packets are rejected, not truncated. English only.
• Narrow domain: conductor and manager decisions for autonomous coding lanes under one decision contract. The labels will not transfer to other policies.
• The adapter is trained for the single-question convention above.

Training

• Recipe (unchanged from Polaris 1 and 2): LoRA r=16, alpha=32, dropout 0.05, on the language-model Linear layers (attention, linear-attention and MLP projections). Candidate cross-entropy over the answer tokens. lr 5e-5, effective batch 8, 3 epochs, linear schedule with 10% warmup, bf16.
• Questions: conductor-questions.json in this repository, for both tracks (unchanged from Polaris 2).
• Data: 1704 rows. The 1396 Polaris 2 rows, plus the 293 new rows above, plus 15 extra copies for the two-label rows. No evaluation or calibration packet is in the training data (checked at build time).
• The training data is not released.

License

Apache-2.0, the same as the base model. Bespoke-Nimble-9B is itself a LoRA-merged Qwen/Qwen3.5-9B, which is also Apache-2.0.