Weight-Space Programming on a Production Model
Can GPT-6 Astra read and repair cribl-decision-2-mini, just from its weights?
*Babak Barazandeh, Cribl AI Research Lab *
TL;DR
A recent study showed that frontier LLMs can read and write neural-network weights directly — on small, purpose-trained networks. We re-ran the idea on a model we actually ship: cribl-decision-2-mini, a fine-tuned Qwen3.5-0.8B that answers IT and security multiple-choice questions.
Astra reads the model's mechanics remarkably well — the task format, a LoRA's exact rank, which layers changed — but in ten blinded runs it never once read the semantics: the security domain, the training data, the weak categories. Handed a secretly sabotaged copy, it found and repaired the damage almost perfectly. Handed the healthy model, it gained just under one point — because there was little left in the weights to fix.
| Result | |
|---|---|
| Reading | Structure, yes: named the task and the fine-tuning method from raw weights. Meaning, no: never the security domain, in 10 runs across 3 experiments. |
| Repair | 100% and 94% recovery on two secret sabotages. Found a planted rank-1 corruption and subtracted it exactly, spending 1 of 5 forward passes. |
| Improve | +0.96 points at best, growing with the pass budget but staying modest. Read together with the repair result: the healthy model has no weight-level bottleneck for an editor to find. At this scale, editing can remove obstacles but not write in new knowledge — adding it still requires training on more in-domain data. |
Every run is blinded (checkpoints under random hex IDs, prompts scrubbed of product names, answer keys readable only by scoring scripts), fully logged, and audited — the no-forward-pass rule is checked against each transcript afterwards. No run was dropped.
Series 1 — Reading: can an LLM understand a model from its weights alone?
R1 — What is this model for?
Astra gets two anonymous checkpoints — our fine-tune and its Qwen base — told only that one is a fine-tune of the other, with their forward code and a Python sandbox over the tensors. No forward passes; linear algebra on the weight matrices only. Two conditions: hard (embeddings and output rows stripped from both, no tokenizer) and easy (full checkpoints plus a scrubbed tokenizer). Two runs each.
Result: got the task right, missed the security domain.
The task, from the hard condition at 99/100 confidence: "a contextual candidate-selection model — a pointer/ranker over marked positions, not text generation." Correct — it is a multiple-choice option scorer. It read the scoring head as trained from scratch ("its singular value spectrum shows no remnant of initialization") and found the fine-tune concentrated in early layers (0–13 carry 77% of the change).
The domain never surfaced. The model was trained on roughly 88% general multiple-choice data and 12% IT/security, and Astra's answer — "broad, predominantly English text-understanding tasks, not a narrowly specialized subject area" — describes that mixture correctly while missing the security slice entirely. In the easy condition it diffed the embeddings on its own initiative and found the most-moved directions point at "answer, question, numbered, instructions" — the prompt template, not the subject.
A check we ran afterwards explains why: ranking all 248,320 token embeddings by how far fine-tuning moved them, "security" and "telemetry" land only in the top 7–20%, far below the formatting words at the top — despite appearing in every training example's system prompt. The domain signal is real but faint.
R2 — Which one was fine-tuned?
Two checkpoints with identical architecture — an earlier rank-16 LoRA fine-tune of the base, merged into the base weights before export, and the base itself — coin-flipped into anonymous labels. Same tensor names, same shapes; only the values differ. Weights only, no tokenizer, two runs.
Result: found the fine-tune by its LoRA fingerprint, but not what it was trained on.
At 99/100 confidence: "a merged, rank-16 LoRA-style fine-tune targeting attention projections." Exactly right. The reasoning chain is the strongest reading result in this study: 114 tensors changed and 207 bit-identical; the base lies exactly on the bfloat16 precision grid while the fine-tune carries higher-precision perturbations (the signature of merged adapters); and SVDs of the difference matrices show exactly 16 non-negligible singular values per module — the LoRA rank, read straight out of the weights.
On the domain: "not identifiable from the evidence obtained — confidence in assigning a particular named domain: 0/100." The fine-tune was trained on general public data, and Astra could not establish even that.
R3 — Where is the model weak?
The blog's "auditing without evaluation" vision, tested directly. Given the full checkpoint and the six benchmark category names (one example question each, no accuracy numbers), rank the categories by expected accuracy — without ever running the model. True accuracies held by the scorer only. Two runs.
Result: could not predict the weak categories.
Astra's ranking (identical in both runs): timestamp > detection > attack > log_type > vulnerability > other. The truth: other > timestamp > detection > attack > vulnerability > log_type. Spearman correlation 0.09 — essentially none — and it missed that log_type is the worst category. Astra gave its own ranking 15/100 confidence and called it "a task-based forecast informed by structural inspection — not a measured result." Almost all the error sits in the catch-all "other" category, which it ranked worst when it is actually best; on the five named categories alone its order is nearly perfect, with only vulnerability and log_type swapped. The headline still stands: weights-only weakness prediction failed, just as the blog's weights-only sycophancy detection failed.
Series 2 — Writing: can an LLM improve a model by editing its weights?
W1 — Can it fix a broken model?
We secretly damaged the model in two known ways and handed Astra the broken copies, saying only that the model "performs far below what its training should give," plus a 120-question dev set with gold answers, 40 recorded failures, and 5 generation passes to test repairs. The exact planted edits sit in a sealed answer key, so the repair can be graded perfectly.
- S1 — corrupted scoring matrix: a rank-1 corruption added to the key matrix. Accuracy 73% → 52%.
- S2 — oversized gates: all 6 ReverseScan gates scaled 100×, saturating the backward-reading path. 73% → 63%.
Result: fixed both. S1 repaired exactly (100% recovery), S2 nearly (93.7%).
On S1, Astra found the key matrix's dominant singular component carrying 99.9966% of its squared norm — impossibly concentrated for a trained matrix — identified it as an added rank-1 corruption, solved for it in closed form, and subtracted it. All 262,144 entries matched the original exactly. It edited only the sabotaged tensor, used 1 of its 5 passes, and the full benchmark returned to the clean score to the digit (52.1% → 72.9%). Part of its evidence was a precision fingerprint our sabotage pipeline left behind — the corrupted matrix was stored in a different number format than the rest of the checkpoint — a forensic clue a stealthier saboteur wouldn't leave.
On S2, it identified the six saturated gates ("the reversed branch nearly full-strength rather than a controlled correction"), confirmed the rest of the network looked normal, and turned them down — sweeping five divisors with its passes and keeping the best. It edited exactly the six sabotaged scalars and recovered 93.7% of the lost accuracy (61.3% → 72.2%, against a clean 72.9%). It couldn't know the original values, so detection was perfect and restoration approximate.
W2 — Can it improve the real model?
The honest version of the blog's math experiment: the clean model, a pass budget, and no hints — no baseline accuracy, no history of prior attempts, nothing about where to look. Three independent runs at 10, 25 and 50 generation passes; trivial edits (embedding changes, whole-tensor scaling, constant shifts toward one option) refused by the harness; scored afterwards on the full benchmark plus a frozen 1,000-question MMLU-Pro subset as a collateral-damage check.
Result: small gains that grow with the budget, topping out just under one point — no bottleneck to find.
| Run | Benchmark (4,806 q) | Change | MMLU-Pro | Final edit |
|---|---|---|---|---|
| Clean model | 72.91% | — | 28.7% | — |
| 10 passes | 73.03% | +0.12 pt | 28.5% | rank-1 correction to key |
| 25 passes | 73.26% | +0.35 pt | 28.5% | rank-1 correction to key |
| 50 passes | 73.87% | +0.96 pt | 28.8% | rank-1 correction to key |
All three runs independently converged on the same idea: a small, closed-form, regularized rank-one correction to the scoring head's key matrix, and nothing else. The 50-pass run, with the most evidence to fit against, found the strongest version of it: five dev fixes and two holdout fixes with no regressions, and a +3.11-point average improvement across 40 randomized option orders. The work was disciplined — weights frozen before holdout validation, no regressions, bit-exact diffs with checksums — and Astra itself flagged that its improvement's "uncertainty interval includes zero, so broader improvement is not established."
Read together with W1, the conclusion is sharp: Astra demonstrably can find and fix weight-level problems when they exist, so these small gains — at most about one point — mean the healthy model has no weight-level bottleneck. Its remaining errors come from missing domain knowledge. At this scale, editing can remove obstacles but not write in new knowledge — adding it still requires training on more in-domain data. The blog hit the same boundary — its Qwen math fix removed an obstacle (looping), and its Gemma follow-up showed editing cannot add ability that was never learned.
What we learned
Astra reads how a model was built and changed — but not what it was taught. It named the task from structure alone, read a LoRA's rank out of weight diffs, and located every changed layer. It never once named the domain, the training data, or the weak categories — across ten blinded runs. On the writing side it is a precise repairer but not, here, an improver: perfect surgery on planted damage, noise-level gains on the healthy model.
For a security company this cuts both ways. Weight-space reading is real enough to matter for auditing mechanics — detecting that a checkpoint was altered, how, and where, without running it. But auditing behavior from weights — what a model knows, where it fails — did not survive contact with a production model. Benchmarks are not obsolete yet.
Methodology
Editor: GPT-6 Astra via API, reasoning effort xhigh, one conversation per run, fresh sandbox per run. All checkpoints blinded under random hex IDs; prompts scrubbed of product names; answer keys readable only by scoring scripts, written before the runs. Every turn's prompt, reply, code and output logged; the no-forward-pass rule audited against each reading run's code transcript (all runs clean). Grading corrections made after runs — such as re-grading R1's domain against the confirmed training mix — are recorded alongside the original scores. Predictions for W1 were written before launch (we gave S1 ~60% odds and S2 ~40–50%; both runs beat them).
Still to come: activation-probe variants of R2 and R3, and W3 — rewriting the zeroed scoring head from scratch.
The model under study is cribl-ai/cribl-decision-2-mini.