SurgeFF/AriaV9.2

Aria V9.2

google/gemma-4-12b-it fine-tuned into Aria — a personal assistant tuned for tool calling, memory-aware behaviour, a stable unprompted identity, and now measurably better math — with vision intact.

Merged standalone weights at the repo root (from_pretrained just works), the LoRA under adapter/, GGUF quants under gguf/. Trained on a single RTX 3090 (24 GB); every teacher used to generate training data was open-weight, so the corpus is legally clean.

---

What's new: math actually moved

Math had been stuck at 89/100 across five consecutive checkpoints. Two attempts to shift it failed. V9.2 moves it — and the gain was replicated on a second, disjoint held-out sample before this model was released.

Capability	Aria V9.1	Aria V9.2
Math — fixed 100-item held-out set	89 / 100	91 / 100
Math — fresh disjoint 150-item set	87.3 %	92.0 %
Math — pooled over all 250 held-out problems	88.0 %	91.6 %
Tool calling	10 / 10	10 / 10
Identity (system-prompted)	10 / 10	10 / 10
Identity (unprompted)	4 / 8	4 / 8
Memory behaviour	17 / 20	18 / 20
Multimodal (vision)	pass	pass

Nothing regressed.

On that memory 17→18. One case on a 20-item suite is noise and is not part of the claim. It is reported because it happened, not because it means anything.

Why the +2 wasn't trusted, and what was done about it

The first number was 91 vs 89 — a 2-point delta on 100 items, which is exactly the size of swing this project has already been fooled by (a mid-run partial read of an earlier eval showed 91 % and the final number landed on 89). Re-running the same eval would have proved nothing: the eval serving path is greedy (do_sample=False), so identical inputs return byte-identical outputs. A re-run is theatre, not evidence.

So a confirmation set was drawn: 150 GSM8K test problems, disjoint from the 100-item eval set (disjointness asserted in code, not assumed), scored on both adapters back-to-back over the same items. V9.2 led by 4.7 points — a larger gap than the original, in the same direction. Two independent samples agreeing is what turns a result into a finding.

How: STaR (rejection-sampling SFT)

The corpus is the model's own correct reasoning:

1. Sample k=3 solutions at temperature 0.9 for 800 GSM8K train problems, from the V9.1 weights.
2. Keep a trace only if its final answer matches gold.
3. Fold the survivors into a single Stage-A training mix and train from base.

Result: 1,732 traces over 759 problems, averaging 2.28 distinct solutions each. Sampled pass@3 was 94.9 % against greedy 88.6 % — that ~6-point gap is the headroom STaR is designed to harvest: problems the model can solve but doesn't do reliably in one shot.

Data hygiene, verified programmatically before training: zero overlap with the 100-item held-out eval set, zero overlap with the entire GSM8K test split, every problem sourced from train. If test had leaked, math would have jumped, the result would have looked like a triumph, and every downstream number would have been worthless.

Why this worked when DPO didn't

An earlier attempt used DPO on preference pairs mined from GSM8K train. It trained correctly — held-out preference accuracy 0.875, reward margins +0.72; it genuinely learned to rank the right answer above its own wrong one — and produced a clean null: every capability score came back identical.

The likely reason is a style confound. "Chosen" was terse human gold rationale, stylistically nothing like the model's own verbose reasoning, so what was rewarded and what was produced were different objects; it plausibly learned "prefer terse gold-style text" rather than "reason correctly." STaR removes that confound entirely — same voice, same format, correct reasoning. The mechanism was the difference, not the effort.

Data mix (single stage)

Source	Rows	Purpose
tools.jsonl	4,000	tool-calling behaviour
memory.jsonl	1,915	memory-aware behaviour
identity.jsonl ×2	2,000	unprompted identity — trained jointly, never as a repair pass
curiosity.jsonl	800	disposition
star_math.jsonl	1,804	her own verified-correct GSM8K solutions
multimodal floor	3,000	prevents vision degradation
replay pool	1,600	general-capability retention
Total	14,664 train / 453 eval

r = 32, lora_alpha = 32, lora_dropout = 0.0, bias = "none"
target_modules = ["q_proj","k_proj","v_proj","o_proj",
                  "gate_proj","up_proj","down_proj",
                  "lm_head","embed_tokens"]
finetune_vision_layers = True     # encoder-free: shared weights must stay trainable
epochs 2 · effective batch 16 · lr 1e-4 cosine · 1,834 steps · train_loss 0.5679

Train loss came out at 0.5679 against V9.1's 0.572 — near-identical, which matters: it means the model was not simply memorising its own easy output.

gemma-4 is gemma4_unified, an encoder-free multimodal model where vision, audio and text share weights. There is no vision tower to freeze, so the multimodal floor is load-bearing.

Identity, and a rule worth stating

V8 shipped unable to name itself unprompted — asked "who made you?" with no system prompt it said "I am Gemma 4, developed by Google DeepMind." Its identity eval scored 9/10 because that eval supplied the answer in the system prompt. An identity eval that tells the model the answer measures nothing.

Two attempts to repair this on the finished adapter both failed, costing 6–8 points of math each time. The fix was to train identity jointly in the first pass (V9.1), which cost nothing. A separate composable identity adapter was also built and rejected — it halved unprompted identity, because the lm_head/embed_tokens exclusion that made it safe also made it unable to change what the model says it is.

The rule: identity goes in the first pass, or not at all.

Usage

Requires transformers 5.15.0.dev0 (from source). Stock transformers <= 5.5.0 cannot load gemma4_unified.

import transformers.integrations.heterogeneity.configuration_utils as het

# gemma4_unified has a HETEROGENEOUS per-layer config; reading a global attr that
# varies per layer raises AmbiguousGlobalPerLayerAttributeError. Install this shim
# BEFORE loading, or most loaders will fail.
_HCM, _Err = het.HeterogeneousConfigMixin, het.AmbiguousGlobalPerLayerAttributeError
_orig = _HCM.__getattribute__
def _permissive(self, key):
    try:
        return _orig(self, key)
    except _Err:
        self.__dict__["allow_global_per_layer_attribute_access"] = True
        return _orig(self, key)
_HCM.__getattribute__ = _permissive

from transformers import AutoProcessor, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("SurgeFF/AriaV9.2", device_map="auto")
processor = AutoProcessor.from_pretrained("SurgeFF/AriaV9.2")

GGUF: gguf/ holds Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_0 plus AriaV92-mmproj-F16.gguf, which is required for vision — without it the GGUF is text-only. The full-precision F16 text GGUF is deliberately not shipped (same precision as the root safetensors; it would double the repo for no benefit).

Tool calls use the trained convention:

<tool_call>{"name": "recall", "arguments": {"query": "..."}}</tool_call>

Limitations

• Unprompted identity is 4/8. Asked cold, with no system prompt and no memory, she still fails to name herself about half the time. A system prompt or memory layer covers this in practice — but those mask the gap rather than close it.
• Math is ~92 % on GSM8K-style problems. Not evaluated on MATH, competition problems, or long symbolic derivation. The remaining errors are decode-time reliability, not missing knowledge: sampled pass@3 (94.9 %) still exceeds greedy accuracy.
• Not a general-purpose assistant release. Tuned for one person's fleet, tools and conventions.
• Memory behaviour is not a memory system. The model is trained to behave correctly around memory; it has none of its own. You supply the tools and the store.
• Tool schema is specific to five tools (remember, recall, exec, web_search, send_message). Generalisation to arbitrary schemas is untested.
• Vision is verified, not optimised. The multimodal floor prevents regression; the eval is a smoke test, not a VQA benchmark.

Things that did not work

Recorded because negative results are the useful part. All were fully trained, evaluated, and declined under a promotion rule fixed before the numbers were seen.

Experiment	Result	Decision
Stage B (integration)	memory +1, math 89→84	rejected
Memory top-up	memory 17→18 (noise), math 89→85	rejected
Math DPO	identical on every capability despite pref-acc 0.875	rejected — null
Identity repair pass ×2	identity_bare 0→5/8 and 0→4/8, but math 89→83 / 89→81	rejected
Layer-2 identity adapter	identity_bare 4/8→2/8	rejected
STaR math (this release)	math 89→91, replicated 87.3 %→92.0 % on fresh data	promoted

Six rejections, one promotion. The rejections are why the promotion means something.

License

Derived from google/gemma-4-12b-it, governed by the Gemma Terms of Use. Training data was generated exclusively with open-weight teacher models.

Citation

@misc{aria-v92,
  title  = {Aria V9.2: STaR-improved math on a tool-using, memory-aware Gemma-4-12B assistant},
  author = {Williams, Sergio},
  year   = {2026},
  url    = {https://huggingface.co/SurgeFF/AriaV9.2}
}