build-small-hackathon/Gemma-26B-A4B-VisualNovel-GGUF

Gemma 4 26B-A4B — Visual Novel (GGUF)

A QLoRA fine-tune of google/gemma-4-26b-a4b-it (26B MoE, ~4B active) for the Ars-Fabula anime visual-novel engine. The model narrates slice-of-life scenes for a locked cast and drives sprites, backgrounds and choices through a bracketed tool protocol ([TOOL: name key="value" choices='[...]']).

Training

• Method: QLoRA on the MoE experts (Path A "un-fuse"): each fused 3D Gemma4TextExperts weight is split into per-expert nn.Linear leaves so LoRA can target experts.(gate_up|down).N (7885 LoRA modules, 1.88% trainable), then merged + re-fused bit-exact to the canonical Gemma4 layout for GGUF.
• Data: 2,752 VN-protocol turns (traced real play, validity-filtered).
• Schedule: 2 epochs, single B200, 688 optimizer steps, final train loss 0.225.
• Why the experts: training the FFN experts (where phrasing lives) is what moved prose quality — attention-only and dense-FFN-only LoRA gave only a shallow restyle. The experts tune keeps the base model's already-low canned-phrase rate while raising vocabulary diversity and sharpening character voice (see Evaluation).

Evaluation

This fine-tune (at its ship config, temperature 1.1) vs the untuned base google/gemma-4-26b-a4b-it, on held-out VN-protocol prompts. Higher is better for ↑ metrics, lower for ↓.

Metric	base gemma-4-26b-a4b-it	26B experts tune
Protocol validity ↑	100% (8/8)	100% (48/48)
Slop-tell density /1k words ↓	2.20	2.11
Type-token ratio (TTR) ↑	0.49	0.53
Cross-scene trigram reuse ↓	0.019	0.004

• The base model is already strong on the protocol, so the win isn't "teaching the format" — it's prose quality at no validity cost. The tune holds 100% validity across 48 held-out scenes (6 seeds) at temperature 1.1, while raising vocabulary diversity (TTR 0.49→0.53) and cutting verbatim cross-scene phrase reuse ~5× (0.019→0.004). Slop-tell density is a wash (2.20 vs 2.11) — the base was never slop-heavy on this list; the tune's gain is voice and variety, not de-cliché-ing.
• Sampling provenance: tune figures are mean over 6 seeds at temp 1.1 (validity over all 48 scenes); the base was sampled once (seed 42) at temp 0.8, its eval default, on the same 8-scene prompt set.

What the metrics mean

• Protocol validity — fraction of generated turns that pass the engine's own validator (vn_validate): well-formed [TOOL: …] calls, parseable choices JSON, and cast-lock (only the locked cast may speak/act). A hard well-formedness gate ("does the turn drive the UI without erroring"), not a taste score.
• Slop-tell density — count of curated "LLM-slop" phrases (stock clichés like "the air hung heavy with unspoken words", "a mix of X and Y") per 1,000 words, via tools/repetition_metrics.py against a hand-curated tell list. Lower = less formulaic. (The list was curated from observed tuned-model failure modes, so it may undercount base-specific clichés — read the base number as a floor, not a like-for-like.)
• Type-token ratio (TTR) — unique words ÷ total words: a vocabulary-diversity proxy. Higher = richer, less word-level repetition.
• Cross-scene trigram reuse — fraction of distinct 3-word sequences that recur across different scenes: a verbatim self-plagiarism proxy. Lower = the model reuses fewer canned spans from scene to scene.

Qualitative read (48 scenes, temp 1.1, seeds 1–6)

• Strengths: genuinely good comedy — per-seed-varied gags with setup / escalation / button (i.e. composing, not memorizing, despite the low 0.225 loss); environmental staging (shows before it tells); distinct, light character voices.
• Weaknesses: romance is the weak suit (stock, on-the-nose, little subtext); the "air heavy / charged with unspoken X" reflex survives the tune but clusters almost entirely in romance scenes; choices lean on an open-up / deflect / stay-silent triad; the occasional garbled line or first↔second-person POV slip.
• Verdict: read it for comedy and cast chemistry; skim the kissing scenes.

Quants

File	Bits	Size	Notes
vn26b-experts-v1-Q4_K_M.gguf	Q4_K_M	16.8 GB	servable default; matches the stock gemma-4-26B-A4B-it-UD-Q4_K_M layout
vn26b-experts-v1-Q8_0.gguf	Q8_0	26.9 GB	near-lossless

MoE fallback (benign): 60/658 tensors (ffn_down / ffn_down_exps, cols 704 & 2112, not ÷256) fall back q4_K→q5_0, q6_K→q8_0 — so the down-projs are higher precision than nominal (why Q4_K_M is 16.8 GB, not ~14 GB).

Serving

Gemma 4 26B-A4B is a custom MoE arch; serve with the atomic-llama-cpp-turboquant fork (the stock llama.cpp lacks the tensor maps). The tuned model emits a reasoning channel (<|channel>thought … <channel|>) that the OpenAI /v1/chat/completions parser mangles — hit the raw /completion endpoint with the embedded chat template and strip a stray leading <channel|>.

Recommended sampling: temperature 1.1, top_p 0.95.

Runtime (Q4_K_M, fork llama-server CUDA build on a single L4, all 30 layers offloaded -ngl 99): prompt eval ≈1620 tok/s, generation ≈61 tok/s.