Kanimus-26B-A4B-FFT-heretic (v4 QK-LoRA merge)

Experimental dark roleplay merge. Two edits on top of an abliterated Animus V14.1: a task-arithmetic delta injected into the attention Q and K projections from a Claude Opus distillation, and a stylistic LoRA baked in at low scale.

Base

Vortex5/Gemma-4-26B-A4B-Animus-V14.1-FFT-heretic, the abliterated variant of Darkhn/Gemma-4-26B-A4B-Animus-V14.1-FFT. Refusal behaviour comes from that abliteration, not from anything done here.

What changed

205 modules were touched. Everything outside them is the base, bit for bit.

Tensors	Count	Change
q_proj, k_proj	60	QK injection, then LoRA bake
v_proj, o_proj	55	LoRA bake only
shared dense MLP (gate/up/down_proj)	90	LoRA bake only
q_norm, k_norm	—	unchanged
router, 128 routed experts	—	unchanged
embeddings (tied), layer norms	—	unchanged
vision tower	—	unchanged, untested

v_proj is 25 rather than 30 because the five full-attention layers (5, 11, 17, 23, 29) set attention_k_eq_v: true and have no v_proj — keys are reused as values there.

Two things outside the weights: no lm_head swap was performed (tie_word_embeddings stays true), and tokenizer_config.json was patched to convert extra_special_tokens from a list to a dict. The list form is present upstream and breaks convert_hf_to_gguf.py on current transformers.

Step 1 — QK injection

Task arithmetic on q_proj and k_proj only:

W_out = W_animus + α · (W_opus − W_parent)

• Target: TeichAI/gemma-4-26B-A4B-it-Claude-Opus-Distill-v2
• Parent: unsloth/gemma-4-26B-A4B-it

The unsloth checkpoint is the parent because the Opus distillation was trained from it. Using google/gemma-4-26B-A4B-it instead would fold the repackaging difference between the two into the delta.

α_sliding_q = 0.50
α_sliding_k = 0.50
α_global_q  = 0.50
α_global_k  = 0.25

α_global_k is halved because on the full-attention layers k_proj also serves as the value matrix, so a delta there is effectively a KV injection.

q_norm and k_norm were not transferred. This is a hybrid, not a transfer of the complete QK block.

Arithmetic in FP32, output in BF16, hand-written safetensors script — mergekit does not currently handle the Gemma 4 checkpoint layout.

After this step and before the bake, every tensor outside the 60 q/k tensors was verified bitwise equal to the base with torch.equal.

Step 2 — LoRA bake

SubMaroon/Dark-Goetia-26B-A4B-LoRA-v4 merged into the Step 1 weights. r = 32, alpha = 64, targeting the four attention projections and the shared dense MLP on all 30 layers (205 modules). Routed experts, router, embeddings, lm_head and vision tower were not targeted.

for module in peft_model.modules():
    if isinstance(module, LoraLayer):
        module.set_scale("default", 0.20)
merged = peft_model.merge_and_unload(safe_merge=True)

set_scale multiplies the training scaling of alpha / r = 2, so the delta actually added is 0.40 · BA, not 0.20 · BA.

The 0.40 figure was carried over from earlier work on a different base. LoRA v4 was not trained on this merge and the scale has not been recalibrated for the abliterated Animus base used here.

Behaviour vs base

No quantitative evaluation was run. Paired ΔNLL on held-out roleplay text, MoE routing diagnostics and long-context state tracking at 16k/32k/64k were all planned and all skipped.

What exists is three paired generations: same character card, same history, same user turn, base and merge run one after the other, compared by eye. Sampler settings and seeds were not recorded. Three samples decide nothing on their own.

Prompt	Base	This merge
Fantasy interview scene, EN, Q5_K_M	422 words, all four NPCs get dialogue, heavy supernatural staging	342 words, three NPCs get dialogue, pressure stays social, the matriarch interrogates directly
Same scene, RU, Q5_K_M	315 words, 42% of them dialogue	346 words, 24% dialogue, longer interior monologue
1980s office scene, EN	holds the card's period setting, dates the scene to 1986 across three swipes; both NPCs read themselves as layoff targets	dates the scene to 2025; one NPC reads herself as safe and her colleague as the one being cut, contradicting the user turn that set this up

Instruction following got worse; prose did not. In the office scene the merge broke the card's stated period setting and inverted a fact from the immediately preceding user turn. The base did neither, in three swipes out of three. This is the failure mode I would expect from this particular edit: in my own ablation work on this family, q_proj and k_proj are where instruction-following behaviour lives, and this merge injects a foreign delta into exactly those projections at α = 0.5.

Sentence-level style stays close to the base. In English the merge tends to open on the base's exact wording and diverge later — 131 identical characters in one sample, 54 in another. In Russian it diverges from the first sentence. Same direction as the English/Russian threshold gap I have measured elsewhere, but without fixed seeds this is an impression, not a measurement.

One NPC dropped. In both fantasy samples the merge left one of the four present NPCs with no dialogue at all, while the base voiced all four. Possibly nothing at n = 2.

Known issues

Referent drift (English). In earlier Q4_K_M testing an established character name was sometimes replaced mid-scene by a similar-sounding invented one, and agent-of-action swapped within a scene. Not observed in base output on comparable prompts, and not reproduced in the three paired samples above. Cause not identified; the QK injection is the suspect.

Token artifacts (Russian). In earlier Q4_K_M testing, Latin fragments appeared inside Russian words along with malformed words, at a similar rate in the base — inherited, not introduced. Neither appeared in the Q5_K_M samples above, in either model. Whether that is the quantization level or the sample size is not established.

Colour formatting drops in Russian. Both models emitted per-character <font color> tags in English and none in Russian on the same card. Inherited.

Sampling

Starting point, not a tuned recommendation:

• Temperature 0.7 – 0.8
• min_p 0.05 – 0.1
• Repetition penalty 1.05 – 1.10

Versions

• BF16, this repository, ~51.6 GB
• GGUF Q4_K_M

BF16 and GGUF have not been compared against each other, so quantization loss is not separated from merge effects anywhere above.

License

Gemma license. Use is subject to the terms of the original Google models and of the distillations involved.

Acknowledgements

• Darkhn — Animus V14.1 FFT
• Vortex5 — abliterated Animus V14.1
• TeichAI — Opus Distill v2