SubMaroon/gemma4-lora-traps

On Gemma 4, v_proj does not exist on 5 of the 30 layers — and your LoRA config does not know it

This is a report created by Claude after hours and days spent training my LoRas for the Goetia merge. You might find this text useful when training your LoRas for the Gemma 4 MoE family.

If you fine-tune google/gemma-4-26B-A4B with a target_modules list containing the string "v_proj", that adapter attaches to 25 layers, not 30. PEFT does not warn you, because it only raises when nothing matched. Your adapter is smaller than you think and asymmetric across depth, and the only visible sign is a trainable-parameter count you probably did not hand-verify.

That is the short version. The long version is more interesting, because on the five layers where v_proj is absent, k_proj is the value matrix — which means an adapter you believe is "queries and keys only" is editing the value path, and an adapter you believe is "values and output only" cannot reach values there at all.

I found this the hard way, by running a controlled experiment that turned out to be measuring something other than what I designed it to measure. Details at the end.

What is already known

The mechanism itself is not my discovery, and I want to be precise about that before adding anything.

The Gemma 4 technical report states it in one sentence, under long-context efficiency: "We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e., values=keys." The official model card puts it as: "To optimize memory for long contexts, global layers feature unified Keys and Values, and apply Proportional RoPE (p-RoPE)."

Devansh has gone furthest in public, and states the consequence explicitly: "Gemma 4 eliminates the V projection in global layers. The key projection is computed, then reused directly as the value, with only RMSNorm applied on the value side as a differentiator in the forward pass." Maarten Grootendorst's visual guide covers K=V as a KV-cache trick. idlemachines noted the normalization asymmetry: "values get normalised too, but magnitude-only, with no learned scale."

And the flag is documented in transformers. Gemma4TextConfig carries a docstring for attention_k_eq_v: "Whether keys and values share the same projection weights. When True, the key projection output is reused as the value projection." One line, in the API reference, with no consequences drawn.

One clarification worth making, because the public write-ups blur it: Gemma 4 has two independent KV-saving mechanisms, and only one of them removes v_proj.

• num_kv_shared_layers — cross-layer sharing. Later layers reuse KV tensors from an earlier non-shared layer. This is the one the official HF launch post and Sebastian Raschka describe. In 26B-A4B it is set to 0, i.e. off.
• attention_k_eq_v — within-layer sharing. On non-sliding layers, values are the key projection. This is the one that sets v_proj to None. In 26B-A4B it is true.

If you read about Gemma 4 KV sharing and concluded it does not affect your adapter, you may have read about the wrong mechanism.

The part nobody seems to have written down

What I could not find anywhere is what this does to a LoRA config. Three consequences, and they compound.

The layout

"attention_k_eq_v": true,
"layer_types": ["sliding_attention", ..., "full_attention", ...]

layer_types puts full_attention at exactly indices 5, 11, 17, 23, 29 — every sixth layer, and always the last. The other 25 are sliding_attention with a 1024-token window. Print the loaded model and the attention blocks are not uniform:

	layers 0–4, 6–10, 12–16, 18–22, 24–28	layers 5, 11, 17, 23, 29
q_proj	2816 → 4096	2816 → 8192
k_proj	2816 → 2048	2816 → 1024
v_proj	2816 → 2048	absent
o_proj	4096 → 2816	8192 → 2816

That is 115 attention projections, not 4 × 30 = 120. The shapes follow from head_dim: 256 / num_key_value_heads: 8 on sliding layers versus global_head_dim: 512 / num_global_key_value_heads: 2 on global ones.

This is not a broken checkpoint. model.safetensors.index.json of google/gemma-4-26B-A4B itself has no self_attn.v_proj.weight key for those five layers. From modeling_gemma4.py:

self.use_alternative_attention = config.attention_k_eq_v and not self.is_sliding
self.v_proj = (
    nn.Linear(config.hidden_size, num_key_value_heads * self.head_dim, bias=config.attention_bias)
    if not self.use_alternative_attention
    else None
)

Note and not self.is_sliding. The flag is global, its effect is not.

Consequence 1: silent partial match

PEFT matches target_modules strings by module-name suffix. There is no ...layers.5.self_attn.v_proj to match, so nothing matches, and nothing is reported — PEFT raises only when the whole list found nothing. So the popular seven-name list gives you v_proj on 25 layers and q/k/o_proj on 30.

This is not hypothetical. Current Gemma 4 fine-tuning guides recommend exactly ["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"] with no caveat about layer coverage. Community adapters use regexes like (mlp|self_attn)\.(up|down|gate|q|k|v|o)_proj that treat v uniformly across depth. Unsloth's guide uses target_modules="all-linear", which sidesteps the problem by accident — it enumerates what exists rather than what you named — but does not explain it. (Per oxen.ai, recent PEFT ships default Gemma 4 target modules scoped to the language model via regex; that fixes vision-tower leakage, not the v_proj count.)

Consequence 2: on those layers, k_proj is the value matrix

Here is the exact order of operations in forward:

key_states   = self.k_proj(hidden_states).view(hidden_shape)
value_states = self.v_proj(hidden_states).view(hidden_shape) if self.v_proj is not None else key_states

key_states   = self.k_norm(key_states)
key_states   = apply_rotary_pos_emb(key_states, cos, sin, unsqueeze_dim=2)
key_states   = key_states.transpose(1, 2)

value_states = self.v_norm(value_states)
value_states = value_states.transpose(1, 2)

Look at where the fallback happens. value_states takes the raw output of k_proj, before k_norm and before RoPE. Then it goes through v_norm, an RMSNorm with with_scale=False. So one projection feeds two paths, normalized differently, and positional information is applied to the key path only.

For anyone trying to reason about attention in terms of separable circuits — where the query-key product decides where to attend and the value-output product decides what gets written into the residual stream — this matters:

• An adapter on q_proj + k_proj is not query-key only. On those five layers it edits values.
• An adapter on v_proj + o_proj has no access to values there. Only o_proj.
• On those five layers the two paths cannot be separated at all. They share one matrix.

And the five layers are the only ones that see the whole context; the other 25 are windowed at 1024 tokens. So anything you care about that involves long context — instruction following deep into a chat, re-reading a system prompt every turn, recalling something from 20k tokens back — lives precisely where the separation you are testing does not exist.

Consequence 3: QK-norm changes what a LoRA delta can even do

q_norm and k_norm are RMSNorm over head_dim, applied after the projection and before RoPE:

query_states = self.q_proj(hidden_states).view(hidden_shape)
query_states = self.q_norm(query_states)
query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)

RMSNorm rescales each head's vector to unit RMS. So a LoRA delta on q_proj or k_proj can change the direction of queries and keys but not their magnitude — the normalization discards it. o_proj has no equivalent per-head constraint.

If you are comparing "adapt q/k" against "adapt v/o" at equal parameter budget on any QK-norm architecture — Gemma 3 and 4, Qwen3, OLMo 2/3 — this asymmetry is part of your result whether you account for it or not. I could not find any discussion of QK-norm interacting with LoRA. The closest published work is on controlling attention logits during pretraining (Anson & Aitchison 2025; Zhai et al., σReparam, ICML 2023), which treats the coupled magnitudes of Q and K as the thing to control — but says nothing about adapters.

Two smaller landmines in the same area

Both are mine as far as I can tell, and both are cheap to avoid:

• Gemma4TextRouter.proj (2816 → 128) is a real nn.Linear, so a loose regex like .*proj$ will catch the MoE router. Adapting expert routing is a far less predictable edit than adjusting attention. Exclude it explicitly.
• gate_proj / up_proj / down_proj in target_modules land on the dense Gemma4TextMLP (2816 → 2112) sitting next to the experts — that is the single shared expert — and not on the 128 routed ones. The names are absorbed by the wrong module, which is why the trainable-parameter count comes out plausible-looking but wrong.

For completeness, the neighbouring traps that are already well documented, so you do not have to rediscover them: Gemma4TextExperts stores weights as stacked nn.Parameter, so bitsandbytes cannot quantize them (Axolotl's expert-quantization docs; bitsandbytes #1849) and PEFT needs target_parameters rather than target_modules to reach them (PEFT docs; unsloth #4907 for the "abnormally low trainable parameter count" symptom). The vision and audio towers reuse the same leaf names, so an unanchored list leaks the adapter into them (oxen.ai; Axolotl multimodal docs). On my first run part of the adapter landed on the vision encoder and the loss flattened almost immediately.

Building the target list so it does what you wrote

Stop passing projection-name strings. Read the module paths off the live model and assert your assumptions:

import re

PROJ = ("q_proj", "k_proj", "v_proj", "o_proj")
LAYER_RE = re.compile(r"language_model\.layers\.(\d+)\.")

layer_mods = {}
for name, _ in model.named_modules():
    if "language_model.layers." not in name:      # anchor: excludes vision/audio towers
        continue
    if not name.endswith(PROJ):
        continue
    li = int(LAYER_RE.search(name).group(1))
    layer_mods.setdefault(li, {})[name.rsplit(".", 1)[-1]] = name

global_layers = sorted(i for i, v in layer_mods.items() if "v_proj" not in v)
assert global_layers == [5, 11, 17, 23, 29], f"layer plan changed: {global_layers}"
assert sum(len(v) for v in layer_mods.values()) == 115

# A genuinely query-key-only arm: skip k_proj on global layers,
# where k_proj is also the value matrix.
targets = [
    layer_mods[li][p]
    for li in sorted(layer_mods)
    for p in ("q_proj", "k_proj")
    if p in layer_mods[li] and not (p == "k_proj" and li in global_layers)
]

FORBIDDEN = ("vision", "audio", "router", "experts", "embed", "lm_head",
             "gate_proj", "up_proj", "down_proj")
assert not [t for t in targets if any(b in t.lower() for b in FORBIDDEN)]

Then re-audit after get_peft_model, because that is where a config can still surprise you: check that the number of trainable tensors is exactly twice the number of targets, and that the set of touched layers and projection types matches your plan. Make both assert, not print. A printed warning scrolls off screen, and an hour of A100 time goes with it. PEFT also ships get_model_status() / get_layer_status(), which is the supported way to see what actually got wrapped.

How I ran into this, and why my own numbers do not settle anything

I wanted to know which half of attention carries writing style and which half is responsible for a fine-tune losing its grip on output format. So: two adapters, same data (1415 train / 74 eval), same seed, r=32, alpha=64, lr 2e-5, 2 epochs, 354 steps, QLoRA 4-bit, one A100 80GB. One on v_proj + o_proj, one on q_proj + k_proj.

	A: v_proj + o_proj	B: q_proj + k_proj
targets	55	60
trainable params	11,182,080	11,796,480
final eval loss	1.9703	2.2056
mean token accuracy	54.96 %	51.38 %

A is below B at every eval checkpoint, monotonically; both plateau; B has 5.5 % more trainable parameters and still loses.

I am not asking you to believe that means anything, for five reasons.

It reproduces a 2021 result. "The value/output side beats the query/key side at equal budget" is Table 5 of the original LoRA paper: on WikiSQL, Wq 70.4 / Wk 70.0 versus Wv 73.0 / Wo 73.2, and Wq+Wk 71.4 versus Wq+Wv 73.7. Yao et al. (IJCAI 2025) added the mechanism: the gradient with respect to W_K contains W_Q, which is near zero early in training, so Q and K are multiplicatively suppressed while V is not.

The split is not the split I thought it was. That is this whole article. Arm B edited values on five layers; arm A never reached values there.

The gap is over-determined. Beyond that, QK-norm handicaps arm B structurally, and my base started at loss 7.43 on this data — very far off. When the base is that far from the target distribution, the run mostly measures which arm can move the output distribution fastest, and that favours o_proj, which writes straight into the residual stream, over q/k, which only reshape a softmax.

My base was not clean, and the contamination is exactly on the seam I was testing. The base is a 15-way MoE merge followed by abliteration. I went back and checked what the abliteration tool actually modifies: attn.o_proj only, on layers 14–26, by unconstrained L-BFGS optimization of the matrix rather than a rank-1 projection. So o_proj had been surgically rewritten in 13 of 30 layers before I started, while q/k/v were untouched. Arm A trains on top of rewritten matrices, arm B on top of pristine ones. I cannot predict the direction of that bias — a rewritten o_proj could be easier or harder to adapt further — but a comparison with a systematic asymmetry like that is not a fair one. Worth stating plainly: if you benchmark anything about attention on an abliterated model, find out which matrices were abliterated first.

(On the other hand, the merge did not lose tensors: the merged checkpoint and google/gemma-4-26B-A4B have byte-identical total_size — 51,611,872,412 — and the shard files differ by 584 bytes, which is the size difference of the safetensors JSON headers. The missing v_proj really is architectural.)

And my behavioural hypothesis was wrong. I expected the value/output half to carry style and the query/key half to be responsible for breaking format. What I saw was the opposite arrangement: the value/output arm carries the style and breaks structured output sooner, while the query/key arm holds formatting but barely transfers style — it describes a character's voice instead of speaking in it. I report that as a negative result rather than dropping it, with the caveat it deserves: those behavioural observations are single generations per setting at one context depth, judged by me, compared at equal adapter weight rather than equal effect size. Equal weight is the wrong normalization when one arm is simply a stronger intervention per unit of weight. That is an anecdote, not a measurement.

What I would actually like to know

• Does the loss gap survive on a clean google/gemma-4-26B-A4B, with three seeds, and with the arms rebuilt so that k_proj on global layers goes to neither side?
• How much of a LoRA delta on q_proj / k_proj survives q_norm / k_norm? If the answer is "not much", then a chunk of the folklore about which projections matter is really a statement about where the normalization sits — and that folklore predates QK-norm becoming standard.
• Those five global layers, where W_V is literally W_K: good place to adapt, or bad? It is the only spot in the model where the two paths are physically tied, and I have no intuition for what a low-rank edit there does.

If you have hit consequence 1 without noticing, or if you have run any of this on a clean base, I would like to hear about it.

References

Architecture and code:

• Gemma 4 Technical Report — https://arxiv.org/html/2607.02770v1
• Gemma 4 model card (Google) — https://ai.google.dev/gemma/docs/core/model_card_4
• google/gemma-4-26B-A4B config — https://huggingface.co/google/gemma-4-26B-A4B/blob/main/config.json
• modeling_gemma4.py — https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma4/modeling_gemma4.py
• configuration_gemma4.py (attention_k_eq_v docstring) — https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma4/configuration_gemma4.py
• Gemma4 model docs — https://huggingface.co/docs/transformers/model_doc/gemma4

Prior public description of K=V:

• Devansh, "Google's Gemma 4 is Weirder than you Realize" — https://machine-learning-made-simple.medium.com/googles-gemma-4-is-weirder-than-you-realize-17d00d95b0d5
• Maarten Grootendorst, "A Visual Guide to Gemma 4" — https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-gemma-4
• idlemachines, "Gemma 4 is not your standard transformer" — https://idlemachines.co.uk/essays/gemma4-architecture

LoRA target-module selection:

• Hu et al., LoRA, 2021, Table 5 / §7.1 — https://arxiv.org/abs/2106.09685
• Yao et al., IJCAI 2025, unequal importance of attention matrices — https://arxiv.org/abs/2410.02247
• Elhage et al., 2021, query-key and output-value circuits — https://transformer-circuits.pub/2021/framework/index.html
• PEFT LoRA developer guide — https://huggingface.co/docs/peft/developer_guides/lora

Already-documented neighbouring traps:

• Axolotl, MoE expert quantization — https://docs.axolotl.ai/docs/expert_quantization.html
• bitsandbytes #1849, fused MoE weights — https://github.com/bitsandbytes-foundation/bitsandbytes/issues/1849
• unsloth #4907, low trainable-parameter count on Gemma 4 MoE — https://github.com/unslothai/unsloth/issues/4907
• oxen.ai, Gemma 4 fine-tuning pipeline — https://ghost.oxen.ai/writing-a-fine-tuning-and-deployment-pipeline-isnt-as-easy-as-it-looks-gemma-4-version/
• Axolotl multimodal docs — https://docs.axolotl.ai/docs/multimodal.html

Attention-logit control (background for consequence 3):

• Anson & Aitchison, "Controlling changes to attention logits", 2025 — https://arxiv.org/abs/2511.21377
• Zhai et al., σReparam, ICML 2023 — https://proceedings.mlr.press/v202/zhai23a/zhai23a.pdf

---

Code excerpts are from huggingface/transformers, Apache License 2.0. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. "Gemma 4" is used descriptively; this article is not affiliated with or endorsed by Google.