P6-E20B-A7B-D3

An experimental, untrained expansion of NVIDIA's Nemotron-3.5 Bolt E4B into a Gemma 4 mixture-of-experts model, made as a base for fine-tuning research. Text only.

How it was built

1. Start from Bolt (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 @ 49843eed), keeping its dense FFN in every layer.
2. Add role-play fine-tunes of gemma-4-E4B-it as experts, mergekit-moe style. Each expert is Bolt's FFN plus one donor's FFN change (donor minus gemma-4-E4B-it, as low-rank factors). Expert 0 is Bolt's own FFN.
   • AuriAetherwiing/G4-E4B-Musica-v1-lora @ b87c464f
   • chimbiwide/Gemma4NPC-E4B @ cbd40b61
   • juiceb0xc0de/bella-bartender-gemma-e4b @ 46c7071a
   • ToastyPigeon/Gemma4-E4B-TIES-3mask-r64 @ 34fda2cb
3. Gemma 4's native MoE block on every layer, beside the dense FFN: five experts, each a complete 10,240-wide FFN, top-1.
4. Router set from hidden states without training, as mergekit-moe's hidden gate does: each expert's row points at Bolt's hidden states on that donor's kind of text; expert 0 takes general and maths text.

20.5B effective parameters (23.98B with the per-layer embeddings, 48 GB in bf16), about 7.3B active. A stock Gemma4ForCausalLM config that loads in transformers and mlx-lm. The chat template is Gemma 4's own; Bolt's variant orders tool-call messages differently, which may suit agentic use better.

Status

No damage a blind LLM judge could see against Bolt's role-play; before any training it predicts held-out human role-play 0.26 nats a token better than Bolt, each scored at its best temperature. Known issue: the router sends only 29% of general text to Bolt's expert, and Musica's expert wins only 14% of Musica's own text. A fitted router is planned.

Fine-tuning notes

These notes are for anyone who trains LoRA adapters on Gemma 4 26B-A4B with Axolotl and releases merged checkpoints. P6 has 26B-A4B's MoE block (a dense FFN beside routed experts in every layer) on an E4B body. Nobody has trained P6 in Axolotl yet. These notes come from reading Axolotl 59b27488, transformers 5.20.2 and PEFT, and from how P6 is built. axolotl-lora.yaml applies them to Axolotl's 26B-A4B bnb-LoRA recipe.

• Retarget the LoRA regex. P6 is a text-only Gemma4ForCausalLM, so its layers are model.layers.N, not model.language_model.layers.N, and Axolotl's 26B-A4B regex matches nothing. Use lora_target_modules: 'model.layers.[\d]+.(_checkpoint_wrapped_module.)?(mlp|self_attn).(up|down|gate|q|k|v|o)_proj'. Only layers 0–23 have k_proj and v_proj; layers 24–41 reuse earlier layers' K/V (E4B's KV sharing).
• Leave the router out. Under top-1 the routing weight is always 1, so router.proj, router.scale and router.per_expert_scale get no useful gradient: training them does nothing. The regex above skips the router, and lora_target_linear would wrap router.proj for no benefit. A later revision of P6 is likely to replace the router.
• Expert LoRA is optional. lora_target_parameters: [experts.gate_up_proj, experts.down_proj] with quantize_moe_experts: true should work as on 26B-A4B (untested). P6 has 5 experts of width 10,240, top-1, against 26B-A4B's 128 of width 704, top-8. Each token updates only the expert it is routed to, so an expert your data rarely reaches barely changes. Without expert LoRA the donor experts stay as built, while attention and the dense FFN (Bolt's own, seen by every token) adapt.
• Turn off the fused LoRA kernels. Axolotl's Gemma 4 README says they don't support KV-sharing layers: set lora_mlp_kernel, lora_qkv_kernel and lora_o_kernel to false. Keep chat_template: gemma4 and eot_tokens: ["<turn|>"].
• Leave the embeddings frozen. That covers E4B's per-layer embedding table (embed_tokens_per_layer, 2.8B) and the embedding table, which is tied to the output head. Adding lm_head to modules_to_save unties it and adds 0.67B.
• Memory and speed. P6's total is close to 26B-A4B's (24.0B against 25.2B), so QLoRA with quantize_moe_experts should need similar VRAM. The per-layer embedding table stays in bf16 (about 5.6 GB). About 7.3B parameters are active per token against 26B-A4B's 3.8B, so expect roughly twice the compute per step.
• Merge with axolotl merge-lora or PEFT, not mergekit. merge-lora folds expert adapters into the bf16 weights. mergekit's Gemma 4 support doesn't model per-layer embeddings or KV sharing. Keep config.json and the chat template as shipped. llama.cpp conversion of P6 is untested.
• Read early loss with care. Bolt is over-confident on human-written role-play: 5.24 nats a token at temperature 1 against 3.82 at its best temperature. A fast early drop on such data is mostly that correcting, not new learning.

Licenses

The terms of every source apply: Bolt, OpenMDW-1.1; gemma-4-E4B-it and Gemma4NPC, Apache 2.0; Musica's base, Apache 2.0 (its adapter card states none); Bella and TIES-3mask, the Gemma license on their cards.