Infatoshi/diffusiongemma-26B-A4B-iq2xxs-4070

DiffusionGemma 26B-A4B IQ2 — pack for RTX 4070 (12 GB)

Ready-to-run IQ2_XXS GGUF of Google’s DiffusionGemma 26B-A4B plus two llama.cpp patches that cut ~2.7 GiB of waste so a desktop 4070 12 GB can run it.

Weights	diffusiongemma-26B-A4B-iq2xxs.gguf (~9.4 GiB)
Arch	Block-diffusion MoE (~25B total / ~3.8B active)
Target	RTX 4070 12 GB, concurrency 1, short–medium prompts
Measured peak	~10.4 GiB (nvidia-smi) on 3090 proxy with ship flags
Speed (3090)	~47–67 tok/s — 4070 will be slower (less bandwidth + CPU-offloaded experts)

Not stock llama.cpp. You need PR #24423 (llama-diffusion-cli) and patches/4070-vram.patch from this repo. Unpatched PR peaks ~13.8 GiB and will OOM a 12 GB card.

---

Quick start (Linux + NVIDIA)

1. Download this repo

# huggingface-cli
hf download Infatoshi/diffusiongemma-26B-A4B-iq2xxs-4070 --local-dir diffusiongemma-4070
cd diffusiongemma-4070

2. Build the patched diffusion runner

# CUDA toolkit required
./scripts/build_llama_diffusion_4070.sh ~/src/llama.cpp-diffusion-4070
export PATH="$HOME/src/llama.cpp-diffusion-4070/build/bin:$PATH"
# or: export CLI=$HOME/src/llama.cpp-diffusion-4070/build/bin/llama-diffusion-cli

Manual equivalent:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/24423/head:diffusiongemma && git checkout diffusiongemma
git apply /path/to/this-repo/patches/4070-vram.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-diffusion-cli

3. Run on a 4070

# Desktop GPU that also drives the display (recommended default)
export CLI=~/src/llama.cpp-diffusion-4070/build/bin/llama-diffusion-cli
export N_CPU_MOE=4
./scripts/run_4070.sh "Why is the sky blue? Answer in two short sentences."

Or one-liner:

llama-diffusion-cli \
  -m diffusiongemma-26B-A4B-iq2xxs.gguf \
  -ngl 99 --n-cpu-moe 4 -n 256 -ub 1 -b 1 \
  --diffusion-eb auto --perf \
  -p "Your prompt here"

Flag	Why
-ngl 99	Layers on GPU
--n-cpu-moe 4	~3 expert layers on CPU RAM (fits 12 GB + display headroom). Headless: try 3
-n 256	One diffusion canvas of 256 tokens
-ub 1 -b 1	Lets the patched CLI size logits to real prompt+canvas (not 2304 ghost tokens)
--diffusion-eb auto	Entropy-bound sampler (default quality path)

System RAM: leave ≥16 GB free for the few MoE layers on CPU (n-cpu-moe).

---

What the patches do

1. Ghost ubatch fix (diffusion-cli.cpp)
   Stock PR sizes n_ubatch = canvas + 2048 even for one-shot prompts → ~2 GiB wasted fp32 logits at vocab 262k. One-shot runs size to the real tokenized prompt instead.

2. Self-cond embedding Q8 (diffusion-gemma.cpp)
   Device SC embedding table was a full F16 dequant copy (1.4 GiB); store as Q8_0 (0.7 GiB). Quality checked vs F16 SC.

Together with --n-cpu-moe 3–4, peak drops from ~13.8 GiB → ~10.4 GiB.

---

Measured numbers (anvil RTX 3090, same flags; 4070 not in fleet)

Config	Peak VRAM	tok/s (3090)
Stock PR #24423, full GPU	~13.83 GiB	~111
Patched, full GPU (ncmoe=0)	~11.44 GiB	~93–112
Ship: patched + ncmoe=3	~10.40–10.42 GiB	~47–67
Desktop margin: ncmoe=4	~10.15 GiB	~58
-n 512 (2 canvases)	use ncmoe=5	~58

• Full experts on GPU still need ~11.4 GiB after patches → over a clean 10.5 GiB desktop budget.
• 4070 bandwidth is lower than 3090; expect roughly half full-GPU speed. Offloaded experts also depend on your CPU/DDR.

Quality (50%-mask pseudo-ppl vs Unsloth Q4_K_M): IQ2 within ~9% of Q4 — usable for chat/coding smoke; not near-BF16.

---

Longer prompts / more tokens

• Each extra ~260 prompt tokens grows ubatch; add about +1 n-cpu-moe.
• -n 512 (two canvases): use N_CPU_MOE=5.
• Interactive multi-turn still uses larger headroom (conversation mode keeps slack).

---

Attribution

Base model	google/diffusiongemma-26B-A4B-it (Apache 2.0)
IQ2 GGUF	Redistributed from hyperspaceai/diffusiongemma-26B-A4B-iq2xxs-GGUF
Runner	ggml-org/llama.cpp PR #24423 + local VRAM patches in this repo
Packaging / measurements	Infatoshi (2026-07)

This repo does not claim the quant methodology; we only package weights + runtime instructions that fit 12 GB after measurement.

---

Troubleshooting

Symptom	Fix
OOM on 12 GB	Raise N_CPU_MOE to 4 or 5; ensure -ub 1 -b 1; confirm patched binary
llama-cli / llama-server fails	Use llama-diffusion-cli only — standard AR runners cannot generate this model
Slow	Expected with CPU MoE; close other GPU apps; more RAM bandwidth helps
Patch won’t apply	PR #24423 moved — open patches/4070-vram.patch and port the two hunks by hand

---

Files

diffusiongemma-26B-A4B-iq2xxs.gguf   # weights (~9.4 GiB)
patches/4070-vram.patch              # required llama.cpp changes
scripts/build_llama_diffusion_4070.sh
scripts/run_4070.sh
README.md