HelloSun/openPangu-R-7B-Diffusion-Q4KM

openPangu-R-7B-Diffusion — Q4_K_M GGUF (llama.cpp)

Quantized Q4_K_M GGUF of openpangu/openPangu-R-7B-Diffusion, together with a llama.cpp patch that adds native support for the model's context-causal block-diffusion decoding.

openPangu-R-7B-Diffusion is a 7B block-diffusion LLM. Generation is not plain next-token autoregression: a block of tokens (default length 32) is denoised in parallel with full (bi-directional) attention inside the block and causal attention for the preceding context, revealing the most confident tokens first. The model can also be run in ordinary autoregressive mode.

Upstream llama.cpp (as of tag b11719) ships the PanguEmbedded architecture (used by the autoregressive openPangu-Embedded) but does not implement the diffusion decoding this model needs, hence the patch.

Files

File	Description
pangu-diffusion-Q4_K_M.gguf	Q4_K_M quantized weights (~4.6 GB), ready to run
pangu-diffusion-llamacpp.patch	llama.cpp patch adding openPangu-R-Diffusion support
build_llamacpp_pangu_diffusion.sh	one-click script: clone llama.cpp → apply patch → build

Quick start

1. Build llama.cpp with the patch (one-click)

bash build_llamacpp_pangu_diffusion.sh
# binaries end up in ./pangu-diffusion-build/llama.cpp/build/bin/

Or manually:

git clone --depth 1 --branch b11719 https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git apply /path/to/pangu-diffusion-llamacpp.patch
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
cmake --build build -j --target llama-diffusion-cli llama-cli llama-quantize

2. Diffusion decoding (recommended)

llama-diffusion-cli applies the model's chat template automatically: pass the raw user message, not a pre-formatted prompt. The rendered prompt is

[unused9]系统：<system>[unused10][unused9]用户：<user>[unused10][unused9]助手：

./build/bin/llama-diffusion-cli \
  -m pangu-diffusion-Q4_K_M.gguf \
  -p "Which is bigger, 9.11 or 9.9?" \
  -ub 256 \
  --diffusion-block-length 32 \
  --diffusion-small-block 4 \
  --diffusion-algorithm 1 \
  --temp 0 -t 16 --seed 42

Recommended sampling (from the official model card): alg = entropy, num_small_blocks = 8 → --diffusion-small-block 4 (with block length 32), top_p = 1, temperature = 0. For coding tasks the authors suggest alg = entropy, num_small_blocks = 32 (→ --diffusion-small-block 1), top_p = 0.8, temperature = 1.

Fast mode (higher throughput): reveal every masked token whose confidence exceeds a threshold in a single denoising step, with one small block per block:

./build/bin/llama-diffusion-cli \
  -m pangu-diffusion-Q4_K_M.gguf \
  -p "<question>" \
  -ub 256 --diffusion-block-length 32 --diffusion-small-block 32 \
  --diffusion-confidence-threshold 0.9 --diffusion-algorithm 4 \
  --temp 0 -t 16 --seed 42

--diffusion-n-seq N runs N sequences through the block decoder in a single (micro-)batch (the prompt is replicated; see Batched decoding below).

--diffusion-algorithm values: 0 origin, 1 entropy, 2 margin, 3 random, 4 confidence. -ub N is the maximum sequence length; the tail that does not fill a whole block is dropped.

3. Autoregressive decoding

./build/bin/llama-cli -m pangu-diffusion-Q4_K_M.gguf \
  -p "[unused9]系统：[unused10][unused9]用户：Hi![unused10][unused9]助手：" -n 128 -t 16

Example

Prompt (translated): “Which is bigger, 9.11 or 9.9?” — command as in §2, fast mode (-ub 256, --diffusion-confidence-threshold 0.9, Q4_K_M):

I need to figure out which is bigger between 9.11 and 9.9. Both are decimals, so I should compare them directly.

First, let's write them with the same number of decimal places to make it easier. 9.11 has two decimal places, and 9.9 has one. So, I can write 9.9 as 9.90 to make it two decimal places.

So, 9.11 vs 9.90.

Now, comparing the tenths place: both have 9 in the tenths place. So, that's the same.

Next, the hundredths place: 9.11 has 1 in the hundredths place, and 9.90 has 0 in the hundredths place. Since 1 is greater than 0, ...

I can think of them as fractions. 9.11 is 911/100, and 9.9 is 990/100.

Since the denominator is the same, I can compare the numerators: 911 vs 990.
911 is less than 990, so 911/100 is less than 990/100, which ...

For a short prompt the output is reproduced token-for-token identically to the Hugging Face reference (see Numerical parity below), e.g. -p "hello" (f16 GGUF): [unused16]Okay, the user just said "hello". That's a simple greeting. I should respond politely and warmly. Maybe they're just starting a conversation.

Speed when you actually use it (measured)

Test environment — the resources actually usable inside the container (docker):

Resource	Value
CPU	Intel Xeon Platinum 8559C
Effective cores	16 vCPU (cgroup cpu.max = 1600000 100000)
Memory (cgroup limit)	≈ 97 GiB
GPU	none usable — /dev/nvidia* nodes exist but cuInit() returns CUDA_ERROR_NO_DEVICE and NVML init fails
Build / runtime	CPU-only, -t 16

These are end-user speeds measured from real runs of the CLI on pangu-diffusion-Q4_K_M.gguf (4.59 GiB, 8.03 B params), i.e. what you actually experience while using it:

What the user experiences	Mode	Speed
Input — waiting for the prompt to be read before output starts	both	≈ 78 t/s short prompt; ≈ 132 t/s for longer prompts (llama-bench pp512)
Output — text generated (autoregressive, streamed live)	llama-cli	11.9 t/s (llama-bench tg128: 11.1 t/s)
Output — block diffusion (quality: block 32 / small-block 4, alg=entropy, temp=0)	llama-diffusion-cli	≈ 4.4 t/s
Output — block diffusion (fast: --diffusion-confidence-threshold 0.9, small-block 32, alg=confidence — the official throughput setting)	llama-diffusion-cli	≈ 5 t/s

Notes:

• Input (prompt processing) is the same causal prefill for both paths; it is what you wait for before the answer appears.

• The AR output speed (≈ 12 t/s) is what llama-cli prints as Generation: X t/s; the diffused text is only flushed at the end of a run (unless --diffusion-visual).

• Block diffusion normally reveals one token per denoising step, i.e. up to block_length (= 32) forward passes per block. With --diffusion-confidence-threshold T every masked token whose confidence exceeds T is revealed in the same step (transfer = argmax(conf) | (conf > T)), which raises throughput — the same mechanism the authors report as giving up to 2.5× over autoregressive decoding on Ascend NPUs. A lower T is faster but less accurate; measured on this CPU (235 tokens, block 32 / small-block 32, alg=confidence, temp=0):

  --diffusion-confidence-threshold	speed
  0.9 (default)	4.9 t/s
  0.7	7.4 t/s
  0.5	8.7 t/s

• The AR figures are the best reference for raw model throughput on this hardware.

What the patch does

1. src/llama-arch.cpp — mark LLM_ARCH_PANGU_EMBED as a diffusion architecture (llm_arch_is_diffusion()), so llama-diffusion-cli accepts the model.
2. examples/diffusion/diffusion.{h,cpp} — add a context-causal block-diffusion decoder (diffusion.block_attention):
   • prefill the prompt with a causal forward (kept in the KV cache);
   • denoise each block with causal_attn = false → full attention over the cached context plus the current block (the KV-cache kq mask keeps every non-empty cached cell when non-causal);
   • re-decode the block each step, dropping its stale keys (llama_memory_seq_rm), reveal the most-confident masked token of the current small block, left to right;
   • commit the block with a causal forward so the KV cache holds correct causal context, then sample the first token of the next block from the last position;
   • diffusion_generate_nseq() generalises this to n_seq sequences sharing one micro-batch (see --diffusion-n-seq).
3. examples/diffusion/diffusion-cli.cpp — read diffusion.block_attention, add --diffusion-small-block/-confidence-threshold/-n-seq, size the context for the batched prefill, and use greedy sampling when temperature == 0.
4. gguf-py + conversion/pangu.py — write the new diffusion.block_attention metadata key for diffusion checkpoints (alongside diffusion.shift_logits and tokenizer.ggml.mask_token_id), and set tokenizer.ggml.add_space_prefix = false (see Tokenizer fix below).

Model metadata in the GGUF

general.architecture            = pangu-embedded
diffusion.shift_logits          = true
diffusion.block_attention       = true
tokenizer.ggml.mask_token_id    = 45830
tokenizer.ggml.add_space_prefix = false

Tokenizer fix

The upstream conversion honours add_prefix_space from tokenizer_config.json, but the slow PanguTokenizer (sentencepiece, legacy: true) does not actually prepend a space. With the flag left on, llama.cpp inserted spurious space tokens after every special token ([unused9]系统：… became [unused9]␠系统：…), which changed the input and the generation. The patch forces tokenizer.ggml.add_space_prefix = false; the tokenizer output then matches the reference exactly (a 22-token prompt becomes 22 tokens, was 37).

Numerical parity

Validated against the Hugging Face reference (inference/generation_utils.py, alg=confidence, num_small_blocks=1, threshold=0.9, temperature=0) on the f16 GGUF: for -p "hello" the generated token IDs are identical, token for token, to the reference. (The earlier mismatch was entirely the tokenizer issue above, not the decoder.)

Batched decoding (--diffusion-n-seq)

--diffusion-n-seq N runs N independent sequences through the block decoder inside a single (micro-)batch. The example CLI replicates the prompt, so this is mainly a way to measure batching; a real deployment would fill the rows with different prompts.

All sequences produce byte-identical output to the single-sequence run (verified for N = 4 and N = 8), i.e. the batching is correct.

Measured on the CPU above (Q4_K_M, block 32 / small-block 32, threshold 0.9, greedy, 235 generated tokens):

--diffusion-n-seq	wall time	aggregate	per sequence
1	44.5 s	5.28 t/s	5.28 t/s
4	162.4 s	5.79 t/s	1.45 t/s
8	321.2 s	5.85 t/s	0.73 t/s

Review / honest verdict: on this CPU batching gives essentially no throughput gain — aggregate speed stays flat (≈5.3–5.9 t/s) and the time grows ~linearly with N, so per-sequence speed drops 1/N. That is expected: a CPU decode step is compute-bound and widens ~linearly with n_seq (there are no tensor cores to fill and no memory-bandwidth headroom to exploit), so the fixed per-step work simply multiplies. Batching is where the official 2.5× over autoregressive decoding comes from on Ascend NPUs/GPUs, not here. We kept it because it is correct, cheap to carry, and a prerequisite for any GPU deployment — but do not expect a CPU speed-up from it.

Notes / limitations

• CPU-only, -t 16, ≈ 4.6 GB file: a 256-token block-diffusion run takes tens of seconds to a few minutes (block_length forwards per block). Throughput can be improved with alg = confidence_threshold, num_small_blocks = 1 (see the official README).
• The decoder reproduces the Hugging Face reference token-for-token on the f16 GGUF (see Numerical parity), and the block-diffusion path was cross-checked against the reference logits (cos ≈ 0.996, f16 vs bf16).
• Quantization + the greedy fast mode can change the reasoning: on the adversarial 9.11 vs 9.9 prompt the Q4_K_M fast mode first states the wrong intermediate answer and only corrects itself from the fraction comparison above, whereas the f16 model matches the reference. Use the f16 GGUF, or alg = entropy, temp = 0, when a case must be exactly reproduced.

Supported features

Implemented (matching the reference inference/generation_utils.py):

Feature	Status
Context-causal block diffusion — full attention inside a block, causal attention over the preceding context	✓
Block-by-block decoding (block length 32)	✓ --diffusion-block-length
Committed-block KV cache with a causal mask; first token of the next block decoded from the last position	✓
Small blocks inside a block (num_small_blocks, optimum 4 or 8)	✓ --diffusion-small-block
Variable-length inference / flexible context	✓
Logit shift (diffusion.shift_logits)	✓
Dedicated mask token (tokenizer.ggml.mask_token_id = 45830)	✓
Reveal order: confidence, entropy, topk_margin	✓ --diffusion-algorithm 4/1/2
confidence_threshold fast mode (multi-token reveal per step)	✓ --diffusion-confidence-threshold
EOS early stop	✓
Batched (multi-sequence) block diffusion	✓ --diffusion-n-seq
Autoregressive decoding	✓ llama-cli

Not implemented:

• Vendor NPU kernels — this uses whatever backend llama.cpp was built with (CPU here).

Original model

• Weights: openpangu/openPangu-R-7B-Diffusion
• Architecture details, benchmarks and citation: see the original model card.