Blackfrost-AI/GLM-5.2-ABLITERATED-NVFP4

GLM-5.2-ABLITERATED — NVFP4 (Blackwell / SM120) 8x RTX 6000 BLACKWELL

The Blackfrost-verified build of the GLM-5.2-ABLITERATED family: an abliterated (refusal-removed) GLM-5.2 quantized to NVFP4 (4-bit) and packaged to serve on 8× NVIDIA RTX PRO 6000 Blackwell (SM120) with vLLM. This repository ships the weights and a complete, reproducible serving recipe — bringing a glm_moe_dsa NVFP4 model up on Blackwell required solving four distinct initialization blockers, all documented below.

⚠️ Uncensored. Refusal directions in the residual stream have been ablated, so the model does not decline requests on content-policy grounds. Read Responsible use before downloading.

---

TL;DR

• Verified by Blackfrost: serves at ~25–27 tok/s single-stream on 8× RTX PRO 6000 (TP=8), OpenAI-compatible endpoint; 0/15 previously-refused prompts refused.
• Base: Blackfrost-AI/GLM-5.2-ABLITERATED-BF16 — reconstructed from the huihui-ai QK3 GGUF (see pipeline).
• This repo: NVFP4 (4-bit, group-size 16), experts-only, ≈420 GB — practical inference on ~768 GB of Blackwell VRAM.
• Everything you need is in deploy-kit/ — one config patch, one launch script, one smoke test.

---

Model Details

Architecture	GlmMoeDsaForCausalLM (glm_moe_dsa) — GLM MoE with Multi-head Latent Attention (MLA) + DeepSeek-style Sparse Attention (DSA)
Parameters	Mixture-of-Experts, ~753B total (NVFP4 footprint ≈ 420 GB)
Layers	78 (first 3 dense, remaining 75 MoE)
Hidden size	6144
Experts	256 routed, 8 active per token, + 1 shared expert
Attention	MLA (kv_lora_rank=512, q_lora_rank=2048) + DSA sparse indexer
Vocab	154,880
Max context	1,048,576 (served at 32,768 by default; raise per VRAM budget)
Quantization	NVFP4 (quant_algo: NVFP4, group-size 16); MLA projections, router gates, shared experts, embeddings, and MTP kept in BF16 (see config.json → quantization_config.ignore)
Chat template	GLM control tokens preserved (<think>, `<

Abliteration

The refusal-mediating directions in the residual stream have been identified and removed, so the model will not refuse on content-policy grounds. This behavior is inherited from the abliterated source (huihui-ai); the quantization here neither adds nor restores any safety behavior. In Blackfrost verification the model engaged on 15/15 previously-refused prompts (0 refusals).

---

How this was made — QK3 → BF16 → NVFP4

The only public form of an abliterated GLM-5.2 is a UD-Q3_K_M GGUF ("QK3") from huihui-ai. The family is reconstructed from it and quantized from a common BF16 source:

huihui-ai/Huihui-GLM-5.2-abliterated  ·  UD-Q3_K_M GGUF  ("QK3", ~343 GB)
        │   streaming dequantization → BF16
        ▼
Blackfrost-AI/GLM-5.2-ABLITERATED-BF16    (BF16 safetensors, ~1.4 TB, 337 shards)
        │   streaming, weight-only NVFP4 (4-bit, group-size 16), experts-only
        ▼
Blackfrost-AI/GLM-5.2-ABLITERATED-NVFP4   (this repo — ≈420 GB, 337 shards)  ← VERIFIED

Only the routed-expert projections (model.layers.*.mlp.experts.*.{gate,up,down}_proj) are quantized to 4-bit; the abliteration-sensitive attention pathway (MLA, DSA indexer, router gates, shared experts) stays BF16 — which is why the de-refusal behavior survives quantization. The sibling FP8 build is produced from the same BF16 by the same experts-only method. All conversions are streaming (shard-by-shard, no calibration data) and were run on 8× RTX PRO 6000 Blackwell (SM120).

---

Quick Start

# 1. Download the weights
hf download Blackfrost-AI/GLM-5.2-ABLITERATED-NVFP4 --local-dir /models/GLM-5.2-ABLITERATED-NVFP4

# 2. (Optional) The shipped config.json is ALREADY patched with the fused_qkv_a_proj fix.
#    This just confirms it — expect "Patched":
python3 deploy-kit/patch_config.py /models/GLM-5.2-ABLITERATED-NVFP4/config.json --check

# 3. Launch (MTP disabled — required on this stack, see Blocker 3;
#    chat templates auto-load from deploy-kit/recipe/)
MTP=0 MODEL=/models/GLM-5.2-ABLITERATED-NVFP4 bash deploy-kit/serve_glm52abl_nvfp4.sh

# 4. Smoke test
bash deploy-kit/smoke_test.sh

First boot takes ~3–4 minutes (weight load + kernel compilation + CUDA-graph capture). Endpoint: http://localhost:8000/v1 (model name glm52abl).

---

Serving Recipe — the four boot blockers

This glm_moe_dsa NVFP4 model does not boot out-of-the-box on Blackwell. Four issues each crash the server at a different init stage. All four fixes are baked into deploy-kit/serve_glm52abl_nvfp4.sh; here is what and why.

#	Stage	Symptom	Root cause	Fix
1	NCCL init	hang/crash at ncclCommInitRank on 8-way TP	serving image bakes NCCL 2.30.4 via LD_PRELOAD; regression on cross-NUMA SYS topology	neutralize the NCCL LD_PRELOAD/path env vars → fall back to bundled NCCL 2.29.7
2	Weight load	AssertionError: shape mismatch [3027 vs 6144] on fused_qkv_a_proj	vLLM fuses MLA q_a_proj + kv_a_proj_with_mqa into fused_qkv_a_proj, which is not in the NVFP4 ignore list → gets 4-bit-quantized and halves its output dim	add *.self_attn.fused_qkv_a_proj to quantization_config.ignore (keep BF16). Use patch_config.py
3	Model init	moe_backend='b12x' is not supported for unquantized MoE	MTP/eagle draft model doesn't inherit the NVFP4 quant config, so its MoE looks unquantized to the b12x backend	disable MTP speculative decoding: MTP=0 (costs ~2–3× decode throughput; correctness unaffected)
4	CUDA-graph capture	custom_all_reduce.cuh:455 'invalid argument'	image bakes VLLM_ENABLE_PCIE_ALLREDUCE=1; the custom C++ PCIe all-reduce kernel fails on SM120 (no NVLink, all-PCIe box)	VLLM_ENABLE_PCIE_ALLREDUCE=0 + --disable-custom-all-reduce → NCCL all-reduce

Full narrative for each blocker (with log excerpts and verification) is in deploy-kit/README.md.

The config patch (Blocker 2)

  "quantization_config": {
    "ignore": [
      ...
      "*.shared_head*",
+     "*.self_attn.fused_qkv_a_proj"
    ]
  }

patch_config.py applies this idempotently (with --check / --revert).

---

Environment (verified working)

Component	Version
GPU	8× NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB each)
Architecture	SM120 (sm_120a)
Interconnect	PCIe (no NVLink)
Serving image	Black Benediction vLLM (voipmonitor/vllm:black-benediction-b12x…cu132-20260608)
vLLM	0.11.2.dev279 (black-benediction b12x)
PyTorch / CUDA	2.12.0 + cu132 / CUDA 13.2
NCCL (effective)	2.29.7 (image's 2.30.4 disabled)

Key serve flags: --quantization modelopt_fp4 --attention-backend B12X_MLA_SPARSE --moe-backend b12x --kv-cache-dtype fp8 --tensor-parallel-size 8 --decode-context-parallel-size 8 --disable-custom-all-reduce --tool-call-parser glm47 --reasoning-parser glm45. The DSA per-layer indexer pattern is passed via --hf-overrides '{"index_topk_pattern":"…"}' (vLLM reads index_topk_pattern, not config.indexer_types) — required for long-context coherence. See the launch script for the exact values.

---

Hardware requirements

• ~420 GB of weights + KV cache. It fits 8× 96 GB Blackwell at --gpu-memory-utilization 0.95, max-model-len 32768, max-num-seqs 2. Longer context / more concurrency needs more headroom.
• SM120 is required for the b12x kernels in this recipe. Other architectures need a different backend (and will not use the b12x env flags).

---

Responsible Use

This model has had safety refusals removed. That makes it useful for red-teaming, security research, evaluation, and unfiltered assistant tasks — and it means the operator must supply the guardrails a user would otherwise rely on.

Prohibited uses:

• Anything involving the sexual exploitation or endangerment of minors.
• Content promoting self-harm or suicide.
• Generation of material that is illegal in your jurisdiction, or that targets real individuals for harassment, doxxing, or fraud.
• Any use prohibited by the upstream GLM license.

You are responsible for adding appropriate safety filtering, human review, and access controls for your deployment. The weights are provided as-is, with no warranty. The license is inherited from the upstream GLM-5.2 base model — review and comply with it before use or redistribution.

Limitations

• 4-bit quantized — expect quality below the full-precision base, especially on long, hard reasoning.
• MTP speculative decoding disabled on this stack (~2–3× slower decode than with MTP). Re-enabling requires a vLLM eagle-loader patch to propagate the quant config to the draft model.
• Custom PCIe all-reduce and NCCL 2.30.4 are disabled for SM120 stability; minor collective-latency overhead.
• Refusal behavior was validated by spot check (0/15), not an exhaustive benchmark.

---

Published by Blackfrost AI. This card documents the verified NVFP4 build of the GLM-5.2-ABLITERATED family, the exact QK3 → BF16 → NVFP4 pipeline used to produce it, and a complete, reproducible serving recipe for 8× RTX PRO 6000 Blackwell.