GrEarl/Kimi-K3-NVFP4A16-Requantized

Kimi-K3 NVFP4A16 — Standard Requantization

Weight-only NVFP4A16 derivative of source revision moonshotai/Kimi-K3@9f62e4e9fffbd0a83ddd60e1c209d828994b3569.

Method

MXFP4 values are dequantized per tensor and requantized with the compressed-tensors memoryless_minmax NVFP4 algorithm. Gate/up experts share one global scale as required by fused MoE loaders; w2 uses its own tensor global scale.

Pinned shard 49, expert 0 real-weight sample versus MXFP4: w1 relative RMSE 0.064053 / cosine 0.998657480; w2 0.063678 / 0.998681610; w3 0.063978 / 0.998660965. All pass relative RMSE < 0.10 and cosine similarity > 0.99.

• Format: compressed-tensors, nvfp4-pack-quantized
• Weights: FP4 E2M1, tensor-group strategy, group size 16
• Local scales: FP8 E4M3FN
• Activations: unquantized (input_activations: null)
• Quantized modules: routed MoE experts only; all ignored K3 modules remain bit-identical
• Source revision: 9f62e4e9fffbd0a83ddd60e1c209d828994b3569
• Source provenance: pinned-staging-config-and-commit-window
• Lossless over-span policy: fail; never silently requantize
• Tensor data size: 1.497 TiB
• Size change versus source tensor data: +5.45%
• Companion artifact: GrEarl/Kimi-K3-NVFP4A16-Transcoded

This is NVFP4A16, not calibrated W4A4 NVFP4. Kimi K3 was trained with MXFP8 activations; FP4 activation quantization would require a separate calibration and quality-validation run.

Runtime validation (2026-07-27 UTC)

A complete Stable LatentMoE block passed on vLLM for this published artifact using the official KimiMoE class. The test first loaded and ran the block through a purpose-built layout path, then rebuilt it and loaded the same 8,072 source tensors through the real K3 model-class chain: KimiK3ForConditionalGeneration.load_weights -> AutoWeightsLoader -> KimiLinearForCausalLM.load_weights -> KimiLinearModel.load_weights -> the generic process_weights_after_loading traversal.

• Evidence run: 20260727T234215Z
• Earlier routed-expert-core run: 20260727T220141Z
• Tested artifact revision: 6d82c0bfeb91dec56fbd1b0726ccb45d7692caf4
• Checkpoint shard: model-00049-of-000096.safetensors (17,916,197,160 bytes)
• Runtime image: vllm/vllm-openai:kimi-k3
• Pinned linux/amd64 image: vllm/vllm-openai@sha256:fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e
• vLLM: 0.1.dev19262+gb6bbf29dd.d20260727
• Hardware: NVIDIA B200, SM100; torch 2.13.0+cu130, CUDA 13.0
• Complete path: FP32 router + correction bias, 7168 -> 3584 latent down, 896 routed experts with real top-k 16 and 3584 -> 3072 -> 3584, routed RMSNorm + 3584 -> 7168 up, and 2 shared experts (7168 -> 6144 -> 7168)
• Loaded source tensors: 8,072 = 8 nonexpert block tensors + 8,064 routed-expert quant tensors; all 896 experts
• SiTU: beta 4.0, linear beta 25.0
• Backend: MARLIN / MarlinExperts
• Observed operators: vllm::moe_forward_shared, _moe_C::moe_wna16_marlin_gemm, _moe_C::grouped_topk, _C::situ_and_mul, vllm_ir::rms_norm, routing alignment, and MoE sum
• Router logits: [2, 896]; selected IDs [2, 16]; 16 unique in-range experts per token; finite normalized weights
• Both direct and real model-class-loader outputs: [2, 7168], finite and nonzero; sampled parameter fingerprints matched exactly and outputs were bit-exact (max_abs_difference=0)
• Complete-block validation run cost: $0.341178; all attempts including the externally canceled run: $3.892933 < $4.5

This verifies the real K3 outer/inner load_weights mapping and generic post-load traversal on a reduced one-block wrapper. DefaultModelLoader index/file traversal was not exercised, and the full model was not constructed.

SGLang runtime was not executed. A zero-cost gate against the official lmsysorg/sglang:kimi-k3 image (linux/amd64 digest sha256:2e8ef3746b2591287db0f37b4470910ab08974bdaf96fc5820bac35f6d3962bc) found that its compressed-tensors selector supports FLOAT tensor-group NVFP4 only as W4A4 with input activation scales; it has no matching NVFP4A16 FLOAT tensor-group MoE path for input_activations: null. The gate therefore returned BLOCKED / DO_NOT_RUN_GPU (20260727T221943Z) before weight download or GPU allocation.

Not verified: the full 93-layer / 2.8T model, all 96 shards resident together, DefaultModelLoader index/file traversal, KDA/MLA/vision integration, TP16, end-to-end token generation, serving quality, or SGLang load/forward/kernel dispatch.

License

Inherits the Kimi K3 License.