Qwen3-VL-8B-Instruct-heretic-NVFP4

Overview

This repository provides an NVFP4 (FP4 E2M1) mixed-precision quantized build of Qwen3-VL-8B-Instruct-heretic in ComfyUI comfy_quant format, intended for use as a Qwen3-VL text encoder wherever one is needed.

This model is a decensored derivative of the official Qwen/Qwen3-VL-8B-Instruct, modified using Heretic.

Vision kept in FP16: ComfyUI's qwen3vl text-encoder does run the vision tower in image/multimodal workflows (sd1_clip.process_tokens → Qwen3VL.preprocess_embed → self.visual), and the vision tower is small but image-quality-sensitive, so leaving it in FP16 preserves visual conditioning fidelity at a modest size cost. This matches the stock Qwen3-VL-8B NVFP4 baseline, which also ships vision in bf16.

Quantization Details

• Backend: ComfyUI convert_to_quant v1.2.6 (comfy_kitchen CUDA NVFP4 kernels)
• Format: comfy_quant mixed precision
  • NVFP4 (FP4 E2M1, 16-element blocks, learned rounding) — all text transformer blocks 0–35 projections
  • FP8 (float8_e4m3fn, tensorwise, RTN) — embed_tokens and lm_head
  • FP16 (bfloat16) — model.norm, all norms/biases, pos_embed, patch_embed, and the entire vision encoder (blocks, merger, deepstack_merger_list)
• No input_scale: ComfyUI quantizes activations dynamically at runtime (per-tensor amax via the NVFP4 layout), so a baked-in activation scale is not required. This matches the stock Qwen3-VL NVFP4 baseline.
• Hardware target: NVIDIA Blackwell (RTX 50 series, SM 12.0) — required for native NVFP4 tensor-core execution (see Runtime below).
• Size: 6.31 GB / 1510 tensors (vs 17.53 GB FP16 source — ~64% smaller).

Layer breakdown

Tier	Layers	Count
FP16 (bf16)	model.norm, all norms/biases, pos_embed, patch_embed, entire vision encoder	kept lossless
FP8 (e4m3fn, tensorwise)	embed_tokens, lm_head	2 weights
NVFP4 (E2M1, block=16)	all text blocks 0–35 projections	252 weights

Runtime / hardware

How the layers execute depends on the GPU (ComfyUI pick_operations gates each format on the device capability):

GPU (SM)	NVFP4 layers	FP8 layers
RTX 5090 / Blackwell (12.0)	native FP4 tensor cores (fast)	native FP8
RTX 4090 / Ada (8.9)	dequantized to bf16 (size only, no speedup)	native FP8

On Blackwell, NVFP4 weights stay packed and run through comfy_kitchen's TensorCoreNVFP4Layout GEMM (FP4 weight × dynamically-quantized FP8 activation). On non-Blackwell GPUs the format is emulated via dequantization, so you get the smaller footprint but not the FP4 speedup.

Usage (ComfyUI)

Place qwen3_vl_8b_nvfp4_full.safetensors in ComfyUI/models/text_encoders/.

Load it with a CLIPLoader node using type qwen3vl_8b, and use it anywhere a Qwen3-VL text encoder is needed.

ComfyUI auto-detects the quantization metadata (Found quantization metadata version 1) and selects MixedPrecisionOps. The vision weights are included (unquantized) so detect_te_model() identifies the file as QWEN3VL_8B and routes to qwen3vl.te() correctly.

License

Apache-2.0 (inherited from the base model).