SierraWing/L3.3-70B-Loki-V2.0-nvfp4

Base model: CrucibleLab/L3.3-70B-Loki-V2.0

Quantization: NVFP4 with FP8 KV cache

Generated with: nvidia-modelopt on B200 hardware

Tested with: vLLM on GB10 DGX Spark (SM121)

Launch command (on SM121):

vllm serve ~/models/L3.3-70B-Loki-V2.0-nvfp4
--max-model-len 65536
--host 0.0.0.0
--port 8000
--gpu-memory-utilization 0.50
--kv-cache-dtype fp8

Quantization Notes

Weights and calibration scales were generated using nvidia-modelopt (hf_ptq.py) with NVFP4 precision and FP8 KV cache quantization. Calibration was performed on B200 hardware. KV cache scales (k_scale/v_scale) were subsequently extracted and injected into the checkpoint on a GB10 DGX Spark (SM121) to produce a self-contained HuggingFace-compatible checkpoint.

Known issues

TRT-LLM PyTorch backend produces incoherent output on SM121

The checkpoint does not include a calibrated q_scale for the FP8 KV cache. vLLM will warn at startup that q_scale is being set to k_scale as a fallback. This is expected and has not been observed to cause meaningful quality degradation in practice.

HuggingFace's model size display shows ~36B parameters due to NVFP4 weight packing. The actual model is 70B parameters (CrucibleLab/L3.3-70B-Loki-V2.0).