Instructions to use LostGentoo/Qwen3.5-4B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LostGentoo/Qwen3.5-4B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="LostGentoo/Qwen3.5-4B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("LostGentoo/Qwen3.5-4B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("LostGentoo/Qwen3.5-4B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LostGentoo/Qwen3.5-4B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LostGentoo/Qwen3.5-4B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/LostGentoo/Qwen3.5-4B-NVFP4

SGLang

How to use LostGentoo/Qwen3.5-4B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LostGentoo/Qwen3.5-4B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LostGentoo/Qwen3.5-4B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use LostGentoo/Qwen3.5-4B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/LostGentoo/Qwen3.5-4B-NVFP4
```

LostGentoo/Qwen3.5-4B-NVFP4

Trusted, reproducible NVFP4 quantization of Qwen/Qwen3.5-4B (Apache-2.0) produced with NVIDIA ModelOpt 0.45 for Blackwell inference (vLLM quantization=modelopt_fp4).

This is a text-chat oriented build of the official VLM wrapper (Qwen3_5ForConditionalGeneration). The vision tower is left unquantized / unused at serve time — send text only.

Why this exists

This release exists to provide a known base + auditable calibration for Qwen3.5-4B in ModelOpt NVFP4 form. It publishes the exact preset, public calibration mix, seed, hashes, and measured speed/quality so the recipe can be re-run independently.

Recipe

Item	Value
Base	`Qwen/Qwen3.5-4B`
Tool	NVIDIA ModelOpt `0.45.0`
Preset	`NVFP4_DEFAULT_CFG` (max calib; group size 16)
KV cache quant	none (`kv_cache_quant_algo: null`)
Excludes	`lm_head`, vision / multimodal modules, `linear_attn.conv1d` / `in_proj_*` (ModelOpt defaults + export)
Calibration	512 public samples, seed 20260730
Mix	~70% `HuggingFaceH4/ultrachat_200k` `train_sft` + ~30% `HuggingFaceFW/fineweb-edu` `sample-10BT`, chat-templated, max seq 4096
Calib SHA-256	`762d54c48ad951318dc641aeb0f2091729bb6f8008ef89126c74ac41eea591b2` (`calib.jsonl`)
Quant host	RTX 5060 Ti (SM120), CUDA 12.8, driver 610.43, torch `2.11.0+cu128`

Scripts live with the producer workspace (build_calib.py, quantize_modelopt.py). AWQ_LITE was attempted but export hit a Qwen fuse/pre_quant_scale shape error; DEFAULT (max) is the published path.

Serve (vLLM ≥ 0.25 recommended; tested 0.26.0)

python -m vllm.entrypoints.openai.api_server \
  --model LostGentoo/Qwen3.5-4B-NVFP4 \
  --served-model-name LostGentoo/Qwen3.5-4B-NVFP4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

vLLM auto-detects ModelOpt NVFP4 (quantization=modelopt_fp4). On Blackwell you should see:

Using CutlassNvFp4LinearKernel for NVFP4 GEMM

Hardware: native FP4 GEMM requires NVIDIA Blackwell. Non-Blackwell will not get the intended speedup.

Tip: Qwen3.5 may emit a “Thinking Process” preamble. Prefer higher max_tokens for evals, or disable thinking in the chat template if your stack supports enable_thinking=false / /no_think.

Measured — gentoo 1× RTX 5060 Ti (SM120), vLLM 0.26.0

Server-side deltas via vllm_probe.py delta + ctxsweep / prefill_probe (unique prompts).

Decode aggregate @6144 ctx

Conc	Agg tok/s
1	87
4	218
8	292
16	341
24	355

Whole-matrix server delta decode ≈ 334 tok/s. Prefill peak @6144 ≈ 17.1k tok/s (conc1 fit ≈ 16.2k). 0 preemptions / sweep cell errors. Kernel: CutlassNvFp4LinearKernel.

Quality (thinking disabled where noted)

Check	Result
50 synthetic product-support spot prompts	50/50 coherent replies
IFEval (`chat_smoke`, n=541, thinking on)	prompt_level_strict 38.3%, inst_level_strict 49.5%
IFEval (`chat_smoke`, n=541, `enable_thinking=false`)	prompt_level_strict 72.1%, inst_level_strict 79.9%
Needle-in-haystack	Pass @4096 with long completion budget; thinking-on was the main source of flakiness
MMLU-Pro (`max_samples=1000`, `enable_thinking=false`)	NVFP4 31.2%, bf16 28.7%, AWQ 28.5%
MMLU-Pro (`max_samples=3000`, `enable_thinking=false`)	NVFP4 31.27%, bf16 29.23% (Δ +2.03 pp)
MMLU-Pro gate vs bf16	Fails the plan threshold: NVFP4 stays ~+2 pp vs bf16 on both 1k and 3k subsets (target `<=1.5 pp`)

Cloud Run RTX PRO 6000 spot

Service qwen35-4b-axion-nvfp4-b6000, vLLM v0.24.0 image + stdlib proxy, max-num-seqs=128. Native fp4_gemm.

Decode @6144	Agg tok/s
C1	162
C16	1006
C64	1486

Prefill peak @6144: 69.7k tok/s. 0 errors / preemptions. Scaled to minScale=0 after the spot.

5060 Ti remains a consumer SM120 signal; B6000 is the G4-class capacity number.

Files

model.safetensors — NVFP4 weights (+ unquantized excluded modules)
hf_quant_config.json — ModelOpt producer metadata
config.json — architectures: [Qwen3_5ForConditionalGeneration], quantization_config.quant_algo: NVFP4
Tokenizer / processor / chat template from the base model

Limitations

VLM architecture quirk: not a pure ForCausalLM checkpoint (vLLM text-only path still works).
Public UltraChat/FineWeb calib — not domain-tuned for any private corpus.
Default thinking mode can burn completion budget; configure your client accordingly.
Consumer Blackwell ≠ datacenter B6000 bandwidth; re-bench on target SKU before capacity planning.

Acknowledgements

Base model: Qwen team (Qwen/Qwen3.5-4B)
Quantization: NVIDIA ModelOpt
Structural recipe validated against vLLM loading behavior and the exported hf_quant_config.json layout

Citation

@misc{lostgentoo_qwen35_4b_nvfp4,
  title  = {Qwen3.5-4B NVFP4 (ModelOpt, public calib)},
  author = {LostGentoo},
  year   = {2026},
  url    = {https://huggingface.co/LostGentoo/Qwen3.5-4B-NVFP4}
}

Downloads last month: 208

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for LostGentoo/Qwen3.5-4B-NVFP4

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(337)

this model