Instructions to use ai-babai/Qwen3.5-9B-NF4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ai-babai/Qwen3.5-9B-NF4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ai-babai/Qwen3.5-9B-NF4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ai-babai/Qwen3.5-9B-NF4")
model = AutoModelForMultimodalLM.from_pretrained("ai-babai/Qwen3.5-9B-NF4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ai-babai/Qwen3.5-9B-NF4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ai-babai/Qwen3.5-9B-NF4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-babai/Qwen3.5-9B-NF4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ai-babai/Qwen3.5-9B-NF4

SGLang

How to use ai-babai/Qwen3.5-9B-NF4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ai-babai/Qwen3.5-9B-NF4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-babai/Qwen3.5-9B-NF4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ai-babai/Qwen3.5-9B-NF4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ai-babai/Qwen3.5-9B-NF4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ai-babai/Qwen3.5-9B-NF4 with Docker Model Runner:
```
docker model run hf.co/ai-babai/Qwen3.5-9B-NF4
```

Qwen3.5-9B NF4

Ready-to-load 4-bit NF4 weights for Qwen/Qwen3.5-9B, packaged in the Hugging Face Transformers + bitsandbytes format. This checkpoint is for Linux/CUDA users who want a persisted NF4 model instead of quantizing the upstream BF16 checkpoint each time it is loaded.

This is the only and default artifact in this repository: 11 sealed model files totaling 7,903,101,799 bytes (7.36 GiB), including four safetensors shards. The weights were derived directly from the exact upstream commit c202236235762e1c871ad0ccb60c8ee5ba337b9a without training.

Quick start

The released artifact was created and reload-verified with these exact core versions:

python -m pip install \
  "torch==2.8.0" \
  "transformers==5.16.1" \
  "bitsandbytes==0.50.2" \
  "safetensors==0.8.0" \
  "tokenizers==0.23.2" \
  accelerate

Install the CUDA build of PyTorch appropriate for your system. The export host used PyTorch 2.8.0+cu128.

import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

model_id = "ai-babai/Qwen3.5-9B-NF4"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.float16,
    device_map={"": 0},
    attn_implementation="sdpa",
).eval()

The quantization configuration is already stored in config.json; do not pass a second runtime BitsAndBytesConfig when loading this persisted checkpoint.

What is preserved

Base model: Qwen/Qwen3.5-9B.
Base revision requested and observed: c202236235762e1c871ad0ccb60c8ee5ba337b9a.
Method: bitsandbytes 4-bit NF4, double quantization enabled, FP16 compute, uint8 quantization storage, no CPU offload.
Quantization scope follows bitsandbytes conversion with no skip-module list. Quantization state is present in both language-model and visual-tower weight namespaces; the visual tower is not claimed to remain wholly FP16/BF16.
Tokenizer, chat template, image processor, processor configuration, model configuration, and generation configuration are included with the weights.
The public prequantized-receipt.json records every artifact filename, byte size, SHA-256 hash, model fingerprint, semantic hashes, package versions, and the GPU used for export.

The original artifact was reloaded locally on an NVIDIA A40. Its complete parameter/buffer fingerprint and unpacked bitsandbytes quantization-state fingerprint matched the pre-save model exactly; processor and configuration semantic hashes also matched. The receipt records 763 tensors/buffers and 358 parameters carrying quantization state.

These checks establish provenance, packaging integrity, and exact persistence of the runtime NF4 state. They are not a BF16 quality evaluation. This release does not claim that NF4 preserves BF16 accuracy, benchmark scores, latency, throughput, RAM use, or VRAM use.

Post-publication CUDA validation

The exact public artifact at commit acce7768149fd2dd5eaad7d188153afbd948b2c0 was downloaded, checked against all 11 sealed-file hashes, and loaded on an NVIDIA RTX 2000 Ada Generation GPU. All model parameters and buffers were on CUDA. Multiple bounded runs completed successfully at the execution level: generation returned nonempty text, reached EOS, and did not hit the 256-token limit.

However, a literal-transcription check on a neutral 640×640 synthetic PNG (SHA-256 793c2af735b159f08ae2f8726e4c5ecc2dd5aa230c9a7bb2fd5c250598536ba2) failed. The image visibly contains A7 B3, ROOM 204, and TOTAL 51.25. With thinking disabled and deterministic greedy generation:

Return only one short line containing the visible text. Do not explain. returned 100%.
A strict three-field JSON transcription prompt returned 100% for all three fields.

The same result was reproduced through a separate inference path using the same verified artifact and image. Therefore, the checks establish artifact integrity, CUDA loading, forward generation, and EOS completion, but this checkpoint did not correctly transcribe that bounded synthetic test case. No BF16 control was run, so this result does not establish that NF4 quantization caused the failure or measure quality relative to the upstream model.

Verification

After downloading the repository, verify the artifact without loading the model:

python verify_artifact.py /path/to/downloaded/model

Expected immutable identifiers:

Item	Value
Upstream commit	`c202236235762e1c871ad0ccb60c8ee5ba337b9a`
Artifact tree SHA-256	`14216ff3908aae760af777b6e27cee14561ecb6ba6cca0f2ca3fdcd82d1e6bf8`
Model fingerprint SHA-256	`6c0149dea87b8b48b9c08a24f78c775e879cffc43f851b58fadaaeb281bdca99`
Quantized parameter-name SHA-256	`86c7cb22eb31ed86a253b14a9012cca2a3d526c8002653f1794f089555c16b1c`
Original receipt SHA-256	`91612ffbbbeff8fa52f0df154f3d77930d19ca8eb763d4693d7d8a7526074bfc`

Reproduction scripts and the observed package lock are included in the repository root. They pin the upstream commit, reject non-Linux/non-CUDA execution, quantize directly from the upstream checkpoint, save safetensors shards, reload them, and compare semantic and tensor/quantization-state fingerprints before writing a receipt. export_nf4.py is a path-free public reconstruction of the executed method; the exporter_sha256 field in the receipt binds the exact historical exporter source. Rebuilding can produce different serialized bytes if the environment or libraries differ.

Limitations

Validated only on Linux with NVIDIA CUDA; CPU, Apple Silicon/MPS, ROCm, and other accelerators were not tested.
The exact core package versions above are the compatibility target. Other versions may change serialization or loading behavior.
This is a post-training weight-only NF4 conversion, not QAT and not a fine-tuned model.
Pre-quantization can reduce repeated load-time conversion work, but it does not by itself establish a speed or memory advantage for your workload.
A bounded synthetic literal-transcription test failed as documented above; do not treat image-text transcription quality as validated by the technical load and generation checks.
The upstream model's capabilities, risks, context-length considerations, and usage guidance still apply. See the upstream model card.

License and attribution

This is a quantized derivative of Qwen/Qwen3.5-9B by the Qwen team. The upstream repository is licensed under Apache License 2.0, which is included as LICENSE. The conversion and release packaging were prepared by ai-babai.

Downloads last month: -

Safetensors

Model size

9B params

Tensor type

F16

Model tree for ai-babai/Qwen3.5-9B-NF4

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Quantized

(498)

this model