Instructions to use KartiOS/Karti-Small-VL-4B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KartiOS/Karti-Small-VL-4B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="KartiOS/Karti-Small-VL-4B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("KartiOS/Karti-Small-VL-4B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("KartiOS/Karti-Small-VL-4B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KartiOS/Karti-Small-VL-4B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KartiOS/Karti-Small-VL-4B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KartiOS/Karti-Small-VL-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/KartiOS/Karti-Small-VL-4B-NVFP4

SGLang

How to use KartiOS/Karti-Small-VL-4B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KartiOS/Karti-Small-VL-4B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KartiOS/Karti-Small-VL-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KartiOS/Karti-Small-VL-4B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KartiOS/Karti-Small-VL-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use KartiOS/Karti-Small-VL-4B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/KartiOS/Karti-Small-VL-4B-NVFP4
```

Karti-Small-VL-4B · v1 · NVFP4

2.5× the throughput, 40% of the size — and it can still see.

NVFP4 build of KartiOS/Karti-Small-VL-4B v1, a 4B vision + tool-calling model for local agents.


Parameters	4.66 B
Precision	NVFP4 (compressed-tensors) · 3.7 GB
Requires	Blackwell, sm_120+ — producing NVFP4 works anywhere, running it does not
Context	32,768
Reference build	BF16 — 9.3 GB

Speed — NVIDIA DGX Spark (GB10)

vLLM 0.27.1, 32k context, FP8 KV cache, temperature 0, single stream, best of 3.

tokens	BF16	NVFP4
128	21.0 tok/s	51.7 tok/s
512	21.1 tok/s	51.8 tok/s
on disk	9.3 GB	3.7 GB
serving footprint	~20 GB	~8 GB

It also degrades far less under contention than BF16, because it moves roughly a third of the bytes per token — on a box already serving other models, that matters more than the single-stream figure.

It can actually see

A community NVFP4 of this same base returned !!!!!!!! for every image — including a solid red square — while passing arithmetic, tool calls and 43 tok/s, at 668 downloads. It had quantized the vision tower: 25 ignored layers, none of them vision.

This build holds out 98 vision modules; all 297 vision tensors stay unquantized, and every release is gated on real images, not text canaries.

PASS  VISION solid red     'red'          PASS  arithmetic 19x23   437
PASS  VISION solid green   'green'        PASS  tool call parses   ✓

Quality

Quantization costs accuracy. Same 603 rows, same endpoint, same decode path.

	base	NVFP4	BF16
Invented-identifier rate	0.378	0.048	0.023
Panel reading	0.595	0.924	0.967

~8× fewer invented identifiers than the untrained base; about half the BF16 margin given back.

Use NVFP4 when throughput or footprint is the constraint. Use BF16 when you need the best accuracy on reading identifiers exactly.

Use

vllm serve KartiOS/Karti-Small-VL-4B-NVFP4 \
  --max-model-len 32768 --kv-cache-dtype fp8 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Recipe

Held out: lm_head · re:.*\.visual\..* (297 tensors) · re:^mtp\..* · re:.*linear_attn\.conv1d\.* — 147 entries, 248 modules quantized.

Calibration is frozen so version-over-version comparisons measure the training, not the quantizer: lmms-lab/flickr30k test, 512 samples, seq 2048, seed 115, multimodal — images through the model's own processor, never text-only. It overlaps none of this program's evaluation sets.

No MTP head in this build, so no built-in speculative decoding; the BF16 has it. Serve the weights directly, never as a LoRA adapter over the base.

From Lumbridge.

Downloads last month: -

Safetensors

Model size

5B params

Tensor type

BF16

Model tree for KartiOS/Karti-Small-VL-4B-NVFP4

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Finetuned

KartiOS/Karti-Small-VL-4B

Quantized

(1)

this model