Instructions to use ajh-code/Mage-VL-FP8-W8A8-W8A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ajh-code/Mage-VL-FP8-W8A8-W8A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ajh-code/Mage-VL-FP8-W8A8-W8A16", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("ajh-code/Mage-VL-FP8-W8A8-W8A16", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ajh-code/Mage-VL-FP8-W8A8-W8A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ajh-code/Mage-VL-FP8-W8A8-W8A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajh-code/Mage-VL-FP8-W8A8-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ajh-code/Mage-VL-FP8-W8A8-W8A16

SGLang

How to use ajh-code/Mage-VL-FP8-W8A8-W8A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ajh-code/Mage-VL-FP8-W8A8-W8A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajh-code/Mage-VL-FP8-W8A8-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ajh-code/Mage-VL-FP8-W8A8-W8A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ajh-code/Mage-VL-FP8-W8A8-W8A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ajh-code/Mage-VL-FP8-W8A8-W8A16 with Docker Model Runner:
```
docker model run hf.co/ajh-code/Mage-VL-FP8-W8A8-W8A16
```

Mage-VL · FP8 W8A8/W8A16

High-fidelity W8A8 prefill with fused W8A16 decode.

Our Quantization series combines the best of Speed, Quality, and Size.

Original Mage-VL · XPO3 NVFP4 / lowest VRAM and fastest prefill · Arands.com · updates

Download

Component	Purpose	Size
`model-*.safetensors`	Complete Mage-VL checkpoint	5.85 GB
`streammind_gate.safetensors`	Upstream proactive streaming gate	1.07 GB
Complete repository	Runtime, processor, codec support, and weights	7.21 GB

The root config.json is a recognized Hugging Face query file, so downloads are tracked and this repository appears as a quantization of microsoft/Mage-VL.

Quick start

Tested on Linux x86-64, Python 3.11, CUDA 13.1, PyTorch 2.13.0+cu130, comfy-kitchen==0.2.22, and NVIDIA Blackwell SM120.

hf download ajh-code/Mage-VL-FP8-W8A8-W8A16 --local-dir mage-vl-quant
cd mage-vl-quant
python3.11 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

CUDA_HOME=/usr/local/cuda-13.1 CUDA_VISIBLE_DEVICES=0 \
  python inference.py --mode offline --model . --image image.jpg \
  --question "Describe this image in detail."

All three decode accelerations are enabled by default and independently configurable. Restore exact ordinary W8A8 execution with:

MAGE_VL_SMALLM_BACKEND=off \
MAGE_VL_FP8_FUSED_GATE_UP=0 \
MAGE_VL_FP8_FUSED_QKV=0 \
CUDA_VISIBLE_DEVICES=0 \
  python inference.py --mode offline --model . --image image.jpg \
  --question "Describe this image."

The first accelerated call JIT-compiles three small CUDA extensions. It took about two minutes in the clean-cache package gate; subsequent processes reuse the cached build. This one-time compilation is excluded from the measurements below.

Measured performance

RTX 5060 Ti, 1,657-token multimodal prompt, greedy decode, exactly 32 new tokens. The accelerated FP8 result uses three warmups and 15 measurements; the BF16, ordinary W8A8, and XPO3 controls use the same local protocol family:

Runtime	Resident allocation	Prefill	Generate 32	Derived decode
BF16	9,044 MiB	0.6011 s	1.4277 s	36.69 tok/s
Ordinary FP8 W8A8	5,579 MiB	0.4541 s	2.0543 s	19.14 tok/s
FP8 W8A8/W8A16 fused	5,579 MiB	0.4535 s	1.0177 s	53.21 tok/s
XPO3 NVFP4 W4A4/W4A16	4,063 MiB	0.4233 s	1.0120 s	51.14 tok/s

These are matched local image-path measurements, not universal end-to-end claims. Accelerated FP8 is 1.40x faster than BF16 and 2.02x faster than ordinary W8A8 for fixed 32-token generation while using 38.3% less resident allocation than BF16. XPO3 remains narrowly fastest overall at 1.41x BF16 and uses 55.1% less resident memory; accelerated FP8 has the highest derived decode rate in this local comparison.

Quantization policy

All 252 Qwen3 language projections use one resident E4M3 checkpoint. Large-M multimodal prefill dynamically quantizes activations and runs W8A8. At M=1, the runtime reads those same weights through W8A16 kernels, then combines QKV dispatch and gate/up/SiLU work to reduce launch overhead. No second weight copy is stored.

The accelerated stack reproduced all 10/10 accepted generated sequences exactly; minimum reference-logit cosine was 0.9999816. The underlying FP8 checkpoint retained 10/10 BF16 first-token decisions with mean final-logit cosine 0.998388.

Validated scope

Gate	Result
Quantized language projections	252 / 252
Image suite	Ten deterministic description, OCR, count, spatial, and detail cases
Packaging	Complete sharded Safetensors checkpoint with root `config.json`
Decode fusion restoration	Exact feature-off and `M>1` fallback; stored weights unchanged
Unchanged BF16 components	Mage-ViT, embeddings, LM head, norms, RoPE, StreamMind
Current hardware target	NVIDIA Blackwell SM120

Image understanding is the validated release path. The upstream video, codec, and StreamMind files are retained for completeness, but their quantized end-to-end paths have not yet received the same release gate.

Validate the download

python validate_release.py

MANIFEST.json records the size and SHA-256 of every distributed file except itself. Hashing the model and StreamMind checkpoints takes a little while.

License and attribution

Mage-VL and this derivative package are released under Apache-2.0. The model architecture, processor, codec utilities, and original BF16 weights derive from microsoft/Mage-VL. The quantized runtime modifications are identified in the bundled source and THIRD_PARTY_NOTICES.md.

Downloads last month: -

Safetensors

Model size

5B params

Tensor type

BF16

F8_E4M3

Model tree for ajh-code/Mage-VL-FP8-W8A8-W8A16

Base model

microsoft/Mage-VL

Quantized

(8)

this model