Instructions to use Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Intel/Qwen3.8-Flash-Next-W4A16-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Intel/Qwen3.8-Flash-Next-W4A16-AutoRound")
model = AutoModelForMultimodalLM.from_pretrained("Intel/Qwen3.8-Flash-Next-W4A16-AutoRound", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound

SGLang

How to use Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/Qwen3.8-Flash-Next-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with Docker Model Runner:
```
docker model run hf.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound
```

Help running it on vllm

by g8i96 - opened 28 days ago

Discussion

g8i96

28 days ago

•

edited 28 days ago

Deploying this model using vllm:

vllm:
    image: vllm/vllm-openai:latest
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    restart: always
    privileged: true
    environment:
      - VLLM_PLE_CPU_OFFLOAD=1
    entrypoint: vllm serve
    command: >
      --model Intel/Qwen3.8-Flash-Next-W4A16-AutoRound
      --port 8000
      --tensor-parallel-size 4
      --enable-expert-parallel
      --language-model-only
      --gpu-memory-utilization 0.95
      --enable-prefix-caching
      --mamba-cache-mode align
      --enable-chunked-prefill
      --async-scheduling
      --enable-auto-tool-choice
      --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
      --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"presence_penalty":0,"repetition_penalty":1}'
      --kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":236223201280,"lazy_offload":false}}'
      --tool-call-parser qwen3_xml
      --reasoning-parser qwen3
      --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}'
      --max-num-seqs 32
      --enable-logging-iteration-details

Produces the error:

 NotImplementedError: Qwen4Exp PLE embedding does not support quantization config INCConfig

Does anyone have a reliable recipe that works well while offloading the PLE embedding?

vcerny

28 days ago

I have an open PR for that https://github.com/vllm-project/vllm/pull/58977 Give me some time to rebase and test.

vcerny

27 days ago

Hi, another PR solving this was merged before mine.. https://github.com/vllm-project/vllm/pull/59990

g8i96 changed discussion status to closed 26 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment