Instructions to use amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8")
model = AutoModelForMultimodalLM.from_pretrained("amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8

SGLang

How to use amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 with Docker Model Runner:
```
docker model run hf.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8
```

Model Overview

Model Architecture: Qwen4ExpForConditionalGeneration
- Input: Text, Image, Video
- Output: Text
Supported Hardware Microarchitecture: AMD MI350 / MI355
ROCm: 7.2.4
PyTorch: 2.11.0+rocm7.2
Transformers: 5.12.1
Operating System(s): Linux
Inference Engine: SGLang
Model Optimizer: AMD-Quark (v0.12 with Patch for PLE Quantization)
- Quantized layers: All MoE experts in the language model, including the shared expert. PLE embedding table is quantized to Per-Tensor FP8. Attention is kept in bf16.
- Weight quantization:
  - MOE OCP MXFP4 for MoE experts, Per-Tensor FP8 for PLE embedding table, Static
- Activation quantization:
  - MOE OCP MXFP4 for MoE experts, Dynamic

This checkpoint quantizes the routed and shared MoE experts to OCP MXFP4, PLE embedding table is quantized to Per-Tensor FP8, everything else is kept in bf16, including the full self-attention and the (Mamba-style) linear-attention projections, the MTP head, router gates, shared-expert gate, vision tower, hyper-connection mixers, and lm_head.

Model Quantization

The model was quantized from Qwen3.8-Flash-Next using AMD-Quark. MoE expert weights and activations are quantized to OCP MXFP4, PLE embedding table is quantized to Per-Tensor FP8.

Quantization script:

from quark.torch import LLMTemplate, ModelQuantizer
from quark.torch.quantization.config.type import ScaleType

SRC = "Qwen/Qwen3.8-Flash-Next"
OUTPUT = "amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8"
NGRAM = "*ple.ple_embedding.ngram_embedding*"
EXCLUDE = [
    "lm_head", "model.visual.*", "mtp.*",
    "*mlp.gate", "*shared_expert_gate*",
    "*.linear_attn.*", "*.self_attn.*", "*hyper_connection*",
    "*ple.conv1d", "*ple.key_proj", "*ple.value_proj",
    "*ple.norm_conv", "*ple.norm_key", "*ple.norm_query",
]

template = LLMTemplate.get("qwen4_exp")
quant_config = template.get_config(
    scheme="mxfp4",
    exclude_layers=EXCLUDE,
    layer_config={NGRAM: "fp8"},
)
quant_config.layer_quant_config[NGRAM].weight.scale_type = ScaleType.bfloat16

ModelQuantizer(quant_config).direct_quantize_checkpoint(
    pretrained_model_path=SRC,
    save_path=OUTPUT,
    weight_converters=template.f2f_weight_converters,
    device="cuda",
    weight_scale_groups=["*ple.ple_embedding.ngram_embedding.shard_*.weight"],
)

For further details or issues, please refer to the AMD-Quark documentation.

Deployment and Evaluation

Use with SGLang

This checkpoint is validated for native MXFP4 deployment on AMD Instinct MI350X and MI355X GPUs. The accuracy measurements in the below section were collected on MI355X with the following environemnt setup:

SGLang commit 0b4f96ff745b7b498e49e613dfeea046a60052fc, from SGLang PR #36601.
Container image rocm/sgl-dev:v0.5.19-rocm724-mi35x-20260916 with above SGLang commit installed from source.

Accuracy

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness.

Benchmark	Qwen/Qwen3.8-Flash-Next	amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 (this model)	Recovery
gsm8k (flexible-extract, 5-shot)	96.97	96.44	99.5%

Reproduction

The GSM8K results were obtained using the SGLang framework, based on the Docker image rocm/sgl-dev:v0.5.19-rocm724-mi35x-20260916, with SGLang source code from this PR installed from source inside the container, and lm-eval installed from source.

Serve the model:

sglang serve --model-path amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8 \
    --tp 4 \
    --attention-backend triton

Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy):

lm_eval --model local-chat-completions \
    --model_args "model=amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=32,timeout=1200,max_retries=3,tokenized_requests=False,max_gen_toks=2048" \
    --tasks gsm8k --num_fewshot 5 --apply_chat_template --gen_kwargs '{"chat_template_kwargs":{"enable_thinking":false}}'

License

Downloads last month: 683

Safetensors

Model size

119B params

Tensor type

BF16

F8_E4M3

I64

Model tree for amd/Qwen3.8-Flash-Next-Quark-MXFP4-PLEFP8

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(258)

this model