Instructions to use Qwen/Qwen3.8-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B

SGLang

How to use Qwen/Qwen3.8-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B
```

we quantized Qwen3.8-27B and compared it with community GGUFs on 4x RTX 5090!

#65

by d4nnyy - opened 1 day ago

Discussion

d4nnyy

1 day ago

qwen put out the 3.8 27b today, so we quantized it from the original weights: 16 files with per-tensor overrides and an imatrix calibrated on our public corpora, from Q8_0 at 28.9 GB down to IQ1_M at 8.5 GB.

after quantizing we measured every qwen3.8 gguf we found on hf in one scenario. published numbers from different repos do not compare, everyone runs their own corpus on their own gpu. so we ran 20 community files from unsloth, lmstudio-community and ggml-org through the same harness as our 16, all on one machine.

harness:

4x RTX 5090
eval_neutral held-out at ctx 4096, 87 chunks
reference is our own BF16 conversion of the original weights, we save its logits once (88 GB) and score every quant against the same file

between 12 and 21 GB our curve is lower than anyone else's. the widest gap is at 13.8 GB, where unsloth's Q3_K_M and our AD-IQ3_S happen to be the same size and ours drifts 33% less, 0.0325 vs 0.0484 mean KLD

but there are a few points where community quants are better. unsloth's UD 2-bit files edge ours below 11 GB, and lmstudio's Q6_K at 22.4 GB beats our nearest 23.1 GB file. past 25 GB it stops mattering whose file you grab, Q6 and Q8 from any publisher agree within noise

below 10 GB every quant of this model degrades fast, our IQ1_M sits at 76.3% top-1. it exists, we would not run it

based on our quantization the best quant for Qwen3.8 on 16 GB hardware is our AD-IQ3_S (13.8 GB) with 92.4% top-1

collection on HF with the imatrix, the per-tensor layout and everything else https://huggingface.co/collections/AtomicChat/qwen-38-27b-6a7c86fe00e317b78f572767

our local ai open source app https://atomic.chat (i'm founder). so feel free to ask any questions and share your feedback!

pymo

1 day ago

How about Q4_k version of yours and others?

TizzyT566

1 day ago

Is there a table? The image is too cluttered

BrokeAmerican

about 22 hours ago

Great table. I guess one has to run q8_x_l. Anything less won't impress people on Instagram. Life is all about impressing others on social media. But you will need 48gb of vram to do so! 👹

apollo-mg

about 19 hours ago

Really appreciate how much of this you published — the corpus, the held-out text, the
contamination scan, the crosscheck, and competitors measured on your own harness rather
than quoted. That's a higher bar than most releases clear, and it's what made the
following possible to check from outside.

Two notes on the MTP head, both from reading the tensor tables only (HTTP range reads of
the GGUF headers — 16 files, 192 MiB transferred, no weights downloaded, so anyone can
reproduce this in a couple of minutes).

1. The q5_k pin didn't land on all 16 files.

The card says the head "is pinned to q5_k in every file here." Eight files match that.
Eight don't:

file	`blk.64` attn_q / ffn_down / eh_proj
`AD-IQ3_S`	IQ4_XS / IQ3_S / Q8_0
`AD-IQ4_XS-IQ3_S`	IQ4_XS / IQ4_XS / Q8_0
`AD-IQ4_XS`	IQ4_XS / Q4_K / Q8_0
`AD-Q4_K`	Q4_K / Q4_K / Q8_0
`AD-Q5_K-Q4_K`	Q4_K / Q5_K / Q8_0
`AD-Q5_K`	Q5_K / Q5_K / Q8_0
`AD-Q6_K-Q5_K`	Q5_K / Q6_K / Q8_0
`AD-Q6_K`	Q6_K / Q6_K / Q8_0

The split is not arbitrary — 15 of 16 files partition exactly on whether the head's natural
type would have hit tensor_requires_imatrix(). Every tier where llama.cpp would have
aborted got the pin; every tier where it wouldn't kept the default. So the pin is doing
what it was written to do, it just isn't reaching the tiers that don't abort.

The one worth a rebuild is AD-IQ3_S — the file the card recommends for 16 GB cards.
Its head FFN is IQ3_S, assigned with no importance data, which is the situation the pin
exists to avoid. IQ3_S and IQ4_XS aren't in the requires-imatrix set, so nothing
refused; it just went through blind.

Worth saying the direction isn't uniformly bad: in all eight of those files eh_proj is
Q8_0 rather than Q5_K, which is better than the pin on the one tensor unique to the
MTP path.

2. Q8_0 is the exception to the pattern.

Its head is Q5_K throughout — byte-identical footprint (278.54 MiB) to AD-IQ1_M's.
Nothing would have aborted at that tier, so the pin cost precision rather than saving the
build. Net effect is that the 28.9 GB file ships a weaker drafter than AD-Q6_K below it,
which is probably backwards for who runs it.

One broader thing your write-up implies but doesn't state. Since blk.64 never
executes in a normal forward pass — the reason the imatrix has nothing for it — the KLD
and top-1 columns are computed with the head not participating. Every number in the table
would be identical if blk.64 were F16 or IQ1_S in every file. That's correct for what
the metric measures, but it does mean the ladder ships a component in every file,
recommends running it with --spec-type draft-mtp, and the quality table beside it is
blind to that component by construction. Draft acceptance rate would be the thing that
sees it, and nobody publishes it.

(Assisted by Claude Opus 5.)

Avifenesh

about 16 hours ago

one card, not 4. nvfp4+q5_k on a pro 6000: 140 tok/s spec, 0.156s ttft, 262k.
https://github.com/avifenesh/memra
https://inference.tiyuvta.ai/app

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment