Instructions to use Qwen/Qwen3.8-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3.8-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Qwen/Qwen3.8-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.8-27B")
model = AutoModelForMultimodalLM.from_pretrained("Qwen/Qwen3.8-27B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3.8-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3.8-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3.8-27B

SGLang

How to use Qwen/Qwen3.8-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3.8-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3.8-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3.8-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3.8-27B with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3.8-27B
```

All UD- GGUFs in this repo have wrong general.file_type metadata

#74

by Phenomellama - opened 15 days ago

Discussion

Phenomellama

15 days ago

•

edited 15 days ago

CLOSED - PUT INTO WRONG PLACE; MODS PLEASE DELETE.

Updating all of my 3.6 quants to 3.8 and found this while working them into Hermes and OpenCode; I know someone posted something about this earlier, but the issue is more extensive. This is a DeepSeek Flash draft of the issue:
If this has already been fixed, please delete the thread.

Every Unsloth Dynamic (UD-*) GGUF in this repo carries an incorrect
"general.file_type" value in its header. I verified this by reading the
GGUF kv section of each file directly (no download needed - the header
is in the first ~11 MB):

File > general.file_type > Displays as > Should be
UD-IQ2_M > 14 (Q4_K_S) > Q4_K_S > IQ2_M
UD-IQ2_XXS > 14 (Q4_K_S) > Q4_K_S > IQ2_XXS
UD-IQ3_XXS > 14 (Q4_K_S) > Q4_K_S > IQ3_XXS
UD-Q2_K_XL > 14 (Q4_K_S) > Q4_K_S > Q2_K
UD-Q3_K_XL > 14 (Q4_K_S) > Q4_K_S > Q3_K_S/M
UD-Q4_K_XL > 14 (Q4_K_S) > Q4_K_S > Q4_K_M
UD-Q5_K_XL > 14 (Q4_K_S) > Q4_K_S > Q5_K_S/M
UD-Q6_K_XL > 14 (Q4_K_S) > Q4_K_S > Q6_K
UD-Q8_K_XL > 15 (Q4_K_M) > Q4_K_M > Q8_0

Every non-UD file in the repo is tagged correctly (IQ4_NL=25, IQ4_XS=30,
Q4_K_S=14, Q6_K=18).

Evidence the file_type values are wrong, not the quants:

File sizes match the real quants: UD-IQ2_XXS is 8.4 GB (~2.4 bpw),
UD-IQ3_XXS is 11.1 GB. A real Q4_K_S of this model is ~15 GB. If the
files were actually Q4_K_S the repo would be ~50% larger.
Older Unsloth repos tag UD files correctly: Qwen3.6-35B-A3B
(UD-IQ4_NL_XL=25, UD-IQ1_M=31), Gemma-4-31B (UD-Q5_K_XL=17),
Granite-4.1-30B (UD-Q6_K_XL=18). So this looks like a regression in
the current UD converter build (this repo was regenerated today,
2026-08-14), not a deliberate scheme.
The UD-* files also have no "general.quantization" string kv, so
there is no other metadata consumers can fall back to.

Impact: any tool that derives the quantization label from
general.file_type misidentifies these files. Concretely, Ollama's
"ollama show" prints Q4_K_S for every UD quant in this repo. llama.cpp
runtime loading is unaffected (per-tensor types drive that), but every
metadata consumer, model UI, and quant-aware script is misled.

Request: could the UD converter write the real base quant into
general.file_type (per llama.cpp's llm_ftype enum, e.g. IQ2_XXS=20,
IQ2_M=23, IQ3_XXS=24, Q2_K=11, Q3_K_M=13, Q4_K_M=16, Q5_K_M=18,
Q6_K=19, Q8_0=8) and re-upload the UD files? Happy to provide the
per-file patch procedure if a re-upload isn't planned.

(For reference, this was previously touched in closed discussion #15
for UD-Q8_K_XL only - this thread covers all UD files.)

Phenomellama changed discussion status to closed 15 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment