Instructions to use bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="bernhardbrieger/Qwen3.8-27B-GPTQ-Int4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bernhardbrieger/Qwen3.8-27B-GPTQ-Int4")
model = AutoModelForMultimodalLM.from_pretrained("bernhardbrieger/Qwen3.8-27B-GPTQ-Int4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bernhardbrieger/Qwen3.8-27B-GPTQ-Int4

SGLang

How to use bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/bernhardbrieger/Qwen3.8-27B-GPTQ-Int4
```

Qwen3.8-27B-GPTQ-Int4

Post-training GPTQ 4-bit quantization of Qwen/Qwen3.8-27B, produced on a RTX 5090 (32 GB) + Ryzen 9 9950X3D + 64 GB DDR5 system with GPTQModel 7.3.4. The checkpoint ships the full multimodal architecture (language + vision tower) plus the Multi-Token-Prediction (MTP) draft layer, and is verified serving 160k context with an FP8 KV cache on vLLM 0.27.1.

Weight footprint drops from 55.56 GB (bf16) to 19.54 GB (Int4) — a 62.23 % reduction — while 11 precision-critical module families are deliberately kept at full precision. Every number on this card is traceable to a file in this repository (config.json, quantize_config.json, quant_log.csv, model.safetensors.index.json) or to a command in the Reproducibility appendix.

A note on how to read this card. The base model publishes no perplexity or benchmark table, and its 55.6 GB bf16 weights cannot be co-resident with this checkpoint on a 32 GB GPU. A base-vs-quant delta is therefore not reported here; the quality section gives absolute numbers and the quantizer's own loss statistics instead. That limitation is stated rather than papered over.

1. Identity & scope

Property	Value
Base model	Qwen/Qwen3.8-27B (Apache-2.0)
Architecture	`Qwen3_5ForConditionalGeneration` (model_type `qwen3_5`), image-text-to-text
Parameters	27,781,427,952 (~27.78 B), as reported by the base model
Quantization	GPTQ, 4-bit, group size 32, symmetric, non-act-ordered
Layers	64 — 48 Gated-DeltaNet linear-attention + 16 full-attention
Native context	262,144 tokens (verified serving at 163,840)
MTP	1 draft layer, 15 tensors, kept bf16, merged into the main shards
License	Apache-2.0 (base and derived)
Quantized by	Bernhard Brieger
Quantization date	2026-08-21

Intended use. General instruction following, coding, tool/function calling, and long-context workloads (retrieval over documents >100k tokens) on hardware that fits ~20 GB of weights. The 4-bit weights use the standard GPTQ packing format, so any runtime that loads GPTQ checkpoints works.

Out of scope. This is a plain quantization of the base model — no fine-tuning, instruction distillation, or safety retraining was applied. It is not intended as a substitute for the base model where full precision is required, and the numbers below characterize this specific checkpoint, not the base model in general.

What is quantized vs. kept at bf16. 400 linear modules (all mlp projections and attention projections across the 64 layers) are quantized to 4-bit: that is 24,326,963,200 of the 27.78 B parameters (87.6 %). The remaining 3.45 B parameters (12.4 %) stay at bf16: the token-embedding and lm_head matrices (1.27 B each), the MTP draft layer (425 M), the vision tower (461 M), and the per-layer layernorms / GDN conv1d / in_proj_a / in_proj_b / output-gate parameters.

2. Quantization recipe

Self-contained recipe — every parameter below is stored in quantize_config.json in this repo.

Toolchain

Component	Version
GPTQModel	7.3.4
transformers	5.15.0
PyTorch	2.13.0+cu130
Python	3.12
GPU	1× NVIDIA RTX 5090, 32 GB
CPU	AMD Ryzen 9 9950X3D, 16C/32T
RAM	64 GB DDR5

GPTQ configuration

Parameter	Value	Note
`bits`	4
`group_size`	32	uniform — full vLLM/SGLang compatibility
`sym`	`true`	symmetric quantization
`desc_act`	`false`	non-act-ordered (deterministic group layout)
`lm_head`	`false`	`lm_head` not quantized
`act_group_aware`	`true`	GPTQModel default, pinned in metadata
`damp_percent`	0.05	auto-increment 0.01 on low damping
`true_sequential`	`true`
`pack_dtype`	int32
Hessian staging dtype	float32
Fallback	`rtn` @ 0.5 % threshold	never triggered (0 fallbacks, §3)

Modules kept at bf16 (the 11 `dynamic` exclusion patterns)

GPTQ exclusions are negative patterns: a matched module is skipped and left at bf16. The recipe below is what is stored in quantize_config.json.dynamic. Each pattern is namespace-agnostic — it matches both the HuggingFace full module path and vLLM's layer prefix, because both consumers anchor the same regular expression at position 0.

Pattern	What it protects	Why
`*.input_layernorm`	input layernorms	precision-critical; not in GPTQModel's supported-module set
`*.post_attention_layernorm`	post-attention layernorms	same
`*.self_attn.q_norm`	query normalization	same
`*.self_attn.k_norm`	key normalization	same
`*.norm`	all other norms (GDN `linear_attn.norm`, final norm, vision merger)	same; catches the rest
`*.linear_attn.conv1d`	GDN causal convolution	`nn.Conv1d` is a quantization candidate — load-bearing
`*.linear_attn.in_proj_a`	GDN `in_proj_a`	`nn.Linear` candidate — load-bearing
`*.linear_attn.in_proj_b`	GDN `in_proj_b`	`nn.Linear` candidate — load-bearing
`*.embed_tokens`	token embeddings	`nn.Embedding` candidate — load-bearing
`visual.*`	entire vision tower	vLLM quantizes its linears with the same config
`mtp.*`	MTP draft layer	keeps speculative-decoding draft weights exact

Why the MTP pattern is written the way it is. vLLM gates MTP on any dynamic key whose text contains mtp (qwen3_5_mtp.py). The form (?:.*\.)?mtp\..* satisfies that gate and matches a root mtp.* without false-positiving on e.g. model.language_model.embed_tokens (no dot before its "mtp"), unlike a naive -:.*mtp.*.

Calibration corpus

Metric	Value
Sequences	2,026
Total tokens	1,802,645
Longest sequence	132,968 tokens
Random seed	42
Ordering	`desc` (longest first)
Concat block size	40,960 (44 full blocks + 1 final; 0 % real-token loss)
Calibration device	CPU (per-layer block to GPU)

Built from a curated pool of public, non-gated sources (see License & provenance): 11 programming languages, an agent/tool-calling pool, general web/wiki prose, a multilingual (DE/FR/IT) pool, and a math pool. The corpus mirrors the deployment profile — code-heavy with a 2×128k long-document tail — so the deepest context regimes (128k/160k) are actually calibrated, not just short sequences.

Pool	Budget (tok)	Used (tok)	Seqs	Length range
code_typescript	185,000	185,815	519	128–4,096
code_python	150,000	151,189	125	256–4,096
code_javascript	115,000	116,018	160	128–4,096
code_csharp	115,000	118,143	110	128–4,096
code_java	75,000	75,152	130	128–4,096
code_cpp	65,000	65,347	66	128–4,096
code_rust	50,000	50,031	97	128–4,096
code_kotlin	40,000	40,788	86	128–4,096
code_shell	45,000	45,404	55	128–4,096
code_swift	15,000	15,500	29	128–4,096
code_sql	15,000	17,268	6	128–4,096
agent	100,000	100,090	182	256–2,048
general (web/wiki/config)	125,000	125,756	131	128–2,048
multilingual (wiki DE/FR/IT)	65,000	65,152	135	128–1,024
math	35,000	35,045	183	64–1,500
Long tiers	12×	12 tiers	754 docs	16k–132,968

The 12 long tiers (four at 32k, four at 16k, two at 64k, two at 128k) add 754 documents reaching up to 132,968 tokens. During quantization GPTQModel repacks the corpus into 40,960-token blocks, so no forward pass ever exceeds 40,960 tokens — this bounds the GDN fp32 value-stream tensor and removes the OOM on the longest sequences structurally.

Run statistics

Stage	Wall time
Model load (bf16, meta-init → GPU per layer)	2 s
Quantize	90.2 min
Save (6 shards)	15 s

Total wall time ~90 min on a single RTX 5090.

3. Quality verification

Three independent checks: the quantizer's own per-module loss, an absolute perplexity, and a functional smoke suite run against the live server.

(a) Quantization loss (from `quant_log.csv`, 400 modules)

Metric	Value
Modules quantized	400 (48 GDN × 6 + 16 full-attn × 7)
Mean per-module loss	1.108e-05
Max per-module loss	6.713e-05
Worst module	`L61 mlp.gate_proj`
Worst cluster	`mlp.gate_proj` at layers 58–61 (~6.3–6.7e-05)
RTN fallbacks	0 (fallback configured but never triggered)

The per-module loss is the squared quantization error GPTQ minimizes per linear layer. A max of 6.7e-05 across all 400 modules, with zero modules ever falling back to the RTN (round-to-nearest) strategy, indicates a clean, uniform fit — no layer required a degraded fallback to converge.

(b) Perplexity (absolute)

Metric	Value
Perplexity	2.729
Tokens scored	n = 260
Method	`max_tokens=1` + `prompt_logprobs=1` over a fixed, neutral English prose sample; PPL = exp(−mean logprob) over the input tokens
Runtime	live vLLM (this checkpoint)

This is an absolute perplexity on a single 260-token sample — a sanity check that the dequantized model assigns sensible probabilities to plain prose, not a quality benchmark. The base model publishes no perplexity, so there is nothing to diff against; the number is reported as-is.

(c) Functional smoke (3/3 pass, live server)

Probe	Result	Detail
Coding generation	PASS	2,273 chars in 11 s (`enable_thinking: false`)
Long-context needle	PASS	target located at 46,037-token context, returned `ZEBRAX42` (2 s)
Tool / function calling	PASS	exact JSON: `{"name":"get_weather","arguments":{"city":"Graz","unit":"celsius"}}` (1 s)

The needle probe exercises the long-context regime that the 2×128k calibration tail was built to protect. The tool-call probe verifies the qwen3_coder parser path end-to-end.

What is not verified here

No base-vs-quant benchmark table. The base card publishes no benchmarks or perplexity, and the 55.6 GB base cannot be co-resident with this checkpoint on the 32 GB test GPU. If a head-to-head is needed, it requires ≥80 GB of VRAM.
MTP speculative decoding is preserved (15/15 tensors, bf16) but was served without the speculative flag in the runs above; see §4.

4. Deployment

Verified serving configuration on vLLM 0.27.1 (Marlin GPTQ kernels, FP8 KV cache, FlashInfer attention), 163,840-token context on an RTX 5090 32 GB:

vllm serve bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 \
  --tensor-parallel-size 1 \
  --max-model-len 163840 \
  --gpu-memory-utilization 0.90 \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --attention-backend FLASHINFER

Weights: 19.54 GB resident. At 160k context with an FP8 KV cache the server holds ~31.4 GB of the 32 GB GPU — it fits, with max_num_seqs 2.
Reasoning / tool parsing: --reasoning-parser qwen3 + --tool-call-parser qwen3_coder are what make structured tool-call output work (see the smoke probe in §3c).

Optional: MTP speculative decoding

The MTP draft layer is present (15 tensors, bf16). To use it:

  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

num_speculative_tokens: 2 is the Qwen-recommended value for a single MTP layer. The MTP weights remain fully usable without the flag (they are simply unused), so the checkpoint runs as a normal model on any setup — speculative decoding is opt-in.

Loading with GPTQModel / transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "bernhardbrieger/Qwen3.8-27B-GPTQ-Int4",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
tok = AutoTokenizer.from_pretrained("bernhardbrieger/Qwen3.8-27B-GPTQ-Int4")

Performance (measured)

One-shot measurement, vLLM 0.27.1, 2026-08-22, batch 1, on the exact §4 flags above (RTX 5090 32 GB). Per context length: 1 warmup request + 3 timed runs, median reported, max_tokens=256, temperature 0, thinking mode off. Prompts were drawn from the §2 calibration corpus such that no prompt is a prefix of another — with --enable-prefix-caching active the server-side hit rate over the whole run was 0.0 %, i.e. every number below is full prefill. Prefill throughput = prompt tokens / TTFT; decode throughput = completion tokens / (total time − TTFT). The KV-cache column is derived from the model config, not measured: FP8 KV holds 32 KiB per token (16 full-attention layers × K+V × 4 KV heads × 256 head-dim × 1 byte), so each entry equals its context length × 32 KiB.

Context	TTFT (s)	Prefill (tok/s)	Decode (tok/s)	KV cache (GiB)
8,192	2.118	3,873	80.2	0.25
16,384	4.418	3,711	79.8	0.50
32,768	9.434	3,475	77.8	1.00
65,536	21.426	3,057	74.4	2.00
131,072	53.476	2,451	68.2	4.00
163,457	72.987	2,240	65.3	4.99

Single-shot snapshot: absolute numbers will differ on other vLLM releases (kernel and scheduling changes). Reproduce with scripts/bench_serve.py against a server launched with the §4 flags.

5. Hardware requirements

Deployment	Minimum VRAM
Weights only (short context)	~20 GB
160k context, FP8 KV, 2 seqs (verified)	32 GB (RTX 5090)

The verified configuration uses the full 32 GB. The 19.54 GB weight footprint is a fixed cost: a GPU below ~20 GB cannot load this checkpoint at any context length. The remaining budget goes to the KV cache, which scales with context length and concurrency — it determines how much context a given card can serve.

6. Reproducibility appendix

Every number in §1–§3 is produced by one of the following. Paths are relative to this repository.

Exact re-quantization needs the calibration corpus (the file referenced in the code below), which is documented in provenance.json but not redistributed with this checkpoint.

6.1 Quantization

The checkpoint is produced by GPTQModel over a 2,026-sequence corpus:

import json
from gptqmodel import GPTQConfig, GPTQModel

DYNAMIC = {
    "-:.*\.input_layernorm$": {},
    "-:.*\.post_attention_layernorm$": {},
    "-:.*\.self_attn\.q_norm$": {},
    "-:.*\.self_attn\.k_norm$": {},
    "-:.*\.norm$": {},
    "-:.*\.linear_attn\.conv1d$": {},
    "-:.*\.linear_attn\.in_proj_a$": {},
    "-:.*\.linear_attn\.in_proj_b$": {},
    "-:.*\.embed_tokens$": {},
    "-:(?:.*\.)?visual\..*": {},
    "-:(?:.*\.)?mtp\..*": {},
}

model = GPTQModel.load(
    "Qwen/Qwen3.8-27B",
    GPTQConfig(bits=4, group_size=32, desc_act=False, sym=True,
               lm_head=False, dynamic=DYNAMIC,
               offload_to_disk=True, offload_to_disk_path="output/offload",
               calibration_data_device="cpu"),
)
seqs = json.load(open("calibration_data.json"))["sequences"]  # see provenance.json
model.quantize(seqs, batch_size=1,
               calibration_sort="desc",
               calibration_concat_size=40960)
model.save("output/Qwen3.8-27B-GPTQ-Int4")

offload_to_disk streams finished layers to NVMe during the layer loop (the default would otherwise stage them on the GPU). Per-module losses land in quant_log.csv; the resolved config is written to quantize_config.json.

6.2 Perplexity

Against a running server on :8000, scoring the fixed 260-token sample with input-token logprobs:

python scripts/measure_ppl_quant.py
# -> quant_ppl  n=260  perplexity=2.729

The script POSTs the fixed sample with max_tokens=1, prompt_logprobs=1 and computes exp(-mean logprob) over the prompt positions.

6.3 Functional smoke

python scripts/smoke_test_api.py
# -> 3 probes: coding / needle_40k / tool_call  (results in smoke_test_api.jsonl)

7. License & provenance

Model weights. Apache-2.0. The base Qwen/Qwen3.8-27B is Apache-2.0 and this derived checkpoint is released under the same license.

Calibration data. The corpus is drawn entirely from public, non-gated datasets. Every source, its license, and (for code) its pinned commit are listed below; the full manifest is provenance.json in this repository. The raw calibration sequences are not redistributed with this checkpoint.

Source	License	Used in
28 code repositories	Apache-2.0 (13) · MIT (13) · BSD-2-Clause (1) · BSD-3-Clause (1)	code pools (11 languages)
codeparrot/codeparrot-clean	per-file upstream license (no dataset-level license declared)	Python pool
wikimedia/wikipedia (EN/DE/FR/IT)	CC BY-SA 3.0 + GFDL	wiki prose, long-document tail
openai/gsm8k	MIT	math pool
allenai/tulu-3-sft-mixture	ODC-BY	agent / tool-calling pool
HuggingFaceTB/SmolLM-Corpus (FineWeb-Edu dedup)	ODC-BY	web prose, long-document tail

Each code repository is pinned to a specific commit in provenance.json (repo + commit SHA + license), so the code portion of the corpus is reconstructable byte-for-byte.

Data attribution. Wikipedia text © Wikipedia contributors, released under CC BY-SA 3.0 and the GFDL. FineWeb-Edu and the Tülu-3 SFT mixture are used under ODC-BY. Attribution is provided here in lieu of per-file notices.

Base model references.

Base model card: https://huggingface.co/Qwen/Qwen3.8-27B
Qwen3.5 / Qwen3.8 technical report: see the links on the base model card.

Downloads last month: 15

Safetensors

Model size

28B params

Tensor type

BF16

I32

Model tree for bernhardbrieger/Qwen3.8-27B-GPTQ-Int4

Base model

Qwen/Qwen3.8-27B

Quantized

(769)

this model