Instructions to use JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm")
model = AutoModelForMultimodalLM.from_pretrained("JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm

SGLang

How to use JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm with Docker Model Runner:
```
docker model run hf.co/JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm
```

Qwen3.6-27B-PrismaQuant-5.2bit-vllm

A 5.2 bits-per-parameter mixed-precision quantization of Qwen/Qwen3.6-27B, produced with PrismaQuant. Packaged in the compressed-tensors format for direct serving with vLLM. ~22 GB on disk (5 safetensors shards), including the multi-token prediction (MTP) module and the vision tower.

This model is experimental only and nothing is guarantee except that the time might be wasted even to download.
** If there are issues, tell me. If it is good, tell someone else **

Highlights

Benchmarks. On a deterministic, prefix-cache-OFF tool-evaluation harness it match or exceed the rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm baseline while being smaller:

Model Tool-eval (3 metrics) Size

This model (PrismaQuant 5.2bpp) 94 / 66.7 / 92.3 22 GB

PrismaAURA 5.5bit (baseline) 93 / 66.7 / 80.8 23 GB

Same harness for both, run solo with prefix caching disabled for determinism. The largest gap is on the third metric (92.3 vs 80.8).

Model	Tool-eval (3 metrics)	Size
This model (PrismaQuant 5.2bpp)	94 / 66.7 / 92.3	22 GB
PrismaAURA 5.5bit (baseline)	93 / 66.7 / 80.8	23 GB

Method

PrismaQuant, 5.2 bpp target, mixed-precision:

Bit placement by Fisher-diagonal sensitivity allocation — more bits are spent on the parameters the loss is most sensitive to.
Per-candidate selection by measured KL against a validated surrogate (the allocator chooses the quantization that minimizes measured output- distribution divergence, rather than relying on a proxy heuristic).
Provenance: stock PrismaQuant allocator, run under our production render-stream cost mode.

The result is a heterogeneous mix of element types across tensors:

NVFP4 (W4A4) for the bulk of the linear weights,
FP8 for sensitivity-flagged tensors,
BF16 for the most sensitive / small tensors kept at full precision.

config.json carries the full quantization_config (quant_method: compressed-tensors, format: mixed-precision); the exact per-tensor scheme is described by mixed_native_manifest.json.

Serving (vLLM)

The model loads via vLLMs native compressed-tensors path:

vllm serve JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm

For fast GDN / linear-attention (gated-delta) serving, install the optional kernel:

pip install causal_conv1d

(Without it, vLLM falls back to a slower path for the linear-attention layers; correctness is unaffected.)

model-0000{1..5}-of-00005.safetensors + model.safetensors.index.json
config.json (with quantization_config), generation_config.json
mixed_native_manifest.json (per-tensor precision map)
MTP module weights (mtp.*) and the Qwen3.6 vision tower (model.visual.*)
tokenizer + chat template

License

Inherited from the base model Qwen/Qwen3.6-27B. Refer to the base model card for the governing license terms.

Provenance (added 2026-07-04 — recorded after the fact, verified from the build environment)

Calibration dataset: diverse-v1.jsonl (4.6 MB, sha256[:16] cc76f4a13c413398) — mixed prose/code/math corpus built by tools/build_diverse_calibration.py. Exact bytes archived in the PrismaQuant repo (calibration-datasets/).
Known caveat (discovered 2026-07-03, applies retroactively): the Fisher probe consumed only 32x1024 tokens (~3%) of that corpus (the "calibration keyhole"). The result stands on its benchmarks; the caveat matters if you try to attribute the win to the calibration mix.
Pipeline: PrismaQuant (Fisher-diagonal allocation, production-render-stream cost, validated-surrogate measured-KL selection, exportable_to_vllm gate); TARGET_BITS selected agentically (5.2 beat the 4.9 KL-kneedle on the agentic harness: 94 vs 89).
Benchmark context: deterministic harness (temp 0, prefix caching OFF), June 2026 stack. Scores from other stacks/harnesses are not comparable; see the repo maintainer for the full serve flags.

Downloads last month: 196

Safetensors

Model size

19B params

Tensor type

F32

BF16

F8_E4M3

Model tree for JasonW2025/Qwen3.6-27B-PrismaQuant-5.2bit-vllm

Base model

Qwen/Qwen3.6-27B

Quantized

(605)

this model

JasonW2025
/

Qwen3.6-27B-PrismaQuant-5.2bit-vllm