Instructions to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ")
model = AutoModelForMultimodalLM.from_pretrained("ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ

SGLang

How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ with Docker Model Runner:
```
docker model run hf.co/ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ
```

Maybe add vLLM FP8 KVCache calibration scales?

by ibaldonl - opened 12 days ago

Discussion

ibaldonl

12 days ago

As the title says, this seems simple to add and should be beneficial for VRAM constrained setups.

anm2211

IST Austria Distributed Algorithms and Systems Lab org 11 days ago

Thanks for the suggestion! In the meantime, could you try vLLM’s existing FP8 KV-cache quantization support and let us know if you observe any major degradation compared with the default KV cache?

We’re currently receiving a large number of requests for additional models and variants, so we unfortunately can’t say when we’ll be able to look into adding calibrated FP8 KV-cache scales for this checkpoint. Any results you can share would be very helpful.

ELVISIO

11 days ago

@anm2211

May I ask whether you plan to release a 2-bit model with a vLLM implementation in the future? I've found ways to scale to NVFP4 KV cache, and I'd like to try fitting it onto a 12GB GPU with partly offload, KV cache calibration and reduced batch tokens.

ibaldonl

11 days ago

I can't do evaluations since the GPU is shared and I have only about 2hs of maintenance window per day unfortunately :-(

anm2211

IST Austria Distributed Algorithms and Systems Lab org 10 days ago

@ELVISIO
We actually tried 2-bit quantization but the results were below our quality standards so we decided not to release it. If you're interested in using llama.cpp, we've released smaller GGUF checkpoints that can fit on a 12 GB GPU. We definitely recommend giving those a try.

@ibaldonl
I ran some tests and didn't observe any noticeable performance degradation, so I can recommend using the FP8 KV cache. If you run into any issues with FP8 KV cache quantization, please let us know here and we'll be happy to take a look.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment