Instructions to use LostGentoo/Qwen3.5-4B-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LostGentoo/Qwen3.5-4B-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="LostGentoo/Qwen3.5-4B-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("LostGentoo/Qwen3.5-4B-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("LostGentoo/Qwen3.5-4B-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LostGentoo/Qwen3.5-4B-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LostGentoo/Qwen3.5-4B-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/LostGentoo/Qwen3.5-4B-AWQ

SGLang

How to use LostGentoo/Qwen3.5-4B-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LostGentoo/Qwen3.5-4B-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LostGentoo/Qwen3.5-4B-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LostGentoo/Qwen3.5-4B-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use LostGentoo/Qwen3.5-4B-AWQ with Docker Model Runner:
```
docker model run hf.co/LostGentoo/Qwen3.5-4B-AWQ
```

Qwen3.5-4B-AWQ (W4A16)

AWQ 4-bit (W4A16) quantization of Qwen/Qwen3.5-4B, built with llm-compressor in compressed-tensors format for native vLLM loading (Marlin kernels).

Qwen3.5-4B is a hybrid-attention VLM: interleaved full self-attention + Gated-DeltaNet linear attention, a vision tower, and an MTP head for speculative decoding.

What is quantized

Component	Precision
LM Linear layers (MLP, full-attn `q/k/v/o_proj`, linear-attn projections)	INT4 W4A16, g128, symmetric
Vision tower + merger/projector	BF16
Embeddings, tied `lm_head`, RMSNorms	BF16
MTP head (`mtp.*`)	BF16 (post-quant splice via `save_mtp_tensors_to_checkpoint`)

248 Linear layers quantized; on-disk ~5.0 GB (vs ~8 GB BF16).

Quality (AWQ vs BF16)

OpenLLM-lite, n=200 per task, greedy logprob MC scoring on RTX 5060 Ti (sm_120):

Task	BF16	AWQ	Recovery
MMLU	0.710	0.700	98.6%
ARC-Challenge	0.545	0.555	101.8%
HellaSwag	0.705	0.670	95.0%
Winogrande	0.685	0.675	98.5%
TruthfulQA MC1	0.295	0.285	96.6%
WikiText-2 PPL	16.77	18.54	+10.6%

Mean MC recovery: 98.1%.

Evals

Full AWQ vs BF16 dump: LostGentoo/awq-quant-evals (qwen35_4b_awq_vs_bf16.json, plus combined quant_quality_evals.json).

Optimal vLLM serve

compressed-tensors is auto-detected - do not pass --quantization awq.

Recommended (text + vision, MTP on)

vllm serve LostGentoo/Qwen3.5-4B-AWQ \
  --trust-remote-code \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --limit-mm-per-prompt '{"image":1}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --generation-config vllm \
  --mamba-cache-mode align \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Notes:

MTP: requires model_mtp.safetensors (included). Older docs may say qwen3_5_mtp; current vLLM remaps that to mtp.
--mamba-cache-mode align: needed for hybrid Gated-DeltaNet + MTP / prefix-cache paths (all is unsupported for Qwen3.5 MTP).
Thinking: default Qwen3.5 thinking can burn tokens; keep it off unless you want CoT. For hard math/coding, set "enable_thinking": true and raise max_tokens.
Vision: keep --limit-mm-per-prompt '{"image":1}' so the VLM path is enabled; omit only for text-only deployments.
Blackwell (sm_120): Marlin W4A16 works; add --enforce-eager only if first bring-up hits compile issues.

Python

from vllm import LLM, SamplingParams

llm = LLM(
    model="LostGentoo/Qwen3.5-4B-AWQ",
    trust_remote_code=True,
    max_model_len=8192,
    limit_mm_per_prompt={"image": 1},
    speculative_config={"method": "mtp", "num_speculative_tokens": 3},
)
sp = SamplingParams(temperature=0.7, top_p=0.8, top_k=20, max_tokens=256)
print(llm.generate(["Explain entropy in one sentence."], sp)[0].outputs[0].text)

OpenAI-compatible client

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "LostGentoo/Qwen3.5-4B-AWQ",
    "messages": [{"role":"user","content":"In one sentence, what is entropy?"}],
    "max_tokens": 128,
    "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Recipe

llm-compressor 0.12: AWQModifier(duo_scaling=False) + QuantizationModifier(W4A16)
Ignore: re:.*visual.*, re:.*lm_head, re:.*mtp.*
Calib: 256 mixed-modal samples from lmms-lab/flickr30k, seq 2048
MTP splice after save from Qwen/Qwen3.5-4B

License

Apache-2.0, inherited from the base model.

Downloads last month: 73

Safetensors

Model size

5B params

Tensor type

I64

I32

BF16

Model tree for LostGentoo/Qwen3.5-4B-AWQ

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(301)

this model