Sunflower-Qwen3.5-9B-bnb-4bit

4-bit (NF4, bitsandbytes) build of Sunbird/Sunflower-Qwen3.5-9B.

Generated by scripts/push_4bit.py.

Source repo	Sunbird/Sunflower-Qwen3.5-9B
Source revision	10156eaa72a329ee5bef01a72d561fb1923274d2
Quantisation	bitsandbytes 4-bit, nf4
Double quantisation	True
Compute dtype	bfloat16
Size on disk	7.9 GB

Why this exists

The full-precision model does not fit a 16 GB GPU. This build is 7.9 GB and runs on a free Colab T4 — Turing GPUs have no fp8 support, so 4-bit is the only way to run the model there.

embed_tokens and lm_head stay in bfloat16 (~4.1 GB) because bitsandbytes quantises nn.Linear only, and this model has a 248k-token vocabulary with untied embeddings.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_ID = "Sunbird/Sunflower-Qwen3.5-9B-bnb-4bit"

# Quantisation config is already in config.json — nothing extra to pass.
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, device_map="cuda")
tokenizer = AutoProcessor.from_pretrained(MODEL_ID).tokenizer

messages = [{"role": "user", "content": "Translate to Luganda: Good morning, how are you?"}]
enc = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to(model.device)

out = model.generate(**enc, max_new_tokens=256, do_sample=False)  # temperature 0
print(tokenizer.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requires bitsandbytes (pip install bitsandbytes).

Limitations

• Transformers only. vLLM cannot load bitsandbytes checkpoints for this architecture. Use the full-precision repo with quantization="fp8" on an L4/A100/H100, or build a compressed-tensors checkpoint with llm-compressor.
• 4-bit is slightly lower quality than bf16/fp8, and generation is slower because weights are dequantised on the fly.

Smoke test at build time:

Translate to Luganda: Good morning, how are you?
-> Wasuze otya, oli otya?