OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit

Gemma 4 E4B it SDNQ Dynamic INT8

This is an int8 quantized version of google/gemma-4-E4B-it using SDNQ (SD.Next Quantization) with the dynamic option, Hadamard Rotation and quantized embeddings.

Note: You need SDNQ v0.2.1 or superior (the quantized embeddings + Hadamard path requires the nn.Embedding fix that landed in 0.2.1).

Usage

Load it with transformers like the original model. You need to import sdnq first so it registers into the transformers quantizer registry — without it, loading fails with a size-mismatch error:

import sdnq  # registers the SDNQ quantizer into transformers
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

messages = [{"role": "user", "content": [{"type": "text", "text": "Explain quantization in one sentence."}]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))