Demondiablo/medgemma-4b-it-int8-w8a8

MedGemma 4B-IT (INT8 W8A8 Dynamic)

This is an INT8 W8A8 Quantized version of google/medgemma-4b-it optimized for high-throughput, low-latency deployment with vLLM and compatible serving engines across all modern GPU architectures.

Quantization Details

• Base Model: google/medgemma-4b-it
• Quantization Method: llm-compressor
• Quantization Scheme: W8A8 (Weights in INT8 per-channel symmetric, activations in dynamic INT8 per-token)
• Format: compressed-tensors (native vLLM format)
• Excluded Modules: lm_head, embed_tokens, multi_modal_projector, and vision tower linear layers preserved for clinical accuracy.
• Quantized On: NVIDIA RTX PRO 6000 Blackwell Server Edition

Deployment & Usage

1. High-Performance Inference with vLLM (Recommended)

from vllm import LLM, SamplingParams

model_name = "Demondiablo/medgemma-4b-it-int8-w8a8"

llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_model_len=4096,
)

sampling_params = SamplingParams(
    temperature=0.2,
    max_tokens=2048,
)

prompt = "You are a clinical AI. Summarize the following prescription: Tablet Thyronorm 50 mcg OD."
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)

2. Inference with Transformers

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

model_name = "Demondiablo/medgemma-4b-it-int8-w8a8"

processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Extract medications: Capsule Desula 21 HS, Thyronorm 50 mg OD."
inputs = processor(text=prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(outputs[0], skip_special_tokens=True))

Hardware Compatibility

INT8 W8A8 acceleration is supported across virtually all modern NVIDIA architectures:

• NVIDIA Turing (T4, RTX 2080 Ti)
• NVIDIA Ampere (A100, A10, RTX 3090, A6000)
• NVIDIA Ada Lovelace (RTX 4090, L4, L40S, RTX 6000 Ada)
• NVIDIA Hopper (H100, H200)
• NVIDIA Blackwell (RTX PRO 6000 Blackwell, B100, B200)