Realtime-Venus-Omni-4bit

This repository provides an optimized 4-bit (NF4) quantized version of inclusionAI/Realtime-Venus-Omni.

It is specifically tailored to run comfortably on 16GB consumer GPUs (such as NVIDIA GeForce RTX 5060 Ti 16GB, RTX 4060 Ti 16GB, RTX 4070 Ti / 4080, etc.) without out-of-memory errors during live duplex multimodal conversations.

---

🚀 Key Improvements & Highlights

1. Selective 4-bit Quantization (Zero Loss in Audio & Vision):
   • Vision Encoder (NaViT SigLIP): Kept in native BF16/FP16 for pristine visual perception.
   • Speech Synthesis (StepAudio TTS Flow & HiFT): Kept in native FP16 & ONNX for 100% natural, crisp 24kHz voice output.
   • LM Head: Kept in BF16 to prevent vocabulary logit degradation and dynamic dequantization spikes.
   • LLM Backbone (Qwen3): Linear layers quantized with NF4 (NormalFloat4) + double quantization.
2. Built-in PyTorch SDPA Optimization:
   • Replaced full $N \times N$ attention expansion in SiglipAttention with PyTorch native scaled_dot_product_attention (SDPA), eliminating the 50GB+ memory spike during multi-frame video ingestion.
3. Hardware Efficiency:
   • Weight Size: Reduced from 18.7 GB to 8.4 GB (~55% reduction).
   • VRAM Consumption: Runs smoothly at ~10.6 GB (idling) to ~12.5 - 13.7 GB (full duplex multimodal streaming), easily fitting within 16GB VRAM.
   • Loading Speed: Loads in ~1 min 20s directly from pre-quantized safetensors.

---

🛠️ Quickstart

1. Requirements

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install transformers accelerate bitsandbytes sentencepiece soundfile librosa opencv-python onnxruntime

2. Loading the Model

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "STIAMO/Realtime-Venus-Omni-4bit"

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    local_files_only=False,
    device_map={"": "cuda:0"},
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
print("Realtime-Venus-Omni 4-bit loaded successfully!")

3. Duplex Conversation

duplex_model = model.as_duplex()
duplex_model.prepare()

# Stream 1 second of audio & video frame
# duplex_model.streaming_prefill(audio_waveform=audio_chunk, frame_list=[pil_frame])
# result = duplex_model.streaming_generate()

---

⚖️ License & Attribution

This model inherits the weights and architecture from inclusionAI/Realtime-Venus-Omni. Please adhere to the original project licenses and legal notices.