Disrupt-linus/FastH3-Omni-MLX-INT4

FastH3-Omni MLX INT4 (Apple Silicon Checkpoint)

Pre-converted 4-bit affine quantized checkpoint for FastH3-Omni (multi-reference video + audio generation) running natively on Apple Silicon Metal via Apple MLX.

Created for the Apple Silicon AI community and directly addressing the request on r/StableDiffusion to eliminate the need for a 65 GB raw download and 50+ GB host RAM conversion step.

---

⚡ Highlights & Benchmarks

Metric	Upstream PyTorch / Safetensors	FastH3-Omni MLX INT4 (Disrupt-linus)
DiT Weights Size	~52.8 GB (BF16)	~11.9 GB (INT4 affine)
Full Distribution	~65 GB	~15.0 GB
Minimum VRAM Required	64 GB+ (MPS/CUDA)	24 GB / 32 GB / 64 GB Apple Silicon
Quantization Format	Uncompressed BF16	INT4 Affine (group_size=64, VSA gate projections retained)
AdaLN Timestep Cache	Dynamically computed (~12 GB runtime)	Pre-baked 8-step PDD schedule (0 MB dynamic allocation)
Target Hardware	Multi-GPU Linux / Datacenter	MacBook Pro / Mac Studio (M1/M2/M3/M4 Pro/Max/Ultra)

---

📦 Directory Structure

FastH3-Omni-MLX-INT4/
├── README.md
├── dit/
│   └── int4/
│       ├── mlx_h3_dit.safetensors    # 11.9 GB: Quantized DiT weights + AdaLN cache + gate compressions
│       └── mlx_h3_dit.json           # Manifest: num_blocks: 50, hidden_dim: 2688, vsa.capable: true
├── vae/
│   ├── fasth3-quality-36L-decoder-nvfp4.safetensors  # 1.3 GB (Quality 36L VAE decoder)
│   ├── fasth3-fast-8L-decoder-nvfp4.safetensors     # 313 MB (Fast 8L real-time decoder)
│   └── lightvae-lite-h3-encoder.safetensors         # 388 MB (LightVAE Lite reference image encoder)
└── root/
    ├── fastvideo_inference.json      # Pipeline orchestration configuration
    ├── audio_scheduler/              # FlowMatchEulerDiscreteScheduler config
    ├── audio_vae/                    # Omnidirectional audio latent VAE (577 MB)
    ├── processor/                    # Multimodal video preprocessor & tokenizer configs
    ├── scheduler/                    # 8-step PDD video flow scheduler config
    ├── text_encoder/                 # Qwen2.5-VL / conditioned text encoder config
    ├── tokenizer/                    # Tokenizer merges, vocab, and configs
    ├── transformer_ref/              # Architecture reference config
    └── vae/                          # Base VAE config

---

🚀 Quickstart & Verification (Python / MLX)

1. Load DiT Directly

import mlx.core as mx
from fastvideo.mlx_runtime.minimax_h3 import load_mlx_h3_checkpoint
from huggingface_hub import snapshot_download

# Download and load directly
model_dir = snapshot_download("Disrupt-linus/FastH3-Omni-MLX-INT4")
dit = load_mlx_h3_checkpoint(f"{model_dir}/dit/int4")

print(f"Loaded: {len(dit.blocks)} blocks")
print(f"VSA Capable: {dit.vsa_capable}")      # True
print(f"Has Refiner: {dit.refiner is not None}") # True
print(f"Pre-baked AdaLN steps: {len(dit._adaln_cache.timesteps)}") # 17 steps

2. End-to-End Inference with FastVideo

git clone -b aryan/fasth3-omni-mlx https://github.com/fastvideo/FastVideo.git
cd FastVideo
pip install mlx torch transformers diffusers huggingface_hub

python -m fastvideo.sample.sample_minimax_h3_mlx \
    --model_path "Disrupt-linus/FastH3-Omni-MLX-INT4" \
    --prompt "A cinematic drone shot through a misty Scandinavian pine forest at dawn, rays of golden sunlight breaking through morning fog, atmospheric ambient bird song and distant wind." \
    --num_frames 49 \
    --height 480 \
    --width 720 \
    --num_inference_steps 8 \
    --output_path "forest_dawn.mp4"

---

🛠 Model Quantization Architecture Details

• Quantized using affine 4-bit integer weights (group_size=64).
• Retained full precision for cross-attention projection layers and normalization layers.
• Embedded static AdaLN timestep conditioning vectors for the 8-step PDD flow scheduler directly into the checkpoint dictionary (_adaln_cache), saving ~12 GB of runtime VRAM during generation passes.
• Maintained exact VSA (Visual-Spatial Alignment) gate compression matrices for multi-reference spatial conditioning.