UnderstandLing/Hy3-colibri-int4

Hy3 — colibrì int4 container (~142 GB)

Pre-converted weights for colibrì — a pure-C engine that runs huge MoE models on consumer hardware by keeping the dense stack resident in RAM and streaming routed experts from disk.

This repo is the first public Colibri-format checkpoint for Tencent Hy3. It lets you run Hy3's 295B-parameter MoE (21B active per token) on a machine with modest RAM and a fast local NVMe drive, instead of an 8×GPU datacenter setup.

⚠️ This is NOT a GGUF / AWQ / GPTQ / MLX / vLLM model. It uses colibrì's own safetensors container: for each quantized weight, name (packed int4/int8) + name.qs (F32 per-row scales), with packing math bit-identical to the engine's C kernels. It only works with the colibrì hy3 engine (see the Hy3 port in this conversion fork).

What we contribute

The upstream tencent/Hy3 weights ship for BF16 / FP8 inference stacks (vLLM, SGLang, Transformers). Community quants (GGUF, AWQ, etc.) target other runtimes and cannot be loaded by colibrì.

This release adds the missing disk-streaming Colibri container plus the runtime pieces needed to use it:

Contribution	Description
Colibri int4 container	107 out-*.safetensors shards (~142 GB) converted from tencent/Hy3-FP8 with convert_hy3.py — shard-at-a-time, resumable, never needs the full FP8 tree on disk at once
Hy3-aware quantization	Mixed precision tuned for streaming: int4 routed experts (disk), int8 attention / shared expert / embeddings, F32 norms + sigmoid router + expert bias; Hy3-FP8 per-tensor weight_scale dequant path (not GLM's block scales)
MTP head shards	6 out-mtp-*.safetensors shards (layer 80, int8) — weights included for future speculative decode; MTP runtime in hy3.c is not wired yet
hy3.c engine	New pure-C forward for hy_v3: GQA attention (64Q / 8KV, Q/K RMSNorm, rotate-half RoPE), sigmoid MoE router with bias + route normalization + router_scaling_factor, batch-union expert matmul, LRU/pin expert cache
Converter + validation	convert_hy3.py, make_hy3_oracle.py, audit_hy3_tensors.py; oracle teacher-forcing validated on a tiny Hy3 fixture before scaling to full weights
Chat / serve integration	Hy3 Hunyuan chat template (｜ delimiters), coli chat, OpenAI-compatible coli serve

Bottom line: same model family as tencent/Hy3, but repackaged so a single NVMe + ~16–64 GB RAM machine can run it via expert streaming — the same idea as jlnsrk/GLM-5.2-colibri-int4, applied to Hy3.

About Hy3 (from the base model)

Hy3 is Tencent's 295B-parameter Mixture-of-Experts instruct model with 21B activated parameters per token, developed by the Tencent Hy Team. See the official model card for benchmarks, agent capabilities, and deployment recipes for GPU stacks.

Property	Value
Architecture	Mixture-of-Experts (MoE), hy_v3
Total parameters	295B
Activated parameters	21B
Layers (excl. MTP)	80 (layer 0 dense, 79 sparse MoE)
Attention	GQA — 64 heads, 8 KV heads, head dim 128
Hidden size	4096
Experts	192 routed + 1 shared, top-8 per token
Context length	256K
Vocabulary	120832

Usage

1. Get the engine (Hy3 port)

git clone https://github.com/JustVugg/colibri.git   # or your Hy3 port fork with c/hy3.c
cd colibri/c
./setup.sh    # builds hy3 + self-test on hy3_tiny fixture

2. Download this repo

Store on a fast local disk (NVMe, native ext4). Avoid WSL /mnt/c (9p) or network mounts — random expert reads are disk-bound.

hf download <your-username>/Hy3-colibri-int4 --local-dir /path/to/hy3_i4

3. Chat

COLI_MODEL=/path/to/hy3_i4 ./coli chat --ram 56 --topp 0.7

Recommended sampling for int4 streaming (differs from the official GPU defaults of temperature=0.9, top_p=1.0 on tencent/Hy3):

• --temp 0.7 --topp 0.7 — colibrì defaults tuned to reduce quantization tail noise

4. OpenAI-compatible API

COLI_MODEL=/path/to/hy3_i4 ./coli serve --ram 56 --host 127.0.0.1 --port 8000

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="hy3-colibri",
    messages=[{"role": "user", "content": "Who are you?"}],
    temperature=0.7,
    top_p=0.7,
)
print(response.choices[0].message.content)

Requirements

• Linux or WSL2 with native ext4 model path
• gcc + OpenMP, AVX2
• ≥16 GB RAM (32–64 GB recommended for a comfortable expert cache)
• ~150 GB free on a fast NVMe SSD
• Optional: CUDA build for future GPU-pinned expert tier (make hy3 CUDA=1)

What's inside

File	Contents
out-*.safetensors (107 shards)	Dense weights (GQA attention, layer-0 FFN, shared expert, embeddings) + 15,168 routed experts (79 MoE layers × 192), int4 with per-row .qs scales; router / norms / bias kept F32
out-mtp-*.safetensors (6 shards)	MTP layer 80 weights (int8) — for future speculative decoding
config.json, tokenizer.json, tokenizer_config.json, generation_config.json	Copied from the Hy3 base repo

On-disk size: ~142 GB (int4 container). Resident RAM (dense only): ~4.1 GB at load time; peak RSS depends on context length, expert cache cap, and --ram budget.

Convert it yourself

If you prefer to reproduce the container locally instead of downloading:

cd colibri/c
pip install torch safetensors huggingface_hub numpy
./coli convert --repo tencent/Hy3-FP8 --model /path/to/hy3_i4

The converter downloads one FP8 shard at a time, dequants, requantizes to the Colibri layout, and deletes the source shard — the full FP8 checkpoint never has to exist on disk at once.

Why not use existing Hy3 quants?

Format	Works with colibrì?
tencent/Hy3-FP8	Input only — must be converted to out-*.safetensors
tencent/Hy3 BF16	Convertible, but larger download
GGUF / llama.cpp / Ollama quants	No — different tensor layout and metadata
AWQ / GPTQ / HF quant repos	No — different scale packing and loaders
vLLM / SGLang	No — GPU serving stacks, not Colibri

Expected performance (honest)

Hy3 is smaller per expert than GLM-5.2 (9 MB vs ~19 MB int4), with 192 experts/layer and 79 MoE layers — cold decode still moves **5–6 GB of expert data per token** from disk. Throughput is disk-bound on a cold cache (often 0.1–0.5 tok/s class on consumer NVMe, highly setup-dependent). Warm LRU / pinned hot experts improve this substantially.

This is not a replacement for vLLM / SGLang deployment on 8× datacenter GPUs. It is a way to run the same model class on hardware you already own.

Status & limitations (early port)

• Beta: the Hy3 hy3.c engine is a new port (validated on oracle fixtures; full-model chat is still being stabilized).
• MTP: weights are included; native speculative decode in hy3.c is not implemented yet.
• CUDA expert tier: planned (make hy3 CUDA=1); not production-ready for Hy3 yet.
• KV disk persistence: not yet ported from GLM's MLA path (GQA layout differs).
• Not loadable in transformers.AutoModelForCausalLM — use colibrì only.

Provenance & license

• Base weights: tencent/Hy3-FP8 and tencent/Hy3 (Apache 2.0).
• Architecture & benchmarks: see tencent/Hy3.
• Engine & converter: colibrì (MIT) + Hy3 port (hy3.c, convert_hy3.py).
• This derivative checkpoint remains under the Apache 2.0 terms of the Hy3 base model.

Citation

If you use the base Hy3 model in academic work, cite Tencent's Hy3 release. If you use this Colibri container, please also credit the colibrì project.