Edge0/Edge0-8B-A1B-preview

Edge0-8b-a1b Preview

An 8B-class sparse MoE that runs on a phone — 1.0 GiB of active memory, experts streamed from SSD.

Edge0-8b-a1b — an 8B MoE LLM that runs at viable speed on portable devices in under 1.0 GiB of active memory (1/4 of its 4.2 GB weight footprint), via the edge0 streaming inference framework.

The key is streaming: experts are memory-mapped and fetched from SSD only as routed, so RAM holds just the active weights. What makes that viable — instead of stalling like plain parameter offloading — is a trained prerouter head that predicts the next token's expert routing one step ahead, hiding storage latency behind compute.

Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.

Model summary

Base model	inclusionAI Ling 3.0 tiny (bailing hybrid, MLA + MoE, ≈7.9B total / ≈1.2B active)
Quantization	4-bit
Layers	24
Experts / active per token	128 / 8 (K=8)
Framework	edge0 (MLX backend)
Contents	base checkpoint + lora_edge0_8b.safetensors + prerouter_edge0_8b.safetensors

The LoRA and prerouter adapters are co-located with the base checkpoint and load automatically — this repository is a complete, ready-to-run model directory for edge0.

Quality

All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 2.8 points on average, with MMLU-Pro above the base (max 100):

Benchmark	edge0-8b (int4)	Ling 3.0 tiny (fp16)
AIME 2026	63.3	73.3
HumanEval	91.5	92.7
GPQA-Diamond	70.7	71.2
MMLU-Pro	70.1	65.8
IFBench	53.9	60.6
Average	69.9	72.7

Performance

Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:

Decode speed	Prefill throughput (cold / warm)	Peak active memory*
23.9–25.3 tok/s	500 / 1428 tok/s	1.0 GiB

*Short contexts; long contexts add KV cache (≈3.3 GiB at 3.3k tokens). Expert weights stream from SSD via mmap and are not resident.

Quick start

pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'

# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview

# Run it
export EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview
edge0 chat --name edge0-8b --prompt "Introduce yourself"

# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-8b --port 8083

For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.

License

Apache 2.0. See LICENSE.