Brooooooklyn/Ornith-1.0-35B-mxfp8-mlx

Ornith-1.0-35B — MXFP8 (mlx-node)

MXFP8 microscaling floating-point quantization of deepreinforce-ai/Ornith-1.0-35B for Apple Silicon, via mlx-node.

Ornith-1.0 is a self-improving family of open-source agentic coding models. The 35B member is a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) post-train.

	Original (BF16)	This Model
Size	~68 GB	36 GB
Format	SafeTensors (sharded)	SafeTensors (sharded)
Precision	BF16 uniform	MXFP8 (FP8 E4M3, gs32) body + 8-bit affine routers + BF16

All Variants

Repo	Format	Size	Decode (tok/s)
Brooooooklyn/Ornith-1.0-35B-UD-Q3_K_XL-mlx	UD-Q3_K_XL	17 GB	111.6
Brooooooklyn/Ornith-1.0-35B-mxfp4-mlx	MXFP4	20 GB	107.8
Brooooooklyn/Ornith-1.0-35B-UD-Q4_K_XL-mlx	UD-Q4_K_XL	22 GB	102.3
Brooooooklyn/Ornith-1.0-35B-nvfp4-mlx	NVFP4	23 GB	94.6
Brooooooklyn/Ornith-1.0-35B-UD-Q5_K_XL-mlx	UD-Q5_K_XL	26 GB	95.4
Brooooooklyn/Ornith-1.0-35B-UD-Q6_K_XL-mlx	UD-Q6_K_XL	31 GB	93.1
Brooooooklyn/Ornith-1.0-35B-UD-Q8_K_XL-mlx	UD-Q8_K_XL	36 GB	91.5
Brooooooklyn/Ornith-1.0-35B-mxfp8-mlx (this model)	MXFP8	36 GB	84.8

Benchmarked on a cool Apple M5 Max: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.)

Performance

Steady-state decode: 84.8 tok/s (1.4x vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), so the active-weight footprint streamed per token is what matters.

Apple-Silicon speed note: this MXFP8 build decodes 84.8 tok/s, about 5% under the equal-size 8-bit affine build UD-Q8 (91.5 tok/s) — Apple Silicon has no native FP8 tensor hardware, so the FP8 block-scale unpack costs a little. Both are strong 8-bit options: pick UD-Q8 for the last few percent of speed, MXFP8 for the microscaling format itself or CUDA/Blackwell portability.

Output Quality

Decoded-text quality was verified against the BF16 reference with a multi-judge review of the actual generated output (not a heuristic): a 4-turn factual chat plus a Python is_balanced() bracket-matching task. This MXFP8 build produced coherent prose, correct facts, and a correct implementation — no runaway generation, repetition loops, or stray tokens — on par with full precision.

Per-Tensor Quantization

Weight	Format	Rationale
switch_mlp.gate_proj/up_proj/down_proj	MXFP8 (FP E4M3, gs32)	MoE expert bulk — microscaled FP
self_attn.q/k/v/o_proj	MXFP8 (gs32)	attention projections
linear_attn.in_proj_qkv/z, out_proj, in_proj_a/b	MXFP8 (gs32)	GatedDeltaNet projections
Router gates (mlp.gate, shared_expert_gate)	8-bit affine	MoE routing accuracy (never FP)
embed_tokens, lm_head	bf16	uniform-FP path keeps embeddings/head full precision
GDN params (A_log, dt_bias)	bf16	state-space dynamics
vision_tower.*	bf16	vision encoder kept full precision

Quantization Strategy

MXFP8 is a microscaling floating-point format (8-bit elements with a shared block exponent per group of 32). Unlike integer-affine quantization, the per-group exponent adapts the dynamic range locally, which suits weight distributions with outliers. MLX runs it natively on Metal via the fp_gather_qmm kernels for MoE experts — no dequantize-to-bf16 fallback. Router gates remain 8-bit affine and the embeddings, head, GatedDeltaNet state params and vision tower stay bf16.

Architecture

Parameter	Value
Total parameters	35.9B (~3B active per token)
Hidden size	2,048
Layers	40 (30 linear GatedDeltaNet + 10 full attention)
Attention heads	16 (2 KV heads, GQA 8:1)
Head dimension	256
Experts	256 per MoE layer, top-8 routing
Vocab size	248,320
Vision	yes (Qwen3.5-VL vision tower, kept bf16)
Max context	262,144 tokens

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Ornith-1.0-35B-mxfp8-mlx');

for await (const event of session.sendStream('Write a Python function to merge two sorted lists.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i Ornith-1.0-35B \
  -o Ornith-1.0-35B-mxfp8-mlx \
  -q --q-mxfp --q-bits 8

The --q-mxfp path upgrades the quantizable linears to MXFP8 microscaled floating-point (shared block exponents, group_size 32). Router gates stay 8-bit affine for routing accuracy; embeddings, head, GatedDeltaNet state params and the vision tower stay bf16.

Acknowledgments

• Unsloth — Per-layer KLD bit-allocation strategy (Dynamic 2.0)
• DeepReinforce — For the Ornith-1.0 model family
• Qwen Team — For the Qwen3.5 base architecture
• Apple MLX — For the Metal-accelerated ML framework

License

MIT (inherited from base model).