Brooooooklyn/Ornith-1.0-35B-nvfp4-mlx

Ornith-1.0-35B — NVFP4 (mlx-node)

NVFP4 microscaling floating-point quantization of deepreinforce-ai/Ornith-1.0-35B for Apple Silicon, via mlx-node.

Ornith-1.0 is a self-improving family of open-source agentic coding models. The 35B member is a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) post-train.

	Original (BF16)	This Model
Size	~68 GB	23 GB
Format	SafeTensors (sharded)	SafeTensors (sharded)
Precision	BF16 uniform	NVFP4 (FP4 E2M1, gs16) FFN + 5/6/8-bit affine + BF16

All Variants

Repo	Format	Size	Decode (tok/s)
Brooooooklyn/Ornith-1.0-35B-UD-Q3_K_XL-mlx	UD-Q3_K_XL	17 GB	111.6
Brooooooklyn/Ornith-1.0-35B-mxfp4-mlx	MXFP4	20 GB	107.8
Brooooooklyn/Ornith-1.0-35B-UD-Q4_K_XL-mlx	UD-Q4_K_XL	22 GB	102.3
Brooooooklyn/Ornith-1.0-35B-nvfp4-mlx (this model)	NVFP4	23 GB	94.6
Brooooooklyn/Ornith-1.0-35B-UD-Q5_K_XL-mlx	UD-Q5_K_XL	26 GB	95.4
Brooooooklyn/Ornith-1.0-35B-UD-Q6_K_XL-mlx	UD-Q6_K_XL	31 GB	93.1
Brooooooklyn/Ornith-1.0-35B-UD-Q8_K_XL-mlx	UD-Q8_K_XL	36 GB	91.5
Brooooooklyn/Ornith-1.0-35B-mxfp8-mlx	MXFP8	36 GB	84.8

Benchmarked on a cool Apple M5 Max: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.)

Performance

Steady-state decode: 94.6 tok/s (1.5x vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), so the active-weight footprint streamed per token is what matters.

Apple-Silicon speed note: this NVFP4 build decodes 94.6 tok/s — essentially identical to the 4-bit affine build UD-Q4 (102.3 tok/s). NVFP4 targets NVIDIA Blackwell FP4 tensor cores; on Apple Silicon it is fully competitive with affine here, so it is a good pick if you also deploy the same checkpoint on CUDA.

Output Quality

Decoded-text quality was verified against the BF16 reference with a multi-judge review of the actual generated output (not a heuristic): a 4-turn factual chat plus a Python is_balanced() bracket-matching task. This NVFP4 build produced coherent prose, correct facts, and a correct implementation — no runaway generation, repetition loops, or stray tokens — on par with full precision.

Per-Tensor Quantization

Weight	Format	Rationale
switch_mlp.gate_proj/up_proj	NVFP4 (FP4 E2M1, gs16, FP8 scales)	MoE expert bulk — Blackwell-style microscaled FP4
switch_mlp.down_proj	5-bit affine	slightly more sensitive — kept affine
self_attn.q/k/v_proj, linear_attn.in_proj_qkv/z	6-bit affine	attention/SSM inputs protected
self_attn.o_proj, linear_attn.out_proj, in_proj_a/b	8-bit affine	sensitive output projections
Router gates (mlp.gate, shared_expert_gate)	8-bit affine	MoE routing accuracy
embed_tokens, lm_head	bf16	embeddings/head full precision
GDN params (A_log, dt_bias)	bf16	state-space dynamics
vision_tower.*	bf16	vision encoder kept full precision

Quantization Strategy

NVFP4 is NVIDIA's FP4 format (E2M1 elements, group_size 16, FP8 E4M3 block scales) designed for Blackwell FP4 tensor cores. MLX runs it natively on Metal, but on Apple Silicon there is no FP4 tensor hardware, so the small groups and FP8 scale unpacking make it slower than integer-affine 4-bit at the same size — it is included here for format completeness and for the CUDA/Blackwell inference path. The qwen3_5 recipe keeps sensitive tensors in affine (down_proj 5-bit, attention/SSM 6-bit, output/router 8-bit) so only the MoE expert gate/up projections are true FP4.

Architecture

Parameter	Value
Total parameters	35.9B (~3B active per token)
Hidden size	2,048
Layers	40 (30 linear GatedDeltaNet + 10 full attention)
Attention heads	16 (2 KV heads, GQA 8:1)
Head dimension	256
Experts	256 per MoE layer, top-8 routing
Vocab size	248,320
Vision	yes (Qwen3.5-VL vision tower, kept bf16)
Max context	262,144 tokens

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Ornith-1.0-35B-nvfp4-mlx');

for await (const event of session.sendStream('Write a Python function to merge two sorted lists.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i Ornith-1.0-35B \
  -o Ornith-1.0-35B-nvfp4-mlx \
  -q --q-mode nvfp4 --q-recipe qwen3_5

NVFP4 (group_size 16, FP8 E4M3 block scales) is applied to the MoE expert gate/up projections via the qwen3_5 recipe (no imatrix). Sensitive tensors fall back to affine: down_proj 5-bit, attention/SSM inputs 6-bit, output projections and routers 8-bit; embeddings, head and vision tower stay bf16.

Acknowledgments

• Unsloth — Per-layer KLD bit-allocation strategy (Dynamic 2.0)
• DeepReinforce — For the Ornith-1.0 model family
• Qwen Team — For the Qwen3.5 base architecture
• Apple MLX — For the Metal-accelerated ML framework

License

MIT (inherited from base model).