Brooooooklyn/Agents-A1-nvfp4-mlx

Agents-A1 — Unsloth NVFP4 + E4M3 FP8 weights (MLX, DGX)

Brooooooklyn/Agents-A1-nvfp4-mlx is an MLX mixed-weight-format quantization of InternScience/Agents-A1, prepared for experimental NVIDIA CUDA inference on Linux aarch64. Agents-A1 is a 35B-A3B Qwen3.5-family mixture-of-experts agent model with 40 language-model layers, 256 experts with top-8 routing, hybrid linear/full attention, and an MTP-capable config. The published source checkpoint does not include an mtp.* tensor subtree.

This model is part of the Qwen Unsloth tensor-class recipe for MLX on macOS and DGX collection.

The source was the all-BF16 checkpoint at revision 55100f11160f545dc45545c677699ace74f6bd10.

Quantization recipe

This is a data-free, weight-only MLX storage port of the Unsloth Qwen3.6 NVFP4 recipe:

• tensors assigned NVFP4 by the recipe are stored as NVFP4, 4-bit with group size 16;
• tensors assigned FP8 by the recipe are stored as raw E4M3 FP8 weight bytes with one BF16 dequantization scale per output channel;
• every excluded tensor stays BF16.

The NVFP4 class uses MLX weight-only quantized matmul with BF16/A16 activations. For the FP8 class, mlx-node reconstructs each weight to BF16 once at load and then uses ordinary A16 matmul or gather-matmul. The serialized fp8_e4m3 form is Uint8 [..., N, K] weight plus BF16 [..., N, 1] scale; it is not MLX mxfp8 and it is not native W8A8 execution.

No imatrix, calibration dataset, AWQ-style pre-scaling, activation calibration, NVFP4 global scale, or FP8 KV-cache calibration was used. This artifact preserves the recipe's tensor-class boundaries and weight storage formats under mlx-node's A16 runtime; it does not claim numerical or performance parity with Unsloth's calibrated W4A4/W8A8 execution.

Tensor class	Stored format
Routed expert switch_mlp.{gate,up,down}_proj, layers 0–31	NVFP4 4/16
Shared expert {gate,up,down}_proj, layers 0–31	NVFP4 4/16
The same routed/shared FFN projections, layers 32–39	E4M3 FP8 weight + per-output BF16 scale
Full-attention {q,k,v,o}_proj	E4M3 FP8 weight + per-output BF16 scale
Linear-attention in_proj_qkv, in_proj_z, out_proj	E4M3 FP8 weight + per-output BF16 scale
lm_head	E4M3 FP8 weight + per-output BF16 scale
Embeddings; router mlp.gate and shared_expert_gate; in_proj_a/b; GDN state, convolution, and norm tensors; all other norms	BF16
MTP tensors	Not present in the published source checkpoint
Vision tower and merger tensors	BF16

The allocation contains 192 NVFP4 modules and 179 E4M3 FP8 modules. The top-level config is nvfp4, 4-bit, group size 16, so the 192 low-class modules inherit that default. The config carries 179 explicit fp8_e4m3 overrides with bits: 8 and group_size: null. The final eight FFN layers intentionally use the higher class; this is the regular, accuracy-oriented 35B recipe rather than the all-FFN-FP4 "Fast" variant.

Target and usage

The canonical native target is aarch64-unknown-linux-gnu: Linux aarch64 with glibc and NVIDIA CUDA 13.0. mlx-node currently validates this experimental, inference-only path on NVIDIA GB10 / DGX Spark (sm_121). It is not a generic CUDA or x86_64 artifact.

At mlx-node 0.0.8, CUDA has no published prebuilt native npm binary. Build mlx-node from source on the DGX host:

git clone --branch v0.0.8 https://github.com/mlx-node/mlx-node.git
cd mlx-node
git submodule update --init --recursive
yarn install
yarn build

Paged attention is Metal-only in this release. Set both eager-mode variables for DGX inference:

MLX_QWEN35_FORCE_EAGER=1 \
MLX_QWEN35_PAGED_OVERRIDE=0 \
  yarn oxnode your-script.ts

For example, your-script.ts can load a locally downloaded copy:

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Agents-A1-nvfp4-mlx');
const result = await session.send('Reply briefly: what can you help with?');
console.log(result.text);

This checkpoint requires the @mlx-node/lm and @mlx-node/core 0.0.8 source tree or a newer release that explicitly supports the same Linux target and serialized modes.

Reproduction

Converter release: mlx-node v0.0.8. The reproducible invocation from the mlx-node repository root was:

mlx convert \
  --input .cache/models/agents-a1 \
  --output .cache/models/agents-a1-unsloth-nvfp4-fp8-dgx-mlx-fresh \
  --model-type qwen3_5_moe \
  --dtype bfloat16 \
  --quantize \
  --q-recipe unsloth \
  --q-mode nvfp4

The resulting five-shard SafeTensors index contains 1,437 tensor entries and reports metadata.total_size = 24,784,342,752 bytes. Its tensor dtypes are 874 BF16, 371 U8, and 192 U32 entries, with 371 scale sidecars and no quantization bias sidecars.

Validation

Static validation confirmed identical quantization and quantization_config blocks, exact index-to-shard closure, 192 inherited NVFP4 groups, 179 complete fp8_e4m3 groups, the expected storage dtypes and shapes, and BF16 preservation for protected tensors. All 333 vision tensor entries remain BF16, and there is no MTP tensor subtree.

The fixed one-token mlx-node load-and-generate smoke test exited successfully with finishReason = "length", numTokens = 1, text = "OK", and rawText = "OK". This one-token text smoke does not validate model quality, long-context behavior, tool use, or the vision path.

Benchmark

macOS A16 fallback only — these are not DGX/CUDA throughput results.

The values below are medians from three fresh child processes, each loading the checkpoint and generating a deterministic 512-token completion on an Apple M5 Max with 128 GiB of unified memory, Darwin 25.5.0/arm64, Node 24.13.1, and @mlx-node/lm, @mlx-node/core, and @mlx-node/core-darwin-arm64 0.0.8. The run used zero warmups, a 60-second cooldown, temperature 0, reasoning effort none, and the same 106-token prompt. Every sample generated all 512 tokens and ended with finishReason = "length".

Metric	macOS A16 fallback median (not DGX/CUDA)
Load time	53,908.621 ms
Time to first token	1,813.556 ms
Prefill throughput	58.449 tokens/s
Decode throughput	59.637 tokens/s
Generation wall time	10,610.061 ms
Total wall time	64,268.905 ms

The prompt and all per-run samples are recorded in benchmark.json. Load time varied strongly because the weights were read from external storage and OS page-cache state differed between fresh processes; treat that median as specific to this run. This fallback benchmark did not exercise DGX, CUDA, native W4A4/W8A8 execution, or the vision path, and it must not be used to infer model quality, memory requirements, or parity with upstream execution.

License and attribution

The source model card declares the Apache-2.0 license. Model capability and training credit belong to InternScience. The tensor-class recipe is credited to Unsloth. This repository converts the pinned BF16 source weights into the mixed NVFP4/plain-E4M3 MLX representation described above.