ATF — Adaptive Tensor Format for Apple Silicon

Introducing ATF (Adaptive Tensor Format) — a custom model format that converts quantized GGUF and MLX checkpoints into a single-file .atf container with hand-written Metal kernels, memory-mapped loads, GPU-resident weights, and an adaptive reasoning router, pushing inference close to the hardware's real memory-bandwidth ceiling in load times measured in seconds.

The problem ATF solves

Many GGUF runtimes dequantize less-common formats (IQ4_NL, Q6_K, UD-Q2_K_XL...) on slow, generic CPU-bound paths — or don't support them on Metal at all. ATF keeps whatever quantization you already chose and swaps that generic path for custom Metal kernels, so decode runs at DRAM-bandwidth-bound speed instead of being stalled on dequantization:

Kernel	Before ATF	With ATF	Max rel. error
IQ4_NL	per-token CPU dequant	74 GB/s	≤ 2.1e-06
Q6_K	unsupported (crash)	62 GB/s	≤ 2.3e-06

That's within reach of the M4's ~84 GB/s memory-bandwidth ceiling — these formats are now bandwidth-bound, not kernel-bound, on Apple hardware.

Models

Model	Size	Quantization	Decode (warm)	Load
Qwen3.5-9B-IQ4_NL.atf	5.5 GB	IQ4_NL + Q6_K lm_head	12.96 tok/s	2.3 s warm / 9.6 s cold
Qwen3.5-9B-mlx4bit.atf	6.9 GB	MLX 4-bit	~12–13 tok/s	seconds
Qwen3.8-27B-UD-Q2_K_XL.atf	9.5 GB	UD-Q2_K_XL (mixed K/IQ)	4.73 tok/s	~15 s

Benchmarks: Apple M4, 16 GB unified memory, greedy decode, 4k context, warm pass. All three run comfortably on a 16 GB Mac.

Getting started

1. Download ATF Chat (DMG) — drag to /Applications. Python/MLX bundled, nothing else to install.
2. Grab a .atf file above into ~/Library/Application Support/atf-chat/models — or let the app download it from the Models tab.
3. Pick the model in the dropdown and start chatting.

OpenAI-compatible API

ATF Chat ships a built-in server, so it drops into tools you already use:

# point any OpenAI client at:
http://localhost:8000/v1/chat/completions

SSE streaming included — works with ChatBox, LibreChat, OpenWebUI, and anything else that speaks the OpenAI chat-completions format.

Features

• A single mmap-able file — no sidecar configs or split shards to manage
• 64k context window — chunked prefill with a preallocated, growing KV cache
• On-demand loading — startup reads a 128-byte header only; models hot-swap with full GPU memory release, no app restart
• Persistent chats — multi-conversation sidebar with Markdown/JSON export
• Full sampling controls — temperature, top_p, repeat penalty, system prompt
• Adaptive reasoning-depth router — every prompt is scored for difficulty and routed to a tier (instant / chat / reasoning / deep) that sets the thinking budget, temperature, and max tokens for that request — no manual mode switch needed for a quick question vs. a hard one
• Fast tokenization — uses gigatoken (Rust-backed) for tokenizer encode/decode instead of the slower pure-Python path

Converting your own models

Bring your own GGUF or MLX checkpoint and get the same custom-kernel speedup:

# From GGUF (keeps its quantization, adds fast Metal kernels)
python -m atf.cli convert model.gguf --raw -o MyModel.atf

# From MLX / safetensors checkpoints
python -m atf.convert_mlx <mlx_dir> -o MyModel.atf
python -m atf.convert <model_dir> -o MyModel.atf

File format

offset 0   magic "ATF1"          4 bytes
offset 4   version major/minor   2 × uint16
...        header (128 bytes)    dims, dtypes, tokenizer, LOD table
...        weights               mmap-able, 16-byte aligned

The current layout: MLX-native tensors carrying GGUF quantization schemes on a fully mmap-able, 16-byte-aligned payload — load time is a GPU upload, not a deserialization pass.

Requirements

• Apple Silicon Mac (M1/M2/M3/M4), 16 GB unified memory minimum
• macOS 13+

Roadmap

Still ahead: storage-tier adaptivity — swapping in lower-bit experts for easy prompts and higher-precision ones for hard prompts on disk/load, not just at inference time — is designed but not yet shipped; today every model loads at a single precision tier. Graph-level fusion (mx.compile) for a further decode speedup is also on the list. Both are where contributions would have the most impact.

License

Model weights follow the Qwen license. The ATF format and runtime are Apache-2.0.