GgufLinear: inference-time GGUF matmul on Apple Silicon — llama.cpp parity
Author: ArthurZucker
Pull request: yes

## What

Opt-in GGUF inference on Apple Silicon. Keeps weights at native quant
(Q4_0 / Q5_0 / Q5_1 / Q8_0 / Q4_K / Q5_K / Q6_K / IQ4_NL / IQ4_XS) and
runs forward through the same Metal kernels llama.cpp ships (packaged
as [`ArthurZ/gguf-kernels`](https://huggingface.co/ArthurZ/gguf-kernels)).

Enabled via `from_pretrained(..., gguf_file=..., gguf_linear=True)`.
Default off.

## Memory

Qwen1.5-MoE-A2.7B Q4_K_M on M3 Max:

| Path | Resident weights |
|---|---:|
| `dtype=torch.bfloat16` | ≈ 28.6 GB |
| `gguf_linear=True` | **≈ 8.8 GB** (3.3 ×) |

## Speed sweep — M3 Max, MAX_NEW=128

| Path | Compile | Attn | MoE Qwen1.5-MoE A2.7B (Q4_K_M) | Dense Llama-3.2-3B (Q4_K_M) |
|---|---|---|---:|---:|
| `generate()` | eager | sdpa | 46.7 | 49.2 |
| **`generate()`** | **compile** | **sdpa** | **103.6 (1.32× llama.cpp)** | **68.8 (1.37× llama.cpp)** |
| `generate_batch` N=1 | eager | sdpa varlen | 37.4 | 21.8 |
| `generate_batch` N=1 | eager | sdpa block-table | 38.5 | 20.3 |
| `generate_batch` N=1 | compile | sdpa block-table | 20.3 | 18.8 |
| `generate_batch` N=8 | eager | sdpa varlen | 72.3 | 62.3 |
| `generate_batch` N=8 | eager | sdpa block-table | 95.3 | 65.5 |
| `generate_batch` N=8 | compile | sdpa block-table | 89.7 | **78.5** |
| `generate_batch` N=1 | eager | mflash varlen | 22.0 | 14.4 |
| **`generate_batch`** N=8 | **eager** | **mflash varlen** | **103.2** | **69.3** |
| llama.cpp `tg256` | — | — | 78.4 | 50.2 |

Aggregate tok/s for N>1.

## What made it fast

1. `GgufLinear` running native Q4_K / Q5_K / Q6_K matvec & matmul Metal kernels — bandwidth-bound decode wins 1.27× vs bf16 matvec.
2. Q4_K simdgroup utilization fix: both matvec + mul_mat_id were dispatched with 64 threads (2 simdgroups) but only used `tiisg`; second simdgroup recomputed the same rows. Added `sgitg` row split. Expert BMM `mul_mat_id` +12 % / +19 %.
3. `_fast_decode_loop` in `generation/utils.py` — pre-allocated output buffer, `argmax(out=)`, in-place Gumbel-max sampling, single `_SYNC_EVERY=16` CPU↔MPS sync.
4. `setup_for_compile` helper — `cache_implementation="static"` + `dynamo.cache_size_limit=512` + `torch.compile(forward, mode="reduce-overhead", dynamic=False)`.
5. `PagedAttentionCache.update` → `index_put_` (was `index_copy_`): **55× per call** on MPS.
6. `paged_decode_attention_f32` Metal kernel — Flash-Decoding online softmax reading K/V through the block-table indirection. No gather. 134 µs/call vs ~16 ms/layer for the gather-then-SDPA fallback.
7. `kv_paged_write_f32` Metal kernel — single-dispatch paged KV write replacing the 5-op Python chain. 6.7 µs/call.
8. Q/K head-permute auto-detect — Qwen2/3-MoE GGUFs ship Q/K in HF layout; applying llama.cpp's reverse-permute on those corrupts the matmul ~140 % relative.
9. CB warmup `nullcontext` (`torch.cuda.stream(None)` was falling through `torch.mps.current_device()` which doesn't exist) — unlocks `generate_batch` + compile on MPS.
10. CB compile defaults: `reduce-overhead` on MPS (was `max-autotune-no-cudagraphs`, autotuning Triton kernels that don't exist on Metal).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45977). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, ggml