Refacto GGUF weight conversion
Author: ArthurZucker
Pull request: yes
Labels: name: GGUF

# What does this PR do?

Refactor GGUF loading to use our weight-conversion pipeline + swap the slow NumPy dequant for a pure-PyTorch port that also runs on MPS / CUDA (vendored from diffusers / existing places)

## 1. Use the weight-conversion framework

- `_GGUF_ARCH_CONVERTERS` is now a flat per-`model_type` list of `WeightRenaming` chains plus `WeightConverter`
- `GGUFQuantizer.update_weight_conversions` injects `GGUFDequantize` as the first op of every `WeightConverter`, like `Fp8Quantizer.update_weight_conversions` injects `Fp8Dequantize`.
- Per-`model_type` coverage matches the legacy loader: `llama`, `mistral`, `phi3`, `cohere`, `qwen2`, `qwen3`, `deci`, `nemotron`, `gemma2`, `gemma3`, `gemma3_text`, `bloom`, `gpt2`, `mamba`, `lfm2`, `falcon`, `stablelm`, `starcoder2`, `qwen2_moe`, `qwen3_moe`, `minimax_m2`, `gpt_oss`, `t5`, `t5encoder`, `umt5`.


## 2. Fast pure-PyTorch dequant kernels
Uses a port of city96 / ComfyUI-GGUF (the same kernels diffusers ships in its GGUF quantizer): `src/transformers/integrations/gguf_dequant.py`. Coverage: `Q4_0/Q4_1`, `Q5_0/Q5_1`, `Q8_0`, `Q2_K…Q6_K`, `IQ4_NL`, `IQ4_XS`, `BF16`, `F16`, `F32`.


## End-to-end loading: this PR vs `main`

Wall-clock for `AutoModelForCausalLM.from_pretrained(..., gguf_file=..., device_map=device)` on Apple Silicon (96 GB unified memory).

| Model | Device | main | This PR | Speedup |
|-------|--------|-----:|--------:|--------:|
| TinyLlama-1.1B Q4_K_M | CPU | 8.95 s | 9.36 s | 0.96× |
| TinyLlama-1.1B Q4_K_M | MPS | 7.79 s | **3.86 s** | **2.02×** |
| Qwen2.5-0.5B Q4_0 | CPU | 11.88 s | 11.26 s | 1.06× |
| Qwen2.5-0.5B Q4_0 | MPS | 12.02 s | **9.13 s** | **1.32×** |
| Llama-3.2-3B Q4_K_M | CPU | 23.61 s | 23.98 s | 0.98× |
| Llama-3.2-3B Q4_K_M | MPS | 21.10 s | **12.15 s** | **1.74×** |
| Gemma-2-2B Q4_K_M | CPU | 26.22 s | **12.54 s** | **2.09×** |
| Gemma-2-2B Q4_K_M | MPS | 16.23 s | 15.88 s | 1.02× |
| Gemma-3-4B Q4_0 | CPU | 24.20 s | 26.73 s | 0.91× |
| Gemma-3-4B Q4_0 | MPS | 21.77 s | **13.30 s** | **1.64×** |
| Qwen1.5-MoE-A2.7B Q4_K_M | CPU | 50.39 s | **18.89 s** | **2.67×** |
| Qwen1.5-MoE-A2.7B Q4_K_M | MPS | 147.65 s | **18.53 s** | **🚀 7.97×** |


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_44794). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: ggml

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/25789898369)

This comment contains `run-slow`, running the specified jobs: 

models: []
quantizations: ["quantization/ggml"]

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/25789898369)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [b1529134](https://github.com/huggingface/transformers/commit/b1529134e9fd7a87588fa51a557c8cce2bc9aca1) | workflow commit (merge commit) |
| PR | [67de05d9](https://github.com/huggingface/transformers/commit/67de05d9b27b64323c504ebe8bf5524c501c9d1a) | branch commit (from PR) |
| main | [8c524a84](https://github.com/huggingface/transformers/commit/8c524a84eeba0fff7ea39af8dda874594b89b2e3) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: ggml

Hi, how's it going with this PR? I've started implementing the on-demand GGUF quantizer as discussed in #40070 , which obviously overlaps with this PR. Currently my work is at https://github.com/woct0rdho/transformers/tree/gguf . I've rebased this PR over the latest main branch and I'd be happy to help finishing the remaining work in this PR. After this PR is merged, I'll open a few PRs for the GGUF quantizer.