Transformer GGUF support philosophy / naive question
Author: luke14free
Pull request: no
Labels: name: Feature request

name: GGUF

Hey there, I am a huge user of both transformers and diffusers and really love the work of the teams at HF. However something is not entirely clear to me regarding the GGUF support by transformers. 

GGUF main idea is to be a format that allows to run big models on machines with limited capabilities.

With this in mind, in diffusers (or in comfyui and llama.cpp) I can load gguf files natively and they mostly just work using less vram than the original model. 

In transformers however while many models are supported in gguf format, they are loaded as gguf but then immediately dequantized back to fp16/32, which takes a lot of time and ultimately the same vram requirements of the full model. So I don't understand why someone would ever want to do it (at that point you are probably better off loading the full model and not wait for dequantization).

It feels like I am missing something very obvious here, so I wanted to ask for guidance to the community 🙏  

It's probably a stupid question but thanks a lot for taking the time to answer me :) 

Comments:
Hi @luke14free, this is a good question! For most people, you should load the original weights and not the GGUF, for exactly the reasons you described. However, loading the GGUF can still be useful; for example, you can test the performance of models using the GGUF weights you intend to deploy, or you can use it for benchmarking GGUF weights versus the originals.

In future, we may also add kernels that allow direct use of GGUF weight formats without dequantization, although this isn't a concrete plan yet!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, I've implemented a proof of concept to load GGUF in transformers with on-demand quantization, see https://github.com/woct0rdho/transformers-qwen3-moe-fused#lora-over-gguf . An example of inference is at [`example_infer_30b_a3b_gguf.py`](https://github.com/woct0rdho/transformers-qwen3-moe-fused/blob/master/example_infer_30b_a3b_gguf.py), and an example of training is at [`example_train_30b_a3b_gguf.py`](https://github.com/woct0rdho/transformers-qwen3-moe-fused/blob/master/example_train_30b_a3b_gguf.py)

For convenience I'm developing it in my transformers-qwen3-moe-fused repo, but it also works with many models that are not Qwen and not MoE.

I've implemented a new quantizer that works like `Bnb4BitHfQuantizer`. But currently I completely bypassed `AutoModel.from_pretrained` when loading the model.

@Rocketknight1 Do you have any idea how to help this get merged in transformers? I guess it takes some effort to sort through things in `AutoModel.from_pretrained` and properly treat GGUF as a quantizer rather than some special flags.

cc @sunmarc @mekkcyber maybe?

Wow nice repo @woct0rdho ! This is really interesting ! We definitely want to treat gguf as a quantizer and not dequantize everything as we do right now. Would you like to take a stab at implementing it ? Note that in v5, the gguf tokenizer is most likely broken (will work on fixing this), so it is better to just init the tokenizer from the original model for now. 

I guess it can be done in the following phases:
1. Parse model config from GGUF. It's big work to support all existing models, so we may start with a few popular models at present
2. Implement GGUF quantizer and load the model. We need:
   - GGUF dequantization in torch. Let's see if we can add it in https://github.com/ggml-org/llama.cpp/pull/25681 . If not, then we need to vendor it
   - Refactor `GGUFQuantizedTensor` to have consistent semantics with bnb `Params4bit` and `Int8Params`
   - `GGUFLinear` and `GGUFEmbedding` that have consistent semantics with bnb `Linear4bit`
   - It should work transparently with PEFT and TRL
3. Replace the old loading mechanism in `AutoModel.from_pretrained` with the quantizer
4. Consider how to deal with MoE. It should work with the existing mechanism of switchable MoE kernels, and work transparently with PEFT
5. Parse tokenizer config from GGUF and load the tokenizer. Before this point we use the tokenizer from the safetensors model
6. Optimize the speed using fused dequant-matmul kernels, like bitsandbytes does. The kernels in libggml already support multiple hardware and we just need to add torch bindings. We may distribute them using HuggingFace Kernels (which is easier to me than how bitsandbytes bundles an army of binaries)
7. Load mmproj

The main motivation for me is to open more possibilities of low-VRAM LoRA training, which is not covered by llama.cpp . Compared to bitsandbytes, GGUF is more actively supporting new model types such as MoE, linear attentions, sparse attentions, and new quant types such as 1-bit quants. Given that transformers is the basis of most LLM training frameworks like Unsloth and Axolotl, it can help a lot.