Community contribution: Adding GGUF support for more architectures
Author: SunMarc
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### Feature request

Recently, we have added the ability to load `gguf` files within [transformers](https://huggingface.co/docs/hub/en/gguf).

<img src="https://github.com/user-attachments/assets/61df6455-6016-449e-a37f-9dfc7f918902" width="600">


The goal was to offer the possibility to users to further train/fine-tune their gguf models. 
<details>
<summary>See Workflow</summary>
1) Load gguf file in transformers: we dequantize the weights to fp32, then we load the weights to be used with PyTorch.

```py 
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
```
2) train/finetune

3) Convert the model back to gguf to use in the ggml ecosystem using [convert_hf_to_gguf](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py) script or using [gguf-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space if you pushed your model on the hub :
```py
tokenizer.save_pretrained('directory')
model.save_pretrained('directory')

!python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}
```
</details>

Let's try to add GGUF support for more architectures! Currently supported architectures are

- [x] Llama
- [x] Mistral
- [x] Qwen2

It would be great to add the support for more architectures such as

- [x] Phi3 https://github.com/huggingface/transformers/pull/31844
- [x] Qwen2Moe https://github.com/huggingface/transformers/pull/33264
- [x] Gemma2 
- [x] T5 https://github.com/huggingface/transformers/pull/33389
- [x] Falcon https://github.com/huggingface/transformers/pull/33437
- [x] Bloom https://github.com/huggingface/transformers/pull/33473
- [x] StableLM https://github.com/huggingface/transformers/pull/33793
- [x] gpt2 https://github.com/huggingface/transformers/pull/34044
- [x] starcoder2 https://github.com/huggingface/transformers/pull/34094
- [ ] llama4
- [ ] Deepseekv3 
- [ ] c4ai-command-a

... and many more (Feel free to suggest more architectures ! The model needs to integrated in transformers)

Adding this feature would require to follow the same protocol as in this [PR](https://github.com/huggingface/transformers/pull/31175/files) : 
1) Update `GGUF_TENSOR_MAPPING` and `GGUF_CONFIG_MAPPING` in order to map the tensor/config of the gguf file to the one on transformers. 
2) Create a `GGUFXXXConverter(XXXConverter)` class to convert the gguf tokenizer to a transformers one. 
3) Write tests


If you are interested to take up the challenge, comment below with the architecture name you want to integrate and open a PR! 

Once you open a PR, feel free to ping @SunMarc @LysandreJik @ArthurZucker for a review ! 

### Motivation

Support for more gguf models

### Your contribution

Reviewing PRs and possibly adding the support for more models

Comments:
@SunMarc I am going to take Qwen2Moe

@SunMarc I want to take Gemma2

@SunMarc May I suggest & take T5? Seems [GGUF version of T5 encoder](https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf) is highly used for getting along with FLUX.

@SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

> @SunMarc May I suggest & take T5? Seems [GGUF version of T5 encoder](https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf) is highly used for getting along with FLUX.

Added @junejae !

> @SunMarc Hello! Unless someone else is working on this model already, may I take MiniCPM-V?

Hi @010kim, thanks for the interest ! MiniCPM-V model relies on `trust_remote_code=True`, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending `trust_remote_code=True` to gguf support, so that the author of the model can add it himself ! 


> Hi @010kim, thanks for the interest ! MiniCPM-V model relies on `trust_remote_code=True`, so I don't think we can add this model for now with gguf support. We don't want to have code in transformers that relies on modeling files that are on the hub. I will think about extending `trust_remote_code=True` to gguf support, so that the author of the model can add it himself !

@SunMarc Thank you so much for your response. It also makes sense the author should work on it. What about Cohere? Can I take it?


Hi @SunMarc 👋🏻
May I work with CLIP model if nobody is working on it?

Hey @jungnerd ! The model you choose needs to be in the conversion script from hf to gguf. See the script [here](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py)

Hey @SunMarc 🙋‍♂️ 
I'd like to try my chance to contribute to this issue, can I take Falcon? 🦅 

Hi @SunMarc, I take bloom if nobody is working on it

Hi @SunMarc, I'd like to handle the work related to Codestrall :)

> Hey @jungnerd ! The model you choose needs to be in the conversion script from hf to gguf. See the script [here](https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py)

There is conversion script for clip model([clip.cpp](https://github.com/monatis/clip.cpp/blob/main/models/convert_hf_to_gguf.py)). Can I use this to contribute?


Hi @SunMarc,
I'm interested in this issue. Would it be okay if I worked on the BLIP model?

> Hi @SunMarc, I'm interested in this issue. Would it be okay if I worked on the BLIP model?

Hi @SunMarc, I'd like to work on the BLIP model, but after researching, I found that it might be challenging due to the Vision model structure. Would it be alright if I switched to working on the Smol model instead?

Hey @SunMarc 🤗 
Gonna continue with granite 🪨 

@SunMarc 
I checked the Smol model and confirmed that it's already functioning well without needing any further work. In the issue mentioned that supporting the Smol model would be beneficial, but is there any specific work required?

If not, I’ll proceed with switching to the dbrx model.

> @SunMarc
> I checked the Smol model and confirmed that it's already functioning well without needing any further work. In the issue mentioned that supporting the Smol model would be beneficial, but is there any specific work required?
> 
> If not, I’ll proceed with switching to the dbrx model.

Oh indeed, this is because it is a llama architecture. 

Hi @SunMarc! I am going to start working on StableLM model

Is any work being done on the Gemma2? If not, I would like to proceed with it!
@SunMarc @KingNish24 

Hi @SunMarc! I suppose GPT2 gguf is not supported yet, if this is a case, I'll take it

> Hi @SunMarc, I'd like to handle the work related to Codestrall :)

Codestrall's tokenizer was just llama tokenizer.
It looks like I don't need to handle codes of Codestrall

> @SunMarc Thank you so much for your response. It also makes sense the author should work on it. What about Cohere? Can I take it?

I went through the codes, and i was able to to load Cohere gguf model, but could not load the tokenizer. This is because Cohere slow tokenizer is not implemented in HuggingFace. (Only FastTokenizer is available for Cohere) Is there a way around to fix this? @SunMarc 

Hey @SunMarc! I"ll take Starcoder2 as next model

Hi @SunMarc! I am going to start working on Mamba

Are you still working on Gemma2? @yijun-lee @KingNish24 ? If not, is it possible for me to try working on it? Thank you! 

> Are you still working on Gemma2? @yijun-lee @KingNish24 ? If not, is it possible for me to try working on it? Thank you!

I’m running behind schedule, but I’m making progress! I’ll handle it. 

> I’m running behind schedule, but I’m making progress! I’ll handle it.

Glad to know! Then is it possible for me to try working on Nemotron? @SunMarc 

Could you please kindly check my PR @SunMarc? Thank you 
[Add Nemotron GGUF Loading Support](https://github.com/huggingface/transformers/pull/34725) 

anyone working on supporting DeepSeek V3?

Deekpseek V3 is not supported yet in transformers but it will be soon with this [PR](https://github.com/huggingface/transformers/pull/35926#issuecomment-2636398699) 

Thanks, it should be quite similar to DeepSeekV2. I have made some progress on it, though mainly on another direction. https://github.com/zinccat/gguf_offload

But it says Architecture qwen2 not supported
the model that I'm using:
gte-Qwen2-7B-instruct-Q4_K_M.gguf

Can we also get support for gemma3 please.
Who is able to do that?
Maybe @kinshuk1207 because you also added support for the original gemma in https://github.com/huggingface/transformers/pull/35887?

Can we also get support for cohere2 as well? or are there issues preventing that from happening?

Which model in particular ?  Please share the model card @huzhekun 

I am specifically personally looking at this model @SunMarc , any other info I should provide?
https://huggingface.co/bartowski/CohereForAI_c4ai-command-a-03-2025-GGUF

No thanks ! I'll add it above as a suggested architecture to add support for 

Would love to get llama4 support
```
GGUF model with architecture llama4 is not supported yet
```

llama4 would be really nice to have indeed, feel free to sumbit a PR if you are interested ! cc @Isotr0py 

Is it okay if I proceed with DeepSeek-V3 or C4AI-Command-R for this? @SunMarc 

Hi @SunMarc! I am going to start working on llama4

Is llama4 supported ?

Hi @SunMarc, can I add support to the Phi4 multimodel?

qwen2vl gguf support

please support this gguf model: 

https://huggingface.co/mradermacher/SkyCaptioner-V1-GGUF

Hi @SunMarc 

Thank you for all your work on expanding GGUF support in Transformers — it’s been great to see the progress! 

I was wondering if GGUF support could be extended to the **NAVER HyperCLOVAX models** (e.g. SEED-Text 0.5B/1.5B, SEED-Vision-3B, SEED-Think-14B).  
Some smaller SEED models already run well via community GGUF conversions, and official support would make workflows much smoother, especially for the reasoning and multimodal variants.

If this sounds valuable, I’d be glad to help or draft an initial PR. 

Hi @SunMarc, it looks like GGUF support for Llama 4 may be incomplete at the moment. I’ve tried loading some Llama 4 GGUF checkpoints and ran into issues.

I’d be happy to take a look and work on adding support for Llama4 here if this is still an open area.


Hi @SunMarc,
Can I work on Llama4 from the checklist? I have an implementation ready for this, please let me know if i can so i can push it and raise a PR for that.

Hi @SunMarc, I would like to add GGUF support for the OLMo-2 architecture.

I checked and the model is already in transformers, llama.cpp supports the olmo2 arch, and there are GGUF checkpoints on the Hub to test with (for example allenai/OLMo-2-1124-7B-Instruct-GGUF and the small allenai/OLMo-2-0425-1B-GGUF). It is not in ggml.py yet and I could not find an existing PR for it.

I plan to follow the same steps as PR #31175: add the config and tensor mappings, add a tokenizer converter, and add a test. Could I take this one? Thanks!

Hi @SunMarc! I'd like to take up cohere2 (c4ai-command-a-03-2025 / c4ai-command-r7b-12-2024) from the checklist — following up on the earlier request for it in this thread.

Quick summary of the approach, following the protocol from the issue description and recent arch additions like #44526:

Add the cohere2 entry to GGUF_CONFIG_MAPPING (tensor mapping should come via gguf-py's MODEL_ARCH.COHERE2 name map)
Wire up the tokenizer converter (Cohere GGUFs use BPE, so starting from GGUFGPTConverter)
Add integration tests using a Q2_K quant of Command R7B (small enough for CI), verifying generation and tokenizer parity against the original checkpoint
Will open a PR shortly!

Hi @SunMarc 👋

I'd like to claim **DeepSeek-V3** from the checklist and add GGUF loading support for it.

A bit of homework before starting:
- `deepseek_v3` is already integrated in `transformers`, so the model prerequisite is met.
- I don't see DeepSeek-V3 in `GGUF_CONFIG_MAPPING` / `GGUF_TENSOR_MAPPING` yet, and I couldn't find an open PR adding it (the last mention in this thread was ~May 2025 and didn't lead to a PR), so it looks unclaimed.
- There are GGUF checkpoints on the Hub to validate against (e.g. the `unsloth/DeepSeek-V3-GGUF` conversions), so I can add a loading test.

Planned approach, following the protocol in the issue description and prior additions (e.g. #31175, and the Qwen2MoE PR #33264 as an MoE reference):
1. Add `deepseek_v3` entries to `GGUF_CONFIG_MAPPING` and `GGUF_TENSOR_MAPPING` in `integrations/ggml.py`, handling the MoE expert tensors and the MLA (multi-head latent attention) projections, which differ from the dense-model mappings.
2. Handle the config/tokenizer conversion as needed.
3. Add a GGUF loading test.

Does this sound right, and is the architecture free to take? Happy to adjust based on your guidance before opening a PR. Thanks!

Hi @SunMarc, I'd like to add GGUF support for Phi-2. The phi model is already in transformers and llama.cpp uses the phi2 architecture. I have a working implementation that maps the phi2 config, registers the GPT2 BPE tokenizer converter, splits the fused attn-qkv projection into separate q/k/v, and derives partial rotary factor from rope.dimension_count. I'll open a PR shortly.