GGUF and interaction with Transformers
Source: transformers

# GGUF and interaction with TransformersThe GGUF file format is used to store models for inference with [GGML](https://github.com/ggerganov/ggml) and other libraries that depend on it, like the very popular [llama.cpp](https://github.com/ggerganov/llama.cpp) or [whisper.cpp](https://github.com/ggerganov/whisper.cpp).It is a file format [supported by the Hugging Face Hub](https://huggingface.co/docs/hub/en/gguf) with features allowing for quick inspection of tensors and metadata within the file.This file format is designed as a "single-file-format" where a single file usually contains both the configurationattributes, the tokenizer vocabulary and other attributes, as well as all tensors to be loaded in the model. Thesefiles come in different formats according to the quantization type of the file. We briefly go over some of them[here](https://huggingface.co/docs/hub/en/gguf#quantization-types).## Support within TransformersWe have added the ability to load `gguf` files within `transformers` in order to offer further training/fine-tuningcapabilities to gguf models, before converting back those models to `gguf` to use within the `ggml` ecosystem. Whenloading a model, we first dequantize it to fp32, before loading the weights to be used in PyTorch.> [!NOTE]> The support is still very exploratory and we welcome contributions in order to solidify it across quantization types> and model architectures.For now, here are the supported model architectures and quantization types:### Supported quantization typesThe initial supported quantization types are decided according to the popular quantized files that have been sharedon the Hub.- F32- Q2_K- Q3_K- Q4_0- Q4_K- Q5_K- Q6_K- Q8_0We take example from the excellent [99991/pygguf](https://github.com/99991/pygguf) Python parser to dequantize the weights.### Supported model architecturesFor now the supported model architectures are the architectures that have been very popular on the Hub, namely:- LLaMa- Mistral- Qwen2## Example usageIn order to load `gguf` files in `transformers`, you should specify the `gguf_file` argument to the `from_pretrained`methods of both tokenizers and models. Here is how one would load a tokenizer and a model, which can be loadedfrom the exact same file:```pyfrom transformers import AutoTokenizer, AutoModelForCausalLMmodel_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)```Now you have access to the full, unquantized version of the model in the PyTorch ecosystem, where you can combine itwith a plethora of other tools.In order to convert back to a `gguf` file, we recommend using the [`convert-hf-to-gguf.py` file](https://github.com/ggerganov/llama.cpp/blob/master/convert-hf-to-gguf.py) from llama.cpp.Here's how you would complete the script above to save the model and export it back to `gguf`:```pytokenizer.save_pretrained('directory')model.save_pretrained('directory')!python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}```