Loading GGUF files support
Author: LysandreJik
Pull request: yes

This PR offers the ability to load `.gguf` files within `transformers`, dequantizing them to float32. 
Doing so enables further training on the GGUF files, before converting them back in the GGUF format for usage in the GGML ecosystem.

We enable this through the `from_gguf` keyword argument of the `from_pretrained` methods of configurations, tokenizers, and PyTorch models. Here is an example of the API:

```py
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
```

> **Note**
> This is still very experimental and we don't expect all files to work seamlessly. We'll work on improving the current implementation and welcome PRs that help us do so.

### Supported quantization types

The initial supported quantization types are decided according to the popular quantized files that have been shared
on the Hub.

- F32
- Q2_K
- Q3_K
- Q4_0
- Q4_K
- Q5_K
- Q6_K
- Q8_0

We take example from, and credit the excellent [99991/pygguf](https://github.com/99991/pygguf) Python parser to dequantize the weights.

### Supported model architectures

For now the supported model architectures are the architectures that have been very popular on the Hub, namely:

- LLaMa
- Mistral

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30391). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks both for the extensive review ! 🚀 

is this correct, or in-progress in v4.41-release?

```
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "NousResearch/Meta-Llama-3-8B-GGUF"
filename = "Meta-Llama-3-8B-Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

print(model)
```

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 6
      3 model_id = "NousResearch/Meta-Llama-3-8B-GGUF"
      4 filename = "Meta-Llama-3-8B-Q4_K_M.gguf"
----> 6 tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
      7 model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
      9 print(model)

File ~/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:899, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    896 tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
    898 if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
--> 899     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    900 else:
    901     if tokenizer_class_py is not None:

File ~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2110, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)
   2107     else:
   2108         logger.info(f"loading file {file_path} from cache at {resolved_vocab_files[file_id]}")
-> 2110 return cls._from_pretrained(
   2111     resolved_vocab_files,
   2112     pretrained_model_name_or_path,
   2113     init_configuration,
   2114     *init_inputs,
   2115     token=token,
   2116     cache_dir=cache_dir,
   2117     local_files_only=local_files_only,
   2118     _commit_hash=commit_hash,
   2119     _is_local=is_local,
   2120     trust_remote_code=trust_remote_code,
   2121     **kwargs,
   2122 )

File ~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2336, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, trust_remote_code, *init_inputs, **kwargs)
   2334 # Instantiate the tokenizer.
   2335 try:
-> 2336     tokenizer = cls(*init_inputs, **init_kwargs)
   2337 except OSError:
   2338     raise OSError(
   2339         "Unable to load vocabulary from file. "
   2340         "Please check that the provided vocabulary is accessible and not corrupted."
   2341     )

File ~/.local/lib/python3.10/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py:100, in GPT2TokenizerFast.__init__(self, vocab_file, merges_file, tokenizer_file, unk_token, bos_token, eos_token, add_prefix_space, **kwargs)
     89 def __init__(
     90     self,
     91     vocab_file=None,
   (...)
     98     **kwargs,
     99 ):
--> 100     super().__init__(
    101         vocab_file,
    102         merges_file,
    103         tokenizer_file=tokenizer_file,
    104         unk_token=unk_token,
    105         bos_token=bos_token,
    106         eos_token=eos_token,
    107         add_prefix_space=add_prefix_space,
    108         **kwargs,
    109     )
    111     self.add_bos_token = kwargs.pop("add_bos_token", False)
    113     pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())

File ~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py:120, in PreTrainedTokenizerFast.__init__(self, *args, **kwargs)
    117     fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
    118 elif gguf_file is not None:
    119     # We need to convert a slow tokenizer to build the backend
--> 120     tokenizer_dict = load_gguf_checkpoint(kwargs.get("vocab_file"))["tokenizer"]
    121     fast_tokenizer = convert_gguf_tokenizer(tokenizer_dict)
    122 elif self.slow_tokenizer_class is not None:
    123     # We need to create and convert a slow tokenizer to build the backend

File ~/.local/lib/python3.10/site-packages/transformers/modeling_gguf_pytorch_utils.py:81, in load_gguf_checkpoint(gguf_checkpoint_path, return_tensors)
     75     logger.error(
     76         "Loading a GGUF checkpoint in PyTorch, requires both PyTorch and GGUF to be installed. Please see "
     77         "https://pytorch.org/ and https://github.com/ggerganov/llama.cpp/tree/master/gguf-py for installation instructions."
     78     )
     79     raise
---> 81 reader = GGUFReader(gguf_checkpoint_path)
     82 fields = reader.fields
     83 reader_keys = list(fields.keys())

File ~/.local/lib/python3.10/site-packages/gguf/gguf_reader.py:85, in GGUFReader.__init__(self, path, mode)
     84 def __init__(self, path: os.PathLike[str] | str, mode: Literal['r' | 'r+' | 'c'] = 'r'):
---> 85     self.data = np.memmap(path, mode = mode)
     86     offs = 0
     87     if self._get(offs, np.uint32, override_order = '<')[0] != GGUF_MAGIC:

File /usr/lib/python3/dist-packages/numpy/core/memmap.py:228, in memmap.__new__(subtype, filename, dtype, mode, offset, shape, order)
    226     f_ctx = nullcontext(filename)
    227 else:
--> 228     f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
    230 with f_ctx as fid:
    231     fid.seek(0, 2)

TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Hi @brandon-lockaby !
I think that GGUF file is broken: https://huggingface.co/NousResearch/Meta-Llama-3-8B-GGUF/discussions/1 - can you try to freshly convert llama-3 using this Space: https://huggingface.co/spaces/ggml-org/gguf-my-repo ?

@younesbelkada 

Same error. Created and loaded from this repo https://huggingface.co/brandonglockaby/Meta-Llama-3-8B-Q4_K_M-GGUF

I should point out, previous attempts are ggufs that work correctly with current releases of llama.cpp and llama-cpp-python

Indeed I was able to repro, this is because the tokenizer is registered as `gpt2` tokenizer, will hhave a look and provide a fix !

@brandon-lockaby - https://github.com/huggingface/transformers/pull/31175 has been merged and might include a fix for the issue you are facing, can you try to re-run the snippet using transformers main branch?

@younesbelkada 

```
pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers
<snip>
Successfully uninstalled transformers-4.41.2
```

Same error related to tokenizer filename, produced with updated repo from gguf-my-repo as well as a gguf from my storage

Hi @brandon-lockaby 
Please see: https://github.com/huggingface/transformers/pull/31358 for the final fix, let me know if that fixes your issue. It fixes the same issue I had locally

> Hi @brandon-lockaby Please see: #31358 for the final fix, let me know if that fixes your issue. It fixes the same issue I had locally

Hi @younesbelkada,

I have tried #31358, and now the tokenizer can be loaded successfully. However, when I attempt to load the GGUF model, an OSError occurs:

```
OSError: QuantFactory/Meta-Llama-3-8B-GGUF does not appear to have a file named config.json. Checkout 'https://huggingface.co/QuantFactory/Meta-Llama-3-8B-GGUF/tree/main' for available files.
```

Here is my code:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig, LlamaForCausalLM

model_id = "QuantFactory/Meta-Llama-3-8B-GGUF"
filename = "Meta-Llama-3-8B.Q4_K_M.gguf"
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = LlamaForCausalLM.from_pretrained(model_id, gguf_file=filename)
```

Many GGUF models on Hugging Face do not have a config.json. So, I tried to load the config from the raw Meta-Llama-3-8B:
```python
config = AutoConfig.from_pretrained("meta-llama/Meta-Llama-3-8B")
model = LlamaForCausalLM.from_pretrained(model_id, gguf_file=filename, config=config)
```
This approach works, but I think this is not an elegant solution. Perhaps more modifications are needed here.

Thank you for your contribution.


Hi @Lin-xs 
Thanks a lot! hmmm indeed there might be a bug when not using autoclasses, can you try to load the model with `AutoModelForCausalLM` instead of `LlamaModelForCausalLM`

> Hi @Lin-xs Thanks a lot! hmmm indeed there might be a bug when not using autoclasses, can you try to load the model with `AutoModelForCausalLM` instead of `LlamaModelForCausalLM`

Thank you, this approach works.

Hi @younesbelkada @Isotr0py ,

I encountered a bug when trying to use `AutoModelForCausalLM` to load the `QuantFactory/Qwen2-7B-GGUF` model. Here is the code I used:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "QuantFactory/Qwen2-7B-GGUF"
filename = "Qwen2-7B.Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
```

The error message I received is:
```
File ~/miniconda3/envs/llama3/lib/python3.11/site-packages/accelerate/utils/modeling.py:358, in set_module_tensor_to_device(module, tensor_name, device, value, dtype, fp16_statistics, tied_params_map)
    356 if value is not None:
    357     if old_value.shape != value.shape:
--> 358         raise ValueError(
    359             f'Trying to set a tensor of shape {value.shape} in \"{tensor_name}\" (which has shape {old_value.shape}), this looks incorrect.'
    360         )
    362     if dtype is None:
    363         # For compatibility with PyTorch load_state_dict which converts state dict dtype to existing dtype in model
    364         value = value.to(old_value.dtype)

ValueError: Trying to set a tensor of shape torch.Size([152064, 3584]) in \"weight\" (which has shape torch.Size([151936, 3584])), this looks incorrect.
```

The same error occurs when I try to load `Qwen/Qwen2-7B-Instruct-GGUF`. It seems that `151936` is the vocab size for Qwen1.5 rather than Qwen2. In [test_ggml.py](https://github.com/huggingface/transformers/blob/eed9ed679878ada2f6d2eefccdbda368cabc88b1/tests/quantization/ggml/test_ggml.py#L45), the attribute `q4_0_qwen2_model_id` is set to `"qwen1_5-0_5b-chat-q4_0.gguf"`, which might cause the test to pass incorrectly.

Could you please take a look at it?

Thanks!

I think probably this is because the default vocab_size of `Qwen2Config` is set to `151936` in [configuration_qwen2.py](https://github.com/huggingface/transformers/blob/9af1b6a80adbac906ba770d23ddf95a147f2f0a0/src/transformers/models/qwen2/configuration_qwen2.py#L96) and the config loaded from Qwen2 gguf file do not have `"vocab_size"`:

```python
from transformers.modeling_gguf_pytorch_utils import load_gguf_checkpoint
from transformers.utils import cached_file

model_id = "Qwen/Qwen2-7B-Instruct-GGUF"
filename = "qwen2-7b-instruct-q2_k.gguf"

gguf_path = cached_file(model_id, filename,)
config_dict = load_gguf_checkpoint(gguf_path, return_tensors=False)["config"]
print(config_dict)
```
the output is
```
{'model_type': 'qwen2',
 '_model_name_or_path': 'qwen2-7b-instruct',
 'num_hidden_layers': 28,
 'max_position_embeddings': 32768,
 'hidden_size': 3584,
 'intermediate_size': 18944,
 'num_attention_heads': 28,
 'num_key_value_heads': 4,
 'rope_theta': 1000000.0,
 'rms_norm_eps': 9.999999974752427e-07,
 'eos_token_id': 151645,
 'pad_token_id': 151643,
 'bos_token_id': 151643}
```

Hi @younesbelkada 

very appreciate the effort to easily use GGUF models w the transformers library :)
just to understand -- does the current functionality convert the weights back to FP 32 and therefore "reverse" the entire quantization scheme when loading the model (instead of dequantizing on the "fly") ?

e.g. when i do the following

```
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q4_K_S.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

print(model.model.layers[0].self_attn.k_proj.weight.dtype)
```
i get:
torch.float32

cc @SunMarc 

That's right ! The goal of this feature is to let users load their gguf in transformers so that they can fine-tune them before reconverting them to gguf format ! 

ok got it! thank you for confirming :)!!