GGUF interaction with Transformers using AutoModel Class
Author: Abdullah-kwl
Pull request: no
Labels: name: GGUF

### Feature request

https://huggingface.co/docs/transformers/main/en/gguf
in above documentation it shows that it loads the gguf model and provided the simple example

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)

but when I run the code it shows the error :
OSError:  TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack.
and some time show your transformer may be not updated 

even after updating transformer it shows the same error not loading  gguf model , add support to load gguf model




### Motivation

it will solve to load gguf model with out the help of other library such as llama.cpp and ollama

### Your contribution

I do not have a complete implementation in mind, but I suggest to start with previous method which mention in https://huggingface.co/docs/transformers/main/en/gguf

Comments:
Hi @Abdullah-kwl 
Thanks for the issue ! Can you make sure you have the latest transformers installed ? `pip install -U transformers`

yes, I am using an updated transformer library.
in collab I am using the same command to download transformers.
pip install -U transformers

Thanks @Abdullah-kwl , will try to repro and report back here

Hi @Abdullah-kwl 
I ran successfully the code on fresh new Google Colab env: https://colab.research.google.com/drive/1rfJZp3DsbavH6IFo-rXUDNvqwFJVIbOb?usp=sharing 
Note we do run a bunch of tests: https://github.com/huggingface/transformers/blob/main/tests/quantization/ggml/test_ggml.py and they do all pass on our end as of today ! 

@younesbelkada yes i again run it now its working there was some dependencies conflict with other libraries but now its running.

but now I am facing problem that my session crashed after using all available RAM, I think its loading model on ram but when I use llama-cpp-python it does not load model on ram and we can easily inference with larger models even more then 7B.
 is there is a way that my session doe not crash due to ram usage.

try out using this model:

model_id = "TheBloke/WestLake-7B-v2-GGUF"
filename = "westlake-7b-v2.Q2_K.gguf"
model = AutoModel.from_pretrained(model_id, gguf_file=filename)
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)

Hello, I have a general question about using GGUF models with AutoModel. 

Is there any difference in the implementation if compared to the [ctransformers](https://github.com/marella/ctransformers) library?
Until now, I used GGUF models in Python with ctransformers, and it is really fast in generating responses. Using GGUF from AutoModel, instead, significantly slow down the inference time. Any clue about this? I am not sure if this transformers compatibility with GGUF models is intended for the the same usage as for ctransformers or llama.cpp or I am doing something wrong. 
Also, is there any way to set up "context_length" and "gpu_layers" offloading from AutoTokenizer?

Thanks!

cc @SunMarc 

Gentle ping @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @Abdullah-kwl, try passing `device_map = "auto"`, so that we make sure to load the model on all available devices ! LMK if this fixes your issue. 

> Hello, I have a general question about using GGUF models with AutoModel.
> 
> Is there any difference in the implementation if compared to the [ctransformers](https://github.com/marella/ctransformers) library?
> Until now, I used GGUF models in Python with ctransformers, and it is really fast in generating responses. Using GGUF from AutoModel, instead, significantly slow down the inference time. Any clue about this? I am not sure if this transformers compatibility with GGUF models is intended for the the same usage as for ctransformers or llama.cpp or I am doing something wrong.
> Also, is there any way to set up "context_length" and "gpu_layers" offloading from AutoTokenizer?
> 
> Thanks!

Hi @Devy99, the goal of the gguf implementation in transformers is to allow users to finetune their quantized model, then convert it back to gguf so that it can run on llama.cpp. 

Clear, thank you! 😁

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing it since the issue has been fixed !