load qwen2-72b-instruct sft awq q4_0 gguf ValueError: Trying to set a tensor of shape torch.Size
Author: liduang
Pull request: no

### System Info

python3.11.9
transformers 4.42.0.dev0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Python 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> model_id = "/data/llm/Qwen/Qwen2-72B-Instruct"
>>> filename = "/data/llm/Qwen/Qwen2-72B-Instruct-SFT-AWQ-fp16-q4_0.gguf"
>>> tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
>>> model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)
Converting and de-quantizing GGUF tensors...:  99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 958/963 [14:25<00:24, Converting and de-quantizing GGUF tensors...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 959/963 [14:25<00:15, Converting and de-quantizing GGUF tensors...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍| 960/963 [14:28<00:10, Converting and de-quantizing GGUF tensors...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 961/963 [14:31<00:06, Converting and de-quantizing GGUF tensors...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 963/963 [14:31<00:00,  1.11it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hicon/LD/Qwen2-sft/transformers/src/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hicon/LD/Qwen2-sft/transformers/src/transformers/modeling_utils.py", line 3834, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hicon/LD/Qwen2-sft/transformers/src/transformers/modeling_utils.py", line 4223, in _load_pretrained_model
    error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hicon/LD/Qwen2-sft/transformers/src/transformers/modeling_utils.py", line 895, in _load_state_dict_into_meta_model
    set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
  File "/home/hicon/miniconda3/envs/qwen2/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 358, in set_module_tensor_to_device
    raise ValueError(
ValueError: Trying to set a tensor of shape torch.Size([151646, 8192]) in "weight" (which has shape torch.Size([151936, 8192])), this look incorrect.


### Expected behavior

TypeError: _BaseAutoModelClass.from_pretrained() missing 1 required positional argument: 'pretrained_model_name_or_path'


Comments:
cc @SunMarc @younesbelkada As it looks like a possible gguf issue 

Hi @liduang 
Thanks for the issue, can you share a model checkpoint on the hub we could look at ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.