RuntimeError: "rshift_cuda" not implemented for 'Half'
Author: davidray222
Pull request: no
Labels: name: bug

### System Info

Hardware details
CPU:Intel(R) Xeon(R) Silver 4410Y
GPU:NVIDIA RTX A6000
Software version
Version of relevant software such as operation system, cuda toolkit, python, auto-gptq, pytorch, transformers, accelerate, etc.
operation system:
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

cuda toolkit:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

Python 3.10.15

Name: auto_gptq
Version: 0.7.1

pytorch:2.2.1+cu121
transformer:4.35.0
accelerate:0.26.0

### Who can help?

https://huggingface.co/jp288881/slim-llm-llama-7b

@ArthurZucker @itazap @mueller

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://huggingface.co/jp288881/slim-llm-llama-7b

### Expected behavior

  I used the quantization method from Aaronhuang-778/SliM-LLM (https://github.com/Aaronhuang-778/SliM-LLM?tab=readme-ov-file) and wanted to load and use the model , but encountered the issue mentioned in the title.I would like to ask if any experts know how to resolve this issue. I have also uploaded my model to Hugging Face for reference.

"Traceback (most recent call last):
  File "/mnt/8tb_raid/david_model/SliM-LLM/AutoGPTQ/test.py", line 31, in <module>
    generated_ids = model.generate(**input_ids)
  File "/mnt/8tb_raid/david_model/SliM-LLM/AutoGPTQ/auto_gptq/modeling/_base.py", line 480, in generate
    return self.model.generate(**kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/generation/utils.py", line 3251, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1163, in forward
    outputs = self.model(
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 913, in forward
    layer_outputs = decoder_layer(
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 640, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 522, in forward
    query_states = self.q_proj(hidden_states)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/david/miniconda3/envs/slimllm6/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/8tb_raid/david_model/SliM-LLM/AutoGPTQ/auto_gptq/nn_modules/qlinear/qlinear_cuda.py", line 319, in forward
    zeros = torch.bitwise_right_shift(
RuntimeError: "rshift_cuda" not implemented for 'Half'
"

Comments:
This isn't really a Transformers issue - it seems like the problem originates in the external quantization library you're using!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.