Unrecognized tensor type ID: AutocastCUDA
Author: fancyerii
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 2
        - gpu_ids: 0
        - main_process_ip: 10.8.0.7
        - main_process_port: 29500
        - rdzv_backend: c10d
        - same_network: False
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

I have similar problem when I try [this tutorial](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing#scrollTo=vT0XjNc2jYKy).

  File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/peft/tuners/lora.py", line 1255, in forward
    result = self.quant_linear_module(x)
  File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/auto_gptq/nn_modules/qlinear/qlinear_cuda_old.py", line 221, in forward
    self.autogptq_cuda.vecquant4matmul_old(x, self.qweight, out, self.scales.float(), self.qzeros, self.group_size)
RuntimeError: Unrecognized tensor type ID: AutocastCUDA

Hardware details
Nvidia A100 40GB

Software
ubuntu 20.04 torch=2.1.0+cu118 auto-gptq=0.4.2+cu117 accelerate=0.23.0

I have found related issue in [here](https://github.com/PanQiWei/AutoGPTQ/issues/374) but found no answer.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing#scrollTo=vT0XjNc2jYKy

### Expected behavior

no error.

Comments:
cc @younesbelkada 


> ### System Info
> * `transformers` version: 4.34.1
> * Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
> * Python version: 3.9.18
> * Huggingface_hub version: 0.17.3
> * Safetensors version: 0.4.0
> * Accelerate version: 0.23.0
> * Accelerate config:    - compute_environment: LOCAL_MACHINE
>   - distributed_type: MULTI_GPU
>   - mixed_precision: fp16
>   - use_cpu: False
>   - debug: False
>   - num_processes: 2
>   - machine_rank: 0
>   - num_machines: 2
>   - gpu_ids: 0
>   - main_process_ip: 10.8.0.7
>   - main_process_port: 29500
>   - rdzv_backend: c10d
>   - same_network: False
>   - main_training_function: main
>   - downcast_bf16: no
>   - tpu_use_cluster: False
>   - tpu_use_sudo: False
>   - tpu_env: []
> * PyTorch version (GPU?): 2.1.0+cu118 (True)
> * Tensorflow version (GPU?): not installed (NA)
> * Flax version (CPU?/GPU?/TPU?): not installed (NA)
> * Jax version: not installed
> * JaxLib version: not installed
> * Using GPU in script?:
> * Using distributed or parallel set-up in script?:
> 
> ### Who can help?
> I have similar problem when I try [this tutorial](https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing#scrollTo=vT0XjNc2jYKy).
> 
> File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/peft/tuners/lora.py", line 1255, in forward result = self.quant_linear_module(x) File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/lili/miniconda3/envs/py39_torch21/lib/python3.9/site-packages/auto_gptq/nn_modules/qlinear/qlinear_cuda_old.py", line 221, in forward self.autogptq_cuda.vecquant4matmul_old(x, self.qweight, out, self.scales.float(), self.qzeros, self.group_size) RuntimeError: Unrecognized tensor type ID: AutocastCUDA
> 
> Hardware details Nvidia A100 40GB
> 
> Software ubuntu 20.04 torch=2.1.0+cu118 auto-gptq=0.4.2+cu117 accelerate=0.23.0
> 
> I have found related issue in [here](https://github.com/PanQiWei/AutoGPTQ/issues/374) but found no answer.
> 
> ### Information
> * [ ]  The official example scripts
> * [ ]  My own modified scripts
> 
> ### Tasks
> * [ ]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
> * [ ]  My own task or dataset (give details below)
> 
> ### Reproduction
> run https://colab.research.google.com/drive/1_TIrmuKOFhuRRiTWN94iLKUFu6ZX4ceb?usp=sharing#scrollTo=vT0XjNc2jYKy
> 
> ### Expected behavior
> no error.

@fancyerii Not a good solution, but can you try downgrading the torch version to 2.0.1+cu118

Also seems to be an autoGPTQ issue rather than a transformers one! 

Closing as duplicate of https://github.com/PanQiWei/AutoGPTQ/issues/374