New update breaks T5, gpt2, opt models (probably all models actually) if bitsandbytes is installed
Author: ViktorThink
Pull request: no
Labels: name: bug

### System Info

It seems like a new update in the repo is causing an error for all models tested, including all the T5, gpt2 and opt models.

The error only occurs if bitsandbytes is installed, I tried an earlier version of bitesandbytes and same problem occured.

I made a colab to showcase it: https://colab.research.google.com/drive/1TSMLP3oPkAb-sBL_9l9KmXtRpP314Axc?usp=sharing

It must have been an update made in the past few hours, since code I used earlier today suddenly raised this error:

```
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1030, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 36, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/modeling_utils.py", line 88, in <module>
    from .utils.bitsandbytes import get_key_to_not_convert, replace_8bit_linear, set_module_8bit_tensor_to_device
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/utils/bitsandbytes.py", line 10, in <module>
    import bitsandbytes as bnb
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 4, in <module>
    import bitsandbytes.functional as F
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/functional.py", line 14, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 41, in <module>
    lib = CUDALibrary_Singleton.get_instance().lib
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 37, in get_instance
    cls._instance.initialize()
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 15, in initialize
    binary_name = evaluate_cuda_setup()
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 136, in evaluate_cuda_setup
    cc = get_compute_capability(cuda)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 112, in get_compute_capability
    return ccs[-1]
IndexError: list index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 462, in from_pretrained
    model_class = _get_model_class(config, cls._model_mapping)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 359, in _get_model_class
    supported_models = model_mapping[type(config)]
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 583, in __getitem__
    return self._load_attr_from_module(model_type, model_name)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 597, in _load_attr_from_module
    return getattribute_from_module(self._modules[module_name], attr)
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 553, in getattribute_from_module
    if hasattr(module, attr):
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1020, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/home/ec2-user/anaconda3/envs/pytorch_p38/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1032, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.t5.modeling_t5 because of the following error (look up to see its traceback):
list index out of range
```

### Who can help?

@younesbelkada
@TimDettmers


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

!pip install  git+https://github.com/huggingface/transformers.git
!pip install bitsandbytes==0.32.1

from transformers import AutoModel
model = AutoModel.from_pretrained("gpt2")

### Expected behavior

It should load the model, but it never does.

Comments:
Hi @ViktorThink . Could you try if the following version works?

`bitsandbytes==0.31.8` or `bitsandbytes==0.31.5`

Oh, I got what was the issue now...

I had bitsandbytes installed on an instance with no cuda device, and that raised the error.

Thank you very much for the quick reply! Highly appreciated!

Thanks for pointing out this issue @ViktorThink !
This should be fixed in https://github.com/huggingface/transformers/pull/18859 that has been merged recently 💪