error loading facebook/opt-30b with text generation pipeline using 8bit mixed precision
Author: morrisalp
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.13.0a0+d0d6b1f (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@patrickvonplaten, @Narsil, @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Running the following on a system with one (NVIDIA A5000) GPU:
```
from transformers import pipeline
model = "facebook/opt-30b"
model_kwargs = {"device_map": "auto", "load_in_8bit": True}
generator = pipeline(task="text-generation", model=model, device=0, model_kwargs=model_kwargs)
```

yields error:
`ValueError: Could not load model facebook/opt-30b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.opt.modeling_opt.OPTForCausalLM'>).`


### Expected behavior

Should be able to create generator with no problem and generate text with `generator.__call__`.

The code works with no error when using smaller opt model checkpoints: "facebook/opt-2.7b", "facebook/opt-6.7b".

Can create model, tokenizer, and generate without pipeline using `AutoModelForCausalLM.from_pretrained(model, device_map="auto")` with `model="facebook/opt-30b"` despite the error message.

Comments:
If you can create your tokenizer and model, just send them to the pipeline function as a quick workaround :-)

@sgugger This actually leads to another error:

```
from transformers import AutoModelForCausalLM, AutoTokenizer
model = "facebook/opt-30b"
model_kwargs = {"device_map": "auto", "load_in_8bit": True}
m = AutoModelForCausalLM.from_pretrained(model, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)
generator = pipeline(task="text-generation", model=m, tokenizer=tokenizer, device=0, model_kwargs=model_kwargs)
```

Yields `NotImplementedError`:

```
File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:980, in Module.to.<locals>.convert(t)
    977 if convert_to_format is not None and t.dim() in (4, 5):
    978     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    979                 non_blocking, memory_format=convert_to_format)
--> 980 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

NotImplementedError: Cannot copy out of meta tensor; no data!
```

Please provide the full traceback, as we can't see what's happening otherwise especially since I can't reproduce locally on my side. cc @younesbelkada who might have better luck reproducing the bug!

hi @morrisalp 
Thanks for the heads up and for flagging the issue! 
Can you please try:

```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model = "facebook/opt-30b"

model_kwargs = {"device_map": "auto", "load_in_8bit": True}
m = AutoModelForCausalLM.from_pretrained(model, **model_kwargs)

tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)
generator = pipeline(task="text-generation", model=m, tokenizer=tokenizer)
```
No need to add `model_kwargs` and `device=0` in addition to what you have added ;) this should work! let us know here

Full traceback for first error (using pipeline only):

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [7], line 1
----> 1 generator = pipeline(task="text-generation", model=model, device=0, model_kwargs=model_kwargs)

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/__init__.py:727, in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    723 # Infer the framework from the model
    724 # Forced if framework already defined, inferred if it's None
    725 # Will load the correct model if possible
    726 model_classes = {"tf": targeted_task["tf"], "pt": targeted_task["pt"]}
--> 727 framework, model = infer_framework_load_model(
    728     model,
    729     model_classes=model_classes,
    730     config=config,
    731     framework=framework,
    732     task=task,
    733     **hub_kwargs,
    734     **model_kwargs,
    735 )
    737 model_config = model.config
    738 hub_kwargs["_commit_hash"] = model.config._commit_hash

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py:266, in infer_framework_load_model(model, config, model_classes, task, framework, **model_kwargs)
    263             continue
    265     if isinstance(model, str):
--> 266         raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.")
    268 framework = "tf" if model.__class__.__name__.startswith("TF") else "pt"
    269 return framework, model

ValueError: Could not load model facebook/opt-30b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.opt.modeling_opt.OPTForCausalLM'>).
```


Full traceback for second error (creating tokenizer and model & passing them to pipeline):

```
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
File <timed exec>:1

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/__init__.py:873, in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    870 if device is not None:
    871     kwargs["device"] = device
--> 873 return pipeline_class(model=model, framework=framework, task=task, **kwargs)

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/text_generation.py:49, in TextGenerationPipeline.__init__(self, *args, **kwargs)
     48 def __init__(self, *args, **kwargs):
---> 49     super().__init__(*args, **kwargs)
     50     self.check_model_type(
     51         TF_MODEL_FOR_CAUSAL_LM_MAPPING if self.framework == "tf" else MODEL_FOR_CAUSAL_LM_MAPPING
     52     )
     53     if "prefix" not in self._preprocess_params:
     54         # This is very specific. The logic is quite complex and needs to be done
     55         # as a "default".
     56         # It also defines both some preprocess_kwargs and generate_kwargs
     57         # which is why we cannot put them in their respective methods.

File /opt/conda/lib/python3.8/site-packages/transformers/pipelines/base.py:778, in Pipeline.__init__(self, model, tokenizer, feature_extractor, modelcard, framework, task, args_parser, device, binary_output, **kwargs)
    776 # Special handling
    777 if self.framework == "pt" and self.device.type != "cpu":
--> 778     self.model = self.model.to(self.device)
    780 # Update config with task specific parameters
    781 task_specific_params = self.model.config.task_specific_params

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:982, in Module.to(self, *args, **kwargs)
    978         return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    979                     non_blocking, memory_format=convert_to_format)
    980     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
--> 982 return self._apply(convert)

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:635, in Module._apply(self, fn)
    633 def _apply(self, fn):
    634     for module in self.children():
--> 635         module._apply(fn)
    637     def compute_should_use_set_data(tensor, tensor_applied):
    638         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    639             # If the new tensor has compatible tensor type as the existing tensor,
    640             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    645             # global flag to let the user control whether they want the future
    646             # behavior of overwriting the existing tensor or not.

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:635, in Module._apply(self, fn)
    633 def _apply(self, fn):
    634     for module in self.children():
--> 635         module._apply(fn)
    637     def compute_should_use_set_data(tensor, tensor_applied):
    638         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    639             # If the new tensor has compatible tensor type as the existing tensor,
    640             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    645             # global flag to let the user control whether they want the future
    646             # behavior of overwriting the existing tensor or not.

    [... skipping similar frames: Module._apply at line 635 (3 times)]

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:635, in Module._apply(self, fn)
    633 def _apply(self, fn):
    634     for module in self.children():
--> 635         module._apply(fn)
    637     def compute_should_use_set_data(tensor, tensor_applied):
    638         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    639             # If the new tensor has compatible tensor type as the existing tensor,
    640             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    645             # global flag to let the user control whether they want the future
    646             # behavior of overwriting the existing tensor or not.

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:658, in Module._apply(self, fn)
    654 # Tensors stored in modules are graph leaves, and we don't want to
    655 # track autograd history of `param_applied`, so we have to use
    656 # `with torch.no_grad():`
    657 with torch.no_grad():
--> 658     param_applied = fn(param)
    659 should_use_set_data = compute_should_use_set_data(param, param_applied)
    660 if should_use_set_data:

File /opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py:980, in Module.to.<locals>.convert(t)
    977 if convert_to_format is not None and t.dim() in (4, 5):
    978     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    979                 non_blocking, memory_format=convert_to_format)
--> 980 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

NotImplementedError: Cannot copy out of meta tensor; no data!
```

@younesbelkada That code gives me the following error, but if this is a GPU OOM error then that is progress :)

```
m = AutoModelForCausalLM.from_pretrained(model, **model_kwargs)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [5], line 1
----> 1 m = AutoModelForCausalLM.from_pretrained(model, **model_kwargs)

File /opt/conda/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py:463, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    461 elif type(config) in cls._model_mapping.keys():
    462     model_class = _get_model_class(config, cls._model_mapping)
--> 463     return model_class.from_pretrained(
    464         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    465     )
    466 raise ValueError(
    467     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    468     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    469 )

File /opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py:2280, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2276         device_map_without_lm_head = {
   2277             key: device_map[key] for key in device_map.keys() if key not in modules_to_not_convert
   2278         }
   2279         if "cpu" in device_map_without_lm_head.values() or "disk" in device_map_without_lm_head.values():
-> 2280             raise ValueError(
   2281                 """
   2282                 Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
   2283                 the quantized model. If you have set a value for `max_memory` you should increase that. To have
   2284                 an idea of the modules that are set on the CPU or RAM you can print model.hf_device_map.
   2285                 """
   2286             )
   2287         del device_map_without_lm_head
   2289 if from_tf:

ValueError: 
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you have set a value for `max_memory` you should increase that. To have
                        an idea of the modules that are set on the CPU or RAM you can print model.hf_device_map.
```

Nice! Yes this error is indeed due to the fact that your int8 model does not fit your available GPU memory. Could you share here what is the hardware you are using (with the avail GPU RAM)? thanks!

You can do something like `nvidia-smi` and post the output here

Sure, here are the specs (with nothing running currently):

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A5000    On   | 00000000:C3:00.0 Off |                  Off |
| 30%   21C    P5    29W / 230W |      2MiB / 24256MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
```

I see, the 30B parameters model needs at least around `30GB` to be loaded in 8-bit. So you'll need more GPU RAM to fit the model here sadly.
However, if you absolutely want to run it, there is hacky solution. You can do:
`pip install --upgrade git+https://github.com/younesbelkada/transformers@bnb_add_custom_map`
and run:
```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
model = "facebook/opt-30b"

device_map = {
    "model.decoder.embed_tokens": 0,
    "model.decoder.embed_positions": 0,
    "model.decoder.final_layer_norm": 0,
    "lm_head": 0,
    "model.decoder.layers.0": 0,
    "model.decoder.layers.1": 0,
    "model.decoder.layers.2": 0,
    "model.decoder.layers.3": 0,
    "model.decoder.layers.4": 0,
    "model.decoder.layers.5": 0,
    "model.decoder.layers.6": 0,
    "model.decoder.layers.7": 0,
    "model.decoder.layers.8": 0,
    "model.decoder.layers.9": 0,
    "model.decoder.layers.10": 0,
    "model.decoder.layers.11": 0,
    "model.decoder.layers.12": 0,
    "model.decoder.layers.13": 0,
    "model.decoder.layers.14": 0,
    "model.decoder.layers.15": 0,
    "model.decoder.layers.16": 0,
    "model.decoder.layers.17": 0,
    "model.decoder.layers.18": 0,
    "model.decoder.layers.19": 0,
    "model.decoder.layers.20": 0,
    "model.decoder.layers.21": 0,
    "model.decoder.layers.22": 0,
    "model.decoder.layers.23": 0,
    "model.decoder.layers.24": 0,
    "model.decoder.layers.25": 0,
    "model.decoder.layers.26": 0,
    "model.decoder.layers.27": 0,
    "model.decoder.layers.28": 0,
    "model.decoder.layers.29": 0,
    "model.decoder.layers.30": 0,
    "model.decoder.layers.31": 0,
    "model.decoder.layers.32": 0,
    "model.decoder.layers.33": 0,
    "model.decoder.layers.34": 0,
    "model.decoder.layers.35": 0,
    "model.decoder.layers.36": 0,
    "model.decoder.layers.37": 0,
    "model.decoder.layers.38": 0,
    "model.decoder.layers.39": 0,
    "model.decoder.layers.40": 0,
    "model.decoder.layers.41": 0,
    "model.decoder.layers.42": "cpu",
    "model.decoder.layers.43": "cpu",
    "model.decoder.layers.44": "cpu",
    "model.decoder.layers.45": "cpu",
    "model.decoder.layers.46": "cpu",
    "model.decoder.layers.47": "cpu",
}

model_kwargs = {"device_map": device_map, "load_in_8bit": True}
m = AutoModelForCausalLM.from_pretrained(model, **model_kwargs)

tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)
generator = pipeline(task="text-generation", model=m, tokenizer=tokenizer) 
```
But keep in mind that the layers that will be set on `cpu` will be kept in their native `dtype` and not converted in `int8`. Also this feature is not supported yet as the integration should done on `bitsandbytes` side, so you may encounter unexpected behaviours but you can always give it a try! 

Related: #19090

Thanks! I mainly wanted to see what the largest LLM I could fit on one of my GPUs would be using mixed precision, and I couldn't tell previously if the 30B model would be OOM due to the other errors...

I see now thanks a lot!
Closing this issue as I consider to be completed, don't hesitate to reopen it if you have more questions !