to is not supported for `8-bit` models 
Author: lborcard
Pull request: no

### System Info

Hi,

I am using a Llama model and wanted to add to pipeline class but it throws me an error when building the pipeline class.
Does anyone have a solution to this?
thank you!


@Narsil 

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction




> ## Model
> model = AutoModelForCausalLM.from_pretrained(
>   model_name,
>   device_map='auto',
>   load_in_8bit=True,
>   max_memory=max_memory)
> 
> ## llm class
> 
> class CustomLLM(LLM):
>     
>     pipeline = pipeline("text-generation",tokenizer = tokenizer, model=model, device="cuda:0")
> 
>     def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
>         prompt_length = len(prompt)
>         response = self.pipeline(prompt, max_new_tokens=num_output)[0]["generated_text"]
> 
>         # only return newly generated tokens
>         return response[prompt_length:]
> 
>     @property
>     def _identifying_params(self) -> Mapping[str, Any]:
>         return {"name_of_model": self.model_name}
> 
>     @property
>     def _llm_type(self) -> str:
>         return "custom"
> 
            " model has already been set to the correct devices and casted to the correct `dtype`."



### Expected behavior

1879 # Checks if the model has been loaded in 8-bit
1880 if getattr(self, "is_loaded_in_8bit", False):
-> 1881 raise ValueError(
1882 ".to is not supported for 8-bit models. Please use the model as it is, since the"
1883 " model has already been set to the correct devices and casted to the correct dtype."

Comments:
cc @younesbelkada 

hi @lborcard 
Indeed the `to` operation is not supported for 8bit models as users will most likely encounter unexpected behaviour.
What version of `transformers` are you using?
A fix has been introduced in https://github.com/huggingface/transformers/pull/21479 and has been documented [here](https://huggingface.co/docs/transformers/pipeline_tutorial#using-pipeline-on-large-models-with-accelerate) - everything should work fine if you use `transformers>=4.29.0`

Hi @younesbelkada ,

Thank you for your answer, I was using version 4.29 but I will try a newer version.
 have a good day

@lborcard 
can you try:
```python
pipeline = pipeline("text-generation",tokenizer = tokenizer, model=model, device=0)
```

I'm still getting this error on the latest version of the transformer. Any work around?

@mrhimanshu can you share an handy reproducible snippet?

I'm also still getting this error when using transformers==4.3.0.0 version. Anyone figured any work-around?

hi everyone,
thanks for raising up the issue, I would greatly appreciate if you could share a reproducible snippet as I can't do anything without it

Thanks @younesbelkada , the error is in the file 
/python3.8/site-packages/transformers/modeling_utils.py
 def half(self, *args):
        # Checks if the model has been loaded in 8-bit
        if getattr(self, "is_quantized", False):
            raise ValueError(
                "`.half()` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the"
                " model has already been casted to the correct `dtype`."
            )
        else:
            return super().half(*args)
            
            
  File "/python3.8/site-packages/transformers/modeling_utils.py", line 1907, in half
    raise ValueError(
ValueError: `.half()` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been casted to the correct `dtype`.           

Please check it out. 

Hi @22Mukesh22 
Thank you for your message, I think you are somehow calling `.half` in your script , can you share a handy small snippet to reproduce?


This .half only comes in picture when passing load_in_8bit=True , else if we remove this from the script , it gives memory error . 
Cuda out of memory , as I have P40 24 GB 4 GPU.   

I am usign the same script "https://github.com/Xirider/finetune-gpt2xl" to finetune the starcoder model.
 

Looking at the repo you shared I think that you are trying to use DeepSpeed + bitsandbytes and purely fine tune the entire model in 8bit or 4bit. This is not supported. 

You should look into PEFT library if you want to fine-tune the model in 8bit or 4bit to fine-tune adapters on top of the base model (which leads to the same results), some examples here: https://github.com/huggingface/peft/tree/main/examples/int8_training 

And the documentation is here: https://huggingface.co/docs/peft/index

Thanks a lot , I will try and update 

ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode. In order to use 8-bit models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism. Therefore you should not specify that you are under any distributed regime in your accelerate config.

@22Mukesh22 can you please update your `accelerate` version?
```bash
pip install --upgrade accelerate
```
Related: https://github.com/huggingface/accelerate/pull/1523

Okay Sure @younesbelkada , i will re run and update if issue still comes.

@younesbelkada still the issues remaisn same . I have upgraded accelerate . It doesn't works 

Hi @22Mukesh22
How do you run your script? can you share the accelerate config you are using? Also let's open a different ticket on accelerate for the issue you are facing and ping me there
Thanks! 

for everyone stumbling into this error, my solution was to use accelerate 0.20.3 and transformers 4.30.2 (not necceserally needed). With those versions the training started correctly.

> for everyone stumbling into this error, my solution was to use accelerate 0.20.3 and transformers 4.30.2 (not necceserally needed). With those versions the training started correctly.

Thanks working!

> for everyone stumbling into this error, my solution was to use accelerate 0.20.3 and transformers 4.30.2 (not necceserally needed). With those versions the training started correctly.

Thank you! This worked for me. I'll try and investigate what went wrong. For reference, this was the traceback:
```
File "finetune.py", line 552, in main
    model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 484, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 2937, in from_pretrained
    dispatch_model(model, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/big_modeling.py", line 391, in dispatch_model
    model.to(device)
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 1897, in to
    raise ValueError(
ValueError: `.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
```

still suffering this issue with accelerate 0.20.3 and transformers 4.30.2, getting "
ValueError: `.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.
"

add that i'm using the bnb_4bit, as follows
```
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
```


> 添加我正在使用bnb_4bit，如下所示
> 
> ```
> quant_config = BitsAndBytesConfig(
>     load_in_4bit=True,
>     bnb_4bit_use_double_quant=True,
>     bnb_4bit_quant_type="nf4",
>     bnb_4bit_compute_dtype=torch.bfloat16
> )
> ```

I also meet the problem when I set CUDA_VISIBLE_DEVICES="0"  in ,sh file. 
However , when I delete this command or I set CUDA_VISIBLE_DEVICES="0,1" or "0,1,2,3"  . It can work.
(But I want to save GPU memory and qlora paper say it can work on one GPU 

> > 添加我正在使用bnb_4bit，如下所示
> > ```
> > quant_config = BitsAndBytesConfig(
> >     load_in_4bit=True,
> >     bnb_4bit_use_double_quant=True,
> >     bnb_4bit_quant_type="nf4",
> >     bnb_4bit_compute_dtype=torch.bfloat16
> > )
> > ```
> 
> I also meet the problem when I set CUDA_VISIBLE_DEVICES="0" in ,sh file. However , when I delete this command or I set CUDA_VISIBLE_DEVICES="0,1" or "0,1,2,3" . It can work. (But I want to save GPU memory and qlora paper say it can work on one GPU



> for everyone stumbling into this error, my solution was to use accelerate 0.20.3 and transformers 4.30.2 (not necceserally needed). With those versions the training started correctly.

 accelerate 0.20.3 works on one GPU and mult GPU(<=4)

I encountered similar issue. I tried CUDA_VISIBLE_DEVICES=1,2,3. But 8-bit llama is automatically loaded to cuda:0.  and I cannot apply ".to('cuda:1') " which gives me ths error, 'to is not supported ....'   


Even if I use non 8-bit model, the model is still automatically loaded to cuda:0 when i sepcify CUDA_VISIBLE_DEVICES=1 sh run.sh. 

    model = LLaMAForCausalLM.from_pretrained(
        "decapoda-research/llama-7b-hf",
        load_in_8bit=False,
        torch_dtype=torch.float16,
        device_map="auto",
    )
  
    print('model_cuda_device {}'.format(model.device))

//output:
model_cuda_device cuda:0



@andotalao24 
To check the devices with a model that has been loaded with `device_map=xxx` you need to call `set(model.hf_device_map.values())`

I have been having the same issue, but I don't know if this is related to hardware. because I got the Error in an 8xA100 with cuda 11.8 but work perfectly in an 8xA100SMX cuda 11.7 (RunPod machines)

Hi @younesbelkada, i was going through the same issue, ISSUE - when we use already loaded quantized model, the pipeline seems to send it device. so I just updated at line 780 in base.py
From
`if self.framework == "pt" and device is not None and not (isinstance(device, int) and device < 0):`
to 
`if self.framework == "pt" and device is not None and not (isinstance(device, int) and device < 0) and not (getattr(self.model, "is_quantized")):`

hi @deepaksen1996 
Thanks! Can you please share a reproducible snippet? 

Hi @younesbelkada, apologies for this late reply
transformers==4.31.0
accelerate==0.21.0
here is the reproducible snippet

```from transformers import pipeline as hf_pipeline
from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
import torch

model_path="tiiuae/falcon-7b"

config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, load_in_8bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b")
params = {
        "max_length":1024,
        "pad_token_id": 0, 
        "device_map":"auto", 
        "load_in_8bit": True,
        # "torch_dtype":"auto"
        }
pipeline = hf_pipeline(
        task="text-generation",
        model=model,
        tokenizer=tokenizer,
        device=0,
        model_kwargs=params,
        )
```

Hi @deepaksen1996 
Thanks for the reproducer. I managed to reproduce it.
Note that `device_map="auto"` will automatically dispatch the model into the correct device(s), hence there is no need to add `device` argument into the model_kwargs. The script below works fine for me:

```python
from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer, pipeline
import torch

model_path="facebook/opt-350m"

config = AutoConfig.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, load_in_8bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_path)
params = {
        "max_length":1024,
        "pad_token_id": 0, 
        "device_map":"auto", 
        "load_in_8bit": True,
        }
pipe = pipeline(
    task="text-generation",
    model=model,
    tokenizer=tokenizer,
    model_kwargs=params,
)

pipe("Hello")
```

How did you solve this issue?

I had the exact same problem and it was fixed by `pip install accelerate==0.21.0`. Hope it helps

Also, my `transformers is 4.36.2`

> I had the exact same problem and it was fixed by `pip install accelerate==0.21.0`. Hope it helps
> 
> Also, my `transformers is 4.36.2`

These specific versions worked for me as well.