Repository: huggingface/accelerate

Issue:
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices.
@younesbelkada 
(Thanks again for developing these great libraries and responding on Github!)

Related issue: https://github.com/huggingface/accelerate/issues/1412

With the bleeding edge `transformers`, I cannot combine PEFT and accelerate to do parameter-efficient fine-tuning with naive pipeline parallelism (i.e., splitting a model loaded on 8-bit across multiple GPUs).
Are both PEFT and accelerate not supporting such use cases? The code is working on earlier `transformers` version so wondering about it. 

```
  File "/home/ec2-user/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1665, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/home/ec2-user/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1768, in _inner_training_loop
    self.model, self.optimizer, self.lr_scheduler
  File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1144, in prepare
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1144, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 995, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1201, in prepare_model
    "You can't train a model that has been loaded in 8-bit precision on multiple devices."
ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices.
```

Here is the subset of outcomes from `pip3 list` regarding the package version:
```
Package                  Version
------------------------ -----------
accelerate               0.19.0
transformers             4.30.0.dev0
peft                     0.3.0
```




Discussion:
> @younesbelkada (Thanks again for developing these great libraries and responding on Github!)
> 
> Related issue: #1412
> 
> With the bleeding edge `transformers`, I cannot combine PEFT and accelerate to do parameter-efficient fine-tuning with naive pipeline parallelism (i.e., splitting a model loaded on 8-bit across multiple GPUs). Are both PEFT and accelerate not supporting such use cases? The code is working on earlier `transformers` version so wondering about it.
> 
> ```
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1665, in train
>     ignore_keys_for_eval=ignore_keys_for_eval,
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/transformers/trainer.py", line 1768, in _inner_training_loop
>     self.model, self.optimizer, self.lr_scheduler
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1144, in prepare
>     self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1144, in <genexpr>
>     self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 995, in _prepare_one
>     return self.prepare_model(obj, device_placement=device_placement)
>   File "/home/ec2-user/.local/lib/python3.7/site-packages/accelerate/accelerator.py", line 1201, in prepare_model
>     "You can't train a model that has been loaded in 8-bit precision on multiple devices."
> ValueError: You can't train a model that has been loaded in 8-bit precision on multiple devices.
> ```
> 
> Here is the subset of outcomes from `pip3 list` regarding the package version:
> 
> ```
> Package                  Version
> ------------------------ -----------
> accelerate               0.19.0
> transformers             4.30.0.dev0
> peft                     0.3.0
> ```

same error. did you solved?

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -1258,9 +1258,11 @@ def prepare_model(self, model: torch.nn.Module, device_placement=None):
             model, "hf_device_map", False
         ):
             model_devices = set(model.hf_device_map.values())
-            if len(model_devices) > 1:
+            if len(model_devices) > 1 and self.distributed_type != DistributedType.NO:
                 raise ValueError(
-                    "You can't train a model that has been loaded in 8-bit precision on multiple devices."
+                    "You can't train a model that has been loaded in 8-bit precision on multiple devices in any distributed mode."
+                    " In order to use 8-bit models that have been loaded across multiple GPUs the solution is to use Naive Pipeline Parallelism."
+                    " Therefore you should not specify that you are under any distributed regime in your accelerate config."
                 )
             current_device = list(model_devices)[0]
             current_device_index = current_device.index if isinstance(current_device, torch.device) else current_device


Test patch:
diff --git a/tests/test_accelerator.py b/tests/test_accelerator.py
--- a/tests/test_accelerator.py
+++ b/tests/test_accelerator.py
@@ -6,7 +6,7 @@
 import torch
 from torch.utils.data import DataLoader, TensorDataset
 
-from accelerate import infer_auto_device_map, init_empty_weights
+from accelerate import DistributedType, infer_auto_device_map, init_empty_weights
 from accelerate.accelerator import Accelerator
 from accelerate.state import GradientState, PartialState
 from accelerate.test_utils import require_multi_gpu, slow
@@ -219,6 +219,8 @@ def test_accelerator_bnb_multi_gpu(self):
         """Tests that the accelerator can be used with the BNB library."""
         from transformers import AutoModelForCausalLM
 
+        PartialState._shared_state = {"distributed_type": DistributedType.MULTI_GPU}
+
         with init_empty_weights():
             model = AutoModelForCausalLM.from_pretrained(
                 "EleutherAI/gpt-neo-125m",
@@ -237,6 +239,31 @@ def test_accelerator_bnb_multi_gpu(self):
         with self.assertRaises(ValueError):
             _ = accelerator.prepare(model)
 
+        PartialState._reset_state()
+
+    @slow
+    @require_multi_gpu
+    def test_accelerator_bnb_multi_gpu_no_distributed(self):
+        """Tests that the accelerator can be used with the BNB library."""
+        from transformers import AutoModelForCausalLM
+
+        with init_empty_weights():
+            model = AutoModelForCausalLM.from_pretrained(
+                "EleutherAI/gpt-neo-125m",
+            )
+            device_map = infer_auto_device_map(model)
+            device_map["lm_head"] = 1
+
+        model = AutoModelForCausalLM.from_pretrained(
+            "EleutherAI/gpt-neo-125m",
+            load_in_8bit=True,
+            device_map=device_map,
+        )
+        accelerator = Accelerator()
+
+        # This should work
+        _ = accelerator.prepare(model)
+
     @require_cuda
     def test_accelerator_cpu_flag_prepare(self):
         model = torch.nn.Linear(10, 10)