internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/accelerate

Issue:
use DeepSpeed  dataloader duplicate per batch 
when I test examples/nlp_example.py， I added a little print log on line 175（https://github.com/huggingface/accelerate/blob/main/examples/nlp_example.py#L145）. 

<img width="1342" alt="截屏2023-03-02 下午6 11 41" src="https://user-images.githubusercontent.com/23132307/222398796-c0a1a30c-db03-4ff9-8824-480646faa9d6.png">

the default_config.yaml ：



You can see that different GPU run different data in the same step. it is ok

` compute_environment: LOCAL_MACHINE 
  deepspeed_config: {}
  distributed_type: MULTI_GPU
  downcast_bf16: 'no'
  dynamo_backend: 'NO'
  fsdp_config: {}
  gpu_ids: all
  machine_rank: 0
  main_training_function: main
  megatron_lm_config: {}
 mixed_precision: bf16
 num_machines: 1
  num_processes: 2
 rdzv_backend: static
 same_network: true
 use_cpu: false`
<img width="1050" alt="截屏2023-03-02 下午6 19 08" src="https://user-images.githubusercontent.com/23132307/222400544-bb6ff673-09d2-4154-b1e3-7bd16752828c.png">



but when use DeepSpeed , different GPU run same data in the same step.  How to understand this? Different cards running the same data?
<img width="924" alt="截屏2023-03-02 下午6 12 50" src="https://user-images.githubusercontent.com/23132307/222399118-12c7f5b8-6019-4f19-b7e9-8b1fc0bdfebc.png">
ed
the default_config.yaml ：
` 
 compute_environment: LOCAL_MACHINE
 deepspeed_config:
 deepspeed_config_file: ./config_blocklm.json
     zero3_init_flag: false
     distributed_type: DEEPSPEED
 downcast_bf16: 'no'
 dynamo_backend: 'NO'
 fsdp_config: {}
 machine_rank: 0
main_training_function: main
 megatron_lm_config: {}
 num_machines: 1
 num_processes: 2
 rdzv_backend: static
 same_network: true
 use_cpu: false`




Discussion:

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -1151,7 +1151,8 @@ def prepare_model(self, model: torch.nn.Module, device_placement=None):
     def _prepare_deepspeed(self, *args):
         deepspeed_plugin = self.state.deepspeed_plugin
 
-        if deepspeed_plugin.deepspeed_config["train_micro_batch_size_per_gpu"] == "auto":
+        is_dataloader_present = any(isinstance(obj, torch.utils.data.DataLoader) for obj in args)
+        if deepspeed_plugin.deepspeed_config["train_micro_batch_size_per_gpu"] == "auto" or is_dataloader_present:
             result = [
                 self._prepare_one(obj, first_pass=True) if isinstance(obj, torch.utils.data.DataLoader) else obj
                 for obj in args


Test patch:
Repository: huggingface/accelerate

Issue:
lr schedulers aren't prepared when using 2.0 nightlies
### System Info

```Shell
- `Accelerate` version: 0.17.0.dev0
- Platform: Linux-5.4.0-125-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Numpy version: 1.23.5
- PyTorch version (GPU?): 2.0.0.dev20230208 (True)
- `Accelerate` default config:
	Not found
```


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Add following snippet post `accelerator.prepare` in the nlp_example.py
```
from accelerate.scheduler import AcceleratedScheduler
if not isinstance(lr_scheduler, AcceleratedScheduler):
        raise ValueError("Oh no! lr scheduler isn't prepared")
```
2. logs will have:
```
ValueError: Oh no! lr scheduler isn't prepared
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 386705
```

Reason: 
1. Because of PR #1017, `_LRScheduler` is being imported instead of  `LRScheduler` as both are present. However, lr schedulers now extend `LRScheduler`, `_LRScheduler` also extends `LRScheduler`. Because of this below line fails and lr scheduler doesn't get prepared:

https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L918

 

### Expected behavior

```Shell
lr scheduler is properly prepared
```



Discussion:

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -102,9 +102,9 @@
 
 
 try:
-    from torch.optim.lr_scheduler import _LRScheduler as LRScheduler
+    from torch.optim.lr_scheduler import LRScheduler
 except ImportError:
-    from torch.optim.lr_scheduler import LRScheduler as LRScheduler
+    from torch.optim.lr_scheduler import _LRScheduler as LRScheduler
 
 logger = get_logger(__name__)
 


Test patch:
Repository: huggingface/accelerate

Issue:
Feature Request for FSDP: Allow to pass param `use_orig_params` to FullyShardedDataParallel beginning from PyTorch v2
### System Info

```Shell
Recently PyTorch has added a new parameter `use_orig_params` to the FullyShardedDataParallel class, which enables to freeze some model layers during the training. See:

* https://dev-discuss.pytorch.org/t/rethinking-pytorch-fully-sharded-data-parallel-fsdp-from-first-principles/1019
* https://github.com/pytorch/pytorch/issues/83204

This feature is especially interesting for training adapters in conjunction with Accelerate. 

1. Please extend FullyShardedDataParallelPlugin to provide this configuration option.
2. And extend `Accelerator::prepare_model` to pass this option as a parameter to the FSDP constructor
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

```Shell
As for 1: Add a configuration option here:

https://github.com/huggingface/accelerate/blob/853823d0ae8e67f5a8b8083df4672e73dfa3f800/src/accelerate/utils/dataclasses.py#L667

As for 2: Add parameter passing here:

https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L1152--L1165
```



Discussion:
cc @pacman100 

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import contextlib
+import inspect
 import math
 import os
 import shutil
@@ -1160,8 +1161,11 @@ def prepare_model(self, model: torch.nn.Module, device_placement=None):
                     "ignored_modules": fsdp_plugin.ignored_modules,
                     "device_id": self.device,
                 }
-                if is_torch_version(">=", "1.13.0"):
+                signature = inspect.signature(FSDP.__init__).parameters.keys()
+                if "limit_all_gathers" in signature:
                     kwargs["limit_all_gathers"] = fsdp_plugin.limit_all_gathers
+                if "use_orig_params" in signature:
+                    kwargs["use_orig_params"] = fsdp_plugin.use_orig_params
                 model = FSDP(model, **kwargs)
             self._models[-1] = model
         elif self.distributed_type == DistributedType.MULTI_CPU:
diff --git a/src/accelerate/utils/dataclasses.py b/src/accelerate/utils/dataclasses.py
--- a/src/accelerate/utils/dataclasses.py
+++ b/src/accelerate/utils/dataclasses.py
@@ -744,6 +744,11 @@ class FullyShardedDataParallelPlugin:
         },
     )
 
+    use_orig_params: bool = field(
+        default=False,
+        metadata={"help": "If True, enables parameter-efficient fine-tuning"},
+    )
+
     def __post_init__(self):
         from torch.distributed.fsdp.fully_sharded_data_parallel import (
             BackwardPrefetch,


Test patch:
Repository: huggingface/accelerate

Issue:
`set_module_tensor_to_device` has wrong shape 
(created by @Narsil who was stuck outside github)
```python 
from transformers import AutoModel
from accelerate.utils import set_module_tensor_to_device
import torch

model = AutoModel.from_pretrained("gpt2")

print(model.wte.weight.shape)
A = torch.zeros((2, 2))
set_module_tensor_to_device(model, 'wte.weight', "cpu", value=A)
print(model.wte.weight.shape)
```
Potential bug:
`set_module_tensor_to_device` doesn't check shapes, meaning you can potentially load invalid checkpoints onto an already loaded model.
This is triggered here in transformers : https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L717
Is it normal ? Is the bug here, or in transformers where we should do the shape checking ?
In anycase, if you have an invalid checkpoint vs config, there will be no error. I'm not sure this is OK.


Discussion:
Couldn't open it myself, had SSO issues, all solved now. Thanks @ArthurZucker 
We can add a check for shape, happy to have a look at a PR.

Resolving patch:
diff --git a/src/accelerate/utils/modeling.py b/src/accelerate/utils/modeling.py
--- a/src/accelerate/utils/modeling.py
+++ b/src/accelerate/utils/modeling.py
@@ -267,6 +267,11 @@ def set_module_tensor_to_device(
         raise ValueError(f"{tensor_name} is on the meta device, we need a `value` to put in on {device}.")
 
     if value is not None:
+        if old_value.shape != value.shape:
+            raise ValueError(
+                f'Trying to set a tensor of shape {value.shape} in "{tensor_name}" (which has shape {old_value.shape}), this look incorrect.'
+            )
+
         if dtype is None:
             # For compatibility with PyTorch load_state_dict which converts state dict dtype to existing dtype in model
             value = value.to(old_value.dtype)


Test patch:
diff --git a/tests/test_modeling_utils.py b/tests/test_modeling_utils.py
--- a/tests/test_modeling_utils.py
+++ b/tests/test_modeling_utils.py
@@ -138,6 +138,16 @@ def test_set_module_tensor_sets_dtype(self):
         set_module_tensor_to_device(model, "linear1.weight", "cpu", value=model.linear1.weight, dtype=torch.float16)
         self.assertEqual(model.linear1.weight.dtype, torch.float16)
 
+    def test_set_module_tensor_checks_shape(self):
+        model = ModelForTest()
+        tensor = torch.zeros((2, 2))
+        with self.assertRaises(ValueError) as cm:
+            set_module_tensor_to_device(model, "linear1.weight", "cpu", value=tensor)
+        self.assertEqual(
+            str(cm.exception),
+            'Trying to set a tensor of shape torch.Size([2, 2]) in "weight" (which has shape torch.Size([4, 3])), this look incorrect.',
+        )
+
     def test_named_tensors(self):
         model = nn.BatchNorm1d(4)
         named_tensors = named_module_tensors(model)
Repository: huggingface/accelerate

Issue:
【BUG Report】accelerate method remove_hook_from_module can't remove cpu_offload hook.
### System Info

First I wanna cpu offloaded model back to cpu.
so i first call the method `accelerate.cpu_offload(module, CUDA_0)` to offload module to cuda.
then I want my module back to cpu, as i called `remove_hook_from_module(module, recurse=True)`

and remove method tells me NoneType error.


if tensor_name not in module._parameters and tensor_name not in module._buffers:
AttributeError: 'NoneType' object has no attribute '_parameters'

so i check the code, and found that.


![image](https://github.com/huggingface/accelerate/assets/53092165/c6868fef-5bc4-4123-852f-747416eb88b7)
```Shell
the hook detach method should return changed module. Also I see the Base Hook definition `ModelHook`. 
````
![image](https://github.com/huggingface/accelerate/assets/53092165/75267d2b-ff2e-465f-877a-81cd735daaa7)

so the only thing is, class `AlignDevicesHook` not support it well for not return module out.

![image](https://github.com/huggingface/accelerate/assets/53092165/60ea198b-7ff9-4106-b10c-cfb368bd0631)





### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

1. call `accelerate.cpu_offload(module, CUDA_0)`  to offload some module to cuda
2. call `remove_hook_from_module(module, recurse=True)` to remove offload hook.




### Expected behavior

detach  hook success.


Discussion:
cc @SunMarc 

Resolving patch:
diff --git a/src/accelerate/hooks.py b/src/accelerate/hooks.py
--- a/src/accelerate/hooks.py
+++ b/src/accelerate/hooks.py
@@ -311,6 +311,7 @@ def detach_hook(self, module):
             for name, device in self.original_devices.items():
                 if device != torch.device("meta"):
                     set_module_tensor_to_device(module, name, device, value=self.weights_map.get(name, None))
+        return module
 
 
 def attach_execution_device_hook(


Test patch:
Repository: huggingface/accelerate

Issue:
pad_across_processes not compatible with PyTorch's nested_tensor
Buddy issue for https://github.com/huggingface/transformers/issues/26687 . The issue is basically identical in accelerate to transformers, so this issue can just be used for tracking and not discussion.


Discussion:

Resolving patch:
diff --git a/src/accelerate/utils/__init__.py b/src/accelerate/utils/__init__.py
--- a/src/accelerate/utils/__init__.py
+++ b/src/accelerate/utils/__init__.py
@@ -101,6 +101,7 @@
     save_offload_index,
 )
 from .operations import (
+    CannotPadNestedTensorWarning,
     broadcast,
     broadcast_object_list,
     concatenate,
diff --git a/src/accelerate/utils/operations.py b/src/accelerate/utils/operations.py
--- a/src/accelerate/utils/operations.py
+++ b/src/accelerate/utils/operations.py
@@ -17,6 +17,7 @@
 """
 
 import pickle
+import warnings
 from functools import update_wrapper, wraps
 from typing import Any, Mapping
 
@@ -525,6 +526,10 @@ def concatenate(data, dim=0):
     return torch.cat(data, dim=dim)
 
 
+class CannotPadNestedTensorWarning(UserWarning):
+    pass
+
+
 @chained_operation
 def pad_across_processes(tensor, dim=0, pad_index=0, pad_first=False):
     """
@@ -543,6 +548,12 @@ def pad_across_processes(tensor, dim=0, pad_index=0, pad_first=False):
     """
 
     def _pad_across_processes(tensor, dim=0, pad_index=0, pad_first=False):
+        if getattr(tensor, "is_nested", False):
+            warnings.warn(
+                "Cannot pad nested tensors without more information. Leaving unprocessed.",
+                CannotPadNestedTensorWarning,
+            )
+            return tensor
         if dim >= len(tensor.shape):
             return tensor
 


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -27,11 +27,13 @@
 from accelerate.test_utils.testing import require_cuda, require_torch_min_version
 from accelerate.test_utils.training import RegressionModel
 from accelerate.utils import (
+    CannotPadNestedTensorWarning,
     check_os_kernel,
     convert_outputs_to_fp32,
     extract_model_from_parallel,
     find_device,
     listify,
+    pad_across_processes,
     patch_environment,
     recursively_apply,
     save,
@@ -226,3 +228,12 @@ def forward(self, x):
                 save(model.state_dict(), save_path, safe_serialization=True)
                 self.assertEqual(len(log.records), 1)
                 self.assertIn("Removed shared tensor", log.output[0])
+
+    @require_torch_min_version(version="1.12")
+    def test_pad_across_processes(self):
+        from torch.nested import nested_tensor
+
+        nt = nested_tensor([[1, 2, 3], [1], [1, 2]])
+        with self.assertWarns(CannotPadNestedTensorWarning):
+            nt2 = pad_across_processes(nt)
+        self.assertIs(nt, nt2)
Repository: huggingface/accelerate

Issue:
`requires_grad` is not properly extracted when registering empty parameters.
```python
def register_empty_parameter(module, name, param):
    old_register_parameter(module, name, param)
    if param is not None:
        param_cls = type(module._parameters[name])
        kwargs = module._parameters[name].__dict__
        module._parameters[name] = param_cls(module._parameters[name].to(device), **kwargs)
```


In `register_empty_parameter`, both the original parameter `module._parameters[name]` and the new parameter `param` don't require grad in my case of `int16` parameters. However, the newly created `param_cls(module._parameters[name].to(device), **kwargs)` ([here](https://github.com/huggingface/accelerate/blob/5ec00da2befbe9fa666f1bee8bdb1dd4daf713e3/src/accelerate/big_modeling.py#L126C22-L126C45)) parameter requires grad anyway, because `requires_grad=False` doesn't make it into `kwargs` for some reason. But since it's integer, it can't require grad and the whole thing fails.

Example notebook: [colab](https://colab.research.google.com/drive/1IwAAKmj9aGaM2N_MKzmyZNYuPemjVEeR?usp=sharing)






Discussion:
Maybe explicitly add `requires_grad` to `kwargs`.

Resolving patch:
diff --git a/src/accelerate/big_modeling.py b/src/accelerate/big_modeling.py
--- a/src/accelerate/big_modeling.py
+++ b/src/accelerate/big_modeling.py
@@ -124,6 +124,7 @@ def register_empty_parameter(module, name, param):
         if param is not None:
             param_cls = type(module._parameters[name])
             kwargs = module._parameters[name].__dict__
+            kwargs["requires_grad"] = param.requires_grad
             module._parameters[name] = param_cls(module._parameters[name].to(device), **kwargs)
 
     def register_empty_buffer(module, name, buffer, persistent=True):


Test patch:
Repository: huggingface/accelerate

Issue:
Python 3.12 distutils deprecation error
### System Info

```Shell
accelerate 0.26.1, Windows 11, Python 3.12.1
```


### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

To reproduce behaviour:
1. In Python 3.12 install latest accelerate with
` pip3 install accelerate --upgrade `
2. Try to initialise accelerate with:
```python
from accelerate import Accelerator
accelerator = Accelerator()
```

This results in the error caused by https://github.com/huggingface/accelerate/blob/f88661b5d9f4977b5a1555a92e64ef3cd359cbc9/src/accelerate/utils/environment.py#L19

![image](https://github.com/huggingface/accelerate/assets/7803161/8b88d142-cf67-43b5-9b02-3f9346baa02a)
The fundamental issue is deprecation of distutils as part of https://peps.python.org/pep-0632/

### Expected behavior

Accelerate should not depend on deprecated package and be compatible with Python 3.12+


Discussion:
Thanks! It looks like we should replace this with `shutil.which` instead (with zero logic changes)

would you like to make a PR? 
> Thanks! It looks like we should replace this with `shutil.which` instead (with zero logic changes)
> 
> would you like to make a PR?

Sure. I’ll look into it next week. Does seem fairly straightforward. 

Resolving patch:
diff --git a/src/accelerate/utils/environment.py b/src/accelerate/utils/environment.py
--- a/src/accelerate/utils/environment.py
+++ b/src/accelerate/utils/environment.py
@@ -16,7 +16,7 @@
 import platform
 import subprocess
 import sys
-from distutils import spawn
+from shutil import which
 from typing import Dict
 
 import torch
@@ -72,8 +72,8 @@ def get_gpu_info():
     """
     if platform.system() == "Windows":
         # If platform is Windows and nvidia-smi can't be found in path
-        # try from systemd rive with default installation path
-        command = spawn.find_executable("nvidia-smi")
+        # try from systemd drive with default installation path
+        command = which("nvidia-smi")
         if command is None:
             command = "%s\\Program Files\\NVIDIA Corporation\\NVSMI\\nvidia-smi.exe" % os.environ["systemdrive"]
     else:


Test patch:
Repository: cloudtools/troposphere

Issue:
Update metadata to include property keys
Within the CloudFormation metadata object for auto scaling launch configurations, it'd be nice to be able to rely on the validators within the template to sanity check the allowed keys within the 'config' dictionary: 

```
"Resources": {
  "MyInstance": {
    "Type": "AWS::EC2::Instance",
    "Metadata" : {
      "AWS::CloudFormation::Init" : {
        "config" : {
          "packages" : {
            :
          },
          "groups" : {
            :
          },
          "users" : {
            :
          },
          "sources" : {
            :
          },
          "files" : {
            :
          },
          "commands" : {
            :
          },
          "services" : {
            :
          }
        }
      }
    },
    "Properties": {
      :
    }
  }
}
```

Currently the implementation of troposphere.cloudformation.InitConfig only contains one key which is 'files'



Discussion:

Resolving patch:
diff --git a/troposphere/cloudformation.py b/troposphere/cloudformation.py
--- a/troposphere/cloudformation.py
+++ b/troposphere/cloudformation.py
@@ -67,7 +67,13 @@ def JSONrepr(self):
 
 class InitConfig(AWSProperty):
     props = {
-        'files': (dict, False)
+        'groups': (dict, False),
+        'users': (dict, False),
+        'sources': (dict, False),
+        'packages': (dict, False),
+        'files': (dict, False),
+        'commands': (dict, False),
+        'services': (dict, False)
     }
 
 


Test patch:
Repository: cloudtools/troposphere

Issue:
Cloudwatch Alarm Threshold Type
The parameter type 'Threshold' within Cloudwatch Alarms is currently of type 'integer' whereas the AWS documentations notes this should be a String.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cw-alarm.html#cfn-cloudwatch-alarms-threshold

I am hitting an issue when using alarms to check instance health - to monitor StatusCheckFailed I have an implementation which sets Threshold to 0.5 to evaluate a healthcheck of sorts. This works in Cloudformation but fails when I try to use it in the troposphere code.

I think the line 'Threshold': (integer, True), should be 'Threshold': (basestring, True), within cloudwatch.py

Any thoughts?



Discussion:
That sounds about right to me - if you want to submit a pull request, I can merge it - or I can go ahead and fix this.  Let me know, and thanks for the issue!


Resolving patch:
diff --git a/troposphere/cloudwatch.py b/troposphere/cloudwatch.py
--- a/troposphere/cloudwatch.py
+++ b/troposphere/cloudwatch.py
@@ -4,7 +4,7 @@
 # See LICENSE file for full license.
 
 from . import AWSObject, AWSProperty, Ref
-from .validators import integer, positive_integer, boolean
+from .validators import positive_integer, boolean
 
 
 class MetricDimension(AWSProperty):
@@ -31,6 +31,6 @@ class Alarm(AWSObject):
         'OKActions': ([basestring, Ref], False),
         'Period': (positive_integer, True),
         'Statistic': (basestring, True),
-        'Threshold': (integer, True),
+        'Threshold': (basestring, True),
         'Unit': (basestring, False),
     }


Test patch: