internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: huggingface/accelerate

Issue:
TypeError: Can't apply _send_to_device on object of type <class 'int'>, only of nested list/tuple/dicts of objects that satisfy _has_to_method.
When I use 0.4.0 version , it is ok. But it is error when using 0.5.1 version. And the data is a dict, including tensor and List[List[int]]


Discussion:
Ah, thanks for flagging this out. Will push a fix this morning!

Resolving patch:
diff --git a/src/accelerate/utils.py b/src/accelerate/utils.py
--- a/src/accelerate/utils.py
+++ b/src/accelerate/utils.py
@@ -201,7 +201,7 @@ def _send_to_device(t, device):
     def _has_to_method(t):
         return hasattr(t, "to")
 
-    return recursively_apply(_send_to_device, tensor, device, test_type=_has_to_method, error_on_other_type=True)
+    return recursively_apply(_send_to_device, tensor, device, test_type=_has_to_method)
 
 
 def get_data_structure(data):


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -20,7 +20,7 @@
 from accelerate.utils import send_to_device
 
 
-TestNamedTuple = namedtuple("TestNamedTuple", "a b")
+TestNamedTuple = namedtuple("TestNamedTuple", "a b c")
 
 
 class UtilsTester(unittest.TestCase):
@@ -31,23 +31,26 @@ def test_send_to_device(self):
         result1 = send_to_device(tensor, device)
         self.assertTrue(torch.equal(result1.cpu(), tensor))
 
-        result2 = send_to_device((tensor, [tensor, tensor]), device)
+        result2 = send_to_device((tensor, [tensor, tensor], 1), device)
         self.assertIsInstance(result2, tuple)
         self.assertTrue(torch.equal(result2[0].cpu(), tensor))
         self.assertIsInstance(result2[1], list)
         self.assertTrue(torch.equal(result2[1][0].cpu(), tensor))
         self.assertTrue(torch.equal(result2[1][1].cpu(), tensor))
+        self.assertEqual(result2[2], 1)
 
-        result2 = send_to_device({"a": tensor, "b": [tensor, tensor]}, device)
+        result2 = send_to_device({"a": tensor, "b": [tensor, tensor], "c": 1}, device)
         self.assertIsInstance(result2, dict)
         self.assertTrue(torch.equal(result2["a"].cpu(), tensor))
         self.assertIsInstance(result2["b"], list)
         self.assertTrue(torch.equal(result2["b"][0].cpu(), tensor))
         self.assertTrue(torch.equal(result2["b"][1].cpu(), tensor))
+        self.assertEqual(result2["c"], 1)
 
-        result3 = send_to_device(TestNamedTuple(a=tensor, b=[tensor, tensor]), device)
+        result3 = send_to_device(TestNamedTuple(a=tensor, b=[tensor, tensor], c=1), device)
         self.assertIsInstance(result3, TestNamedTuple)
         self.assertTrue(torch.equal(result3.a.cpu(), tensor))
         self.assertIsInstance(result3.b, list)
         self.assertTrue(torch.equal(result3.b[0].cpu(), tensor))
         self.assertTrue(torch.equal(result3.b[1].cpu(), tensor))
+        self.assertEqual(result3.c, 1)
Repository: huggingface/accelerate

Issue:
Should `get_linear_schedule_with_warmup` consider `gradient_accumulation_steps`?
I think `num_training_steps` should be `len(train_dataloader) * num_epochs // gradient_accumulation_steps`


Discussion:
Could you develop a bit which file/example you are talking about and at which line?
Sorry, I mean `accelerate/examples/nlp_example.py`  `line 128`, `num_training_steps` should be `len(train_dataloader) * num_epochs // gradient_accumulation_steps`. Now it doesn't consider `gradient_accumulation_steps`.
Thanks for the precision! That's completely correct, would you like to make a PR to fix this?
Sorry, I'm not so familar with making PR. Could you please help make it?
Futhermore, I think adding   `assert (len(train_dataloader) * num_epochs) % gradient_accumulation_steps == 0` is required, but it's kind of trivial.

Resolving patch:
diff --git a/examples/nlp_example.py b/examples/nlp_example.py
--- a/examples/nlp_example.py
+++ b/examples/nlp_example.py
@@ -125,7 +125,7 @@ def collate_fn(examples):
     lr_scheduler = get_linear_schedule_with_warmup(
         optimizer=optimizer,
         num_warmup_steps=100,
-        num_training_steps=len(train_dataloader) * num_epochs,
+        num_training_steps=(len(train_dataloader) * num_epochs) // gradient_accumulation_steps,
     )
 
     # Now we train the model


Test patch:
Repository: huggingface/accelerate

Issue:
typing (list/tuple/dict) in accelerate.utils
Hi, 

I can't help noticing the typing in accelerate.utils is a bit messy.

The type check for `dict` is actually `Mapping` (which I think is a better approach as it allows user defined `Mapping`, maybe there needs some doc work.
And the type check for list/tuple is `(list, tuple)` which I believe `Sequence` could works better. 

Also, https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils.py#L119-L122 has specific behavior for type `namedtuple` and may not work with some other type, maybe it's better to use EAFP here?

```python
    try:
        return type(obj)(generator)
    except TypeError:
        return type(obj)(*list(generator))
```


Discussion:
`Mapping` only checks the class implements a couple of methods, it offers no guarantee that you can instantiate an object of that class with a dictionary. Same for the check for `list/tuple`, `Sequence` does not guarantee you can instantiate the object with a generator.

Your suggestion for the re-instantiation of a list/tuple/namedtuple sounds great! If you'd like to make a PR of it that would be awesome!

Resolving patch:
diff --git a/src/accelerate/utils.py b/src/accelerate/utils.py
--- a/src/accelerate/utils.py
+++ b/src/accelerate/utils.py
@@ -118,11 +118,11 @@ def honor_type(obj, generator):
     """
     Cast a generator to the same type as obj (list, tuple or namedtuple)
     """
-    # There is no direct check whether an object if of type namedtuple sadly, this is a workaround.
-    if isinstance(obj, tuple) and hasattr(obj, "_fields"):
-        # Can instantiate a namedtuple from a generator directly, contrary to a tuple/list.
+    try:
+        return type(obj)(generator)
+    except TypeError:
+        # Some objects may not be able to instantiate from a generator directly
         return type(obj)(*list(generator))
-    return type(obj)(generator)
 
 
 def is_torch_tensor(tensor):


Test patch:
Repository: huggingface/accelerate

Issue:
Crash with fp16 in pytorch 1.11
Crash due to new signature of `autocast.__exit__()` in pytorch 1.11

See below, now args are required:
https://github.com/pytorch/pytorch/blob/0b1f3bd158bec2cf4ab6bfde573d691c65b0742d/torch/autocast_mode.py#L201

Error:
```
Traceback (most recent call last):
 File "/Midgard/home/mrabadan/anaconda3/envs/multitask/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/Midgard/home/mrabadan/anaconda3/envs/multitask/lib/python3.8/site-packages/accelerate/accelerator.py", line 573, in autocast
    autocast_context.__exit__()
TypeError: __exit__() missing 3 required positional arguments: 'exc_type', 'exc_val', and 'exc_tb'
```


Discussion:
Could you explain how you got this error? Executing a few examples with mixed precision with PyTorch 1.11 and I didn't get it.
I run as `pytorch -m torch.distributed.run myscript` and I set the `fp16` argument in the Accelerator to true. Using master.
Like I said, I have no problem with this, so I need a clear reproducer to be able to help.
There you go. Tried on three different machines with fresh installs.

Minimal conda env:
```
name: test
channels:
  - pytorch
dependencies:
  - cudatoolkit=11.3
  - pip
  - pytorch=1.11
  - pip:
    - accelerate
    - packaging
```

Example script to reproduce:
```python
import torch
import accelerate

accelerator = accelerate.Accelerator(fp16=True)

a = torch.ones((2,10)).cuda()
criterion = torch.nn.CrossEntropyLoss()

with accelerator.autocast():
    out = criterion(a,a)
```

Result:
```
Traceback (most recent call last):
  File "/home/miquelmr/multitask/test2.py", line 10, in <module>
    out = criterion(a,a)
  File "/home/miquelmr/miniconda3/envs/test/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/home/miquelmr/miniconda3/envs/test/lib/python3.9/site-packages/accelerate/accelerator.py", line 573, in autocast
    autocast_context.__exit__()
TypeError: __exit__() missing 3 required positional arguments: 'exc_type', 'exc_val', and 'exc_tb'
```
Thanks a lot, I understand the problem now! Will fix this today.

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -14,6 +14,7 @@
 
 import gc
 import os
+import sys
 import warnings
 from contextlib import contextmanager
 from typing import List, Optional, Union
@@ -705,7 +706,7 @@ def autocast(self):
 
             autocast_context.__enter__()
             yield
-            autocast_context.__exit__()
+            autocast_context.__exit__(*sys.exc_info())
         else:
             yield
 


Test patch:
Repository: huggingface/accelerate

Issue:
DeepSpeed integration doesn't seem to work with generated config
I have used `accelerate config` to generate config that uses deepspeed

Config:
```
compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  offload_optimizer_device: NONE
  zero_stage: 3
distributed_type: DEEPSPEED
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 8
use_cpu: false
```

If I run this, I get a key error:

```
  File "/.../site-packages/accelerate/accelerator.py", line 240, in mixed_precision
    if self.state.deepspeed_plugin.deepspeed_config["fp16"]["enabled"]:
KeyError: 'fp16'
```

I am on python 3.7 and using accelerate  0.6.1 and deepspeed 0.6.1

I suspect the issue is caused by this line:
https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L274

which is querying the config that is written on this line:
https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils.py#L726
which does not have the keys in question




Discussion:
This has already been reported in #198 
It's been a long time, I know. We are going to start looking into this next week :-) 
Ah sorry it's actually different, a fix is incoming!

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -284,9 +284,10 @@ def use_fp16(self):
     @property
     def mixed_precision(self):
         if self.distributed_type == DistributedType.DEEPSPEED:
-            if self.state.deepspeed_plugin.deepspeed_config["fp16"]["enabled"]:
+            config = self.state.deepspeed_plugin.deepspeed_config
+            if config.get("fp16", {}).get("enabled", False):
                 mixed_precision = "fp16"
-            elif self.state.deepspeed_plugin.deepspeed_config["bf16"]["enabled"]:
+            elif config.get("bf16", {}).get("enabled", False):
                 mixed_precision = "bf16"
             else:
                 mixed_precision = "no"


Test patch:
Repository: huggingface/accelerate

Issue:
`psutil` required by utils/modeling.py but it isn't declared as a dependency
### System Info

```Shell
Accelerate `0.10.0.dev0` on Debian Bullseye running Python 3.10.5.


File "/opt/venv/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 276, in get_max_memory
    import psutil
ModuleNotFoundError: No module named 'psutil'
```

I'm not sure if you have any minimum version you need to satisfy for `psutil` as a runtime dependency but I see that there are no constraints on it as a `test` dependency in setup.py.

If you don't have any requirements, I'm happy to just add it myself and open a patch PR.

Thanks!
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

Use `device_map="auto"` when loading any model that supports it.

### Expected behavior

```Shell
I expect that `psutil` is declared as a runtime dependency of the `accelerate` package instead of having to install it myself.
```



Discussion:
It is already in our extra "tests" (see [here](https://github.com/huggingface/accelerate/blob/29eef234c9151ed4b11632ee15d356b0aea9c20c/setup.py#L22)), did you mean that you'd like it in the main dependencies?
Yes, please! It's required for some runtime functionality in at least one place.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -19,7 +19,6 @@
 extras["quality"] = ["black ~= 22.0", "isort >= 5.5.4", "flake8 >= 3.8.3"]
 extras["docs"] = []
 extras["test"] = [
-    "psutil",
     "pytest",
     "pytest-xdist",
     "pytest-subtests",
@@ -60,7 +59,7 @@
         ]
     },
     python_requires=">=3.7.0",
-    install_requires=["numpy>=1.17", "packaging>=20.0", "pyyaml", "torch>=1.4.0"],
+    install_requires=["numpy>=1.17", "packaging>=20.0", "psutil", "pyyaml", "torch>=1.4.0"],
     extras_require=extras,
     classifiers=[
         "Development Status :: 5 - Production/Stable",


Test patch:
Repository: huggingface/accelerate

Issue:
How to set the torch elastic params (e.g., max_restarts)? 
Hi All,

I am using `torch==1.9.0` and `accelerate=0.9.0`

I use the following to start my training script:
`accelerate launch --multi_gpu --mixed_precision=fp16 --num_machines 1 --num_processes 8 train.py`

As seen, the training runs in a single machine with 8 GPUs.  and the following information is printed.

```
[INFO] 2022-09-04 18:43:32,155 api: Starting elastic_operator with launch configs:
  entrypoint       : run_clm.py
  min_nodes        : 1
  max_nodes        : 1
  nproc_per_node   : 8
  run_id           : none
  rdzv_backend     : static
  rdzv_endpoint    : 127.0.0.1:29500
  rdzv_configs     : {'rank': 0, 'timeout': 900}
  max_restarts     : 3
  monitor_interval : 5
  log_dir          : None
  metrics_cfg      : {}
```

We can see that accelerate uses the torch's elastic run in the backend. The thing I want to is to **set `max_restarts` to `0` and  `rdzv_configs['timeout']` to `0`**. How can I do it?

Many thanks!




Discussion:

Resolving patch:
diff --git a/src/accelerate/commands/launch.py b/src/accelerate/commands/launch.py
--- a/src/accelerate/commands/launch.py
+++ b/src/accelerate/commands/launch.py
@@ -270,6 +270,25 @@ def launch_command_parser(subparsers=None):
         default=None,
         help="The port to use to communicate with the machine of rank 0.",
     )
+    # Rendezvous related arguments
+    parser.add_argument(
+        "--rdzv_conf",
+        type=str,
+        default="",
+        help="Additional rendezvous configuration (<key1>=<value1>,<key2>=<value2>,...).",
+    )
+    parser.add_argument(
+        "--max_restarts",
+        type=int,
+        default=0,
+        help="Maximum number of worker group restarts before failing.",
+    )
+    parser.add_argument(
+        "--monitor_interval",
+        type=float,
+        default=5,
+        help="Interval, in seconds, to monitor the state of workers.",
+    )
     parser.add_argument(
         "--main_training_function",
         type=str,


Test patch:
Repository: huggingface/accelerate

Issue:
`notebook_launcher` cannot use MPS device on M1 or M2 Mac
### System Info

```Shell
- `Accelerate` version: 0.13.0.dev0
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.10.4
- Numpy version: 1.23.1
- PyTorch version (GPU?): 1.13.0.dev20220909 (False)
- `Accelerate` default config:
	- compute_environment: LOCAL_MACHINE
	- distributed_type: MPS
	- mixed_precision: no
	- use_cpu: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- main_process_ip: None
	- main_process_port: None
	- rdzv_backend: static
	- same_network: False
	- main_training_function: main
	- deepspeed_config: {}
	- fsdp_config: {}
	- downcast_bf16: no
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:
1. Stable Diffusion Textual Inversion Colab script: https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb
2. Run script on M1 Max, expecting `accelerate.notebook_launcher` to report running the training on *GPU (or MPS)*
3. Observe it runs the training on *CPU* instead

### Expected behavior

```Shell
With a previous PR, `mps` M1 GPU support was introduced to Accelerate: https://github.com/huggingface/accelerate/pull/596

However `notebook_launcher` needs to be adapted to make use of this change as well.
```



Discussion:

Resolving patch:
diff --git a/src/accelerate/launchers.py b/src/accelerate/launchers.py
--- a/src/accelerate/launchers.py
+++ b/src/accelerate/launchers.py
@@ -121,12 +121,17 @@ def notebook_launcher(function, args=(), num_processes=None, use_fp16=False, mix
                 start_processes(launcher, args=args, nprocs=num_processes, start_method="fork")
 
         else:
-            # No need for a distributed launch otherwise as it's either CPU or one GPU.
-            if torch.cuda.is_available():
+            # No need for a distributed launch otherwise as it's either CPU, GPU or MPS.
+            use_mps_device = "false"
+            if torch.backends.mps.is_available():
+                print("Launching training on MPS.")
+                use_mps_device = "true"
+            elif torch.cuda.is_available():
                 print("Launching training on one GPU.")
             else:
                 print("Launching training on CPU.")
-            function(*args)
+            with patch_environment(use_mps_device=use_mps_device):
+                function(*args)
 
 
 def debug_launcher(function, args=(), num_processes=2):


Test patch:
Repository: huggingface/accelerate

Issue:
RuntimeError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.
I normally just throw everything into `accelerator.prepare` that has any tensor buffers or parameters. However, if an object subclasses `torch.nn.Module`, has no parameters and we are in a multi-GPU context, then `accelerator.prepare` tries to wrap it in DistributedDataParallel, which throws an error:

```
RuntimeError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.
```

The quick fix is to simply move the object from `accelerator.prepare` to `obj.to(accelerator.device)` but it's nice to be able to leave everything in `accelerator.prepare` so I can add parameters, turn `requires_grad` on/off and not have to move code around.

I can open a new PR to fix this; I imagine it's just checking if any of the `model.parameters()` have `require_grad` set to true. If they don't then it's a no-op because it's already moved to `accelerator.device`.

https://github.com/huggingface/accelerate/blob/693d46826e32507376d44f99967df4710886c984/src/accelerate/accelerator.py#L781-L783




Discussion:
Thanks for the very clear issue! Your proposed fix sounds right, do you want to open a PR with it?

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -777,10 +777,11 @@ def prepare_model(self, model: torch.nn.Module, device_placement=None):
         if device_placement:
             model = model.to(self.device)
         if self.distributed_type == DistributedType.MULTI_GPU:
-            kwargs = self.ddp_handler.to_kwargs() if self.ddp_handler is not None else {}
-            model = torch.nn.parallel.DistributedDataParallel(
-                model, device_ids=[self.local_process_index], output_device=self.local_process_index, **kwargs
-            )
+            if any(p.requires_grad for p in model.parameters()):
+                kwargs = self.ddp_handler.to_kwargs() if self.ddp_handler is not None else {}
+                model = torch.nn.parallel.DistributedDataParallel(
+                    model, device_ids=[self.local_process_index], output_device=self.local_process_index, **kwargs
+                )
         elif self.distributed_type == DistributedType.FSDP:
             from torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP
 


Test patch:
Repository: huggingface/accelerate

Issue:
cannot import name 'LRScheduler' from 'torch.optim.lr_scheduler'
### System Info

```Shell
env: nvcr.io/nvidia/pytorch:22.12-py3
PyTorch Version 1.14.0a0+410ce96

accelerate-0.15.0
transformers==4.25.1
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

1. nvidia-docker run -ti --rm nvcr.io/nvidia/pytorch:22.12-py3 bash
2. pip install transformers accelerate
3. accelerate env

### Expected behavior

```Shell
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 5, in <module>
    from accelerate.commands.accelerate_cli import main
  File "/usr/local/lib/python3.8/dist-packages/accelerate/__init__.py", line 7, in <module>
    from .accelerator import Accelerator
  File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 102, in <module>
    from torch.optim.lr_scheduler import LRScheduler as LRScheduler
ImportError: cannot import name 'LRScheduler' from 'torch.optim.lr_scheduler' (/usr/local/lib/python3.8/dist-packages/torch/optim/lr_scheduler.py)
```
```



Discussion:

Resolving patch:
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -101,9 +101,9 @@
     import torch_xla.distributed.xla_multiprocessing as xmp
 
 
-if is_torch_version("<=", "1.13.5"):
+try:
     from torch.optim.lr_scheduler import _LRScheduler as LRScheduler
-else:
+except ImportError:
     from torch.optim.lr_scheduler import LRScheduler as LRScheduler
 
 logger = get_logger(__name__)


Test patch:
Repository: huggingface/accelerate

Issue:
Reduce support for multi_cpu
As far as I can tell, reduce for multi_cpu actually just performs a within-process instead of an across-process reduction. 

https://github.com/huggingface/accelerate/blob/36beea9a1be9251a3ad07f7c10fd425c96954f46/src/accelerate/utils/operations.py#L430-L443


Discussion:
Indeed. Would you like to open a PR with a fix?
Sure - I'll look into it

Resolving patch:
diff --git a/src/accelerate/utils/operations.py b/src/accelerate/utils/operations.py
--- a/src/accelerate/utils/operations.py
+++ b/src/accelerate/utils/operations.py
@@ -436,6 +436,9 @@ def _reduce_across_processes(tensor, reduction="mean"):
         elif state.distributed_type.value in CUDA_DISTRIBUTED_TYPES:
             torch.distributed.all_reduce(cloned_tensor, ReduceOp.SUM)
             return cloned_tensor
+        elif state.distributed_type == DistributedType.MULTI_CPU:
+            torch.distributed.all_reduce(cloned_tensor, ReduceOp.SUM)
+            return cloned_tensor
         else:
             if reduction == "sum":
                 return cloned_tensor.sum()


Test patch: