muellerzr/github-pr-history huggingface dataset:
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml
new file mode 100644
index 000000000..69a101307
--- /dev/null
+++ b/.github/ISSUE_TEMPLATE/bug-report.yml
@@ -0,0 +1,59 @@
+name: "\U0001F41B Bug Report"
+description: Submit a bug report to help us improve Accelerate
+labels: [ "bug" ]
+body:
+  - type: textarea
+    id: system-info
+    attributes:
+      label: System Info
+      description: Please share your accelerate configuration with us. You can run the command `accelerate env` and copy-paste its outputs below
+      render: Shell
+      placeholder: accelerate version, OS, python version, numpy version, torch version, and accelerate's configuration
+    validations:
+      required: true
+  
+  - type: checkboxes
+    id: information-scripts-examples
+    attributes:
+      label: Information
+      description: 'The problem arises when using:'
+      options:
+        - label: "The official example scripts"
+        - label: "My own modified scripts"
+  
+  - type: checkboxes
+    id: information-tasks
+    attributes:
+      label: Tasks
+      description: "The tasks I am working on are:"
+      options:
+        - label: "One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)"
+        - label: "My own task or dataset (give details below)"
+  
+  - type: textarea
+    id: reproduction
+    validations:
+      required: true
+    attributes:
+      label: Reproduction
+      description: |
+        Please provide a code sample that reproduces the problem you ran into. It can be a Colab link or just a code snippet.
+        If you have code snippets, error messages, stack traces please provide them here as well.
+        Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
+        Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
+
+      placeholder: |
+        Steps to reproduce the behavior:
+          
+          1.
+          2.
+          3.
+
+  - type: textarea
+    id: expected-behavior
+    validations:
+      required: true
+    attributes:
+      label: Expected behavior
+      description: "A clear and concise description of what you would expect to happen."
+      render: Shell



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/docker/accelerate-gpu/Dockerfile b/docker/accelerate-gpu/Dockerfile
index 6195ae2ae..f495d9063 100644
--- a/docker/accelerate-gpu/Dockerfile
+++ b/docker/accelerate-gpu/Dockerfile
@@ -22,7 +22,7 @@ SHELL ["/bin/bash", "-c"]
 # Activate the conda env and install torch + accelerate
 RUN source activate accelerate && \
     python3 -m pip install --no-cache-dir \
-    git+https://github.com/huggingface/accelerate@nightly-runner#egg=accelerate[test,test_trackers] \
+    git+https://github.com/huggingface/accelerate#egg=accelerate[test,test_trackers] \
     --extra-index-url https://download.pytorch.org/whl/cu113
 
 # Stage 2



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/test_utils/training.py b/src/accelerate/test_utils/training.py
index c26587abe..7345b93cb 100644
--- a/src/accelerate/test_utils/training.py
+++ b/src/accelerate/test_utils/training.py
@@ -17,8 +17,6 @@
 from torch.utils.data import DataLoader
 
 from accelerate.utils.dataclasses import DistributedType
-from datasets import load_dataset
-from transformers import AutoTokenizer
 
 
 class RegressionDataset:
@@ -51,6 +49,9 @@ def forward(self, x=None):
 
 
 def mocked_dataloaders(accelerator, batch_size: int = 16):
+    from datasets import load_dataset
+    from transformers import AutoTokenizer
+
     tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
     data_files = {"train": "tests/test_samples/MRPC/train.csv", "validation": "tests/test_samples/MRPC/dev.csv"}
     datasets = load_dataset("csv", data_files=data_files)



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Thanks folks, for the library and the fix!
diff --git a/docker/accelerate-cpu/Dockerfile b/docker/accelerate-cpu/Dockerfile
index c536fb70c..0c7073f7c 100644
--- a/docker/accelerate-cpu/Dockerfile
+++ b/docker/accelerate-cpu/Dockerfile
@@ -27,8 +27,11 @@ RUN python3 -m pip install --no-cache-dir \
 # Stage 2
 FROM python:3.6-slim AS build-image
 COPY --from=compile-image /opt/venv /opt/venv
-RUN useradd -ms /bin/bash user
-USER user
+# Install apt libs
+RUN apt-get update && \
+    apt-get install -y curl git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists*
 
 # Make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
diff --git a/docker/accelerate-gpu/Dockerfile b/docker/accelerate-gpu/Dockerfile
index 299f573d8..5eabb00f7 100644
--- a/docker/accelerate-gpu/Dockerfile
+++ b/docker/accelerate-gpu/Dockerfile
@@ -30,9 +30,13 @@ FROM nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04 AS build-image
 COPY --from=compile-image /opt/conda /opt/conda
 ENV PATH /opt/conda/bin:$PATH
 
-RUN echo "source activate accelerate" >> /.bashrc
+# Install apt libs
+RUN apt-get update && \
+    apt-get install -y curl git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists*
+
+RUN echo "source activate accelerate" >> ~/.profile
 
-RUN useradd -ms /bin/bash user
-USER user
 # Activate the virtualenv
 CMD ["/bin/bash"]
\ No newline at end of file



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/setup.py b/setup.py
index dcfe3fd8e..b53519525 100644
--- a/setup.py
+++ b/setup.py
@@ -19,6 +19,7 @@
 extras["quality"] = ["black ~= 22.0", "isort >= 5.5.4", "flake8 >= 3.8.3"]
 extras["docs"] = []
 extras["test"] = [
+    "psutil", 
     "pytest",
     "pytest-xdist",
     "pytest-subtests",
@@ -27,7 +28,7 @@
     "scipy",
     "sklearn"
 ]
-extras["test_trackers"] = ["wandb", "comet-ml", "tensorflow"]
+extras["test_trackers"] = ["wandb", "comet-ml", "tensorflow>=2.6.2", "tensorboard"]
 extras["dev"] = extras["quality"] + extras["test"]
 
 extras["sagemaker"] = [


Review comments: body: No, `psutil` should be an optional dependency as it's only used for big model inference when `device_map="auto"`.
diff_hunk: @@ -55,7 +55,7 @@
         ]
     },
     python_requires=">=3.6.0",
-    install_requires=["numpy>=1.17", "packaging>=20.0", "pyyaml", "torch>=1.4.0"],
+    install_requires=["numpy>=1.17", "packaging>=20.0", "psutil", "pyyaml", "torch>=1.4.0"],
body: Makes sense, will move it over there. Thanks!
diff_hunk: @@ -55,7 +55,7 @@
         ]
     },
     python_requires=">=3.6.0",
-    install_requires=["numpy>=1.17", "packaging>=20.0", "pyyaml", "torch>=1.4.0"],
+    install_requires=["numpy>=1.17", "packaging>=20.0", "psutil", "pyyaml", "torch>=1.4.0"],

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/tests/test_examples.py b/tests/test_examples.py
index 6ec2a143a..3c875e02d 100644
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -21,6 +21,7 @@
 import unittest
 from unittest import mock
 
+from accelerate import Accelerator
 from accelerate.test_utils.examples import compare_against_test
 from accelerate.test_utils.testing import TempDirTestCase, slow
 from accelerate.utils import write_basic_config
@@ -142,11 +143,11 @@ def test_checkpointing_by_epoch(self):
     def test_checkpointing_by_steps(self):
         testargs = f"""
         examples/by_feature/checkpointing.py
-        --checkpointing_steps 2
+        --checkpointing_steps 1
         --output_dir {self.tmpdir}
         """.split()
         _ = subprocess.run(self._launch_args + testargs, stdout=subprocess.PIPE, env=os.environ)
-        self.assertTrue(os.path.exists(os.path.join(self.tmpdir, "step_4")))
+        self.assertTrue(os.path.exists(os.path.join(self.tmpdir, "step_5")))
 
     def test_load_states_by_epoch(self):
         testargs = f"""
@@ -163,14 +164,20 @@ def test_load_states_by_epoch(self):
     def test_load_states_by_steps(self):
         testargs = f"""
         examples/by_feature/checkpointing.py
-        --resume_from_checkpoint {os.path.join(self.tmpdir, "step_4")}
+        --resume_from_checkpoint {os.path.join(self.tmpdir, "step_5")}
         """.split()
         output = subprocess.run(
             self._launch_args + testargs, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
         ).stdout
-        self.assertNotIn("epoch 0:", output)
-        self.assertIn("epoch 1:", output)
-        self.assertIn("epoch 2:", output)
+        num_processes = Accelerator().num_processes
+        if num_processes > 1:
+            self.assertNotIn("epoch 0:", output)
+            self.assertNotIn("epoch 1:", output)
+            self.assertIn("epoch 2:", output)
+        else:
+            self.assertNotIn("epoch 0:", output)
+            self.assertIn("epoch 1:", output)
+            self.assertIn("epoch 2:", output)
 
     @slow
     def test_cross_validation(self):



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 5020c6cc4..4613e1e8d 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -14,7 +14,7 @@ jobs:
     - name: Install Python dependencies
       run: pip install setuptools==59.5.0; pip install -e .[test,test_trackers]
     - name: Run Tests
-      run: make test
+      run: make test_cpu
       
   test_examples:
     runs-on: ubuntu-latest
@@ -27,4 +27,4 @@ jobs:
     - name: Install Python dependencies
       run: pip install setuptools==59.5.0; pip install -e .[test] tensorboard
     - name: Run Tests
-      run: make test_examples
\ No newline at end of file
+      run: make test_examples
diff --git a/Makefile b/Makefile
index db7893565..d2822153b 100644
--- a/Makefile
+++ b/Makefile
@@ -24,8 +24,12 @@ style:
 	python utils/style_doc.py src/accelerate docs/source --max_len 119
 	
 # Run tests for the library
-test:
-	python -m pytest -n auto --dist=loadfile -s -v ./tests/ --ignore=./tests/test_examples.py
+test_cpu:
+	python -m pytest -s -v ./tests/ --ignore=./tests/test_examples.py
+
+test_cuda:
+	python -m pytest -s -v ./tests/ --ignore=./tests/test_examples.py --ignore=./tests/test_scheduler.py --ignore=./tests/test_cpu.py
+	python -m pytest -s -v ./tests/test_cpu.py ./tests/test_scheduler.py
 
 test_examples:
-	python -m pytest -n auto --dist=loadfile -s -v ./tests/test_examples.py
+	python -m pytest -s -v ./tests/test_examples.py
diff --git a/tests/test_cpu.py b/tests/test_cpu.py
index 72a8b2e7d..63ee69a97 100644
--- a/tests/test_cpu.py
+++ b/tests/test_cpu.py
@@ -18,6 +18,6 @@
 from accelerate.test_utils import test_script
 
 
-class MultiTPUTester(unittest.TestCase):
+class MultiCPUTester(unittest.TestCase):
     def test_cpu(self):
         debug_launcher(test_script.main)



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/utils/offload.py b/src/accelerate/utils/offload.py
index 660bc521a..a562e9283 100644
--- a/src/accelerate/utils/offload.py
+++ b/src/accelerate/utils/offload.py
@@ -21,44 +21,13 @@
 import torch
 
 
-def offload_state_dict(save_dir: Union[str, os.PathLike], state_dict: Dict[str, torch.Tensor]):
-    """
-    Offload a state dict in a given folder.
-
-    Args:
-        save_dir (`str` or `os.PathLike`): The directory in which to offload the state dict.
-        state_dict (`Dict[str, torch.Tensor]`): The dictionary of tensors to offload.
-    """
-    os.makedirs(save_dir, exist_ok=True)
-    index = {}
-    for name, parameter in state_dict.items():
-        tensor_file = os.path.join(save_dir, f"{name}.dat")
-        array = parameter.numpy()
-        index[name] = {"dtype": str(array.dtype), "shape": list(array.shape)}
-        if array.ndim == 0:
-            array = array[None]
-        file_array = np.memmap(tensor_file, dtype=array.dtype, mode="w+", shape=array.shape)
-        file_array[:] = array[:]
-        file_array.flush()
-
-    # Update index
-    index_file = os.path.join(save_dir, "index.json")
-    if os.path.isfile(index_file):
-        with open(index_file, "r", encoding="utf-8") as f:
-            current_index = json.load(f)
-    else:
-        current_index = {}
-    current_index.update(index)
-
-    with open(index_file, "w", encoding="utf-8") as f:
-        json.dump(current_index, f, indent=2)
-
-
 def offload_weight(weight, weight_name, offload_folder, index=None):
     array = weight.numpy()
     tensor_file = os.path.join(offload_folder, f"{weight_name}.dat")
     if index is not None:
         index[weight_name] = {"dtype": str(array.dtype), "shape": list(array.shape)}
+    if array.ndim == 0:
+        array = array[None]
     file_array = np.memmap(tensor_file, dtype=array.dtype, mode="w+", shape=array.shape)
     file_array[:] = array[:]
     file_array.flush()
@@ -82,6 +51,23 @@ def save_offload_index(index, offload_folder):
         json.dump(current_index, f, indent=2)
 
 
+def offload_state_dict(save_dir: Union[str, os.PathLike], state_dict: Dict[str, torch.Tensor]):
+    """
+    Offload a state dict in a given folder.
+
+    Args:
+        save_dir (`str` or `os.PathLike`): The directory in which to offload the state dict.
+        state_dict (`Dict[str, torch.Tensor]`): The dictionary of tensors to offload.
+    """
+    os.makedirs(save_dir, exist_ok=True)
+    index = {}
+    for name, parameter in state_dict.items():
+        index = offload_weight(parameter, name, save_dir, index=index)
+
+    # Update index
+    save_offload_index(index, save_dir)
+
+
 class PrefixedDataset(Mapping):
     """
     Will access keys in a given dataset by adding a prefix.



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/README.md b/README.md
index a3fc74cef..44b25778e 100644
--- a/README.md
+++ b/README.md
@@ -212,6 +212,7 @@ If you like the simplicity of 🤗 Accelerate but would prefer a higher-level ab
 
 * [Animus](https://github.com/Scitator/animus) is a minimalistic framework to run machine learning experiments. Animus highlights common "breakpoints" in ML experiments and provides a unified interface for them within [IExperiment](https://github.com/Scitator/animus/blob/main/animus/core.py#L76).
 * [Catalyst](https://github.com/catalyst-team/catalyst#getting-started) is a PyTorch framework for Deep Learning Research and Development. It focuses on reproducibility, rapid experimentation, and codebase reuse so you can create something new rather than write yet another train loop. Catalyst provides a [Runner](https://catalyst-team.github.io/catalyst/api/core.html#runner) to connect all parts of the experiment: hardware backend, data transformations, model train, and inference logic.
+* [fastai](https://github.com/fastai/fastai#installing) is a PyTorch framework for Deep Learning that simplifies training fast and accurate neural nets using modern best practices. fastai provides a [Learner](https://docs.fast.ai/learner.html#Learner) to handle the training, fine-tuning, and inference of deep learning algorithms.
 * [Kornia](https://kornia.readthedocs.io/en/latest/get-started/introduction.html) is a differentiable library that allows classical computer vision to be integrated into deep learning models. Kornia provides a [Trainer](https://kornia.readthedocs.io/en/latest/x.html#kornia.x.Trainer) with the specific purpose to train and fine-tune the supported deep learning algorithms within the library.
 * [pytorch-accelerated](https://github.com/Chris-hughes10/pytorch-accelerated) is a lightweight training library, with a streamlined feature set centred around a general-purpose [Trainer](https://pytorch-accelerated.readthedocs.io/en/latest/trainer.html), that places a huge emphasis on simplicity and transparency; enabling users to understand exactly what is going on under the hood, but without having to write and maintain the boilerplate themselves!
 



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/setup.py b/setup.py
index 7d463cf88..dcfe3fd8e 100644
--- a/setup.py
+++ b/setup.py
@@ -55,7 +55,7 @@
         ]
     },
     python_requires=">=3.6.0",
-    install_requires=["torch>=1.4.0", "pyyaml", "numpy>=1.17"],
+    install_requires=["numpy>=1.17", "packaging>=20.0", "pyyaml", "torch>=1.4.0"],
     extras_require=extras,
     classifiers=[
         "Development Status :: 5 - Production/Stable",


Review comments: body: If we're alphabetically sorting, let's really do it ;-)
diff_hunk: @@ -55,7 +55,7 @@
         ]
     },
     python_requires=">=3.6.0",
-    install_requires=["torch>=1.4.0", "pyyaml", "numpy>=1.17"],
+    install_requires=["numpy>=1.17", "pyyaml", "torch>=1.4.0", "packaging>=20.0"],
body: Fixed, heh
diff_hunk: @@ -55,7 +55,7 @@
         ]
     },
     python_requires=">=3.6.0",
-    install_requires=["torch>=1.4.0", "pyyaml", "numpy>=1.17"],
+    install_requires=["numpy>=1.17", "pyyaml", "torch>=1.4.0", "packaging>=20.0"],

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/hooks.py b/src/accelerate/hooks.py
index 0a5bca96d..dea10ba50 100644
--- a/src/accelerate/hooks.py
+++ b/src/accelerate/hooks.py
@@ -260,6 +260,24 @@ def detach_hook(self, module):
                     set_module_tensor_to_device(module, name, device, value=self.weights_map.get(name, None))
 
 
+def attach_execution_device_hook(module: torch.nn.Module, execution_device: Union[int, str, torch.device]):
+    """
+    Recursively attaches `AlignDevicesHook` to all submodules of a given model to make sure they have the right
+    execution device
+
+    Args:
+        module (`torch.nn.Module`):
+            The module where we want to attach the hooks.
+        execution_device (`int`, `str` or `torch.device`):
+            The device on which inputs and model weights should be placed before the forward pass.
+    """
+    if not hasattr(module, "_hf_hook") and len(module.state_dict()) > 0:
+        add_hook_to_module(module, AlignDevicesHook(execution_device))
+
+    for child in module.children():
+        attach_execution_device_hook(child, execution_device)
+
+
 def attach_align_device_hook(
     module: torch.nn.Module,
     execution_device: Optional[torch.device] = None,
@@ -383,6 +401,7 @@ def attach_align_device_hook_on_blocks(
             place_submodules=True,
         )
         add_hook_to_module(module, hook)
+        attach_execution_device_hook(module, execution_device[module_name])
     elif module_name in execution_device:
         attach_align_device_hook(
             module,
@@ -395,6 +414,7 @@ def attach_align_device_hook_on_blocks(
         if not hasattr(module, "_hf_hook"):
             hook = AlignDevicesHook(execution_device=execution_device[module_name], io_same_device=(module_name == ""))
             add_hook_to_module(module, hook)
+        attach_execution_device_hook(module, execution_device[module_name])
     elif module_name == "":
         hook = AlignDevicesHook(io_same_device=True)
         add_hook_to_module(module, hook)



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Not yet, will have a look before merging (though I don't expect much difference since there is not that many submodules).
diff --git a/.github/workflows/build-docker-images.yml b/.github/workflows/build-docker-images.yml
new file mode 100644
index 000000000..6b7987e51
--- /dev/null
+++ b/.github/workflows/build-docker-images.yml
@@ -0,0 +1,53 @@
+name: Build Docker images (scheduled)
+
+on:
+  repository_dispatch:
+  schedule:
+    - cron: "0 1 * * *"
+
+concurrency:
+  group: docker-image-builds
+  cancel-in-progress: false
+
+jobs:
+  latest-cpu:
+    name: "Latest Accelerate CPU [dev]"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+      - name: Check out code
+        uses: actions/checkout@v2
+      - name: Login to DockerHub
+        uses: docker/login-action@v1
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_PASSWORD }}
+
+      - name: Build and Push CPU
+        uses: docker/build-push-action@v2
+        with:
+          context: ./docker/accelerate-cpu
+          push: true
+          tags: huggingface/accelerate-cpu
+
+  latest-cuda:
+    name: "Latest Accelerate GPU [dev]"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v1
+      - name: Check out code
+        uses: actions/checkout@v2
+      - name: Login to DockerHub
+        uses: docker/login-action@v1
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_PASSWORD }}
+
+      - name: Build and Push GPU
+        uses: docker/build-push-action@v2
+        with:
+          context: ./docker/accelerate-gpu
+          push: true
+          tags: huggingface/accelerate-gpu
\ No newline at end of file
diff --git a/docker/accelerate-cpu/Dockerfile b/docker/accelerate-cpu/Dockerfile
index 06379e69b..c536fb70c 100644
--- a/docker/accelerate-cpu/Dockerfile
+++ b/docker/accelerate-cpu/Dockerfile
@@ -16,7 +16,6 @@ ENV VIRTUAL_ENV=/opt/venv
 RUN python3 -m venv ${VIRTUAL_ENV}
 # Make sure we use the virtualenv
 ENV PATH="${VIRTUAL_ENV}/bin:$PATH"
-
 WORKDIR /workspace
 # Install specific CPU torch wheel to save on space
 RUN python3 -m pip install --upgrade --no-cache-dir pip
@@ -28,6 +27,8 @@ RUN python3 -m pip install --no-cache-dir \
 # Stage 2
 FROM python:3.6-slim AS build-image
 COPY --from=compile-image /opt/venv /opt/venv
+RUN useradd -ms /bin/bash user
+USER user
 
 # Make sure we use the virtualenv
 ENV PATH="/opt/venv/bin:$PATH"
diff --git a/docker/accelerate-gpu/Dockerfile b/docker/accelerate-gpu/Dockerfile
index b371042e1..299f573d8 100644
--- a/docker/accelerate-gpu/Dockerfile
+++ b/docker/accelerate-gpu/Dockerfile
@@ -31,5 +31,8 @@ COPY --from=compile-image /opt/conda /opt/conda
 ENV PATH /opt/conda/bin:$PATH
 
 RUN echo "source activate accelerate" >> /.bashrc
+
+RUN useradd -ms /bin/bash user
+USER user
 # Activate the virtualenv
 CMD ["/bin/bash"]
\ No newline at end of file



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
index 45181a092..c98ce0bb1 100644
--- a/.github/workflows/stale.yml
+++ b/.github/workflows/stale.yml
@@ -3,6 +3,7 @@ name: Stale Bot
 on:
   schedule:
     - cron: "0 15 * * *"
+  workflow_dispatch:
 
 jobs:
   close_stale_issues:
@@ -12,7 +13,7 @@ jobs:
     env:
       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
-    - uses: actions/@checkoutv2
+    - uses: actions/checkout@v2
     
     - name: Setup Python
       uses: actions/setup-python@v1



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/examples/cv_example.py b/examples/cv_example.py
index a80b42d13..483bddac3 100644
--- a/examples/cv_example.py
+++ b/examples/cv_example.py
@@ -73,7 +73,7 @@ def __getitem__(self, idx):
 
 def training_function(config, args):
     # Initialize accelerator
-    accelerator = Accelerator(cpu=args.cpu, mixed_precision=args.mix_precision)
+    accelerator = Accelerator(cpu=args.cpu, mixed_precision=args.mixed_precision)
 
     # Sample hyper-parameters for learning rate, batch size, seed and a few other HPs
     lr = config["lr"]



Discussion: body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/accelerate/pr_388). All of your documentation changes will be reflected on that endpoint.
diff --git a/src/accelerate/scheduler.py b/src/accelerate/scheduler.py
index 8efce8dea..1fac5f012 100644
--- a/src/accelerate/scheduler.py
+++ b/src/accelerate/scheduler.py
@@ -61,7 +61,9 @@ def step(self, *args, **kwargs):
             # num_processes steps per training step
             num_processes = AcceleratorState().num_processes
             for _ in range(num_processes):
-                self.scheduler.step(*args, **kwargs)
+                # Special case when using OneCycle and `drop_last` was not used
+                if getattr(self.scheduler, "total_steps", 0) <= self.scheduler.last_epoch:
+                    self.scheduler.step(*args, **kwargs)
 
     # Passthroughs
     def get_last_lr(self):


Review comments: body: Should we guard around the `last_epoch` attribute here as well somehow?
diff_hunk: @@ -61,7 +61,9 @@ def step(self, *args, **kwargs):
             # num_processes steps per training step
             num_processes = AcceleratorState().num_processes
             for _ in range(num_processes):
-                self.scheduler.step(*args, **kwargs)
+                # Special case when using OneCycle and `drop_last` was not used
+                if getattr(self.scheduler, "total_steps", 0) <= self.scheduler.last_epoch:
body: `last_epoch` is part of the base scheduler object, so it's always present: https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L24
diff_hunk: @@ -61,7 +61,9 @@ def step(self, *args, **kwargs):
             # num_processes steps per training step
             num_processes = AcceleratorState().num_processes
             for _ in range(num_processes):
-                self.scheduler.step(*args, **kwargs)
+                # Special case when using OneCycle and `drop_last` was not used
+                if getattr(self.scheduler, "total_steps", 0) <= self.scheduler.last_epoch:

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/utils/imports.py b/src/accelerate/utils/imports.py
index 3f10b83cf..b492f2f9c 100644
--- a/src/accelerate/utils/imports.py
+++ b/src/accelerate/utils/imports.py
@@ -12,8 +12,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
 import importlib
+import sys
+
+
+# The package importlib_metadata is in a different place, depending on the Python version.
+if sys.version_info < (3, 8):
+    import importlib_metadata
+else:
+    import importlib.metadata as importlib_metadata
 
 
 try:
@@ -45,7 +52,15 @@ def is_tpu_available():
 
 
 def is_deepspeed_available():
-    return importlib.util.find_spec("deepspeed") is not None
+    package_exists = importlib.util.find_spec("deepspeed") is not None
+    # Check we're not importing a "deepspeed" directory somewhere but the actual library by trying to grab the version
+    # AND checking it has an author field in the metadata that is HuggingFace.
+    if package_exists:
+        try:
+            _ = importlib_metadata.metadata("deepspeed")
+            return True
+        except importlib_metadata.PackageNotFoundError:
+            return False
 
 
 def is_tensorflow_available():


Review comments: body: ```suggestion
```
If I'm not mistaken this is unnecessary. You can do `_deepspeed_metadata.get("version")` if it's needed?
diff_hunk: @@ -45,7 +52,18 @@ def is_tpu_available():
 
 
 def is_deepspeed_available():
-    return importlib.util.find_spec("deepspeed") is not None
+    package_exists = importlib.util.find_spec("deepspeed") is not None
+    # Check we're not importing a "deepspeed" directory somewhere but the actual library by trying to grab the version
+    # AND checking it has an author field in the metadata that is HuggingFace.
+    if package_exists:
+        try:
+            _ = importlib_metadata.version("deepspeed")
body: `importlib_metadata.metadata` will also then raise the same error if it's not installed
diff_hunk: @@ -45,7 +52,18 @@ def is_tpu_available():
 
 
 def is_deepspeed_available():
-    return importlib.util.find_spec("deepspeed") is not None
+    package_exists = importlib.util.find_spec("deepspeed") is not None
+    # Check we're not importing a "deepspeed" directory somewhere but the actual library by trying to grab the version
+    # AND checking it has an author field in the metadata that is HuggingFace.
+    if package_exists:
+        try:
+            _ = importlib_metadata.version("deepspeed")

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/commands/launch.py b/src/accelerate/commands/launch.py
index f283dfbc5..ff4522162 100644
--- a/src/accelerate/commands/launch.py
+++ b/src/accelerate/commands/launch.py
@@ -324,7 +324,7 @@ def deepspeed_launcher(args):
     current_env["USE_DEEPSPEED"] = "true"
     current_env["DEEPSPEED_ZERO_STAGE"] = str(args.zero_stage)
     current_env["GRADIENT_ACCUMULATION_STEPS"] = str(args.gradient_accumulation_steps)
-    current_env["DEEPSPEED_OFFLOAD_OPTIMIZER_DEVICE"] = str(args.offload_optimizer_device)
+    current_env["DEEPSPEED_OFFLOAD_OPTIMIZER_DEVICE"] = str(args.offload_optimizer_device).lower()
 
     process = subprocess.Popen(cmd, env=current_env)
     process.wait()



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/data_loader.py b/src/accelerate/data_loader.py
index a827633c0..50aeb57b6 100644
--- a/src/accelerate/data_loader.py
+++ b/src/accelerate/data_loader.py
@@ -326,12 +326,21 @@ class DataLoaderDispatcher(DataLoader):
     """
 
     def __init__(self, dataset, split_batches: bool = False, **kwargs):
+        shuffle = False
+        if version.parse(torch.__version__) >= version.parse("1.11.0"):
+            from torch.utils.data.datapipes.iter.combinatorics import ShufflerIterDataPipe
+
+            # We need to save the shuffling state of the DataPipe
+            if isinstance(dataset, ShufflerIterDataPipe):
+                shuffle = dataset._shuffle_enabled
         super().__init__(dataset, **kwargs)
         self.split_batches = split_batches
         if version.parse(torch.__version__) < version.parse("1.8.0"):
             raise ImportError(
                 "Using `DataLoaderDispatcher` requires PyTorch 1.8.0 minimum. You have {torch.__version__}."
             )
+        if shuffle:
+            torch.utils.data.graph_settings.apply_shuffle_settings(dataset, shuffle=shuffle)
 
     def __iter__(self):
         state = AcceleratorState()


Review comments: body: This will break compatibility with older versions of PyTorch as I believe datapipes were just introduced in PyTorch 1.11.
diff_hunk: @@ -17,6 +17,7 @@
 
 import torch
 from torch.utils.data import BatchSampler, DataLoader, IterableDataset
+from torch.utils.data.datapipes.iter.combinatorics import ShufflerIterDataPipe
body: There should be a versioning test of PyTorch first, then this instance check. :-)
diff_hunk: @@ -326,12 +327,15 @@ class DataLoaderDispatcher(DataLoader):
     """
 
     def __init__(self, dataset, split_batches: bool = False, **kwargs):
+        shuffle = dataset._shuffle_enabled if isinstance(dataset, ShufflerIterDataPipe) else False
body: ```suggestion
        if version.parse(torch.__version__) >= version.parse("1.11.99"):
```
Just in case we get many micro releases ;-)
diff_hunk: @@ -326,12 +326,21 @@ class DataLoaderDispatcher(DataLoader):
     """
 
     def __init__(self, dataset, split_batches: bool = False, **kwargs):
+        shuffle = False
+        if version.parse(torch.__version__) >= version.parse("1.11.0"):
body: Do you mean `"1.10.99"` ?
diff_hunk: @@ -326,12 +326,21 @@ class DataLoaderDispatcher(DataLoader):
     """
 
     def __init__(self, dataset, split_batches: bool = False, **kwargs):
+        shuffle = False
+        if version.parse(torch.__version__) >= version.parse("1.11.0"):
body: Arf, review before coffee. I read > and thought for a minute it needed PyTorch 1.12, you can ignore :-)
diff_hunk: @@ -326,12 +326,21 @@ class DataLoaderDispatcher(DataLoader):
     """
 
     def __init__(self, dataset, split_batches: bool = False, **kwargs):
+        shuffle = False
+        if version.parse(torch.__version__) >= version.parse("1.11.0"):

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 6fd24c321..cb904a9b5 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -881,14 +881,15 @@ def get_state_dict(self, model):
                 is_zero_3 = self.state.deepspeed_plugin.zero_stage == 3
 
         if is_zero_3:
-            state_dict = model._zero3_consolidated_fp16_state_dict()
+            state_dict = model._zero3_consolidated_16bit_state_dict()
         else:
             model = self.unwrap_model(model)
             state_dict = model.state_dict()
 
-        for k in state_dict:
-            if state_dict[k].dtype == torch.float16:
-                state_dict[k] = state_dict[k].float()
+        if state_dict is not None:
+            for k in state_dict:
+                if state_dict[k].dtype == torch.float16:
+                    state_dict[k] = state_dict[k].float()
 
         return state_dict
 



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 6fd24c321..b9780330a 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -881,7 +881,7 @@ def get_state_dict(self, model):
                 is_zero_3 = self.state.deepspeed_plugin.zero_stage == 3
 
         if is_zero_3:
-            state_dict = model._zero3_consolidated_fp16_state_dict()
+            state_dict = model._zero3_consolidated_16bit_state_dict()
         else:
             model = self.unwrap_model(model)
             state_dict = model.state_dict()



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: > Thanks for fixing! Should we have a minimal version of DeepSpeed? I don't think this was always supported.

Hello, yes, we should have minimal version of DeepSpeed. I will club it with all the other changes that I am planning to have to improve the DeeSpeed integration.
diff --git a/src/accelerate/checkpointing.py b/src/accelerate/checkpointing.py
index b0e2c04a2..b043d5d20 100644
--- a/src/accelerate/checkpointing.py
+++ b/src/accelerate/checkpointing.py
@@ -102,7 +102,7 @@ def save_accelerator_state(
     states["torch_cuda_manual_seed"] = torch.cuda.get_rng_state_all()
     # ^^ safe to call this function even if cuda is not available
     if is_tpu_available():
-        states["xm_seed"] = torch.tensor(xm.get_rng_state())
+        states["xm_seed"] = xm.get_rng_state()
     output_states_file = os.path.join(output_dir, states_name)
     torch.save(states, output_states_file)
     logger.info(f"Random states saved in {output_states_file}")



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/launchers.py b/src/accelerate/launchers.py
index cfd8929d9..d3cd2e1a2 100644
--- a/src/accelerate/launchers.py
+++ b/src/accelerate/launchers.py
@@ -80,7 +80,7 @@ def notebook_launcher(function, args=(), num_processes=None, use_fp16=False, mix
     else:
         if num_processes is None:
             raise ValueError(
-                "You have to specify the number of GPUs you would like to use, add `num_process=...` to your call."
+                "You have to specify the number of GPUs you would like to use, add `num_processes=...` to your call."
             )
 
         if num_processes > 1:



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/__init__.py b/src/accelerate/__init__.py
index b74c683b4..cb38abce6 100644
--- a/src/accelerate/__init__.py
+++ b/src/accelerate/__init__.py
@@ -10,6 +10,7 @@
     DeepSpeedPlugin,
     DistributedDataParallelKwargs,
     DistributedType,
+    FullyShardedDataParallelPlugin,
     GradScalerKwargs,
     InitProcessGroupKwargs,
     find_executable_batch_size,
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index ba34587e2..6fd24c321 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -163,15 +163,18 @@ def __init__(
             assert isinstance(
                 deepspeed_plugin, DeepSpeedPlugin
             ), "`deepspeed_plugin` must be a DeepSpeedPlugin object."
+            os.environ["USE_DEEPSPEED"] = "true"  # use DeepSpeed if plugin is provided
+
+        if fsdp_plugin is None:  # init from env variables
+            fsdp_plugin = FullyShardedDataParallelPlugin() if os.environ.get("USE_FSDP", "false") == "true" else None
+        else:
+            if not isinstance(fsdp_plugin, FullyShardedDataParallelPlugin):
+                raise TypeError("`fsdp_plugin` must be a FullyShardedDataParallelPlugin object.")
+            os.environ["USE_FSDP"] = "true"  # use FSDP if plugin is provided
 
         if os.environ.get("USE_FSDP", "false") == "true":
             if version.parse(torch.__version__) < version.parse("1.12.0.dev20220418+cu113"):
                 raise ValueError("FSDP requires PyTorch >= 1.12.0.dev20220418+cu113")
-            if fsdp_plugin is None:  # init from env variables
-                fsdp_plugin = FullyShardedDataParallelPlugin()
-            else:
-                if not isinstance(fsdp_plugin, FullyShardedDataParallelPlugin):
-                    raise TypeError("`fsdp_plugin` must be a FullyShardedDataParallelPlugin object.")
 
         # Kwargs handlers
         self.ddp_handler = None



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Thanks a lot!! @sgugger @pacman100
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index 8ff69e048..01a55e646 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -133,17 +133,24 @@ def get_cluster_input():
         main_training_function = "main"
 
     if distributed_type in [DistributedType.MULTI_CPU, DistributedType.MULTI_GPU, DistributedType.TPU]:
-        machine_type = str(distributed_type).split(".")[0].replace("MULTI_", "")
+        machine_type = str(distributed_type).split(".")[1].replace("MULTI_", "")
         if machine_type == "TPU":
             machine_type += " cores"
         else:
-            machine_type += "'s"
+            machine_type += "(s)"
         num_processes = _ask_field(
             f"How many {machine_type} should be used for distributed training? [1]:",
             lambda x: int(x),
             default=1,
             error_message="Please enter an integer.",
         )
+    elif distributed_type in [DistributedType.FSDP, DistributedType.DEEPSPEED]:
+        num_processes = _ask_field(
+            "How many GPU(s) should be used for distributed training? [1]:",
+            lambda x: int(x),
+            default=1,
+            error_message="Please enter an integer.",
+        )
     else:
         num_processes = 1
 



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index 559fda41c..8ff69e048 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -132,12 +132,20 @@ def get_cluster_input():
     else:
         main_training_function = "main"
 
-    num_processes = _ask_field(
-        "How many devices should be used for (potentially) distributed training? [1]:",
-        lambda x: int(x),
-        default=1,
-        error_message="Please enter an integer.",
-    )
+    if distributed_type in [DistributedType.MULTI_CPU, DistributedType.MULTI_GPU, DistributedType.TPU]:
+        machine_type = str(distributed_type).split(".")[0].replace("MULTI_", "")
+        if machine_type == "TPU":
+            machine_type += " cores"
+        else:
+            machine_type += "'s"
+        num_processes = _ask_field(
+            f"How many {machine_type} should be used for distributed training? [1]:",
+            lambda x: int(x),
+            default=1,
+            error_message="Please enter an integer.",
+        )
+    else:
+        num_processes = 1
 
     if distributed_type != DistributedType.TPU:
         mixed_precision = _ask_field(



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index cfab0f7f8..559fda41c 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -133,7 +133,7 @@ def get_cluster_input():
         main_training_function = "main"
 
     num_processes = _ask_field(
-        "How many processes in total will you use? [1]: ",
+        "How many devices should be used for (potentially) distributed training? [1]:",
         lambda x: int(x),
         default=1,
         error_message="Please enter an integer.",


Review comments: body: ```suggestion
        "How many devices should be used for (potentially) distributed training? [1]:",
```
diff_hunk: @@ -133,7 +133,7 @@ def get_cluster_input():
         main_training_function = "main"
 
     num_processes = _ask_field(
-        "How many processes in total will you use? [1]: ",
+        "How many devices should be used for (potentially) distributed training? [1:]",

Discussion: body: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/accelerate/pr_343). All of your documentation changes will be reflected on that endpoint.
diff --git a/docs/source/index.mdx b/docs/source/index.mdx
index c41ae4a0e..9d9000097 100644
--- a/docs/source/index.mdx
+++ b/docs/source/index.mdx
@@ -52,7 +52,7 @@ Changing it to work with accelerate is really easy and only adds a few lines of
 + device = accelerator.device
   my_model.to(device)
   # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
++ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
 +     my_model, my_optimizer, my_training_dataloader
 + )
 



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: It's the least I can do, thank's for merging.
diff --git a/src/accelerate/memory_utils.py b/src/accelerate/memory_utils.py
index 4ffc5d10a..422ead770 100644
--- a/src/accelerate/memory_utils.py
+++ b/src/accelerate/memory_utils.py
@@ -82,5 +82,7 @@ def decorator(*args, **kwargs):
                     gc.collect()
                     torch.cuda.empty_cache()
                     batch_size //= 2
+                else:
+                    raise
 
     return decorator
diff --git a/tests/test_memory_utils.py b/tests/test_memory_utils.py
index cc1b3e827..42fa151d0 100644
--- a/tests/test_memory_utils.py
+++ b/tests/test_memory_utils.py
@@ -80,3 +80,12 @@ def mock_training_loop_function(batch_size, arg1, arg2):
             mock_training_loop_function(128, "hello", "world")
             self.assertIn("Batch size was passed into `f`", cm.exception.args[0])
             self.assertIn("`f(arg1='hello', arg2='world')", cm.exception.args[0])
+
+    def test_any_other_error(self):
+        @find_executable_batch_size(starting_batch_size=16)
+        def mock_training_loop_function(batch_size):
+            raise ValueError("Oops, we had an error!")
+
+        with self.assertRaises(ValueError) as cm:
+            mock_training_loop_function()
+            self.assertIn("Oops, we had an error!", cm.exception.args[0])



Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/memory_utils.py b/src/accelerate/memory_utils.py
index 73a298caf..4ffc5d10a 100644
--- a/src/accelerate/memory_utils.py
+++ b/src/accelerate/memory_utils.py
@@ -73,6 +73,8 @@ def decorator(*args, **kwargs):
                 f"Remove this as the decorator already does so: `{function.__name__}({arg_str})`"
             )
         while True:
+            if batch_size == 0:
+                raise RuntimeError("No executable batch size found, reached zero.")
             try:
                 return function(batch_size, *args, **kwargs)
             except Exception as e:
@@ -80,7 +82,5 @@ def decorator(*args, **kwargs):
                     gc.collect()
                     torch.cuda.empty_cache()
                     batch_size //= 2
-                else:
-                    raise
 
     return decorator
diff --git a/tests/test_memory_utils.py b/tests/test_memory_utils.py
index 510d79c71..cc1b3e827 100644
--- a/tests/test_memory_utils.py
+++ b/tests/test_memory_utils.py
@@ -50,6 +50,26 @@ def mock_training_loop_function(batch_size, arg1):
         self.assertListEqual(batch_sizes, [128, 64, 32, 16, 8])
         self.assertListEqual([bs, arg1], [8, "hello"])
 
+    def test_start_zero(self):
+        @find_executable_batch_size(starting_batch_size=0)
+        def mock_training_loop_function(batch_size):
+            pass
+
+        with self.assertRaises(RuntimeError) as cm:
+            mock_training_loop_function()
+            self.assertIn("No executable batch size found, reached zero.", cm.exception.args[0])
+
+    def test_approach_zero(self):
+        @find_executable_batch_size(starting_batch_size=16)
+        def mock_training_loop_function(batch_size):
+            if batch_size > 0:
+                raise_fake_out_of_memory()
+            pass
+
+        with self.assertRaises(RuntimeError) as cm:
+            mock_training_loop_function()
+            self.assertIn("No executable batch size found, reached zero.", cm.exception.args[0])
+
     def test_verbose_guard(self):
         @find_executable_batch_size(starting_batch_size=128)
         def mock_training_loop_function(batch_size, arg1, arg2):


Review comments: body: ```suggestion
    def test_start_zero(self):
```
diff_hunk: @@ -50,6 +50,26 @@ def mock_training_loop_function(batch_size, arg1):
         self.assertListEqual(batch_sizes, [128, 64, 32, 16, 8])
         self.assertListEqual([bs, arg1], [8, "hello"])
 
+    def test_zero(self):

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index 69464ea0b..f9d35ea48 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -64,7 +64,7 @@ def get_cluster_input():
     else:
         use_cpu = False
 
-    deepspeed_config = None
+    deepspeed_config = {}
     if distributed_type in [DistributedType.MULTI_GPU, DistributedType.NO]:
         use_deepspeed = _ask_field(
             "Do you want to use DeepSpeed? [yes/NO]: ",
@@ -78,7 +78,6 @@ def get_cluster_input():
                 is_deepspeed_available()
             ), "DeepSpeed is not installed => run `pip3 install deepspeed` or build it from source"
 
-        deepspeed_config = {}
         if distributed_type == DistributedType.DEEPSPEED:
             deepspeed_config["zero_stage"] = _ask_field(
                 "What should be your DeepSpeed's ZeRO optimization stage (0, 1, 2, 3)? [2]: ",
@@ -99,6 +98,7 @@ def get_cluster_input():
                 default=1,
             )
 
+    fsdp_config = {}
     if distributed_type in [DistributedType.MULTI_GPU]:
         use_fsdp = _ask_field(
             "Do you want to use FullyShardedDataParallel? [yes/NO]: ",
@@ -108,7 +108,6 @@ def get_cluster_input():
         )
         if use_fsdp:
             distributed_type = DistributedType.FSDP
-        fsdp_config = {}
         if distributed_type == DistributedType.FSDP:
             fsdp_config["sharding_strategy"] = _ask_field(
                 "What should be your sharding strategy ([1] FULL_SHARD, [2] SHARD_GRAD_OP)? [1]: ",


Review comments: body: ```suggestion
    fsdp_config = {}
```
Can you remove the same thing at line 112 as a result?
diff_hunk: @@ -99,6 +99,7 @@ def get_cluster_input():
                 default=1,
             )
 
+    fsdp_config = None

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._