diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
index 0da843ec2..84c4e03f6 100644
--- a/docs/source/_toctree.yml
+++ b/docs/source/_toctree.yml
@@ -33,6 +33,8 @@
     title: How to avoid CUDA Out-of-Memory
   - local: usage_guides/sagemaker
     title: Using Accelerate on SageMaker
+  - local: usage_guides/mps
+    title: How to use Apple Silicon M1 GPUs
   title: How-To Guides
 - sections:
   - local: concept_guides/gradient_synchronization
diff --git a/docs/source/usage_guides/mps.mdx b/docs/source/usage_guides/mps.mdx
new file mode 100644
index 000000000..43a00f104
--- /dev/null
+++ b/docs/source/usage_guides/mps.mdx
@@ -0,0 +1,81 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+**Pre-requisites**: To install torch with mps support, 
+please follow this nice medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1).
+
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query:
+
+```
+ Which type of machine are you using?. 
+ ```
+
+This will generate a config file that will be used automatically to properly set 
+the default options when doing `accelerate launch`, such as the one shown below:
+
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
+
+After this configuration has been made, here is how you run the CV example 
+(from the root of the repo) with MPS enabled:
+
+```bash
+accelerate launch /examples/cv_example.py --data_dir images
+```
+
+## A few caveats to be aware of
+
+1. For `nlp_example.py` the metrics are too bad when compared to CPU-only training. 
+This means certain operations in BERT model are going wrong using mps device and this needs to be fixed by PyTorch.
+2. Distributed setups `gloo` and `nccl` are not working with `mps` device. 
+This means that currently only single GPU of `mps` device type can be used.
+
+Finally, please, remember that, 🤗 `Accelerate` only integrates MPS backend, therefore if you
+have any problems or questions with regards to MPS backend usage, please, file an issue with [PyTorch GitHub](https://github.com/pytorch/pytorch/issues).
\ No newline at end of file
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index bbd8db435..9cd4b7da3 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -29,7 +29,7 @@
 from .logging import get_logger
 from .optimizer import AcceleratedOptimizer
 from .scheduler import AcceleratedScheduler
-from .state import AcceleratorState, GradientState
+from .state import AcceleratorState, GradientState, parse_flag_from_env
 from .tracking import LOGGER_TYPE_TO_CLASS, GeneralTracker, filter_trackers
 from .utils import (
     MODEL_NAME,
@@ -282,7 +282,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and not parse_flag_from_env("USE_MPS_DEVICE"):
                 raise ValueError(err.format(mode="fp16", requirement="a GPU"))
             kwargs = self.scaler_handler.to_kwargs() if self.scaler_handler is not None else {}
             if self.distributed_type == DistributedType.FSDP:
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index ba677e3eb..1ba5839e2 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -28,9 +28,9 @@
 
 def get_cluster_input():
     distributed_type = _ask_field(
-        "Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU): ",
+        "Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU [4] MPS): ",
         _convert_distributed_mode,
-        error_message="Please enter 0, 1, 2 or 3.",
+        error_message="Please enter 0, 1, 2, 3 or 4.",
     )
 
     machine_rank = 0
diff --git a/src/accelerate/commands/config/config_utils.py b/src/accelerate/commands/config/config_utils.py
index fe72be267..9dd1f4c99 100644
--- a/src/accelerate/commands/config/config_utils.py
+++ b/src/accelerate/commands/config/config_utils.py
@@ -37,7 +37,7 @@ def _convert_compute_environment(value):
 
 def _convert_distributed_mode(value):
     value = int(value)
-    return DistributedType(["NO", "MULTI_CPU", "MULTI_GPU", "TPU"][value])
+    return DistributedType(["NO", "MULTI_CPU", "MULTI_GPU", "TPU", "MPS"][value])
 
 
 def _convert_sagemaker_distributed_mode(value):
diff --git a/src/accelerate/commands/launch.py b/src/accelerate/commands/launch.py
index 599e710f0..863f903c1 100644
--- a/src/accelerate/commands/launch.py
+++ b/src/accelerate/commands/launch.py
@@ -63,6 +63,12 @@ def launch_command_parser(subparsers=None):
         action="store_true",
         help="Whether or not this should launch a distributed GPU training.",
     )
+    parser.add_argument(
+        "--use_mps_device",
+        default=False,
+        action="store_true",
+        help="Whether or not this should use MPS-enabled GPU device on MacOS machines.",
+    )
     parser.add_argument(
         "--use_deepspeed",
         default=False,
@@ -320,6 +326,7 @@ def simple_launcher(args):
 
     current_env = os.environ.copy()
     current_env["USE_CPU"] = str(args.cpu or args.use_cpu)
+    current_env["USE_MPS_DEVICE"] = str(args.use_mps_device)
     if args.num_machines > 1:
         current_env["MASTER_ADDR"] = args.main_process_ip
         current_env["MASTER_PORT"] = str(args.main_process_port)
@@ -752,11 +759,18 @@ def launch_command(args):
     # Get the default from the config file.
     if args.config_file is not None or os.path.isfile(default_config_file) and not args.cpu:
         defaults = load_config_from_file(args.config_file)
-        if not args.multi_gpu and not args.tpu and not args.use_deepspeed and not args.use_fsdp:
+        if (
+            not args.multi_gpu
+            and not args.tpu
+            and not args.use_deepspeed
+            and not args.use_fsdp
+            and not args.use_mps_device
+        ):
             args.use_deepspeed = defaults.distributed_type == DistributedType.DEEPSPEED
             args.multi_gpu = defaults.distributed_type == DistributedType.MULTI_GPU
             args.tpu = defaults.distributed_type == DistributedType.TPU
             args.use_fsdp = defaults.distributed_type == DistributedType.FSDP
+            args.use_mps_device = defaults.distributed_type == DistributedType.MPS
         if defaults.compute_environment == ComputeEnvironment.LOCAL_MACHINE:
             # Update args with the defaults
             for name, attr in defaults.__dict__.items():
diff --git a/src/accelerate/state.py b/src/accelerate/state.py
index a4f6bac2a..bd339048f 100644
--- a/src/accelerate/state.py
+++ b/src/accelerate/state.py
@@ -206,7 +206,26 @@ def __init__(
                 self.distributed_type = DistributedType.NO
                 self.num_processes = 1
                 self.process_index = self.local_process_index = 0
-                self.device = torch.device("cuda" if torch.cuda.is_available() and not cpu else "cpu")
+                if parse_flag_from_env("USE_MPS_DEVICE") and not cpu:
+                    if not torch.backends.mps.is_available():
+                        if not torch.backends.mps.is_built():
+                            raise AssertionError(
+                                "MPS not available because the current PyTorch install was not "
+                                "built with MPS enabled. Please install torch version >=1.12.0 on "
+                                "your Apple silicon Mac running macOS 12.3 or later with a native "
+                                "version (arm64) of Python"
+                            )
+                        else:
+                            raise AssertionError(
+                                "MPS not available because the current MacOS version is not 12.3+ "
+                                "and/or you do not have an MPS-enabled device on this machine."
+                            )
+                    else:
+                        self.device = torch.device("mps")
+                elif cpu or not torch.cuda.is_available():
+                    self.device = torch.device("cpu")
+                else:
+                    self.device = torch.device("cuda")
                 self.mixed_precision = mixed_precision
             self.initialized = True
 
diff --git a/src/accelerate/utils/dataclasses.py b/src/accelerate/utils/dataclasses.py
index 350d95320..50c80adb1 100644
--- a/src/accelerate/utils/dataclasses.py
+++ b/src/accelerate/utils/dataclasses.py
@@ -123,6 +123,7 @@ class DistributedType(str, enum.Enum):
     DEEPSPEED = "DEEPSPEED"
     FSDP = "FSDP"
     TPU = "TPU"
+    MPS = "MPS"
 
 
 class SageMakerDistributedType(str, enum.Enum):


Review comments: body: Can we do something similar to this [here](https://github.com/huggingface/accelerate/blob/main/src/accelerate/test_utils/testing.py#L94-L99) so that cuda tests can also be ran? As so far tests are only running on the CPU here 😄 
diff_hunk: @@ -272,7 +272,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and os.environ.get("USE_MPS_DEVICE", "false") == "false":
body: There might be other truthy values we're not catching here, can se use something like `parse_flag_from_env`?
diff_hunk: @@ -272,7 +272,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and os.environ.get("USE_MPS_DEVICE", "false") == "false":
body: ```suggestion
    current_env["USE_MPS_DEVICE"] = str(args.use_mps_device)
```
We can't rely on users always setting lowercased values, so we should handle any casing.
diff_hunk: @@ -320,6 +326,7 @@ def simple_launcher(args):
 
     current_env = os.environ.copy()
     current_env["USE_CPU"] = str(args.cpu or args.use_cpu)
+    current_env["USE_MPS_DEVICE"] = str(args.use_mps_device).lower()
body: Give instructions here maybe?
diff_hunk: @@ -208,7 +208,24 @@ def __init__(
                 self.distributed_type = DistributedType.NO
                 self.num_processes = 1
                 self.process_index = self.local_process_index = 0
-                self.device = torch.device("cuda" if torch.cuda.is_available() and not cpu else "cpu")
+                if os.environ.get("USE_MPS_DEVICE", "false") == "true" and not cpu:
+                    if not torch.backends.mps.is_available():
+                        if not torch.backends.mps.is_built():
+                            raise AssertionError(
+                                "MPS not available because the current PyTorch install was not "
+                                "built with MPS enabled."
body: Hello, `use_mps_device` is a boolean argument using argparse action, so I think this should be fine.
diff_hunk: @@ -320,6 +326,7 @@ def simple_launcher(args):
 
     current_env = os.environ.copy()
     current_env["USE_CPU"] = str(args.cpu or args.use_cpu)
+    current_env["USE_MPS_DEVICE"] = str(args.use_mps_device).lower()
body: Done.
diff_hunk: @@ -208,7 +208,24 @@ def __init__(
                 self.distributed_type = DistributedType.NO
                 self.num_processes = 1
                 self.process_index = self.local_process_index = 0
-                self.device = torch.device("cuda" if torch.cuda.is_available() and not cpu else "cpu")
+                if os.environ.get("USE_MPS_DEVICE", "false") == "true" and not cpu:
+                    if not torch.backends.mps.is_available():
+                        if not torch.backends.mps.is_built():
+                            raise AssertionError(
+                                "MPS not available because the current PyTorch install was not "
+                                "built with MPS enabled."
body: Done.
diff_hunk: @@ -320,6 +326,7 @@ def simple_launcher(args):
 
     current_env = os.environ.copy()
     current_env["USE_CPU"] = str(args.cpu or args.use_cpu)
+    current_env["USE_MPS_DEVICE"] = str(args.use_mps_device).lower()
body: Done.
diff_hunk: @@ -272,7 +272,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and os.environ.get("USE_MPS_DEVICE", "false") == "false":
body: Hello, I didn't understand. Let's take this offline to better understand.
diff_hunk: @@ -272,7 +272,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and os.environ.get("USE_MPS_DEVICE", "false") == "false":
body: Specify what specific question related to MPS needs to be answered here :) 
diff_hunk: @@ -0,0 +1,93 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this. 
+New device `mps` maps computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer Official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked. This will generate a config file that will be used automatically to properly set the
+default options when doing
body: Resolving based on Sylvains note about the tests
diff_hunk: @@ -272,7 +272,7 @@ def __init__(
             self.native_amp = is_torch_version(">=", "1.6")
             if not self.native_amp:
                 raise ValueError(err.format(mode="fp16", requirement="PyTorch >= 1.6"))
-            if not torch.cuda.is_available():
+            if not torch.cuda.is_available() and os.environ.get("USE_MPS_DEVICE", "false") == "false":
body: Add a line under each header, I believe this is why the URLs aren't showing correctly and why everything's formatting seems to be broken: https://moon-ci-docs.huggingface.co/docs/accelerate/pr_596/en/usage_guides/mps#accelerated-pytorch-training-on-mac
diff_hunk: @@ -0,0 +1,93 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
body: ```suggestion
Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
```
diff_hunk: @@ -0,0 +1,93 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this. 
+New device `mps` maps computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
body: Done.
diff_hunk: @@ -0,0 +1,93 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
body: Done

diff_hunk: @@ -0,0 +1,93 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this. 
+New device `mps` maps computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer Official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked. This will generate a config file that will be used automatically to properly set the
+default options when doing
body: We can break this up a bit to make it more digestible/readable + syntax highlighting

```suggestion
```python
# installing torch with m1 support on mac
# install latest python from https://www.python.org/downloads/release/python-3106/
# check the platform
import platform
platform.platform()
```
```python out
'macOS-12.5-arm64-arm-64bit' 
```
```python
# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
# install torch 1.12 via the below command
# pip3 install torch torchvision torchaudio
# test the `mps` device support
import torch
torch.has_mps
```
```python out
True
```
```python
a = torch.Tensor([10,11])
a.to("mps")
```
```python out
/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
tensor([10.0000, 11.0000], device='mps:0')
```
```
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
body: I can't make a suggestion for this one, but I recommend something like this:

and answer the questions asked, specifically choose `MPS` for the query:
```
Which type of machine are you using?
```
This will generate a config file that will be used automatically to properly set the
default options when doing
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query `Which type of machine are you using?` . 
+This will generate a config file that will be used automatically to properly set the
+default options when doing
body: Move this section to after you've ran `accelerate config` and state that this is what the generated configuration looks like
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query `Which type of machine are you using?` . 
+This will generate a config file that will be used automatically to properly set the
+default options when doing
+
+```bash
+accelerate launch my_script.py --args_to_my_script
+```
+
+For instance, here is how you would run the CV example (from the root of the repo) with MPS enabled:
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
body: I feel the above block is fine as it might be too distracting to have multiple blocks which can break the flow when the user is reading 
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
body: But then it would be breaking the flow of commands. I have followed generic commands `accelerate config` and `accelerate launch script.py args` with specific example so that user can quickly get the gist. 
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query `Which type of machine are you using?` . 
+This will generate a config file that will be used automatically to properly set the
+default options when doing
+
+```bash
+accelerate launch my_script.py --args_to_my_script
+```
+
+For instance, here is how you would run the CV example (from the root of the repo) with MPS enabled:
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
body: If you're using comments inside a code-block in the documentation, you are doing something wrong; you can put the comments as sentences **then** enter a code-block. Also you should use the doctest syntax to highlight the inputs and outputs, as it's not clear otherwise.
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
body: I'd probably change it to be:

```
This will generate a config file that will be used automatically to properly set the default options when doing `accelerate launch`, such as the one shown below:
```
And just don't show the full `accelerate launch` snippet, since here we want to focus on the config not the launch.

The next part would then say:
```
After this configuration has been made, here is how you run the CV example (from the root of the repo) with MPS enabled:
```

This way each part is focused on what matters to that section
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query `Which type of machine are you using?` . 
+This will generate a config file that will be used automatically to properly set the
+default options when doing
+
+```bash
+accelerate launch my_script.py --args_to_my_script
+```
+
+For instance, here is how you would run the CV example (from the root of the repo) with MPS enabled:
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
body: Oh, okay, then maybe I can just leave it at the link to the medium blogpost instead of summarizing it this way. 
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
body: Done. Removed this entire block and just left it with a link to the medium blogpost explaining this in detail. 
diff_hunk: @@ -0,0 +1,100 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+### Pre-requisites: Installing torch with mps support
+
+Following this great medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1),
+below are the steps to make sure torch is installed with `mps` support.
+
+```
+# installing torch with m1 support on mac
+# install latest python from https://www.python.org/downloads/release/python-3106/
+# check the platform
+import platform
+platform.platform()
+'macOS-12.5-arm64-arm-64bit' 
+# This is compatible as the macOS version is above 12.3 and it is the ARM64 native version of Python
+# install torch 1.12 via the below command
+# pip3 install torch torchvision torchaudio
+# test the `mps` device support
+import torch
+torch.has_mps
+True
+a = torch.Tensor([10,11])
+a.to("mps")
+/Users/mac/ml/lib/python3.10/site-packages/torch/_tensor_str.py:103: UserWarning: The operator 'aten::bitwise_and.Tensor_out' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at  /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
+  nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
+tensor([10.0000, 11.0000], device='mps:0')
+```
body: This should be rewritten or removed now that the PyTorch issue has been solved (in nightly at least), no?
diff_hunk: @@ -0,0 +1,81 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+**Pre-requisites**: To install torch with mps support, 
+please follow this nice medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1).
+
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query:
+
+```
+ Which type of machine are you using?. 
+ ```
+
+This will generate a config file that will be used automatically to properly set 
+the default options when doing `accelerate launch`, such as the one shown below:
+
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
+
+After this configuration has been made, here is how you run the CV example 
+(from the root of the repo) with MPS enabled:
+
+```bash
+accelerate launch /examples/cv_example.py --data_dir images
+```
+
+## A few caveats to be aware of
+
+1. For `nlp_example.py` the metrics are too bad when compared to CPU-only training. 
+This means certain operations in BERT model are going wrong using mps device and this needs to be fixed by PyTorch.
body: Done. Please refer this PR #629 
diff_hunk: @@ -0,0 +1,81 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerated PyTorch Training on Mac
+
+With PyTorch v1.12 release, developers and researchers can take advantage of Apple silicon GPUs for significantly faster model training. 
+This unlocks the ability to perform machine learning workflows like prototyping and fine-tuning locally, right on Mac.
+Apple's Metal Performance Shaders (MPS) as a backend for PyTorch enables this and can be used via the new `"mps"` device. 
+This will map computational graphs and primitives on the MPS Graph framework and tuned kernels provided by MPS.
+For more information please refer official documents [Introducing Accelerated PyTorch Training on Mac](https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/)
+and [MPS BACKEND](https://pytorch.org/docs/stable/notes/mps.html).
+
+### Benefits of Training and Inference using Apple M1 Chips
+
+1. Enables users to train larger networks or batch sizes locally
+2. Reduces data retrieval latency and provides the GPU with direct access to the full memory store due to unified memory architecture. 
+Therefore, improving end-to-end performance.
+3. Reduces costs associated with cloud-based development or the need for additional local GPUs.
+
+**Pre-requisites**: To install torch with mps support, 
+please follow this nice medium article [GPU-Acceleration Comes to PyTorch on M1 Macs](https://medium.com/towards-data-science/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1).
+
+
+## How it works out of the box
+
+On your machine(s) just run:
+
+```bash
+accelerate config
+```
+
+and answer the questions asked, specifically choose `MPS` for the query:
+
+```
+ Which type of machine are you using?. 
+ ```
+
+This will generate a config file that will be used automatically to properly set 
+the default options when doing `accelerate launch`, such as the one shown below:
+
+```bash
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MPS
+downcast_bf16: 'no'
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: 'no'
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
+
+After this configuration has been made, here is how you run the CV example 
+(from the root of the repo) with MPS enabled:
+
+```bash
+accelerate launch /examples/cv_example.py --data_dir images
+```
+
+## A few caveats to be aware of
+
+1. For `nlp_example.py` the metrics are too bad when compared to CPU-only training. 
+This means certain operations in BERT model are going wrong using mps device and this needs to be fixed by PyTorch.

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: `notebook_launcher` doesn't seem to be modified in order to utilize this change that supports MPS device.