diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 806f0094d..a0278218b 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -1,6 +1,6 @@
 name: Run Tests
 
-on: [pull_request]
+# on: [pull_request]
 
 env:
   HF_HOME: ~/hf_cache
diff --git a/Makefile b/Makefile
index 9cae9d74d..100a1484a 100644
--- a/Makefile
+++ b/Makefile
@@ -58,4 +58,4 @@ test_prod:
 	$(MAKE) test_core
 
 test_rest:
-	python -m pytest -s -v ./tests/test_examples.py::FeatureExamplesTests -k "not by_step and not by_epoch"
\ No newline at end of file
+	python -m pytest -s -v ./tests/test_examples.py::FeatureExamplesTests -k "not by_step and not by_epoch"
diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml
index c2fd77282..0da843ec2 100644
--- a/docs/source/_toctree.yml
+++ b/docs/source/_toctree.yml
@@ -1,38 +1,68 @@
-- sections: 
+- sections:
   - local: index
     title: 🤗 Accelerate
-  - local: quicktour
-    title: Quick tour
-  - local: installation
+  - local: basic_tutorials/install
     title: Installation
-  title: Get started
+  - local: quicktour
+    title: Quicktour
+  title: Getting started
+- sections:
+  - local: basic_tutorials/overview
+    title: Overview
+  - local: basic_tutorials/migration
+    title: Migrating to 🤗 Accelerate
+  - local: basic_tutorials/launch
+    title: Launching distributed code
+  - local: basic_tutorials/notebook
+    title: Launching distributed training from Jupyter Notebooks
+  title: Tutorials
 - sections:
-  - local: big_modeling
-    title: Handling big models
-  - local: gradient_accumulation
-    title: Gradient accumulation
-  - local: sagemaker
-    title: Amazon SageMaker
-  title: Guides
+  - local: usage_guides/gradient_accumulation
+    title: Performing gradient accumulation
+  - local: usage_guides/fsdp
+    title: Fully Sharded Data Parallelism
+  - local: usage_guides/checkpoint
+    title: Saving and loading training states
+  - local: usage_guides/deepspeed
+    title: How to use DeepSpeed
+  - local: usage_guides/tracking
+    title: Using experiment trackers
+  - local: usage_guides/big_modeling
+    title: How to use large models with small resources
+  - local: usage_guides/memory
+    title: How to avoid CUDA Out-of-Memory
+  - local: usage_guides/sagemaker
+    title: Using Accelerate on SageMaker
+  title: How-To Guides
 - sections:
-  - local: accelerator
-    title: Accelerator
-  - local: launcher
-    title: Notebook Launcher
-  - local: kwargs
-    title: Kwargs Handlers
-  - local: checkpoint
-    title: Checkpointing
-  - local: internal
-    title: Internals
-  - local: tracking
-    title: Experiment Tracking
-  - local: fsdp
-    title: Fully Sharded Data Parallel
-  - local: memory
-    title: Memory Utilities
-  - local: deepspeed
-    title: DeepSpeed
-  - local: utilities
-    title: General Utilities
-  title: API Reference
+  - local: concept_guides/gradient_synchronization
+    title: Gradient synchronization
+  - local: concept_guides/deferring_execution
+    title: Executing and deferring jobs
+  - local: concept_guides/training_tpu
+    title: TPU best practices
+  title: Concepts and fundamentals
+- sections: 
+  - local: package_reference/accelerator
+    title: Main Accelerator class
+  - local: package_reference/state
+    title: Stateful configuration classes
+  - local: package_reference/cli
+    title: The Command Line
+  - local: package_reference/torch_wrappers
+    title: Torch wrapper classes
+  - local: package_reference/tracking
+    title: Experiment trackers
+  - local: package_reference/launchers
+    title: Distributed launchers
+  - local: package_reference/deepspeed
+    title: DeepSpeed utilities
+  - local: package_reference/logging
+    title: Logging
+  - local: package_reference/big_modeling
+    title: Working with large models
+  - local: package_reference/kwargs
+    title: Kwargs handlers
+  - local: package_reference/utilities
+    title: Utility functions and classes
+  title: "Reference"
\ No newline at end of file
diff --git a/docs/source/accelerator.mdx b/docs/source/accelerator.mdx
deleted file mode 100644
index 268b09d1f..000000000
--- a/docs/source/accelerator.mdx
+++ /dev/null
@@ -1,65 +0,0 @@
-<!--Copyright 2021 The HuggingFace Team. All rights reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
-an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
-specific language governing permissions and limitations under the License.
--->
-
-# Accelerator
-
-The [`Accelerator`] is the main class provided by 🤗 Accelerate. It serves at the main entrypoint for
-the API. To quickly adapt your script to work on any kind of setup with 🤗 Accelerate just:
-
-1. Initialize an [`Accelerator`] object (that we will call `accelerator` in the rest of this
-   page) as early as possible in your script.
-2. Pass along your model(s), optimizer(s), dataloader(s) to the [`~Accelerator.prepare`] method.
-3. (Optional but best practice) Remove all the `.cuda()` or `.to(device)` in your code and let the
-   `accelerator` handle device placement for you.
-4. Replace the `loss.backward()` in your code by `accelerator.backward(loss)`.
-5. (Optional, when using distributed evaluation) Gather your predictions and labels before storing them or using
-   them for metric computation using [`~Accelerator.gather`].
-
-This is all that is needed in most cases. For more advanced cases or a nicer experience here are the functions you
-should search for and replace by the corresponding methods of your `accelerator`:
-
-- `print` statements should be replaced by [`~Accelerator.print`] to be only printed once per
-  process.
-- Use [`~Accelerator.is_local_main_process`] for statements that should be executed once per server.
-- Use [`~Accelerator.is_main_process`] for statements that should be executed once only.
-- Use [`~Accelerator.wait_for_everyone`] to make sure all processes join that point before continuing
-  (useful before a model save for instance).
-- Use [`~Accelerator.unwrap_model`] to unwrap your model before saving it.
-- Use [`~Accelerator.save`] instead of `torch.save`.
-- Use [`~Accelerator.clip_grad_norm_`] instead of `torch.nn.utils.clip_grad_norm_` and
-  [`~Accelerator.clip_grad_value_`] instead of `torch.nn.utils.clip_grad_value_`.
-
-To perform gradient accumulation use [`~Accelerator.accumulate`] and specify a `gradient_accumulation_steps`. 
-This will also automatically ensure the gradients are synced or unsynced when on multi-device training, check if the step should
-actually be performed, and auto-scale the loss:
-
-```python
-accelerator = Accelerator(gradient_accumulation_steps=2)
-model, optimizer, training_dataloader = accelerator.prepare(model, optimizer, training_dataloader)
-
-for input, label in training_dataloader:
-    with accelerator.accumulate(model):
-        predictions = model(input)
-        loss = loss_function(predictions, label)
-        accelerator.backward(loss)
-        optimizer.step()
-        scheduler.step()
-        optimizer.zero_grad()
-```
-
-<Tip warning={true}>
-
-Using this with `dispatch_batches=True`  (which is the default for iterable datasets) is currently not supported.
-
-</Tip>
-
-[[autodoc]] Accelerator
diff --git a/docs/source/basic_tutorials/install.mdx b/docs/source/basic_tutorials/install.mdx
new file mode 100644
index 000000000..19630f575
--- /dev/null
+++ b/docs/source/basic_tutorials/install.mdx
@@ -0,0 +1,99 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+New features are added every day that haven't been released yet. To try them out yourself, install
+from the GitHub repository:
+
+```bash
+pip install git+https://github.com/huggingface/accelerate
+```
+
+If you're working on contributing to the library or wish to play with the source code and see live 
+results as you run the code, an editable version can be installed from a locally-cloned version of the 
+repository:
+
+```bash
+git clone https://github.com/huggingface/accelerate
+cd accelerate
+pip install -e .
+```
+
+## Configuring 🤗 Accelerate
+
+After installing, you need to configure 🤗 Accelerate for how the current system is setup for training. 
+To do so run the following and answer the questions prompted to you:
+
+```bash
+accelerate config
+```
+
+To write a barebones configuration that doesn't include options such as DeepSpeed configuration or running on TPUs, you can quickly run:
+
+```bash
+python -c "from accelerate.utils import write_basic_config; write_basic_config(mixed_precision='fp16')"
+```
+🤗 Accelerate will automatically utilize the maximum number of GPUs available and set the mixed precision mode.
+
+To check that your configuration looks fine, run:
+
+```bash
+accelerate env
+```
+
+An example output is shown below, which describes two GPUs on a single machine with no mixed precision being used:
+
+```bash
+- `Accelerate` version: 0.11.0.dev0
+- Platform: Linux-5.10.0-15-cloud-amd64-x86_64-with-debian-11.3
+- Python version: 3.7.12
+- Numpy version: 1.19.5
+- PyTorch version (GPU?): 1.12.0+cu102 (True)
+- `Accelerate` default config:
+        - compute_environment: LOCAL_MACHINE
+        - distributed_type: MULTI_GPU
+        - mixed_precision: no
+        - use_cpu: False
+        - num_processes: 2
+        - machine_rank: 0
+        - num_machines: 1
+        - main_process_ip: None
+        - main_process_port: None
+        - main_training_function: main
+        - deepspeed_config: {}
+        - fsdp_config: {}
+```
\ No newline at end of file
diff --git a/docs/source/basic_tutorials/launch.mdx b/docs/source/basic_tutorials/launch.mdx
new file mode 100644
index 000000000..741920f26
--- /dev/null
+++ b/docs/source/basic_tutorials/launch.mdx
@@ -0,0 +1,178 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+      accelerator = Accelerator()
+
+      model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+          model, optimizer, training_dataloader, scheduler
+      )
+
+      for batch in training_dataloader:
+          optimizer.zero_grad()
+          inputs, targets = batch
+          outputs = model(inputs)
+          loss = loss_function(outputs, targets)
+          accelerator.backward(loss)
+          optimizer.step()
+          scheduler.step()
+
++ if __name__ == "__main__":
++     main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+<Tip warning={true}>
+
+  It's recommended you run `accelerate config` before using `accelerate launch` to configure your environment to your liking. 
+  Otherwise 🤗 Accelerate will use very basic defaults depending on your system setup.
+
+</Tip>
+
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you having to remember what each of them are.
+
+<Tip>
+
+  If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`.
+
+</Tip>
+
+You can launch your script quickly by using:
+
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, e.g., if GPUs are available, it will use all of them by default without the mixed precision.
+Here is how you would use all GPUs and train with mixed precision disabled:
+
+```bash
+accelerate launch --multi_gpu {script_name.py} {--arg1} {--arg2} ...
+```
+
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision while avoiding all of the warnings: 
+
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} {--arg1} {--arg2} ...
+```
+
+For a complete list of parameters you can pass in, run:
+
+```bash
+accelerate launch -h
+```
+
+<Tip>
+
+  Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
+
+</Tip>
+
+For a visualization of this difference, that earlier `accelerate launch` on multi-gpu would look something like so with `torchrun`:
+
+```bash
+MIXED_PRECISION="fp16" torchrun --nproc_per_node=2 --num_machines=1 {script_name.py} {--arg1} {--arg2} ...
+```
+
+## Why you should always use `accelerate config`
+
+Why is it useful to the point you should **always** run `accelerate config`? 
+
+Remember that earlier call to `accelerate launch` as well as `torchrun`?
+Post configuration, to run that script with the needed parts you just need to use `accelerate launch` outright, without passing anything else in:
+
+```bash
+accelerate launch {script_name.py} {--arg1} {--arg2} ...
+```
+
+
+## Custom Configurations
+
+As briefly mentioned earlier, `accelerate launch` should be mostly used through combining set configurations 
+made with the `accelerate config` command. These configs are saved to a `default_config.yaml` file in your cache folder for 🤗 Accelerate. 
+This cache folder is located at (with decreasing order of priority):
+
+- The content of your environment variable `HF_HOME` suffixed with `accelerate`.
+- If it does not exist, the content of your environment variable `XDG_CACHE_HOME` suffixed with
+  `huggingface/accelerate`.
+- If this does not exist either, the folder `~/.cache/huggingface/accelerate`.
+
+To have multiple configurations, the flag `--config_file` can be passed to the `accelerate launch` command paired 
+with the location of the custom yaml. 
+
+An example yaml may look something like the following for two GPUs on a single machine using `fp16` for mixed precision:
+```yaml
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MULTI_GPU
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: fp16
+num_machines: 1
+num_processes: 2
+use_cpu: false
+```
+
+Launching a script from the location of that custom yaml file looks like the following:
+```bash
+accelerate launch --config_file {path/to/config/my_config_file.yaml} {script_name.py} {--arg1} {--arg2} ...
+```
\ No newline at end of file
diff --git a/docs/source/basic_tutorials/migration.mdx b/docs/source/basic_tutorials/migration.mdx
new file mode 100644
index 000000000..ab703c960
--- /dev/null
+++ b/docs/source/basic_tutorials/migration.mdx
@@ -0,0 +1,123 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code to use 🤗 Accelerate!
+You'll see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, write out a very basic PyTorch training loop. 
+
+<Tip>
+
+    We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand.
+
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+[`Accelerator`] is the main force behind utilizing all the possible options for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+- device = 'cuda'
++ device = accelerator.device
+  model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`~Accelerator.prepare`]. 🤗 Accelerate will
+make sure everything is setup in the current environment for you to start training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. By default when using `device_placement=True`, all of the objects that can be sent to the right device will be.
+If you need to work with data that isn't passed to [~Accelerator.prepare] but should be on the active device, you should pass in the `device` you made earlier. 
+
+<Tip warning={true}>
+
+    Accelerate will only prepare objects that inherit from their respective PyTorch classes (such as `torch.optim.Optimizer`).
+
+</Tip>
+
+### Modifying the training loop
+
+Finally, three lines of code need to be changed in the training loop. 🤗 Accelerate's DataLoader classes will automatically handle the device placement by default,
+and [`~Accelerator.backward`] should be used for performing the backward pass:
+
+```diff
+-   inputs = inputs.to(device)
+-   targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+-   loss.backward()
++   accelerator.backward(loss)
+```
+
+With that, your training loop is now ready to use 🤗 Accelerate!
+
+## The finished code
+
+Below is the final version of the converted code: 
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
diff --git a/docs/source/basic_tutorials/notebook.mdx b/docs/source/basic_tutorials/notebook.mdx
new file mode 100644
index 000000000..903a992d3
--- /dev/null
+++ b/docs/source/basic_tutorials/notebook.mdx
@@ -0,0 +1,429 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching Multi-Node Training from a Jupyter Environment
+
+This tutorial teaches you how to fine tune a computer vision model with 🤗 Accelerate from a Jupyter Notebook on a distributed system.
+You will also learn how to setup a few requirements needed for ensuring your environment is configured properly, your data has been prepared properly, and finally how to launch training.
+
+<Tip>
+
+    This tutorial is also available as a Jupyter Notebook [here](https://github.com/huggingface/notebooks/blob/main/examples/accelerate_examples/simple_cv_example.ipynb)
+
+</Tip>
+
+## Configuring the Environment
+
+Before any training can be performed, a 🤗 Accelerate config file must exist in the system. Usually this can be done by running the following in a terminal and answering the prompts:
+
+```bash
+accelerate config
+```
+
+However, if general defaults are fine and you are *not* running on a TPU, 🤗Accelerate has a utility to quickly write your GPU configuration into a config file via [`utils.write_basic_config`].
+
+The following code will restart Jupyter after writing the configuration, as CUDA code was called to perform this. 
+
+<Tip warning={true}>
+
+    CUDA can't be initialized more than once on a multi-node system. It's fine to debug in the notebook and have calls to CUDA, but in order to finally train a full cleanup and restart will need to be performed.
+    
+</Tip>
+
+```python
+import os
+from accelerate.utils import write_basic_config
+
+write_basic_config()  # Write a config file
+os._exit(00)  # Restart the notebook
+```
+
+## Preparing the Dataset and Model
+
+Next you should prepare your dataset. As mentioned at earlier, great care should be taken when preparing the `DataLoaders` and model to make sure that **nothing** is put on *any* GPU. 
+
+If you do, it is recommended to put that specific code into a function and call that from within the notebook launcher interface, which will be shown later. 
+
+Make sure the dataset is downloaded based on the directions [here](https://github.com/huggingface/accelerate/tree/main/examples#simple-vision-example)
+
+```python
+import os, re, torch, PIL
+import numpy as np
+
+from torch.optim.lr_scheduler import OneCycleLR
+from torch.utils.data import DataLoader, Dataset
+from torchvision.transforms import Compose, RandomResizedCrop, Resize, ToTensor
+
+from accelerate import Accelerator
+from accelerate.utils import set_seed
+from timm import create_model
+```
+
+First you need to create a function to extract the class name based on a filename:
+
+```python
+import os
+
+data_dir = "../../images"
+fnames = os.listdir(data_dir)
+fname = fnames[0]
+print(fname)
+```
+
+```python out
+beagle_32.jpg
+```
+
+In the case here, the label is `beagle`. Using regex you can extract the label from the filename:
+
+```python
+import re
+
+
+def extract_label(fname):
+    stem = fname.split(os.path.sep)[-1]
+    return re.search(r"^(.*)_\d+\.jpg$", stem).groups()[0]
+```
+
+```python
+extract_label(fname)
+```
+
+And you can see it properly returned the right name for our file:
+
+```python out
+"beagle"
+```
+
+Next a `Dataset` class should be made to handle grabbing the image and the label:
+
+```python
+class PetsDataset(Dataset):
+    def __init__(self, file_names, image_transform=None, label_to_id=None):
+        self.file_names = file_names
+        self.image_transform = image_transform
+        self.label_to_id = label_to_id
+
+    def __len__(self):
+        return len(self.file_names)
+
+    def __getitem__(self, idx):
+        fname = self.file_names[idx]
+        raw_image = PIL.Image.open(fname)
+        image = raw_image.convert("RGB")
+        if self.image_transform is not None:
+            image = self.image_transform(image)
+        label = extract_label(fname)
+        if self.label_to_id is not None:
+            label = self.label_to_id[label]
+        return {"image": image, "label": label}
+```
+
+Now to build the dataset. Outside the training function you can find and declare all the filenames and labels and use them as references inside the 
+launched function:
+
+```python
+fnames = [os.path.join("../../images", fname) for fname in fnames if fname.endswith(".jpg")]
+```
+
+Next gather all the labels:
+
+```python
+all_labels = [extract_label(fname) for fname in fnames]
+id_to_label = list(set(all_labels))
+id_to_label.sort()
+label_to_id = {lbl: i for i, lbl in enumerate(id_to_label)}
+```
+
+Next, you should make a `get_dataloaders` function that will return your built dataloaders for you. As mentioned earlier, if data is automatically 
+sent to the GPU or a TPU device when building your `DataLoaders`, they must be built using this method. 
+
+```python
+def get_dataloaders(batch_size: int = 64):
+    "Builds a set of dataloaders with a batch_size"
+    random_perm = np.random.permutation(len(fnames))
+    cut = int(0.8 * len(fnames))
+    train_split = random_perm[:cut]
+    eval_split = random_perm[:cut]
+
+    # For training a simple RandomResizedCrop will be used
+    train_tfm = Compose([RandomResizedCrop((224, 224), scale=(0.5, 1.0)), ToTensor()])
+    train_dataset = PetsDataset([fnames[i] for i in train_split], image_transform=train_tfm, label_to_id=label_to_id)
+
+    # For evaluation a deterministic Resize will be used
+    eval_tfm = Compose([Resize((224, 224)), ToTensor()])
+    eval_dataset = PetsDataset([fnames[i] for i in eval_split], image_transform=eval_tfm, label_to_id=label_to_id)
+
+    # Instantiate dataloaders
+    train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=batch_size, num_workers=4)
+    eval_dataloader = DataLoader(eval_dataset, shuffle=False, batch_size=batch_size * 2, num_workers=4)
+    return train_dataloader, eval_dataloader
+```
+
+Finally, you should import the scheduler to be used later:
+
+```python
+from torch.optim.lr_scheduler import CosineAnnealingLR
+```
+
+## Writing the Training Function
+
+Now you can build the training loop. [`notebook_launcher`] works by passing in a function to call that will be ran across the distributed system.
+
+Here is a basic training loop for the animal classification problem:
+
+<Tip>
+
+    The code has been split up to allow for explainations on each section. A full version that can be copy and pasted will be available at the end
+
+</Tip>
+
+
+```python
+def training_loop(mixed_precision="fp16", seed: int = 42, batch_size: int = 64):
+    set_seed(seed)
+    accelerator = Accelerator(mixed_precision=mixed_precision)
+```
+
+First you should set the seed and create an [`Accelerator`] object as early in the training loop as possible.
+
+<Tip warning={true}>
+
+    If training on the TPU, your training loop should take in the model as a parameter and it should be instantiated 
+    outside of the training loop function. See the [TPU best practices](../concept_guides/training_tpu) 
+    to learn why
+
+</Tip>
+
+Next you should build your dataloaders and create your model:
+
+```python
+    train_dataloader, eval_dataloader = get_dataloaders(batch_size)
+    model = create_model("resnet50d", pretrained=True, num_classes=len(label_to_id))
+```
+
+<Tip>
+
+    You build the model here so that the seed also controls the new weight initialization
+
+</Tip>
+
+As you are performing transfer learning in this example, the encoder of the model starts out frozen so the head of the model can be 
+trained only initially:
+
+```python
+    for param in model.parameters():
+        param.requires_grad = False
+    for param in model.get_classifier().parameters():
+        param.requires_grad = True
+```
+
+Normalizing the batches of images will make training a little faster:
+
+```python
+    mean = torch.tensor(model.default_cfg["mean"])[None, :, None, None]
+    std = torch.tensor(model.default_cfg["std"])[None, :, None, None]
+```
+
+To make these constants available on the active device, you should set it to the Accelerator's device:
+
+```python
+    mean = mean.to(accelerator.device)
+    std = std.to(accelerator.device)
+```
+
+Next instantiate the rest of the PyTorch classes used for training:
+
+```python
+    optimizer = torch.optim.Adam(params=model.parameters(), lr=3e-2 / 25)
+    lr_scheduler = OneCycleLR(optimizer=optimizer, max_lr=3e-2, epochs=5, steps_per_epoch=len(train_dataloader))
+```
+
+Before passing everything to [`~Accelerator.prepare`].
+
+<Tip>
+
+    There is no specific order to remember, you just need to unpack the objects in the same order you gave them to the prepare method.
+
+</Tip>
+
+```python
+    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
+    )
+```
+
+Now train the model:
+
+```python
+    for epoch in range(5):
+        model.train()
+        for batch in train_dataloader:
+            inputs = (batch["image"] - mean) / std
+            outputs = model(inputs)
+            loss = torch.nn.functional.cross_entropy(outputs, batch["label"])
+            accelerator.backward(loss)
+            optimizer.step()
+            lr_scheduler.step()
+            optimizer.zero_grad()
+```
+
+The evaluation loop will look slightly different compared to the training loop. The number of elements passed as well as the overall 
+total accuracy of each batch will be added to two constants:
+
+```python
+        model.eval()
+        accurate = 0
+        num_elems = 0
+```
+
+Next you have the rest of your standard PyTorch loop:
+
+```python
+        for batch in eval_dataloader:
+            inputs = (batch["image"] - mean) / std
+            with torch.no_grad():
+                outputs = model(inputs)
+            predictions = outputs.argmax(dim=-1)
+```
+
+Before finally the last major difference. 
+
+When performing distributed evaluation, the predictions and labels need to be passed through 
+[`~Accelerator.gather`] so that all of the data is available on the current device and a properly calculated metric can be achieved:
+
+```python
+            accurate_preds = accelerator.gather(predictions) == accelerator.gather(batch["label"])
+            num_elems += accurate_preds.shape[0]
+            accurate += accurate_preds.long().sum()
+```
+
+Now you just need to calculate the actual metric for this problem, and you can print it on the main process using [`~Accelerator.print`]:
+
+```python
+        eval_metric = accurate.item() / num_elems
+        accelerator.print(f"epoch {epoch}: {100 * eval_metric:.2f}")
+```
+
+A full version of this training loop is available below:
+
+```python
+def training_loop(mixed_precision="fp16", seed: int = 42, batch_size: int = 64):
+    set_seed(seed)
+    # Initialize accelerator
+    accelerator = Accelerator(mixed_precision=mixed_precision)
+    # Build dataloaders
+    train_dataloader, eval_dataloader = get_dataloaders(batch_size)
+
+    # Instantiate the model (you build the model here so that the seed also controls new weight initaliziations)
+    model = create_model("resnet50d", pretrained=True, num_classes=len(label_to_id))
+
+    # Freeze the base model
+    for param in model.parameters():
+        param.requires_grad = False
+    for param in model.get_classifier().parameters():
+        param.requires_grad = True
+
+    # You can normalize the batches of images to be a bit faster
+    mean = torch.tensor(model.default_cfg["mean"])[None, :, None, None]
+    std = torch.tensor(model.default_cfg["std"])[None, :, None, None]
+
+    # To make this constant available on the active device, set it to the accelerator device
+    mean = mean.to(accelerator.device)
+    std = std.to(accelerator.device)
+
+    # Intantiate the optimizer
+    optimizer = torch.optim.Adam(params=model.parameters(), lr=3e-2 / 25)
+
+    # Instantiate the learning rate scheduler
+    lr_scheduler = OneCycleLR(optimizer=optimizer, max_lr=3e-2, epochs=5, steps_per_epoch=len(train_dataloader))
+
+    # Prepare everything
+    # There is no specific order to remember, you just need to unpack the objects in the same order you gave them to the
+    # prepare method.
+    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
+    )
+
+    # Now you train the model
+    for epoch in range(5):
+        model.train()
+        for batch in train_dataloader:
+            inputs = (batch["image"] - mean) / std
+            outputs = model(inputs)
+            loss = torch.nn.functional.cross_entropy(outputs, batch["label"])
+            accelerator.backward(loss)
+            optimizer.step()
+            lr_scheduler.step()
+            optimizer.zero_grad()
+
+        model.eval()
+        accurate = 0
+        num_elems = 0
+        for batch in eval_dataloader:
+            inputs = (batch["image"] - mean) / std
+            with torch.no_grad():
+                outputs = model(inputs)
+            predictions = outputs.argmax(dim=-1)
+            accurate_preds = accelerator.gather(predictions) == accelerator.gather(batch["label"])
+            num_elems += accurate_preds.shape[0]
+            accurate += accurate_preds.long().sum()
+
+        eval_metric = accurate.item() / num_elems
+        # Use accelerator.print to print only on the main process.
+        accelerator.print(f"epoch {epoch}: {100 * eval_metric:.2f}")
+```
+
+## Using the notebook_launcher
+
+All that's left is to use the [`notebook_launcher`].
+
+You pass in the function, the arguments (as a tuple), and the number of processes to train on. (See the [documentation](../package_reference/launchers) for more information)
+
+```python
+from accelerate import notebook_launcher
+```
+
+```python
+args = ("fp16", 42, 64)
+notebook_launcher(training_loop, args, num_processes=2)
+```
+
+In the case of running on the TPU, it would look like so:
+
+```python
+model = create_model("resnet50d", pretrained=True, num_classes=len(label_to_id))
+
+args = (model, "fp16", 42, 64)
+notebook_launcher(training_loop, args, num_processes=8)
+```
+
+As it's running it will print the progress as well as state how many devices you ran on. This tutorial was ran with two GPUs:
+
+```python out
+Launching training on 2 GPUs.
+epoch 0: 88.12
+epoch 1: 91.73
+epoch 2: 92.58
+epoch 3: 93.90
+epoch 4: 94.71
+```
+
+And that's it!
+
+## Conclusion
+
+This notebook showed how to perform distributed training from inside of a Jupyter Notebook. Some key notes to remember:
+
+- Make sure to save any code that use CUDA (or CUDA imports) for the function passed to [`notebook_launcher`]
+- Set the `num_processes` to be the number of devices used for training (such as number of GPUs, CPUs, TPUs, etc)
+- If using the TPU, declare your model outside the training loop function
\ No newline at end of file
diff --git a/docs/source/basic_tutorials/overview.mdx b/docs/source/basic_tutorials/overview.mdx
new file mode 100644
index 000000000..59ff9cbae
--- /dev/null
+++ b/docs/source/basic_tutorials/overview.mdx
@@ -0,0 +1,21 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Overview
+
+Welcome to the 🤗 Accelerate tutorials! These introductory guides will help catch you up to speed on working with 🤗 Accelerate.
+You'll learn how to modify your code to have it work with the API seamlessly, how to launch your script properly,
+and more!
+
+These tutorials assume some basic knowledge of Python and familiarity with the PyTorch framework.
+
+If you have any questions about 🤗 Accelerate, feel free to join and ask the community on our [forum](https://discuss.huggingface.co/c/accelerate/18).
\ No newline at end of file
diff --git a/docs/source/concept_guides/deferring_execution.mdx b/docs/source/concept_guides/deferring_execution.mdx
new file mode 100644
index 000000000..4297e2567
--- /dev/null
+++ b/docs/source/concept_guides/deferring_execution.mdx
@@ -0,0 +1,95 @@
+# Deferring Executions
+
+When you run your usual script, instructions are executed in order. Using 🤗 Accelerate to deploy your script on several
+GPUs at the same time introduces a complication: while each process executes all instructions in order, some may be
+faster than others.
+
+You might need to wait for all processes to have reached a certain point before executing a given instruction. For
+instance, you shouldn't save a model before being sure every process is done with training, and you wouldn't want to 
+continue training before all the model weights have been loaded in. To do this, just write the following line in your code:
+
+```
+accelerator.wait_for_everyone()
+```
+
+This instruction will block all the processes that arrive first until all the other processes have reached that
+point (if you run your script on just one GPU or CPU, this won't do anything).
+
+A few example cases for when to use this utility are listed below:
+
+<Tip>
+
+    Some of these are utilized with the [`~Accelerator.main_process_first`] context manager, which utilizes [`~Accelerator.wait_for_everyone`] to 
+    run a particular set of code on the main process beforehand before triggering and launching the other processes
+
+</Tip>
+
+## Downloading a Dataset 
+
+When downloading a dataset, you should download it first on the main process and then loading the cached dataset in afterwards
+
+<Tip>
+
+    `load_dataset` will perform a lock under the hood to stop multiple downloads from happening at once, but if you are downloading something 
+    not using this library you should use this method.
+    
+</Tip>
+
+```python
+with accelerator.main_process_first():
+    datasets = load_dataset("glue", "mrpc")
+```
+
+Under the hood this is the same as calling: 
+
+```python
+# First do something on the main process
+if accelerator.is_main_process:
+    datasets = load_dataset("glue", "mrpc")
+else:
+    accelerator.wait_for_everyone()
+
+# And then send it to the rest of them
+if not accelerator.is_main_process:
+    datasets = load_dataset("glue", "mrpc")
+else:
+    accelerator.wait_for_everyone()
+```
+
+## Saving the `state_dict`
+
+When saving the `state_dict` of the model, since you would normally save one file on just the main process
+you should specify that:
+
+```python
+if accelerator.is_main_process:
+    model = accelerator.unwrap_model(model)
+    torch.save(model.state_dict(), "weights.pth")
+```
+
+## Loading in the `state_dict`
+
+When loading in the `state_dict` to a model, optimizer, or scheduler, you should wait 
+for all workers to have the weights loaded in before moving on to training
+
+```python
+with accelerator.main_process_first():
+    state = torch.load("weights.pth")
+    model.load_state_dict(state)
+```
+
+## Applying a multi-worker CPU operation 
+
+Applying a `map()` operation on multiple workers, such as tokenizing should be done on the 
+main process first, and then propagated to each one. 
+
+```python
+datasets = load_dataset("glue", "mrpc")
+
+with accelerator.main_process_first():
+    tokenized_datasets = datasets.map(
+        tokenize_function,
+        batched=True,
+        remove_columns=["idx", "sentence1", "sentence2"],
+    )
+```
\ No newline at end of file
diff --git a/docs/source/concept_guides/gradient_synchronization.mdx b/docs/source/concept_guides/gradient_synchronization.mdx
new file mode 100644
index 000000000..02b5adf0d
--- /dev/null
+++ b/docs/source/concept_guides/gradient_synchronization.mdx
@@ -0,0 +1,117 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`~Accelerator.prepare`] and passing in your model.
+
+```diff
++ from accelerate import Accelerator
++ accelerator = Accelerator()
+  import torch.nn as nn
+- from torch.nn.parallel import DistributedDataParallel
+
+  model = nn.Linear(10,10)
++ model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this risk slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
+need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
+can you avoid this overhead?
+
+## Solving the slowdown problem
+
+Since you are skipping these batches, their gradients do not need to be synchronized until the point where `.backward()` is actually called. 
+PyTorch cannot automagically tell when you need to do this, but they do provide a tool to help through the [`no_sync`](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync) context manager
+that is added to your model after converting it to DDP.
+
+Under this context manager, PyTorch will skip synchronizing the gradients when `.backward()` is called, and the first call to `.backward()` outside this 
+context manager will trigger the synchronization. See an example below:
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for index, batch in enumerate(dataloader):
+    inputs, targets = batch
+    # Trigger gradient synchronization on the last batch
+    if index != (len(dataloader) - 1):
+        with ddp_model.no_sync():
+            # Gradients only accumulate
+            outputs = ddp_model(inputs)
+            loss = loss_func(outputs)
+            accelerator.backward(loss)
+    else:
+        # Gradients finally sync
+        outputs = ddp_model(inputs)
+        loss = loss_func(outputs)
+        accelerator.backward(loss)
+```
+
+In 🤗 Accelerate to make this an API that can be called no matter the training device (though it may not do anything if you are not in a distributed system!),
+`ddp_model.no_sync` gets replaced with [`~Accelerator.no_sync`] and operates the same way:
+
+```diff
+  ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+  for index, batch in enumerate(dataloader):
+      inputs, targets = batch
+      # Trigger gradient synchronization on the last batch
+      if index != (len(dataloader)-1):
+-         with ddp_model.no_sync():
++         with accelerator.no_sync(model):
+              # Gradients only accumulate
+              outputs = ddp_model(inputs)
+              loss = loss_func(outputs, targets)
+              accelerator.backward(loss)
+      else:
+          # Gradients finally sync
+          outputs = ddp_model(inputs)
+          loss = loss_func(outputs)
+          accelerator.backward(loss)
+```
+
+As you may expect, the [`~Accelerator.accumulate`] function wraps around this conditional check by keeping track of the current batch number, leaving you with the final
+gradient accumulation API:
+
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for batch in dataloader:
+    with accelerator.accumulate(model):
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+```
\ No newline at end of file
diff --git a/docs/source/concept_guides/training_tpu.mdx b/docs/source/concept_guides/training_tpu.mdx
new file mode 100644
index 000000000..32736ce31
--- /dev/null
+++ b/docs/source/concept_guides/training_tpu.mdx
@@ -0,0 +1,164 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Training on TPUs with 🤗 Accelerate
+
+Training on TPUs can be slightly different than training on multi-gpu, even with 🤗 Accelerate. This guide aims to show you 
+where you should be careful and why, as well as the best practices in general.
+
+## Training in a Notebook
+
+The main carepoint when training on TPUs comes from the [`notebook_launcher`]. As mentioned in the [notebook tutorial](../usage_guides/notebook), you need to 
+restructure your training code into a function that can get passed to the [`notebook_launcher`] function and be careful about not declaring any tensors on the GPU.
+
+While on a TPU that last part is not as important, a critical part to understand is that when you launch code from a notebook you do so through a process called **forking**. 
+When launching from the command-line, you perform **spawning**, where a python process is not currently running and you *spawn* a new process in. Since your Jupyter notebook is already 
+utilizing a python process, you need to *fork* a new process from it to launch your code. 
+
+Where this becomes important is in regards to declaring your model. On forked TPU processes, it is recommended that you instantiate your model *once* and pass this into your 
+training function. This is different than training on GPUs where you create `n` models that have their gradients synced and back-propagated at certain moments. Instead one 
+model instance is shared between all the nodes and it is passed back and forth. This is important especially when training on low-resource TPUs such as those provided in Kaggle kernels or
+on Google Colaboratory. 
+
+Below is an example of a training function passed to the [`notebook_launcher`] if training on CPUs or GPUs:
+
+<Tip>
+
+    This code snippet is based off the one from the `simple_nlp_example` notebook found [here](https://github.com/huggingface/notebooks/blob/main/examples/accelerate/simple_nlp_example.ipynb) with slight 
+    modifications for the sake of simplicity
+
+</Tip>
+
+```python
+def training_function():
+    # Initialize accelerator
+    accelerator = Accelerator()
+    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+    train_dataloader, eval_dataloader = create_dataloaders(
+        train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+    )
+
+    # Instantiate optimizer
+    optimizer = AdamW(params=model.parameters(), lr=hyperparameters["learning_rate"])
+
+    # Prepare everything
+    # There is no specific order to remember, we just need to unpack the objects in the same order we gave them to the
+    # prepare method.
+    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader
+    )
+
+    num_epochs = hyperparameters["num_epochs"]
+    # Now we train the model
+    for epoch in range(num_epochs):
+        model.train()
+        for step, batch in enumerate(train_dataloader):
+            outputs = model(**batch)
+            loss = outputs.loss
+            accelerator.backward(loss)
+
+            optimizer.step()
+            optimizer.zero_grad()
+```
+
+```python
+from accelerate import notebook_launcher
+
+notebook_launcher(training_function)
+```
+
+<Tip>
+
+    The `notebook_launcher` will default to 8 processes if 🤗 Accelerate has been configured for a TPU
+
+</Tip>
+
+If you use this example and declare the model *inside* the training loop, then on a low-resource system you will potentially see an error 
+like:
+
+```
+ProcessExitedException: process 0 terminated with signal SIGSEGV
+```
+
+This error is *extremely* cryptic but the basic explaination is you ran out of system RAM. You can avoid this entirely by reconfiguring the training function to 
+accept a single `model` argument, and declare it in an outside cell:
+
+```python
+# In another Jupyter cell
+model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+```
+
+```diff
++ def training_function(model):
+      # Initialize accelerator
+      accelerator = Accelerator()
+-     model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+      train_dataloader, eval_dataloader = create_dataloaders(
+          train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+      )
+  ...
+```
+
+And finally calling the training function with:
+
+```diff
+  from accelerate import notebook_launcher
+- notebook_launcher(training_function)
++ notebook_launcher(training_function, (model,))
+```
+
+<Tip>
+
+    The above workaround is only needed when launching a TPU instance from a Jupyter Notebook on a low-resource server such as Google Colaboratory or Kaggle. If 
+    using a script or launching on a much beefier server declaring the model beforehand is not needed.
+
+</Tip>
+
+## Mixed Precision and Global Variables 
+
+As mentioned in the [mixed precision tutorial](../usage_guides/mixed_precision), 🤗 Accelerate supports fp16 and bf16, both of which can be used on TPUs.
+That being said, ideally `bf16` should be utilized as it is extremely efficient to use.
+
+There are two "layers" when using `bf16` and 🤗 Accelerate on TPUs, at the base level and at the operation level. 
+
+At the base level, this is enabled when passing `mixed_precision="bf16"` to `Accelerator`, such as:
+```python
+accelerator = Accelerator(mixed_precision="bf16")
+```
+By default this will cast `torch.float` and `torch.double` to `bfloat16` on TPUs. 
+The specific configuration being set is an environmental variable of `XLA_USE_BF16` is set to `1`.
+
+There is a futher configuration you can perform which is setting the `XLA_DOWNCAST_BF16` environmental variable. If set to `1`, then 
+`torch.float` is `bfloat16` and `torch.double` is `float32`.
+
+This is performed in the `Accelerator` object when passing `downcast_bf16=True`:
+```python
+accelerator = Accelerator(mixed_precision="bf16", downcast_bf16=True)
+```
+
+Using downcasting instead of bf16 everywhere is good for when you are trying to calculate metrics, log values, and more where raw bf16 tensors would be unusable. 
+
+## Training Times on TPUs
+
+As you launch your script, you may notice that training seems exceptionally slow at first. This is because TPUs
+first run through a few batches of data to see how much memory to allocate before finally utilizing this configured 
+memory allocation extremely efficiently. 
+
+If you notice that your evaluation code to calculate the metrics of your model takes longer due to a larger batch size being used, 
+it is recommended to keep the batch size the same as the training data if it is too slow. Otherwise the memory will reallocate to this 
+new batch size after the first few iterations. 
+
+<Tip>
+
+    Just because the memory is allocated does not mean it will be used or that the batch size will increase when going back to your training dataloader.
+
+</Tip>
\ No newline at end of file
diff --git a/docs/source/index.mdx b/docs/source/index.mdx
index 16593d554..3f0f58802 100644
--- a/docs/source/index.mdx
+++ b/docs/source/index.mdx
@@ -1,4 +1,4 @@
-<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
 the License. You may obtain a copy of the License at
@@ -12,121 +12,60 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
-
-## Features
-
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
 ```diff
 + from accelerate import Accelerator
-
 + accelerator = Accelerator()
-  # Use the device given by the *accelerator* object.
-+ device = accelerator.device
-  my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
-+     my_model, my_optimizer, my_training_dataloader
+
++ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++     model, optimizer, training_dataloader, scheduler
 + )
 
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
+  for batch in training_dataloader:
+      optimizer.zero_grad()
       inputs, targets = batch
       inputs = inputs.to(device)
       targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
+      outputs = model(inputs)
+      loss = loss_function(outputs, targets)
 +     accelerator.backward(loss)
-      my_optimizer.step()
+      optimizer.step()
+      scheduler.step()
 ```
 
-and with this, your script can now run in a distributed environment (multi-GPU, TPU).
+Built on `torch_xla` and `torch.distributed`, 🤗 Accelerate takes care of the heavy lifting, so you don't have to write any custom code to adapt to these platforms.
+Convert existing codebases to utilize [DeepSpeed](usage_guides/deepspeed), perform [fully sharded data parallelism](usage_guides/fsdp), and have automatic support for mixed-precision training! 
 
-You can even simplify your script a bit by letting 🤗 Accelerate handle the device placement for you (which is safer,
-especially for TPU training):
+<Tip> 
 
-```diff
-+ from accelerate import Accelerator
+  To get a better idea of this process, make sure to check out the [Tutorials](basic_tutorials/overview)! 
 
-+ accelerator = Accelerator()
-- my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
+</Tip>
 
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
-
-## Script launcher
-
-No need to remember how to use `torch.distributed.launch` or to write a specific launcher for TPU training! 🤗
-Accelerate comes with a CLI tool that will make your life easier when launching distributed scripts.
-
-On your machine(s) just run:
 
+This code can then be launched on any system through Accelerate's CLI interface:
 ```bash
-accelerate config
+accelerate launch {my_script.py}
 ```
 
-and answer the questions asked. This will generate a config file that will be used automatically to properly set the
-default options when doing
-
-```bash
-accelerate launch my_script.py --args_to_my_script
-```
-
-For instance, here is how you would run the NLP example (from the root of the repo):
-
-```bash
-accelerate launch examples/nlp_example.py
-```
-
-## Supported integrations
-
-- CPU only
-- single GPU
-- multi-GPU on one node (machine)
-- multi-GPU on several nodes (machines)
-- TPU
-- FP16 with native AMP (apex on the roadmap)
-- DeepSpeed (experimental support)
+<div class="mt-10">
+  <div class="w-full flex flex-col space-y-4 md:space-y-0 md:grid md:grid-cols-2 md:gap-y-4 md:gap-x-5">
+    <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg" href="/docs/accelerate/basic_tutorials/overview"
+      ><div class="w-full text-center bg-gradient-to-br from-blue-400 to-blue-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">Tutorials</div>
+      <p class="text-gray-700">Learn the basics and become familiar with using 🤗 Accelerate. Start here if you are using 🤗 Accelerate for the first time!</p>
+    </a>
+    <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg" href="/docs/accelerate/utility_guides/gradient_accumulation"
+      ><div class="w-full text-center bg-gradient-to-br from-indigo-400 to-indigo-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">How-to guides</div>
+      <p class="text-gray-700">Practical guides to help you achieve a specific goal. Take a look at these guides to learn how to use 🤗 Accelerate to solve real-world problems.</p>
+    </a>
+    <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg" href="/docs/accelerate/concept_guides/gradient_synchronization"
+      ><div class="w-full text-center bg-gradient-to-br from-pink-400 to-pink-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">Conceptual guides</div>
+      <p class="text-gray-700">High-level explanations for building a better understanding of important topics such as avoiding subtle nuances and pitfalls in distributed training and DeepSpeed.</p>
+   </a>
+    <a class="!no-underline border dark:border-gray-700 p-5 rounded-lg shadow hover:shadow-lg" href="/docs/accelerate/package_reference/accelerator"
+      ><div class="w-full text-center bg-gradient-to-br from-purple-400 to-purple-500 rounded-lg py-1.5 font-semibold mb-5 text-white text-lg leading-relaxed">Reference</div>
+      <p class="text-gray-700">Technical descriptions of how 🤗 Accelerate classes and methods work.</p>
+    </a>
+  </div>
+</div>
\ No newline at end of file
diff --git a/docs/source/installation.mdx b/docs/source/installation.mdx
deleted file mode 100644
index 26c2698aa..000000000
--- a/docs/source/installation.mdx
+++ /dev/null
@@ -1,96 +0,0 @@
-<!---
-Copyright 2021 The HuggingFace Team. All rights reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Installation
-
-🤗 Accelerate is tested on Python 3.6+, and PyTorch 1.6.0+.
-
-You should install 🤗 Accelerate in a [virtual environment](https://docs.python.org/3/library/venv.html). If you're
-unfamiliar with Python virtual environments, check out the [user guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). Create a virtual environment with the version of Python you're going
-to use and activate it.
-
-Now, if you want to use 🤗 Accelerate, you can install it with pip.
-
-## Installation with pip
-
-First you need to install PyTorch. Please refer to the
-[PyTorch installation page](https://pytorch.org/get-started/locally/#start-locally) regarding the specific install command for your platform.
-
-When PyTorch has been installed, 🤗 Accelerate can be installed using pip as follows:
-
-```bash
-pip install accelerate
-```
-
-Alternatively, for CPU-support only, you can install 🤗 Accelerate and PyTorch in one line with:
-
-```bash
-pip install accelerate[torch]
-```
-
-To check 🤗 Accelerate is properly installed, run the following command:
-
-```bash
-python -c "TODO write"
-```
-
-## Installing from source
-
-Here is how to quickly install `accelerate` from source:
-
-```bash
-pip install git+https://github.com/huggingface/accelerate
-```
-
-Note that this will install not the latest released version, but the bleeding edge `main` version, which you may want to use in case a bug has been fixed since the last official release and a new release hasn't  been yet rolled out.
-
-While we strive to keep `main` operational at all times, if you notice some issues, they usually get fixed within a few hours or a day and you're more than welcome to help us detect any problems by opening an [Issue](https://github.com/huggingface/accelerate/issues) and this way, things will get fixed even sooner.
-
-Again, you can run:
-
-```bash
-python -c "TODO write"
-```
-
-to check 🤗 Accelerate is properly installed.
-
-## Editable install
-
-If you want to constantly use the bleeding edge `main` version of the source code, or if you want to contribute to the library and need to test the changes in the code you're making, you will need an editable install. This is done by cloning the repository and installing with the following commands:
-
-``` bash
-git clone https://github.com/huggingface/accelerate.git
-cd accelerate
-pip install -e .
-```
-
-This command performs a magical link between the folder you cloned the repository to and your python library paths, and it'll look inside this folder in addition to the normal library-wide paths. So if normally your python packages get installed into:
-```
-~/anaconda3/envs/main/lib/python3.7/site-packages/
-```
-now this editable install will reside where you clone the folder to, e.g. `~/accelerate/` and python will search it too.
-
-Do note that you have to keep that `accelerate` folder around and not delete it to continue using the 🤗 Accelerate library.
-
-Now, let's get to the real benefit of this installation approach. Say, you saw some new feature just has been committed into `main`. If you have already performed all the steps above, to update your accelerate repo to include all the latest commits, all you need to do is to `cd` into that cloned repository folder and update the clone to the latest version:
-
-```bash
-cd ~/accelerate/
-git pull
-```
-
-There is nothing else to do. Your python environment will find the bleeding edge version of 🤗 Accelerate on the next run.
-
diff --git a/docs/source/launcher.mdx b/docs/source/launcher.mdx
deleted file mode 100644
index fb672d5ad..000000000
--- a/docs/source/launcher.mdx
+++ /dev/null
@@ -1,28 +0,0 @@
-<!--Copyright 2021 The HuggingFace Team. All rights reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
-an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
-specific language governing permissions and limitations under the License.
--->
-
-# Notebook Launcher
-
-Launch your training function inside a notebook. Currently supports launching a training with TPUs on [Google
-Colab](https://colab.research.google.com/) and [Kaggle kernels](https://www.kaggle.com/code), as well as training on
-several GPUs (if the machine on which you are running your notebook has them).
-
-An example can be found in [this notebook](https://github.com/huggingface/notebooks/blob/master/examples/accelerate/simple_nlp_example.ipynb).
-
-<Tip warning={true}>
-
-Your `Accelerator` object should only be defined inside the training function. This is because the
-initialization should be done inside the launcher only.
-
-</Tip>
-
-[[autodoc]] notebook_launcher
diff --git a/docs/source/package_reference/accelerator.mdx b/docs/source/package_reference/accelerator.mdx
new file mode 100644
index 000000000..fb20f1a66
--- /dev/null
+++ b/docs/source/package_reference/accelerator.mdx
@@ -0,0 +1,163 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerator
+
+The [`Accelerator`] is the main class provided by 🤗 Accelerate. 
+It serves at the main entrypoint for the API. 
+
+## Quick adaptation of your code
+
+To quickly adapt your script to work on any kind of setup with 🤗 Accelerate just:
+
+1. Initialize an [`Accelerator`] object (that we will call `accelerator` throughout this page) as early as possible in your script.
+2. Pass your dataloader(s), model(s), optimizer(s), and scheduler(s) to the [`~Accelerator.prepare`] method.
+3. Remove all the `.cuda()` or `.to(device)` from your code and let the `accelerator` handle the device placement for you. 
+
+<Tip>
+
+    Step three is optional, but considered a best practice.
+
+</Tip>
+
+4. Replace `loss.backward()` in your code with `accelerator.backward(loss)`
+5. Gather your predictions and labels before storing them or using them for metric computation using [`~Accelerator.gather`]
+
+<Tip warning={true}>
+
+    Step five is mandatory when using distributed evaluation
+    
+</Tip>
+
+In most cases this is all that is needed. The next section lists a few more advanced use cases and nice features
+you should search for and replace by the corresponding methods of your `accelerator`:
+
+## Advanced recommendations
+
+### Printing
+
+`print` statements should be replaced by [`~Accelerator.print`] to be printed once per process
+
+```diff
+- print("My thing I want to print!")
++ accelerator.print("My thing I want to print!")
+```
+
+### Executing processes
+
+#### Once on a single server
+
+For statements that should be executed once per server, use [`~Accelerator.is_local_main_process`]:
+
+```python
+if accelerator.is_local_main_process:
+    do_thing_once_per_server()
+```
+
+A function can be wrapped using the [`~Accelerator.on_local_main_process`] function to achieve the same 
+behavior on a function's execution:
+
+```python
+@accelerator.on_local_main_process
+def do_my_thing():
+    "Something done once per server"
+    do_thing_once_per_server()
+```
+
+#### Only ever once across all servers
+
+For statements that should only ever be executed once, use [`~Accelerator.is_main_process`]:
+
+```python
+if accelerator.is_main_process:
+    do_thing_once()
+```
+
+A function can be wrapped using the [`~Accelerator.on_main_process`] function to achieve the same 
+behavior on a function's execution:
+
+```python
+@accelerator.on_main_process
+def do_my_thing():
+    "Something done once per server"
+    do_thing_once()
+```
+
+#### On specific processes
+
+If a function should be ran on a specific overall or local process index, there are similar decorators 
+to achieve this:
+
+```python
+@accelerator.on_local_process(local_process_idx=0)
+def do_my_thing():
+    "Something done on process index 0 on each server"
+    do_thing_on_index_zero_on_each_server()
+```
+
+```python
+@accelerator.on_process(process_index=0)
+def do_my_thing():
+    "Something done on process index 0"
+    do_thing_on_index_zero()
+```
+
+### Synchronicity control
+
+Use [`~Accelerator.wait_for_everyone`] to make sure all processes join that point before continuing. (Useful before a model save for instance)
+
+### Saving and loading
+
+Use [`~Accelerator.unwrap_model`] before saving to remove all special model wrappers added during the distributed process. 
+
+```python
+model = MyModel()
+model = accelerator.prepare(model)
+# Unwrap
+model = accelerator.unwrap_model(model)
+```
+
+Use [`~Accelerator.save`] instead of `torch.save`:
+
+```diff
+  state_dict = model.state_dict()
+- torch.save(state_dict, "my_state.pkl")
++ accelerator.save(state_dict, "my_state.pkl")
+```
+
+### Operations
+
+Use [`~Accelerator.clip_grad_norm_`] instead of ``torch.nn.utils.clip_grad_norm_`` and [`~Accelerator.clip_grad_value_`] instead of ``torch.nn.utils.clip_grad_value``
+
+### Gradient Accumulation
+
+To perform gradient accumulation use [`~Accelerator.accumulate`] and specify a gradient_accumulation_steps. 
+This will also automatically ensure the gradients are synced or unsynced when on 
+multi-device training, check if the step should actually be performed, and auto-scale the loss:
+
+```diff
+- accelerator = Accelerator()
++ accelerator = Accelerator(gradient_accumulation_steps=2)
+
+  for (input, label) in training_dataloader:
++     with accelerator.accumulate(model):
+          predictions = model(input)
+          loss = loss_function(predictions, labels)
+          accelerator.backward(loss)
+          optimizer.step()
+          scheduler.step()
+          optimizer.zero_grad()
+```
+
+## Overall API documentation:
+
+[[autodoc]] Accelerator
\ No newline at end of file
diff --git a/docs/source/package_reference/big_modeling.mdx b/docs/source/package_reference/big_modeling.mdx
new file mode 100644
index 000000000..e54ac0807
--- /dev/null
+++ b/docs/source/package_reference/big_modeling.mdx
@@ -0,0 +1,41 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Working with large models
+
+## Dispatching and Offloading Models
+
+[[autodoc]] big_modeling.init_empty_weights
+[[autodoc]] big_modeling.cpu_offload
+[[autodoc]] big_modeling.disk_offload
+[[autodoc]] big_modeling.dispatch_model
+[[autodoc]] big_modeling.load_checkpoint_and_dispatch
+
+## Model Hooks
+
+### Hook Classes
+
+[[autodoc]] hooks.ModelHook
+[[autodoc]] hooks.AlignDevicesHook
+[[autodoc]] hooks.SequentialHook
+
+### Adding Hooks
+
+[[autodoc]] hooks.add_hook_to_module
+[[autodoc]] hooks.attach_execution_device_hook
+[[autodoc]] hooks.attach_align_device_hook
+[[autodoc]] hooks.attach_align_device_hook_on_blocks
+
+### Removing Hooks
+
+[[autodoc]] hooks.remove_hook_from_module
+[[autodoc]] hooks.remove_hook_from_submodules
\ No newline at end of file
diff --git a/docs/source/package_reference/cli.mdx b/docs/source/package_reference/cli.mdx
new file mode 100644
index 000000000..092b72850
--- /dev/null
+++ b/docs/source/package_reference/cli.mdx
@@ -0,0 +1,152 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# The Command Line 
+
+Below is a list of all the available commands 🤗 Accelerate with their parameters
+
+## accelerate config
+
+**Command**:
+
+`accelerate config` or `accelerate-config`
+
+Launches a series of prompts to create and save a `default_config.yml` configuration file for your training system. Should 
+always be ran first on your machine.
+
+**Usage**: 
+
+```bash
+accelerate config [arguments]
+```
+
+**Optional Arguments**:
+* `--config_file CONFIG_FILE` (`str`) -- The path to use to store the config file. Will default to a file named default_config.yaml in the cache location, which is the content
+                        of the environment `HF_HOME` suffixed with 'accelerate', or if you don't have such an environment variable, your cache directory
+                        (`~/.cache` or the content of `XDG_CACHE_HOME`) suffixed with `huggingface`.
+* `-h`, `--help` (`bool`) -- Show a help message and exit
+
+## accelerate env
+
+**Command**:
+
+`accelerate env` or `accelerate-env`
+
+Lists the contents of the passed 🤗 Accelerate configuration file. Should always be used when opening an issue on the [GitHub repository](https://github.com/huggingface/accelerate).
+
+**Usage**:
+
+```bash
+accelerate env [arguments]
+```
+
+**Optional Arguments**:
+* `--config_file CONFIG_FILE` (`str`) -- The path to use to store the config file. Will default to a file named default_config.yaml in the cache location, which is the content
+                        of the environment `HF_HOME` suffixed with 'accelerate', or if you don't have such an environment variable, your cache directory
+                        (`~/.cache` or the content of `XDG_CACHE_HOME`) suffixed with `huggingface`.
+* `-h`, `--help` (`bool`) -- Show a help message and exit
+
+## accelerate launch
+
+**Command**:
+
+`accelerate launch` or `accelerate-launch`
+
+Launches a specified script on a distributed system with the right parameters.
+
+**Usage**: 
+
+```bash
+accelerate launch [arguments] {training_script} --{training_script-argument-1} --{training_script-argument-2} ...
+```
+
+**Positional Arguments**:
+
+- `{training_script}` -- The full path to the script to be launched in parallel
+- `--{training_script-argument-1}` -- Arguments of the training script
+
+**Optional Arguments**:
+
+* `-h`, `--help` (`bool`) -- Show a help message and exit
+* `--config_file CONFIG_FILE` (`str`)-- The config file to use for the default values in the launching script.
+* `--cpu` (`bool`) -- Whether or not to force the training on the CPU.
+* `--mixed_precision {no,fp16,bf16}` (`str`) -- Whether or not to use mixed precision training. Choose between FP16 and BF16 (bfloat16) training. BF16 training is only supported on
+                    Nvidia Ampere GPUs and PyTorch 1.10 or later.
+* `--multi_gpu` (`bool`, defaults to `False`) -- Whether or not this should launch a distributed GPU training.
+* `-m`, `--module` (`bool`) -- Change each process to interpret the launch script as a Python module, executing with the same behavior as 'python -m'.
+* `--no_python` (`bool`) -- Skip prepending the training script with 'python' - just execute it directly. Useful when the script is not a Python script.
+
+The rest of these arguments are configured through `accelerate config` and are read in from the specified `--config_file` (or default configuration) for their 
+values. They can also be passed in manually.
+
+**Machine Configuration Arguments**:
+
+The following arguments are useful for customization of worker machines
+* `--machine_rank MACHINE_RANK` (`int`) -- The rank of the machine on which this script is launched.
+* `--num_machines NUM_MACHINES` (`int`) -- The total number of machines used in this training.
+* `--num_processes NUM_PROCESSES` (`int`) -- The total number of processes to be launched in parallel.
+* `--main_process_ip MAIN_PROCESS_IP` (`str`) -- The IP address of the machine of rank 0.
+* `--main_process_port MAIN_PROCESS_PORT` (`int`) -- The port to use to communicate with the machine of rank 0.
+* `--num_cpu_threads_per_process NUM_CPU_THREADS_PER_PROCESS` (`int`) -- The number of CPU threads per process. Can be tuned for optimal performance.
+
+
+**DeepSpeed Arguments**:
+
+The following arguments are only useful when `use_deepspeed` is passed: 
+* `--use_deepspeed` (`bool`) -- Whether to use deepspeed.
+* `--deepspeed_config_file DEEPSPEED_CONFIG_FILE` (`str`) -- DeepSpeed config file.
+* `--zero_stage ZERO_STAGE` (`str`) -- DeepSpeed's ZeRO optimization stage
+* `--offload_optimizer_device OFFLOAD_OPTIMIZER_DEVICE` (`str`) -- Decides where (none|cpu|nvme) to offload optimizer states
+* `--offload_param_device OFFLOAD_PARAM_DEVICE` (`str`) -- Decides where (none|cpu|nvme) to offload parameters
+* `--gradient_accumulation_steps GRADIENT_ACCUMULATION_STEPS` (`int`) -- Number of gradient_accumulation_steps used in your training script
+* `--gradient_clipping GRADIENT_CLIPPING` (`float`) -- gradient clipping value used in your training script
+The following arguments are related to using ZeRO Stage-3
+* `--zero3_init_flag ZERO3_INIT_FLAG` (`bool`) -- Decides Whether (true|false) to enable `deepspeed.zero.Init` for constructing massive models
+* `--zero3_save_16bit_model ZERO3_SAVE_16BIT_MODEL` (`bool`) -- Decides Whether (true|false) to save 16-bit model weights when using ZeRO Stage-3
+
+**Fully Sharded Data Parallelism Arguments**:
+
+The following arguments are only useful when `use_fdsp` is passed:
+* `--use_fsdp` (`bool`) -- Whether to use fsdp.
+* `--offload_params OFFLOAD_PARAMS` (`bool`) -- Decides Whether (true|false) to offload parameters and gradients to CPU.
+* `--min_num_params MIN_NUM_PARAMS` (`int`) -- FSDP's minimum number of parameters for Default Auto Wrapping.
+* `--sharding_strategy SHARDING_STRATEGY` (`str`) -- FSDP's Sharding Strategy.
+
+**TPU Arguments**:
+
+The following arguments are only useful when `tpu` is passed:
+* `--tpu` (`bool`) - Whether or not this should launch a TPU training.
+* `--main_training_function MAIN_TRAINING_FUNCTION` (`str`) -- The name of the main function to be executed in your script.
+
+**AWS SageMaker Arguments**:
+
+The following arguments are only useful when training in SageMaker
+* `--aws_access_key_id AWS_ACCESS_KEY_ID` (`str`) -- The AWS_ACCESS_KEY_ID used to launch the Amazon SageMaker training job
+* `--aws_secret_access_key AWS_SECRET_ACCESS_KEY` (`str`) -- The AWS_SECRET_ACCESS_KEY used to launch the Amazon SageMaker training job
+
+## accelerate test
+
+`accelerate test` or `accelerate-test`
+
+Runs `accelerate/test_utils/test_script.py` to verify that 🤗 Accelerate has been properly configured on your system and runs. 
+
+**Usage**: 
+
+```bash
+accelerate test [arguments]
+```
+
+**Optional Arguments**:
+* `--config_file CONFIG_FILE` (`str`) -- The path to use to store the config file. Will default to a file named default_config.yaml in the cache location, which is the content
+                        of the environment `HF_HOME` suffixed with 'accelerate', or if you don't have such an environment variable, your cache directory
+                        (`~/.cache` or the content of `XDG_CACHE_HOME`) suffixed with `huggingface`.
+* `-h`, `--help` (`bool`) -- Show a help message and exit
diff --git a/docs/source/package_reference/deepspeed.mdx b/docs/source/package_reference/deepspeed.mdx
new file mode 100644
index 000000000..fee886f4e
--- /dev/null
+++ b/docs/source/package_reference/deepspeed.mdx
@@ -0,0 +1,25 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Utilities for DeepSpeed
+
+[[autodoc]] utils.DeepSpeedPlugin
+
+[[autodoc]] utils.DummyOptim
+
+[[autodoc]] utils.DummyScheduler
+
+[[autodoc]] utils.DeepSpeedEngineWrapper
+
+[[autodoc]] utils.DeepSpeedOptimizerWrapper
+
+[[autodoc]] utils.DeepSpeedSchedulerWrapper
diff --git a/docs/source/kwargs.mdx b/docs/source/package_reference/kwargs.mdx
similarity index 100%
rename from docs/source/kwargs.mdx
rename to docs/source/package_reference/kwargs.mdx
diff --git a/docs/source/package_reference/launchers.mdx b/docs/source/package_reference/launchers.mdx
new file mode 100644
index 000000000..6f37f0af0
--- /dev/null
+++ b/docs/source/package_reference/launchers.mdx
@@ -0,0 +1,19 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launchers
+
+Functions for launching training on distributed processes.
+
+
+[[autodoc]] accelerate.notebook_launcher
+[[autodoc]] accelerate.debug_launcher
\ No newline at end of file
diff --git a/docs/source/package_reference/logging.mdx b/docs/source/package_reference/logging.mdx
new file mode 100644
index 000000000..675af41ee
--- /dev/null
+++ b/docs/source/package_reference/logging.mdx
@@ -0,0 +1,24 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Logging with Accelerate
+
+Accelerate has its own logging utility to handle logging while in a distributed system.
+To utilize this replace cases of `logging` with `accelerate.logging`:
+```diff
+- import logging
++ from accelerate.logging import get_logger
+- logger = logging.getLogger(__name__)
++ logger = get_logger(__name__)
+```
+
+[[autodoc]] logging.get_logger
\ No newline at end of file
diff --git a/docs/source/package_reference/state.mdx b/docs/source/package_reference/state.mdx
new file mode 100644
index 000000000..f1f5ef974
--- /dev/null
+++ b/docs/source/package_reference/state.mdx
@@ -0,0 +1,23 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Stateful Classes
+
+Below are variations of a [singleton class](https://en.wikipedia.org/wiki/Singleton_pattern) in the sense that all
+instances share the same state, which is initialized on the first instantiation.
+
+These classes are immutable and store information about certain configurations or 
+states.
+
+[[autodoc]] state.AcceleratorState
+
+[[autodoc]] state.GradientState
\ No newline at end of file
diff --git a/docs/source/internal.mdx b/docs/source/package_reference/torch_wrappers.mdx
similarity index 64%
rename from docs/source/internal.mdx
rename to docs/source/package_reference/torch_wrappers.mdx
index 4ff1d6ff9..4ac8ae572 100644
--- a/docs/source/internal.mdx
+++ b/docs/source/package_reference/torch_wrappers.mdx
@@ -10,44 +10,24 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
 specific language governing permissions and limitations under the License.
 -->
 
-# Internals
+# Wrapper classes for torch Dataloaders, Optimizers, and Schedulers
 
-## Gradient Accumulation states
+The internal classes Accelerate uses to prepare objects for distributed training
+when calling [`~Accelerator.prepare`].
 
-[[autodoc]] state.GradientState
-
-## Optimizer
-
-[[autodoc]] optimizer.AcceleratedOptimizer
-
-## DataLoader
-
-The main work on your PyTorch `DataLoader` is done by the following function:
+## Datasets and DataLoaders
 
 [[autodoc]] data_loader.prepare_data_loader
 
-### DataLoaderShard
-
-[[autodoc]] data_loader.DataLoaderShard
-
-### BatchSamplerShard
-
 [[autodoc]] data_loader.BatchSamplerShard
-
-### IterableDatasetShard
-
 [[autodoc]] data_loader.IterableDatasetShard
+[[autodoc]] data_loader.DataLoaderShard
+[[autodoc]] data_loader.DataLoaderDispatcher
 
-## Scheduler
-
-[[autodoc]] scheduler.AcceleratedScheduler
-
-## Distributed Config
-
-### AcceleratorState
+## Optimizers 
 
-[[autodoc]] state.AcceleratorState
+[[autodoc]] optimizer.AcceleratedOptimizer
 
-## Tracking
+## Schedulers 
 
-[[autodoc]] tracking.GeneralTracker
+[[autodoc]] scheduler.AcceleratedScheduler
\ No newline at end of file
diff --git a/docs/source/package_reference/tracking.mdx b/docs/source/package_reference/tracking.mdx
new file mode 100644
index 000000000..5e7a97f24
--- /dev/null
+++ b/docs/source/package_reference/tracking.mdx
@@ -0,0 +1,26 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Experiment Tracking
+
+## The Base Tracker Class
+
+[[autodoc]] tracking.GeneralTracker
+
+## Integrated Trackers
+
+[[autodoc]] tracking.TensorBoardTracker
+    - __init__
+[[autodoc]] tracking.WandBTracker
+    - __init__
+[[autodoc]] tracking.CometMLTracker
+    - __init__
diff --git a/docs/source/utilities.mdx b/docs/source/package_reference/utilities.mdx
similarity index 97%
rename from docs/source/utilities.mdx
rename to docs/source/package_reference/utilities.mdx
index bfcaeb60e..0c64953fc 100644
--- a/docs/source/utilities.mdx
+++ b/docs/source/package_reference/utilities.mdx
@@ -44,8 +44,6 @@ These include data operations that mimic the same `torch` ops but can be used on
 
 These functionalities check the state of the current working environment including information about the operating system itself, what it can support, and if particular dependencies are installed. 
 
-[[autodoc]] utils.get_max_memory
-
 [[autodoc]] utils.is_bf16_available
 
 [[autodoc]] utils.is_torch_version
@@ -58,6 +56,12 @@ These functionalities check the state of the current working environment includi
 
 When setting up 🤗 Accelerate for the first time, rather than running `accelerate config` [~utils.write_basic_config] can be used as an alternative for quick configuration.
 
+## Memory
+
+[[autodoc]] utils.get_max_memory
+
+[[autodoc]] utils.find_executable_batch_size
+
 ## Modeling
 
 These utilities relate to interacting with PyTorch models
diff --git a/docs/source/quicktour.mdx b/docs/source/quicktour.mdx
index 4b144ba05..812b20226 100644
--- a/docs/source/quicktour.mdx
+++ b/docs/source/quicktour.mdx
@@ -18,7 +18,7 @@ Let's have a look at the 🤗 Accelerate main features and traps to avoid.
 
 To use 🤗 Accelerate in your own script, you have to change four things:
 
-1. Import the [`Accelerator`] main class instantiate one in an `accelerator` object:
+1. Import the [`Accelerator`] main class and instantiate one in an `accelerator` object:
 
 ```python
 from accelerate import Accelerator
@@ -28,7 +28,7 @@ accelerator = Accelerator()
 
 This should happen as early as possible in your training script as it will initialize everything necessary for
 distributed training. You don't need to indicate the kind of environment you are in (just one machine with a GPU, one
-match with several GPUs, several machines with multiple GPUs or a TPU), the library will detect this automatically.
+machines with several GPUs, several machines with multiple GPUs or a TPU), the library will detect this automatically.
 
 2. Remove the call `.to(device)` or `.cuda()` for your model and input data. The `accelerator` object
 will handle this for you and place all those objects on the right device for you. If you know what you're doing, you
@@ -40,8 +40,8 @@ To fully deactivate the automatic device placement, pass along `device_placement
 
 <Tip warning={true}>
 
-If you place your objects manually on the proper device, be careful to create your optimizer after putting your
-model on `accelerator.device` or your training will fail on TPU.
+    If you place your objects manually on the proper device, be careful to create your optimizer after putting your
+    model on `accelerator.device` or your training will fail on TPU.
 
 </Tip>
 
@@ -61,9 +61,9 @@ use `shuffle=True` or any kind of random sampler).
 
 <Tip>
 
-The actual batch size for your training will be the number of devices used multiplied by the batch size you set in
-your script: for instance training on 4 GPUs with a batch size of 16 set when creating the training dataloader will
-train at an actual batch size of 64.
+    The actual batch size for your training will be the number of devices used multiplied by the batch size you set in
+    your script: for instance training on 4 GPUs with a batch size of 16 set when creating the training dataloader will
+    train at an actual batch size of 64.
 
 </Tip>
 
@@ -76,16 +76,16 @@ training loop.
 
 <Tip warning={true}>
 
-You should only pass the learning rate scheduler to [`~Accelerator.prepare`] when the scheduler needs to be stepped
-at each optimizer step.
+    You should only pass the learning rate scheduler to [`~Accelerator.prepare`] when the scheduler needs to be stepped
+    at each optimizer step.
 
 </Tip>
 
 <Tip warning={true}>
 
-Your training dataloader may change length when going through this method: if you run on X GPUs, it will have its
-length divided by X (since your actual batch size will be multiplied by X), unless you set
-`split_batches=True`.
+    Your training dataloader may change length when going through this method: if you run on X GPUs, it will have its
+    length divided by X (since your actual batch size will be multiplied by X), unless you set
+    `split_batches=True`.
 
 </Tip>
 
@@ -133,27 +133,32 @@ for inputs, targets in validation_dataloader:
 
 <Tip warning={true}>
 
-As for the training dataloader, passing your validation dataloader through
-[`~Accelerator.prepare`] may change it: if you run on X GPUs, it will have its length divided by X
-(since your actual batch size will be multiplied by X), unless you set `split_batches=True`.
+    Similar to the training dataloader, passing your validation dataloader through
+    [`~Accelerator.prepare`] may change it: if you run on X GPUs, it will have its length divided by X
+    (since your actual batch size will be multiplied by X), unless you set `split_batches=True`.
+
+</Tip>
 
 Any instruction using your training dataloader length (for instance if you need the number of total training steps
-to create a learning rate scheduler) should go after the call to [`~Accelerator.prepare`].
+to create a learning rate scheduler) should go after the call to [`~Accelerator.prepare`]. 
 
-As some data at the end of the dataset may be duplicated so the batch can divide equally to all workers, metrics should be 
-calculated through the [`~Accelerator.gather_for_metrics`] method to automatically remove the duplicated data.
+Some data at the end of the dataset may be duplicated so the batch can be divided equally among all workers. As a result, metrics
+should be calculated through the [`~Accelerator.gather_for_metrics`] method to automatically remove the duplicated data while gathering.
 
-If for some reason you don't wish to have this automatically done, [`~Accelerator.gather`] can be used instead to gather 
-the data across all processes and this can manually be done instead.
+<Tip>
+
+    If for some reason you don't wish to have this automatically done, [`~Accelerator.gather`] can be used instead to gather 
+    the data across all processes and this can manually be done instead.
 
 </Tip>
 
+
 <Tip warning={true}>
 
-The [`~Accelerator.gather`] and [`~Accelerator.gather_for_metrics`] methods require the tensors to be all the same size on each process. If
-you have tensors of different sizes on each process (for instance when dynamically padding to the maximum length in
-a batch), you should use the [`~Accelerator.pad_across_processes`] method to pad you tensor to the
-biggest size across processes.
+    The [`~Accelerator.gather`] and [`~Accelerator.gather_for_metrics`] methods require the tensors to be all the same size on each process. If
+    you have tensors of different sizes on each process (for instance when dynamically padding to the maximum length in
+    a batch), you should use the [`~Accelerator.pad_across_processes`] method to pad you tensor to the
+    biggest size across processes.
 
 </Tip>
 
@@ -163,7 +168,7 @@ You can use the regular commands to launch your distributed training (like `torc
 PyTorch), they are fully compatible with 🤗 Accelerate. The only caveat here is that 🤗 Accelerate uses the environment
 to determine all useful information, so `torch.distributed.launch` should be used with the flag `--use_env`.
 
-🤗 Accelerate also provides a CLI tool that unifies all launcher, so you only have to remember one command. To use it,
+🤗 Accelerate also provides a CLI tool that unifies all launchers, so you only have to remember one command. To use it,
 just run:
 
 ```bash
@@ -207,7 +212,10 @@ If you stored the config file in a non-default location, you can indicate it to
 accelerate launch --config_file path_to_config.yaml path_to_script.py --args_for_the_script
 ```
 
-You can also override any of the arguments determined by your config file, see TODO: insert ref here.
+You can also override any of the arguments determined by your config file. 
+To see the complete list of parameters that you can pass in, run `accelerate launch -h`. 
+
+Check out the [Launch tutorial](basic_tutorials/launch) for more information about launching your scripts. 
 
 
 ## Launching training from a notebook
@@ -227,11 +235,14 @@ notebook_launcher(training_function)
 
 <Tip warning={true}>
 
-Your `Accelerator` object should only be defined inside the training function. This is because the
-initialization should be done inside the launcher only.
+    Your [`Accelerator`] object should only be defined inside the training function. This is because the
+    initialization should be done inside the launcher only.
 
 </Tip>
 
+Check out the [Notebook Launcher tutorial](basic_tutorials/notebook) for more information about training on TPUs. 
+
+
 ## Training on TPU
 
 If you want to launch your script on TPUs, there are a few caveats you should be aware of. Behind the scenes, the TPUs
@@ -242,7 +253,7 @@ optimizations takes some time.
 The good news is that this compilation will be cached so the second step and all the following will be much faster. The
 bad news is that it only applies if all of your steps do exactly the same operations, which implies:
 
-- having all tensors of the same length in all your lengths
+- having all tensors of the same length in all your batches
 - having static code (i.e., not a for loop of length that could change from step to step)
 
 Having any of the things above change between two steps will trigger a new compilation which will, once again, take a
@@ -271,6 +282,8 @@ passed your model to [`~Accelerator.prepare`]) will break the tying. You will ne
 after. You can find an example of this in the [run_clm_no_trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py) script in
 the Transformers repository.
 
+Check out the [TPU tutorial](concept_guides/training_tpu) for more information about training on TPUs. 
+
 
 ## Other caveats
 
@@ -357,18 +370,21 @@ Note that since all the model parameters are references to tensors, this will lo
 ## Saving/loading entire states
 
 When training your model, you may want to save the current state of the model, optimizer, random generators, and potentially LR schedulers to be restored in the _same script_.
-You can use `accelerator.save_state` and `accelerator.load_state` respectively to do so, just by simply passing in a save location. 
-If you have registered any other stateful items to be stored through `accelerator.register_for_checkpointing` they will also be saved and/or loaded.
+You can use [`~Accelerator.save_state`] and [`~Accelerator.load_state`] respectively to do so, just by simply passing in a save location. 
+If you have registered any other stateful items to be stored through [`~Accelerator.register_for_checkpointing`] they will also be saved and/or loaded.
+
 <Tip>
-    Every object passed to `register_for_checkpointing` must have a `load_state_dict` and `save_dict` function to be stored
+
+    Every object passed to [`~Accelerator.register_for_checkpointing`] must have a `load_state_dict` and `state_dict` function to be stored
+
 </Tip>
 
 
 ### Gradient clipping
 
 If you are using gradient clipping in your script, you should replace the calls to
-`torch.nn.utils.clip_grad_norm_` or `torch.nn.utils.clip_grad_value_` with `accelerator.clip_grad_norm_`
-and `accelerator.clip_grad_value_` respectively.
+`torch.nn.utils.clip_grad_norm_` or `torch.nn.utils.clip_grad_value_` with [`~Accelerator.clip_grad_norm_`]
+and [`~Accelerator.clip_grad_value_`] respectively.
 
 
 ### Mixed Precision training
@@ -397,6 +413,26 @@ if not accelerator.optimizer_step_was_skipped:
     lr_scheduler.step()
 ```
 
+### Gradient Accumulation 
+
+To perform gradient accumulation use [`~Accelerator.accumulate`] and specify a `gradient_accumulation_steps`. 
+This will also automatically ensure the gradients are synced or unsynced when on multi-device training, check if the step should
+actually be performed, and auto-scale the loss:
+
+```python
+accelerator = Accelerator(gradient_accumulation_steps=2)
+model, optimizer, training_dataloader = accelerator.prepare(model, optimizer, training_dataloader)
+
+for input, label in training_dataloader:
+    with accelerator.accumulate(model):
+        predictions = model(input)
+        loss = loss_function(predictions, label)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+        optimizer.zero_grad()
+```
+
 ### DeepSpeed
 
 DeepSpeed support is experimental, so the underlying API will evolve in the near future and may have some slight
@@ -405,7 +441,7 @@ will be added in a next version.
 
 <Tip warning={true}>
 
-The [`notebook_launcher`] does not support the DeepSpeed integration yet.
+    The [`notebook_launcher`] does not support the DeepSpeed integration yet.
 
 </Tip>
 
@@ -415,7 +451,7 @@ Internally, the library works by first analyzing the environment in which the sc
 kind of distributed setup is used, how many different processes there are and which one the current script is in. All
 that information is stored in the [`~AcceleratorState`].
 
-This class is initialized the first time you instantiate a [`Accelerator`] as well as performing any
+This class is initialized the first time you instantiate an [`~Accelerator`] as well as performing any
 specific initialization your distributed setup needs. Its state is then uniquely shared through all instances of
 [`~state.AcceleratorState`].
 
@@ -448,18 +484,18 @@ setting the same seed in the main random number generator in all processes.
 
 <Tip warning={true}>
 
-Synchronization of the main torch (or CUDA or XLA) random number generator will affect any other potential random
-artifacts you could have in your dataset (like random data augmentation) in the sense that all processes will get
-the same random numbers from the torch random modules (so will apply the same random data augmentation if it's
-controlled by torch).
+    Synchronization of the main torch (or CUDA or XLA) random number generator will affect any other potential random
+    artifacts you could have in your dataset (like random data augmentation) in the sense that all processes will get
+    the same random numbers from the torch random modules (so will apply the same random data augmentation if it's
+    controlled by torch).
 
 </Tip>
 
 <Tip>
 
-The randomization part of your custom sampler, batch sampler or iterable dataset should be done using a local
-`torch.Generator` object (in PyTorch >= 1.6), see the traditional `RandomSampler`, as an example.
+    The randomization part of your custom sampler, batch sampler or iterable dataset should be done using a local
+    `torch.Generator` object (in PyTorch >= 1.6), see the traditional `RandomSampler`, as an example.
 
 </Tip>
 
-For more details about the internals, see the [Internals page](internal).
+For more details about the internals, see the [Internals page](package_reference/torch_wrappers).
diff --git a/docs/source/big_modeling.mdx b/docs/source/usage_guides/big_modeling.mdx
similarity index 87%
rename from docs/source/big_modeling.mdx
rename to docs/source/usage_guides/big_modeling.mdx
index 006c8bc4e..243b354d0 100644
--- a/docs/source/big_modeling.mdx
+++ b/docs/source/usage_guides/big_modeling.mdx
@@ -31,7 +31,7 @@ While this works very well for regularly sized models, this workflow has some cl
 
 <Tip warning={true}>
 
-This API is quite new and still in its experimental stage. While we strive to provide a stable API, it's possible some small parts of the public API will change in the future.
+    This API is quite new and still in its experimental stage. While we strive to provide a stable API, it's possible some small parts of the public API will change in the future.
 
 </Tip>
 
@@ -57,7 +57,7 @@ initializes an empty model with a bit more than 100B parameters. Behind the scen
 
 <Tip warning={true}>
 
-You can't move a model initialized like this on CPU or another device directly, since it doesn't have any data. It's also very likely that a forward pass with that empty model will fail, as not all operations are supported on the meta device.
+    You can't move a model initialized like this on CPU or another device directly, since it doesn't have any data. It's also very likely that a forward pass with that empty model will fail, as not all operations are supported on the meta device.
 
 </Tip>
 
@@ -199,7 +199,7 @@ This way, you model can run for inference even if it doesn't fit on one of the G
 
 <Tip warning={true}>
 
-This only supports inference of your model, not training. Most of the computation happens behind `torch.no_grad()` context managers to avoid spending some GPU memory with intermediate activations.
+    This only supports inference of your model, not training. Most of the computation happens behind `torch.no_grad()` context managers to avoid spending some GPU memory with intermediate activations.
 
 </Tip>
 
@@ -209,7 +209,7 @@ You can let 🤗 Accelerate handle the device map computation by setting `device
 
 <Tip>
 
-You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+    You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
 
 </Tip>
 
@@ -222,7 +222,7 @@ When you have more GPU memory available than the model size, here the difference
 
 <Tip>
 
-The options `"auto"` and `"balanced"` produce the same results for now, but the behavior of `"auto"` might change in the future if we find a strategy that makes more sense, while `"balanced"` will stay stable.
+    The options `"auto"` and `"balanced"` produce the same results for now, but the behavior of `"auto"` might change in the future if we find a strategy that makes more sense, while `"balanced"` will stay stable.
 
 </Tip>
 
@@ -238,9 +238,9 @@ device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB",
 
 <Tip warning={true}>
 
-When a first allocation happens in PyTorch, it loads CUDA kernels which take about 1-2GB of memory depending on the GPU. Therefore you always have less usable memory than the actual size of the GPU. To see how much memory is actually used do `torch.ones(1).cuda()` and look at the memory usage.
+    When a first allocation happens in PyTorch, it loads CUDA kernels which take about 1-2GB of memory depending on the GPU. Therefore you always have less usable memory than the actual size of the GPU. To see how much memory is actually used do `torch.ones(1).cuda()` and look at the memory usage.
 
-Therefore when you create memory maps with `max_memory` make sure to adjust the avaialble memory accordingly to avoid out-of-memory errors.
+    Therefore when you create memory maps with `max_memory` make sure to adjust the avaialble memory accordingly to avoid out-of-memory errors.
 
 </Tip>
 
@@ -271,7 +271,7 @@ device_map = {"block1": 0, "block2.linear1": 1, "block2.linear2": 1}
 
 <Tip>
 
-To be the most efficient, make sure your device map puts the parameters on the GPUs in a sequential manner (e.g. don't put one of the first weights on GPU 0, then weights on GPU 1 and the last weight back to GPU 0) to avoid making many transfers of data between the GPUs.
+    To be the most efficient, make sure your device map puts the parameters on the GPUs in a sequential manner (e.g. don't put one of the first weights on GPU 0, then weights on GPU 1 and the last weight back to GPU 0) to avoid making many transfers of data between the GPUs.
 
 </Tip>
 
@@ -285,20 +285,4 @@ We are aware of the current limitations in the API:
 - [`load_checkpoint_and_dispatch`] and [`load_checkpoint_in_model`] do not perform any check on the correctness of your state dict compared to your model at the moment (this will be fixed in a future version), so you may get some weird errors if trying to load a checkpoint with mismatched or missing keys.
 - The model parallelism used when your model is split on several GPUs is naive and not optimized, meaning that only one GPU works at a given time and the other sits idle.
 - When weights are offloaded on the CPU/hard drive, there is no pre-fetching (yet, we will work on this for future versions) which means the weights are put on the GPU when they are needed and not before.
-- Hard-drive offloading might be very slow if the hardware you run on does not have fast communication between disk and CPU (like NVMes).
-
-## API doc
-
-[[autodoc]] cpu_offload
-
-[[autodoc]] disk_offload
-
-[[autodoc]] dispatch_model
-
-[[autodoc]] infer_auto_device_map
-
-[[autodoc]] init_empty_weights
-
-[[autodoc]] load_checkpoint_and_dispatch
-
-[[autodoc]] load_checkpoint_in_model
+- Hard-drive offloading might be very slow if the hardware you run on does not have fast communication between disk and CPU (like NVMes).
\ No newline at end of file
diff --git a/docs/source/checkpoint.mdx b/docs/source/usage_guides/checkpoint.mdx
similarity index 100%
rename from docs/source/checkpoint.mdx
rename to docs/source/usage_guides/checkpoint.mdx
diff --git a/docs/source/deepspeed.mdx b/docs/source/usage_guides/deepspeed.mdx
similarity index 98%
rename from docs/source/deepspeed.mdx
rename to docs/source/usage_guides/deepspeed.mdx
index c0385c352..8a4238732 100644
--- a/docs/source/deepspeed.mdx
+++ b/docs/source/usage_guides/deepspeed.mdx
@@ -472,25 +472,11 @@ model, eval_dataloader = accelerator.prepare(model, eval_dataloader)
 
 1. Current integration doesn’t support Pipeline Parallelism of DeepSpeed.
 2. Current integration doesn’t support `mpu`, limiting the tensor parallelism which is supported in Megatron-LM. 
-3. Current integration doesn’t support multiple models for a given `accelerator` object. 
+3. Current integration doesn’t support multiple models. 
 
+## DeepSpeed Resources
 
-## Internals
-
-[[autodoc]] utils.DeepSpeedPlugin
-
-[[autodoc]] utils.DummyOptim
-
-[[autodoc]] utils.DummyScheduler
-
-[[autodoc]] utils.DeepSpeedEngineWrapper
-
-[[autodoc]] utils.DeepSpeedOptimizerWrapper
-
-[[autodoc]] utils.DeepSpeedSchedulerWrapper
-
-
-## Main DeepSpeed Resources
+The documentation for the internals related to deepspeed can be found [here](../package_reference/deepspeed).
 
 - [Project's github](https://github.com/microsoft/deepspeed)
 - [Usage docs](https://www.deepspeed.ai/getting-started/)
diff --git a/docs/source/fsdp.mdx b/docs/source/usage_guides/fsdp.mdx
similarity index 73%
rename from docs/source/fsdp.mdx
rename to docs/source/usage_guides/fsdp.mdx
index 32f982786..a561e4ecd 100644
--- a/docs/source/fsdp.mdx
+++ b/docs/source/usage_guides/fsdp.mdx
@@ -39,10 +39,14 @@ For instance, here is how you would run the NLP example (from the root of the re
 compute_environment: LOCAL_MACHINE
 deepspeed_config: {}
 distributed_type: FSDP
+downcast_bf16: 'no'
 fsdp_config:
-  min_num_params: 2000
-  offload_params: false
-  sharding_strategy: 1
+  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
+  fsdp_backward_prefetch_policy: BACKWARD_PRE
+  fsdp_offload_params: false
+  fsdp_sharding_strategy: 1
+  fsdp_state_dict_type: FULL_STATE_DICT
+  fsdp_transformer_layer_cls_to_wrap: GPT2Block
 machine_rank: 0
 main_process_ip: null
 main_process_port: null
@@ -60,9 +64,13 @@ accelerate launch examples/nlp_example.py
 Currently, `Accelerate` supports the following config through the CLI:
 
 ```bash
-`Sharding Strategy`: [1] FULL_SHARD, [2] SHARD_GRAD_OP
-`Min Num Params`: FSDP\'s minimum number of parameters for Default Auto Wrapping.
-`Offload Params`: Decides Whether to offload parameters and gradients to CPU.
+`Sharding Strategy`: [1] FULL_SHARD (shards optimizer states, gradients and parameters), [2] SHARD_GRAD_OP (shards optimizer states and gradients), [3] NO_SHARD
+`Offload Params`: Decides Whether to offload parameters and gradients to CPU
+`Auto Wrap Policy`: [1] TRANSFORMER_BASED_WRAP, [2] SIZE_BASED_WRAP, [3] NO_WRAP 
+`Transformer Layer Class to Wrap`: When using `TRANSFORMER_BASED_WRAP`, user specifies transformer layer class name (case-sensitive) to wrap ,e.g, `BertLayer`, `GPTJBlock`, `T5Block`...
+`Min Num Params`: minimum number of parameters when using `SIZE_BASED_WRAP`
+`Backward Prefetch`: [1] BACKWARD_PRE, [2] BACKWARD_POST, [3] NO_PREFETCH
+`State Dict Type`: [1] FULL_STATE_DICT, [2] LOCAL_STATE_DICT, [3] SHARDED_STATE_DICT  
 ```
 
 ## A few caveats to be aware of
@@ -76,19 +84,18 @@ Currently, `Accelerate` supports the following config through the CLI:
 However, below is the recommended way to prepare model and optimizer while using FSDP:
 
 ```diff
-model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", return_dict=True)
+  model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", return_dict=True)
 + model = accelerator.prepare(model)
 
-optimizer = torch.optim.AdamW(params=model.parameters(), lr=lr)
+  optimizer = torch.optim.AdamW(params=model.parameters(), lr=lr)
 
-- model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(model,
--        optimizer, train_dataloader, eval_dataloader, lr_scheduler
+- model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
+-        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
 -    )
 
 + optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
 +         optimizer, train_dataloader, eval_dataloader, lr_scheduler
-+        )
-
++    )
 ```
 
 - In case of a single model, if you have created the optimizer with multiple parameter groups and called prepare with them together,
@@ -110,11 +117,9 @@ optimizer = torch.optim.AdamW(params=model.parameters(), lr=lr)
   ```
 
 
-- In case of multiple models, it is necessary to prepare the models before creating optimizers or else it will throw an error.
-- Mixed precision is currently not supported with FSDP.
+- In case of multiple models, it is necessary to prepare the models before creating optimizers or else it will throw an error. 
+Then pass the optimizers to the prepare call in the same order as corresponding models else `accelerator.save_state()` and `accelerator.load_state()` will result in wrong/unexpected behaviour.
+- This feature is incompatible with `--predict_with_generate` in the `run_translation.py` script of 🤗 `Transformers` library.
 
-For more control, users can leverage the `FullyShardedDataParallelPlugin` wherein they can specify `auto_wrap_policy`, `backward_prefetch` and `ignored_modules`.
-After creating an instance of this class, users can pass it to the Accelerator class instantiation.
+For more control, users can leverage the `FullyShardedDataParallelPlugin`. After creating an instance of this class, users can pass it to the Accelerator class instantiation.
 For more information on these options, please refer to the PyTorch [FullyShardedDataParallel](https://github.com/pytorch/pytorch/blob/0df2e863fbd5993a7b9e652910792bd21a516ff3/torch/distributed/fsdp/fully_sharded_data_parallel.py#L236) code.
-
-[[autodoc]] utils.FullyShardedDataParallelPlugin
diff --git a/docs/source/gradient_accumulation.mdx b/docs/source/usage_guides/gradient_accumulation.mdx
similarity index 86%
rename from docs/source/gradient_accumulation.mdx
rename to docs/source/usage_guides/gradient_accumulation.mdx
index f6ab857d8..f9f5dc62a 100644
--- a/docs/source/gradient_accumulation.mdx
+++ b/docs/source/usage_guides/gradient_accumulation.mdx
@@ -71,13 +71,15 @@ First the code shown earlier will be converted to utilize 🤗 Accelerate withou
 ```
 
 <Tip warning={true}>
-In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization.
+
+  In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the [Concepts tutorial](concept_guides/gradient_synchronization)!
+
 </Tip>
 
 ## Letting 🤗 Accelerate handle gradient accumulation
 
 All that is left now is to let 🤗 Accelerate handle the gradient accumulation for us. To do so you should pass in a `gradient_accumulation_steps` parameter to [`Accelerator`], dictating the number 
-of steps to perform before each call to `step()` and how to automatically adjust the loss during the call to [`Accelerator.backward`]:
+of steps to perform before each call to `step()` and how to automatically adjust the loss during the call to [`~Accelerator.backward`]:
 
 ```diff
   from accelerate import Accelerator
@@ -85,19 +87,19 @@ of steps to perform before each call to `step()` and how to automatically adjust
 + accelerator = Accelerator(gradient_accumulation_steps=2)
 ```
 
-From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
-You just wrap it around the entire training part of your code: 
+From here you can use the [`~Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
+You just wrap it around the entire training part of our code: 
 
 ```diff
 - for index, batch in enumerate(training_dataloader):
 + for batch in training_dataloader:
-+    with accelerator.accumulate(model):
++     with accelerator.accumulate(model):
           optimizer.zero_grad()
           inputs, targets = batch
           outputs = model(inputs)
 ```
 
-and you can remove all the special checks for the step number and the loss adjustment:
+You can remove all the special checks for the step number and the loss adjustment:
 
 ```diff
 - loss = loss / gradient_accumulation_steps
diff --git a/docs/source/memory.mdx b/docs/source/usage_guides/memory.mdx
similarity index 95%
rename from docs/source/memory.mdx
rename to docs/source/usage_guides/memory.mdx
index e578e43d7..9c5674977 100644
--- a/docs/source/memory.mdx
+++ b/docs/source/usage_guides/memory.mdx
@@ -48,4 +48,4 @@ def training_function(args):
 +   inner_training_loop()
 ```
 
-[[autodoc]] utils.find_executable_batch_size
\ No newline at end of file
+To find out more, check the documentation [here](package_reference/utilities#accelerate.find_executable_batch_size)
\ No newline at end of file
diff --git a/docs/source/sagemaker.mdx b/docs/source/usage_guides/sagemaker.mdx
similarity index 90%
rename from docs/source/sagemaker.mdx
rename to docs/source/usage_guides/sagemaker.mdx
index 563f0c742..6195783e9 100644
--- a/docs/source/sagemaker.mdx
+++ b/docs/source/usage_guides/sagemaker.mdx
@@ -54,7 +54,7 @@ accelerate config
 
 <Tip>
 
-🤗 Accelerate is not saving any of your credentials.
+    🤗 Accelerate is not saving any of your credentials.
 
 </Tip>
 
@@ -72,8 +72,8 @@ directory. After training, artifacts in this directory are uploaded to S3:
 
 <Tip warning={true}>
 
-SageMaker doesn’t support argparse actions. If you want to use, for example, boolean hyperparameters, you need to
-specify type as bool in your script and provide an explicit True or False value for this hyperparameter. [[REF]](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#prepare-a-pytorch-training-script).
+    SageMaker doesn’t support argparse actions. If you want to use, for example, boolean hyperparameters, you need to
+    specify type as bool in your script and provide an explicit True or False value for this hyperparameter. [[REF]](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#prepare-a-pytorch-training-script).
 
 </Tip>
 
@@ -92,7 +92,7 @@ arguments needed by your training script as named arguments.
 
 <Tip>
 
-If you run one of the example scripts, don't forget to add `accelerator.save('/opt/ml/model')` to it.
+    If you run one of the example scripts, don't forget to add `accelerator.save('/opt/ml/model')` to it.
 
 </Tip>
 
diff --git a/docs/source/tracking.mdx b/docs/source/usage_guides/tracking.mdx
similarity index 85%
rename from docs/source/tracking.mdx
rename to docs/source/usage_guides/tracking.mdx
index 25e44b3c9..2fb68c0f2 100644
--- a/docs/source/tracking.mdx
+++ b/docs/source/usage_guides/tracking.mdx
@@ -19,12 +19,9 @@ There are a large number of experiment tracking API's available, however getting
 
 Currently `Accelerate` supports three trackers out-of-the-box:
 
-
-[[autodoc]] tracking.TensorBoardTracker
-
-[[autodoc]] tracking.WandBTracker
-
-[[autodoc]] tracking.CometMLTracker
+- TensorBoard
+- WandB
+- CometML
 
 To use any of them, pass in the selected type(s) to the `log_with` parameter in [`Accelerate`]:
 ```python
@@ -107,6 +104,9 @@ import wandb
 
 
 class MyCustomTracker(GeneralTracker):
+    name = "wandb"
+    requires_logging_directory = False
+
     def __init__(self, run_name: str):
         self.run_name = run_name
         wandb.init(self.run_name)
@@ -137,27 +137,27 @@ accelerator = Accelerator(log_with=[tracker, "all"])
 
 If a library has an API that does not follow a strict `.log` with an overall dictionary such as Neptune.AI, logging can be done manually under an `if accelerator.is_main_process` statement:
 ```diff
-from accelerate import Accelerator
+  from accelerate import Accelerator
 + import neptune.new as neptune
 
-accelerator = Accelerator()
+  accelerator = Accelerator()
 + run = neptune.init(...)
 
-my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
-device = accelerator.device
-my_model.to(device)
-
-for iteration in config["num_iterations"]:
-    for batch in my_training_dataloader:
-        my_optimizer.zero_grad()
-        inputs, targets = batch
-        inputs = inputs.to(device)
-        targets = targets.to(device)
-        outputs = my_model(inputs)
-        loss = my_loss_function(outputs, targets)
-        total_loss += loss
-        accelerator.backward(loss)
-        my_optimizer.step()
-+       if accelerator.is_main_process:
-+           run["logs/training/batch/loss"].log(loss)
+  my_model, my_optimizer, my_training_dataloader = accelerate.prepare(my_model, my_optimizer, my_training_dataloader)
+  device = accelerator.device
+  my_model.to(device)
+
+  for iteration in config["num_iterations"]:
+      for batch in my_training_dataloader:
+          my_optimizer.zero_grad()
+          inputs, targets = batch
+          inputs = inputs.to(device)
+          targets = targets.to(device)
+          outputs = my_model(inputs)
+          loss = my_loss_function(outputs, targets)
+          total_loss += loss
+          accelerator.backward(loss)
+          my_optimizer.step()
++         if accelerator.is_main_process:
++             run["logs/training/batch/loss"].log(loss)
 ```
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 977cd5d24..41fc1e3d4 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -140,10 +140,20 @@ class Accelerator:
             A list of `KwargHandler` to customize how the objects related to distributed training or mixed precision
             are created. See [kwargs](kwargs) for more information.
 
-    Attributes
+    **Attributes:**
 
         - **device** (`torch.device`) -- The device to use.
+        - **distributed_type** ([`~utils.DistributedType`]) -- The distributed training configuration.
+        - **local_process_index** (`int`) -- The process index on the current machine.
+        - **mixed_precision** (`str`) -- The configured mixed precision mode.
+        - **num_processes** (`int`) -- The total number of processes used for training.
+        - **optimizer_step_was_skipped** (`bool`) -- Whether or not the optimizer update was skipped (because of
+          gradient overflow in mixed precision), in which
+        case the learning rate should not be changed.
+        - **process_index** (`int`) -- The overall index of the current process among all processes.
         - **state** ([`~state.AcceleratorState`]) -- The distributed setup state.
+        - **sync_gradients** (`bool`) -- Whether the gradients are currently being synced across all processes.
+        - **use_distributed** (`bool`) -- Whether the current configuration is for distributed training.
     """
 
     def __init__(
@@ -308,6 +318,9 @@ def __init__(
 
     @property
     def use_distributed(self):
+        """
+        Whether the Accelerator is configured for distributed training
+        """
         return self.distributed_type != DistributedType.NO and self.num_processes > 1
 
     @property
diff --git a/src/accelerate/state.py b/src/accelerate/state.py
index 27a627ae9..a4f6bac2a 100644
--- a/src/accelerate/state.py
+++ b/src/accelerate/state.py
@@ -47,20 +47,18 @@ def parse_choice_from_env(key, default="no"):
 # Inspired by Alex Martelli's 'Borg'.
 class AcceleratorState:
     """
-    This is a variation of a [singleton class](https://en.wikipedia.org/wiki/Singleton_pattern) in the sense that all
-    instance of `AcceleratorState` share the same state, which is initialized on the first instantiation.
+    Singleton class that has information about the current training environment.
 
-    Attributes:
+    **Attributes:**
 
         - **device** (`torch.device`) -- The device to use.
-        - **sync_gradients** (`bool`) -- Whether to sync the gradients or not
-        - **distributed_type** (`~accelerate.state.DistributedType`) -- The type of distributed environment currently
+        - **distributed_type** ([`~accelerate.state.DistributedType`]) -- The type of distributed environment currently
           in use.
-        - **num_processes** (`int`) -- The number of processes currently launched in parallel.
-        - **process_index** (`int`) -- The index of the current process.
         - **local_process_index** (`int`) -- The index of the current process on the current server.
         - **mixed_precision** (`str`) -- Whether or not the current script will use mixed precision. If you are using
           mixed precision, define if you want to use FP16 or BF16 (bfloat16) as the floating point.
+        - **num_processes** (`int`) -- The number of processes currently launched in parallel.
+        - **process_index** (`int`) -- The index of the current process.
     """
 
     _shared_state = {}
@@ -250,13 +248,10 @@ def _check_initialized(self, mixed_precision=None, cpu=None):
 
 class GradientState:
     """
-    This is a variation of a [singleton class](https://en.wikipedia.org/wiki/Singleton_pattern) in the sense that all
-    instance of `GradientState` share the same state, which is initialized on the first instantiation.
+    Singleton class that has information related to gradient synchronization for gradient accumulation
 
-    This specific state revolves around whether gradients should be synced and if we have reached the end of a prepared
-    dataloader Attributes:
+    **Attributes:**
 
-        - **sync_gradients** (`bool`) -- Whether the gradients should be synced
         - **end_of_dataloader** (`bool`) -- Whether we have reached the end the current dataloader
         - **remainder** (`int`) -- The number of extra samples that were added from padding the dataloader
     """


Review comments: body: missing the use of the context manager below
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Performing gradient accumulation with 🤗 Accelerate
+
+Gradient accumulation is a technique where you can train on bigger batch sizes than 
+your machine would normally be able to fit into memory. This is done by accumulating gradients over
+several batches, and only performing the step through the optimizer after a certain number of batches have been performed.
+
+While technically standard gradient accumulation code would work fine in a distributed setup, it is not the most efficient
+method for doing so and you may experience considerable slowdowns!
+
+In this tutorial you will see how to quickly setup gradient accumulation and perform it with the utilties provided in 🤗 Accelerate,
+which can total to adding just one line of code!
+
+This example will use a very simplistic PyTorch training loop that performs gradient accumulation every two batches:
+
+```python
+device = 'cuda'
+model.to(device)
+
+gradient_accumulation_steps = 2
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    loss.backward()
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+## Converting it to 🤗 Accelerate
+
+First the code shown earlier will be converted to utilize 🤗 Accelerate without the special gradient accumulation helper:
+
+```python
+from accelerate import Accelerator
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    accelerator.backward(loss)
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+<Tip warning={true}>
+In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the Concepts tutorial!
+</Tip>
+
+## Letting 🤗 Accelerate handle gradient accumulation
+
+All that is left now is to let 🤗 Accelerate handle the gradient accumulation for us. To do so you should pass in a `gradient_accumulation_steps` parameter to `Accelerator`, dictating the number 
+of steps to perform before each call to `step()` and `backward()`:
+
+```diff
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
+```
+
+From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
body: Thanks 😄 *Should* be set now, appreciate your thorough review on it! 🙇 
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Performing gradient accumulation with 🤗 Accelerate
+
+Gradient accumulation is a technique where you can train on bigger batch sizes than 
+your machine would normally be able to fit into memory. This is done by accumulating gradients over
+several batches, and only performing the step through the optimizer after a certain number of batches have been performed.
+
+While technically standard gradient accumulation code would work fine in a distributed setup, it is not the most efficient
+method for doing so and you may experience considerable slowdowns!
+
+In this tutorial you will see how to quickly setup gradient accumulation and perform it with the utilties provided in 🤗 Accelerate,
+which can total to adding just one line of code!
+
+This example will use a very simplistic PyTorch training loop that performs gradient accumulation every two batches:
+
+```python
+device = 'cuda'
+model.to(device)
+
+gradient_accumulation_steps = 2
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    loss.backward()
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+## Converting it to 🤗 Accelerate
+
+First the code shown earlier will be converted to utilize 🤗 Accelerate without the special gradient accumulation helper:
+
+```python
+from accelerate import Accelerator
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    accelerator.backward(loss)
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+<Tip warning={true}>
+In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the Concepts tutorial!
+</Tip>
+
+## Letting 🤗 Accelerate handle gradient accumulation
+
+All that is left now is to let 🤗 Accelerate handle the gradient accumulation for us. To do so you should pass in a `gradient_accumulation_steps` parameter to `Accelerator`, dictating the number 
+of steps to perform before each call to `step()` and `backward()`:
+
+```diff
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
+```
+
+From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
body: looking forward to this feature!
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Performing gradient accumulation with 🤗 Accelerate
+
+Gradient accumulation is a technique where you can train on bigger batch sizes than 
+your machine would normally be able to fit into memory. This is done by accumulating gradients over
+several batches, and only performing the step through the optimizer after a certain number of batches have been performed.
+
+While technically standard gradient accumulation code would work fine in a distributed setup, it is not the most efficient
+method for doing so and you may experience considerable slowdowns!
+
+In this tutorial you will see how to quickly setup gradient accumulation and perform it with the utilties provided in 🤗 Accelerate,
+which can total to adding just one line of code!
+
+This example will use a very simplistic PyTorch training loop that performs gradient accumulation every two batches:
+
+```python
+device = 'cuda'
+model.to(device)
+
+gradient_accumulation_steps = 2
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    loss.backward()
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+## Converting it to 🤗 Accelerate
+
+First the code shown earlier will be converted to utilize 🤗 Accelerate without the special gradient accumulation helper:
+
+```python
+from accelerate import Accelerator
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
+    accelerator.backward(loss)
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
+```
+
+<Tip warning={true}>
+In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the Concepts tutorial!
+</Tip>
+
+## Letting 🤗 Accelerate handle gradient accumulation
+
+All that is left now is to let 🤗 Accelerate handle the gradient accumulation for us. To do so you should pass in a `gradient_accumulation_steps` parameter to `Accelerator`, dictating the number 
+of steps to perform before each call to `step()` and `backward()`:
+
+```diff
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
+```
+
+From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
body: ```suggestion
🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that allows the same PyTorch code be ran across any distributed configuration by adding just four lines of code!
body: ```suggestion
platforms separately so you don't have to. In minutes, existing codebases can be converted to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that allows the same PyTorch code be ran across any distributed configuration by adding just four lines of code!
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
+platforms separately so you don't have to. In minutes existing codebases can be converted to utilize [DeepSpeed](https://github.com/microsoft/DeepSpeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
body: ```suggestion
<Tip> To get a better idea of this process, make sure to check out the [Tutorials](https://huggingface.co/docs/accelerate/basic_tutorials/overview) ! </Tip>
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that allows the same PyTorch code be ran across any distributed configuration by adding just four lines of code!
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
+platforms separately so you don't have to. In minutes existing codebases can be converted to utilize [DeepSpeed](https://github.com/microsoft/DeepSpeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
 
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+For an idea of just how easy it can be, here's a brief example of the changes needed on a basic PyTorch script:
+<Tip>To get a better idea of this process, make sure to check out the [Tutorials](https://huggingface.co/docs/accelerate/basic_tutorials/overview)!</Tip>
body: I think for even greater impact, you can move the code snippet up here so users can immediately see the four lines of code and how easy it is to launch.
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
body: Maybe to avoid sounding too negative, this is a bit better?

```suggestion
Built on `torch_xla` and `torch.distributed`, 🤗 Accelerate takes care of the heavy lifting, so you don't have to write any custom code to adapt to these platforms.
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
body: ```suggestion
platforms separately so you don't have to. Convert existing codebases to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
+platforms separately so you don't have to. In minutes, existing codebases can be converted to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
body: I think the Markdown link isn't rendered because there needs to be a newline after `<Tip>`.

```suggestion
<Tip> 

To get a better idea of this process, make sure to check out the [Tutorials](https://huggingface.co/docs/accelerate/basic_tutorials/overview)! 

</Tip>
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
+platforms separately so you don't have to. In minutes, existing codebases can be converted to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
 
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+For an idea of just how easy it can be, here's a brief example of the changes needed on a basic PyTorch script:
+<Tip> To get a better idea of this process, make sure to check out the [Tutorials](https://huggingface.co/docs/accelerate/basic_tutorials/overview) ! </Tip>
body: The spacing is slightly off in this snippet for the lines removed and added.

```suggestion
-   inputs = inputs.to(device)
```
diff_hunk: @@ -12,121 +12,56 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
-## Features
+Built on `torch_xla` and `torch.distributed`, Accelerate wraps around the tough and problematic code changes that need to occur on each of these
+platforms separately so you don't have to. In minutes, existing codebases can be converted to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
 
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+For an idea of just how easy it can be, here's a brief example of the changes needed on a basic PyTorch script:
+<Tip> To get a better idea of this process, make sure to check out the [Tutorials](https://huggingface.co/docs/accelerate/basic_tutorials/overview) ! </Tip>
 
 ```diff
-+ from accelerate import Accelerator
-
-+ accelerator = Accelerator()
-  # Use the device given by the *accelerator* object.
-+ device = accelerator.device
-  my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
-
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
-      inputs = inputs.to(device)
-      targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
++from accelerate import Accelerator
++accelerator = Accelerator()
 
-and with this, your script can now run in a distributed environment (multi-GPU, TPU).
-
-You can even simplify your script a bit by letting 🤗 Accelerate handle the device placement for you (which is safer,
-especially for TPU training):
-
-```diff
-+ from accelerate import Accelerator
++model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++    model, optimizer, training_dataloader, scheduler
++)
 
-+ accelerator = Accelerator()
-- my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
-
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
-
-## Script launcher
-
-No need to remember how to use `torch.distributed.launch` or to write a specific launcher for TPU training! 🤗
-Accelerate comes with a CLI tool that will make your life easier when launching distributed scripts.
-
-On your machine(s) just run:
-
-```bash
-accelerate config
-```
-
-and answer the questions asked. This will generate a config file that will be used automatically to properly set the
-default options when doing
-
-```bash
-accelerate launch my_script.py --args_to_my_script
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+-    inputs = inputs.to(device)
body: ```suggestion
New features are added every day that haven't been released yet. To try them out yourself, install
```
diff_hunk: @@ -0,0 +1,91 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+As 🤗 Accelerate is rapidly under development, new features are added every day that haven't been released yet. To try them out yourself install
body: ```suggestion
from the GitHub repository:
```
diff_hunk: @@ -0,0 +1,91 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+As 🤗 Accelerate is rapidly under development, new features are added every day that haven't been released yet. To try them out yourself install
+from the Github Repository:
body: ```suggestion
After installing, you need to configure 🤗 Accelerate for how the current system is setup for training. 
```
diff_hunk: @@ -0,0 +1,91 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+As 🤗 Accelerate is rapidly under development, new features are added every day that haven't been released yet. To try them out yourself install
+from the Github Repository:
+
+```bash
+git clone https://github.com/huggingface/accelerate
+cd accelerate
+pip install -e .
+```
+
+## Configuring 🤗 Accelerate
+
+After installing, you need to tell Accelerate how the current system is setup for training. 
body: ```suggestion
This tutorial will detail how to easily convert existing PyTorch code to use 🤗 Accelerate!
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
body: ```suggestion
You'll see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
body: ```suggestion
To begin, write out a very basic PyTorch training loop. 
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
body: ```suggestion
We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand.
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
body: ```suggestion
[`Accelerator`] is the main force behind utilizing all the possible options for distributed training!
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
body: ```suggestion
make sure everything is setup in the current environment for you to start training:
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+-device = 'cuda'
++device = accelerator.device
+model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+then make sure that everything is setup for the environment you are currently in to start your training:
body: ```suggestion
These objects are returned in the same order they were sent in with. By default, `device_placement=True`, so your model is automatically moved to the proper device.
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+-device = 'cuda'
++device = accelerator.device
+model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+then make sure that everything is setup for the environment you are currently in to start your training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. If `device_placement=True` is passed into the [`Accelerator`] object (the default), 
body: I think you also need to add a newline before and after the `<Tip>` here so the code format is rendered.

```suggestion
Accelerate will only prepare objects that inherit from their respective PyTorch classes (such as `torch.optim.Optimizer`).
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+-device = 'cuda'
++device = accelerator.device
+model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+then make sure that everything is setup for the environment you are currently in to start your training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. If `device_placement=True` is passed into the [`Accelerator`] object (the default), 
+performing this action will also automatically move your model to the proper device as well.
+
+<Tip warning={true}>
+Accelerate will only prepare objects that inherit from their respective PyTorch classes (such as `torch.optim.Optimizer`)
body: ```suggestion
Finally, three lines of code need to be changed in the training loop. 🤗 Accelerate's DataLoader classes will automatically handle the device placement by default,
```
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+-device = 'cuda'
++device = accelerator.device
+model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+then make sure that everything is setup for the environment you are currently in to start your training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. If `device_placement=True` is passed into the [`Accelerator`] object (the default), 
+performing this action will also automatically move your model to the proper device as well.
+
+<Tip warning={true}>
+Accelerate will only prepare objects that inherit from their respective PyTorch classes (such as `torch.optim.Optimizer`)
+</Tip>
+
+### Modifying the training loop
+
+Finally, three lines of code need changing in our training loop. 🤗 Accelerate's DataLoader classes will automatically handle the device placement by default,
body: Love that you added a complete version of all the snippets so users have the option of just copying the code once!
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code into utilizing 🤗 Accelerate!
+You'll be able to see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, first write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+This class will be the main force helping you with utilizing everything possible for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+-device = 'cuda'
++device = accelerator.device
+model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+then make sure that everything is setup for the environment you are currently in to start your training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. If `device_placement=True` is passed into the [`Accelerator`] object (the default), 
+performing this action will also automatically move your model to the proper device as well.
+
+<Tip warning={true}>
+Accelerate will only prepare objects that inherit from their respective PyTorch classes (such as `torch.optim.Optimizer`)
+</Tip>
+
+### Modifying the training loop
+
+Finally, three lines of code need changing in our training loop. 🤗 Accelerate's DataLoader classes will automatically handle the device placement by default,
+and [`Accelerator.backward`] should be used for performing the backward pass:
+
+```diff
+-   inputs = inputs.to(device)
+-   targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+-   loss.backward()
++   accelerator.backward(loss)
+```
+
+With that, your training loop is now ready to use 🤗 Accelerate!
+
+## The finished code
body: ```suggestion
In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
body: The right navbar doesn't display code format so the ticks will just show up :/

```suggestion
## Using accelerate launch
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
body: Need some newlines here too :)
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
body: ```suggestion
Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
body: ```suggestion
For example, here is how to use `accelerate launch` with a single GPU:
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
body: ```suggestion
In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+Accelerate can take some liberties towards a few hyperparameters, such as if a GPU is available it will use all of them by default, and no mixed precision will be used. 
body: I would maybe clarify that you'll get a warning if you don't pass in the needed parameters. This is kind of unclear until now.

```suggestion
would also launch that same script on two GPUs using mixed precision while avoiding all of the warnings: 
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+Accelerate can take some liberties towards a few hyperparameters, such as if a GPU is available it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision, but also avoiding all of the warnings: 
body: ```suggestion
For a complete list of parameters you can pass in, run:
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+Accelerate can take some liberties towards a few hyperparameters, such as if a GPU is available it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision, but also avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a full list of all parameters you can pass in, run:
body: ```suggestion
- If this does not exist either, the folder `~/.cache/huggingface/accelerate`.
```
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+Accelerate can take some liberties towards a few hyperparameters, such as if a GPU is available it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision, but also avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a full list of all parameters you can pass in, run:
+```bash
+accelerate launch -h
+```
+
+<Tip>
+Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
+</Tip>
+
+## Custom Configurations
+
+As briefly mentioned earlier, `accelerate launch` should be mostly used through combining set configurations 
+made with the `accelerate config` command. These configs are saved to a `default_config.yaml` file in your cache folder for 🤗 Accelerate. 
+This cache folder is located at (with decreasing order of priority):
+
+- The content of your environment variable `HF_HOME` suffixed with `accelerate`.
+- If it does not exist, the content of your environment variable `XDG_CACHE_HOME` suffixed with
+  `huggingface/accelerate`.
+- If this does not exist either, the folder `~/.cache/huggingface/accelerate`
body: I would also move the snippet at the end of this page under here since that's what it's showing here right?
diff_hunk: @@ -0,0 +1,144 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial you were introduced with how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using `accelerate launch`
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various different torch spawn methods, all of the environmental variables that would be expected for them can be modified here as well.
+For example, here is how to use `accelerate launch` but only use a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+Accelerate can take some liberties towards a few hyperparameters, such as if a GPU is available it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision, but also avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a full list of all parameters you can pass in, run:
+```bash
+accelerate launch -h
+```
+
+<Tip>
+Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
+</Tip>
+
+## Custom Configurations
+
+As briefly mentioned earlier, `accelerate launch` should be mostly used through combining set configurations 
+made with the `accelerate config` command. These configs are saved to a `default_config.yaml` file in your cache folder for 🤗 Accelerate. 
+This cache folder is located at (with decreasing order of priority):
+
+- The content of your environment variable `HF_HOME` suffixed with `accelerate`.
+- If it does not exist, the content of your environment variable `XDG_CACHE_HOME` suffixed with
+  `huggingface/accelerate`.
+- If this does not exist either, the folder `~/.cache/huggingface/accelerate`
+
+To have multiple configurations, the flag `--config_file` can be passed to the `accelerate launch` command paired 
body: I think you can create a separate Troubleshoot page for all these caveats. I think a user just getting started doesn't necessarily need to know all these smaller details. Likewise, I think the Internal mechanism section is probably better in the Concepts and Fundamentals section since the user doesn't need to know exactly how it works under the hood. 
diff_hunk: @@ -271,6 +276,8 @@ passed your model to [`~Accelerator.prepare`]) will break the tying. You will ne
 after. You can find an example of this in the [run_clm_no_trainer](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py) script in
 the Transformers repository.
 
+Check out the [TPU tutorial](https://huggingface.co/docs/accelerate/concept_guides/training_tpu) for more information about training on TPUs. 
+
 
 ## Other caveats
body: Adding a link to it would make it easier for users to navigate there.
diff_hunk: @@ -49,29 +49,29 @@ for index, batch in enumerate(training_dataloader):
 First the code shown earlier will be converted to utilize 🤗 Accelerate without the special gradient accumulation helper:
 
 ```diff
-+ from accelerate import Accelerator
-+ accelerator = Accelerator()
-
-+ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
-+     model, optimizer, training_dataloader, scheduler
-+ )
-
-  for index, batch in enumerate(training_dataloader):
-      optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = model(inputs)
-      loss = loss_function(outputs, targets)
-      loss = loss / gradient_accumulation_steps
-+     accelerator.backward(loss)
-      if (index+1) % gradient_accumulation_steps == 0:
-          optimizer.step()
-          scheduler.step()
++from accelerate import Accelerator
++accelerator = Accelerator()
+
++model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++    model, optimizer, training_dataloader, scheduler
++)
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+-    inputs = inputs.to(device)
+-    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
++    accelerator.backward(loss)
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
 ```
 
 <Tip warning={true}>
-In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization.
+In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the Concepts tutorial!
body: ```suggestion
You can remove all the special checks for the step number and the loss adjustment:
```
diff_hunk: @@ -80,35 +80,39 @@ All that is left now is to let 🤗 Accelerate handle the gradient accumulation
 of steps to perform before each call to `step()` and how to automatically adjust the loss during the call to [`Accelerator.backward`]:
 
 ```diff
-  from accelerate import Accelerator
-- accelerator = Accelerator()
-+ accelerator = Accelerator(gradient_accumulation_steps=2)
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
 ```
 
 From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
-You just wrap it around the entire training part of your code: 
+You just wrap it around the entire training part of our code: 
 
 ```diff
-- for index, batch in enumerate(training_dataloader):
-+ for batch in training_dataloader:
-+    with accelerator.accumulate(model):
-          optimizer.zero_grad()
-          inputs, targets = batch
-          outputs = model(inputs)
+-for index, batch in enumerate(training_dataloader):
++for batch in training_dataloader:
++   with accelerator.accumulate(model):
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
 ```
 
 and you can remove all the special checks for the step number and the loss adjustment:
body: Add some newlines so `DataLoaders` is properly rendered, and I'd also add a link to the Gradient Sync concept guide.
diff_hunk: @@ -80,35 +80,39 @@ All that is left now is to let 🤗 Accelerate handle the gradient accumulation
 of steps to perform before each call to `step()` and how to automatically adjust the loss during the call to [`Accelerator.backward`]:
 
 ```diff
-  from accelerate import Accelerator
-- accelerator = Accelerator()
-+ accelerator = Accelerator(gradient_accumulation_steps=2)
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
 ```
 
 From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
-You just wrap it around the entire training part of your code: 
+You just wrap it around the entire training part of our code: 
 
 ```diff
-- for index, batch in enumerate(training_dataloader):
-+ for batch in training_dataloader:
-+    with accelerator.accumulate(model):
-          optimizer.zero_grad()
-          inputs, targets = batch
-          outputs = model(inputs)
+-for index, batch in enumerate(training_dataloader):
++for batch in training_dataloader:
++   with accelerator.accumulate(model):
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
 ```
 
 and you can remove all the special checks for the step number and the loss adjustment:
 
 ```diff
-- loss = loss / gradient_accumulation_steps
-  accelerator.backward(loss)
-- if (index+1) % gradient_accumulation_steps == 0:
-  optimizer.step()
-  scheduler.step()
+-loss = loss / gradient_accumulation_steps
+accelerator.backward(loss)
+-if (index+1) % gradient_accumulation_steps == 0:
+optimizer.step()
+scheduler.step()
 ```
 
 As you can see the [`Accelerator`] is able to keep track of the batch number you are on and it will automatically know whether to step through the prepared optimizer and how to adjust the loss. 
 
+<Tip warning={true}>
body: I think we can probably remove this header.
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
body: ```suggestion
training in a distributed setup. But how does this risk slowing down your code?
```
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++from accelerate import Accelerator
++accelerator = Accelerator()
+import torch.nn as nn
+-from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10,10)
++model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this automatically cause a risk of slowing down your code?
body: ```suggestion
The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
```
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++from accelerate import Accelerator
++accelerator = Accelerator()
+import torch.nn as nn
+-from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10,10)
++model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this automatically cause a risk of slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`, all instances of the model
body: This was a little hard to follow, is this what you meant?

```suggestion
need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
```
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++from accelerate import Accelerator
++accelerator = Accelerator()
+import torch.nn as nn
+-from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10,10)
++model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this automatically cause a risk of slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`, all instances of the model
+need to have updated their gradients collated and updated before moving onto the next batch of data. But when performing 
body: ```suggestion
gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
```
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++from accelerate import Accelerator
++accelerator = Accelerator()
+import torch.nn as nn
+-from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10,10)
++model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this automatically cause a risk of slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`, all instances of the model
+need to have updated their gradients collated and updated before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skipping `.backward()` until `n` batches has been reached. This 
body: Is `them` referring to the batches of data?

```suggestion
can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
```
diff_hunk: @@ -0,0 +1,111 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+## Background into PyTorch distributed
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++from accelerate import Accelerator
++accelerator = Accelerator()
+import torch.nn as nn
+-from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10,10)
++model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this automatically cause a risk of slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`, all instances of the model
+need to have updated their gradients collated and updated before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skipping `.backward()` until `n` batches has been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. But how 
body: Add newlines here as well :)
diff_hunk: @@ -0,0 +1,27 @@
+# Deferring Executions
+
+When you run your usual script, instructions are executed in order. Using 🤗 Accelerate to deploy your script on several
+GPUs at the same time introduces a complication: while each process executes all instructions in order, some may be
+faster than others.
+
+You might need to wait for all processes to have reached a certain point before executing a given instruction. For
+instance, you shouldn't save a model before being sure every process is done with training, and you wouldn't want to 
+continue training before all the model weights have been loaded in. To do this, just write the following line in your code:
+
+```
+accelerator.wait_for_everyone()
+```
+
+This instruction will block all the processes that arrive first until all the other processes have reached that
+point (if you run your script on just one GPU or CPU, this won't do anything).
+
+A few example cases for when to use this utility are listed below:
+
+<Tip>
body: Add newlines here too
diff_hunk: @@ -0,0 +1,146 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Training on TPUs with 🤗 Accelerate
+
+Training on TPUs can be slightly different than training on multi-gpu, even with 🤗 Accelerate. This guide aims to show you 
+where you should be careful and why, as well as the best practices in general.
+
+## Training in a Notebook
+
+The main carepoint when training on TPUs comes from the `notebook_launcher`. As mentioned in the [notebook tutorial](https://huggingface.co/docs/accelerate/basic_tutorials/notebook), you need to 
+restructure your training code into a function that can get passed to the `notebook_launcher` function and be careful about not declaring any tensors on the GPU.
+
+While on a TPU that last part is not as important, a critical part to understand is that when you launch code from a notebook you do so through a process called **forking**. 
+When launching from the command-line, you perform **spawning**, where a python process is not currently running and you *spawn* a new process in. Since your Jupyter notebook is already 
+utilizing a python process, you need to *fork* a new process from it to launch your code. 
+
+Where this becomes important is in regards to declaring your model. On forked TPU processes, it is recommended that you instantiate your model *once* and pass this into your 
+training function. This is different than training on GPUs where you create `n` models that have their gradients synced and back-propagated at certain moments. Instead one 
+model instance is shared between all the nodes and it is passed back and forth. This is important especially when training on low-resource TPUs such as those provided in Kaggle kernels or
+on Google Colaboratory. 
+
+Below is an example of a training function passed to the `notebook_launcher` if training on CPUs or GPUs:
+
+<Tip>
body: At the time the concepts tutorial didn't exist 😃  will definitely add it in!
diff_hunk: @@ -49,29 +49,29 @@ for index, batch in enumerate(training_dataloader):
 First the code shown earlier will be converted to utilize 🤗 Accelerate without the special gradient accumulation helper:
 
 ```diff
-+ from accelerate import Accelerator
-+ accelerator = Accelerator()
-
-+ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
-+     model, optimizer, training_dataloader, scheduler
-+ )
-
-  for index, batch in enumerate(training_dataloader):
-      optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = model(inputs)
-      loss = loss_function(outputs, targets)
-      loss = loss / gradient_accumulation_steps
-+     accelerator.backward(loss)
-      if (index+1) % gradient_accumulation_steps == 0:
-          optimizer.step()
-          scheduler.step()
++from accelerate import Accelerator
++accelerator = Accelerator()
+
++model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++    model, optimizer, training_dataloader, scheduler
++)
+
+for index, batch in enumerate(training_dataloader):
+    optimizer.zero_grad()
+    inputs, targets = batch
+-    inputs = inputs.to(device)
+-    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss = loss / gradient_accumulation_steps
++    accelerator.backward(loss)
+    if (index+1) % gradient_accumulation_steps == 0:
+        optimizer.step()
+        scheduler.step()
 ```
 
 <Tip warning={true}>
-In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization.
+In its current state, this code is not going to perform gradient accumulation efficiently due to a process called gradient synchronization. Read more about that in the Concepts tutorial!
body: This actually is a mistake because this warning is no longer needed! But I added a link to the gradient sync tutorial int he earlier mention of it
diff_hunk: @@ -80,35 +80,39 @@ All that is left now is to let 🤗 Accelerate handle the gradient accumulation
 of steps to perform before each call to `step()` and how to automatically adjust the loss during the call to [`Accelerator.backward`]:
 
 ```diff
-  from accelerate import Accelerator
-- accelerator = Accelerator()
-+ accelerator = Accelerator(gradient_accumulation_steps=2)
+from accelerate import Accelerator
+-accelerator = Accelerator()
++accelerator = Accelerator(gradient_accumulation_steps=2)
 ```
 
 From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
-You just wrap it around the entire training part of your code: 
+You just wrap it around the entire training part of our code: 
 
 ```diff
-- for index, batch in enumerate(training_dataloader):
-+ for batch in training_dataloader:
-+    with accelerator.accumulate(model):
-          optimizer.zero_grad()
-          inputs, targets = batch
-          outputs = model(inputs)
+-for index, batch in enumerate(training_dataloader):
++for batch in training_dataloader:
++   with accelerator.accumulate(model):
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
 ```
 
 and you can remove all the special checks for the step number and the loss adjustment:
 
 ```diff
-- loss = loss / gradient_accumulation_steps
-  accelerator.backward(loss)
-- if (index+1) % gradient_accumulation_steps == 0:
-  optimizer.step()
-  scheduler.step()
+-loss = loss / gradient_accumulation_steps
+accelerator.backward(loss)
+-if (index+1) % gradient_accumulation_steps == 0:
+optimizer.step()
+scheduler.step()
 ```
 
 As you can see the [`Accelerator`] is able to keep track of the batch number you are on and it will automatically know whether to step through the prepared optimizer and how to adjust the loss. 
 
+<Tip warning={true}>
body: ```suggestion
Convert existing codebases to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
```
diff_hunk: @@ -12,121 +12,60 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
-
-## Features
-
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
 ```diff
 + from accelerate import Accelerator
-
 + accelerator = Accelerator()
-  # Use the device given by the *accelerator* object.
-+ device = accelerator.device
-  my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
-
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
-      inputs = inputs.to(device)
-      targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
 
-and with this, your script can now run in a distributed environment (multi-GPU, TPU).
-
-You can even simplify your script a bit by letting 🤗 Accelerate handle the device placement for you (which is safer,
-especially for TPU training):
-
-```diff
-+ from accelerate import Accelerator
-
-+ accelerator = Accelerator()
-- my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
-+     my_model, my_optimizer, my_training_dataloader
++ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++     model, optimizer, training_dataloader, scheduler
 + )
 
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
+  for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+-   inputs = inputs.to(device)
+-   targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
++   accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
 ```
 
-## Script launcher
+Built on `torch_xla` and `torch.distributed`, 🤗 Accelerate takes care of the heavy lifting, so you don't have to write any custom code to adapt to these platforms.
+platforms separately so you don't have to. Convert existing codebases to utilize [DeepSpeed](https://huggingface.co/docs/accelerate/deepspeed), perform [fully sharded data parallelism](https://huggingface.co/docs/accelerate/fsdp), and have automatic support for mixed-precision training! 
body: ```suggestion
Some data at the end of the dataset may be duplicated so the batch can be divided equally among all workers. As a result, metrics
```
diff_hunk: @@ -137,17 +137,22 @@ As for the training dataloader, passing your validation dataloader through
 [`~Accelerator.prepare`] may change it: if you run on X GPUs, it will have its length divided by X
 (since your actual batch size will be multiplied by X), unless you set `split_batches=True`.
 
+</Tip>
+
 Any instruction using your training dataloader length (for instance if you need the number of total training steps
-to create a learning rate scheduler) should go after the call to [`~Accelerator.prepare`].
+to create a learning rate scheduler) should go after the call to [`~Accelerator.prepare`]. 
 
-As some data at the end of the dataset may be duplicated so the batch can divide equally to all workers, metrics should be 
-calculated through the [`~Accelerator.gather_for_metrics`] method to automatically remove the duplicated data.
+Some data at the end of the dataset may be duplicated so the batch can divide equally to all workers. As a result metrics
body: Should be un-commented before merging.
diff_hunk: @@ -1,6 +1,6 @@
 name: Run Tests
 
-on: [pull_request]
+# on: [pull_request]
body: Installation should stay in Getting Started.
diff_hunk: @@ -1,38 +1,67 @@
-- sections: 
+- sections:
   - local: index
     title: 🤗 Accelerate
   - local: quicktour
-    title: Quick tour
-  - local: installation
+    title: Quicktour of 🤗 Accelerate
+  title: Getting started
+- sections:
+  - local: basic_tutorials/overview
+    title: Overview
+  - local: basic_tutorials/install
body: ```suggestion
    title: Quicktour
```
I'm pretty sure the user won't expect a quicktour of another lib here ;-)
diff_hunk: @@ -1,38 +1,67 @@
-- sections: 
+- sections:
   - local: index
     title: 🤗 Accelerate
   - local: quicktour
-    title: Quick tour
-  - local: installation
+    title: Quicktour of 🤗 Accelerate
body: That's an editable install, a source install is just installing from git and should be detailed here, before you speak about editable install for developping.
diff_hunk: @@ -0,0 +1,91 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+New features are added every day that haven't been released yet. To try them out yourself, install
+from the GitHub repository:
+
+```bash
+git clone https://github.com/huggingface/accelerate
+cd accelerate
+pip install -e .
+```
body: You should insist here that running `accelerate config` beforehand is recommended.
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
body: You should give a code example of this.
diff_hunk: @@ -0,0 +1,29 @@
+# Deferring Executions
+
+When you run your usual script, instructions are executed in order. Using 🤗 Accelerate to deploy your script on several
+GPUs at the same time introduces a complication: while each process executes all instructions in order, some may be
+faster than others.
+
+You might need to wait for all processes to have reached a certain point before executing a given instruction. For
+instance, you shouldn't save a model before being sure every process is done with training, and you wouldn't want to 
+continue training before all the model weights have been loaded in. To do this, just write the following line in your code:
+
+```
+accelerator.wait_for_everyone()
+```
+
+This instruction will block all the processes that arrive first until all the other processes have reached that
+point (if you run your script on just one GPU or CPU, this won't do anything).
+
+A few example cases for when to use this utility are listed below:
+
+<Tip>
+
+Some of these are utilized with the [`Accelerator.main_process_first`] context manager, which utilizes [`Accelerator.wait_for_everyone`] to 
+run a particular set of code on the main process beforehand before triggering and launching the other processes
+
+</Tip>
+
+- Downloading a dataset on the main process first and then loading the cached dataset in afterwards
body: Use 4 lines of indentation.
diff_hunk: @@ -12,121 +12,60 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
-
-## Features
-
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
 ```diff
 + from accelerate import Accelerator
-
 + accelerator = Accelerator()
-  # Use the device given by the *accelerator* object.
-+ device = accelerator.device
-  my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
-
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
-      inputs = inputs.to(device)
-      targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
 
-and with this, your script can now run in a distributed environment (multi-GPU, TPU).
-
-You can even simplify your script a bit by letting 🤗 Accelerate handle the device placement for you (which is safer,
-especially for TPU training):
-
-```diff
-+ from accelerate import Accelerator
-
-+ accelerator = Accelerator()
-- my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
-+     my_model, my_optimizer, my_training_dataloader
++ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++     model, optimizer, training_dataloader, scheduler
 + )
 
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
+  for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+-   inputs = inputs.to(device)
+-   targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
++   accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
body: I would show first the sample where we don't change the device as letting Accelerate handle device placement changes more lines of code.
diff_hunk: @@ -12,121 +12,60 @@ specific language governing permissions and limitations under the License.
 
 # Accelerate
 
-Run your *raw* PyTorch training script on any kind of device.
-
-## Features
-
-- 🤗 Accelerate provides an easy API to make your scripts run with mixed precision and in any kind of distributed
-  setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run
-  seamlessly on your local machine for debugging or your training environment.
-
-- 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment and
-  then launch the scripts.
-
-
-## Easy to integrate
-
-A traditional training loop in PyTorch looks like this:
-
-```python
-my_model.to(device)
-
-for batch in my_training_dataloader:
-    my_optimizer.zero_grad()
-    inputs, targets = batch
-    inputs = inputs.to(device)
-    targets = targets.to(device)
-    outputs = my_model(inputs)
-    loss = my_loss_function(outputs, targets)
-    loss.backward()
-    my_optimizer.step()
-```
-
-Changing it to work with accelerate is really easy and only adds a few lines of code:
+🤗 Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just four lines of code! In short, training and inference at scale made simple, efficient and adaptable.
 
 ```diff
 + from accelerate import Accelerator
-
 + accelerator = Accelerator()
-  # Use the device given by the *accelerator* object.
-+ device = accelerator.device
-  my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerator.prepare(
-+     my_model, my_optimizer, my_training_dataloader
-+ )
-
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
-      inputs = inputs.to(device)
-      targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
-```
 
-and with this, your script can now run in a distributed environment (multi-GPU, TPU).
-
-You can even simplify your script a bit by letting 🤗 Accelerate handle the device placement for you (which is safer,
-especially for TPU training):
-
-```diff
-+ from accelerate import Accelerator
-
-+ accelerator = Accelerator()
-- my_model.to(device)
-  # Pass every important object (model, optimizer, dataloader) to *accelerator.prepare*
-+ my_model, my_optimizer, my_training_dataloader = accelerate.prepare(
-+     my_model, my_optimizer, my_training_dataloader
++ model, optimizer, training_dataloader, scheduler = accelerator.prepare(
++     model, optimizer, training_dataloader, scheduler
 + )
 
-  for batch in my_training_dataloader:
-      my_optimizer.zero_grad()
-      inputs, targets = batch
--     inputs = inputs.to(device)
--     targets = targets.to(device)
-      outputs = my_model(inputs)
-      loss = my_loss_function(outputs, targets)
-      # Just a small change for the backward instruction
--     loss.backward()
-+     accelerator.backward(loss)
-      my_optimizer.step()
+  for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+-   inputs = inputs.to(device)
body: The old page had way more information, which we should keep.
diff_hunk: @@ -0,0 +1,17 @@
+<!--Copyright 2021 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Accelerator
+
+The main [`Accelerator`] class:
body: ````suggestion
```python
````
Why remove this? We lose all formatting after...
diff_hunk: @@ -24,8 +24,8 @@ which can total to adding just one new line of code!
 
 This example will use a very simplistic PyTorch training loop that performs gradient accumulation every two batches:
 
-```python
-device = "cuda"
+```
body: Please use 4 spaces to indent
diff_hunk: @@ -86,18 +88,18 @@ of steps to perform before each call to `step()` and how to automatically adjust
 ```
 
 From here you can use the [`Accelerator.accumulate`] context manager from inside your training loop to automatically perform the gradient accumulation for you!
-You just wrap it around the entire training part of your code: 
+You just wrap it around the entire training part of our code: 
 
 ```diff
 - for index, batch in enumerate(training_dataloader):
 + for batch in training_dataloader:
 +    with accelerator.accumulate(model):
-          optimizer.zero_grad()
-          inputs, targets = batch
-          outputs = model(inputs)
+       optimizer.zero_grad()
body: Some content seems to be lost here as there is no usage guide.
diff_hunk: @@ -0,0 +1,19 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launchers
+
+Functions for launching training on distributed processes.
body: ```suggestion
In this case, 🤗 Accelerate will make some hyperparameter decisions for you, e.g., if GPUs are available, it will use all of them by default without the mixed precision.
```
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
body: ```suggestion
Here is how you would use all GPUs and train with mixed precision disabled:
```
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
body: maybe specify that this is one machine with two GPUs?
diff_hunk: @@ -0,0 +1,91 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Installation and Configuration
+
+Before you start, you will need to setup your environment, install the appropriate packages, and configure 🤗 Accelerate. 🤗 Accelerate is tested on **Python 3.7+**.
+
+## Installing 🤗 Accelerate
+
+🤗 Accelerate is available on pypi and conda, as well as on GitHub. Details to install from each are below:
+
+### pip 
+
+To install 🤗 Accelerate from pypi, perform:
+
+```bash
+pip install accelerate
+```
+
+### conda
+
+🤗 Accelerate can also be installed with conda with:
+
+```bash
+conda install -c conda-forge accelerate
+```
+
+### Source
+
+New features are added every day that haven't been released yet. To try them out yourself, install
+from the GitHub repository:
+
+```bash
+git clone https://github.com/huggingface/accelerate
+cd accelerate
+pip install -e .
+```
+
+## Configuring 🤗 Accelerate
+
+After installing, you need to configure 🤗 Accelerate for how the current system is setup for training. 
+To do so run the following and answer the questions prompted to you:
+
+```bash
+accelerate config
+```
+
+To write a barebones configuration that doesn't include options such as DeepSpeed configuration or running on TPUs, you can quickly run:
+
+```bash
+python -c "from accelerate.utils import write_basic_config; write_basic_config(mixed_precision='fp16')"
+```
+🤗 Accelerate will automatically utilize the maximum number of GPUs available and set the mixed precision mode.
+
+To check that your configuration looks fine, run:
+
+```bash
+accelerate env
+```
+
+An example output is shown below:
body: ```suggestion
This command wraps around all of the different commands needed to launch your script on various platforms, without you having to remember what each of them are.
```
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
body: ```suggestion
If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`.
```
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
body: why curly brackets here? i think also something like that would be clear enough:
```suggestion
accelerate launch your_script.py --arg1 --arg2 ...
```
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
body: what's the benefit of that? what happens if you start a random script with accelerate?
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision while avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a complete list of parameters you can pass in, run:
+```bash
+accelerate launch -h
+```
+
+<Tip>
+
+Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
body: regarding path_to_script.py: maybe just be consistent with the above name used.
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision while avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a complete list of parameters you can pass in, run:
+```bash
+accelerate launch -h
+```
+
+<Tip>
+
+Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
+
+</Tip>
+
+## Custom Configurations
+
+As briefly mentioned earlier, `accelerate launch` should be mostly used through combining set configurations 
+made with the `accelerate config` command. These configs are saved to a `default_config.yaml` file in your cache folder for 🤗 Accelerate. 
+This cache folder is located at (with decreasing order of priority):
+
+- The content of your environment variable `HF_HOME` suffixed with `accelerate`.
+- If it does not exist, the content of your environment variable `XDG_CACHE_HOME` suffixed with
+  `huggingface/accelerate`.
+- If this does not exist either, the folder `~/.cache/huggingface/accelerate`.
+
+To have multiple configurations, the flag `--config_file` can be passed to the `accelerate launch` command paired 
+with the location of the custom yaml. 
+
+An example yaml may look something like the following:
+```yaml
+compute_environment: LOCAL_MACHINE
+deepspeed_config: {}
+distributed_type: MULTI_GPU
+fsdp_config: {}
+machine_rank: 0
+main_process_ip: null
+main_process_port: null
+main_training_function: main
+mixed_precision: fp16
+num_machines: 1
+num_processes: 1
+use_cpu: false
+```
+
+An example of launching a script with a yaml located in a custom location would look something like so:
+```bash
+accelerate launch --config_file path/to/config/my_config_file.yaml path_to_script.py --args_for_the_script
body: `prepare` makes the "Setting the right device" a bit obsolete no? maybe make a note when to use `prepare` or `model.to`
diff_hunk: @@ -0,0 +1,119 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Migrating your code to 🤗 Accelerate
+
+This tutorial will detail how to easily convert existing PyTorch code to use 🤗 Accelerate!
+You'll see that by just changing a few lines of code, 🤗 Accelerate can perform its magic and get you on 
+your way towards running your code on distributed systems with ease!
+
+## The base training loop
+
+To begin, write out a very basic PyTorch training loop. 
+
+<Tip>
+We are under the presumption that `training_dataloader`, `model`, `optimizer`, `scheduler`, and `loss_function` have been defined beforehand.
+</Tip>
+
+```python
+device = "cuda"
+model.to(device)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    inputs = inputs.to(device)
+    targets = targets.to(device)
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    loss.backward()
+    optimizer.step()
+    scheduler.step()
+```
+
+## Add in 🤗 Accelerate
+
+To start using 🤗 Accelerate, first import and create an [`Accelerator`] instance:
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+```
+[`Accelerator`] is the main force behind utilizing all the possible options for distributed training!
+
+### Setting the right device
+
+The [`Accelerator`] class knows the right device to move any PyTorch object to at any time, so you should
+change the definition of `device` to come from [`Accelerator`]:
+
+```diff
+- device = 'cuda'
++ device = accelerator.device
+  model.to(device)
+```
+
+### Preparing your objects
+
+Next you need to pass all of the important objects related to training into [`Accelerator.prepare`]. 🤗 Accelerate will
+make sure everything is setup in the current environment for you to start training:
+
+```
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+```
+These objects are returned in the same order they were sent in with. By default, `device_placement=True`, so your model is automatically moved to the proper device.
+performing this action will also automatically move your model to the proper device as well.
body: agree, a few examples would be useful here. also a graphic could be useful.
diff_hunk: @@ -0,0 +1,29 @@
+# Deferring Executions
+
+When you run your usual script, instructions are executed in order. Using 🤗 Accelerate to deploy your script on several
+GPUs at the same time introduces a complication: while each process executes all instructions in order, some may be
+faster than others.
+
+You might need to wait for all processes to have reached a certain point before executing a given instruction. For
+instance, you shouldn't save a model before being sure every process is done with training, and you wouldn't want to 
+continue training before all the model weights have been loaded in. To do this, just write the following line in your code:
+
+```
+accelerator.wait_for_everyone()
+```
+
+This instruction will block all the processes that arrive first until all the other processes have reached that
+point (if you run your script on just one GPU or CPU, this won't do anything).
+
+A few example cases for when to use this utility are listed below:
+
+<Tip>
+
+Some of these are utilized with the [`Accelerator.main_process_first`] context manager, which utilizes [`Accelerator.wait_for_everyone`] to 
+run a particular set of code on the main process beforehand before triggering and launching the other processes
+
+</Tip>
+
+- Downloading a dataset on the main process first and then loading the cached dataset in afterwards
body: shouldn't you use the target to get a loss?
```suggestion
            ddp_model(input, target).backward()
```
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++ from accelerate import Accelerator
++ accelerator = Accelerator()
+  import torch.nn as nn
+- from torch.nn.parallel import DistributedDataParallel
+
+  model = nn.Linear(10,10)
++ model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this risk slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
+need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
+can you avoid this overhead?
+
+## Solving the slowdown problem
+
+Since you are skipping these batches, their gradients do not need to be synchronized until the point where `.backward()` is actually called. 
+PyTorch cannot automagically tell when you need to do this, but they do provide a tool to help through the [`no_sync`](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync) context manager
+that is added to your model after converting it to DDP.
+
+Under this context manager, PyTorch will skip synchronizing the gradients when `.backward()` is called, and the first call to `.backward()` outside this 
+context manager will trigger the synchronization. See an example below:
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for index, batch in enumerate(dataloader):
+    input, target = batch
+    # Trigger gradient synchronization on the last batch
+    if index != (len(dataloader) - 1):
+        with ddp_model.no_sync():
+            # Gradients only accumulate
+            ddp_model(input).backward()
body: same as above
```suggestion
        ddp_model(input, target).backward()
```
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++ from accelerate import Accelerator
++ accelerator = Accelerator()
+  import torch.nn as nn
+- from torch.nn.parallel import DistributedDataParallel
+
+  model = nn.Linear(10,10)
++ model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this risk slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
+need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
+can you avoid this overhead?
+
+## Solving the slowdown problem
+
+Since you are skipping these batches, their gradients do not need to be synchronized until the point where `.backward()` is actually called. 
+PyTorch cannot automagically tell when you need to do this, but they do provide a tool to help through the [`no_sync`](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync) context manager
+that is added to your model after converting it to DDP.
+
+Under this context manager, PyTorch will skip synchronizing the gradients when `.backward()` is called, and the first call to `.backward()` outside this 
+context manager will trigger the synchronization. See an example below:
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for index, batch in enumerate(dataloader):
+    input, target = batch
+    # Trigger gradient synchronization on the last batch
+    if index != (len(dataloader) - 1):
+        with ddp_model.no_sync():
+            # Gradients only accumulate
+            ddp_model(input).backward()
+    else:
+        # Gradients finally sync
+        ddp_model(input).backward()
body: same as above
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++ from accelerate import Accelerator
++ accelerator = Accelerator()
+  import torch.nn as nn
+- from torch.nn.parallel import DistributedDataParallel
+
+  model = nn.Linear(10,10)
++ model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this risk slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
+need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
+can you avoid this overhead?
+
+## Solving the slowdown problem
+
+Since you are skipping these batches, their gradients do not need to be synchronized until the point where `.backward()` is actually called. 
+PyTorch cannot automagically tell when you need to do this, but they do provide a tool to help through the [`no_sync`](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync) context manager
+that is added to your model after converting it to DDP.
+
+Under this context manager, PyTorch will skip synchronizing the gradients when `.backward()` is called, and the first call to `.backward()` outside this 
+context manager will trigger the synchronization. See an example below:
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for index, batch in enumerate(dataloader):
+    input, target = batch
+    # Trigger gradient synchronization on the last batch
+    if index != (len(dataloader) - 1):
+        with ddp_model.no_sync():
+            # Gradients only accumulate
+            ddp_model(input).backward()
+    else:
+        # Gradients finally sync
+        ddp_model(input).backward()
+```
+
+In 🤗 Accelerate to make this an API that can be called no matter the training device (though it may not do anything if you are not in a distributed system!),
+`ddp_model.no_sync` gets replaced with [`Accelerator.no_sync`] and operates the same way:
+
+```diff
+  ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+  for index, batch in enumerate(dataloader):
+      input, target = batch
+      # Trigger gradient synchronization on the last batch
+      if index != (len(dataloader)-1):
+-         with ddp_model.no_sync():
++         with accelerator.no_sync(model):
+              # Gradients only accumulate
+              ddp_model(input).backward()
body: If you want to make it a bit more concise here, you could just show the right way here and just say: if you initialize the model inside the function this might happen <show error message and explanation>.
diff_hunk: @@ -0,0 +1,154 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Training on TPUs with 🤗 Accelerate
+
+Training on TPUs can be slightly different than training on multi-gpu, even with 🤗 Accelerate. This guide aims to show you 
+where you should be careful and why, as well as the best practices in general.
+
+## Training in a Notebook
+
+The main carepoint when training on TPUs comes from the `notebook_launcher`. As mentioned in the [notebook tutorial](https://huggingface.co/docs/accelerate/basic_tutorials/notebook), you need to 
+restructure your training code into a function that can get passed to the `notebook_launcher` function and be careful about not declaring any tensors on the GPU.
+
+While on a TPU that last part is not as important, a critical part to understand is that when you launch code from a notebook you do so through a process called **forking**. 
+When launching from the command-line, you perform **spawning**, where a python process is not currently running and you *spawn* a new process in. Since your Jupyter notebook is already 
+utilizing a python process, you need to *fork* a new process from it to launch your code. 
+
+Where this becomes important is in regards to declaring your model. On forked TPU processes, it is recommended that you instantiate your model *once* and pass this into your 
+training function. This is different than training on GPUs where you create `n` models that have their gradients synced and back-propagated at certain moments. Instead one 
+model instance is shared between all the nodes and it is passed back and forth. This is important especially when training on low-resource TPUs such as those provided in Kaggle kernels or
+on Google Colaboratory. 
+
+Below is an example of a training function passed to the `notebook_launcher` if training on CPUs or GPUs:
+
+<Tip>
+
+This code snippet is based off the one from the `simple_nlp_example` notebook found [here](https://github.com/huggingface/notebooks/blob/main/examples/accelerate/simple_nlp_example.ipynb) with slight 
+modifications for the sake of simplicity
+
+</Tip>
+
+```python
+def training_function():
+    # Initialize accelerator
+    accelerator = Accelerator()
+    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+    train_dataloader, eval_dataloader = create_dataloaders(
+        train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+    )
+
+    # Instantiate optimizer
+    optimizer = AdamW(params=model.parameters(), lr=hyperparameters["learning_rate"])
+
+    # Prepare everything
+    # There is no specific order to remember, we just need to unpack the objects in the same order we gave them to the
+    # prepare method.
+    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader
+    )
+
+    num_epochs = hyperparameters["num_epochs"]
+    # Now we train the model
+    for epoch in range(num_epochs):
+        model.train()
+        for step, batch in enumerate(train_dataloader):
+            outputs = model(**batch)
+            loss = outputs.loss
+            accelerator.backward(loss)
+
+            optimizer.step()
+            optimizer.zero_grad()
+```
+
+```python
+from accelerate import notebook_launcher
+
+notebook_launcher(training_function)
+```
+
+<Tip>
+
+The `notebook_launcher` will default to 8 processes if 🤗 Accelerate has been configured for a TPU
+
+</Tip>
+
+If you use this example and declare the model *inside* the training loop, then on a low-resource system you will potentially see an error 
+like:
+
+```
+ProcessExitedException: process 0 terminated with signal SIGSEGV
+```
+
+This error is *extremely* cryptic but the basic explaination is you ran out of system RAM. You can avoid this entirely by reconfiguring the training function to 
+accept a single `model` argument, and declare it in an outside cell:
+
+```python
+# In another Jupyter cell
+model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+```
+
+```diff
++ def training_function(model):
+      # Initialize accelerator
+      accelerator = Accelerator()
+-     model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+      train_dataloader, eval_dataloader = create_dataloaders(
+          train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+      )
+  ...
+```
+And finally calling the training function with:
+
+```diff
+  from accelerate import notebook_launcher
+- notebook_launcher(training_function)
++ notebook_launcher(training_function, (model,))
+```
+
body: can you maybe explain a bit what the difference to the first setting is and what the impact is and when you should use which?
diff_hunk: @@ -0,0 +1,154 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Training on TPUs with 🤗 Accelerate
+
+Training on TPUs can be slightly different than training on multi-gpu, even with 🤗 Accelerate. This guide aims to show you 
+where you should be careful and why, as well as the best practices in general.
+
+## Training in a Notebook
+
+The main carepoint when training on TPUs comes from the `notebook_launcher`. As mentioned in the [notebook tutorial](https://huggingface.co/docs/accelerate/basic_tutorials/notebook), you need to 
+restructure your training code into a function that can get passed to the `notebook_launcher` function and be careful about not declaring any tensors on the GPU.
+
+While on a TPU that last part is not as important, a critical part to understand is that when you launch code from a notebook you do so through a process called **forking**. 
+When launching from the command-line, you perform **spawning**, where a python process is not currently running and you *spawn* a new process in. Since your Jupyter notebook is already 
+utilizing a python process, you need to *fork* a new process from it to launch your code. 
+
+Where this becomes important is in regards to declaring your model. On forked TPU processes, it is recommended that you instantiate your model *once* and pass this into your 
+training function. This is different than training on GPUs where you create `n` models that have their gradients synced and back-propagated at certain moments. Instead one 
+model instance is shared between all the nodes and it is passed back and forth. This is important especially when training on low-resource TPUs such as those provided in Kaggle kernels or
+on Google Colaboratory. 
+
+Below is an example of a training function passed to the `notebook_launcher` if training on CPUs or GPUs:
+
+<Tip>
+
+This code snippet is based off the one from the `simple_nlp_example` notebook found [here](https://github.com/huggingface/notebooks/blob/main/examples/accelerate/simple_nlp_example.ipynb) with slight 
+modifications for the sake of simplicity
+
+</Tip>
+
+```python
+def training_function():
+    # Initialize accelerator
+    accelerator = Accelerator()
+    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+    train_dataloader, eval_dataloader = create_dataloaders(
+        train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+    )
+
+    # Instantiate optimizer
+    optimizer = AdamW(params=model.parameters(), lr=hyperparameters["learning_rate"])
+
+    # Prepare everything
+    # There is no specific order to remember, we just need to unpack the objects in the same order we gave them to the
+    # prepare method.
+    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader
+    )
+
+    num_epochs = hyperparameters["num_epochs"]
+    # Now we train the model
+    for epoch in range(num_epochs):
+        model.train()
+        for step, batch in enumerate(train_dataloader):
+            outputs = model(**batch)
+            loss = outputs.loss
+            accelerator.backward(loss)
+
+            optimizer.step()
+            optimizer.zero_grad()
+```
+
+```python
+from accelerate import notebook_launcher
+
+notebook_launcher(training_function)
+```
+
+<Tip>
+
+The `notebook_launcher` will default to 8 processes if 🤗 Accelerate has been configured for a TPU
+
+</Tip>
+
+If you use this example and declare the model *inside* the training loop, then on a low-resource system you will potentially see an error 
+like:
+
+```
+ProcessExitedException: process 0 terminated with signal SIGSEGV
+```
+
+This error is *extremely* cryptic but the basic explaination is you ran out of system RAM. You can avoid this entirely by reconfiguring the training function to 
+accept a single `model` argument, and declare it in an outside cell:
+
+```python
+# In another Jupyter cell
+model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+```
+
+```diff
++ def training_function(model):
+      # Initialize accelerator
+      accelerator = Accelerator()
+-     model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+      train_dataloader, eval_dataloader = create_dataloaders(
+          train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+      )
+  ...
+```
+And finally calling the training function with:
+
+```diff
+  from accelerate import notebook_launcher
+- notebook_launcher(training_function)
++ notebook_launcher(training_function, (model,))
+```
+
+## Mixed Precision and Global Variables 
+
+As mentioned in the [mixed precision tutorial](https://huggingface.co/docs/accelerate/usage_guides/mixed_precision), 🤗 Accelerate supports fp16 and bf16, both of which can be used on TPUs.
+That being said, ideally `bf16` should be utilized as it is extremely efficient to use.
+
+There are two "layers" when using `bf16` and 🤗 Accelerate on TPUs, at the base level and at the operation level. 
+
+At the base level, this is enabled when passing `mixed_precision="bf16"` to `Accelerator`, such as:
+```python
+accelerator = Accelerator(mixed_precision="bf16")
+```
+By default this will cast `torch.float` and `torch.double` to `bfloat16` on TPUs. 
+The specific configuration being set is an environmental variable of `XLA_USE_BF16` is set to `1`.
+
+There is a futher configuration you can perform which is setting the `XLA_DOWNCAST_BF16` environmental variable. If set to `1`, then 
+`torch.float` is `bfloat16` and `torch.double` is `float32`.
body: It will remove the boilerplate needed when calling `torchrun` XYZ. I can expand this further to point it out, by translating the earlier bash with accelerate launch to show the equivalence. 
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+Just put `accelerate launch` at the start of your command, and pass in additional arguments and parameters to your script afterwards like normal!
+
+Since this runs the various torch spawn methods, all of the expected environment variables can be modified here as well.
+For example, here is how to use `accelerate launch` with a single GPU:
+```bash
+CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...
+```
+
+You can also use `accelerate launch` without performing `accelerate config` first, but you may need to manually pass in the right configuration parameters.
+In this case, 🤗 Accelerate will make some hyperparameter decisions for you, such as if a GPU is available, it will use all of them by default, and no mixed precision will be used. 
+Here is how you would use all GPUs and train on no mixed precision:
+```bash
+accelerate launch --multi_gpu {script_name.py} --arg1 --arg2 ...
+```
+To get more specific you should pass in the needed parameters yourself. For instance, here is how you 
+would also launch that same script on two GPUs using mixed precision while avoiding all of the warnings: 
+```bash
+accelerate launch --multi_gpu --mixed_precision=fp16 --num_processes=2 {script_name.py} --arg1 --arg2 ...
+```
+
+For a complete list of parameters you can pass in, run:
+```bash
+accelerate launch -h
+```
+
+<Tip>
+
+Even if you are not using 🤗 Accelerate in your code, you can still use the launcher for starting your scripts!
body: I skipped over calculating the loss in this case, to simply show calling `.backward()`. but I'll include a loss calculation to make it a bit more clear
diff_hunk: @@ -0,0 +1,109 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Gradient Synchronization
+
+PyTorch's distributed module operates by communicating back and forth between all of the GPUs in your system.
+This communication takes time, and ensuring all processes know the states of each other happens at particular triggerpoints
+when using the `ddp` module. 
+
+These triggerpoints are added to the PyTorch model, specifically their `forward()` and `backward()` methods. 
+This happens when the model is wrapped with `DistributedDataParallel`:
+```python
+import torch.nn as nn
+from torch.nn.parallel import DistributedDataParallel
+
+model = nn.Linear(10, 10)
+ddp_model = DistributedDataParallel(model)
+```
+In 🤗 Accelerate this conversion happens automatically when calling [`Accelerator.prepare`] and passing in your model.
+
+```diff
++ from accelerate import Accelerator
++ accelerator = Accelerator()
+  import torch.nn as nn
+- from torch.nn.parallel import DistributedDataParallel
+
+  model = nn.Linear(10,10)
++ model = accelerator.prepare(model)
+```
+
+## The slowdown in gradient accumulation
+
+You now understand that PyTorch adds hooks to the `forward` and `backward` method of your PyTorch model when 
+training in a distributed setup. But how does this risk slowing down your code?
+
+In DDP (distributed data parallel), the specific order in which processes are performed and ran are expected
+at specific points and these must also occur at roughly the same time before moving on.
+
+The most direct example is when you update all of the parameters in a model through `.backward()`. All instances of the model
+need to have updated their gradients, collated, and updated again before moving onto the next batch of data. But when performing 
+gradient accumulation, you accumulate `n` losses and skip `.backward()` until `n` batches have been reached. This 
+can cause a significant slowdown since all the processes need to communicate with them more times than needed. How 
+can you avoid this overhead?
+
+## Solving the slowdown problem
+
+Since you are skipping these batches, their gradients do not need to be synchronized until the point where `.backward()` is actually called. 
+PyTorch cannot automagically tell when you need to do this, but they do provide a tool to help through the [`no_sync`](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync) context manager
+that is added to your model after converting it to DDP.
+
+Under this context manager, PyTorch will skip synchronizing the gradients when `.backward()` is called, and the first call to `.backward()` outside this 
+context manager will trigger the synchronization. See an example below:
+```python
+ddp_model, dataloader = accelerator.prepare(model, dataloader)
+
+for index, batch in enumerate(dataloader):
+    input, target = batch
+    # Trigger gradient synchronization on the last batch
+    if index != (len(dataloader) - 1):
+        with ddp_model.no_sync():
+            # Gradients only accumulate
+            ddp_model(input).backward()
body: I made it less concise since one example shows running a regular script vs this one is for the tpu launcher in a notebook, but will think on this a bit more
diff_hunk: @@ -0,0 +1,154 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Training on TPUs with 🤗 Accelerate
+
+Training on TPUs can be slightly different than training on multi-gpu, even with 🤗 Accelerate. This guide aims to show you 
+where you should be careful and why, as well as the best practices in general.
+
+## Training in a Notebook
+
+The main carepoint when training on TPUs comes from the `notebook_launcher`. As mentioned in the [notebook tutorial](https://huggingface.co/docs/accelerate/basic_tutorials/notebook), you need to 
+restructure your training code into a function that can get passed to the `notebook_launcher` function and be careful about not declaring any tensors on the GPU.
+
+While on a TPU that last part is not as important, a critical part to understand is that when you launch code from a notebook you do so through a process called **forking**. 
+When launching from the command-line, you perform **spawning**, where a python process is not currently running and you *spawn* a new process in. Since your Jupyter notebook is already 
+utilizing a python process, you need to *fork* a new process from it to launch your code. 
+
+Where this becomes important is in regards to declaring your model. On forked TPU processes, it is recommended that you instantiate your model *once* and pass this into your 
+training function. This is different than training on GPUs where you create `n` models that have their gradients synced and back-propagated at certain moments. Instead one 
+model instance is shared between all the nodes and it is passed back and forth. This is important especially when training on low-resource TPUs such as those provided in Kaggle kernels or
+on Google Colaboratory. 
+
+Below is an example of a training function passed to the `notebook_launcher` if training on CPUs or GPUs:
+
+<Tip>
+
+This code snippet is based off the one from the `simple_nlp_example` notebook found [here](https://github.com/huggingface/notebooks/blob/main/examples/accelerate/simple_nlp_example.ipynb) with slight 
+modifications for the sake of simplicity
+
+</Tip>
+
+```python
+def training_function():
+    # Initialize accelerator
+    accelerator = Accelerator()
+    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+    train_dataloader, eval_dataloader = create_dataloaders(
+        train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+    )
+
+    # Instantiate optimizer
+    optimizer = AdamW(params=model.parameters(), lr=hyperparameters["learning_rate"])
+
+    # Prepare everything
+    # There is no specific order to remember, we just need to unpack the objects in the same order we gave them to the
+    # prepare method.
+    model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(
+        model, optimizer, train_dataloader, eval_dataloader
+    )
+
+    num_epochs = hyperparameters["num_epochs"]
+    # Now we train the model
+    for epoch in range(num_epochs):
+        model.train()
+        for step, batch in enumerate(train_dataloader):
+            outputs = model(**batch)
+            loss = outputs.loss
+            accelerator.backward(loss)
+
+            optimizer.step()
+            optimizer.zero_grad()
+```
+
+```python
+from accelerate import notebook_launcher
+
+notebook_launcher(training_function)
+```
+
+<Tip>
+
+The `notebook_launcher` will default to 8 processes if 🤗 Accelerate has been configured for a TPU
+
+</Tip>
+
+If you use this example and declare the model *inside* the training loop, then on a low-resource system you will potentially see an error 
+like:
+
+```
+ProcessExitedException: process 0 terminated with signal SIGSEGV
+```
+
+This error is *extremely* cryptic but the basic explaination is you ran out of system RAM. You can avoid this entirely by reconfiguring the training function to 
+accept a single `model` argument, and declare it in an outside cell:
+
+```python
+# In another Jupyter cell
+model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+```
+
+```diff
++ def training_function(model):
+      # Initialize accelerator
+      accelerator = Accelerator()
+-     model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
+      train_dataloader, eval_dataloader = create_dataloaders(
+          train_batch_size=hyperparameters["train_batch_size"], eval_batch_size=hyperparameters["eval_batch_size"]
+      )
+  ...
+```
+And finally calling the training function with:
+
+```diff
+  from accelerate import notebook_launcher
+- notebook_launcher(training_function)
++ notebook_launcher(training_function, (model,))
+```
+
body: Thanks for pointing this out! Will make sure to have a `notebook.mdx` as part of the `basic_tutorials` before merging
diff_hunk: @@ -0,0 +1,19 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launchers
+
+Functions for launching training on distributed processes.
body: Accident 😓 
diff_hunk: @@ -24,8 +24,8 @@ which can total to adding just one new line of code!
 
 This example will use a very simplistic PyTorch training loop that performs gradient accumulation every two batches:
 
-```python
-device = "cuda"
+```
body: With curly brackets the reader knows right away that it's meant to represent templating, though I didn't propagate this well enough through the docs so will make sure to do so
diff_hunk: @@ -0,0 +1,148 @@
+<!--Copyright 2022 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+-->
+
+# Launching your 🤗 Accelerate scripts
+
+In the previous tutorial, you were introduced to how to modify your current training script to use 🤗 Accelerate.
+The final version of that code is shown below:
+
+```python
+from accelerate import Accelerator
+
+accelerator = Accelerator()
+
+model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+    model, optimizer, training_dataloader, scheduler
+)
+
+for batch in training_dataloader:
+    optimizer.zero_grad()
+    inputs, targets = batch
+    outputs = model(inputs)
+    loss = loss_function(outputs, targets)
+    accelerator.backward(loss)
+    optimizer.step()
+    scheduler.step()
+```
+
+But how do you run this code and have it utilize the special hardware available to it?
+
+First you should rewrite the above code into a function, and make it callable as a script. For example:
+
+```diff
+  from accelerate import Accelerator
+  
++ def main():
+
+    accelerator = Accelerator()
+
+    model, optimizer, training_dataloader, scheduler = accelerator.prepare(
+        model, optimizer, training_dataloader, scheduler
+    )
+
+    for batch in training_dataloader:
+        optimizer.zero_grad()
+        inputs, targets = batch
+        outputs = model(inputs)
+        loss = loss_function(outputs, targets)
+        accelerator.backward(loss)
+        optimizer.step()
+        scheduler.step()
+
++ if __name__ == "__main__":
++   main()
+```
+
+Next you need to launch it with `accelerate launch`. 
+
+## Using accelerate launch
+
+🤗 Accelerate has a special CLI command to help you launch your code in your system through `accelerate launch`.
+This command wraps around all of the different commands needed to launch your script on various platforms, without you
+having to remember what each of them are.
+
+<Tip>
+
+If you are familiar with launching scripts in PyTorch yourself such as with `torchrun`, you can still do this. It is not required to use `accelerate launch`
+
+</Tip>
+
+You can launch your script quickly by using:
+```bash
+accelerate launch {script_name.py} --arg1 --arg2 ...
body: Move the file back to the root maybe? It would be more consistent.
diff_hunk: @@ -1,38 +1,68 @@
-- sections: 
+- sections:
   - local: index
     title: 🤗 Accelerate
-  - local: quicktour
-    title: Quick tour
-  - local: installation
+  - local: basic_tutorials/install

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: @pacman100 the difference between the two is something like the following:
- How-To guides are how do you do things with **Accelerate**, e.g. how to perform gradient accumulation, how to perform logging, how to perform tracking (think our example scripts!)
- Concepts and Fundamentals is given this thing accelerate does, here's an understanding of some of the harder-to-grasp magic it does underneath that you should know more about as you become more familiar with the library. E.g. Gradient synchronization is how we do gradient accumulation, so that's one example where we took an applied concept taught in the higher-level tutorial and took it to a lower level.

- Re: example scripts, it'll be in this PR eventually, if not a follow up. Still thinking on a good design for integration. Right now this is a very tl;dr version to make sure everyone is aligned (which we are!) before getting too deep, and now I'll move towards migrating the full docs in (avoiding your reference issues, etc, etc)