diff --git a/.gitignore b/.gitignore
index b6e47617d..3b89c957d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -127,3 +127,6 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# VSCode
+.vscode
diff --git a/README.md b/README.md
index 781eea3c9..fcc20a12a 100644
--- a/README.md
+++ b/README.md
@@ -34,7 +34,98 @@
 <p>Run your *raw* PyTorch training script on any kind of device
 </h3>
 
-🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then runs seamlessly on your local machine for debugging or your training environment.
+🤗 Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boiler code needed to use multi-GPUs/TPU/fp16.
+
+🤗 Accelerate abstracts exactly and only the boiler code related to multi-GPUs/TPU/fp16 and let the rest of your code unchanged.
+
+Here is an example:
+
+<table>
+<tr>
+<th> Original training code (CPU or mono-GPU only)</th>
+<th> With Accelerate for CPU/GPU/multi-GPUs/TPUs/fp16 </th>
+</tr>
+<tr>
+<td>
+
+```python
+import torch
+import torch.nn.functional as F
+from datasets import load_dataset
+
+
+
+device = 'cpu'
+
+model = torch.nn.Transformer().to(device)
+optim = torch.optim.Adam(model.parameters())
+
+dataset = load_dataset('my_dataset')
+data = torch.utils.data.Dataloader(dataset)
+
+
+
+
+model.train()
+for epoch in range(10):
+    for source, targets in data:
+        source = source.to(device)
+        targets = targets.to(device)
+
+        optimizer.zero_grad()
+
+        output = model(source, targets)
+        loss = F.cross_entropy(output, targets)
+
+        loss.backward()
+
+        optimizer.step()
+```
+
+</td>
+<td>
+
+```python
+  import torch
+  import torch.nn.functional as F
+  from datasets import load_dataset
+
++ from accelerate import Accelerator
++ accelerator = Accelerator()
++ device = accelerator.device
+
+  model = torch.nn.Transformer().to(device)
+  optim = torch.optim.Adam(model.parameters())
+
+  dataset = load_dataset('my_dataset')
+  data = torch.utils.data.Dataloader(dataset)
+
++ model, optim, data = accelerator.prepare(
+                            model, optim, data)
+
+  model.train()
+  for epoch in range(10):
+      for source, targets in data:
+          source = source.to(device)
+          targets = targets.to(device)
+
+          optimizer.zero_grad()
+
+          output = model(source, targets)
+          loss = F.cross_entropy(output, targets)
+
++         accelerate.backward(loss)
+
+          optimizer.step()
+```
+
+</td>
+</tr>
+</table>
+
+As you can see on this example, by adding 5-lines to any standard PyTorch training script you can now run on any kind of single or distributed node setting (single CPU, single GPU, multi-GPUs and TPUs) as well as with or without mixed precision (fp16).
+
+The same code can then in paticular run without modification on your local machine for debugging or your training environment.
 
 🤗 Accelerate also provides a CLI tool that allows you to quickly configure and test your training environment then launch the scripts.
 
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 000000000..cb1d2645b
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,75 @@
+# In this folder we showcase various full examples using `Accelerate`
+
+## Simple NLP example
+
+The [simple_example.py](./simple_example.py) script is a simple example to train a Bert model on a classification task ([GLUE's MRPC]()).
+
+The same script can be run in any of the following configurations:
+- single CPU or single GPU
+- multi GPUS (using PyTorch distributed mode)
+- (multi) TPUs
+- fp16 (mixed-precision) or fp32 (normal precision)
+
+To run it in each of these various modes, use the following commands:
+- single CPU:
+    * from a server without GPU
+        ```bash
+        python ./simple_example.py
+        ```
+    * from any server by passing `cpu=True` to the `Accelerator`.
+        ```bash
+        python ./simple_example.py --cpu
+        ```
+    * from any server with Accelerate launcher
+        ```bash
+        accelerate launch --cpu ./simple_example.py
+        ```
+- single GPU:
+    ```bash
+    python ./simple_example.py  # from a server with a GPU
+    ```
+- with fp16 (mixed-precision)
+    * from any server by passing `fp16=True` to the `Accelerator`.
+        ```bash
+        python ./simple_example.py --fp16
+        ```
+    * from any server with Accelerate launcher
+        ```bash
+        accelerate launch --fb16 ./simple_example.py
+- multi GPUS (using PyTorch distributed mode)
+    * With Accelerate config and launcher
+        ```bash
+        accelerate config  # This will create a config file on your server
+        accelerate launch ./simple_example.py  # This will run the script on your server
+        ```
+    * With traditional PyTorch launcher
+        ```bash
+        python -m torch.distributed.launch --nproc_per_node 2 --use_env ./simple_example.py
+        ```
+- multi GPUs, multi node (several machines, using PyTorch distributed mode)
+    * With Accelerate config and launcher, on each machine:
+        ```bash
+        accelerate config  # This will create a config file on each server
+        accelerate launch ./simple_example.py  # This will run the script on each server
+        ```
+    * With PyTorch launcher only
+        ```bash
+        python -m torch.distributed.launch --nproc_per_node 2 \
+            --use_env \
+            --node_rank 0 \
+            --master_addr master_node_ip_address \
+            ./simple_example.py  # On the first server
+        python -m torch.distributed.launch --nproc_per_node 2 \
+            --use_env \
+            --node_rank 1 \
+            --master_addr master_node_ip_address \
+            ./simple_example.py  # On the second server
+        ```
+- (multi) TPUs
+    * With Accelerate config and launcher
+        ```bash
+        accelerate config  # This will create a config file on your TPU server
+        accelerate launch ./simple_example.py  # This will run the script on each server
+        ```
+    * In PyTorch:
+        Add an `xmp.spawn` line in your script as you usually do.
diff --git a/examples/simple_example.py b/examples/simple_example.py
index dbbfd22f8..88c81e3a0 100644
--- a/examples/simple_example.py
+++ b/examples/simple_example.py
@@ -11,11 +11,28 @@
 )
 
 
+########################################################################
+# This is a fully working simple example to use Accelerate
+#
+# This example train a Bert base model on GLUE MRPC
+# in any of the following settings (with the same script):
+#   - single CPU or single GPU
+#   - multi GPUS (using PyTorch distributed mode)
+#   - (multi) TPUs
+#   - fp16 (mixed-precision) or fp32 (normal precision)
+#
+# To run it in each of these various modes, follow the instructions
+# in the readme for examples:
+# https://github.com/huggingface/accelerate/examples
+#
+########################################################################
+
+
 MAX_GPU_BATCH_SIZE = 16
 EVAL_BATCH_SIZE = 32
 
 
-def training_function(config):
+def training_function(config, args):
     # Sample hyper-parameters for learning rate, batch size, seed and a few other HPs
     lr = config["lr"]
     num_epochs = int(config["num_epochs"])
@@ -62,7 +79,7 @@ def collate_fn(examples):
 
     set_seed(seed)
     # Initialize accelerator
-    accelerator = Accelerator(device_placement=False)
+    accelerator = Accelerator(device_placement=False, fp16=args.fp16, cpu=args.cpu)
 
     # Instantiate the model (we build the model here so that the seed also control new weights initialization)
     model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", return_dict=True)
@@ -117,8 +134,20 @@ def collate_fn(examples):
 
 
 def main():
+    parser = argparse.ArgumentParser(description="Simple example of training script.")
+    parser.add_argument(
+        "--fp16",
+        action="store_true",
+        help="If passed, will use FP16 training.",
+    )
+    parser.add_argument(
+        "--cpu",
+        action="store_true",
+        help="If passed, will train on the CPU.",
+    )
+    args = parser.parse_args()
     config = {"lr": 2e-5, "num_epochs": 3, "correct_bias": True, "seed": 42, "batch_size": 16}
-    training_function(config)
+    training_function(config, args)
 
 
 if __name__ == "__main__":
diff --git a/src/accelerate/accelerator.py b/src/accelerate/accelerator.py
index 96e3b2c86..a18f31372 100644
--- a/src/accelerate/accelerator.py
+++ b/src/accelerate/accelerator.py
@@ -14,7 +14,7 @@ class Accelerator:
 
     Args:
         device_placement (:obj:`bool`, `optional`, defaults to :obj:`True`):
-            Whether or not the accelerator should put objects on device (tensors yielded by the datalaoder, model,
+            Whether or not the accelerator should put objects on device (tensors yielded by the dataloader, model,
             etc...).
         split_batches (:obj:`bool`, `optional`, defaults to :obj:`False`):
             Whether or not the accelerator should split the batches yielded by the dataloaders across the devices. If
@@ -38,6 +38,7 @@ def __init__(
         self, device_placement: bool = True, split_batches: bool = False, fp16: bool = None, cpu: bool = False
     ):
         self.state = AcceleratorState(fp16=fp16, cpu=cpu, _from_accelerator=True)
+
         self.device_placement = device_placement
         self.split_batches = split_batches
 
@@ -99,6 +100,13 @@ def prepare(self, *args):
         """
         Prepare all objects passed in :obj:`args` for distributed training and mixed precision, then return them in the
         same order.
+
+        Accepts the following type of objects:
+
+            - :obj:`torch.utils.data.DataLoader`: PyTorch Dataloader
+            - :obj:`torch.nn.Module`: PyTorch Module
+            - :obj:`torch.optim.Optimizer`: PyTorch Optimizer
+
         """
         # On TPUs, putting the model on the XLA device will create new parameters, so the corresponding optimizer will
         # have parameters disconnected from the model (so no training :-( ).
diff --git a/src/accelerate/commands/launch.py b/src/accelerate/commands/launch.py
index d559d0b0b..601cba147 100644
--- a/src/accelerate/commands/launch.py
+++ b/src/accelerate/commands/launch.py
@@ -32,6 +32,9 @@ def launch_command_parser(subparsers=None):
     parser.add_argument(
         "--fp16", default=False, action="store_true", help="Whether or not to use mixed precision training."
     )
+    parser.add_argument(
+        "--cpu", default=False, action="store_true", help="Whether or not to force the training on the CPU."
+    )
     parser.add_argument(
         "--num_processes", type=int, default=None, help="The total number of processes to be launched in parallel."
     )
@@ -77,6 +80,7 @@ def simple_launcher(args):
     cmd.extend(args.training_script_args)
 
     current_env = os.environ.copy()
+    current_env["USE_CPU"] = str(args.cpu)
     current_env["USE_FP16"] = str(args.fp16)
 
     process = subprocess.Popen(cmd, env=current_env)
@@ -143,7 +147,7 @@ def launch_command(args):
         raise ValueError("You can only pick one between `--multi_gpu` and `--tpu`.")
 
     # Get the default from the config file.
-    if args.config_file is not None or os.path.isfile(default_config_file):
+    if args.config_file is not None or os.path.isfile(default_config_file) and not args.cpu:
         defaults = LaunchConfig.from_json_file(json_file=args.config_file)
         if not args.multi_gpu and not args.tpu:
             args.multi_gpu = defaults.distributed_type == DistributedType.MULTI_GPU
@@ -157,9 +161,9 @@ def launch_command(args):
             args.num_processes = 1
 
     # Use the proper launcher
-    if args.multi_gpu:
+    if args.multi_gpu and not args.cpu:
         multi_gpu_launcher(args)
-    elif args.tpu:
+    elif args.tpu and not args.cpu:
         tpu_launcher(args)
     else:
         simple_launcher(args)


Review comments: body: ```suggestion
+ accelerator = Accelerator(device_placement=False)
```
The default is still `True` as it's safer for the user (and it still takes one line of code to have the default to `False`). If the table has space enough, I'd add the same example with the device placement handled by `accelerate` (like the current example below).
diff_hunk: @@ -34,7 +34,98 @@
 <p>Run your *raw* PyTorch training script on any kind of device
 </h3>
 
-🤗 Accelerate provides an easy API to make your scripts run with mixed precision and on any kind of distributed setting (multi-GPUs, TPUs etc.) while still letting you write your own training loop. The same code can then run on your local machine for debugging or your training environment.
+🤗 Accelerate was created for PyTorch users who like to write the training loop of PyTorch models but are reluctant to write and maintain the boiler code needed to use multi-GPUs/TPU/fp16.
+
+🤗 Accelerate abstracts exactly and only the boiler code related to multi-GPUs/TPU/fp16 and let the rest of your code unchanged.
+
+Here is an example:
+
+<table>
+<tr>
+<th> Original training code (CPU or mono-GPU only)</th>
+<th> With Accelerate for CPU/GPU/multi-GPUs/TPUs/fp16 </th>
+</tr>
+<tr>
+<td>
+
+```python
+import torch
+import torch.nn.functional as F
+from datasets import load_dataset
+
+
+
+device = 'cpu'
+
+model = torch.nn.Transformer().to(device)
+optim = torch.optim.Adam(model.parameters())
+
+dataset = load_dataset('my_dataset')
+data = torch.utils.data.Dataloader(dataset)
+
+
+
+
+model.train()
+for epoch in range(10):
+    for source, targets in data:
+        source = source.to(device)
+        targets = targets.to(device)
+
+        optimizer.zero_grad()
+
+        output = model(source, targets)
+        loss = F.cross_entropy(output, targets)
+
+        loss.backward()
+
+        optimizer.step()
+```
+
+</td>
+<td>
+
+```python
+  import torch
+  import torch.nn.functional as F
+  from datasets import load_dataset
+
++ from accelerate import Accelerator
++ accelerator = Accelerator()