diff --git a/docs/source/big_modeling.mdx b/docs/source/big_modeling.mdx
index a5cb3d4ed..051d1bda7 100644
--- a/docs/source/big_modeling.mdx
+++ b/docs/source/big_modeling.mdx
@@ -203,6 +203,65 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM for the model weights:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+When a first allocation happens in PyTorch, it loads CUDA kernels which take about 1-2GB of memory depending on the GPU. Therefore you always have less usable memory than the actual size of the GPU. To see how much memory is actually used do `torch.ones(1).cuda()` and look at the memory usage.
+
+Therefore when you create memory maps with `max_memory` make sure to adjust the avaialble memory accordingly to avoid out-of-memory errors.
+
+</Tip>
+
+Additionally, if you do some additional operations with your outputs without placing them back on the CPU (for instance inside the `generate` method of Transformers) and if you placed your inputs on a GPU, that GPU will consume more memory than the others (Accelerate always place the output back to the device of the input). Therefore if you would like to optimize the maximum batch size and you have many GPUs, give the first GPU less memory. For example, with BLOOM-176B on 8x80 A100 setup the close to ideal map is:
+
+```python
+max_memory = {0: "30GIB", 1: "46GIB", 2: "46GIB", 3: "46GIB", 4: "46GIB", 5: "46GIB", 6: "46GIB", 7: "46GIB"}
+```
+as you can see we gave the remaining 7 GPUs ~50% more memory than GPU 0.
+
+If you opt to fully design the `device_map` yourself, it should be a dictionary with keys being module names of your model and values being a valid device identifier (for instance an integer for the GPUs) or `"cpu"` for CPU offload, `"disk"` for disk offload. The keys need to cover the whole model, you can then define your device map as you wish: for instance if your model has two blocks (let's say `block1` and `block2`) which each contain three linear layers (let's say `linear1`, `linear2` and `linear3`), a valid device map can be:
+
+```python
+device_map = {"block1": 0, "block2": 1}
+```
+
+another one that is valid could be:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 0, "block2.linear2": 1, "block2.linear3": 1}
+```
+
+On the other hand, this one is not valid as it does not cover every parameter of the model:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 1, "block2.linear2": 1}
+```
+
+<Tip>
+
+To be the most efficient, make sure your device map puts the parameters on the GPUs in a sequential manner (e.g. don't put one of the first weights on GPU 0, then weights on GPU 1 and the last weight back to GPU 0) to avoid making many transfers of data between the GPUs.
+
+</Tip>
+
 ## Limits and further development
 
 We are aware of the current limitations in the API:


Review comments: body: I think including the ticks here may have broken the right navbar in the docs?
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a `device_map`
body: Arg!
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a `device_map`
body: I think here you may want to say "want to use more than ... for model weights". The addition is to be specific that it's just "model weights". I don't think accelerate controls activation and temp memory, does it?

So should it say - don't forget to leave memory for activations?

We really need to try to come up with the math that converts batch size into how much memory it'd need.

One way is to do it empirically by simply measuring with a given bs and reporting back to user a breakdown - this is how much was used for weights and how much for the activations/temps. But may not succeed on OOM.
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
body: ```suggestion
When a first allocation happens in PyTorch, it loads CUDA kernels which take about 1-2GB of memory depending on the GPU. Therefore you always have less usable memory than the actual size of the GPU. To see how much memory is actually used do `torch.ones(1).cuda()` and look at the memory usage.

Therefore when you create memory maps with `max_memory` make sure to adjust the avaialble memory accordingly to avoid out-of-memory errors.
```

if you're open to a more detailed explanation of what's going on and how to measure it.
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+PyTorch takes some space on a GPU, which can be up to 2GB. You should account for this when passing along `max_memory` to avoid out-of-memory errors.
body: Should it recommend that one puts the params in a sequential order and not manually organize them so that they fit better? e.g. putting a huge 176 embedding somewhere with a few tiny weights would balance things nicer, but it'd be detrimental to performance.
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+PyTorch takes some space on a GPU, which can be up to 2GB. You should account for this when passing along `max_memory` to avoid out-of-memory errors.
+
+</Tip>
+
+If you opt to fully design the `device_map` yourself, it should be a dictionary with keys being module names of your model and values being a valid device identifier (for instance an integer for the GPUs) or `"cpu"` for CPU offload, `"disk"` for disk offload. The keys need to cover the whole model, you can then define your device map as you wish: for instance if your model has two blocks (let's say `block1` and `block2`) which each contain three linear layers (let's say `linear1`, `linear2` and `linear3`), a valid device map can be:
+
+```python
+device_map = {"block1": 0, "block2": 1}
+```
+
+another one that is valid could be:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 0, "block2.linear2": 1, "block2.linear3": 1}
+```
+
+On the other hand, this one is not valid as it does not cover every parameter of the model:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 1, "block2.linear2": 1}
+```
+
body: ````suggestion

Additionally, it's important to know that the first GPU consumes more memory than the rest of the GPUs for managing activations, therefore if you would like to optimize the maximum batch size and you have many GPUs. Give the first GPU less memory. For example, with BLOOM-176B on 8x80 A100 setup the close to ideal map is:
```
    return {0: '30GIB', 1: '46GIB', 2: '46GIB', 3: '46GIB', 4: '46GIB', 5: '46GIB', 6: '46GIB', 7: '46GIB'}
```
as you can see we gave the remaining 7 GPUs ~50% more memory than GPU 0.
````
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+PyTorch takes some space on a GPU, which can be up to 2GB. You should account for this when passing along `max_memory` to avoid out-of-memory errors.
+
+</Tip>
+
body: This part is only for `generate`. Will rephrase a bit after accepting your suggestion.
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+PyTorch takes some space on a GPU, which can be up to 2GB. You should account for this when passing along `max_memory` to avoid out-of-memory errors.
+
+</Tip>
+
body: Good point!
diff_hunk: @@ -203,6 +203,50 @@ This only supports inference of your model, not training. Most of the computatio
 
 </Tip>
 
+## Designing a device map
+
+You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+
+<Tip>
+
+You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
+
+</Tip>
+
+First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
+
+Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM:
+
+```python
+from accelerate import infer_auto_device_map
+
+device_map = infer_auto_device_map(my_model, max_memory={0: "10GiB", 1: "10GiB", "cpu": "30GiB"})
+```
+
+<Tip warning={true}>
+
+PyTorch takes some space on a GPU, which can be up to 2GB. You should account for this when passing along `max_memory` to avoid out-of-memory errors.
+
+</Tip>
+
+If you opt to fully design the `device_map` yourself, it should be a dictionary with keys being module names of your model and values being a valid device identifier (for instance an integer for the GPUs) or `"cpu"` for CPU offload, `"disk"` for disk offload. The keys need to cover the whole model, you can then define your device map as you wish: for instance if your model has two blocks (let's say `block1` and `block2`) which each contain three linear layers (let's say `linear1`, `linear2` and `linear3`), a valid device map can be:
+
+```python
+device_map = {"block1": 0, "block2": 1}
+```
+
+another one that is valid could be:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 0, "block2.linear2": 1, "block2.linear3": 1}
+```
+
+On the other hand, this one is not valid as it does not cover every parameter of the model:
+
+```python
+device_map = {"block1": 0, "block2.linear1": 1, "block2.linear2": 1}
+```
+

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Should have addressed all your comments @stas00, let me know if all looks right to you and I'll merge :-)