diff --git a/docs/source/big_modeling.mdx b/docs/source/big_modeling.mdx
index 051d1bda7..006c8bc4e 100644
--- a/docs/source/big_modeling.mdx
+++ b/docs/source/big_modeling.mdx
@@ -205,7 +205,7 @@ This only supports inference of your model, not training. Most of the computatio
 
 ## Designing a device map
 
-You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+You can let 🤗 Accelerate handle the device map computation by setting `device_map` to one of the supported options (`"auto"`, `"balanced"`, `"balanced_low_0"`, `"sequential"`) or create one yourself, if you want more control over where each layer should go.
 
 <Tip>
 
@@ -213,6 +213,19 @@ You can derive all sizes of the model (and thus compute a `device_map`) on a mod
 
 </Tip>
 
+All the options will produce the same result when you don't have enough GPU memory to accomodate the whole model (which is to fit everything that can on the GPU, then offload weights on the CPU or even on the disk if there is not enough RAM). 
+
+When you have more GPU memory available than the model size, here the difference between each option:
+- `"auto"` and `"balanced"` evenly split the model on all available GPUs, making it possible for you to use a batch size greater than 1.
+- `"balanced_low_0"` evenly splits the model on all GPUs except the first one, and only puts on GPU 0 what does not fit on the others. This option is great when you need to use GPU 0 for some processing of the outputs, like when using the `generate` function for Transformers models
+- `"sequential"` will fit what it can on GPU 0, then move on GPU 1 and so forth (so won't use the last GPUs if it doesn't need to).
+
+<Tip>
+
+The options `"auto"` and `"balanced"` produce the same results for now, but the behavior of `"auto"` might change in the future if we find a strategy that makes more sense, while `"balanced"` will stay stable.
+
+</Tip>
+
 First note that you can limit the memory used on each GPU by using the `max_memory` argument (available in [`infer_auto_device_map`] and in all functions using it). When setting `max_memory`, you should pass along a dictionary containing the GPU identifiers (for instance `0`, `1` etc.) and the `"cpu"` key for the maximum RAM you want used for CPU offload. The values can either be an integer (in bytes) or a string representing a number with its unit, such as `"10GiB"` or `"10GB"`.
 
 Here is an example where we don't want to use more than 10GiB on each of two GPUs and no more than 30GiB of CPU RAM for the model weights:
diff --git a/src/accelerate/big_modeling.py b/src/accelerate/big_modeling.py
index 907dfdac4..311111ad4 100644
--- a/src/accelerate/big_modeling.py
+++ b/src/accelerate/big_modeling.py
@@ -24,6 +24,7 @@
     OffloadedWeightsLoader,
     check_device_map,
     extract_submodules_state_dict,
+    get_balanced_memory,
     infer_auto_device_map,
     load_checkpoint_in_model,
     offload_state_dict,
@@ -278,7 +279,8 @@ def load_checkpoint_and_dispatch(
             A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer
             name, once a given module name is inside, every submodule of it will be sent to the same device.
 
-            To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`.
+            To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more
+            information about each option see [here](big_modeling#designing-a-device-map).
         max_memory (`Dict`, *optional*):
             A dictionary device identifier to maximum memory. Will default to the maximum memory available for each GPU
             and the available CPU RAM if unset.
@@ -302,7 +304,20 @@ def load_checkpoint_and_dispatch(
             called directly during the forward, for instance if a `dense` linear layer is registered, but at forward,
             `dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly.
     """
-    if device_map == "auto":
+    if isinstance(device_map, str) and device_map not in ["auto", "balanced", "balanced_low_0", "sequential"]:
+        raise ValueError(
+            "If passing a string for `device_map`, please choose 'auto', 'balanced', 'balanced_low_0' or "
+            "'sequential'."
+        )
+    if device_map != "sequential":
+        max_memory = get_balanced_memory(
+            model,
+            max_memory=max_memory,
+            no_split_module_classes=no_split_module_classes,
+            dtype=dtype,
+            low_zero=(device_map == "balanced_low_0"),
+        )
+    if isinstance(device_map, str):
         device_map = infer_auto_device_map(
             model, max_memory=max_memory, no_split_module_classes=no_split_module_classes, dtype=dtype
         )
diff --git a/src/accelerate/utils/__init__.py b/src/accelerate/utils/__init__.py
index 350efc899..f6b83d2f9 100644
--- a/src/accelerate/utils/__init__.py
+++ b/src/accelerate/utils/__init__.py
@@ -37,6 +37,7 @@
     convert_file_size_to_int,
     dtype_byte_size,
     find_tied_parameters,
+    get_balanced_memory,
     get_max_layer_size,
     get_max_memory,
     infer_auto_device_map,
diff --git a/src/accelerate/utils/modeling.py b/src/accelerate/utils/modeling.py
index 62c4f9761..e412ce0a0 100644
--- a/src/accelerate/utils/modeling.py
+++ b/src/accelerate/utils/modeling.py
@@ -326,6 +326,101 @@ def load_offloaded_weights(model, index, offload_folder):
         set_module_tensor_to_device(model, param_name, "cpu", value=torch.tensor(weight))
 
 
+def get_balanced_memory(
+    model: nn.Module,
+    max_memory: Optional[Dict[Union[int, str], Union[int, str]]] = None,
+    no_split_module_classes: Optional[List[str]] = None,
+    dtype: Optional[Union[str, torch.dtype]] = None,
+    low_zero: bool = False,
+):
+    """
+    Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU.
+
+    <Tip>
+
+    All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the
+    meta device (as it would if initialized within the `init_empty_weights` context manager).
+
+    </Tip>
+
+    Args:
+        model (`torch.nn.Module`): The model to analyze.
+        max_memory (`Dict`, *optional*):
+            A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset.
+        no_split_module_classes (`List[str]`, *optional*):
+            A list of layer class names that should never be split across device (for instance any layer that has a
+            residual connection).
+        dtype (`str` or `torch.dtype`, *optional*):
+            If provided, the weights will be converted to that type when loaded.
+        low_zero (`bool`, *optional*):
+            Minimizes the number of weights on GPU 0, which is convenient when it's used for other operations (like the
+            Transformers generate function).
+    """
+    # Get default / clean up max_memory
+    max_memory = get_max_memory(max_memory)
+
+    if not torch.cuda.is_available():
+        return max_memory
+
+    num_devices = len([d for d in max_memory if torch.device(d).type == "cuda"])
+    module_sizes = compute_module_sizes(model, dtype=dtype)
+    per_gpu = module_sizes[""] // (num_devices - 1 if low_zero else num_devices)
+
+    # We can't just set the memory to model_size // num_devices as it will end being too small: each GPU will get
+    # slightly less layers and some layers will end up offload at the end. So this function computes a buffer size to
+    # add which is the biggest of:
+    # - the size of no split block (if applicable)
+    # - the mean of the layer sizes
+    if no_split_module_classes is None:
+        no_split_module_classes = []
+    elif not isinstance(no_split_module_classes, (list, tuple)):
+        no_split_module_classes = [no_split_module_classes]
+
+    # Identify the size of the no_split_block modules
+    if len(no_split_module_classes) > 0:
+        no_split_children = {}
+        for name, size in module_sizes.items():
+            if name == "":
+                continue
+            submodule = model
+            for submodule_name in name.split("."):
+                submodule = getattr(submodule, submodule_name)
+            class_name = submodule.__class__.__name__
+            if class_name in no_split_module_classes and class_name not in no_split_children:
+                no_split_children[class_name] = size
+
+            if set(no_split_children.keys()) == set(no_split_module_classes):
+                break
+        buffer = max(no_split_children.values()) if len(no_split_children) > 0 else 0
+    else:
+        buffer = 0
+
+    # Compute mean of final modules. In the first dict of module sizes, leaves are the parameters
+    leaves = [n for n in module_sizes if len([p for p in module_sizes if p.startswith(n) and len(p) > len(n)]) == 0]
+    module_sizes = {n: v for n, v in module_sizes.items() if n not in leaves}
+    # Once removed, leaves are the final modules.
+    leaves = [n for n in module_sizes if len([p for p in module_sizes if p.startswith(n) and len(p) > len(n)]) == 0]
+    mean_leaves = int(sum([module_sizes[n] for n in leaves]) / len(leaves))
+    buffer = int(1.25 * max(buffer, mean_leaves))
+    if low_zero:
+        per_gpu += buffer
+        gpu_zero = 0
+    else:
+        gpu_zero = per_gpu
+        per_gpu += buffer
+
+    max_memory = get_max_memory(max_memory)
+    for i in range(num_devices):
+        # We still leave slightly more space on GPU 0 and only apply the buffer on the other devices.
+        max_memory[i] = min(gpu_zero if i == 0 else per_gpu, max_memory[i])
+
+    if low_zero:
+        min_zero = max(0, module_sizes[""] - sum([max_memory[i] for i in range(1, num_devices)]))
+        max_memory[0] = min(min_zero, max_memory[0])
+
+    return max_memory
+
+
 def infer_auto_device_map(
     model: nn.Module,
     max_memory: Optional[Dict[Union[int, str], Union[int, str]]] = None,
diff --git a/tests/test_modeling_utils.py b/tests/test_modeling_utils.py
index 15e164218..799170257 100644
--- a/tests/test_modeling_utils.py
+++ b/tests/test_modeling_utils.py
@@ -26,6 +26,7 @@
     clean_device_map,
     compute_module_sizes,
     find_tied_parameters,
+    get_balanced_memory,
     infer_auto_device_map,
     load_checkpoint_in_model,
     named_module_tensors,
@@ -358,3 +359,13 @@ def test_infer_auto_device_map(self):
         device_map = infer_auto_device_map(model, max_memory={0: 400, 1: 500})
         expected = {"0": 0, "2.linear2": 0, "1": 1, "2.linear1": 1, "2.batchnorm": 1}
         self.assertDictEqual(device_map, expected)
+
+    @require_cuda
+    def test_get_balanced_memory(self):
+        model = ModelForTest()
+        # model has size 236: linear1 64, batchnorm 72, linear2 100
+        max_memory = get_balanced_memory(model, max_memory={0: 200, 1: 200})
+        self.assertDictEqual({0: 118, 1: 200}, max_memory)
+
+        max_memory = get_balanced_memory(model, max_memory={0: 300, 1: 300})
+        self.assertDictEqual({0: 118, 1: 215}, max_memory)


Review comments: body: Docstring format nit
diff_hunk: @@ -326,6 +326,95 @@ def load_offloaded_weights(model, index, offload_folder):
         set_module_tensor_to_device(model, param_name, "cpu", value=torch.tensor(weight))
 
 
+def get_balanced_memory(
+    model: nn.Module,
+    max_memory: Optional[Dict[Union[int, str], Union[int, str]]] = None,
+    no_split_module_classes: Optional[List[str]] = None,
+    dtype: Optional[Union[str, torch.dtype]] = None,
+    low_zero: bool = False,
+):
+    """
+    Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU.
+
+    <Tip>
+
+    All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the
+    meta device (as it would if initialized within the `init_empty_weights` context manager).
+
+    </Tip>
+
+    Args:
+        model (`torch.nn.Module`): The model to analyze.
body: I don't see any nit ;-) 
diff_hunk: @@ -326,6 +326,95 @@ def load_offloaded_weights(model, index, offload_folder):
         set_module_tensor_to_device(model, param_name, "cpu", value=torch.tensor(weight))
 
 
+def get_balanced_memory(
+    model: nn.Module,
+    max_memory: Optional[Dict[Union[int, str], Union[int, str]]] = None,
+    no_split_module_classes: Optional[List[str]] = None,
+    dtype: Optional[Union[str, torch.dtype]] = None,
+    low_zero: bool = False,
+):
+    """
+    Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU.
+
+    <Tip>
+
+    All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the
+    meta device (as it would if initialized within the `init_empty_weights` context manager).
+
+    </Tip>
+
+    Args:
+        model (`torch.nn.Module`): The model to analyze.
body: If it resonates I would rename it to something more self-documenting? e.g. `minimize_gpu0_memory` or `minimize_first_gpu_memory` 
diff_hunk: @@ -326,6 +326,95 @@ def load_offloaded_weights(model, index, offload_folder):
         set_module_tensor_to_device(model, param_name, "cpu", value=torch.tensor(weight))
 
 
+def get_balanced_memory(
+    model: nn.Module,
+    max_memory: Optional[Dict[Union[int, str], Union[int, str]]] = None,
+    no_split_module_classes: Optional[List[str]] = None,
+    dtype: Optional[Union[str, torch.dtype]] = None,
+    low_zero: bool = False,
body: ```suggestion
All the options will produce the same result when you don't have enough GPU memory to accomodate the whole model (which is to fit everything that can on the GPU, then offload weights on the CPU or even on the disk if there is not enough RAM). 

When you have more GPU memory available than the model size, here the difference between each option:
```
Let's add a break here
diff_hunk: @@ -205,14 +205,25 @@ This only supports inference of your model, not training. Most of the computatio
 
 ## Designing a device map
 
-You can let 🤗 Accelerate handle the device map computation by setting `device_map="auto"` or create one yourself, if you want more control over where each layer should go.
+You can let 🤗 Accelerate handle the device map computation by setting `device_map` to one of the supported options (`"auto"`, `"balanced"`, `"balanced_low_0"`, `"sequential"`) or create one yourself, if you want more control over where each layer should go.
 
 <Tip>
 
 You can derive all sizes of the model (and thus compute a `device_map`) on a model that is on the meta device.
 
 </Tip>
 
+All the options will produce the same result when you don't have enough GPU memory to accomodate the whole model (which is to fit everything that can on the GPU, then offload weights on the CPU or even on the disk if there is not enough RAM). When you have more GPU memory available than the model size, here the difference between each option:

Discussion: body: _The documentation is not available anymore as the PR was closed or merged._
body: Yes I will update it once the other PR is merged :-) 
body: > This leads to equal allocation across all GPUs. Ideally it should be reworked to leave the gpu0 as close to empty as possible. So probably first trying to spread the weights across all but the first gpu, while leaving enough memory for activation calculation and temps. And only then assign any remaining weights to the first gpu.

That is exactly what the option `"balanced_low_0"` will do.