Repository: Lightning-AI/pytorch-lightning
Pull request: Update `ClusterEnvironment` docs
## What does this PR do?

Updates to cluster docs:
- Merged docs for general purpose cluster and SLURM cluster into one doc
- Added docs for Cluster Environment
- Updated Grid docs

Fixes #7073

## Before submitting
- [ ] Was this discussed/approved via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together?
- [x] Did you make sure to update the documentation with your changes? (if necessary)
- [ ] Did you write any new necessary tests? (not for typos and docs)
- [ ] Did you verify new and existing tests pass locally with your changes?
- [ ] Did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- For CHANGELOG separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

 - [x] Is this pull request ready for review? (if not, please submit in draft mode)
 - [x] Check that all items from **Before submitting** are resolved
 - [x] Make sure the title is self-explanatory and the description concisely explains the PR
 - [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
DDP for multi-node jobs in non-slurm environments
## 🐛 Bug

When running DDP with num_nodes > 1 without a SLURM environment, the global rank is not computed correctly: global rank ends up being always equal to local rank, disregarding number of nodes. My hunch for why this is happening is that num_processes is not computed correctly.

My current solution is to create a custom plugin and explicitly pass num_nodes, please see below. I would need a bit of guidance to see how we can fix the current pytorch-lightning code in a less hacky way.

The following code works using the latest dev version (1.3.0rc1)

```
...

trainer = pl.Trainer(
  gpus=hparams.gpus,
  max_epochs=hparams.max_epochs,
  num_nodes=hparams.num_nodes,
  distributed_backend='ddp' if hparams.gpus > 1 else None,
  sync_batchnorm=hparams.gpus > 1,
  precision=hparams.precision,
  fast_dev_run=hparams.fast_dev_run,
  plugins=CustomPlugin(num_nodes=hparams.num_nodes),
  logger=logger)

...
```

```
class CustomEnvironment(LightningEnvironment):
    def __init__(self, num_nodes):
        super().__init__()
        self._creates_children = num_nodes > 1

    def creates_children(self):
        return self._creates_children

    def global_rank(self):
        rank = None
        if 'RANK' in os.environ:
            rank = int(os.environ.get("RANK"))
        return rank

    def master_address(self):
        return os.environ.get("MASTER_ADDR", "127.0.0.1")

    def master_port(self):
        return int(os.environ.get("MASTER_PORT", super().master_port()))

    def world_size(self):
        world_size = None
        if 'WORLD_SIZE' in os.environ:
           world_size = int(os.environ.get('WORLD_SIZE'))
        return world_size

    def node_rank(self):
        return int(os.environ.get("NODE_RANK", super().node_rank()))

    def local_rank(self) -> int:
        return int(os.environ.get("LOCAL_RANK", super().local_rank()))


class CustomPlugin(DDPPlugin):
    def __init__(self, num_nodes, **kwargs: Union[Any, Dict[str, Any]]) -> None:
        self.num_nodes = num_nodes
        super().__init__(
            cluster_environment=CustomEnvironment(num_nodes),
            find_unused_parameters=False,
            **kwargs,
        )

    @property
    def distributed_sampler_kwargs(self):
        distributed_sampler_kwargs = dict(num_replicas=self.world_size, rank=self.global_rank)
        return distributed_sampler_kwargs

    def set_world_ranks(self):
        self.local_rank = self.cluster_environment.local_rank()
        self.node_rank = self.cluster_environment.node_rank()
        self.global_rank = self.cluster_environment.global_rank()
        self.world_size = self.cluster_environment.world_size()
        
        # local run, this falls back to the current logic which is broken for
        # ddp and num_nodes > 1, I suspect the problem is in self.num_processes
        if self.global_rank is None:
            if self.num_nodes > 1:
                raise ValueError("Expected num_nodes > 1.")
            self.global_rank = self.num_processes * self.node_rank + self.local_rank
        if self.world_size is None:
            if self.num_nodes > 1:
                raise ValueError("Expected num_nodes > 1.")
            self.world_size = self.num_nodes * self.num_processes
```

Hints from issue discussion:
How did you set the `NODE_RANK` environment variable when launching the processes? 
With the following debug script:

```python
import os
import torch
from torch.utils.data import Dataset
from pytorch_lightning import LightningModule, Trainer


class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):

    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        print("step", self.trainer.global_rank, self.trainer.local_rank, self.trainer.node_rank, self.trainer.world_size)
        loss = self(batch).sum()
        self.log("train_loss", loss)
        return {"loss": loss}

    def validation_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("valid_loss", loss)
        return {"x": loss}

    def test_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("test_loss", loss)
        return {"y": loss}

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)


def run():
    train_data = torch.utils.data.DataLoader(RandomDataset(32, 64), batch_size=2, num_workers=0)
    val_data = torch.utils.data.DataLoader(RandomDataset(32, 64), batch_size=2, num_workers=0)

    model = BoringModel()
    trainer = Trainer(
        gpus=2,
        num_nodes=2,
        accelerator="ddp",
        default_root_dir=os.getcwd(),
        limit_train_batches=1,
        limit_val_batches=1,
        num_sanity_val_steps=0,
        max_epochs=1,
        weights_summary=None,
    )
    print("init", trainer.global_rank, trainer.local_rank, trainer.node_rank, trainer.world_size)
    trainer.fit(model, train_dataloader=train_data, val_dataloaders=val_data)


if __name__ == '__main__':
    run()
```

If we run with e.g.` NODE_RANK=1 python debug.py`
The stdout prints (global_rank, local_rank, node_rank, world_size):

```
init 2 0 1 4
init 3 1 1 4
```

for the two processes on that node, which looks correct to me.
EDIT: the code here works on 1.3.0rc2
Apologies, I just noticed you said **1.3.0.rc1** and I thought you are running the latest.  But actually, the latest release is rc2 which has some cluster related things fixed. Mind try the update?
You may have to run master branch because of #7061 which only very recently got fixed.
It's a bit weird, I get:

2021-04-17 18:01:18 [1,3]<stderr>:  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 113, in _call_children_scripts
2021-04-17 18:01:18 [1,3]<stderr>:    assert self.local_rank == 0

Exactly 6 times (out of 8 gpus, which makes sense), then the two local_rank = 0 continue with training. Is this how it is supposed to work by design?
With my example code above or with yours? Try mine first so we can be sure everything is setup correctly.
The local_rank 0 should launch all other processes on that machine and then continue training, but not until the all processes have initialized. 

On node 0 you need to call
`NODE_RANK=0 MASTER_ADDR=... MASTER_PORT=... python debug.py`
On node 1 you need to call
`NODE_RANK=1 MASTER_ADDR=... MASTER_PORT=... python debug.py`

With my setup.

I see the problem: currently the platform I am currently using launches 8 processes automatically (creates_children must be set to return True), that's why 6 of them fail on the assert.

Can we customize the creates_children return value of the environment without a custom one?
Update. If I set 1 process per node in the platform, it works great, thanks for the support and for the great tool.

If there's any interest in making `creates_children` customizable I can make a PR.
You are welcome. But what is this mysterious platform you are talking about? Does it have a name?

What did you have in mind? A setter? 
When we created the cluster environments, the intention was that they are a sort of look up table, a source of information about the type of environment. The idea is that this information is mostly static, because the cluster environment is not really expected to change dynamically. But I'm open to suggestion in #6303 
Yes :) AzureML.

I thought about an option to Trainer: Trainer(..., environment_creates_children=True, ...), which would entail that PL must assume the environment creates world_size processes (False by default).

Let me take a look
That would interfere with all cluster environments (currently we have slurm, torchelastic, and lightning). 
The best way currently would be to implement a `AzureMLEnvironment` inheriting from `ClusterEnvironment`.
That's the intended way to enable new clusters in Lightning.

Is there a straightforward way for the user to specify to use the `AzureMLEnvironment` ?

Commit message: update duplicated title

Patch to review:
diff --git a/docs/source/advanced/cluster.rst b/docs/source/advanced/cluster.rst
deleted file mode 100644
index 34538cb85b474..0000000000000
--- a/docs/source/advanced/cluster.rst
+++ /dev/null
@@ -1,57 +0,0 @@
-
-.. _non-slurm:
-
-Computing cluster
-=================
-
-With Lightning it is easy to run your training script on a computing cluster without almost any modifications to the script.
-This guide shows how to run a training job on a general purpose cluster.
-
-Also, check :doc:`../extensions/accelerators` as a new and more general approach to a cluster setup.
-
---------
-
-
-Cluster setup
--------------
-
-To setup a multi-node computing cluster you need:
-
-1) Multiple computers with PyTorch Lightning installed
-2) A network connectivity between them with firewall rules that allow traffic flow on a specified *MASTER_PORT*.
-3) Defined environment variables on each node required for the PyTorch Lightning multi-node distributed training
-
-PyTorch Lightning follows the design of `PyTorch distributed communication package <https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization>`_. and requires the following environment variables to be defined on each node:
-
-- *MASTER_PORT* - required; has to be a free port on machine with NODE_RANK 0
-- *MASTER_ADDR* - required (except for NODE_RANK 0); address of NODE_RANK 0 node
-- *WORLD_SIZE* - required; how many nodes are in the cluster
-- *NODE_RANK* - required; id of the node in the cluster
-
-
-Training script design
-----------------------
-
-To train a model using multiple nodes, do the following:
-
-1.  Design your :ref:`lightning_module` (no need to add anything specific here).
-
-2.  Enable DDP in the trainer
-
-    .. code-block:: python
-
-       # train on 32 GPUs across 4 nodes
-       trainer = Trainer(gpus=8, num_nodes=4, accelerator='ddp')
-
-
-Submit a job to the cluster
----------------------------
-
-To submit a training job to the cluster you need to run the same training script on each node of the cluster.
-This means that you need to:
-
-1. Copy all third-party libraries to each node (usually means - distribute requirements.txt file and install it).
-
-2. Copy all your import dependencies and the script itself to each node.
-
-3. Run the script on each node.
diff --git a/docs/source/clouds/cloud_training.rst b/docs/source/clouds/cloud_training.rst
index 127bee6478dfd..ad4b96d83118f 100644
--- a/docs/source/clouds/cloud_training.rst
+++ b/docs/source/clouds/cloud_training.rst
@@ -1,8 +1,10 @@
+.. _grid:
+
 ################
 AWS/GCP training
 ################
 Lightning has a native solution for training on AWS/GCP at scale (Lightning-Grid).
-Grid is in private early-access now but you can request access at `grid.ai <https://www.grid.ai/>`_.
+Go to `grid.ai <https://www.grid.ai/>`_ to create an account.
 
 We've designed Grid to work for Lightning users without needing to make ANY changes to their code.
 
diff --git a/docs/source/clouds/slurm.rst b/docs/source/clouds/cluster.rst
similarity index 56%
rename from docs/source/clouds/slurm.rst
rename to docs/source/clouds/cluster.rst
index d482dc77ab456..0199a547c6c9b 100644
--- a/docs/source/clouds/slurm.rst
+++ b/docs/source/clouds/cluster.rst
@@ -2,22 +2,89 @@
 
     from pytorch_lightning.trainer.trainer import Trainer
 
+*****************
+Computing cluster
+*****************
+
+With Lightning it is easy to run your training script on a computing cluster without almost any modifications to the script.
+In this guide, we cover
+
+1.  General purpose cluster (not managed)
+
+2.  SLURM cluster
+
+3.  Custom cluster environment
+
+4.  General tips for multi-node training
+
+--------
+
+.. _non-slurm:
+
+1. General purpose cluster
+==========================
+
+This guide shows how to run a training job on a general purpose cluster. We recommend beginners to try this method
+first because it requires the least amount of configuration and changes to the code.
+To setup a multi-node computing cluster you need:
+
+1) Multiple computers with PyTorch Lightning installed
+2) A network connectivity between them with firewall rules that allow traffic flow on a specified *MASTER_PORT*.
+3) Defined environment variables on each node required for the PyTorch Lightning multi-node distributed training
+
+PyTorch Lightning follows the design of `PyTorch distributed communication package <https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization>`_. and requires the following environment variables to be defined on each node:
+
+- *MASTER_PORT* - required; has to be a free port on machine with NODE_RANK 0
+- *MASTER_ADDR* - required (except for NODE_RANK 0); address of NODE_RANK 0 node
+- *WORLD_SIZE* - required; how many nodes are in the cluster
+- *NODE_RANK* - required; id of the node in the cluster
+
+
+Training script setup
+---------------------
+
+To train a model using multiple nodes, do the following:
+
+1.  Design your :ref:`lightning_module` (no need to add anything specific here).
+
+2.  Enable DDP in the trainer
+
+    .. code-block:: python
+
+       # train on 32 GPUs across 4 nodes
+       trainer = Trainer(gpus=8, num_nodes=4, accelerator='ddp')
+
+
+Submit a job to the cluster
+---------------------------
+
+To submit a training job to the cluster you need to run the same training script on each node of the cluster.
+This means that you need to:
+
+1. Copy all third-party libraries to each node (usually means - distribute requirements.txt file and install it).
+2. Copy all your import dependencies and the script itself to each node.
+3. Run the script on each node.
+
+
+--------
+
+
 .. _slurm:
 
-Computing cluster (SLURM)
-=========================
+2. SLURM managed cluster
+========================
 
-Lightning automates the details behind training on a SLURM-powered cluster.
+Lightning automates the details behind training on a SLURM-powered cluster. In contrast to the general purpose
+cluster above, the user does not start the jobs manually on each node and instead submits it to SLURM which
+schedules the resources and time for which the job is allowed to run.
 
-.. _multi-node:
 
-----------
+Training script design
+----------------------
 
-Multi-node training
--------------------
 To train a model using multiple nodes, do the following:
 
-1.  Design your :doc:`lightning module <../common/lightning_module>`.
+1.  Design your :ref:`lightning_module` (no need to add anything specific here).
 
 2.  Enable DDP in the trainer
 
@@ -94,31 +161,6 @@ To train a model using multiple nodes, do the following:
 
         sbatch submit.sh
 
-.. note::
-    When running in DDP mode, any errors in your code will show up as an NCCL issue.
-    Set the `NCCL_DEBUG=INFO` flag to see the ACTUAL error.
-
-
-Normally now you would need to add a
-:class:`~torch.utils.data.distributed.DistributedSampler` to your dataset, however
-Lightning automates this for you. But if you still need to set a sampler set the Trainer flag
-:paramref:`~pytorch_lightning.Trainer.replace_sampler_ddp` to ``False``.
-
-Here's an example of how to add your own sampler (again, not needed with Lightning).
-
-.. testcode::
-
-    # in your LightningModule
-    def train_dataloader(self):
-        dataset = MyDataset()
-        dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
-        dataloader = Dataloader(dataset, sampler=dist_sampler)
-        return dataloader
-
-    # in your training script
-    trainer = Trainer(replace_sampler_ddp=False)
-
-----------
 
 Wall time auto-resubmit
 -----------------------
@@ -136,7 +178,6 @@ To get this behavior make sure to add the correct signal to your SLURM script
     # 90 seconds before training ends
     SBATCH --signal=SIGUSR1@90
 
-----------
 
 Building SLURM scripts
 ----------------------
@@ -198,11 +239,94 @@ See also the multi-node examples
     )
 
 
-The other option is that you generate scripts on your own via a bash command or use another library.
+The other option is that you generate scripts on your own via a bash command or use our
+:doc:`native solution <../clouds/cloud_training>`.
 
 ----------
 
+.. _custom-cluster:
+
+3. Custom cluster
+=================
+
+Lightning provides an interface for providing your own definition of a cluster environment. It mainly consists of
+parsing the right environment variables to access information such as world size, global and local rank (process id),
+and node rank (node id). Here is an example of a custom
+:class:`~pytorch_lightning.plugins.environments.cluster_environment.ClusterEnvironment`:
+
+.. testcode::
+
+    import os
+    from pytorch_lightning.plugins.environments import ClusterEnvironment
+
+    class MyClusterEnvironment(ClusterEnvironment):
+
+        def creates_children(self) -> bool:
+            # return True if the cluster is managed (you don't launch processes yourself)
+            return True
+
+        def world_size(self) -> int:
+            return int(os.environ["WORLD_SIZE"])
+
+        def global_rank(self) -> int:
+            return int(os.environ["RANK"])
+
+        def local_rank(self) -> int:
+            return int(os.environ["LOCAL_RANK"])
+
+        def node_rank(self) -> int:
+            return int(os.environ["NODE_RANK"])
+
+        def master_address(self) -> str:
+            return os.environ["MASTER_ADDRESS"]
+
+        def master_port(self) -> int:
+            return int(os.environ["MASTER_PORT"])
+
+
+    trainer = Trainer(plugins=[MyClusterEnvironment()])
+
+
+----------
+
+4. General tips for multi-node training
+=======================================
+
+Debugging flags
+---------------
+
+When running in DDP mode, some errors in your code can show up as an NCCL issue.
+Set the ``NCCL_DEBUG=INFO`` environment variable to see the ACTUAL error.
+
+.. code-block:: bash
+
+    python NCCL_DEBUG=INFO train.py ...
+
+
+Distributed sampler
+-------------------
+
+Normally now you would need to add a
+:class:`~torch.utils.data.distributed.DistributedSampler` to your dataset, however
+Lightning automates this for you. But if you still need to set a sampler set the Trainer flag
+:paramref:`~pytorch_lightning.Trainer.replace_sampler_ddp` to ``False``.
+
+Here's an example of how to add your own sampler (again, not needed with Lightning).
+
+.. testcode::
+
+    # in your LightningModule
+    def train_dataloader(self):
+        dataset = MyDataset()
+        dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+        dataloader = Dataloader(dataset, sampler=dist_sampler)
+        return dataloader
+
+    # in your training script
+    trainer = Trainer(replace_sampler_ddp=False)
+
+
 Self-balancing architecture (COMING SOON)
------------------------------------------
+=========================================
 
 Here Lightning distributes parts of your module across available GPUs to optimize for speed and memory.
diff --git a/docs/source/extensions/plugins.rst b/docs/source/extensions/plugins.rst
index 8b712bdce0244..76f1731ff592d 100644
--- a/docs/source/extensions/plugins.rst
+++ b/docs/source/extensions/plugins.rst
@@ -43,6 +43,10 @@ PrecisionPlugin
 - Gradient clipping
 
 
+Futhermore, for multi-node training Lightning provides cluster environment plugins that allow the advanced user
+to configure Lighting to integrate with a :ref:`custom-cluster`.
+
+
 .. image:: ../_static/images/accelerator/overview.svg
 
 
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 53b93a649d215..d299fb560ff1a 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -93,8 +93,8 @@ PyTorch Lightning Documentation
    :caption: Common Use Cases
 
    clouds/cloud_training
+   clouds/cluster
    advanced/amp
-   clouds/slurm
    common/child_modules
    common/debugging
    common/loggers
@@ -114,7 +114,6 @@ PyTorch Lightning Documentation
    advanced/pruning_quantization
    advanced/transfer_learning
    advanced/tpu
-   advanced/cluster
    common/test_set
    common/production_inference
 
diff --git a/docs/source/starter/new-project.rst b/docs/source/starter/new-project.rst
index efdfa6fb23012..ff2c91580203d 100644
--- a/docs/source/starter/new-project.rst
+++ b/docs/source/starter/new-project.rst
@@ -751,7 +751,7 @@ Once you define and train your first Lightning model, you might want to try othe
 - :ref:`Automatically scale your batch size <advanced/training_tricks:Auto scaling of batch size>`
 - :doc:`Automatically find a good learning rate <../advanced/lr_finder>`
 - :ref:`Load checkpoints directly from S3 <common/weights_loading:Checkpoint Loading>`
-- :doc:`Scale to massive compute clusters <../clouds/slurm>`
+- :doc:`Scale to massive compute clusters <../clouds/cluster>`
 - :doc:`Use multiple dataloaders per train/val/test loop <../advanced/multiple_loaders>`
 - :ref:`Use multiple optimizers to do reinforcement learning or even GANs <common/optimizers:Use multiple optimizers (like GANs)>`
 


Code review comments:
diff_hunk: @@ -198,11 +239,94 @@ See also the multi-node examples
     )
 
 
-The other option is that you generate scripts on your own via a bash command or use another library.
+The other option is that you generate scripts on your own via a bash command or use our
+:doc:`native solution <../clouds/cloud_training>`.
 
 ----------
 
+.. _custom-cluster:
+
+3. Custom cluster
+=================
+
+Lightning provides an interface for providing your own definition of a cluster environment. It mainly consists of
+parsing the right environment variables to access information such as world size, global and local rank (process id),
+and node rank (node id). Here is an example of a custom
+:class:`~pytorch_lightning.plugins.environments.cluster_environment.ClusterEnvironment`:
+
+.. testcode::
+
+    import os
+    from pytorch_lightning.plugins.environments import ClusterEnvironment
+
+    class MyClusterEnvironment(ClusterEnvironment):
+
+        def creates_children(self) -> bool:
+            # return True if the cluster is managed (you don't launch processes yourself)
+            return True
+
+        def world_size(self) -> int:
+            return int(os.environ["WORLD_SIZE"])
+
+        def global_rank(self) -> int:
+            return int(os.environ["RANK"])
+
+        def local_rank(self) -> int:
+            return int(os.environ["LOCAL_RANK"])
+
+        def node_rank(self) -> int:
+            return int(os.environ["NODE_RANK"])
+
+        def master_address(self) -> str:
+            return os.environ["MASTER_ADDRESS"]
+
+        def master_port(self) -> int:
+            return int(os.environ["MASTER_PORT"])
+
+
+    trainer = Trainer(plugins=[MyClusterEnvironment()])
+
+
+----------
+
+4. General tips for multi-node training
+=======================================
+
+Debugging flags
+---------------
+
+When running in DDP mode, some errors in your code can show up as an NCCL issue.
+Set the ``NCCL_DEBUG=INFO`` environment variable to see the ACTUAL error.
+
+.. code-block:: bash
+
+    python NCCL_DEBUG=INFO train.py ...
+
+
+Distributed sampler
+-------------------
+
+Normally now you would need to add a
+:class:`~torch.utils.data.distributed.DistributedSampler` to your dataset, however
+Lightning automates this for you. But if you still need to set a sampler set the Trainer flag
+:paramref:`~pytorch_lightning.Trainer.replace_sampler_ddp` to ``False``.
+
+Here's an example of how to add your own sampler (again, not needed with Lightning).
+
+.. testcode::
+
+    # in your LightningModule
+    def train_dataloader(self):
+        dataset = MyDataset()
+        dist_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
+        dataloader = Dataloader(dataset, sampler=dist_sampler)
+        return dataloader
+
+    # in your training script
+    trainer = Trainer(replace_sampler_ddp=False)
+
+
 Self-balancing architecture (COMING SOON)
path: docs/source/clouds/cluster.rst
text: @user2:
@user1 should we remove this?

If not, this should be in training tricks

@user1:
I'd remove this as we've moved past this point with fully sharded

diff_hunk: @@ -1,8 +1,10 @@
+.. _grid:
+
 ################
 AWS/GCP training
 ################
 Lightning has a native solution for training on AWS/GCP at scale (Lightning-Grid).
path: docs/source/clouds/cloud_training.rst
text: @user1:
This seems outdated
```suggestion
Lightning has a native solution for training on AWS/GCP at scale.
```