Repository: Lightning-AI/pytorch-lightning
Pull request: Raise exceptions when torch distributed is not available
## What does this PR do?
Avoids init_dist_connection failing silently

If torch.distributed is not available, the process_group cannot be created. Later, the runtime will fail when wrapping model or after training steps (when call distributed_c10d). So we should fail faster at `init_dist_connection` step
<!--
Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context.
List any dependencies that are required for this change.

If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

The following links the related issue to the PR (https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
-->

partially fixes #10348

### Does your PR introduce any breaking changes? If yes, please list them.

<!-- FILL IN or None -->

## Before submitting

- [x] Was this **discussed/approved** via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- [x] Did you make sure to **update the documentation** with your changes? (if necessary)
- [ ] Did you write any **new necessary tests**? (not for typos and docs)
- [x] Did you verify new and **existing tests pass** locally with your changes?
- [x] Did you list all the **breaking changes** introduced by this pull request?
- [x] Did you **update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)**? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

- [x] Is this pull request ready for review? (if not, please submit in draft mode)
- [x] Check that all items from **Before submitting** are resolved
- [x] Make sure the title is self-explanatory and the description concisely explains the PR
- [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?

Make sure you had fun coding 🙃


Problem statement:
Module 'torch.distributed' has no attribute 'ProcessGroup' when importing PyTorch Lightning
## 🐛 Bug

When importing PyTorch Lightning, it throws an `AttributeError: module 'torch.distributed' has no attribute 'ProcessGroup'`. I guess it comes from the fact that I am on macOS (M1) and PyTorch does not provide `torch.distributed` with its pre-built package. Indeed, `torch.distributed.is_available()` is `False`. 

### To Reproduce

```
import pytorch_lightning
```

### Environment

- PyTorch Lightning Version: 1.5.0
- PyTorch Version: 1.10
- Python version: 3.9
- OS: macOS
- How you installed PyTorch: `conda`
- Any other relevant information:

Hints from issue discussion:
Hello @Riccorl 
Thanks for reporting. Can you please show us the full error so we can check in which module it occurs?

I'm not sure why you get `torch.distributed.is_available() = False` on MacOS, it should be True. It is for me.
> Hello @Riccorl Thanks for reporting. Can you please show us the full error so we can check in which module it occurs?

This is the stack trace

```
Traceback (most recent call last):
  File "/Users/ric/Documents/PhD/Projects/invero-xl/invero_xl/train.py", line 7, in <module>
    import pytorch_lightning as pl
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/__init__.py", line 20, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/callbacks/__init__.py", line 26, in <module>
    from pytorch_lightning.callbacks.pruning import ModelPruning
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/callbacks/pruning.py", line 31, in <module>
    from pytorch_lightning.core.lightning import LightningModule
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/core/__init__.py", line 16, in <module>
    from pytorch_lightning.core.lightning import LightningModule
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 39, in <module>
    from pytorch_lightning.trainer.connectors.logger_connector.fx_validator import _FxValidator
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/trainer/__init__.py", line 16, in <module>
    from pytorch_lightning.trainer.trainer import Trainer
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 30, in <module>
    from pytorch_lightning.accelerators import Accelerator, IPUAccelerator
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/accelerators/__init__.py", line 13, in <module>
    from pytorch_lightning.accelerators.accelerator import Accelerator  # noqa: F401
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 26, in <module>
    from pytorch_lightning.plugins.precision import ApexMixedPrecisionPlugin, NativeMixedPrecisionPlugin, PrecisionPlugin
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/plugins/__init__.py", line 8, in <module>
    from pytorch_lightning.plugins.plugins_registry import (  # noqa: F401
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/plugins/plugins_registry.py", line 20, in <module>
    from pytorch_lightning.plugins.training_type.training_type_plugin import TrainingTypePlugin
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/__init__.py", line 1, in <module>
    from pytorch_lightning.plugins.training_type.ddp import DDPPlugin  # noqa: F401
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 68, in <module>
    from torch.distributed.optim import DistributedOptimizer
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/torch/distributed/optim/__init__.py", line 37, in <module>
    from .post_localSGD_optimizer import PostLocalSGDOptimizer
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/torch/distributed/optim/post_localSGD_optimizer.py", line 2, in <module>
    import torch.distributed.algorithms.model_averaging.averagers as averagers
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/torch/distributed/algorithms/model_averaging/averagers.py", line 5, in <module>
    import torch.distributed.algorithms.model_averaging.utils as utils
  File "/Users/ric/mambaforge/envs/srl/lib/python3.9/site-packages/torch/distributed/algorithms/model_averaging/utils.py", line 10, in <module>
    params: Iterator[torch.nn.Parameter], process_group: dist.ProcessGroup
AttributeError: module 'torch.distributed' has no attribute 'ProcessGroup'
```


> I'm not sure why you get `torch.distributed.is_available() = False` on MacOS, it should be True. It is for me.

I installed PyTorch like this:

```
conda install pytorch -c pytorch
```

But I guess that the problem is the ARM build (I'm on an M1 cpu).
We can fix this easily as the error comes from a typing annotation, but we'll also have to add a M1 CI job when it becomes available.
Current init_dist_connection() will do nothing if torch.distributed.is_avalible = False. To wrap model with `DistributedDataParallel()`, something like `torch.distributed.init_process_group(
     backend='nccl', world_size=N, init_method='...'
      )`
Is required right?
Should we raise exceptions at set_distributed() in ddp if torch.distributed.is_avalible = False?
@four4fish Yes you are right. I tested it and we get this message: 

> RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

I agree, for users like @Riccorl we could inform them directly that DDP is not available when `torch.distributed.is_avalible = False`.

Btw, doesn't your PR directly solve the import problem of this issue? I couldn't find other places.
@Riccorl I had a similar issue on M1 macbook, it only happens with pytorch=1.10.
 **Downgrading torch to '1.9.1.post3' resolved the issue for me.** 
@awaelchli  I think after the import PR, import lightning didn't fail. But when trainer calls ddp setup_distributed(), which calls init_dist_connection() will check `torch.distributed.is_avalible` before create process group. Because `torch.distributed.is_avalible=False` no process_group was created, so the future ddp will fail. Where is this runtime error happens exactly? when wrap the model?

I was proposing: should we throw exception in init_dist_connect() If `torch.distributed.is_avalible=false` ?

I encountered this same issue. I'm building PyTorch Lightning 1.5.0 and PyTorch 1.10.0 from source using the Spack package manager on macOS 10.15.7. Unfortunately, PyTorch distributed doesn't seem to build for me on macOS: https://github.com/pytorch/pytorch/issues/68002

It sounds like requiring distributed support was an accident and will be removed in future releases. Let me know which PR solves this and I'll add a patch to the 1.5.0 release in Spack.

Commit message: Raise exceptions when torch distributed is not avalible

Patch to review:
diff --git a/pytorch_lightning/utilities/distributed.py b/pytorch_lightning/utilities/distributed.py
index ef0f0b5ef3e97..d968f2eb46d1d 100644
--- a/pytorch_lightning/utilities/distributed.py
+++ b/pytorch_lightning/utilities/distributed.py
@@ -381,7 +381,9 @@ def init_dist_connection(
     world_size = world_size if world_size is not None else cluster_environment.world_size()
     os.environ["MASTER_ADDR"] = cluster_environment.main_address
     os.environ["MASTER_PORT"] = str(cluster_environment.main_port)
-    if torch.distributed.is_available() and not torch.distributed.is_initialized():
+    if not torch.distributed.is_available():
+        raise RuntimeError("Torch distributed is not avalible, can not initialized distributed process group")
+    if not torch.distributed.is_initialized():
         log.info(f"initializing distributed: GLOBAL_RANK: {global_rank}, MEMBER: {global_rank + 1}/{world_size}")
         torch.distributed.init_process_group(
             torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs


Code review comments:
diff_hunk: @@ -381,7 +381,9 @@ def init_dist_connection(
     world_size = world_size if world_size is not None else cluster_environment.world_size()
     os.environ["MASTER_ADDR"] = cluster_environment.main_address
     os.environ["MASTER_PORT"] = str(cluster_environment.main_port)
-    if torch.distributed.is_available() and not torch.distributed.is_initialized():
+    if not torch.distributed.is_available():
+        raise RuntimeError("Torch distributed is not avalible, can not initialized distributed process group")
path: pytorch_lightning/utilities/distributed.py
text: @user1:
```suggestion
        raise RuntimeError("torch.distributed is not available. Cannot initialize distributed process group")
```

@user2:
Could you apply ananthsub's suggestion again? It seems like your force-pushing reverted the change here.