Repository: Lightning-AI/pytorch-lightning
Pull request: Custom Plugin is_distributed
## What does this PR do?
Adds support for custom plugins to specify if they are distributed.

<!--
Please include a summary of the change and which issue is fixed.
 Please also include relevant motivation and context.
 List any dependencies that are required for this change.

If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

The following links the related issue to the PR (https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
-->
Fixes #6535 

## Before submitting
- [x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together?
- [ ] Did you make sure to update the documentation with your changes? (if necessary)
- [ ] Did you write any new necessary tests? (not for typos and docs)
- [x] Did you verify new and existing tests pass locally with your changes?
- [ ] Did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- For CHANGELOG separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

 - [x] Is this pull request ready for review? (if not, please submit in draft mode)
 - [ ] Check that all items from **Before submitting** are resolved
 - [x] Make sure the title is self-explanatory and the description concisely explains the PR
 - [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
Custom Distributed Plugin Auto Add Distributed Sampler
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
I have a custom distributed plugin, but it currently does not work PTL's automatic distributed sampler.

Plugin looks like this:
```
class MyPlugin(ParallelPlugin):
    @property
    def distributed_sampler_kwargs(self):
        ...
```
But in `data_loading.py`, when deciding whether to add a distributed data loader, PTL looks at `accelerator_connector.is_distributed`:
```
need_dist_sampler = self.accelerator_connector.is_distributed and not isinstance(
            dataloader.sampler, DistributedSampler
        )
```

And `self.accelerator_connector.is_distributed` only returns `True` if the built-in plugins are used, not any custom plugin:
```
@property
def is_distributed(self) -> bool:
    is_distributed = self.use_ddp or self.use_ddp2 or self.use_horovod
    if self.on_tpu:
        is_distributed |= self.training_type_plugin.is_distributed
    return is_distributed
```

Therefore, with a custom plugin, the distributed sampler is not set.

How can a custom plugin set itself to be distributed, so this property, and any other properties related to distributed training will automatically be set to the correct value?

cc @SeanNaren @justusschock @awaelchli

Hints from issue discussion:

Commit message: dont return for tpu

Patch to review:
diff --git a/pytorch_lightning/trainer/connectors/accelerator_connector.py b/pytorch_lightning/trainer/connectors/accelerator_connector.py
index 99d716f6b5a8c..f3b92144bdc64 100644
--- a/pytorch_lightning/trainer/connectors/accelerator_connector.py
+++ b/pytorch_lightning/trainer/connectors/accelerator_connector.py
@@ -273,6 +273,11 @@ def use_deepspeed(self) -> bool:
 
     @property
     def is_distributed(self) -> bool:
+        # Used for custom plugins.
+        # Custom plugins should implement is_distributed property.
+        if hasattr(self.training_type_plugin, 'is_distributed') and not \
+                self.on_tpu:
+            return self.training_type_plugin.is_distributed
         is_distributed = self.use_ddp or self.use_ddp2 or self.use_horovod
         if self.on_tpu:
             is_distributed |= self.training_type_plugin.is_distributed


Code review comments:
diff_hunk: @@ -273,6 +273,11 @@ def use_deepspeed(self) -> bool:
 
     @property
     def is_distributed(self) -> bool:
+        # Used for custom plugins.
+        # Custom plugins should implement is_distributed property.
+        if hasattr(self.training_type_plugin, 'is_distributed') and not \
+                self.on_tpu:
path: pytorch_lightning/trainer/connectors/accelerator_connector.py
text: @user1:
```suggestion
        if hasattr(self.training_type_plugin, 'is_distributed') and not self.on_tpu:
```
we have 120 line not 80