Repository: Lightning-AI/pytorch-lightning

Issue:
"Unsupported `ReduceOp` for distributed computing" warning when using Result without distributed
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

step_result.py imports pytorch_lightning.metrics.converters

converters.py raises the following warning if torch.distributed.ReduceOp cannot be imported:

> rank_zero_warn('Unsupported `ReduceOp` for distributed computing.')

I don't use and don't want to use distributed training, but this warning is printed to stdout non-stop at one warning per second rate.

### To Reproduce

Use Result object without a distributed package available.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->


### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

This warning should be printed once at most. 
If I don't use distributed, I don't need to see this warning at all.

### Environment

* CUDA:
        - GPU:
                - GeForce RTX 2080
        - available:         True
        - version:           10.2
* Packages:
        - numpy:             1.18.1
        - pyTorch_debug:     False
        - pyTorch_version:   1.6.0
        - pytorch-lightning: 0.9.0
        - tensorboard:       2.2.0
        - tqdm:              4.47.0
* System:
        - OS:                Windows
        - architecture:
                - 64bit
                - WindowsPE
        - processor:         Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
        - python:            3.7.1
        - version:           10.0.19041

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Yeah, that is PyTorch issue as most it does not support several distributed features on Windows...
May send a PR with some friendly message or update it docs?
I think I understand what you are saying. But, pytorch-lightning code is the one that prints the warning message. Since it keeps printing the same warning like it is in an infinite loop, upgrading from 0.8 to 0.9 made pytorch-lightning unusable for me, and for anyone using Windows I imagine. 

If you say pytorch-lightning 0.9+ won't support Windows, that's one thing. Otherwise, this is an issue.


> I think I understand what you are saying. But, pytorch-lightning code is the one that prints the warning message. 

yes, we warn you that you requested something which is not supported on your OS, pt does not give it to you either, the diff is the PT even does not tell you and is standard computing

> Since it keeps printing the same warning like it is in an infinite loop, upgrading from 0.8 to 0.9 made pytorch-lightning unusable for me, and for anyone using Windows I imagine.

so you just complain that you get this message several times? @awaelchli

> If you say pytorch-lightning 0.9+ won't support Windows, that's one thing. Otherwise, this is an issue.

this is a misunderstanding, it is not PL who does not support distributed on Win but PyTorch itself, PL is just wrapper above PT... so you shall create an issue there...
> this is a misunderstanding, it is not PL who does not support distributed on Win but PyTorch itself, PL is just wrapper above PT... so you shall create an issue there...

I think I failed to explain the problem. I use PT without any issues, because I don't use distributed anyway. I don't use distributed when I use PL either, but PL core imports distributed anyway, and prints that warning, and prints it once per every second (I don't know why).

Edit: and this only started to happen in 0.9 because the import line came with the Result object.

ok, that is fair that we shall tell you just once or even do not import it if you do not need it...
@ozen mind send a PR with patching this importing?
cc: @awaelchli @yukw777 
OK, couple of findings:

- As I understand `rank_zero_only` covers distributed processes, but not DataLoader worker processes. The worker processes are started at the beginning of the training and validation runs at every epoch, resulted in a lot of warnings, especially if you use a large number of workers.
- `from pytorch_lightning.metrics.converters import _sync_ddp_if_available` line in `core/step_result.py` can be moved inside the if clause where it is used if `sync_dist=True`. But this alone won't solve the problem because metric converters are imported elsewhere too.
- The ReduceOp import in `metrics/converters.py` that causes the warning is used for two things: type hints and in the function `_sync_ddp_if_available`. If we moved the import inside _sync_ddp_if_available the problem would be solved, but I don't know if it's OK to remove the type hints.
To keep it simple, you can also ignore the warnings in the command-line with

     >>> python -W "ignore::UserWarning::0" file_to_run.py
I have mentioned multi-process data loading as the cause of the spam of warnings. In fact, this is a problem only in Windows when `import pytorch_lightning` statement is outside of `if __name__ == '__main__':` block. The reason of this is [described here](https://pytorch.org/docs/stable/data.html#platform-specific-behaviors). Moving the import statements inside the if block solves the problem.

We may still consider changing the code a little bit. The origin of the warning is `pytorch_lightning.metrics.converters` module. 

```python
try:
    from torch.distributed import ReduceOp
except ImportError:
    class ReduceOp:
        SUM = None
    rank_zero_warn("Unsupported `ReduceOp` for distributed computing")
```

Note that ReduceOp.SUM is not used. `ReduceOp` class is imported only for the type hints (i.e. `reduce_op: Optional[ReduceOp] = None`). However on the same module, group parameters do not follow this practice for type hinting and use `Optional[Any]`.

An example:

```python
def tensor_metric(group: Optional[Any] = None, reduce_op: Optional[ReduceOp] = None) -> Callable:
```

If we change the hints into

```python
def tensor_metric(group: Optional[Any] = None, reduce_op: Optional[Any] = None) -> Callable:
```

we can remove the import ReduceOp statement together with the warning.

Or maybe we want type hints for group as well:

```python
def tensor_metric(group: Optional[group] = None, reduce_op: Optional[ReduceOp] = None) -> Callable:
```

and add a try-catch import for group too since it has the same problem with the ReduceOp class.

I can send a PR depending on the decision.

Resolving patch:
diff --git a/pytorch_lightning/metrics/converters.py b/pytorch_lightning/metrics/converters.py
--- a/pytorch_lightning/metrics/converters.py
+++ b/pytorch_lightning/metrics/converters.py
@@ -26,18 +26,14 @@
 import torch
 from torch.utils.data._utils.collate import np_str_obj_array_pattern
 
-from pytorch_lightning.utilities import rank_zero_warn
 from pytorch_lightning.utilities.apply_func import apply_to_collection
 
-try:
+if torch.distributed.is_available():
     from torch.distributed import ReduceOp
-except ImportError:
-
+else:
     class ReduceOp:
         SUM = None
 
-    rank_zero_warn("Unsupported `ReduceOp` for distributed computing")
-
 
 def _apply_to_inputs(func_to_apply: Callable, *dec_args, **dec_kwargs) -> Callable:
     """
diff --git a/pytorch_lightning/metrics/sklearns.py b/pytorch_lightning/metrics/sklearns.py
--- a/pytorch_lightning/metrics/sklearns.py
+++ b/pytorch_lightning/metrics/sklearns.py
@@ -19,17 +19,13 @@
 
 from pytorch_lightning import _logger as lightning_logger
 from pytorch_lightning.metrics.metric import NumpyMetric
-from pytorch_lightning.utilities import rank_zero_warn
 
-try:
+if torch.distributed.is_available():
     from torch.distributed import group
-except ImportError:
-
+else:
     class group:
         WORLD = None
 
-    rank_zero_warn("Unsupported `group` for distributed computing.")
-
 
 class SklearnMetric(NumpyMetric):
     """


Test patch: