Repository: Lightning-AI/pytorch-lightning

Issue:
 **gather_all_tensors_if_available**  share the same underlying storage for all GPUs
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

Hi, one of new features in   #2528  `gather_all_tensors_if_available` has a  `list` copy bug,  and this would lead that tensors in all GPUs  are the wrongly same as one GPU,  since they share the same storage:

https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/metrics/converters.py#L304

```
gathered_result = world_size * [torch.zeros_like(result)]
```
change into:
```
gathered_result = [torch.zeros_like(result) for _ in range(world_size)]
```






Discussion:
@justusschock @SkafteNicki 
@ShomyLiu good catch, would you be up for sending a PR? Please, note that the function is not used anywhere yet, but are there for future changes to the metric package. 
@SkafteNicki  it's my pleasure for a PR. I will finish this as soon as possible. 

Yeah, it's a new function to wrap the `torch.distributed.all_gather`.  But I think it is a very common use case;  especially, when using DDP mode, we always need to gather all the outputs cross all the GPUs.

@ShomyLiu Yes, I agree that it is a common use case.
Please ping me when PR is ready.

Resolving patch:
diff --git a/pytorch_lightning/metrics/converters.py b/pytorch_lightning/metrics/converters.py
--- a/pytorch_lightning/metrics/converters.py
+++ b/pytorch_lightning/metrics/converters.py
@@ -301,7 +301,7 @@ def gather_all_tensors_if_available(result: Union[torch.Tensor],
 
         world_size = torch.distributed.get_world_size(group)
 
-        gathered_result = world_size * [torch.zeros_like(result)]
+        gathered_result = [torch.zeros_like(result) for _ in range(world_size)]
 
         # sync and broadcast all
         torch.distributed.barrier(group=group)


Test patch:
diff --git a/tests/metrics/test_converters.py b/tests/metrics/test_converters.py
--- a/tests/metrics/test_converters.py
+++ b/tests/metrics/test_converters.py
@@ -15,6 +15,7 @@
     _numpy_metric_conversion,
     _tensor_metric_conversion,
     sync_ddp_if_available,
+    gather_all_tensors_if_available,
     tensor_metric,
     numpy_metric
 )
@@ -134,6 +135,17 @@ def _ddp_test_fn(rank, worldsize, add_offset: bool, reduction_mean=False):
             'Sync-Reduce does not work properly with DDP and Tensors'
 
 
+def _ddp_test_gather_all_tensors(rank, worldsize):
+    _setup_ddp(rank, worldsize)
+
+    tensor = torch.tensor([rank])
+    gather_tensors = gather_all_tensors_if_available(tensor)
+    mannual_tensors = [torch.tensor([i]) for i in range(worldsize)]
+
+    for t1, t2 in zip(gather_tensors, mannual_tensors):
+        assert(t1.equal(t2))
+
+
 @pytest.mark.skipif(sys.platform == "win32" , reason="DDP not available on windows")
 def test_sync_reduce_ddp():
     """Make sure sync-reduce works with DDP"""
@@ -164,6 +176,16 @@ def test_sync_reduce_simple():
         'Sync-Reduce does not work properly without DDP and Tensors'
 
 
+@pytest.mark.skipif(sys.platform == "win32" , reason="DDP not available on windows")
+def test_gather_all_tensors_ddp():
+    """Make sure gather_all_tensors works with DDP"""
+    tutils.reset_seed()
+    tutils.set_random_master_port()
+
+    worldsize = 2
+    mp.spawn(_ddp_test_gather_all_tensors, args=(worldsize, ), nprocs=worldsize)
+
+
 def _test_tensor_metric(is_ddp: bool):
     @tensor_metric()
     def tensor_test_metric(*args, **kwargs):