Repository: pytorch/ignite

Issue:
Distributed tests fail 
Recently while working on `ignite.contrib.metrics.regression` tests, I was trying to add distributed tests to some metrics that didn't have distributed tests. While i was running those tests when `world_size > 1` the result computed by the metric implemented in `ignite` differs from the one computed by `sklearn`(if imported from sklearn) / `numpy`. And here are examples for that:

- [Test wave hedges distance](https://gist.github.com/KickItLikeShika/497e80b2ebf2b99a9775e095868b62af)
`_test_distrib_integration()` fails in case of `world_size > 1` and if i didn't comment `all_gather()` in `_test_distrib_compute()` it fails as well!

- [Test fractional absolute error](https://gist.github.com/KickItLikeShika/8b6919f08b4d5ba531a363580517f791)
The same failures in the same two tests `_test_distrib_compute()` and `_test_distrib_integration()`.

related: #1284 


Discussion:
If I understand correctly the test, every process generates random numbers and compute the metric. Then, the result is compared to a score computed with numpy. 

But you have to uncomment the gather part otherwise it can’t be similar. 
* https://gist.github.com/KickItLikeShika/497e80b2ebf2b99a9775e095868b62af#file-test_wave_hedges_distance-py-L121
* https://gist.github.com/KickItLikeShika/8b6919f08b4d5ba531a363580517f791#file-test_fractional_absolute_error-py-L128
@sdesrozis Thanks for your comment! You understand the test correctly. I'm already doing `all_gather()` and i know it should be done, but the point is the `_test_distrib_compute()` doesn't work if i did `all_gather()`. And `_test_distrib_integration()` doesn't work no matter i'm gathering or not. Btw, the same problem happens with many other tests, for example: `test_mean_...`, `test_fractional_bias`. And i will try the other tests, it mostly will fail the same way...
I don't know if it's related, but i see a weird pattern here. Dor metrics implemented in `sklearn` and we import `sklearn` to use it for computing the metric it works fine, I guess the problem is with metrics we computed it with `numpy`. I have tried to add distributed tests for all metrics in `ignite.contrib.metrics.regression` and all the metrics we import `sklearn` to compute works fine.
I will have a look asap
Ah ah take a look, the metrics are not distributed. It means a reduction is needed in the implementation. I could fix it tomorrow. 
@sdesrozis That's great to hear! If you provided me some hints about what should be done i will be happy to help
@sdesrozis I think we can leave this task to @KickItLikeShika . 

@KickItLikeShika check here: https://pytorch.org/ignite/metrics.html#metrics-and-distributed-computations
and here : https://pytorch.org/ignite/distributed.html#ignite-distributed-utils for the list of collective ops.
Please, try to figure out how it works by your own and ask if you tried everything and you become blocked for sometime. Thanks !

Resolving patch:
diff --git a/ignite/contrib/metrics/regression/fractional_absolute_error.py b/ignite/contrib/metrics/regression/fractional_absolute_error.py
--- a/ignite/contrib/metrics/regression/fractional_absolute_error.py
+++ b/ignite/contrib/metrics/regression/fractional_absolute_error.py
@@ -4,6 +4,7 @@
 
 from ignite.contrib.metrics.regression._base import _BaseRegression
 from ignite.exceptions import NotComputableError
+from ignite.metrics.metric import reinit__is_reduced, sync_all_reduce
 
 
 class FractionalAbsoluteError(_BaseRegression):
@@ -32,21 +33,26 @@ class FractionalAbsoluteError(_BaseRegression):
         device: specifies which device updates are accumulated on. Setting the
             metric's device to be the same as your ``update`` arguments ensures the ``update`` method is
             non-blocking. By default, CPU.
+
+    .. versionchanged:: 0.5.0
+        - Works with DDP.
     """
 
+    @reinit__is_reduced
     def reset(self) -> None:
-        self._sum_of_errors = 0.0
+        self._sum_of_errors = torch.tensor(0.0, device=self._device)
         self._num_examples = 0
 
     def _update(self, output: Tuple[torch.Tensor, torch.Tensor]) -> None:
-        y_pred, y = output
+        y_pred, y = output[0].detach(), output[1].detach()
         errors = 2 * torch.abs(y.view_as(y_pred) - y_pred) / (torch.abs(y_pred) + torch.abs(y.view_as(y_pred)))
-        self._sum_of_errors += torch.sum(errors).item()
+        self._sum_of_errors += torch.sum(errors).to(self._device)
         self._num_examples += y.shape[0]
 
+    @sync_all_reduce("_num_examples", "_sum_of_errors")
     def compute(self) -> float:
         if self._num_examples == 0:
             raise NotComputableError(
                 "FractionalAbsoluteError must have at least one example before it can be computed."
             )
-        return self._sum_of_errors / self._num_examples
+        return self._sum_of_errors.item() / self._num_examples


Test patch:
diff --git a/tests/ignite/contrib/metrics/regression/test_fractional_absolute_error.py b/tests/ignite/contrib/metrics/regression/test_fractional_absolute_error.py
--- a/tests/ignite/contrib/metrics/regression/test_fractional_absolute_error.py
+++ b/tests/ignite/contrib/metrics/regression/test_fractional_absolute_error.py
@@ -1,7 +1,10 @@
+import os
+
 import numpy as np
 import pytest
 import torch
 
+import ignite.distributed as idist
 from ignite.contrib.metrics.regression import FractionalAbsoluteError
 from ignite.engine import Engine
 from ignite.exceptions import NotComputableError
@@ -71,11 +74,11 @@ def update_fn(engine, batch):
             idx = (engine.state.iteration - 1) * batch_size
             y_true_batch = np_y[idx : idx + batch_size]
             y_pred_batch = np_y_pred[idx : idx + batch_size]
-            return idx, torch.from_numpy(y_pred_batch), torch.from_numpy(y_true_batch)
+            return torch.from_numpy(y_pred_batch), torch.from_numpy(y_true_batch)
 
         engine = Engine(update_fn)
 
-        m = FractionalAbsoluteError(output_transform=lambda x: (x[1], x[2]))
+        m = FractionalAbsoluteError()
         m.attach(engine, "fab")
 
         np_y = y.numpy()
@@ -104,3 +107,158 @@ def get_test_cases():
         test_cases = get_test_cases()
         for y_pred, y, batch_size in test_cases:
             _test(y_pred, y, batch_size)
+
+
+def _test_distrib_compute(device):
+
+    rank = idist.get_rank()
+    torch.manual_seed(12)
+
+    def _test(metric_device):
+        metric_device = torch.device(metric_device)
+        m = FractionalAbsoluteError(device=metric_device)
+        torch.manual_seed(10 + rank)
+
+        y_pred = torch.rand(size=(100,), device=device)
+        y = torch.rand(size=(100,), device=device)
+
+        m.update((y_pred, y))
+
+        # gather y_pred, y
+        y_pred = idist.all_gather(y_pred)
+        y = idist.all_gather(y)
+
+        np_y = y.cpu().numpy()
+        np_y_pred = y_pred.cpu().numpy()
+
+        np_sum = (2 * np.abs((np_y_pred - np_y)) / (np.abs(np_y_pred) + np.abs(np_y))).sum()
+        np_len = len(np_y_pred)
+        np_ans = np_sum / np_len
+
+        assert m.compute() == pytest.approx(np_ans)
+
+    for _ in range(3):
+        _test("cpu")
+        if device.type != "xla":
+            _test(idist.device())
+
+
+def _test_distrib_integration(device):
+
+    rank = idist.get_rank()
+    torch.manual_seed(12)
+
+    def _test(n_epochs, metric_device):
+        metric_device = torch.device(metric_device)
+        n_iters = 80
+        s = 16
+        n_classes = 2
+
+        offset = n_iters * s
+        y_true = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
+        y_preds = torch.rand(size=(offset * idist.get_world_size(),)).to(device)
+
+        def update(engine, i):
+            return (
+                y_preds[i * s + rank * offset : (i + 1) * s + rank * offset],
+                y_true[i * s + rank * offset : (i + 1) * s + rank * offset],
+            )
+
+        engine = Engine(update)
+
+        fae = FractionalAbsoluteError(device=metric_device)
+        fae.attach(engine, "fae")
+
+        data = list(range(n_iters))
+        engine.run(data=data, max_epochs=n_epochs)
+
+        assert "fae" in engine.state.metrics
+
+        res = engine.state.metrics["fae"]
+        if isinstance(res, torch.Tensor):
+            res = res.cpu().numpy()
+
+        np_y = y_true.cpu().numpy()
+        np_y_pred = y_preds.cpu().numpy()
+
+        np_sum = (2 * np.abs((np_y_pred - np_y)) / (np.abs(np_y_pred) + np.abs(np_y))).sum()
+        np_len = len(np_y_pred)
+        np_ans = np_sum / np_len
+
+        assert pytest.approx(res) == np_ans
+
+    metric_devices = ["cpu"]
+    if device.type != "xla":
+        metric_devices.append(idist.device())
+    for metric_device in metric_devices:
+        for _ in range(2):
+            _test(n_epochs=1, metric_device=metric_device)
+            _test(n_epochs=2, metric_device=metric_device)
+
+
+@pytest.mark.distributed
+@pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
+@pytest.mark.skipif(torch.cuda.device_count() < 1, reason="Skip if no GPU")
+def test_distrib_gpu(distributed_context_single_node_nccl):
+    device = torch.device(f"cuda:{distributed_context_single_node_nccl['local_rank']}")
+    _test_distrib_compute(device)
+    _test_distrib_integration(device)
+
+
+@pytest.mark.distributed
+@pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
+def test_distrib_cpu(distributed_context_single_node_gloo):
+    device = torch.device("cpu")
+    _test_distrib_compute(device)
+    _test_distrib_integration(device)
+
+
+@pytest.mark.distributed
+@pytest.mark.skipif(not idist.has_hvd_support, reason="Skip if no Horovod dist support")
+@pytest.mark.skipif("WORLD_SIZE" in os.environ, reason="Skip if launched as multiproc")
+def test_distrib_hvd(gloo_hvd_executor):
+    device = torch.device("cpu" if not torch.cuda.is_available() else "cuda")
+    nproc = 4 if not torch.cuda.is_available() else torch.cuda.device_count()
+
+    gloo_hvd_executor(_test_distrib_compute, (device,), np=nproc, do_init=True)
+    gloo_hvd_executor(_test_distrib_integration, (device,), np=nproc, do_init=True)
+
+
+@pytest.mark.multinode_distributed
+@pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
+@pytest.mark.skipif("MULTINODE_DISTRIB" not in os.environ, reason="Skip if not multi-node distributed")
+def test_multinode_distrib_cpu(distributed_context_multi_node_gloo):
+    device = torch.device("cpu")
+    _test_distrib_compute(device)
+    _test_distrib_integration(device)
+
+
+@pytest.mark.multinode_distributed
+@pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
+@pytest.mark.skipif("GPU_MULTINODE_DISTRIB" not in os.environ, reason="Skip if not multi-node distributed")
+def test_multinode_distrib_gpu(distributed_context_multi_node_nccl):
+    device = torch.device(f"cuda:{distributed_context_multi_node_nccl['local_rank']}")
+    _test_distrib_compute(device)
+
+
+@pytest.mark.tpu
+@pytest.mark.skipif("NUM_TPU_WORKERS" in os.environ, reason="Skip if NUM_TPU_WORKERS is in env vars")
+@pytest.mark.skipif(not idist.has_xla_support, reason="Skip if no PyTorch XLA package")
+def test_distrib_single_device_xla():
+    device = idist.device()
+    _test_distrib_compute(device)
+    _test_distrib_integration(device)
+
+
+def _test_distrib_xla_nprocs(index):
+    device = idist.device()
+    _test_distrib_compute(device)
+    _test_distrib_integration(device)
+
+
+@pytest.mark.tpu
+@pytest.mark.skipif("NUM_TPU_WORKERS" not in os.environ, reason="Skip if no NUM_TPU_WORKERS in env vars")
+@pytest.mark.skipif(not idist.has_xla_support, reason="Skip if no PyTorch XLA package")
+def test_distrib_xla_nprocs(xmp_executor):
+    n = int(os.environ["NUM_TPU_WORKERS"])
+    xmp_executor(_test_distrib_xla_nprocs, args=(), nprocs=n)