Repository: pytorch/ignite

Issue:
Improve Accumulation metric on tensors to work with Apex
## 🚀 Feature Request
It seems that the ignite is not compatible with apex. When i am trying to use the (automatic mixed precision)amp from the Apex for the GPU Acceleration, I have encountered the following error. Look around the Ignite source code and I find that it uses the original torch.Tensor. Is there any timeline for Ignite to be compatible with Apex?
```
m.update(loss)
--
File "/usr/local/lib/python3.6/site-packages/ignite/metrics/metric.py", line 575, in wrapper
func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/ignite/metrics/accumulation.py", line 73, in update
self.accumulator = self._op(self.accumulator, output)
File "/usr/local/lib/python3.6/site-packages/ignite/metrics/accumulation.py", line 130, in _mean_op
return a + x
File "/usr/local/lib64/python3.6/site-packages/apex/amp/wrap.py", line 62, in wrapper
.format(types))
NotImplementedError: Do not know how to handle these types to promote: {'FloatTensor', 'DoubleTensor'}
```
Apex: https://github.com/NVIDIA/apex

## Environment

- PyTorch Version (1.4.0):
- Ignite Version (0.4.4):
- OS (Linux):
- Python version:  python 3.6
- Apex Version (0.1)


Discussion:
Thanks for reporting @youshaox! Normally, Ignite is compatible with Apex, please take a look at this notebook: https://github.com/pytorch/ignite/blob/master/examples/notebooks/Cifar100_bench_amp.ipynb
in older versions we also provided image segmentation example with Apex: https://github.com/pytorch/ignite/blob/0.4.4/examples/references/segmentation/pascal_voc2012/code/scripts/training.py

However, let me still investigate what happens with accumulation metric. Meanwhile, could you please provide a reproducible small code snippet ? Thanks 
@youshaox I could reproduce the issue you reported. A workaround in this case can be:
```diff
- m.update(loss)
+ m.update(loss.item())
```

We can improve the code of `VariableAccumulation`: https://github.com/pytorch/ignite/blob/5837b2d40ba8302e986a5a5115b734990cbe201b/ignite/metrics/accumulation.py#L68-L73
which may fail to auto promote types if we are using apex which patches that. Maybe, something like that:
```python
        if isinstance(output, torch.Tensor):
            output = output.detach()
            if not (output.device == self._device and output.dtype == self.accumulator.dtype):
                output = output.to(self.accumulator)

        self.accumulator = self._op(self.accumulator, output)
```

Could help with this issue, however struggling to reproduce it with autocast, @vfdev-5 would you still have the code snippet ? thanks
@fco-dv it is apex only related failure. To reproduce it, try to add two tensors of different types: f32 and f64 once apex is fully initialized (following above discussion).

Resolving patch:
diff --git a/ignite/metrics/accumulation.py b/ignite/metrics/accumulation.py
--- a/ignite/metrics/accumulation.py
+++ b/ignite/metrics/accumulation.py
@@ -67,8 +67,8 @@ def update(self, output: Union[float, torch.Tensor]) -> None:
 
         if isinstance(output, torch.Tensor):
             output = output.detach()
-            if output.device != self._device:
-                output = output.to(self._device)
+            if not (output.device == self._device and output.dtype == self.accumulator.dtype):
+                output = output.to(self.accumulator)
 
         self.accumulator = self._op(self.accumulator, output)
 


Test patch:
diff --git a/tests/ignite/metrics/test_accumulation.py b/tests/ignite/metrics/test_accumulation.py
--- a/tests/ignite/metrics/test_accumulation.py
+++ b/tests/ignite/metrics/test_accumulation.py
@@ -1,8 +1,11 @@
 import os
+from importlib.util import find_spec
 
 import numpy as np
 import pytest
 import torch
+from torch.nn import Linear
+from torch.optim import SGD
 
 import ignite.distributed as idist
 from ignite.engine import Engine
@@ -391,6 +394,34 @@ def _test_distrib_accumulator_device(device):
         ), f"{type(m.accumulator.device)}:{m.accumulator.device} vs {type(metric_device)}:{metric_device}"
 
 
+def _test_apex_average(device, amp_mode, opt_level):
+    assert amp_mode == "apex"
+    assert device == "cuda"
+
+    model = Linear(1, 1)
+
+    if device:
+        model.to(device)
+
+    model.weight.data.zero_()
+    model.bias.data.zero_()
+    optimizer = SGD(model.parameters(), 0.1)
+
+    from apex import amp
+
+    model, optimizer = amp.initialize(model, optimizer, opt_level=opt_level)
+
+    mean_var = VariableAccumulation(lambda a, x: a + x)
+    y_true = torch.rand(100).float().to(device)
+
+    for y in y_true:
+        mean_var.update(y)
+
+    a, n = mean_var.compute()
+    assert a.item() == pytest.approx(y_true.sum().item())
+    assert n == len(y_true)
+
+
 @pytest.mark.distributed
 @pytest.mark.skipif(not idist.has_native_dist_support, reason="Skip if no native dist support")
 @pytest.mark.skipif(torch.cuda.device_count() < 1, reason="Skip if no GPU")
@@ -482,3 +513,13 @@ def _test_distrib_xla_nprocs(index):
 def test_distrib_xla_nprocs(xmp_executor):
     n = int(os.environ["NUM_TPU_WORKERS"])
     xmp_executor(_test_distrib_xla_nprocs, args=(), nprocs=n)
+
+
+@pytest.mark.skipif(not torch.cuda.is_available(), reason="Skip if no GPU")
+@pytest.mark.skipif(not find_spec("apex"), reason="Skip if no APEX")
+def test_apex_average_on_cuda():
+    device = "cuda"
+    _test_apex_average(device, amp_mode="apex", opt_level="O0")
+    _test_apex_average(device, amp_mode="apex", opt_level="O1")
+    _test_apex_average(device, amp_mode="apex", opt_level="O2")
+    _test_apex_average(device, amp_mode="apex", opt_level="O3")