Repository: pytorch/ignite

Issue:
More metrics for regression tasks and forecasting
Following the discussion in #305 
We propose to implement the following list of  metrics

D1 = magnitude of error
- Mean Error
- Mean Normalized Bias
- Fractional Bias

D2 = absolute error
- Median Absolute Error
- Mean Absolute Relative Error / Mean Absolute Percentage Error
- Median Absolute Percentage Error
- Median Relative Absolute Error
- Geometric Mean Absolute Error
- Geometric Mean Relative Absolute Error
- Wave Hedges Distance
- Fractional absolute error
- Canberra Metric

based on the paper: 
https://arxiv.org/pdf/1809.03006.pdf
See appendix 2, page 30.

What do you think ?

Implementation to put into [contrib](https://github.com/pytorch/ignite/tree/master/ignite/contrib/metrics)

Discussion:
vfdev-5: @weihuangxu thanks for the PR, could you please put PR template back.
weihuangxu: @vfdev-5 Sure. I have fixed the template. There is an issue regarding the devision in mean absolute relative error. If the ground truth is 0, it may raise numerical issue. I am not sure if we should raise an error or just show Nan in the output. What do you think?
vfdev-5: @weihuangxu maybe we could add a flag ~ `percent=True` to return Mean Absolute Percentage Error = MARE * 100 ?

> There is an issue regarding the devision in mean absolute relative error. If the ground truth is 0, it may raise numerical issue. I am not sure if we should raise an error or just show Nan in the output. What do you think?

@weihuangxu good question, not sure how to regularize this properly. 
Maybe we can add something like:
```
a = 0
if error < tol (1e-20):
     MARE = 0
else:
     MARE = Inf
``` 
@zasdfgbnm any ideas ?


zasdfgbnm: @vfdev-5 Considering metric arithmetic (https://github.com/pytorch/ignite/issues/321) is coming soon, would it be helpful to complicate the API by adding a `percent` argument? Builtin metrics like `Precision` does not have such thing.

For the divide by zero error, I would suggest just ignore the case that ground truth is zero, doing so the final result would automatically becomes `inf`,  and the user should know  preprocess their dataset if they want MARE.
vfdev-5: @zasdfgbnm good point with `percent`. Ok, let's skip this flag.

> For the divide by zero error, I would suggest just ignore the case that ground truth is zero, doing so the final result would automatically becomes inf, and the user should know preprocess their dataset if they want MARE.

So you suggest to ignore the cases with gt = 0 ? 

I did not get your point about "the user should know preprocess their dataset if they want MARE." ...
zasdfgbnm: @vfdev-5 as from the definition, MARE = sum(|error/gt|), gt is something given in the dataset, which I assume the users should but might not know well. If something in the dataset has gt=0, why do they insist on using ill-defined metrics like MARE? So all we need to do is to notify the user that they should not use this metric, by returning a `inf` or `nan` or raise an Exception.

So, considering that if we don't add any special handle of case gt=0, the user would know that they are doing things wrongly by getting a `inf` or `nan` result. But if we add 
```python
if error < tol (1e-20):
     MARE = 0
else:
     MARE = Inf
```
the user might sometimes get 0 instead of `nan`, which I consider as a silent failure.
vfdev-5: @zasdfgbnm Ok, I see your point. Let's raise an exception `NotComputableError` with a message that the ground truth is zero.
weihuangxu: @TheCodez Thanks for pointing out. 

@vfdev-5 I added the error message in update step when the ground truth has 0 value. 
vfdev-5: @weihuangxu thanks! Some comments in the code and could you also please check if docs are correctly rendered with formula:
```
cd docs
pip install -r docs/requirements.txt
make html
cd build && python3 -m http.server 1234
```
and go in the browser to 0.0.0.0:1234

Patch:
diff --git a/ignite/contrib/handlers/param_scheduler.py b/ignite/contrib/handlers/param_scheduler.py
index 0ec27bbc..5c26dd11 100644
--- a/ignite/contrib/handlers/param_scheduler.py
+++ b/ignite/contrib/handlers/param_scheduler.py
@@ -8,28 +8,37 @@ class ParamScheduler(object):
     training.
 
     Args:
-        optimizer (`torch.optim.Optimizer`): the optimizer to use
+        optimizer (`torch.optim.Optimizer` or dict): the optimizer or parameters group to use
         param_name (str): name of optimizer's parameter to update
         save_history (bool, optional): whether to log the parameter values
             (default=False)
     """
     def __init__(self, optimizer, param_name, save_history=False):
-        self.optimizer = optimizer
+
+        if isinstance(optimizer, dict):
+            self.optimizer_param_groups = [optimizer]
+        else:
+            self.optimizer_param_groups = optimizer.param_groups
         self.param_name = param_name
         self.save_history = save_history
         self.event_index = 0
 
-    def __call__(self, engine):
+    def __call__(self, engine, name=None):
+
         value = self.get_param()
-        for param_group in self.optimizer.param_groups:
+
+        for param_group in self.optimizer_param_groups:
             param_group[self.param_name] = value
 
+        if name is None:
+            name = self.param_name
+
         if self.save_history:
             if not hasattr(engine.state, 'param_history'):
                 setattr(engine.state, 'param_history', {})
-            engine.state.param_history.setdefault(self.param_name, [])
-            values = [pg[self.param_name] for pg in self.optimizer.param_groups]
-            engine.state.param_history[self.param_name].append(values)
+            engine.state.param_history.setdefault(name, [])
+            values = [pg[self.param_name] for pg in self.optimizer_param_groups]
+            engine.state.param_history[name].append(values)
 
         self.event_index += 1
 
@@ -44,7 +53,7 @@ class CyclicalScheduler(ParamScheduler):
     cycle of some size.
 
     Args:
-        optimizer (`torch.optim.Optimizer`): the optimizer to use
+        optimizer (`torch.optim.Optimizer` or dict): the optimizer or parameters group to use
         param_name (str): name of optimizer's parameter to update
         start_value (float): value at start of cycle
         end_value (float) : value at the middle of the cycle
@@ -77,13 +86,13 @@ class CyclicalScheduler(ParamScheduler):
         self.cycle_mult = cycle_mult
         self.cycle = 0
 
-    def __call__(self, engine):
+    def __call__(self, engine, name=None):
         if self.event_index != 0 and self.event_index % self.cycle_size == 0:
             self.event_index = 0
             self.cycle_size *= self.cycle_mult
             self.cycle += 1
 
-        return super(CyclicalScheduler, self).__call__(engine)
+        return super(CyclicalScheduler, self).__call__(engine, name)
 
 
 class LinearCyclicalScheduler(CyclicalScheduler):
@@ -91,7 +100,7 @@ class LinearCyclicalScheduler(CyclicalScheduler):
     adjusts it back to 'start_value' for a half-cycle.
 
     Args:
-        optimizer (`torch.optim.Optimizer`): the optimizer to use
+        optimizer (`torch.optim.Optimizer` or dict): the optimizer or parameters group to use
         param_name (str): name of optimizer's parameter to update
         start_value (float): value at start of cycle
         end_value (float) : value at the middle of the cycle
@@ -130,7 +139,7 @@ class CosineAnnealingScheduler(CyclicalScheduler):
     wave (as suggested in [Smith17]_).
 
     Args:
-        optimizer (`torch.optim.Optimizer`): the optimizer to use
+        optimizer (`torch.optim.Optimizer` or dict): the optimizer or parameters group to use
         param_name (str): name of optimizer's parameter to update
         start_value (float): value at start of cycle
         end_value (float) : value at the end of the cycle
@@ -156,6 +165,24 @@ class CosineAnnealingScheduler(CyclicalScheduler):
         # Anneals the learning rate from 1e-1 to 1e-3 over the course of 1 epoch.
         #
 
+    .. code-block:: python
+
+        from ignite.contrib.handlers.param_scheduler import CosineAnnealingScheduler
+        from ignite.contrib.handlers.param_scheduler import LinearCyclicalScheduler
+
+        optimizer = SGD(
+            [
+                {"params": model.base.parameters(), 'lr': 0.001),
+                {"params": model.fc.parameters(), 'lr': 0.01),
+            ]
+        )
+
+        scheduler1 = LinearCyclicalScheduler(optimizer.param_groups[0], 'lr', 1e-7, 1e-5, len(train_loader))
+        trainer.add_event_handler(Events.ITERATION_COMPLETED, scheduler1, "lr (base)")
+
+        scheduler2 = CosineAnnealingScheduler(optimizer.param_groups[1], 'lr', 1e-5, 1e-3, len(train_loader))
+        trainer.add_event_handler(Events.ITERATION_COMPLETED, scheduler2, "lr (fc)")
+
     .. [Smith17] Smith, Leslie N. "Cyclical learning rates for training neural networks."
                  Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on. IEEE, 2017
     """
@@ -177,7 +204,7 @@ class ConcatScheduler(ParamScheduler):
     switch to the next scheduler.
 
     Args:
-        optimizer (`torch.optim.Optimizer`): the optimizer to use
+        optimizer (`torch.optim.Optimizer` or dict): the optimizer or parameters group to use
         param_name (str): name of optimizer's parameter to update
         schedulers_list (list): List of three tuple of the order (scheduler_cls,
             scheduler_kwds, duration).
@@ -232,6 +259,7 @@ class ConcatScheduler(ParamScheduler):
         self._schedulers_list = schedulers_list
         self._schedulers_index = 0
         self._next_scheduler_switch = 0
+        self.optimizer = optimizer
 
     def _next_scheduler(self):
         scheduler_cls, scheduler_kwds, self._next_scheduler_switch = \
@@ -249,10 +277,10 @@ class ConcatScheduler(ParamScheduler):
         self._scheduler = scheduler_cls(**kwds)
         self._schedulers_index = (self._schedulers_index + 1) % len(self._schedulers_list)
 
-    def __call__(self, engine):
+    def __call__(self, engine, name=None):
         if self._next_scheduler_switch is not None:
             self._next_scheduler_switch -= 1
             if self._next_scheduler_switch <= 0:
                 self._next_scheduler()
 
-        return self._scheduler(engine)
+        return self._scheduler(engine, name)
diff --git a/ignite/contrib/metrics/__init__.py b/ignite/contrib/metrics/__init__.py
index 9f0d40b1..4f097761 100644
--- a/ignite/contrib/metrics/__init__.py
+++ b/ignite/contrib/metrics/__init__.py
@@ -1,4 +1,3 @@
-
 from ignite.contrib.metrics.average_precision import AveragePrecision
 from ignite.contrib.metrics.roc_auc import ROC_AUC
 from ignite.contrib.metrics.mae import MaximumAbsoluteError
diff --git a/ignite/contrib/metrics/regression/__init__.py b/ignite/contrib/metrics/regression/__init__.py
new file mode 100644
index 00000000..3bf5ebd6
--- /dev/null
+++ b/ignite/contrib/metrics/regression/__init__.py
@@ -0,0 +1,1 @@
+from ignite.contrib.metrics.regression.mean_absolute_relative_error import MeanAbsoluteRelativeError
diff --git a/ignite/contrib/metrics/regression/mean_absolute_relative_error.py b/ignite/contrib/metrics/regression/mean_absolute_relative_error.py
new file mode 100644
index 00000000..fc7fb7e2
--- /dev/null
+++ b/ignite/contrib/metrics/regression/mean_absolute_relative_error.py
@@ -0,0 +1,39 @@
+from __future__ import division
+from ignite.exceptions import NotComputableError
+from ignite.metrics import Metric
+import torch
+
+
+class MeanAbsoluteRelativeError(Metric):
+    r"""
+    Calculate Mean Absolute Relative Error
+
+    :math:`\text{MARE} = \frac{1}{n}\sum _j^n\frac{\left|A_j-P_j\right|}{\left|A_j\right|}`
+
+    where, :math:`A_j` is the ground truth and :math:`P_j` is the predicted value.
+
+    More details can be found in the reference `Botchkarev 2018`_ .
+
+    - `update` must receive output of the form `(y_pred, y)`
+
+    .. _`Botchkarev 2018`:
+        https://arxiv.org/ftp/arxiv/papers/1809/1809.03006.pdf
+    """
+
+    def reset(self):
+        self._sum_of_absolute_relative_errors = 0.0
+        self._num_samples = 0
+
+    def update(self, output):
+        y_pred, y = output
+        if (y == 0).any():
+            raise NotComputableError('The ground truth has 0')
+        absolute_error = torch.abs(y_pred - y.view_as(y_pred)) / torch.abs(y.view_as(y_pred))
+        self._sum_of_absolute_relative_errors += torch.sum(absolute_error).item()
+        self._num_samples += y.size()[0]
+
+    def compute(self):
+        if self._num_samples == 0:
+            raise NotComputableError('MeanAbsoluteRelativeError must have at least'
+                                     'one sample before it can be computed')
+        return self._sum_of_absolute_relative_errors / self._num_samples
diff --git a/ignite/engine/engine.py b/ignite/engine/engine.py
index e05ad519..b896cdef 100644
--- a/ignite/engine/engine.py
+++ b/ignite/engine/engine.py
@@ -142,6 +142,26 @@ class Engine(object):
         self._event_handlers[event_name].append((handler, args, kwargs))
         self._logger.debug("added handler for event %s ", event_name)
 
+    def has_event_handler(self, handler, event_name=None):
+        """Check if the specified event has the specified handler.
+
+        Args:
+            handler (Callable): the callable event handler.
+            event_name: The event the handler attached to. Set this
+                to ``None`` to search all events.
+        """
+        if event_name is not None:
+            if event_name not in self._event_handlers:
+                return False
+            events = [event_name]
+        else:
+            events = self._event_handlers
+        for e in events:
+            for h, _, _ in self._event_handlers[e]:
+                if h == handler:
+                    return True
+        return False
+
     def _check_signature(self, fn, fn_description, *args, **kwargs):
         exception_msg = None
 
diff --git a/ignite/metrics/metric.py b/ignite/metrics/metric.py
index b682001a..03b6e2c2 100644
--- a/ignite/metrics/metric.py
+++ b/ignite/metrics/metric.py
@@ -70,6 +70,8 @@ class Metric(object):
         engine.state.metrics[name] = self.compute()
 
     def attach(self, engine, name):
-        engine.add_event_handler(Events.EPOCH_STARTED, self.started)
-        engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)
         engine.add_event_handler(Events.EPOCH_COMPLETED, self.completed, name)
+        if not engine.has_event_handler(self.started, Events.EPOCH_STARTED):
+            engine.add_event_handler(Events.EPOCH_STARTED, self.started)
+        if not engine.has_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED):
+            engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)


Test patch:
diff --git a/tests/ignite/contrib/metrics/regression/__init__.py b/tests/ignite/contrib/metrics/regression/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py b/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py
new file mode 100644
index 00000000..3c580823
--- /dev/null
+++ b/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py
@@ -0,0 +1,62 @@
+from ignite.contrib.metrics.regression import MeanAbsoluteRelativeError
+from ignite.exceptions import NotComputableError
+import torch
+from pytest import approx
+from pytest import raises
+
+
+def test_mean_absolute_relative_error():
+    a = torch.rand(4)
+    b = torch.rand(4)
+    c = torch.rand(4)
+    d = torch.rand(4)
+    ground_truth = torch.rand(4)
+
+    m = MeanAbsoluteRelativeError()
+
+    m.update((a, ground_truth))
+    abs_error_a = torch.sum(torch.abs(ground_truth - a) / torch.abs(ground_truth))
+    num_samples_a = a.size()[0]
+    sum_error = abs_error_a
+    sum_samples = num_samples_a
+    MARE_a = sum_error / sum_samples
+    assert m.compute() == approx(MARE_a.item())
+
+    m.update((b, ground_truth))
+    abs_error_b = torch.sum(torch.abs(ground_truth - b) / torch.abs(ground_truth))
+    num_samples_b = b.size()[0]
+    sum_error += abs_error_b
+    sum_samples += num_samples_b
+    MARE_b = sum_error / sum_samples
+    assert m.compute() == approx(MARE_b.item())
+
+    m.update((c, ground_truth))
+    abs_error_c = torch.sum(torch.abs(ground_truth - c) / torch.abs(ground_truth))
+    num_samples_c = c.size()[0]
+    sum_error += abs_error_c
+    sum_samples += num_samples_c
+    MARE_c = sum_error / sum_samples
+    assert m.compute() == approx(MARE_c.item())
+
+    m.update((d, ground_truth))
+    abs_error_d = torch.sum(torch.abs(ground_truth - d) / torch.abs(ground_truth))
+    num_samples_d = d.size()[0]
+    sum_error += abs_error_d
+    sum_samples += num_samples_d
+    MARE_d = sum_error / sum_samples
+    assert m.compute() == approx(MARE_d.item())
+
+
+def test_zero_div():
+    a = torch.tensor([2.0, -1.0, -1.0, 2.0])
+    ground_truth = torch.tensor([0.0, 0.5, 0.2, 1.0])
+
+    m = MeanAbsoluteRelativeError()
+    with raises(NotComputableError):
+        m.update((a, ground_truth))
+
+
+def test_zero_sample():
+    m = MeanAbsoluteRelativeError()
+    with raises(NotComputableError):
+        m.compute()
diff --git a/tests/ignite/engine/test_engine.py b/tests/ignite/engine/test_engine.py
index 14e2b887..2030c697 100644
--- a/tests/ignite/engine/test_engine.py
+++ b/tests/ignite/engine/test_engine.py
@@ -130,6 +130,26 @@ def test_adding_multiple_event_handlers():
         handler.assert_called_once_with(engine)
 
 
+def test_has_event_handler():
+    engine = DummyEngine()
+    handlers = [MagicMock(), MagicMock()]
+    m = MagicMock()
+    for handler in handlers:
+        engine.add_event_handler(Events.STARTED, handler)
+    engine.add_event_handler(Events.COMPLETED, m)
+
+    for handler in handlers:
+        assert engine.has_event_handler(handler, Events.STARTED)
+        assert engine.has_event_handler(handler)
+        assert not engine.has_event_handler(handler, Events.COMPLETED)
+        assert not engine.has_event_handler(handler, Events.EPOCH_STARTED)
+
+    assert not engine.has_event_handler(m, Events.STARTED)
+    assert engine.has_event_handler(m, Events.COMPLETED)
+    assert engine.has_event_handler(m)
+    assert not engine.has_event_handler(m, Events.EPOCH_STARTED)
+
+
 def test_args_and_kwargs_are_passed_to_event():
     engine = DummyEngine()
     kwargs = {'a': 'a', 'b': 'b'}
diff --git a/tests/ignite/metrics/test_metric.py b/tests/ignite/metrics/test_metric.py
index 3754bca1..675169b8 100644
--- a/tests/ignite/metrics/test_metric.py
+++ b/tests/ignite/metrics/test_metric.py
@@ -1,5 +1,5 @@
 from ignite.metrics import Metric
-from ignite.engine import State
+from ignite.engine import Engine, State
 import torch
 from mock import MagicMock
 
@@ -69,3 +69,47 @@ def test_no_grad():
     state = State(output=(y_pred, y))
     engine = MagicMock(state=state)
     metric.iteration_completed(engine)
+
+
+def test_attach():
+    class CountMetric(Metric):
+        def __init__(self, value):
+            self.reset_count = 0
+            super(CountMetric, self).__init__()
+            self.reset_count = 0
+            self.compute_count = 0
+            self.update_count = 0
+            self.value = value
+
+        def reset(self):
+            self.reset_count += 1
+
+        def compute(self):
+            self.compute_count += 1
+            return self.value
+
+        def update(self, output):
+            self.update_count += 1
+
+    def process_function(*args, **kwargs):
+        return 1
+
+    engine = Engine(process_function)
+    m1 = CountMetric(123)
+    m2 = CountMetric(456)
+    m1.attach(engine, "m1")
+    m2.attach(engine, "m2_1")
+    m2.attach(engine, "m2_2")
+    engine.run(range(10), 5)
+
+    assert engine.state.metrics["m1"] == 123
+    assert engine.state.metrics["m2_1"] == 456
+    assert engine.state.metrics["m2_2"] == 456
+
+    assert m1.reset_count == 5
+    assert m1.compute_count == 5
+    assert m1.update_count == 50
+
+    assert m2.reset_count == 5
+    assert m2.compute_count == 10
+    assert m2.update_count == 50