Repository: pytorch/ignite

Issue:
More metrics for regression tasks and forecasting
Following the discussion in #305 
We propose to implement the following list of  metrics

D1 = magnitude of error
- Mean Error
- Mean Normalized Bias
- Fractional Bias

D2 = absolute error
- Median Absolute Error
- Mean Absolute Relative Error / Mean Absolute Percentage Error
- Median Absolute Percentage Error
- Median Relative Absolute Error
- Geometric Mean Absolute Error
- Geometric Mean Relative Absolute Error
- Wave Hedges Distance
- Fractional absolute error
- Canberra Metric

based on the paper: 
https://arxiv.org/pdf/1809.03006.pdf
See appendix 2, page 30.

What do you think ?

Implementation to put into [contrib](https://github.com/pytorch/ignite/tree/master/ignite/contrib/metrics)


Discussion:
@vfdev-5 I was interested in working on some of these metrics. Can I help out?
@anmolsjoshi yes, you can implement one or several metrics from the list and send a PR as always :)
Would there be memory issue with `Median XXX Error`? It requires recording all histories.
@zasdfgbnm yes it can potentially lead to a memory issue if we implement it by storing everything. 
Today we already have `EpochMetric` that stores all predictions and ground-truths. BTW it worth to note that in the docs of `EpochMetric`.

@vfdev-5 @anmolsjoshi @zasdfgbnm I think metrics are so cool. I would like to help with this issue. May I know what metrics you have been working on right now? I can start with some of the rest metrics.
@weihuangxu I would say you can start with any metric you like (for instance avoid taking one with median) and send a PR. At the moment, there is no PRs, so they are all available :)

Resolving patch:
diff --git a/ignite/contrib/metrics/__init__.py b/ignite/contrib/metrics/__init__.py
--- a/ignite/contrib/metrics/__init__.py
+++ b/ignite/contrib/metrics/__init__.py
@@ -1,4 +1,3 @@
-
 from ignite.contrib.metrics.average_precision import AveragePrecision
 from ignite.contrib.metrics.roc_auc import ROC_AUC
 from ignite.contrib.metrics.mae import MaximumAbsoluteError
diff --git a/ignite/contrib/metrics/regression/__init__.py b/ignite/contrib/metrics/regression/__init__.py
new file mode 100644
--- /dev/null
+++ b/ignite/contrib/metrics/regression/__init__.py
@@ -0,0 +1 @@
+from ignite.contrib.metrics.regression.mean_absolute_relative_error import MeanAbsoluteRelativeError
diff --git a/ignite/contrib/metrics/regression/mean_absolute_relative_error.py b/ignite/contrib/metrics/regression/mean_absolute_relative_error.py
new file mode 100644
--- /dev/null
+++ b/ignite/contrib/metrics/regression/mean_absolute_relative_error.py
@@ -0,0 +1,39 @@
+from __future__ import division
+from ignite.exceptions import NotComputableError
+from ignite.metrics import Metric
+import torch
+
+
+class MeanAbsoluteRelativeError(Metric):
+    r"""
+    Calculate Mean Absolute Relative Error
+
+    :math:`\text{MARE} = \frac{1}{n}\sum _j^n\frac{\left|A_j-P_j\right|}{\left|A_j\right|}`
+
+    where, :math:`A_j` is the ground truth and :math:`P_j` is the predicted value.
+
+    More details can be found in the reference `Botchkarev 2018`_ .
+
+    - `update` must receive output of the form `(y_pred, y)`
+
+    .. _`Botchkarev 2018`:
+        https://arxiv.org/ftp/arxiv/papers/1809/1809.03006.pdf
+    """
+
+    def reset(self):
+        self._sum_of_absolute_relative_errors = 0.0
+        self._num_samples = 0
+
+    def update(self, output):
+        y_pred, y = output
+        if (y == 0).any():
+            raise NotComputableError('The ground truth has 0')
+        absolute_error = torch.abs(y_pred - y.view_as(y_pred)) / torch.abs(y.view_as(y_pred))
+        self._sum_of_absolute_relative_errors += torch.sum(absolute_error).item()
+        self._num_samples += y.size()[0]
+
+    def compute(self):
+        if self._num_samples == 0:
+            raise NotComputableError('MeanAbsoluteRelativeError must have at least'
+                                     'one sample before it can be computed')
+        return self._sum_of_absolute_relative_errors / self._num_samples


Test patch:
diff --git a/tests/ignite/contrib/metrics/regression/__init__.py b/tests/ignite/contrib/metrics/regression/__init__.py
new file mode 100644
diff --git a/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py b/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py
new file mode 100644
--- /dev/null
+++ b/tests/ignite/contrib/metrics/regression/test_mean_absolute_relative_error.py
@@ -0,0 +1,62 @@
+from ignite.contrib.metrics.regression import MeanAbsoluteRelativeError
+from ignite.exceptions import NotComputableError
+import torch
+from pytest import approx
+from pytest import raises
+
+
+def test_mean_absolute_relative_error():
+    a = torch.rand(4)
+    b = torch.rand(4)
+    c = torch.rand(4)
+    d = torch.rand(4)
+    ground_truth = torch.rand(4)
+
+    m = MeanAbsoluteRelativeError()
+
+    m.update((a, ground_truth))
+    abs_error_a = torch.sum(torch.abs(ground_truth - a) / torch.abs(ground_truth))
+    num_samples_a = a.size()[0]
+    sum_error = abs_error_a
+    sum_samples = num_samples_a
+    MARE_a = sum_error / sum_samples
+    assert m.compute() == approx(MARE_a.item())
+
+    m.update((b, ground_truth))
+    abs_error_b = torch.sum(torch.abs(ground_truth - b) / torch.abs(ground_truth))
+    num_samples_b = b.size()[0]
+    sum_error += abs_error_b
+    sum_samples += num_samples_b
+    MARE_b = sum_error / sum_samples
+    assert m.compute() == approx(MARE_b.item())
+
+    m.update((c, ground_truth))
+    abs_error_c = torch.sum(torch.abs(ground_truth - c) / torch.abs(ground_truth))
+    num_samples_c = c.size()[0]
+    sum_error += abs_error_c
+    sum_samples += num_samples_c
+    MARE_c = sum_error / sum_samples
+    assert m.compute() == approx(MARE_c.item())
+
+    m.update((d, ground_truth))
+    abs_error_d = torch.sum(torch.abs(ground_truth - d) / torch.abs(ground_truth))
+    num_samples_d = d.size()[0]
+    sum_error += abs_error_d
+    sum_samples += num_samples_d
+    MARE_d = sum_error / sum_samples
+    assert m.compute() == approx(MARE_d.item())
+
+
+def test_zero_div():
+    a = torch.tensor([2.0, -1.0, -1.0, 2.0])
+    ground_truth = torch.tensor([0.0, 0.5, 0.2, 1.0])
+
+    m = MeanAbsoluteRelativeError()
+    with raises(NotComputableError):
+        m.update((a, ground_truth))
+
+
+def test_zero_sample():
+    m = MeanAbsoluteRelativeError()
+    with raises(NotComputableError):
+        m.compute()