Repository: sktime/sktime
Pull request: added mape_loss and cosmetic fixes to notebooks (removed kernel)
#### Reference Issues/PRs
Fixes #499.

#### What does this implement/fix? Explain your changes.
Implements MAPE loss function.

#### Does your contribution introduce a new dependency? If yes, which one?
No new dependencies introduced

#### PR checklist

##### For all contributions
- [x] I've added myself to the [list of contributors](https://github.com/alan-turing-institute/sktime/blob/master/.all-contributorsrc).
- [ ] Optionally, I've updated sktime's [CODEOWNERS](https://github.com/alan-turing-institute/sktime/blob/master/CODEOWNERS) to receive notifications about future changes to these files.
- [x] I've added unit tests and made sure they pass locally.

#### Any other comments?
Another option of implementing this solution would be to re-use sk-learn implementation via simple wrapper. But its small enough that perhaps its better to keep it as a separate implementation.

Problem statement:
MAPE loss function implementation
**Is your feature request related to a problem? Please describe.**
A need for MAPE loss function to be able to compare forecasting results.

**Describe the solution you'd like**
MAPE loss function compatible with sktime

**Describe alternatives you've considered**
None

**Additional context**
NA

Hints from issue discussion:
Implemented in PR #500. Please have a look. Thx

Commit message: Merge branch 'master' into ftr_tch_add_mape

Patch to review:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index dd8a695399e..be82f301f19 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -6,6 +6,12 @@ All notable changes to this project will be documented in this file. We keep tra
 [0.5.0] - 2020-xx-xx
 --------------------
 
+Added
+~~~~~
+* Added mape_loss function @tch
+* Started unit tests for mape_loss and smape_loss functions @tch
+* Removed dependence on pycharm kernel from some of example notebooks @tch
+
 
 [0.4.3] - 2020-10-20
 --------------------
diff --git a/examples/catch22.ipynb b/examples/catch22.ipynb
index 7097b769cb9..f2e7c6b8df9 100644
--- a/examples/catch22.ipynb
+++ b/examples/catch22.ipynb
@@ -99,7 +99,9 @@
    "outputs": [
     {
      "data": {
-      "text/plain": "Catch22()"
+      "text/plain": [
+       "Catch22()"
+      ]
      },
      "execution_count": 3,
      "metadata": {},
@@ -114,6 +116,11 @@
   {
    "cell_type": "code",
    "execution_count": 4,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -147,27 +154,24 @@
    "source": [
     "transformed_data_uv = c22_uv.transform(IPD_X_train)\n",
     "print(transformed_data_uv.head())"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   },
   {
    "cell_type": "markdown",
+   "metadata": {},
    "source": [
     "The transform `Catch22` method will process all 22 features.\n",
     "For individual features, the transform_single_feature method can be used when provided with a numeric feature ID or the feature name."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": 5,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -181,33 +185,32 @@
    "source": [
     "transformed_feature_uv = c22_uv._transform_single_feature(IPD_X_train, \"CO_f1ecac\")\n",
     "print(transformed_feature_uv)"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   },
   {
    "cell_type": "markdown",
+   "metadata": {},
    "source": [
     "### Multivariate\n",
     "\n",
     "Transformation of multivariate data is supported by `Catch22`.\n",
     "The default procedure will concatenate each column prior to transformation."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": 6,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "data": {
-      "text/plain": "Catch22()"
+      "text/plain": [
+       "Catch22()"
+      ]
      },
      "execution_count": 6,
      "metadata": {},
@@ -217,17 +220,16 @@
    "source": [
     "c22_mv = Catch22()\n",
     "c22_mv.fit(BM_X_train, BM_y_train)"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": 7,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -261,13 +263,7 @@
    "source": [
     "transformed_data_mv = c22_mv.transform(BM_X_train)\n",
     "print(transformed_data_mv.head())"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   },
   {
    "cell_type": "markdown",
@@ -286,7 +282,9 @@
    "outputs": [
     {
      "data": {
-      "text/plain": "Catch22ForestClassifier(random_state=0)"
+      "text/plain": [
+       "Catch22ForestClassifier(random_state=0)"
+      ]
      },
      "execution_count": 8,
      "metadata": {},
@@ -319,9 +317,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "name": "pycharm-28faf070",
+   "display_name": "Python 3",
    "language": "python",
-   "display_name": "PyCharm (sktime-fork)"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -333,7 +331,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.6"
+   "version": "3.8.6"
   }
  },
  "nbformat": 4,
diff --git a/examples/dictionary_based_classification.ipynb b/examples/dictionary_based_classification.ipynb
index e2fc9584b94..f8c824ac83f 100644
--- a/examples/dictionary_based_classification.ipynb
+++ b/examples/dictionary_based_classification.ipynb
@@ -242,6 +242,7 @@
   },
   {
    "cell_type": "markdown",
+   "metadata": {},
    "source": [
     "## 6. Temporal Dictionary Ensemble (TDE)\n",
     "\n",
@@ -250,14 +251,16 @@
     "A Gaussian processes regressor is used to select new parameter sets to evaluate for the ensemble, predicting the accuracy of a set of parameter values using past classifier performances.\n",
     "\n",
     "Inheriting the cBOSS ensemble structure, the number of parameter samples $k$, time limit $t$ and max ensemble size $s$ remain as parameters to be set accounting for memory and time requirements."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": 6,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -286,28 +289,25 @@
     "\n",
     "tde_preds = tde.predict(X_test)\n",
     "print(\"TDE Accuracy: \" + str(metrics.accuracy_score(y_test, tde_preds)))"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   },
   {
    "cell_type": "markdown",
+   "metadata": {},
    "source": [
     "## 7. WEASEL+MUSE (Multivariate Symbolic Extension)\n",
     "\n",
     "WEASEL+MUSE is the multivariate extension of WEASEL."
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": 7,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -323,20 +323,14 @@
     "\n",
     "muse_preds = muse.predict(X_test_mv)\n",
     "print(\"MUSE Accuracy: \" + str(metrics.accuracy_score(y_test_mv, muse_preds)))"
-   ],
-   "metadata": {
-    "collapsed": false,
-    "pycharm": {
-     "name": "#%%\n"
-    }
-   }
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "name": "pycharm-28faf070",
+   "display_name": "Python 3",
    "language": "python",
-   "display_name": "PyCharm (sktime-fork)"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -348,7 +342,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.8.6"
   }
  },
  "nbformat": 4,
diff --git a/examples/interval_based_classification.ipynb b/examples/interval_based_classification.ipynb
index 9e1d37bffea..7ca2b611e01 100644
--- a/examples/interval_based_classification.ipynb
+++ b/examples/interval_based_classification.ipynb
@@ -152,9 +152,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "name": "pycharm-28faf070",
+   "display_name": "Python 3",
    "language": "python",
-   "display_name": "PyCharm (sktime-fork)"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -166,7 +166,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.6"
+   "version": "3.8.6"
   }
  },
  "nbformat": 4,
diff --git a/sktime/performance_metrics/forecasting/__init__.py b/sktime/performance_metrics/forecasting/__init__.py
index a5e6b53cb76..a27e803f771 100644
--- a/sktime/performance_metrics/forecasting/__init__.py
+++ b/sktime/performance_metrics/forecasting/__init__.py
@@ -1,19 +1,20 @@
 #!/usr/bin/env python3 -u
-# coding: utf-8
+# -*- coding: utf-8 -*-
 # copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
 
-__author__ = ["Markus Löning"]
+__author__ = ["Markus Löning", "Tomasz Chodakowski"]
 __all__ = [
     "MASE",
     "sMAPE",
     "mase_loss",
     "smape_loss",
-    "make_forecasting_scorer"
+    "mape_loss",
+    "make_forecasting_scorer",
 ]
 
 from sktime.performance_metrics.forecasting._classes import MASE
-from sktime.performance_metrics.forecasting._classes import \
-    make_forecasting_scorer
+from sktime.performance_metrics.forecasting._classes import make_forecasting_scorer
 from sktime.performance_metrics.forecasting._classes import sMAPE
 from sktime.performance_metrics.forecasting._functions import mase_loss
 from sktime.performance_metrics.forecasting._functions import smape_loss
+from sktime.performance_metrics.forecasting._functions import mape_loss
diff --git a/sktime/performance_metrics/forecasting/_classes.py b/sktime/performance_metrics/forecasting/_classes.py
index 0d84d7a1ae3..9ebc1a834f6 100644
--- a/sktime/performance_metrics/forecasting/_classes.py
+++ b/sktime/performance_metrics/forecasting/_classes.py
@@ -1,12 +1,12 @@
+# -*- coding: utf-8 -*-
 from sktime.performance_metrics.forecasting._functions import mase_loss
-from sktime.performance_metrics.forecasting._functions import smape_loss
+from sktime.performance_metrics.forecasting._functions import smape_loss, mape_loss
 
-__author__ = ['Markus Löning']
-__all__ = ["MetricFunctionWrapper", "make_forecasting_scorer", "MASE", "sMAPE"]
+__author__ = ["Markus Löning", "Tomasz Chodakowski"]
+__all__ = ["MetricFunctionWrapper", "make_forecasting_scorer", "MASE", "sMAPE", "MAPE"]
 
 
 class MetricFunctionWrapper:
-
     def __init__(self, fn, name=None, greater_is_better=False):
         self.fn = fn
         self.name = name if name is not None else fn.__name__
@@ -18,25 +18,34 @@ def __call__(self, y_test, y_pred, *args, **kwargs):
 
 def make_forecasting_scorer(fn, name=None, greater_is_better=False):
     """Factory method for creating metric classes from metric functions"""
-    return MetricFunctionWrapper(fn, name=name,
-                                 greater_is_better=greater_is_better)
+    return MetricFunctionWrapper(fn, name=name, greater_is_better=greater_is_better)
 
 
 class MASE(MetricFunctionWrapper):
-
     def __init__(self):
         name = "MASE"
         fn = mase_loss
         greater_is_better = False
-        super(MASE, self).__init__(fn=fn, name=name,
-                                   greater_is_better=greater_is_better)
+        super(MASE, self).__init__(
+            fn=fn, name=name, greater_is_better=greater_is_better
+        )
 
 
 class sMAPE(MetricFunctionWrapper):
-
     def __init__(self):
         name = "sMAPE"
         fn = smape_loss
         greater_is_better = False
-        super(sMAPE, self).__init__(fn=fn, name=name,
-                                    greater_is_better=greater_is_better)
+        super(sMAPE, self).__init__(
+            fn=fn, name=name, greater_is_better=greater_is_better
+        )
+
+
+class MAPE(MetricFunctionWrapper):
+    def __init__(self):
+        name = "MAPE"
+        fn = mape_loss
+        greater_is_better = False
+        super(sMAPE, self).__init__(
+            fn=fn, name=name, greater_is_better=greater_is_better
+        )
diff --git a/sktime/performance_metrics/forecasting/_functions.py b/sktime/performance_metrics/forecasting/_functions.py
index 8f93423f03a..d6be80f4fa0 100644
--- a/sktime/performance_metrics/forecasting/_functions.py
+++ b/sktime/performance_metrics/forecasting/_functions.py
@@ -8,8 +8,8 @@
 from sktime.utils.validation.series import check_time_index
 from sktime.utils.validation.forecasting import check_y
 
-__author__ = ["Markus Löning"]
-__all__ = ["mase_loss", "smape_loss"]
+__author__ = ["Markus Löning", "Tomasz Chodakowski"]
+__all__ = ["mase_loss", "smape_loss", "mape_loss"]
 
 
 def mase_loss(y_test, y_pred, y_train, sp=1):
@@ -97,3 +97,41 @@ def smape_loss(y_test, y_pred):
     nominator = np.abs(y_test - y_pred)
     denominator = np.abs(y_test) + np.abs(y_pred)
     return np.mean(2.0 * nominator / denominator)
+
+
+def mape_loss(y_test, y_pred):
+    """Mean absolute percentage error (MAPE)
+        MAPE output is non-negative floating point where the best value is 0.0.
+        There is no limit on how large the error can be, particulalrly when `y_test`
+        values are close to zero. In such cases the function returns a large value
+        instead of `inf`.
+
+    Parameters
+    ----------
+    y_test : pandas Series of shape = (fh,) where fh is the forecasting horizon
+        Ground truth (correct) target values.
+    y_pred : pandas Series of shape = (fh,)
+        Estimated target values.
+
+    Returns
+    -------
+    loss : float
+        MAPE loss expressed as a fractional number rather than percentage point.
+
+
+    Examples
+    --------
+    >>> from sklearn.metrics import mean_absolute_error
+    >>> y_test = pd.Series([1, -1, 2])
+    >>> y_pred = pd.Series([2, -2, 4])
+    >>> mape_loss(y_test, y_pred)
+    1.0
+    """
+
+    y_test = check_y(y_test)
+    y_pred = check_y(y_pred)
+    check_equal_time_index(y_test, y_pred)
+
+    eps = np.finfo(np.float64).eps
+
+    return np.mean(np.abs(y_test - y_pred) / np.maximum(np.abs(y_test), eps))
diff --git a/sktime/performance_metrics/tests/__init__.py b/sktime/performance_metrics/tests/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/sktime/performance_metrics/tests/_config.py b/sktime/performance_metrics/tests/_config.py
new file mode 100644
index 00000000000..20a67f49356
--- /dev/null
+++ b/sktime/performance_metrics/tests/_config.py
@@ -0,0 +1,22 @@
+#!/usr/bin/env python3 -u
+# -*- coding: utf-8 -*-
+
+__author__ = ["Tomasz Chodakowski"]
+__all__ = [
+    "TEST_YS",
+    "TEST_YS_ZERO",
+]
+
+import pandas as pd
+
+from sktime.utils._testing.forecasting import _make_series
+
+RANDOM_SEED = 42
+TEST_YS = [
+    _make_series(n_timepoints=50, random_state=RANDOM_SEED),
+    _make_series(n_timepoints=1, random_state=RANDOM_SEED),
+    pd.Series([0.0000001, 0.0000002, 0.0000003]),
+]
+
+
+TEST_YS_ZERO = [pd.Series([0.0, 0.0, 0.0])]
diff --git a/sktime/performance_metrics/tests/test_performance_metrics_forecasting.py b/sktime/performance_metrics/tests/test_performance_metrics_forecasting.py
new file mode 100644
index 00000000000..79d61910cac
--- /dev/null
+++ b/sktime/performance_metrics/tests/test_performance_metrics_forecasting.py
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3 -u
+# -*- coding: utf-8 -*-
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+__author__ = ["Tomasz Chodakowski"]
+
+import pytest
+
+from sktime.performance_metrics.forecasting import mape_loss
+from sktime.performance_metrics.forecasting import smape_loss
+
+from sktime.performance_metrics.tests._config import TEST_YS, TEST_YS_ZERO
+
+
+@pytest.mark.parametrize("test_y", TEST_YS_ZERO)
+def test_all_losses_near_zero(test_y):
+    # mape should be large when predicitons are close to zero
+    y_test = test_y
+    y_pred = y_test + 0.001
+    assert mape_loss(y_test, y_pred) > 1e6
+
+
+@pytest.mark.parametrize("test_y", TEST_YS)
+def test_mape_loss(test_y):
+    y_test = test_y
+    y_pred = y_test
+    assert mape_loss(y_test, y_pred) == pytest.approx(0.0)
+
+    y_pred = y_test * 1.1
+    assert mape_loss(y_test, y_pred) == pytest.approx(0.1)
+
+    y_pred = y_test * 0.9
+    assert mape_loss(y_test, y_pred) == pytest.approx(0.1)
+
+    y_pred = y_test * 1.000001
+    assert mape_loss(y_test, y_pred) == pytest.approx(0.000001)
+
+    y_pred = y_test * 2
+    assert mape_loss(y_test, y_pred) == pytest.approx(1.0)
+
+    y_pred = y_test * 100
+    assert mape_loss(y_test, y_pred) == pytest.approx(99.0)
+
+
+@pytest.mark.parametrize("test_y", TEST_YS)
+def test_smape_loss(test_y):
+    y_test = test_y
+    y_pred = y_test
+    assert smape_loss(y_test, y_pred) == pytest.approx(0.0)
+    assert smape_loss(y_pred, y_test) == pytest.approx(0.0)
+
+    y_pred = y_test * 1.1
+    assert smape_loss(y_test, y_pred) == pytest.approx(0.095238095238)
+    assert smape_loss(y_pred, y_test) == pytest.approx(0.095238095238)
+
+    y_pred = y_test * 1.000001
+    assert smape_loss(y_test, y_pred) == pytest.approx(0.000001)
+    assert smape_loss(y_pred, y_test) == pytest.approx(0.000001)
+
+    y_pred = y_test * 2.0
+    assert smape_loss(y_test, y_pred) == pytest.approx(0.6666666)
+    assert smape_loss(y_pred, y_test) == pytest.approx(0.6666666)
+
+    y_pred = y_test * 100
+    assert smape_loss(y_test, y_pred) == smape_loss(y_pred, y_test)


Code review comments:
diff_hunk: @@ -0,0 +1,65 @@
+#!/usr/bin/env python3 -u
+# -*- coding: utf-8 -*-
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+
+__author__ = ["Tomasz Chodakowski"]
+
+import pytest
+
+from sktime.performance_metrics.forecasting import mape_loss
+from sktime.performance_metrics.forecasting import smape_loss
+
+from sktime.performance_metrics.tests._config import TEST_YS, TEST_YS_ZERO
+
+
+@pytest.mark.parametrize("test_y", TEST_YS_ZERO)
+def test_all_losses_near_zero(test_y):
path: sktime/performance_metrics/tests/test_performance_metrics_forecasting.py
text: @user1:
I'd either rename the function to `test_mape_near_zero` or add the other losses to the function. 

@author:
Thanks Markus, I have changed the name to test_mape_loss_near_zero but will revisit it to introduce near zero checks for all losses as this would be helpful. However, there is small discrepancy between the way sMAPE/MASE and MAPE handle such situation. The former return nan when encountering 0 denominator while MAPE returns large value (that comes from division by a small value - machine epsilon). The second approach seem pragmatic and taken by scikit learn as well but happy to change that behaviour, either way, to be consistent. What's the view here?

@user1:
I haven't thought much about this, but following scikit-learn seems sensible. 

@author:
OK. Lets get this PR in and I will do next one with proposed changes to sMAPE (handle special case where y_test and y_pred are both 0 -> return 0 instead of nan) and MASE (special case where reference, in-sample mae is 0 -> return relatively large error instead of nan) this should make it consistent and follow the approach in scikit learn