Repository: sktime/sktime

Issue:
[ENH] MAPE/sMAPE `symmetric` flag default should be switched
Current parameter defaults of MAPE, MdAPE metric classes etc heavily frustrates user expectations, since the default parameter settings actually produce sMAPE and sMdAPE etc (as pe common definition) and not MAPE, MdAPE (as per common definition).

This should be switched around, so default invocation of MAPE metric classes indeed give MAPE (as commonly defined).

The switch is easy - just flip the default to `symmetric=False` - but we will need to add a deprecation message since this is part of a widely used public interface.

See the [documentation on deprecation policy](https://www.sktime.org/en/stable/developer_guide/deprecation.html), we're probably looking at deprecation in 0.11.0, and removal in 0.12.0.

I.e., adding a "will be deprecated" warning now, then a "will be removed" warning in 0.11.0, then removing in 0.12.0.

Good first issue for someone to get involved with `sktime`.

---

from discussions, @k7e7n7t:

https://github.com/alan-turing-institute/sktime/discussions/1508

> I ran mean_absolute_percentage_error in cell 41 of 01-forecasting.ipynb and got 0.145427686270316.
> I then ran MeanAbsolutePercentageError class with the default symmetric = False (which make it asymmetric just like MAPE I assume) but I get  0.13189432350948402 which I would expect from sMAPE, but not with the flag set to False.  If I set symmetric = True it does match.  This seems backwards to me.  What am I missing?


Discussion:
I don't think it's a real bug. The issue is that default is `symmetric=True`. Have a look at the docs:
https://www.sktime.org/en/stable/api_reference/auto_generated/sktime.performance_metrics.forecasting.MedianAbsolutePercentageError.html

This is not very intuitive. Users expect that MAPE is MAPE and not sMAPE... 
e.g. Have a look at the forecasting metrics code here: 
https://github.com/ltsaprounis/dsf-ts-forecasting/blob/main/notebooks/02-forecasting-metrics.ipynb

MAPE and sMAPE have the expected behaviour when you manually specify `symmetric`. 

I think this needs to change, but we'll have back-compatibility issues if/when we change it. I would also prefer a separate sMAPE class to avoid things like that in the future. 

Tested it on the code in main: 
```python
from sktime.performance_metrics.forecasting import MeanAbsolutePercentageError
import numpy as np

y_forecast = np.array([10, 10, 10])
y_actual_1 = np.array([8, 8, 8])
y_actual_2 = np.array([12, 12, 12])


mape = MeanAbsolutePercentageError(symmetric=False)
print("This should be MAPE:")
print(f"forecast 1: {mape(y_actual_1, y_forecast): 0.2f}")
print(f"forecast 2: {mape(y_actual_2, y_forecast): 0.2f}")

print("\nThis should be sMAPE:")
smape = MeanAbsolutePercentageError(symmetric=True)
print(f"forecast 1: {smape(y_actual_1, y_forecast): 0.2f}")
print(f"forecast 2: {smape(y_actual_2, y_forecast): 0.2f}")

print("\nsimple NUMPY implementation:")

def calc_mape(actual, fcast): 
    return np.mean(np.abs(actual-fcast)/actual)

def calc_smape(actual, fcast): 
    return 2 * np.mean(np.abs((actual-fcast)/(actual+fcast)))

print("\nThis should be MAPE:")
print(f"forecast 1: {calc_mape(y_actual_1, y_forecast): 0.2f}")
print(f"forecast 2: {calc_mape(y_actual_2, y_forecast): 0.2f}")

print("\nThis should be sMAPE:")
print(f"forecast 1: {calc_smape(y_actual_1, y_forecast): 0.2f}")
print(f"forecast 2: {calc_smape(y_actual_2, y_forecast): 0.2f}")
```
ah, yes! Makes sense, @ltsaprounis.
It's not a bug since the implementation agrees with the specification.

But it is a quite heavy frustration of user expectations that, if you write `MeanAbsolutePercentageError()`, you do *not actually get the MAPE* as commonly defined.

I think we need to change the default to `True`, and that needs deprecation since it is part of the public interface.

I'll change this issue to a ticket to do this, if we are all agreed.

FYI @k7e7n7t.

Resolving patch:
diff --git a/sktime/forecasting/model_selection/_tune.py b/sktime/forecasting/model_selection/_tune.py
--- a/sktime/forecasting/model_selection/_tune.py
+++ b/sktime/forecasting/model_selection/_tune.py
@@ -36,6 +36,7 @@ def __init__(
         strategy="refit",
         n_jobs=None,
         pre_dispatch=None,
+        backend="loky",
         refit=False,
         scoring=None,
         verbose=0,
@@ -47,6 +48,7 @@ def __init__(
         self.strategy = strategy
         self.n_jobs = n_jobs
         self.pre_dispatch = pre_dispatch
+        self.backend = backend
         self.refit = refit
         self.scoring = scoring
         self.verbose = verbose
@@ -194,7 +196,9 @@ def _fit(self, y, X=None, fh=None, **fit_params):
         scoring = check_scoring(self.scoring)
         scoring_name = f"test_{scoring.name}"
 
-        parallel = Parallel(n_jobs=self.n_jobs, pre_dispatch=self.pre_dispatch)
+        parallel = Parallel(
+            n_jobs=self.n_jobs, pre_dispatch=self.pre_dispatch, backend=self.backend
+        )
 
         def _fit_and_score(params):
             # Clone forecaster.
@@ -336,6 +340,9 @@ class ForecastingGridSearchCV(BaseGridSearch):
     error_score: numeric value or the str 'raise', optional (default=np.nan)
         The test score returned when a forecaster fails to be fitted.
     return_train_score: bool, optional (default=False)
+    backend: str, optional (default="loky")
+        Specify the parallelisation backend implementation in joblib, where
+        "loky" is used by default.
 
     Attributes
     ----------
@@ -439,6 +446,7 @@ def __init__(
         verbose=0,
         return_n_best_forecasters=1,
         pre_dispatch="2*n_jobs",
+        backend="loky",
     ):
         super(ForecastingGridSearchCV, self).__init__(
             forecaster=forecaster,
@@ -450,6 +458,7 @@ def __init__(
             verbose=verbose,
             return_n_best_forecasters=return_n_best_forecasters,
             pre_dispatch=pre_dispatch,
+            backend=backend,
         )
         self.param_grid = param_grid
 
@@ -529,6 +538,9 @@ class ForecastingRandomizedSearchCV(BaseGridSearch):
         Pass an int for reproducible output across multiple
         function calls.
     pre_dispatch: str, optional (default='2*n_jobs')
+    backend: str, optional (default="loky")
+        Specify the parallelisation backend implementation in joblib, where
+        "loky" is used by default.
 
     Attributes
     ----------
@@ -564,6 +576,7 @@ def __init__(
         return_n_best_forecasters=1,
         random_state=None,
         pre_dispatch="2*n_jobs",
+        backend="loky",
     ):
         super(ForecastingRandomizedSearchCV, self).__init__(
             forecaster=forecaster,
@@ -575,6 +588,7 @@ def __init__(
             verbose=verbose,
             return_n_best_forecasters=return_n_best_forecasters,
             pre_dispatch=pre_dispatch,
+            backend=backend,
         )
         self.param_distributions = param_distributions
         self.n_iter = n_iter
diff --git a/sktime/performance_metrics/forecasting/_functions.py b/sktime/performance_metrics/forecasting/_functions.py
--- a/sktime/performance_metrics/forecasting/_functions.py
+++ b/sktime/performance_metrics/forecasting/_functions.py
@@ -8,6 +8,8 @@
 the lower the better.
 """
 
+from warnings import warn
+
 import numpy as np
 from scipy.stats import gmean
 from sklearn.metrics import mean_absolute_error as _mean_absolute_error
@@ -1444,6 +1446,8 @@ def geometric_mean_squared_error(
     return np.average(output_errors, weights=multioutput)
 
 
+# TODO: change default to symmetric=False in v0.12.0, edit doc string,
+# and remove warning
 def mean_absolute_percentage_error(
     y_true,
     y_pred,
@@ -1536,6 +1540,11 @@ def mean_absolute_percentage_error(
     >>> mean_absolute_percentage_error(y_true, y_pred, multioutput=[0.3, 0.7])
     0.5668686868686869
     """
+    warn(
+        "In the percentage error metric functions the default argument "
+        "symmetric=True is changing to symmetric=False in v0.12.0.",
+        FutureWarning,
+    )
     _, y_true, y_pred, multioutput = _check_reg_targets(y_true, y_pred, multioutput)
     if horizon_weight is not None:
         check_consistent_length(y_true, horizon_weight)
@@ -1556,6 +1565,8 @@ def mean_absolute_percentage_error(
     return np.average(output_errors, weights=multioutput)
 
 
+# TODO: change default to symmetric=False in v0.12.0, edit doc string,
+# and remove warning
 def median_absolute_percentage_error(
     y_true,
     y_pred,
@@ -1652,6 +1663,11 @@ def median_absolute_percentage_error(
     >>> median_absolute_percentage_error(y_true, y_pred, multioutput=[0.3, 0.7])
     0.5066666666666666
     """
+    warn(
+        "In the percentage error metric functions the default argument "
+        "symmetric=True is changing to symmetric=False in v0.12.0.",
+        FutureWarning,
+    )
     _, y_true, y_pred, multioutput = _check_reg_targets(y_true, y_pred, multioutput)
     if horizon_weight is None:
         output_errors = np.median(
@@ -1674,6 +1690,8 @@ def median_absolute_percentage_error(
     return np.average(output_errors, weights=multioutput)
 
 
+# TODO: change default to symmetric=False in v0.12.0, edit doc string,
+# and remove warning
 def mean_squared_percentage_error(
     y_true,
     y_pred,
@@ -1725,7 +1743,7 @@ def mean_squared_percentage_error(
         If True, returns root mean squared error (RMSPE)
         If False, returns mean squared error (MSPE)
 
-    symmetric : bool, default=False
+    symmetric : bool, default=True
         Calculates symmetric version of metric if True.
 
     Returns
@@ -1778,6 +1796,11 @@ def mean_squared_percentage_error(
     square_root=True, symmetric=False)
     0.7504665536595034
     """
+    warn(
+        "In the percentage error metric functions the default argument "
+        "symmetric=True is changing to symmetric=False in v0.12.0.",
+        FutureWarning,
+    )
     _, y_true, y_pred, multioutput = _check_reg_targets(y_true, y_pred, multioutput)
     if horizon_weight is not None:
         check_consistent_length(y_true, horizon_weight)
@@ -1801,6 +1824,8 @@ def mean_squared_percentage_error(
     return np.average(output_errors, weights=multioutput)
 
 
+# TODO: change default to symmetric=False in v0.12.0, edit doc string,
+# and remove warning
 def median_squared_percentage_error(
     y_true,
     y_pred,
@@ -1856,7 +1881,7 @@ def median_squared_percentage_error(
         If True, returns root mean squared error (RMSPE)
         If False, returns mean squared error (MSPE)
 
-    symmetric : bool, default=False
+    symmetric : bool, default=True
         Calculates symmetric version of metric if True.
 
     Returns
@@ -1910,6 +1935,11 @@ def median_squared_percentage_error(
     square_root=True, symmetric=False)
     0.7428571428571428
     """
+    warn(
+        "In the percentage error metric functions the default argument "
+        "symmetric=True is changing to symmetric=False in v0.12.0.",
+        FutureWarning,
+    )
     _, y_true, y_pred, multioutput = _check_reg_targets(y_true, y_pred, multioutput)
     perc_err = _percentage_error(y_true, y_pred, symmetric=symmetric)
     if horizon_weight is None:


Test patch: