Repository: sktime/sktime
Issue category: Feature Request

Issue:
[ENH] allow `coverage` to be specified in `evaluate` using interval forecats metrics
I am trying to evaluate the model. and one of the outputs that i need is y_pred_interval. For that i add the EmpiricalCoverage metric. It is calculating intervals, but with fixed coverage of 0.9. How to affect it and calculate y_pred_intervals with the different coverage? 

 Example code:

```python
evaluate(
    model,
    cv=cv,
    y=y_df.loc[index],
    X=x_df.loc[index],
    return_data=True,
    backend='loky',
    scoring=[MeanAbsolutePercentageError(), EmpiricalCoverage(), CRPS()],
)
```


Hints:
hm, indeed it appears it is not possible to set this - compared to `PinballLoss` where an alpha can be attached to the loss. It feels like an omission that should be fixed.
The fix might not be straightforward, and we would need to think about how the code should even look like.

Because the `coverage` is usually passed to `predict_interval` and not to the metric. The option to pass `alpha` to `PinballLoss` has always been a bit of a hack here.

Let's say a user wants the coverage of 0.9 and 0.8 intervals both, what would be a good way to pass it? Open design question!
I would go so far and classify this as a bug, since a user should reasonably expect to be able to pass the coverage or alpha at which the metric should be computed, for `evaluate`.
One solution I can see:

* every quantile metric should accept `alpha` as arg, but optional
* every interval metric should accept `coverage` as arg, but optional
* there is no further logic for now, this is only used in cases such as `evaluate` or tuning to allow a user to attach this information to the metric.
    * logic for this is already present in `evaluate`, namely in `_get_pred_args_from_metric`. It should therefore suffice to give all the metrics that do not have these yet parameters `alpha` and `coverage`.

The code above would then become:

```python
evaluate(
    model,
    cv=cv,
    y=y_df.loc[index],
    X=x_df.loc[index],
    return_data=True,
    backend='loky',
    scoring=[MeanAbsolutePercentageError(), EmpiricalCoverage(coverage=0.8), CRPS()],
)
```


So, as simpler recipe:

1. for every interval metric
2. add `coverage` as `__init__` arg, and write `self.coverage=coverage` in the init.
3. add the parameter to the documentation, this can be coped from `predict_interval`.

Same for `alpha` and `predict_quantiles`, and quantile metrics.

Patch:
diff --git a/sktime/performance_metrics/forecasting/probabilistic/_classes.py b/sktime/performance_metrics/forecasting/probabilistic/_classes.py
index 76405a316f9..961a3b274cd 100644
--- a/sktime/performance_metrics/forecasting/probabilistic/_classes.py
+++ b/sktime/performance_metrics/forecasting/probabilistic/_classes.py
@@ -358,6 +358,22 @@ def _check_alpha(self, alpha):
 
         return alpha
 
+    def _check_coverage(self, coverage):
+        """Check coverage input and coerce to np.ndarray."""
+        if coverage is None:
+            return None
+
+        if isinstance(coverage, float):
+            coverage = [coverage]
+
+        if not isinstance(coverage, np.ndarray):
+            coverage = np.asarray(coverage)
+
+        if not all((coverage > 0) & (coverage < 1)):
+            raise ValueError("Coverage must be between 0 and 1.")
+
+        return coverage
+
     def _handle_multioutput(self, loss, multioutput):
         """Handle output according to multioutput parameter.
 
@@ -417,14 +433,27 @@ class PinballLoss(_BaseProbaForecastingErrorMetric):
 
     Parameters
     ----------
-    multioutput : string "uniform_average" or "raw_values" determines how\
-        multioutput results will be treated.
+    multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
+            (n_outputs,), default='uniform_average'
+        Defines whether and how to aggregate metric for across variables.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
 
     score_average : bool, optional, default = True
         specifies whether scores for each quantile should be averaged.
 
-    alpha (optional) : float, list or np.ndarray, specifies what quantiles to \
-        evaluate metric at.
+        * If True, metric/loss is averaged over all quantiles present in ``y_pred``.
+        * If False, metric/loss is not averaged over quantiles.
+
+    alpha (optional) : float, list of float, or 1D array-like, default=None
+        quantiles to evaluate metric at.
+        Can be specified if no explicit quantiles are present in the direct use of
+        the metric, for instance in benchmarking via ``evaluate``, or tuning
+        via ``ForecastingGridSearchCV``.
 
     Examples
     --------
@@ -539,11 +568,27 @@ class EmpiricalCoverage(_BaseProbaForecastingErrorMetric):
 
     Parameters
     ----------
-    multioutput : string "uniform_average" or "raw_values" determines how\
-        multioutput results will be treated.
+    multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
+            (n_outputs,), default='uniform_average'
+        Defines whether and how to aggregate metric for across variables.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
 
     score_average : bool, optional, default = True
-        specifies whether scores for each quantile should be averaged.
+        specifies whether scores for each coverage value should be averaged.
+
+        * If True, metric/loss is averaged over all coverages present in ``y_pred``.
+        * If False, metric/loss is not averaged over coverages.
+
+    coverage (optional) : float, list of float, or 1D array-like, default=None
+        nominal coverage to evaluate metric at.
+        Can be specified if no explicit coverages are present in the direct use of
+        the metric, for instance in benchmarking via ``evaluate``, or tuning
+        via ``ForecastingGridSearchCV``.
     """
 
     _tags = {
@@ -551,9 +596,14 @@ class EmpiricalCoverage(_BaseProbaForecastingErrorMetric):
         "lower_is_better": False,
     }
 
-    def __init__(self, multioutput="uniform_average", score_average=True):
+    def __init__(
+        self, multioutput="uniform_average", score_average=True, coverage=None
+    ):
         self.score_average = score_average
         self.multioutput = multioutput
+        self.coverage = coverage
+        self._coverage = self._check_coverage(coverage)
+        self.metric_args = {"coverage": self._coverage}
         super().__init__(score_average=score_average, multioutput=multioutput)
 
     def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
@@ -598,7 +648,8 @@ def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
     def get_test_params(self):
         """Retrieve test parameters."""
         params1 = {}
-        return [params1]
+        params2 = {"coverage": 0.5}
+        return [params1, params2]
 
 
 class IntervalWidth(_BaseProbaForecastingErrorMetric):
@@ -606,11 +657,27 @@ class IntervalWidth(_BaseProbaForecastingErrorMetric):
 
     Parameters
     ----------
-    multioutput : string "uniform_average" or "raw_values" determines how\
-        multioutput results will be treated.
+    multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
+            (n_outputs,), default='uniform_average'
+        Defines whether and how to aggregate metric for across variables.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
 
     score_average : bool, optional, default = True
-        specifies whether scores for each quantile should be averaged.
+        specifies whether scores for each coverage value should be averaged.
+
+        * If True, metric/loss is averaged over all coverages present in ``y_pred``.
+        * If False, metric/loss is not averaged over coverages.
+
+    coverage (optional) : float, list of float, or 1D array-like, default=None
+        nominal coverage to evaluate metric at.
+        Can be specified if no explicit coverages are present in the direct use of
+        the metric, for instance in benchmarking via ``evaluate``, or tuning
+        via ``ForecastingGridSearchCV``.
     """
 
     _tags = {
@@ -618,9 +685,14 @@ class IntervalWidth(_BaseProbaForecastingErrorMetric):
         "lower_is_better": True,
     }
 
-    def __init__(self, multioutput="uniform_average", score_average=True):
+    def __init__(
+        self, multioutput="uniform_average", score_average=True, coverage=None
+    ):
         self.score_average = score_average
         self.multioutput = multioutput
+        self.coverage = coverage
+        self._coverage = self._check_coverage(coverage)
+        self.metric_args = {"coverage": self._coverage}
         super().__init__(score_average=score_average, multioutput=multioutput)
 
     def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
@@ -662,7 +734,8 @@ def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
     def get_test_params(self):
         """Retrieve test parameters."""
         params1 = {}
-        return [params1]
+        params2 = {"coverage": 0.5}
+        return [params1, params2]
 
 
 class ConstraintViolation(_BaseProbaForecastingErrorMetric):
@@ -670,11 +743,27 @@ class ConstraintViolation(_BaseProbaForecastingErrorMetric):
 
     Parameters
     ----------
-    multioutput : string "uniform_average" or "raw_values" determines how\
-        multioutput results will be treated.
+    multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
+            (n_outputs,), default='uniform_average'
+        Defines whether and how to aggregate metric for across variables.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
 
     score_average : bool, optional, default = True
-        specifies whether scores for each quantile should be averaged.
+        specifies whether scores for each coverage value should be averaged.
+
+        * If True, metric/loss is averaged over all coverages present in ``y_pred``.
+        * If False, metric/loss is not averaged over coverages.
+
+    coverage (optional) : float, list of float, or 1D array-like, default=None
+        nominal coverage to evaluate metric at.
+        Can be specified if no explicit coverages are present in the direct use of
+        the metric, for instance in benchmarking via ``evaluate``, or tuning
+        via ``ForecastingGridSearchCV``.
     """
 
     _tags = {
@@ -682,9 +771,14 @@ class ConstraintViolation(_BaseProbaForecastingErrorMetric):
         "lower_is_better": True,
     }
 
-    def __init__(self, multioutput="uniform_average", score_average=True):
+    def __init__(
+        self, multioutput="uniform_average", score_average=True, coverage=None
+    ):
         self.score_average = score_average
         self.multioutput = multioutput
+        self.coverage = coverage
+        self._coverage = self._check_coverage(coverage)
+        self.metric_args = {"coverage": self._coverage}
         super().__init__(score_average=score_average, multioutput=multioutput)
 
     def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
@@ -732,7 +826,8 @@ def _evaluate_by_index(self, y_true, y_pred, multioutput, **kwargs):
     def get_test_params(self):
         """Retrieve test parameters."""
         params1 = {}
-        return [params1]
+        params2 = {"coverage": 0.5}
+        return [params1, params2]
 
 
 PANDAS_DF_MTYPES = ["pd.DataFrame", "pd-multiindex", "pd_multiindex_hier"]
@@ -763,8 +858,15 @@ def evaluate(self, y_true, y_pred, multioutput=None, **kwargs):
         y_pred : return object of probabilistic predictition method scitype:y_pred
             must be at fh and for variables equal to those in y_true
 
-        multioutput : string "uniform_average" or "raw_values" determines how\
-            multioutput results will be treated.
+        multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
+                (n_outputs,), default='uniform_average'
+            Defines whether and how to aggregate metric for across variables.
+
+            * If 'uniform_average' (default), errors are mean-averaged across variables.
+            * If array-like, errors are weighted averaged across variables,
+            values as weights.
+            * If 'raw_values', does not average errors across variables,
+            columns are retained.
 
         Returns
         -------
@@ -849,14 +951,19 @@ class LogLoss(_BaseDistrForecastingMetric):
     multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
             (n_outputs,), default='uniform_average'
         Defines whether and how to aggregate metric for across variables.
-        If 'uniform_average' (default), errors are mean-averaged across variables.
-        If array-like, errors are weighted averaged across variables, values as weights.
-        If 'raw_values', does not average errors across variables, columns are retained.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
+
     multivariate : bool, optional, default=False
-        if True, behaves as multivariate log-loss
-        log-loss is computed for entire row, results one score per row
-        if False, is univariate log-loss
-        log-loss is computed per variable marginal, results in many scores per row
+
+        * if True, behaves as multivariate log-loss:
+          the log-loss is computed for entire row, results one score per row
+        * if False, is univariate log-loss:
+          the log-loss is computed per variable marginal, results in many scores per row
     """
 
     def __init__(self, multioutput="uniform_average", multivariate=False):
@@ -904,14 +1011,19 @@ class SquaredDistrLoss(_BaseDistrForecastingMetric):
     multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
             (n_outputs,), default='uniform_average'
         Defines whether and how to aggregate metric for across variables.
-        If 'uniform_average' (default), errors are mean-averaged across variables.
-        If array-like, errors are weighted averaged across variables, values as weights.
-        If 'raw_values', does not average errors across variables, columns are retained.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
+
     multivariate : bool, optional, default=False
-        if True, behaves as multivariate squared loss
-        squared loss is computed for entire row, results one score per row
-        if False, is univariate squared loss
-        squared loss is computed per variable marginal, results in many scores per row
+
+        * if True, behaves as multivariate squared loss:
+          the score is computed for entire row, results one score per row
+        * if False, is univariate squared loss:
+          the score is computed per variable marginal, results in many scores per row
     """
 
     def __init__(self, multioutput="uniform_average", multivariate=False):
@@ -957,14 +1069,19 @@ class CRPS(_BaseDistrForecastingMetric):
     multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
             (n_outputs,), default='uniform_average'
         Defines whether and how to aggregate metric for across variables.
-        If 'uniform_average' (default), errors are mean-averaged across variables.
-        If array-like, errors are weighted averaged across variables, values as weights.
-        If 'raw_values', does not average errors across variables, columns are retained.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
+
     multivariate : bool, optional, default=False
-        if True, behaves as multivariate CRPS (sum of scores)
-        CRPS is computed for entire row, results one score per row
-        if False, returns univariate CRPS, per variable
-        CRPS is computed per variable marginal, results in many scores per row
+
+        * if True, behaves as multivariate CRPS:
+          the score is computed for entire row, results one score per row
+        * if False, is univariate CRPS:
+          the score is computed per variable marginal, results in many scores per row
     """  # noqa: E501
 
     def __init__(self, multioutput="uniform_average", multivariate=False):
@@ -1020,14 +1137,19 @@ class AUCalibration(_BaseDistrForecastingMetric):
     multioutput : {'raw_values', 'uniform_average'} or array-like of shape \
             (n_outputs,), default='uniform_average'
         Defines whether and how to aggregate metric for across variables.
-        If 'uniform_average' (default), errors are mean-averaged across variables.
-        If array-like, errors are weighted averaged across variables, values as weights.
-        If 'raw_values', does not average errors across variables, columns are retained.
+
+        * If 'uniform_average' (default), errors are mean-averaged across variables.
+        * If array-like, errors are weighted averaged across variables,
+          values as weights.
+        * If 'raw_values', does not average errors across variables,
+          columns are retained.
+
     multivariate : bool, optional, default=False
-        if True, behaves as multivariate metric (sum of scores)
-        The metric is computed for entire row, results one score per row
-        if False, returns univariate metric, per variable
-        The metric is computed per variable marginal, results in many scores per row
+
+        * if True, behaves as multivariate metric (sum of scores):
+          the metric is computed for entire row, results one score per row
+        * if False, is univariate metric, per variable:
+          the metric is computed per variable marginal, results in many scores per row
     """  # noqa: E501
 
     def __init__(self, multioutput="uniform_average", multivariate=False):


Test patch:
diff --git a/sktime/forecasting/model_evaluation/tests/test_evaluate.py b/sktime/forecasting/model_evaluation/tests/test_evaluate.py
index ed0becaa66d..ccf1912bff8 100644
--- a/sktime/forecasting/model_evaluation/tests/test_evaluate.py
+++ b/sktime/forecasting/model_evaluation/tests/test_evaluate.py
@@ -39,7 +39,9 @@
 )
 from sktime.performance_metrics.forecasting.probabilistic import (
     CRPS,
+    ConstraintViolation,
     EmpiricalCoverage,
+    IntervalWidth,
     LogLoss,
     PinballLoss,
 )
@@ -54,6 +56,11 @@
 
 METRICS = [MeanAbsolutePercentageError(symmetric=True), MeanAbsoluteScaledError()]
 PROBA_METRICS = [CRPS(), EmpiricalCoverage(), LogLoss(), PinballLoss()]
+INTERVAL_METRICS_WITH_PARAMS = [
+    EmpiricalCoverage(coverage=0.95),
+    ConstraintViolation(coverage=[0.7, 0.8]),
+    IntervalWidth(coverage=[0.7, 0.8]),
+]
 
 # list of parallelization backends to test
 BACKENDS = _get_parallel_test_fixtures("estimator")
@@ -358,6 +365,34 @@ def test_evaluate_probabilistic(n_columns, scoring):
         pass
 
 
+@pytest.mark.skipif(
+    not run_test_for_class([evaluate] + INTERVAL_METRICS_WITH_PARAMS),
+    reason="run test only if softdeps are present and incrementally (if requested)",
+)
+@pytest.mark.parametrize("n_columns", [1, 2])
+@pytest.mark.parametrize("scoring", INTERVAL_METRICS_WITH_PARAMS)
+def test_evaluate_probabilistic_with_params(n_columns, scoring):
+    """Check that evaluate works with interval, quantile, and distribution forecasts."""
+    y = _make_series(n_columns=n_columns)
+
+    forecaster = NaiveForecaster()
+    cv = SlidingWindowSplitter()
+    out = evaluate(
+        forecaster,
+        cv,
+        y,
+        X=None,
+        scoring=scoring,
+        error_score="raise",
+        return_data=True,
+    )
+    scoring_coverage = scoring._coverage
+    assert "y_pred_interval" in out.columns
+    df_intervals = pd.concat(out["y_pred_interval"].to_list())
+    columns = {col[1] for col in df_intervals.columns if len(col) == 3}
+    assert all(coverage in columns for coverage in scoring_coverage)
+
+
 @pytest.mark.skipif(
     not run_test_for_class(evaluate),
     reason="run test only if softdeps are present and incrementally (if requested)",