Repository: sktime/sktime

Issue:
Add weights hyperparemeter to EnsembleForecaster API
**Is your feature request related to a problem? Please describe.**
EnsembleForecaster does not currently accept weights. With #1136  and #1139 being worked on it would be nice to optionally allow users to pass weights to `EnsembleForecaster`. 

This would give users an out-of-the-box way to use ainverse error based weighting strategy in forecasting ensembles. Users could also supply other types of weights (information criterion, etc).  

**Describe the solution you'd like**
An update to the `EnsembleForecaster` API to allow weights to be passed (number weights should equal number of forecasters passed). 

This would require altering the `__init__` to store the weights, adding a check (in `_fit`?), and then updating `_predict` to use versions of the `aggfunc` that allow weights (for examples of this see use of _weighted_percentile, _weighted_geometric_mean, and np.average in performance_metrics\_functions.py).


**Describe alternatives you've considered**
N/A

**Additional context**
N/A



Discussion:
Hm, how does this relate to the online weighted ensemble and the hedge forecasters?
Would be good if you could look at both in detail, to avoid duplication or interface inconsistency.
@fkiraly they are definately related. 

As a start, this would essentially be the "non-online" version of the `PredictionWeightedEnsemble` from the tutorial (which is built from the online ensemble). 

Since `OnlineEnsembleForecaster` inherits from `EnsembleForecaster` a second option (but not necessarily something we'd need to do) would be to also add a weights hyperparameter to the `OnlineEnsembleForecaster` that would initialize the weights based on user input (default would be to initialize as 1 / # of forecasters like currently done). 

I'd also like to improve the documentation in this area so its a bit more accessible, but have a few questions to verify my understanding. We can work on that separately. 
>I'd also like to improve the documentation in this area so its a bit more accessible, but have a few questions to verify my understanding. We can work on that separately.

Yes, indeed - especially the methodological documentation. I have to admit I'm not exactly sure which algorithms are implemented there exactly, and this needs to be clarified and improved in the docs.
Yep -- it is hard to figure that out from the documentation we have currently. I actually want take a crack at the user guide for forecasting to help make things more clear (and of course fix the docstrings as I go through that). 
Hi @RNKuhns I'd like to work on this 🚀 It's closely related to [#1082](https://github.com/alan-turing-institute/sktime/pull/1082) that I'm working on now.
@GuzalBulatova that works for me. @aiwalter and I have been talking about some options for the functionality. Maybe the three of us can find a quick time to talk. 

After digging into the online ensemblers per @fkiraly's comment above. I think that the current differences between that and `EnsembleForecaster` are currently that the online ensemblers use weights and update those weights (the online part). 

I think that it would be reasonable if the only difference was that the online ensembles update their weights (`EnsembleForecaster` would fix its weights based on the new `weights` hyperparameter).

Resolving patch:
diff --git a/sktime/forecasting/base/_meta.py b/sktime/forecasting/base/_meta.py
--- a/sktime/forecasting/base/_meta.py
+++ b/sktime/forecasting/base/_meta.py
@@ -70,6 +70,23 @@ def _fit_forecaster(forecaster, y, X, fh):
             for forecaster in forecasters
         )
 
+    def _update(self, y, X=None, update_params=True):
+        """Update fitted parameters.
+
+        Parameters
+        ----------
+        y : pd.Series
+        X : pd.DataFrame
+        update_params : bool, optional, default=True
+
+        Returns
+        -------
+        self : an instance of self.
+        """
+        for forecaster in self.forecasters_:
+            forecaster.update(y, X, update_params=update_params)
+        return self
+
     def _predict_forecasters(
         self, fh=None, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA
     ):
diff --git a/sktime/forecasting/compose/_column_ensemble.py b/sktime/forecasting/compose/_column_ensemble.py
--- a/sktime/forecasting/compose/_column_ensemble.py
+++ b/sktime/forecasting/compose/_column_ensemble.py
@@ -106,6 +106,23 @@ def _fit(self, y, X=None, fh=None):
 
         return self
 
+    def _update(self, y, X=None, update_params=True):
+        """Update fitted parameters.
+
+        Parameters
+        ----------
+        y : pd.Series
+        X : pd.DataFrame
+        update_params : bool, optional, default=True
+
+        Returns
+        -------
+        self : an instance of self.
+        """
+        for _, forecaster, index in self.forecasters_:
+            forecaster.update(y.iloc[:, index], X, update_params=update_params)
+        return self
+
     def _predict(self, fh=None, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA):
 
         y_pred = np.zeros((len(fh), len(self.forecasters_)))
diff --git a/sktime/forecasting/compose/_ensemble.py b/sktime/forecasting/compose/_ensemble.py
--- a/sktime/forecasting/compose/_ensemble.py
+++ b/sktime/forecasting/compose/_ensemble.py
@@ -1,13 +1,20 @@
 #!/usr/bin/env python3 -u
 # -*- coding: utf-8 -*-
-# copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
-"""Implements forecaster for creating forecasts from ensembles of other forecasters."""
+# copyright: sktime developers, BSD-3-Clause License (see LICENSE file).
+"""Implements ensemble forecasters.
 
-__author__ = ["Markus Löning"]
+Creates univariate (optionally weighted)
+combination of the predictions from underlying forecasts.
+"""
+
+__author__ = ["mloning", "GuzalBulatova"]
 __all__ = ["EnsembleForecaster"]
 
+import numpy as np
 import pandas as pd
 
+from scipy.stats import gmean
+from sklearn.utils.stats import _weighted_percentile
 from sktime.forecasting.base._base import DEFAULT_ALPHA
 from sktime.forecasting.base._meta import _HeterogenousEnsembleForecaster
 
@@ -30,6 +37,8 @@ class EnsembleForecaster(_HeterogenousEnsembleForecaster):
         -1 means using all processors.
     aggfunc : str, {'mean', 'median', 'min', 'max'}, default='mean'
         The function to aggregate prediction from individual forecasters.
+    weights : list of floats
+        Weights to apply in aggregation.
 
     Examples
     --------
@@ -40,7 +49,8 @@ class EnsembleForecaster(_HeterogenousEnsembleForecaster):
     >>> y = load_airline()
     >>> forecasters = [("trend", PolynomialTrendForecaster()),\
                         ("naive", NaiveForecaster())]
-    >>> forecaster = EnsembleForecaster(forecasters=forecasters, n_jobs=2)
+    >>> forecaster = EnsembleForecaster(forecasters=forecasters,\
+                                        aggfunc="mean", weights=[1, 10])
     >>> forecaster.fit(y=y, X=None, fh=[1,2,3])
     EnsembleForecaster(...)
     >>> y_pred = forecaster.predict()
@@ -48,14 +58,15 @@ class EnsembleForecaster(_HeterogenousEnsembleForecaster):
 
     _required_parameters = ["forecasters"]
     _tags = {
-        "univariate-only": True,
+        "univariate-only": False,
         "requires-fh-in-fit": False,
         "handles-missing-data": False,
     }
 
-    def __init__(self, forecasters, n_jobs=None, aggfunc="mean"):
+    def __init__(self, forecasters, n_jobs=None, aggfunc="mean", weights=None):
         super(EnsembleForecaster, self).__init__(forecasters=forecasters, n_jobs=n_jobs)
         self.aggfunc = aggfunc
+        self.weights = weights
 
     def _fit(self, y, X=None, fh=None):
         """Fit to training data.
@@ -77,23 +88,6 @@ def _fit(self, y, X=None, fh=None):
         self._fit_forecasters(forecasters, y, X, fh)
         return self
 
-    def _update(self, y, X=None, update_params=True):
-        """Update fitted parameters.
-
-        Parameters
-        ----------
-        y : pd.Series
-        X : pd.DataFrame
-        update_params : bool, optional, default=True
-
-        Returns
-        -------
-        self : an instance of self.
-        """
-        for forecaster in self.forecasters_:
-            forecaster.update(y, X, update_params=update_params)
-        return self
-
     def _predict(self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA):
         """Return the predicted reduction.
 
@@ -113,16 +107,80 @@ def _predict(self, fh, X=None, return_pred_int=False, alpha=DEFAULT_ALPHA):
             raise NotImplementedError()
 
         y_pred = pd.concat(self._predict_forecasters(fh, X), axis=1)
+        y_pred = _aggregate(y=y_pred, aggfunc=self.aggfunc, weights=self.weights)
+
+        return y_pred
+
+
+def _aggregate(y, aggfunc, weights):
+    """Apply aggregation function by row.
 
-        valid_aggfuncs = ("median", "mean", "min", "max")
-        if self.aggfunc not in valid_aggfuncs:
-            raise ValueError(f"Invalid `aggfunc`. Please use one of {valid_aggfuncs}")
-
-        if self.aggfunc == "median":
-            return y_pred.median(axis=1)
-        elif self.aggfunc == "min":
-            return y_pred.min(axis=1)
-        elif self.aggfunc == "max":
-            return y_pred.max(axis=1)
-        else:
-            return y_pred.mean(axis=1)
+    Parameters
+    ----------
+    y : pd.DataFrame
+        Multivariate series to transform.
+    aggfunc : str
+        Aggregation function used for transformation.
+    weights : list of floats
+        Weights to apply in aggregation.
+
+    Returns
+    -------
+    y_agg: pd.Series
+        Transformed univariate series.
+    """
+    if weights is None:
+        aggfunc = _check_aggfunc(aggfunc, weighted=False)
+        y_agg = aggfunc(y, axis=1)
+    else:
+        aggfunc = _check_aggfunc(aggfunc, weighted=True)
+        y_agg = aggfunc(y, axis=1, weights=np.array(weights))
+
+    return pd.Series(y_agg, index=y.index)
+
+
+def _check_aggfunc(aggfunc, weighted=False):
+    _weighted = "weighted" if weighted else "unweighted"
+    valid_aggfuncs = {
+        "mean": {"unweighted": np.mean, "weighted": np.average},
+        "median": {"unweighted": np.median, "weighted": _weighted_median},
+        "min": {"unweighted": np.min, "weighted": _weighted_min},
+        "max": {"unweighted": np.max, "weighted": _weighted_max},
+        "gmean": {"unweighted": gmean, "weighted": gmean},
+    }
+    if aggfunc not in valid_aggfuncs.keys():
+        raise ValueError("Aggregation function %s not recognized." % aggfunc)
+    return valid_aggfuncs[aggfunc][_weighted]
+
+
+def _weighted_median(y, axis=1, weights=None):
+    w_median = np.apply_along_axis(
+        func1d=_weighted_percentile,
+        axis=axis,
+        arr=y.values,
+        sample_weight=weights,
+        percentile=50,
+    )
+    return w_median
+
+
+def _weighted_min(y, axis=1, weights=None):
+    w_min = np.apply_along_axis(
+        func1d=_weighted_percentile,
+        axis=axis,
+        arr=y.values,
+        sample_weight=weights,
+        percentile=0,
+    )
+    return w_min
+
+
+def _weighted_max(y, axis=1, weights=None):
+    w_max = np.apply_along_axis(
+        func1d=_weighted_percentile,
+        axis=axis,
+        arr=y.values,
+        sample_weight=weights,
+        percentile=100,
+    )
+    return w_max


Test patch:
diff --git a/sktime/forecasting/compose/tests/test_ensemble.py b/sktime/forecasting/compose/tests/test_ensemble.py
new file mode 100644
--- /dev/null
+++ b/sktime/forecasting/compose/tests/test_ensemble.py
@@ -0,0 +1,116 @@
+#!/usr/bin/env python3 -u
+# -*- coding: utf-8 -*-
+"""copyright: sktime developers, BSD-3-Clause License (see LICENSE file)."""
+
+__author__ = ["GuzalBulatova"]
+
+import numpy as np
+import pandas as pd
+import pytest
+import sys
+
+from scipy.stats import gmean
+from sktime.forecasting.compose import EnsembleForecaster
+from sktime.forecasting.exp_smoothing import ExponentialSmoothing
+from sktime.forecasting.naive import NaiveForecaster
+from sktime.forecasting.trend import PolynomialTrendForecaster
+
+from sktime.utils._testing.forecasting import make_forecasting_problem
+
+
+@pytest.mark.parametrize(
+    "forecasters",
+    [
+        [("trend", PolynomialTrendForecaster()), ("naive", NaiveForecaster())],
+        [("trend", PolynomialTrendForecaster()), ("ses", ExponentialSmoothing())],
+    ],
+)
+def test_avg_mean(forecasters):
+    """Assert `mean` aggfunc returns the same values as `average` with equal weights."""
+    y = make_forecasting_problem()
+    forecaster = EnsembleForecaster(forecasters)
+    forecaster.fit(y, fh=[1, 2, 3])
+    mean_pred = forecaster.predict()
+
+    forecaster_1 = EnsembleForecaster(forecasters, aggfunc="mean", weights=[1, 1])
+    forecaster_1.fit(y, fh=[1, 2, 3])
+    avg_pred = forecaster_1.predict()
+
+    pd.testing.assert_series_equal(mean_pred, avg_pred)
+
+
+@pytest.mark.parametrize("aggfunc", ["median", "mean", "min", "max", "gmean"])
+@pytest.mark.parametrize(
+    "forecasters",
+    [
+        [("trend", PolynomialTrendForecaster()), ("naive", NaiveForecaster())],
+    ],
+)
+def test_aggregation_unweighted(forecasters, aggfunc):
+    """Assert aggfunc returns the correct values."""
+    y = make_forecasting_problem()
+    forecaster = EnsembleForecaster(forecasters=forecasters, aggfunc=aggfunc)
+    forecaster.fit(y, fh=[1, 2, 3])
+    actual_pred = forecaster.predict()
+
+    predictions = []
+    if aggfunc == "gmean":
+        aggfunc = gmean
+    for _, forecaster in forecasters:
+        f = forecaster
+        f.fit(y)
+        f_pred = f.predict(fh=[1, 2, 3])
+        predictions.append(f_pred)
+    predictions = pd.DataFrame(predictions)
+    expected_pred = predictions.apply(func=aggfunc, axis=0)
+
+    pd.testing.assert_series_equal(actual_pred, expected_pred)
+
+
+@pytest.mark.parametrize("aggfunc", ["mean", "gmean"])
+@pytest.mark.parametrize("weights", [[1.44, 1.2]])
+@pytest.mark.parametrize(
+    "forecasters",
+    [
+        [("trend", PolynomialTrendForecaster()), ("naive", NaiveForecaster())],
+    ],
+)
+@pytest.mark.skipif(sys.version_info < (3, 7), reason="requires python3.7 or higher")
+def test_aggregation_weighted(forecasters, aggfunc, weights):
+    """Assert weighted aggfunc returns the correct values."""
+    y = make_forecasting_problem()
+    forecaster = EnsembleForecaster(
+        forecasters=forecasters, aggfunc=aggfunc, weights=weights
+    )
+    forecaster.fit(y, fh=[1, 2, 3])
+    actual_pred = forecaster.predict()
+
+    predictions = []
+    for _, forecaster in forecasters:
+        f = forecaster
+        f.fit(y)
+        f_pred = f.predict(fh=[1, 2, 3])
+        predictions.append(f_pred)
+
+    predictions = pd.DataFrame(predictions)
+    if aggfunc == "mean":
+        func = np.average
+    else:
+        func = gmean
+    expected_pred = predictions.apply(func=func, axis=0, weights=weights)
+
+    pd.testing.assert_series_equal(actual_pred, expected_pred)
+
+
+@pytest.mark.parametrize("aggfunc", ["miin", "maximum", ""])
+@pytest.mark.parametrize(
+    "forecasters",
+    [[("trend", PolynomialTrendForecaster()), ("naive", NaiveForecaster())]],
+)
+def test_invalid_aggfuncs(forecasters, aggfunc):
+    """Check if invalid aggregation functions return Error."""
+    y = make_forecasting_problem()
+    forecaster = EnsembleForecaster(forecasters=forecasters, aggfunc=aggfunc)
+    forecaster.fit(y, fh=[1, 2])
+    with pytest.raises(ValueError, match=r"not recognized"):
+        forecaster.predict()