Repository: sktime/sktime

Issue:
[BUG] StatsForecastAutoCES: predict() got an unexpected keyword argument 'level'
**Describe the bug**

StatsForecastAutoCES model complains about `TypeError: predict_in_sample() got an unexpected keyword argument 'level'` when `.predict()` is called after `.fit()`


**To Reproduce**
```python
from sktime.forecasting.statsforecast import StatsForecastAutoCES
from sktime.datasets import load_airline

y = load_airline()

forecaster = StatsForecastAutoCES(season_length=12)
forecaster.fit(y)

forecaster.predict(fh=[1,2,3])

```

**Additional context**

```
Traceback (most recent call last):
  File "/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-17-4d83c1149447>", line 1, in <module>
    forecaster.predict(fh=[1,2,3])
  File "/lib/python3.9/site-packages/sktime/forecasting/base/_base.py", line 404, in predict
    y_pred = self._predict(fh=fh, X=X_inner)
  File "/lib/python3.9/site-packages/sktime/forecasting/base/adapters/_generalised_statsforecast.py", line 192, in _predict
    out_of_sample_point_predictions = self._predict_in_or_out_of_sample(
  File "/lib/python3.9/site-packages/sktime/forecasting/base/adapters/_generalised_statsforecast.py", line 92, in _predict_in_or_out_of_sample
    predictions = self._forecaster.predict(
TypeError: predict() got an unexpected keyword argument 'level'
```

**Versions**
statsforecast 1.4.0 version
M1 Mac chip.

@yarnabrina 



Discussion:
Hi @arnaujc91, it's not really a bug. These models were added in `sktime` in May I think, and since v1.5.0 `statsforecast` support `level` arguments for `AutoCES`. You can see it in the [release notes](https://github.com/Nixtla/statsforecast/releases/tag/v1.5.0), and this is the relevant [PR](https://github.com/Nixtla/statsforecast/pull/394).
@fkiraly should we update the lower bound to `1.5.0` from `0.5.2` (released more than a year ago)?
Ok i see, i have v. 1.4.0 and there is not working, that is a problem actually because when you use, for example the [ForecastingGridSearchCV](https://www.sktime.net/en/latest/api_reference/auto_generated/sktime.forecasting.model_selection.ForecastingGridSearchCV.html) class, it automatically uses the method `predict` and the error shows up. So somehow would be great if an error shows up asking to update the version of statsforecast or sth, i could not figure out why this was happening, if it was because of your code or statsforecast code.
how about we change the adapter to check whether the level arg is present, and only pass it then?

Some people will have older `statsforecast` versions, e.g., since private pypi mirrors may lag behind newer releases.
alternatively, there is no need to raise the lower bound for `statsforecast` if this is estiator specific - we can simply add the `1.5.0` bound to the estimator. Although it does come from the adapter, so perhaps my first suggestion is better?
Sure, i think we could do a try/catch exception and then depending if the error is thrown we pass it or not. I guess that would work.
agree, @arnaujc91 - except that I always prefer explicit condition checking above try/catch, that's better style.

In this case, it would be via `inspect.signature`.

Would you like to make a PR?
Assuming @yarnabrina agrees to this change proposal to the adapter.
The offending line is the one that passes `level`, as in your traceback.

The one thing to check is that it is indeed a named argument rather than a kwarg that's unwrapped somewhere.
Sounds good! I had no idea about the existence of [inspect.signature](https://docs.python.org/3/library/inspect.html), thanks for sharing!

We can go for that one then
> Sounds good! I had no idea about the existence of [inspect.signature](https://docs.python.org/3/library/inspect.html), thanks for sharing!

It's really useful if you frequently write interfaces to inconsistent patterns...
> Assuming @yarnabrina agrees to this change proposal to the adapter.

Unfortunately, I don't. I disagree with first suggestion, and not very enthusiastic about second. I'll give my reasoning below.

Whether `level` exists or not will change the tags (e.g. `capability:pred_int`) as well. Suggestion 1 will then require presence of  `statsforecast` in init to update tags dynamically, if I'm not wrong. Do we want that?

I don't have anything against suggestion 2, except I am not sure all other statsforecast estimators in sktime support all other versions between 0.5.2 and 1.4.0, by which I mean suporting all features that sktime is using them for. Also, there has been change in major version from 0 to 1, and I'll prefer to update the lower bound.

My suggestion is to update lower bound now in `pyproject.toml` (one time activity) and specify in estimator dependency as well. In future if new estimators are added that were absent in previous versions, we'll specify as such. But I'm unclear what to do if the behaviour changes for existing estimators in future.
> Suggestion 1 will then require presence of statsforecast in init to update tags dynamically, if I'm not wrong. Do we want that?

If it happens after `super.__init__`, it's fine, since the `super.__init__` call raises the exception "statsforecast not present".

More generally, I think the adapter should be able to cope with statsforecast estimators that do not have the `level` argument. It claims to be a general adapter, so it should cover both cases of estimators with and without prediction interval capability.

> My suggestion is to update lower bound now

That would not deal with a potential estimator that is added later that doesn't have `level`. That could happen, no?
From the release note linked above:

> [FEAT] AutoCES prediction intervals in https://github.com/Nixtla/statsforecast/pull/394 (now StatsForecast is fully probabilistic)

I doubt Nixtla will want to change this, and I think they will maintain same API.

But it's better to be safe (and I get ehe general argument), so @arnaujc91 can continue with the `inspect` check as discussed above.
Ok - but please contradict me if you still genuinely disagree, @yarnabrina!
We should optimally go with a consensus after convincing rather than you being "talked into" an option...


Resolving patch:
diff --git a/sktime/forecasting/arch.py b/sktime/forecasting/arch.py
--- a/sktime/forecasting/arch.py
+++ b/sktime/forecasting/arch.py
@@ -48,14 +48,13 @@ def __init__(
 
         super().__init__()
 
-    def _instantiate_model(self):
-        # import inside method to avoid hard dependency
+    def _get_statsforecast_class(self):
         from statsforecast.models import GARCH as _GARCH
 
-        return _GARCH(
-            p=self.p,
-            q=self.q,
-        )
+        return _GARCH
+
+    def _get_statsforecast_params(self):
+        return {"p": self.p, "q": self.q}
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
@@ -110,13 +109,13 @@ def __init__(
 
         super().__init__()
 
-    def _instantiate_model(self):
-        # import inside method to avoid hard dependency
+    def _get_statsforecast_class(self):
         from statsforecast.models import ARCH as _ARCH
 
-        return _ARCH(
-            p=self.p,
-        )
+        return _ARCH
+
+    def _get_statsforecast_params(self):
+        return {"p": self.p}
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
diff --git a/sktime/forecasting/base/adapters/_generalised_statsforecast.py b/sktime/forecasting/base/adapters/_generalised_statsforecast.py
--- a/sktime/forecasting/base/adapters/_generalised_statsforecast.py
+++ b/sktime/forecasting/base/adapters/_generalised_statsforecast.py
@@ -1,5 +1,9 @@
 # copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
 """Implements adapter for StatsForecast models."""
+from inspect import signature
+from typing import Dict
+from warnings import warn
+
 import pandas
 
 from sktime.forecasting.base import BaseForecaster
@@ -27,10 +31,48 @@ def __init__(self):
         super().__init__()
 
         self._forecaster = None
+        pred_supported = self._check_supports_pred_int()
+        self._support_pred_int_in_sample = pred_supported["int_in_sample"]
+        self._support_pred_int = pred_supported["int"]
 
-    def _instantiate_model(self):
+        self.set_tags(
+            **{"capability:pred_int:insample": self._support_pred_int_in_sample}
+        )
+        self.set_tags(**{"capability:pred_int": self._support_pred_int})
+
+    def _get_statsforecast_class(self):
         raise NotImplementedError("abstract method")
 
+    def _get_statsforecast_params(self):
+        return self.get_params()
+
+    def _get_init_statsforecast_params(self):
+        statsforecast_class = self._get_statsforecast_class()
+        return list(signature(statsforecast_class.__init__).parameters.keys())
+
+    def _get_validated_statsforecast_params(self):
+        sktime_params = self._get_statsforecast_params()
+        sktime_default_params = self.get_param_defaults().keys()
+        statsforecast_params = self._get_init_statsforecast_params()
+
+        for sktime_param in sktime_params.keys():
+            if sktime_param not in statsforecast_params:
+                sktime_params.pop(sktime_param)
+                if sktime_param not in sktime_default_params:
+                    warn(
+                        f"Keyword argument '{sktime_param}' will be omitted as it is"
+                        f" not found in the __init__ method "
+                        f"from {self._get_statsforecast_class()}. "
+                        f"Check your statsforecast version"
+                        f"to find out the right API parameters."
+                    )
+        return sktime_params
+
+    def _instantiate_model(self):
+        cls = self._get_statsforecast_class()
+        params = self._get_validated_statsforecast_params()
+        return cls(**params)
+
     def _fit(self, y, X, fh):
         """Fit forecaster to training data.
 
@@ -86,12 +128,20 @@ def _predict_in_or_out_of_sample(
         level_arguments = None if levels is None else [100 * level for level in levels]
 
         if fh_type == "in-sample":
-            predictions = self._forecaster.predict_in_sample(level=level_arguments)
+            predict_method = self._forecaster.predict_in_sample
+            # Before v1.5.0 (from statsforecast) not all foreasters
+            # have a "level" keyword argument in `predict_in_sample`
+            level_kw = (
+                {"level": level_arguments} if self._support_pred_int_in_sample else {}
+            )
+            predictions = predict_method(**level_kw)
             point_predictions = predictions["fitted"]
         elif fh_type == "out-of-sample":
-            predictions = self._forecaster.predict(
-                maximum_forecast_horizon, X=X, level=level_arguments
-            )
+            predict_method = self._forecaster.predict
+            # Before v1.5.0 (from statsforecast) not all foreasters
+            # have a "level" keyword argument in `predict`
+            level_kw = {"level": level_arguments} if self._support_pred_int else {}
+            predictions = predict_method(maximum_forecast_horizon, X=X, **level_kw)
             point_predictions = predictions["mean"]
 
         if isinstance(point_predictions, pandas.Series):
@@ -268,6 +318,61 @@ def _predict_interval(self, fh, X, coverage, legacy_interface=True):
 
         return final_interval_predictions
 
+    def _check_supports_pred_int(self) -> Dict[str, bool]:
+        """
+        Check if prediction intervals will work with forecaster.
+
+        Check if `level` argument is available in `predict_in_sample` and
+        `predict` methods from the `statsforecast` forecaster.
+        A tuple of booleans (`support_pred_int_in_sample`, `support_pred_int`)
+        is returned where the user is informed which of the two, if any,
+        support interval predictions.
+
+         Furthermore, will throw a warning to let the user know that he should consider
+         upgrading statsforecast version as both or one of the two methods might
+         not be able to produce confidence intervals.
+
+        Returns
+        -------
+        Dict of bool
+            A dict containing two boolean values:
+            - `support_pred_int_in_sample`: True if prediction intervals are supported
+              in `predict_in_sample`, False otherwise.
+            - `support_pred_int`: True if prediction intervals are supported
+              in `predict`, False otherwise.
+        """
+        statsforecast_class = self._get_statsforecast_class()
+        if (
+            "level"
+            not in signature(statsforecast_class.predict_in_sample).parameters.keys()
+        ):
+            support_pred_int_in_sample = False
+            import statsforecast
+
+            warn(
+                f" {statsforecast_class.__name__} from "
+                f"statsforecast v{statsforecast.__version__} "
+                f"does not support prediction of intervals in `predict_in_sample`. "
+                f"Consider upgrading to a newer version."
+            )
+        else:
+            support_pred_int_in_sample = True
+
+        if "level" not in signature(statsforecast_class.predict).parameters.keys():
+            support_pred_int = False
+            import statsforecast
+
+            warn(
+                f" {statsforecast_class.__name__} from "
+                f"statsforecast v{statsforecast.__version__} "
+                f"does not support prediction of intervals in `predict`. "
+                f"Consider upgrading to a newer version."
+            )
+        else:
+            support_pred_int = True
+
+        return {"int_in_sample": support_pred_int_in_sample, "int": support_pred_int}
+
 
 class StatsForecastBackAdapter:
     """StatsForecast Back Adapter.
diff --git a/sktime/forecasting/statsforecast.py b/sktime/forecasting/statsforecast.py
--- a/sktime/forecasting/statsforecast.py
+++ b/sktime/forecasting/statsforecast.py
@@ -247,45 +247,48 @@ def __init__(
 
         super().__init__()
 
-    def _instantiate_model(self):
-        # import inside method to avoid hard dependency
-        from statsforecast.models import AutoARIMA as _AutoARIMA
-
-        return _AutoARIMA(
-            d=self.d,
-            D=self.D,
-            max_p=self.max_p,
-            max_q=self.max_q,
-            max_P=self.max_P,
-            max_Q=self.max_Q,
-            max_order=self.max_order,
-            max_d=self.max_d,
-            max_D=self.max_D,
-            start_p=self.start_p,
-            start_q=self.start_q,
-            start_P=self.start_P,
-            start_Q=self.start_Q,
-            stationary=self.stationary,
-            seasonal=self.seasonal,
-            ic=self.information_criterion,
-            stepwise=self.stepwise,
-            nmodels=self.n_fits,
-            trace=self.trace,
-            approximation=self.approximation,
-            method=self.method,
-            truncate=self.truncate,
-            test=self.test,
-            test_kwargs=self.offset_test_args,
-            seasonal_test=self.seasonal_test,
-            seasonal_test_kwargs=self.seasonal_test_args,
-            allowdrift=self.trend,
-            allowmean=self.with_intercept,
-            blambda=self.blambda,
-            biasadj=self.biasadj,
-            parallel=self.parallel,
-            num_cores=self.n_jobs,
-            season_length=self.sp,
-        )
+    def _get_statsforecast_class(self):
+        """Get the class of the statsforecast forecaster."""
+        from statsforecast.models import AutoARIMA
+
+        return AutoARIMA
+
+    def _get_statsforecast_params(self):
+        return {
+            "d": self.d,
+            "D": self.D,
+            "max_p": self.max_p,
+            "max_q": self.max_q,
+            "max_P": self.max_P,
+            "max_Q": self.max_Q,
+            "max_order": self.max_order,
+            "max_d": self.max_d,
+            "max_D": self.max_D,
+            "start_p": self.start_p,
+            "start_q": self.start_q,
+            "start_P": self.start_P,
+            "start_Q": self.start_Q,
+            "stationary": self.stationary,
+            "seasonal": self.seasonal,
+            "ic": self.information_criterion,
+            "stepwise": self.stepwise,
+            "nmodels": self.n_fits,
+            "trace": self.trace,
+            "approximation": self.approximation,
+            "method": self.method,
+            "truncate": self.truncate,
+            "test": self.test,
+            "test_kwargs": self.offset_test_args,
+            "seasonal_test": self.seasonal_test,
+            "seasonal_test_kwargs": self.seasonal_test_args,
+            "allowdrift": self.trend,
+            "allowmean": self.with_intercept,
+            "blambda": self.blambda,
+            "biasadj": self.biasadj,
+            "parallel": self.parallel,
+            "num_cores": self.n_jobs,
+            "season_length": self.sp,
+        }
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
@@ -360,15 +363,18 @@ def __init__(
 
         super().__init__()
 
-    def _instantiate_model(self):
-        """Create underlying forecaster instance."""
+    def _get_statsforecast_class(self):
+        """Get the class of the statsforecast forecaster."""
         from statsforecast.models import AutoTheta
 
-        return AutoTheta(
-            season_length=self.season_length,
-            decomposition_type=self.decomposition_type,
-            model=self.model,
-        )
+        return AutoTheta
+
+    def _get_statsforecast_params(self):
+        return {
+            "season_length": self.season_length,
+            "decomposition_type": self.decomposition_type,
+            "model": self.model,
+        }
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
@@ -452,13 +458,18 @@ def __init__(
 
         super().__init__()
 
-    def _instantiate_model(self):
+    def _get_statsforecast_class(self):
         """Create underlying forecaster instance."""
         from statsforecast.models import AutoETS
 
-        return AutoETS(
-            season_length=self.season_length, model=self.model, damped=self.damped
-        )
+        return AutoETS
+
+    def _get_statsforecast_params(self):
+        return {
+            "season_length": self.season_length,
+            "model": self.model,
+            "damped": self.damped,
+        }
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
@@ -527,11 +538,17 @@ def __init__(self, season_length: int = 1, model: str = "Z"):
 
         super().__init__()
 
-    def _instantiate_model(self):
-        """Create underlying forecaster instance."""
+    def _get_statsforecast_class(self):
+        """Get the class of the statsforecast forecaster."""
         from statsforecast.models import AutoCES
 
-        return AutoCES(season_length=self.season_length, model=self.model)
+        return AutoCES
+
+    def _get_statsforecast_params(self):
+        return {
+            "season_length": self.season_length,
+            "model": self.model,
+        }
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):
@@ -633,14 +650,16 @@ def __init__(
                     "forecaster."
                 )
 
-    def _instantiate_model(self):
-        """Create underlying forecaster instance."""
+    def _get_statsforecast_class(self):
         from statsforecast.models import MSTL
 
-        return MSTL(
-            season_length=self.season_length,
-            trend_forecaster=self._trend_forecaster,
-        )
+        return MSTL
+
+    def _get_statsforecast_params(self):
+        return {
+            "season_length": self.season_length,
+            "trend_forecaster": self._trend_forecaster,
+        }
 
     @classmethod
     def get_test_params(cls, parameter_set="default"):


Test patch: