Repository: sktime/sktime

Issue:
[BUG] using custom `extract_path` does not work in `sktime.datasets.load_UCR_UEA_dataset`
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
Error occurs when I use my own download path.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
#my code
x, y = load_UCR_UEA_dataset(name="xxx", return_X_y=True, return_type="numpy3d", extract_path="data/xxx")

#traceback
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    dataloader.load_data("Synthetic Control", "data/Synthetic Control")
  File "dataloader.py", line 4, in load_data
    x, y = load_UCR_UEA_dataset(name=dataset_name, return_X_y=True, return_type="numpy3d", extract_path=stored_path)
  File "/home/anaconda3/envs/pt/lib/python3.9/site-packages/sktime/datasets/_single_problem_loaders.py", line 124, in load_UCR_UEA_dataset
    return _load_dataset(name, split, return_X_y, return_type, extract_path)
  File "/home/anaconda3/envs/pt/lib/python3.9/site-packages/sktime/datasets/_data_io.py", line 249, in _load_dataset
    if name in _list_available_datasets(check_path):
UnboundLocalError: local variable 'check_path' referenced before assignment

#original code
if extract_path is None:
    # default for first check is sktime/datasets/data
    check_path = os.path.join(MODULE, "data")

def _get_data_from(path):
    return _load_provided_dataset(name, split, return_X_y, return_type, path)

# if the dataset exists in check_path = sktime/datasets/data, retrieve it from there
if name in _list_available_datasets(check_path):
    return _get_data_from(check_path)

# now we know the dataset is not in check_path
# so we need to check whether it is already in the download/cache path
# download path is extract_path/local_data, defaults to sktime/datasets/local_data
if extract_path is None:
    extract_path = os.path.join(MODULE, "local_data")

# in either case below, we need to ensure the directory exists
_mkdir_if_not_exist(extract_path)

# search if the dataset is already in the extract path after download
if name in _list_available_datasets(extract_path):
    return _get_data_from(extract_path)

```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
When I trace back to the original code, "check_path" is defined only when the extract_path is not specified. If I use my own extract_path, "check_path" is not defined when checking available datasets.

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:

from sktime import show_versions; show_versions()
-->
the newest version
</details>

<!-- Thanks for contributing! -->


Discussion:
steenrotsman: The - crude - way in which I tested locally was by seeing if `load_UCR_UEA_dataset(name="ArrowHead", extract_path="data/xxx")` raised a `FileNotFoundError` instead of an `UnboundLocalError`.
fkiraly: well, we could add that - that is better than nothing. `except FileNotFoundError as e` and otherwise raise.

Patch:
diff --git a/README.md b/README.md
index bd4cae0c5..89c679a9c 100644
--- a/README.md
+++ b/README.md
@@ -82,6 +82,7 @@ sktime also provides **interfaces to related libraries**, for example [scikit-le
 | **[Time Series Distances/Kernels]** | maturing | [Tutorial](https://github.com/sktime/sktime/blob/main/examples/03_transformers.ipynb) · [API Reference](https://www.sktime.net/en/latest/api_reference/dists_kernels.html) · [Extension Template](https://github.com/sktime/sktime/blob/main/extension_templates/dist_kern_panel.py) |
 | **[Time Series Alignment]** | experimental | [API Reference](https://www.sktime.net/en/latest/api_reference/alignment.html) · [Extension Template](https://github.com/sktime/sktime/blob/main/extension_templates/alignment.py) |
 | **[Annotation]** | experimental | [Extension Template](https://github.com/sktime/sktime/blob/main/extension_templates/annotation.py) |
+| **[Time Series Splitters]** | maturing | [Extension Template](https://github.com/sktime/sktime/blob/main/extension_templates/split.py) | |
 | **[Distributions and simulation]** | experimental |  |
 
 [forecasting]: https://github.com/sktime/sktime/tree/main/sktime/forecasting
@@ -93,6 +94,7 @@ sktime also provides **interfaces to related libraries**, for example [scikit-le
 [time series alignment]: https://github.com/sktime/sktime/tree/main/sktime/alignment
 [transformations]: https://github.com/sktime/sktime/tree/main/sktime/transformations
 [distributions and simulation]: https://github.com/sktime/sktime/tree/main/sktime/proba
+[time series splitters]: https://github.com/sktime/sktime/tree/main/sktime/split
 [parameter fitting]: https://github.com/sktime/sktime/tree/main/sktime/param_est
 
 
diff --git a/docs/source/api_reference/forecasting.rst b/docs/source/api_reference/forecasting.rst
index e7b87652b..246b081a9 100644
--- a/docs/source/api_reference/forecasting.rst
+++ b/docs/source/api_reference/forecasting.rst
@@ -258,6 +258,8 @@ All "ARIMA" and "Auto-ARIMA" models below include SARIMAX capability.
 Auto-ARIMA models
 ~~~~~~~~~~~~~~~~~
 
+.. currentmodule:: sktime.forecasting.arima
+
 .. autosummary::
     :toctree: auto_generated/
     :template: class.rst
diff --git a/docs/source/developer_guide/dependencies.rst b/docs/source/developer_guide/dependencies.rst
index 92f4f3b4a..c08339849 100644
--- a/docs/source/developer_guide/dependencies.rst
+++ b/docs/source/developer_guide/dependencies.rst
@@ -52,11 +52,19 @@ Estimators with a soft dependency need to ensure the following:
    To do this, add a ``# doctest: +SKIP`` to the end of each line in the doctest to skip it entirely.
    See ``forecasting.arima.ARIMA`` as as an example. If concerned that skipping the test will reduce test coverage,
    consider exposing the doctest example as a pytest test function instead, see below how to handle soft dependencies in pytest functions.
-*  Decorate all ``pytest`` tests that import soft dependencies with a ``@pytest.mark.skipif(...)`` conditional on a check to ``_check_soft_dependencies``
-   for your new soft dependency, with ``severity="none"``.  Be sure that all soft dependencies imported for testing are imported within the test function itself,
-   rather than for the whole module!  This decorator will then skip your test, including imports, unless the system has the required packages installed.
-   This prevents crashes for any users running ``check_estimator`` on all estimators, or a full local ``pytest`` run without the required soft dependency.
-   See the tests in ``forecasting.tests.test_pmdarima`` for a concrete example.
+*  Decorate all ``pytest`` tests that import soft dependencies with a ``@pytest.mark.skipif(...)`` conditional on a soft dependency check.
+   If the test is specific to a single estimator or object, use ``run_test_for_class`` from ``sktime.tests.test_switch``
+   to mediate the condition through the class tags.
+   Otherwise, use ``_check_soft_dependencies`` for your new soft dependency, with ``severity="none"``.
+   Be sure that all soft dependencies imported for testing are imported within the test function itself,
+   rather than at root level (at the top) of the module.
+   This decorator will then skip your test, including imports,
+   unless the system has the required packages installed.
+   This prevents crashes for any users running ``check_estimator`` on all estimators,
+   or a full local ``pytest`` run without the required soft dependency.
+   See the tests in ``forecasting.tests.test_pmdarima`` for a concrete example of
+   ``run_test_for_class`` usage to decorate a test. See ``utils.tests.test_plotting``
+   for an example of ``_check_soft_dependencies`` usage.
 
 Adding and maintaining soft dependencies
 ----------------------------------------
diff --git a/extension_templates/split.py b/extension_templates/split.py
new file mode 100644
index 000000000..d795d298f
--- /dev/null
+++ b/extension_templates/split.py
@@ -0,0 +1,269 @@
+"""Extension template for parameter estimators.
+
+Purpose of this implementation template:
+    quick implementation of new estimators following the template
+    NOT a concrete class to import! This is NOT a base class or concrete class!
+    This is to be used as a "fill-in" coding template.
+
+How to use this implementation template to implement a new estimator:
+- make a copy of the template in a suitable location, give it a descriptive name.
+- work through all the "todo" comments below
+- fill in code for mandatory methods, and optionally for optional methods
+- do not write to reserved variables: _tags, _tags_dynamic
+- you can add more private methods, but do not override BaseEstimator's private methods
+    an easy way to be safe is to prefix your methods with "_custom"
+- change docstrings for functions and the file
+- ensure interface compatibility by sktime.utils.estimator_checks.check_estimator
+- once complete: use as a local library, or contribute to sktime via PR
+- more details:
+  https://www.sktime.net/en/stable/developer_guide/add_estimators.html
+
+Mandatory implements:
+    splitting (iloc reference)            - _split(self, y)
+
+Optional implements:
+    splitting (loc reference)             - _split_loc(self, y)
+    get number of splits                  - get_n_splits(self, y)
+
+Testing - required for sktime test framework and check_estimator usage:
+    get default parameters for test instance(s) - get_test_params(cls)
+
+copyright: sktime developers, BSD-3-Clause License (see LICENSE file)
+"""
+# todo: write an informative docstring for the file or module, remove the above
+# todo: add an appropriate copyright notice for your estimator
+#       estimators contributed to sktime should have the copyright notice at the top
+#       estimators of your own do not need to have permissive or BSD-3 copyright
+
+# todo: uncomment the following line, enter authors' GitHub IDs
+# __author__ = [authorGitHubID, anotherAuthorGitHubID]
+
+
+from sktime.split.base import BaseSplitter
+
+# todo: add any necessary imports here
+
+# todo: if any imports are sktime soft dependencies:
+# make sure to fill in the "python_dependencies" tag with the package import name
+
+
+class MySplitter(BaseSplitter):
+    """Custom splitter. todo: write docstring.
+
+    todo: describe your custom splitter here
+
+    Parameters
+    ----------
+    parama : int
+        descriptive explanation of parama
+    paramb : string, optional (default='default')
+        descriptive explanation of paramb
+    paramc : boolean, optional (default= whether paramb is not the default)
+        descriptive explanation of paramc
+    and so on
+    est : sktime.estimator, BaseEstimator descendant
+        descriptive explanation of est
+    est2: another estimator
+        descriptive explanation of est2
+    and so on
+    """
+
+    # todo: fill out estimator tags here
+    #  tags are inherited from parent class if they are not set
+    _tags = {
+        #
+        # behavioural tags
+        # ----------------
+        #
+        # internal support for hierarchical and panel data
+        "split_hierarchical": False,
+        # valid values: True, False
+        # if False, splitter broadcasts over instances for hierarchical data
+        # if True, internal _split must support pd.MultiIndex
+        #
+        # which of _split and _split_loc is called in split_series by default
+        "split_series_uses": "iloc",
+        # valid values: "iloc" or "loc"
+        # determines whether split_series under the hood
+        # calls split ("iloc") or split_loc ("loc"). Setting this can give
+        # performance advantages, e.g., if "loc" is faster to obtain.
+        #
+        # dependency tags: python version and soft dependencies
+        # -----------------------------------------------------
+        #
+        # python version requirement
+        "python_version": None,
+        # valid values: str, PEP 440 valid python version specifiers
+        # raises exception at construction if local python version is incompatible
+        #
+        # soft dependency requirement
+        "python_dependencies": None
+        # valid values: str or list of str
+        # raises exception at construction if modules at strings cannot be imported
+    }
+
+    # todo: add any hyper-parameters and components to constructor
+    def __init__(self, est, parama, est2=None, paramb="default", paramc=None):
+        # estimators should precede parameters
+        #  if estimators have default values, set None and initialize below
+
+        # todo: write any hyper-parameters and components to self
+        self.est = est
+        self.parama = parama
+        self.paramb = paramb
+        self.paramc = paramc
+
+        # leave this as is
+        super().__init__()
+
+        # todo: optional, parameter checking logic (if applicable) should happen here
+        # if writes derived values to self, should *not* overwrite self.parama etc
+        # instead, write to self._parama, self._newparam (starting with _)
+
+        # todo: default estimators should have None arg defaults
+        #  and be initialized here
+        #  do this only with default estimators, not with parameters
+        # if est2 is None:
+        #     self.estimator = MyDefaultEstimator()
+
+        # todo: if tags of estimator depend on component tags, set these here
+        #  only needed if estimator is a composite
+        #  tags set in the constructor apply to the object and override the class
+        #
+        # example 1: conditional setting of a tag
+        # if est.foo == 42:
+        #   self.set_tags(handles-missing-data=True)
+        # example 2: cloning tags from component
+        #   self.clone_tags(est2, ["enforce_index_type", "handles-missing-data"])
+
+    # todo: implement this, mandatory
+    def _split(self, y):
+        """Get iloc references to train/test splits of `y`.
+
+        private _split containing the core logic, called from split
+
+        Parameters
+        ----------
+        y : pd.Index
+            Index of time series to split
+
+        Yields
+        ------
+        train : 1D np.ndarray of dtype int
+            Training window indices, iloc references to training indices in y
+        test : 1D np.ndarray of dtype int
+            Test window indices, iloc references to test indices in y
+        """
+        # todo: implement the core logic of your splitter here
+        # ensure to avoid side effects to self or y
+        #
+        # example:
+        # for train, test in some_logic(y):
+        #     yield train, test
+
+    # todo: consider implementing this, optional
+    # if not implementing, delete this - default is as below and present in base class
+    def _split_loc(self, y):
+        """Get loc references to train/test splits of `y`.
+
+        private _split containing the core logic, called from split_loc
+
+        Default implements using split and y.index to look up the loc indices.
+        Can be overridden for faster implementation.
+
+        Parameters
+        ----------
+        y : pd.Index
+            index of time series to split
+
+        Yields
+        ------
+        train : pd.Index
+            Training window indices, loc references to training indices in y
+        test : pd.Index
+            Test window indices, loc references to test indices in y
+        """
+        for train, test in self.split(y):
+            # default gets loc index from iloc index
+            yield y[train], y[test]
+
+    # todo: consider implementing this, optional
+    # only implement if the result does not depend on y
+    # if not implementing, delete this - default is as below and present in base class
+    def get_n_splits(self, y) -> int:
+        """Return the number of splits.
+
+        Parameters
+        ----------
+        y : pd.Series or pd.Index, optional (default=None)
+            Time series to split
+
+        Returns
+        -------
+        n_splits : int
+            The number of splits.
+        """
+        return len(list(self.split(y)))
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the splitter.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+
+        # todo: set the testing parameters for the object
+        # Testing parameters can be dictionary or list of dictionaries
+        # Testing parameter choice should cover internal cases well.
+        #
+        # this method can, if required, use:
+        #   class properties (e.g., inherited); parent class test case
+        #   imported objects such as estimators from sktime or sklearn
+        # important: all such imports should be *inside get_test_params*, not at the top
+        #            since imports are used only at testing time
+        #
+        # The parameter_set argument is not used for automated, module level tests.
+        #   It can be used in custom, estimator specific tests, for "special" settings.
+        # A parameter dictionary must be returned *for all values* of parameter_set,
+        #   i.e., "parameter_set not available" errors should never be raised.
+        #
+        # A good parameter set should primarily satisfy two criteria,
+        #   1. Chosen set of parameters should have a low testing time,
+        #      ideally in the magnitude of few seconds for the entire test suite.
+        #       This is vital for the cases where default values result in
+        #       "big" models which not only increases test time but also
+        #       run into the risk of test workers crashing.
+        #   2. There should be a minimum two such parameter sets with different
+        #      sets of values to ensure a wide range of code coverage is provided.
+        #
+        # example 1: specify params as dictionary
+        # any number of params can be specified
+        # params = {"est": value0, "parama": value1, "paramb": value2}
+        #
+        # example 2: specify params as list of dictionary
+        # note: Only first dictionary will be used by create_test_instance
+        # params = [{"est": value1, "parama": value2},
+        #           {"est": value3, "parama": value4}]
+        # return params
+        #
+        # example 3: parameter set depending on param_set value
+        #   note: only needed if a separate parameter set is needed in tests
+        # if parameter_set == "special_param_set":
+        #     params = {"est": value1, "parama": value2}
+        #     return params
+        #
+        # # "default" params - always returned except for "special_param_set" value
+        # params = {"est": value3, "parama": value4}
+        # return params
diff --git a/sktime/datasets/_data_io.py b/sktime/datasets/_data_io.py
index cf9549612..e5e11b827 100644
--- a/sktime/datasets/_data_io.py
+++ b/sktime/datasets/_data_io.py
@@ -241,6 +241,8 @@ def _load_dataset(name, split, return_X_y, return_type=None, extract_path=None):
     if extract_path is None:
         # default for first check is sktime/datasets/data
         check_path = os.path.join(MODULE, "data")
+    else:
+        check_path = extract_path
 
     def _get_data_from(path):
         return _load_provided_dataset(name, split, return_X_y, return_type, path)
diff --git a/sktime/performance_metrics/forecasting/_classes.py b/sktime/performance_metrics/forecasting/_classes.py
index 08d331383..e8311bd53 100644
--- a/sktime/performance_metrics/forecasting/_classes.py
+++ b/sktime/performance_metrics/forecasting/_classes.py
@@ -573,6 +573,10 @@ class BaseForecastingErrorMetricFunc(BaseForecastingErrorMetric):
         else:
             func = self.func
 
+        return self._evaluate_func(func=func, y_true=y_true, y_pred=y_pred, **params)
+
+    def _evaluate_func(self, func, y_true, y_pred, **params):
+        """Call func with kwargs subset to func parameters."""
         # import here for now to avoid interaction with getmembers in tests
         # todo: clean up ancient getmembers in test_metrics_classes
         from functools import partial
@@ -584,6 +588,15 @@ class BaseForecastingErrorMetricFunc(BaseForecastingErrorMetric):
             func_params = func_params.intersection(params.keys())
             params = {key: params[key] for key in func_params}
 
+        # deal with sklearn specific parameter constraints
+        # as these are a decorator, they obfuscate python native inspection
+        # via signature, so have to be dealt with separately
+        if hasattr(func, "_skl_parameter_constraints"):
+            constr = func._skl_parameter_constraints
+            if isinstance(constr, dict):
+                constr_params = set(constr.keys()).intersection(params.keys())
+                params = {key: params[key] for key in constr_params}
+
         res = func(y_true=y_true, y_pred=y_pred, **params)
         return res
 
@@ -609,6 +622,16 @@ class _DynamicForecastingErrorMetric(BaseForecastingErrorMetricFunc):
 
         self.set_tags(**{"lower_is_better": lower_is_better})
 
+    def _evaluate(self, y_true, y_pred, **kwargs):
+        """Evaluate the desired metric on given inputs."""
+        # this dict should contain all parameters
+        params = kwargs
+        params.update({"multioutput": self.multioutput, "multilevel": self.multilevel})
+
+        func = self.func
+
+        return self._evaluate_func(func=func, y_true=y_true, y_pred=y_pred, **params)
+
     @classmethod
     def get_test_params(cls, parameter_set="default"):
         """Return testing parameter settings for the estimator.
diff --git a/sktime/regression/deep_learning/__init__.py b/sktime/regression/deep_learning/__init__.py
index cb7052b8a..7e144d0c0 100644
--- a/sktime/regression/deep_learning/__init__.py
+++ b/sktime/regression/deep_learning/__init__.py
@@ -1,8 +1,14 @@
 """Deep learning based regressors."""
 __all__ = [
     "CNNRegressor",
+    "MCDCNNRegressor",
+    "ResNetRegressor",
+    "SimpleRNNRegressor",
     "TapNetRegressor",
 ]
 
 from sktime.regression.deep_learning.cnn import CNNRegressor
+from sktime.regression.deep_learning.mcdcnn import MCDCNNRegressor
+from sktime.regression.deep_learning.resnet import ResNetRegressor
+from sktime.regression.deep_learning.rnn import SimpleRNNRegressor
 from sktime.regression.deep_learning.tapnet import TapNetRegressor
diff --git a/sktime/split/base/_base_splitter.py b/sktime/split/base/_base_splitter.py
index b1325307e..64a3564df 100644
--- a/sktime/split/base/_base_splitter.py
+++ b/sktime/split/base/_base_splitter.py
@@ -113,10 +113,10 @@ class BaseSplitter(BaseObject):
         Parameters
         ----------
         y : pd.Index or time series in sktime compatible time series format,
-                time series can be in any Series, Panel, or Hierarchical mtype format
+            time series can be in any Series, Panel, or Hierarchical mtype format
             Index of time series to split, or time series to split
             If time series, considered as index of equivalent pandas type container:
-                pd.DataFrame, pd.Series, pd-multiindex, or pd_multiindex_hier mtype
+            pd.DataFrame, pd.Series, pd-multiindex, or pd_multiindex_hier mtype
 
         Yields
         ------
@@ -144,8 +144,8 @@ class BaseSplitter(BaseObject):
 
         Parameters
         ----------
-        y : pd.Index or time series in sktime compatible time series format
-            Time series to split, or index of time series to split
+        y : pd.Index
+            Index of time series to split
 
         Yields
         ------
@@ -197,8 +197,10 @@ class BaseSplitter(BaseObject):
         Parameters
         ----------
         y : pd.Index or time series in sktime compatible time series format,
-                time series can be in any Series, Panel, or Hierarchical mtype format
-            Time series to split, or index of time series to split
+            time series can be in any Series, Panel, or Hierarchical mtype format
+            Index of time series to split, or time series to split
+            If time series, considered as index of equivalent pandas type container:
+            pd.DataFrame, pd.Series, pd-multiindex, or pd_multiindex_hier mtype
 
         Yields
         ------
@@ -240,10 +242,11 @@ class BaseSplitter(BaseObject):
 
         Parameters
         ----------
-        y : time series in sktime compatible time series format,
-                time series can be in any Series, Panel, or Hierarchical mtype format
-            e.g., pd.Series, pd.DataFrame, np.ndarray
-            Time series to split, or index of time series to split
+        y : pd.Index or time series in sktime compatible time series format,
+            time series can be in any Series, Panel, or Hierarchical mtype format
+            Index of time series to split, or time series to split
+            If time series, considered as index of equivalent pandas type container:
+            pd.DataFrame, pd.Series, pd-multiindex, or pd_multiindex_hier mtype
 
         Yields
         ------
@@ -387,8 +390,11 @@ class BaseSplitter(BaseObject):
 
         Parameters
         ----------
-        y : pd.Series or pd.Index, optional (default=None)
-            Time series to split
+        y : pd.Index or time series in sktime compatible time series format,
+            time series can be in any Series, Panel, or Hierarchical mtype format
+            Index of time series to split, or time series to split
+            If time series, considered as index of equivalent pandas type container:
+            pd.DataFrame, pd.Series, pd-multiindex, or pd_multiindex_hier mtype
 
         Returns
         -------


Test patch:
diff --git a/sktime/datasets/tests/test_data_io.py b/sktime/datasets/tests/test_data_io.py
index 196526c26..9e6126268 100644
--- a/sktime/datasets/tests/test_data_io.py
+++ b/sktime/datasets/tests/test_data_io.py
@@ -84,6 +84,16 @@ def test_load_UCR_UEA_dataset():
     assert X.shape == (42, 1) and y.shape == (42,)
 
 
+def test_load_UCR_UEA_local():
+    """Tests load_UCR_UEA_dataset looks for local file if extract_path is set.
+
+    A FileNotFoundError indicates the function looked for the file and everything up
+    to that point went fine.
+    """
+    with pytest.raises(FileNotFoundError):
+        load_UCR_UEA_dataset(name="UnitTest", extract_path=" ")
+
+
 _CHECKS = {
     "uschange": {
         "columns": ["Income", "Production", "Savings", "Unemployment"],
diff --git a/sktime/performance_metrics/forecasting/tests/test_metrics.py b/sktime/performance_metrics/forecasting/tests/test_metrics.py
index 1b2d07c4f..12f9709bf 100644
--- a/sktime/performance_metrics/forecasting/tests/test_metrics.py
+++ b/sktime/performance_metrics/forecasting/tests/test_metrics.py
@@ -129,3 +129,18 @@ def test_make_scorer():
     scorer = make_forecasting_scorer(rmsle, name="RMSLE")
 
     scorer.evaluate(pd.Series([1, 2, 3]), pd.Series([1, 2, 4]))
+
+
+def test_make_scorer_sklearn():
+    """Test make_forecasting_scorer and the failure case in #5715.
+
+    Naive adaptation fails on newer sklearn versions due to
+    decoration with sklearn's custom input constraint wrapper.
+    """
+    from sklearn.metrics import mean_absolute_error
+
+    from sktime.performance_metrics.forecasting import make_forecasting_scorer
+
+    scorer = make_forecasting_scorer(mean_absolute_error, name="RMSLE")
+
+    scorer.evaluate(pd.Series([1, 2, 3]), pd.Series([1, 2, 4]))